Blog

Solinea Customer Case on Kubernetes.io: How We Architected and Run Kubernetes on OpenStack at Scale at Yahoo! JAPAN

 

Originally posted on kubernetes.io

Editor’s note: today’s post is by the Infrastructure Engineering team at Yahoo! JAPAN, talking about how they run OpenStack on Kubernetes. This post has been translated and edited for context with permission — originally published on the Yahoo! JAPAN engineering blog


Intro
This post outlines how Yahoo! JAPAN, with help from Google and Solinea, built an automation tool chain for “one-click” code deployment to Kubernetes running on OpenStack. 

We’ll also cover the basic security, networking, storage, and performance needs to ensure production readiness. 

Finally, we will discuss the ecosystem tools used to build the CI/CD pipeline, Kubernetes as a deployment platform on VMs/bare metal, and an overview of Kubernetes architecture to help you architect and deploy your own clusters. 

Preface
Since our company started using OpenStack in 2012, our internal environment has changed quickly. Our initial goal of virtualizing hardware was achieved with OpenStack. However, due to the progress of cloud and container technology, we needed the capability to launch services on various platforms. This post will provide our example of taking applications running on OpenStack and porting them to Kubernetes.

Coding Lifecycle
The goal of this project is to create images for all required platforms from one application code, and deploy those images onto each platform. For example, when code is changed at the code registry, bare metal images, Docker containers and VM images are created by CI (continuous integration) tools, pushed into our image registry, then deployed to each infrastructure platform.

website-yja-kubernetes_ci-pipeline

We use following products in our CICD pipeline:

Function

Product

Code registry

GitHub Enterprise

CI tools

Jenkins

Image registry

Artifactory

Bug tracking system

JIRA

deploying Bare metal platform

OpenStack Ironic

deploying VM platform

OpenStack

deploying container platform

Kubernetes

Image Creation. Each image creation workflow is shown in the next diagram.

VM Image Creation:

website-yja-kubernetes_vm-image-creation

 


Bare Metal Image Creation:

website-yja-kubernetes_bare-metal-image-creation
push code to GitHub
hook to Jenkins master
Launch job at Jenkins slave 
checkout Packer repository
Run Service Job
Download base bare metal image by build script
build script execute diskimage-builder with Packer to create bare metal image
Upload new created image to Glance
Upload the image to Artifactory

Container Image Creation:

website-yja-kubernetes_container-image-creation

push code to GitHub
hook to Jenkins master
Launch job at Jenkins slave 
checkout Dockerfile repository
Run Service Job
Download base docker image from Artifactory
If no docker image found at Artifactory, download from Docker Hub
Execute docker build and create image 
Upload the image to Artifactory

Platform Architecture

Let’s focus on the container workflow to walk through how we use Kubernetes as a deployment platform. This platform architecture is as below.
website-yja-kubernetes_platform-architecture

Function

Product

Infrastructure Services

OpenStack

Container Host

CentOS

Container Cluster Manager

Kubernetes

Container Networking

Project Calico

Container Engine

Docker

Container Registry

Artifactory

Service Registry

etcd

Source Code Management

GitHub Enterprise

CI tool

Jenkins

Infrastructure Provisioning

Terraform

Logging

Fluentd, Elasticsearch, Kibana

Metrics

Heapster, Influxdb, Grafana

Service Monitoring

Prometheus

We use CentOS for Container Host (OpenStack instances) and install Docker, Kubernetes, Calico, etcd and so on. Of course, it is possible to run various container applications on Kubernetes. In fact, we run OpenStack as one of those applications. That’s right, OpenStack on Kubernetes on OpenStack. We currently have more than 30 OpenStack clusters, that quickly become hard to manage and operate. As such, we wanted to create a simple, base OpenStack cluster to provide the basic functionality needed for Kubernetes and make our OpenStack environment easier to manage.

Kubernetes Architecture

Let me explain Kubernetes architecture in some more detail. The architecture diagram is below.

website-yja-kubernetes_kubernetes-architecture

Product

Description

OpenStack Keystone

Kubernetes Authentication and Authorization

OpenStack Cinder

External volume used from Pod (grouping of multiple containers)

kube-apiserver

Configure and validate objects like Pod or Services (definition of access to services in container) through REST API

kube-scheduler

Allocate Pods to each node

kube-controller-manager

Execute Status management, manage replication controller

kubelet

Run on each node as agent and manage Pod

calico

Enable inter-Pod connection using BGP

kube-proxy

Configure iptable NAT tables to configure IP and load balance (ClusterIP)

etcd

Distribute KVS to store Kubernetes and Calico information

etcd-proxy

Run on each node and transfer client request to etcd clusters

Tenant Isolation

To enable multi-tenant usage like OpenStack, we utilize OpenStack Keystone for authentication and authorization.

Authentication

With a Kubernetes plugin, OpenStack Keystone can be used for Authentication. By Adding authURL of Keystone at startup Kubernetes API server, we can use OpenStack OS_USERNAME and OS_PASSWORD for Authentication.

Authorization

We currently use the ABAC (Attribute-Based Access Control) mode of Kubernetes Authorization. We worked with a consulting company, Solinea, who helped create a utility to convert OpenStack Keystone user and tenant information to Kubernetes JSON policy file that maps Kubernetes ABAC user and namespace information to OpenStack tenants. We then specify that policy file when launching Kubernetes API Server. This utility also creates namespaces from tenant information. These configurations enable Kubernetes to authenticate with OpenStack Keystone and operate in authorized namespaces.

Volumes and Data Persistence

Kubernetes provides “Persistent Volumes” subsystem which works as persistent storage for Pods. “Persistent Volumes” is capable to support cloud-provider storage, it is possible to utilize OpenStack cinder-volume by using OpenStack as cloud provider.

Networking

Flannel and various networking exists as networking model for Kubernetes, we used Project Calico for this project. Yahoo! JAPAN recommends to build data center with pure L3 networking like redistribute ARP validation or IP CLOS networking, Project Calico matches this direction. When we apply overlay model like Flannel, we cannot access to Pod IP from outside of Kubernetes clusters. But Project Calico makes it possible. We also use Project Calico for Load Balancing we describe later.

In Project Calico, broadcast production IP by BGP working on BIRD containers (OSS routing software) launched on each nodes of Kubernetes. By default, it broadcast in cluster only. By setting peering routers outside of clusters, it makes it possible to access a Pod from outside of the clusters.

website-yja-kubernetes_networking

External Service Load Balancing

There are multiple choices of external service load balancers (access to services from outside of clusters) for Kubernetes such as NodePort, LoadBalancer and Ingress. We could not find solution which exactly matches our requirements. However, we found a solution that almost matches our requirements by broadcasting Cluster IP used for Internal Service Load Balancing (access to services from inside of clusters) with Project Calico BGP which enable External Load Balancing at Layer 4 from outside of clusters.

website-yja-kubernetes_loadbalancing

Service Discovery

Service Discovery is possible at Kubernetes by using SkyDNS addon. This is provided as cluster internal service, it is accessible in cluster like ClusterIP. By broadcasting ClusterIP by BGP, name resolution works from outside of clusters. By combination of Image creation workflow and Kubernetes, we built the following tool chain which makes it easy from code push to deployment.

website-yja-kubernetes_workflow_k8s_all

Summary

In summary, by combining Image creation workflows and Kubernetes, Yahoo! JAPAN, with help from Google and Solinea, successfully built an automated tool chain which makes it easy to go from code push to deployment, while taking multi-tenancy, authn/authz, storage, networking, service discovery and other necessary factors for production deployment. We hope you found the discussion of ecosystem tools used to build the CI/CD pipeline, Kubernetes as a deployment platform on VMs/bare-metal, and the overview of Kubernetes architecture to help you architect and deploy your own clusters. Thank you to all of the people who helped with this project. –Norifumi Matsuya, Hirotaka Ichikawa, Masaharu Miyamoto and Yuta Kinoshita. This post has been translated and edited for context with permission — originally published on the Yahoo! JAPAN engineer blog where this was one in a series of posts focused on Kubernetes.

Proof for Tech Pros:
Join our Bi-weekly Open Infrastructure Demo
solinea.com/next
Proof For Managers:
Get to Know Solinea Services:
solinea.com/services
Proof For Executives:
Arrange a Free 20-min Open Infrastructure Strategy Session and Solutions Overview
solinea.com/contact