How to build scalable, reliable and stable Kubernetes cluster atop OpenStack Bo Wang bo.wang@easystack.cn HouMing Wang houming.wang@easystack.cn
Cluster resources management Cluster data persistence Contents Integrate kuryr-kubernetes as CNI plugin Integrate manila as storage provisioner
Architecture of Kubernetes Cluster master nodes slave nodes kube-proxy apiserver end-user pods kube-proxy flanneld etcd scheduler kubelet controller manager flanneld docker kubelet containers docker system daemons
Cluster Resource Management – why Pods and system daemons compete for resources Pods can consume all the available capacity on a node by default Resource starvation What ever happened in our environment: • kube-proxy, prometheus were evicted • dockerd does not response in time • etcd cluster crash System daemons crash and pods evicting
Cluster Resource Management – how categories components solution ref kubernetes system daemons kubelet ， docker configure [1] –kube-reserved etcd ， flanneld ， apiserver OS system daemons configure [1] --system-reserved eviction thresholds kubelet configure [1] --eviction-hard kube-system pods kube-scheduler, configure [2] kube-controller, kube-proxy, guaranteed QoS class prometheus, fluentd end-user pods configure [2] needed QoS class [1] Reserve Compute Resources for System Daemons: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ [2] Configure Quality of Service for Pods: https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/
Cluster Resource Management – example Node Capacity 32Gi of memory, 16 CPUs and 100Gi of Storage kube-reserved --kube-reserved=cpu=1,memory=2Gi,ephemeral-storage=1Gi system-reserved --system-reserved=cpu=500m,memory=1Gi,ephemeral-storage=1Gi eviction-threshold --eviction-hard=memory.available<500Mi,nodefs.available<10% available for pods 14.5 CPUs, 28.5Gi memory, 98Gi local storage pod eviction occurs in the following order: • BestEffort • Burstable • Guaranteed
Cluster resources management Cluster data persistence Contents Integrate kuryr-kubernetes as CNI plugin Integrate manila as storage provisioner
Cluster Data Persistence All cluster data stored in local storage of VM instance. VM destroyed, data lost. move essential data into persistent volumes separately as needed. etcd data kubernetes object resources, Done in upstream container network configurations [1] https://bugs.launchpad.net/magnum/+bug/1697655 [2] https://review.openstack.org/#/c/473789/ monitor data nodes info, configure volumes for prometheus pods pods info logging data kubernetes daemons log, configure volumes for elasticsearch pods system daemons logs, container logs
Etcd Cluster Independent Deployment “Fast disks are the most critical factor for etcd deployment performance and stability. etcd is very sensitive to disk write latency.” “Few etcd deployments require a lot of CPU capacity.” [1] slave nodes master nodes LB apiserver flanneld flanneld etcd nodes etcd high performance volumes [1] https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md
Cluster resources management Cluster data persistence Contents Integrate kuryr-kubernetes as CNI plugin Integrate manila as storage provisioner
Integrate kuryr-kubernetes as CNI plugin eth0 eth1 eth0 eth1 No IP No IP 10.0.0.6 10.0.0.5 kuryr bridge kuryr bridge k8s api server kubelet tap-xxx tap-xxx tap-yyy tap-yyy eth0 eth0 kubele t eth0 eth0 kuryr-cni Pod1 Pod2 Pod1 Pod2 10.0.0.9 10.0.0.7 10.0.0.10 10.0.0.8 kuryr kuryr-cni controller iptables iptables kube-proxy kube-proxy master node slave node Neutron Server
Integrate kuryr-kubernetes as CNI plugin difference with upstream reasons ref kuryr only for ip allocation 1. iptables has better performance than neutron lbaasv2 kube-proxy for service --> pod 2. kuryr does not support k8s services in following kinds: [1] [2] LoadBalancer; NodePort; Endpoint-less; Specify cluster ip add implementation of portmapping into cni plugin should support hostPort [3] kuryr-cni network topology of pods and vms with kube-proxy, macvlan do not go through the host system [4] iptables trunk port is not enabled in our product stop watching k8s events in theory, watching events should have better performance kubelet --> kuryr-cni --> kuryr-controller but in our test, kuryr-cni came into time out errors against concurrent pods creating. simplify the process to sequential call [1] https://bugs.launchpad.net/kuryr-kubernetes/+bug/1684118 [2] https://bugs.launchpad.net/kuryr-kubernetes/+bug/1697942 [3] https://github.com/kubernetes-incubator/bootkube/issues/662 [4] https://github.com/kubernetes/kubernetes/issues/53089
Cluster resources management Cluster data persistence Contents Integrate kuryr-kubernetes as CNI plugin Integrate manila as storage provisioner
Integrate manila as storage provisioner Deployments/RC with one replica Deployments/RC with multi-replicas Pod Pod1 Pod2 Pod3 ReadWriteMany ReadWriteOnce Cinder NFS persistent volume persistent volume Block Storage Shared File System Cinder Manila
Integrate manila as storage provisioner Manually leveraging manila to provide NFS PV for k8s pods nfs-pv.yaml Manila k8s get share Create PV with export location share location Create share Create PVC match PV nfs-pvc.yaml Create share network Create Pods mount PVC Multiple pods read/write share
Integrate manila as storage provisioner Add manila as an external storage provisioner[1][2] to provide PV dynamically for Pods manila storage class: K8s cluster kubeconfig cloudconfig watch easystack manila k8s PVC events provisioner pods apiserver manila pvc: [3] openstack manila [1] https://kubernetes.io/docs/concepts/storage/persistent-volumes/ [2] https://github.com/kubernetes-incubator/external-storage/ [3] https://github.com/kubernetes-incubator/external-storage/pull/429
Magnum Q: Cloud these happen in magnum? A: Yes, we did all these work based on internal magnum. Related BP in magnum launchpad: etcd cluster independent deployment: https://blueprints.launchpad.net/magnum/+spec/deploy-etcd-cluster-independently • integrate kuryr-kubernetes with magnum: https://blueprints.launchpad.net/magnum/+spec/integrate-kuryr-kubernetes • integrate manila with magnum: https://blueprints.launchpad.net/magnum/+spec/magnum-manila-integration •
Recommend
More recommend
