A Kubernetes Operator for etcd jetstack.io Open Source Tools - PowerPoint PPT Presentation

Presented by James Laverack, Solutions Engineer A Kubernetes Operator for etcd jetstack.io

Open Source Tools Subscription Consulting Training Enhancing the Kubernetes Reference architecture, online Consulting and engineering to Learn and engage directly experience with open source training and SLA support 24x7 for make the most of Kubernetes and alongside our team, with courses tooling. Includes provisioning, your production Kubernetes move you to production quickly for all stages of your Kubernetes stateful services and security deployment journey related projects jetstack.io

“We need to run etcd in Kubernetes.” jetstack.io

jetstack.io

etcd-0.example.com LEADER client-app.example.com etcd-1.example.com CLIENT APPLICATION etcd.example.com etcd-2.example.com jetstack.io

“What about a stateful set?” jetstack.io

jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

etcd-0.example.com LEADER etcd-1.example.com etcd-0.example.com etcd-1.example.com etcd-2.example.com etcd-2.example.com jetstack.io

etcd-0.example.com LEADER etcd-1.example.com etcd-0.example.com etcd-1.example.com etcd-2.example.com etcd-3.example.com etcd-2.example.com jetstack.io

etcd-0.example.com LEADER etcd-3.example.com etcd-1.example.com etcd-2.example.com jetstack.io

etcd-0.example.com LEADER etcd-1.example.com etcd-0.example.com etcd-1.example.com etcd-2.example.com etcd-2.example.com jetstack.io

etcd-0.example.com LEADER etcd-1.example.com etcd-0.example.com etcd-1.example.com jetstack.io

jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

Pre-stop hook: Init container: err := removePeerFromEtcd(peer) if !bootstrap and firstLaunch { if err != nil { err := addPeerToEtcd(peer) // Uhh… if err != nil { // Shut down anyway? // Comms failure? } fail() } markAsLaunched() } jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

Pre-stop hook: Init container: err := removePeerFromEtcd(peer) if !bootstrap and firstLaunch { if err != nil { err := addPeerToEtcd(peer) // Uhh… if err != nil { // Shut down anyway? // Comms failure? } Only if we’re really fail() removing a peer, not if } we just move the Pod... markAsLaunched() } jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

Pre-stop hook: Init container: err := removePeerFromEtcd(peer) if !bootstrap and firstLaunch { if err != nil { err := addPeerToEtcd(peer) // Uhh… if err != nil { // Shut down anyway? // Comms failure? } Only if we’re really fail() removing a peer, not if } we just move the Pod... markAsLaunched() } jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

Pre-stop hook: Init container: err := removePeerFromEtcd(peer) if !bootstrap and firstLaunch { if err != nil { err := addPeerToEtcd(peer) // Uhh… if err != nil { // Shut down anyway? // Comms failure? } Only if we’re really fail() removing a peer, not if } we just move the Pod... markAsLaunched() If we are bootstrapping, how } do we determine the domain names of all the other peers? Detect bootstrap Query the STS to check the by trying to dial the size? What if someone scales cluster? What if it’s it while we’re bootstrapping? just a comms failure? Means the etcd pod needs How do we recover permissions to list stateful from that? sets... Detect firstLaunch by presence of file on the filesystem? What if we crash halfway through adding ourselves to If we scale to zero do we delete all the etcd? Can we retry safely? PVCs and all the data? jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

We need an Operator. jetstack.io

“Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components.” jetstack.io kubernetes.io/docs/concepts/extend-kubernetes/operator

“The Operator pattern aims to capture the key aim of a human operator […] Human operators […] have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems.” jetstack.io kubernetes.io/docs/concepts/extend-kubernetes/operator

An Operator encodes knowledge. jetstack.io

cert-manager.io strimzi.io And many more jetstack.io

jetstack.io coreos.com/blog/introducing-operators

“How do you actually build an Operator?” jetstack.io

EtcdCluster jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

$ kubectl api-resources NAME SHORTNAMES APIGROUP NAMESPACED KIND deployments deploy apps true Deployment replicasets rs apps true ReplicaSet statefulsets sts apps true StatefulSet pods po true Pod etcdclusters etcd.improbable.io true EtcdCluster < many lines omitted > jetstack.io

eco-system jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

- apiGroups: - apiGroups: - apiGroups: - etcd.improbable.io - apps - "" resources: resources: resources: - etcdclusters - replicasets - services verbs: verbs: verbs: - get - create - create - list - get - get - watch - list - list - watch - watch jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

jetstack.io

jetstack.io github.com/kubernetes-sigs/kubebuilder

Operator logic jetstack.io

Replica Set, Persistent Volume Claim, EtcdCluster Service, etc. 1 2 etcd-cluster-operator jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

Replica Set, 2 Persistent Volume Claim, EtcdCluster Service, etc. 1 3 etcd-cluster-operator jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

EtcdCluster EtcdPeer jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

1 3 EtcdCluster 2 EtcdPeer, Service 5 EtcdPeer Replica Set, Persistent 4 Volume Claim jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

Design considerations jetstack.io

Be level-triggered. jetstack.io

Do one thing at a time. jetstack.io

The cache might lie to you. jetstack.io

Deploying an etcd Cluster jetstack.io

apiVersion: etcd.improbable.io/v1alpha1 kind: EtcdCluster metadata: name: my-etcd namespace: foo spec: replicas: 3 jetstack.io

foo CLUSTER my-etcd PEER PEER PEER my-etcd my-etcd-0 my-etcd-1 my-etcd-2 jetstack.io Custom Resource Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

¯\_( ツ )_/¯ 1 3 EtcdCluster 2 EtcdPeer, Service 5 EtcdPeer Replica Set, Persistent 4 Volume Claim jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

apiVersion: etcd.improbable.io/v1alpha1 kind: EtcdPeer metadata: name: my-etcd-0 Namespace: foo spec: clusterName: my-etcd bootstrap: initialClusterState: New static: initialCluster: - name: my-etcd-0 host: my-etcd-0.my-etcd.foo.svc - name: my-etcd-1 host: my-etcd-1.my-etcd.foo.svc - name: my-etcd-2 host: my-etcd-2.my-etcd.foo.svc jetstack.io

foo my-etcd-0.my-etcd.foo.svc PEER CLUSTER my-etcd-0 my-etcd-0 my-etcd my-etcd my-etcd-0 my-etcd.foo.svc jetstack.io Custom Resource Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

my-etcd-0.my-etcd.foo.svc my-etcd-0.my-etcd.foo.svc my-etcd-1.my-etcd.foo.svc my-etcd-2.my-etcd.foo.svc my-etcd-1.my-etcd.foo.svc my-etcd-2.my-etcd.foo.svc jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

Scale up jetstack.io

apiVersion: etcd.improbable.io/v1alpha1 kind: EtcdCluster metadata: name: my-etcd spec: replicas: 3 5 jetstack.io

$ kubectl scale etcdcluster my-etcd --replicas 5 jetstack.io

my-etcd-0.my-etcd.foo.svc my-etcd-0.my-etcd.foo.svc my-etcd-1.my-etcd.foo.svc my-etcd-2.my-etcd.foo.svc my-etcd-3.my-etcd.foo.svc my-etcd-1.my-etcd.foo.svc my-etcd-2.my-etcd.foo.svc jetstack.io Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

foo CLUSTER my-etcd PEER PEER PEER PEER my-etcd my-etcd-0 my-etcd-1 my-etcd-2 my-etcd-3 jetstack.io Custom Resource Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

apiVersion: etcd.improbable.io/v1alpha1 kind: EtcdPeer metadata: name: my-etcd-3 namespace: foo spec: clusterName: my-etcd bootstrap: initialClusterState: Existing static: initialCluster: - name: my-etcd-0 host: my-etcd-0.my-etcd.foo.svc - name: my-etcd-1 host: my-etcd-1.my-etcd.foo.svc - name: my-etcd-2 host: my-etcd-2.my-etcd.foo.svc - name: my-etcd-3 host: my-etcd-3.my-etcd.foo.svc jetstack.io

foo CLUSTER my-etcd PEER PEER PEER PEER my-etcd-0 my-etcd-1 my-etcd-2 my-etcd-3 my-etcd-0 my-etcd-1 my-etcd-2 my-etcd-3 my-etcd my-etcd-0 my-etcd-1 my-etcd-2 my-etcd-3 jetstack.io Custom Resource Kubernetes Icons Set — The Linux Foundation — CC-BY-4.0

Scale down jetstack.io

$ kubectl scale etcdcluster my-etcd --replicas 1 jetstack.io