Kubernetes APIs Under the Hood @pwittrock
Who Am I? Phillip Wittrock (@pwittrock) Software Engineer at Google working on GKE and OSS Kubernetes My mission is to make using Kubernetes simple and enjoyable You might have come across me through… ● Kubectl ● Kubebuilder ● Kubernetes Steering Committee @pwittrock
Kubernetes Refresher ● Nodes are machines in a cluster that run Containers in Pods ● Pods are created and managed by higher level abstractions such as ReplicaSets ● ReplicaSets managed by higher level abstractions such as Deployments ● Deployments (and all other user owned objects) defined in files and created / updated with `kubectl apply` ● APIs (deployments, replicasets, pods, nodes) == Resource Types and Objects == Resources @pwittrock
Kubernetes APIs Are... Declarative , Asynchronous, Level-Triggered , Observable, Discoverable, Versioned , Access Controlled, Extensible, ... @pwittrock
Kubernetes APIs are… Declarative apiVersion: apps/v1 Configuration for a Deployment that kind: Deployment metadata: manages 3 Pods each running an nginx name: nginx-deployment container labels: {app: nginx} spec: Deployment resource is replicas: 3 selector: declared in a file matchLabels: {app: nginx} template: metadata: labels: {app: nginx} Create or update the resource in the cluster by run spec: kubectl apply on a file or directory containers: [ {name: nginx, kubectl apply -f deploy.yaml image: 'nginx:1.7.9'}] @pwittrock
Create Deployment Example apiVersion: apps/v1 kubectl apply -f kind: Deployment deploy.yaml metadata: deploy.yaml name: nginx-deployment labels: {app: nginx} spec: 1. Discover EndPoints replicas: 3 selector: 2. HTTP POST matchLabels: {app: nginx} template: 3. ??? metadata: labels: {app: nginx} spec: apiserver containers: [ Pod Pod Nodes {name: nginx, Pod image: 'nginx:1.7.9'}] @pwittrock
Lifecycle - Resources, Controllers and Webhooks ● APIs declared as Resources - provide storage and endpoints ● APIs actuated by Controllers - execute the business logic ● APIs admitted by Webhooks - defaulting, validation, conversion @pwittrock
Kubernetes APIs are... Asynchronous & Observable Foo Resources stored in etcd by the apiserver API endpoints Resource: Foo objects (CRUD storage) Foo objects Stores Foo objects Foo Resources Stuff Asynchronous watch notification on object create / update / delete Controller: Foo Controller Does Stuff Loose coupling between Controller and API endpoints: Storage doesn’t know about @pwittrock Controllers
Kubernetes APIs are... Level Triggered Object Controller default/baz default / Reconcile Reconcile baz Queue Function Watch events: Create Update Update Tips: ● Internal cleanup with Reconcile on Batch events together ownerReferences namespace/name ● External cleanup with into single Reconcile call only, not the event finalizers @pwittrock
Controller Workflow ● apiserver streams Watch Event to Controller Controller ● Controller Reads Object + Related Reconcile: Objects (e.g. Deployment + Read Cluster Watch Event ReplicaSets) State Reconcile: ● Controller Creates new owned Update Cluster objects, updates owned objects, Objects updates object status apiserver @pwittrock
Kubectl Apply: Create Deployment Deployment Controller watch evt create ReplicaSet apply create deploy. ReplicaSet apiserver yaml Controller create Pod(s) update Pod(s) Node(s) update Pod(s) (Pod) Scheduler @pwittrock
Kubectl Apply: Update Deployment Deployment Controller watch evt Scale up new ReplicaSet / Scale down old ReplicaSet apply patch deploy. ReplicaSet apiserver yaml Controller delete Pod(s) create Pod(s) update Pod(s) Node(s) update Pod(s) (Pod) Scheduler @pwittrock
Resources apiVersion: v1 kind: Pod metadata metadata: name: my-app Resource Types declare APIs ● namespace: default … ● Resources provide storage for spec: objects containers: - args: [sh] ● Standardized schema structure image: gcr.io/some-project/udptest spec Discoverable API endpoints and ● imagePullPolicy: Always name: client schema … dnsPolicy: ClusterFirst ● “Just work” with declarative … tooling - e.g. kubectl apply status: status podIP: 10.8.3.11 … @pwittrock
TypeMeta apiVersion: apps /v1 ● Kind (Deployment) kind: Deployment metadata: ○ Name of the API (e.g. Deployment) name: nginx-deployment ● Group (apps) labels: {app: nginx} spec: ○ Like a package in go, java, etc (e.g. apps ) replicas: 3 ● Version (v1) selector: ... ○ Ensures backwards compatibility of: Defaulted template: Fields & Schema ... @pwittrock
ObjectMeta apiVersion: apps/v1 Name and * Namespace uniquely identify an ● kind: Deployment metadata: object for a given Resource name: nginx-deployment ● Annotations are arbitrary key-value pairs namespace: default labels: {app: nginx} that cannot be queried spec: Labels are key-value pairs that may be ● replicas: 3 selector: queried (selected) ... template: ... @pwittrock
Spec and Status apiVersion: apps/v1 kind: Deployment metadata: ● Spec name: nginx-deployment labels: {app: nginx} Object Desired State (e.g. how ○ spec: many replicas to run, template for replicas: 3 selector: Pods, etc) matchLabels: {...} ● Status (not shown) template: metadata: ○ Defines the observed state for an labels: {...} spec: object (e.g. how many replicas are ... running) @pwittrock
Deployment name: nginx Resource Wiring labels :run=nginx selector :run=nginx ReplicaSet Labels/Selectors locate objects ● name: nginx-65899c769f ● Label - generated objects labels :run=nginx selector :run=nginx ● Selector - find labeled objects owner : Deployment nginx ● OwnerReference on generated objects Pod name: nginx-65899c769f-6slpx labels :run=nginx Tip: Objects with owner references are Pod automatically garbage collected when selector :run=nginx name: nginx-65899c769f-fbgcv all of their owners have been deleted labels : run=nginx owner: ReplicaSet nginx-65899c769f @pwittrock
apiVersion: apps/v1 kind: Deployment Synchronous Defaulting metadata: name: nginx-deployment and Validation labels: {app: nginx} spec: # server defaults this value Unspecified optional fields may be ● # to 1 if unset replicas: 1 defaulted by the apiserver before the object is stored # make sure these match the # template labels Simple Schema validation performed ● selector: through OpenAPI matchLabels: {...} template: ● Complex validation performed by the metadata: apiserver before the object is stored labels: {...} @pwittrock
Kubernetes APIs are… Extensible Mutating Webhook + CustomResourceDefinition Deployment Service + (CRD) (or Pod) Deployment (or Pod) Foo Foo Foo Admission Resource Controller Actuation: Defaulting, Validation, Version Storage, Schema, Display, level-triggered, Conversion etc asynchronous Tip: build your own APIs in go using @pwittrock kubebuilder
apiVersion: v1 Updating Resources kind: Service metadata: name: nginx Gotchas labels: {app: nginx} spec: selector: app: nginx ● Spec has shared ownership across multiple parts ports: - protocol: TCP of the system port: 80 ● Controllers or other actors may update the Spec # not set by owner! with new fields which must be retained across # don’t overwrite! clusterIp: 10.0.171.239 updates to the object # not set by owner! ● Both an issue for Controllers and for users # don’t overwrite! managing Resources using config loadBalancerIp: 78.1124.19 type: LoadBalancer ● Need to either read-update-write or apply @pwittrock
Kind: Pod Observing Objects - ... spec: readinessGates: Status and Events - conditionType: "www.example.com/feature-1" status: conditions : ● Actuation performed asynchronously - type: Ready Status published to users, tools and other ● status: "False" lastProbeTime: null controllers through Status field lastTransitionTime: ... Conditions: key/value pairs that communicate ● - type: "www.example.com/feature-1" status: "False" status (current) to other tools (part of Status lastProbeTime: null field) lastTransitionTime: ... ● Events: separate objects that communicate containerStatuses: - containerID: docker://abcd… past events to users ready: true @pwittrock
Converting API versions ● Different versions of an API may have different representations default/baz Foo Object ○ Changing default values and field names / field types requires a new version ● All versions of the same API are logically equivalent The same object may be read or written in any ● version -- the underlying object remains the same -- v1beta2 v1beta1 v1 but the endpoints are different. API Endpoints @pwittrock
Classes of APIs Composites Decorators Operators Autoscalers, Resource Spark, Airflow Tuners Cloud Native Abstractions Tekton, Knative @pwittrock
Kubernetes APIs Are... Declarative , Asynchronous, Level-Triggered , Observable, Discoverable, Versioned, Access Controlled, Extensible, ... @pwittrock
Kubebuilder Workshop https://github.com/DirectXMan12/kubebuilder-workshops/tr ee/software-architecture-2019 @pwittrock
Rate today ’s session Session page on conference website O’Reilly Events App
Questions? @pwittrock
Recommend
More recommend
