DevOps: Where is My PodPod
Hello! I am smalltown MaiCoin Site Reliability Engineer Taipei HashiCorp UG Organizer AWS UG Taiwan Staff
Pets vs Cattle GUI Driven API Driven ● ● Ticket Based Self Service ● ● Hand Crafted Automated ● ● Reserved On Demand ● ● Scale-Up Scale-Out ● ● Smart Hardware Smart Apps ● ● Proprietary Open Source ● ● Waterfall Ops Agile DevOps ● ● ... ... ● ●
Kubernetes = Cattle Pattern
After Using Kubernetes?
Livestock Industry Requires Expertise System Range of Species Animal Health Breeding Feeding Product
The Same Thing Happened in K8S App Not Redundancy Pod is Pending Interfere W/ Node Not Ready Each Other Pod Not in Right Node Out of Resource
Yes, You are Involved in Livestock Industry Now! 真的變成 “ 碼農 ” 了 ...
Cluster Pattern Pod Arrangement Resource Management
Cluster Pattern Pod Arrangement Resource Management
How to Arrange Application Workload? If There are 3 Applications, 3 Environments (Alpha, Beta, ● Production) ... Run All Application Instances on a Single Cluster? ● A Separate Cluster for Each Application Instance? ● A Combination of the Above? ●
One Large Shared Cluster 👎 Efficient Resource Usage Beta Prod 👎 Cheap Alpha Beta Prod 👎 Efficient Administration 👏 Single Point of Failure Alpha Alpha Beta Prod 👏 No Hard Security Isolation 👏 No Hard Multi-Tenancy 👏 Many Users 👏 Clusters Can't Grow Infinitely Large
Many Small Single-Use Clusters Beta Alpha Prod 👎 Reduced Blast Radius 👎 Isolation Beta Alpha Prod 👎 Few Users 👏 Inefficient Resource Usage 👏 Expensive Beta Alpha Prod 👏 Complex Administration
Cluster per Application Alpha Beta 👎 Cluster Can be Customised for an Prod App Alpha Beta Alpha 👏 Different Environments in the Prod Beta Same Cluster Prod
Cluster per Environment 👎 Isolation of the Prod Alpha Environment 👎 Cluster can be Customised for an Beta Environment Prod 👎 Lock Down Access to Prod Cluster 👏 Lack of Isolation Between Apps 👏 App Requirements are Not Localised
Which One is Better? Depends on Your Use Case ● Trade-Off the Pros and Cons of the Different Approaches ● The Choice is Not Limited to the Above Examples ● It can be Any Combination of Them! ● Ref
Multiple (Availability) Zones Multiple, Isolated Locations Within Each Region ● Protect your Application Against (Availability) Zone ● Disruption
Network Latency Take AWS for Example, Inter-AZ Network Latency Depends ● on Different Region, General Below 10 ms Does It Matter? ●
Persistent Volume High Efficiency Storage and Pod Need to Stay in the Same ● (Availability) Zone What is the Problem? ●
Extra Cost AWS/Azure/GCP Regional Data Transfer is Charged at $ ● 0.01/GB Large Amount of Data Transfer will Lead to Huge Cost ● (GitLab)
Cluster Pattern Pod Arrangement Resource Management
How to Put Pod in the Right Node Dedicated Nodes ● Nodes with Special Hardware ● Taint based Evictions ●
Node Selector apiVersion: v1 land:grass ❤ kind: Pod ... spec: containers: - name: cattle image: cattle land:grass imagePullPolicy: IfNotPresent nodeSelector: land: grass
Node Affinity - Required apiVersion: v1 kind: Pod metadata: name: with-node-affinity http://kubernetes.io/land: spec: pasture-1 or pasture-2 ❤ affinity: nodeAffinity: required DuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/land operator: In kubernetes.io/land: pasture-1 values: - pasture-1 - pasture-2 ...
Node Affinity - Preferred ╮（╯＿╰）╭ apiVersion: v1 kind: Pod metadata: name: with-node-affinity http://kubernetes.io/land: spec: pasture-1 or pasture-1 😣 affinity: nodeAffinity: preferred DuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: kubernetes.io/land kubernetes.io/land: pasture-3 operator: In values: - pasture-1 - pasture-2 ...
Taint apiVersion: v1 kind: Pod metadata: name: cattle labels: env: test spec: containers: - name: cattle image: cattle land=mud:NoSchedule imagePullPolicy: IfNotPresent
Toleration ... spec: containers: - name: pig image: pig imagePullPolicy: IfNotPresent tolerations: - key: "land" operator: "Equal" land=mud:NoSchedule value: "mud" effect: "NoSchedule"
Inter-Pod Affinity apiVersion: v1 kind: Pod metadata: name: with-pod-affinity spec: affinity: podAffinity : required DuringSchedulingIgnoredDuringExecution: - labelSelector: land:grass land:grass matchExpressions: - key: species operator: In values: - cattle topologyKey: failure-domain.beta.kubernetes.io/land land:mud land:mud
Inter-Pod Anti-Affinity apiVersion: v1 kind: Pod metadata: name: with-pod-affinity spec: affinity: podAntiAffinity : required DuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: species land:grass operator: In values: - cattle topologyKey: failure-domain.beta.kubernetes.io/land land:mud
Why Need PodTopologySpread? apiVersion: v1 kind: Pod metadata: name: with-pod-affinity spec: affinity: podAffinity : required DuringSchedulingIgnoredDuringExecution: - labelSelector: land:grass land:grass matchExpressions: - key: species operator: In values: - cattle topologyKey: failure-domain.beta.kubernetes.io/land land:mud land:mud
How PodTopologySpread Work? spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: land land:grass land:grass whenUnsatisfiable: DoNotSchedule labelSelector: skew=3 skew=0 species: cattle ❌ ✅
Cluster Pattern Pod Arrangement Resource Management
Why Need Resource Management? Avoid Out of Control Application Affect Others ● Application Support Scale Out Ability ● Easy to Plan Cluster Overall Capability ● Ensure The Most Important Application Survival and Safety ● ... ●
Everyone Knows Resource Request & Limit Ref
When K8S Users Ignore You 😇 Default Memory Requests and Limits for a Namespace ● Default CPU Requests and Limits for a Namespace ● Minimum and Maximum Memory Constraints for a ● Namespace Minimum and Maximum CPU Constraints for a Namespace ● Memory and CPU Quotas for a Namespace ● Pod Quota for a Namespace ●
But Do You Know Pod QoS? ● Guaranteed : Every Container in the Pod Must Have a Memory/CPU Limit and a Memory/CPU Request, and They Must be the Same ● Burstable : Not Meet the Criteria for QoS Class Guaranteed, and At Least one Container in the Pod has a Memory or CPU Request ● BestEffort : Not Have Any Memory or CPU Limits or Requests
When Out of Resource... BestEffort Pods ● Burstable Pods Whose Resource Usage Exceeds Its Request ● Burstable Pods Whose Resource Usage is Beneath Its ● Request Guaranteed Pods ●
Pod Disruptions Voluntary and Involuntary Disruptions ● Dealing with Disruptions ● Ensure Pod Requests Appropriate Resources ○ Replicate Your Application ○ Spread Applications Across Racks (Using Anti-Affinity) ○ or Across Zones (if Using a Multi-Zone Cluster)
Perform a Disruptive Action on All the Nodes Accept Downtime ● Failover to Another Complete Replica Cluster ● Use Pod Disruption Budget ●
Pod Disruption Budget (1/6) PDB = At Least 2 of The 3 Pods to be Available at All Times
Pod Disruption Budget (2/6) PDB = At Least 2 of The 3 Pods to be Available at All Times
Pod Disruption Budget (3/6) PDB = At Least 2 of The 3 Pods to be Available at All Times
Pod Disruption Budget (4/6) PDB = At Least 2 of The 3 Pods to be Available at All Times
Pod Disruption Budget (5/6) PDB = At Least 2 of The 3 Pods to be Available at All Times
Pod Disruption Budget (6/6) PDB = At Least 2 of The 3 Pods to be Available at All Times
Pod Priority and Preemption apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority-nonpreempting value: 1000000 preemptionPolicy: Never/PreemptLowerPriority globalDefault: false/true description: "Pod Priority and Preemption"
THANKS! ANY QUESTIONS? You can find me at my office: Frontend Engineer ● Backend Engineer ●
Recommend
More recommend
