Testing applications with traffic control In containers Alban - PowerPoint PPT Presentation

Testing applications with traffic control In containers Alban Crequy https://goo.gl/ZLLlv3 ContainerCon North America - August 2016

Alban Crequy ∘ Worked on rkt the last 1.5 years ∘ Currently tech lead on rkt ∘ In 2014, worked on traffic control for multimedia applications in cars (tcmmd) https://github.com/alban

Berlin-based software company building foundational Linux technologies Some examples of what we work on... OS Tree git for operating system binaries

Find out more about us… Blog: http://kinvolk.io/blog Github: https://github.com/kinvolk Twitter: https://twitter.com/kinvolkio Email: hello@kinvolk.io

Plan What is traffic control and how does it work on Linux ∘ How it can be used for testing a microservices application ∘ Demo ∘ With CoreOS Linux, Kubernetes, Weave Scope ∘

What is traffic control? How does it work on Linux?

Traffic control, why? fair distribution ∘ of bandwidth client reserve ∘ bandwidth to THE specific web server client INTERNET applications avoid ∘ client bufferbloat

Queuing disciplines (qdisc) Network scheduling algorithm ∘ which packet to emit next? ∘ when? ∘ Configurable at run-time: ∘ /sbin/tc ∘ Netlink ∘ Default on new network interfaces: sysctl net.core.default_qdisc ∘ eth0 THE INTERNET qdisc

Stochastic Fairness Queueing (sfq) round robin FIFO 0 THE INTERNET FIFO 1 ... eth0 FIFO n

Traffic control for testing?

Network emulator (netem) latency packet loss corrupt ... bandwidth eth0 THE INTERNET netem

Testing with containers Testing framework configure “netem” qdiscs: bandwidth, latency, packet drop... container 1 container 2 eth0 eth0

The demo application

microservices-demo https://github.com/microservices-demo/microservices-demo

Some micro-services payment catalogue front-end Firefox orders-db orders

Kubernetes

Kubernetes objects - Kubernetes “Pods” - Group of container(s) running together - Kubernetes “Replication Sets” - Control the number of “Pods” replicas running - Kubernetes “Service” - Directing the traffic to “Pods” pod pod service pod

Testing with traffic control in Kubernetes configure network ∘ pod pod simulator play scenarios ∘ pod pod controls Latency tc tc ∘ Bandwidth ∘ Kubernetes Kubernetes Packet drop ∘ node 1 node 2

Weave Scope

Weave Scope

Testing with Weave Scope pod pod pod pod pod pod pod pod Scope App Scope Scope tc tc Probe Probe Kubernetes Kubernetes node 1 Node 2

Demo Reproduce the demo yourself: https://github.com/kinvolk/demo

Plugins in Scope - Unix socket in /var/run/scope/plugins/ - Protocols: report and control - Write your own plugins for your testing needs Scope App report control control Scope tc plugin Probe report

Testing framework for web apps Selenium

Testing more complex scenarios (my “wishlist”)

Add latency on a specific connection payment catalogue front-end Firefox orders-db orders latency=100ms

How to define classes of traffic dest_ip=10.0.4.* dest_ip=10.0.5.* other eth0 interface netem latency=100ms

u32: filter on content eth0 interface HTB root qdisc (type = HTB) filters HTB root class (type = HTB) (type=u32) other ip=10.0.4.* ip=10.0.5.* HTB HTB HTB leaf classes (type = HTB) netem netem netem leaf qdiscs (type = netem) latency=10ms

Filtering with cBPF/eBPF if (skb->protocol…) BPF_JMP... return TC_H_MAKE(TC_H_ROOT, BPF_LD... compilation mark); BPF_RET... clang... -march=bpf userspace upload kernel in the kernel: eth0 x86_64 code - bpf() JIT compilation - Netlink BPF netem netem

eBPF maps tc userspace eBPF map kernel eth0 Build statistics ∘ Make them available to ∘ x86_64 code BPF the testing framework netem netem

The End Try the demos yourself: https://github.com/kinvolk/demo Read more: https://kinvolk.io/blog/ The slides: https://goo.gl/ZLLlv3

Questions?