Enab abling DP DPDK DK/SR-IOV f for r November 2017 2017 - PowerPoint PPT Presentation

Enab abling DP DPDK DK/SR-IOV f for r November 2017 2017 contai ainer erized zed V Virtual al Net Network Func unctions w ns with h Zun Zun Bin Zhou [NFV Researcher, Lenovo] Hongbin Lu [Zun PTL,Huawei] Yaguang Tang [NFV Researcher, Lenovo] Shunli Zhou [Zun Core, Fiberhome]

Age genda da ➡ Introduction to Zun ➡ Zun Container for NFV • Challenges & Gaps • SR-IOV support in Zun • Container with DPDK ➡ Performance Benchmark Testing • Setup • Results ➡ Demo ➡ Conclusion

Which E Emerging Technologies I Interest st Open enStac ack Us User ers? ● Containers are the most interesting emerging technologies. ● 75% of OpenStack users interests in containers.

Introduce Zun Zun ➡ How to use containers on OpenStack? ➡ Existing solutions • Integrate containers into Nova • Example: Nova-docker, Nova-lxd • Install Container Orchestration Engine (COEs) on VMs. • Example: Magnum, Kubespray • OpenStack Container service: Zun

Introduce Zun Zun ● OpenStack Container service ● Provide API for provisioning and managing containers without VMs ○ Speed ○ Simplicity ● Arbitrary memory and vCPUs ● Containers as first class resource ○ Keystone RBAC for individual container ○ Neutron port(s) for each container ○ Cinder volume(s) bind-mount

Introduce Zun Zun ➡ Nova-docker • Use Nova to manage containers • Suitable if VMs and SSH Create Run VMs Containers containers are the same Migrate List Exec ➡ Obstacles ... Delete ... • VMs and containers are different • Container specified features are not exposed Nova Zun

Introduce Zun Zun ➡ Magnum • Provision Nova instances Magnum Zun • Install a COE Tenant 1 Tenant 2 Tenant 3 Tenant 1 Tenant 2 Tenant 3 • Run containers on the Contain Contain Contain Contain Contain Contain COE ➡ Pros: ers ers ers ers ers ers • Strong Isolation COE COE COE Zun ➡ Cons: • Low resource utilization Virtualization Virtualization (optional) • Virtualization penalty Baremetal Baremetal

Introduce Zun Zun ➡ Concepts: • Contai ainer er : A single container • create, update, delete, start, stop, kill, … • network-attach, add-security-group, … • attach, exec, commit, log, ... • Cap apsule e (Experimental): A group of containers that are co- located, have shared network and volumes. • create, list, delete, …

Introduce Zun Zun ➡ Zun API • Provide REST APIs • Manage all compute nodes Keystone Cinder • Scheduling containers ➡ Zun Compute • Compute node agent Zun API Zun Compute Docker • Manage local containers • Track compute resources ➡ Kuryr • Bind neutron ports to Neutron Kuryr containers

Age genda da ➡ Introduction to Zun ➡ Zun Container for NFV • Challenges & Gaps • SR-IOV support in Zun • Container with DPDK ➡ Performance Benchmark Testing • Setup • Results ➡ Demo ➡ Conclusion

Co Contai ainer f for NF NFV ➡ What is NFV • A new way to design, deploy and manage network services • Replace hardware with software • Move network functions to commodity hardware ➡ Benefits of NFV • Fast provisioning • Quick scale up and down • Easy upgrade and relocate • Reduce cost • No vendor hardware locked-in

Co Contai ainer f for NF NFV ➡ VM or Containers? • Time to provision: container boots faster • Resource consumption: container has less memory footprint • Package management: Docker makes it easy • Configurability: container is better • Portability: container image is smaller • Security: VM provides better isolation • Use Clear Container to improve security

Challe lleng nges & Gaps of usin ing containe iners ➡ Lack of supports of NFV required features in container VM NFV Req features Container ecosystem • Container runtime SR-IOV Yes Weak • Container orchestration DPDK Yes Weak • OpenStack integration ➡ Use Zun to reduce the gaps CPU pinning Yes Weak NUMA Yes Weak Hugepage Yes Weak

Enab able S e SR-IOV i in Zu Zun ➡ What is SR-IOV? • A standardized mechanism to virtualize PCIe devices • Make a single PCIe Ethernet controller (PF) to appear as multiple PCIe devices (VF) • PF: Physical Function • VF: Virtual Function • Passthrough VF to container • Bypass virtual switch layer

Enab able S e SR-IOV i in Zu Zun ➡ Enable SR-IOV in Zun • Create VFs in compute nodes • Configure Neutron • Configure Zun • Whitelist PCI devices (e.g. pci_passthrough_whitelist = { "devname": "eth3", "physical_network": "physnet2"}) • Enable PCI filters (e.g. enabled_filters = ...,PciPassthroughFilter) • Configure Kuryr • Enable SR-IOV driver

Enab able S e SR-IOV i in Zu Zun 1.Create a SR-IOV port 2.Create a container 2 User Zun API 3.Pick a host that has available VFs 4.Assign a VF to the port 3 1 5.Create a container 4 Neutron Zun Compute 6.Docker calls its network plugin (Kuryr) to setup the network 5 7.Kuryr retrieve VF’s information 7 from the neutron port and 6 Kuryr Docker perform port binding

Age genda da ➡ Introduction to Zun ➡ Zun Container for NFV • Challenges & Gaps • SR-IOV support in Zun • Container with DPDK ➡ Performance Benchmark Testing • Setup • Results ➡ Demo ➡ Conclusion

Co Contai ainer w with DP DPDK DK DPDK PMD ● physical nic ○ igb_uio ○ vfio-pci ● virtual hardware ○ virtio_user vhost software ● net_pcap (kernel stack)

DP DPDK DK & & S SR-IOV for c container er SR-IOV in kernel SR-IOV in userland Container Container DPDK DPDK Container Container DPDK Passthrough netns netns Host kernel ETHx ETHx PF driver Host kernel VF driver VF driver VF VF PF VF VF

Per erforman ance Ben enchmark T Tes esting Case 1 (non DPDK) Case 2 ● Container with SR-IOV & ● Zun Container with SR-IOV DPDK (kernel land) ● Zun Container with OVS ● Container with SR-IOV & DPDK (user land) networking

Tes esting s set etup non DPDK Testing ● iperf3 with udp Role Hardware OS network CPU Controller Think system x3650 M5 Ubuntu 82599ES Intel(R) E5- 16.04.3 10Gb 2680 v3 @ DPDK Testing 2.50GHz ● L2FWD as containerized compute Think system x3650 M5 Ubuntu 82599ES Intel(R) E5- 16.04.3 10Gb 2680 v3 @ 2.50GHz VNF ● RFC 2544 standard throughput testing Software version other ● DPDK-pktgen as packet DPDK 17.05 generator Openvswitch 2.8.1

Zu Zun networking w witho hout SR-IOV OV container container O O PF PF V V S S container container zun-compute zun-compute Linux Linux bridge Server1 Server2 bridge

Zu Zun n networking with S SR-IOV OV container VF VF container container VF VF container zun-compute zun-compute Server1 Server2

Container n network B Benchmarking

DPDK testing ng tuning ● Hugepage size ● PCIe NUMA ● Isolate CPU cores for tx/rx pktgen ● Disable isolated cpu core interrupts BOOT_IMAGE=/vmlinuz-4.4.0-87-generic root=/dev/mapper/docker2--vg-root ro default_hugepagesz=1G hugepagesz=2M hugepagesz=1G hugepages=8 iommu=pt intel_iommu=on isolcpus=5,6,7,8,9,10 nohz=on nohz_full=5,6,7,8,9,10 rcu_nocbs=5,6,7,8,9,10

Testing scenar Te ario 1 1 ● Userland SR-IOV used by container ● DPDK application l2fwd inside container dpdk-devbind --bind=igb_uio 0000:06:10.2 docker run -v /dev/hugepages/:/dev/hug VF1 VF1 epages --net=none -- privileged --name test2 VNF VF1 -dit 14ce48b74dd9 pktgen l2fwd VF2 VF2 l2fwd -l 5-6 -n 4 --huge-dir /dev/hugepages --socket- mem 1024,1024 -- -q 8 -p 1 Container Server1 Server2

● containers using SR-IOV by kernel netns Te Testing scenar ario 2 2 ● DPDK application l2fwd inside container $ neutron port-create sriov -- name sriov_port -- binding:vnic_type direct $ zun run --net port=sriov_port VF1 VF1 dpdk-test VNF NETNS VF1 pktgen l2fwd -l 5-6 -n 4 --huge-dir l2fwd /dev/hugepages --socket-mem VF2 1024,1024 -- VF2 vdev=’eth_pcap0,iface=eth0’ -- -q 8 -p 1 Container Server1 Server2

Container DPDK Benchmarking

Dem Demo ht https://yout utu. u.be/EwghP hPOVZLq Lq0

Conclusi sion SR-IOV & DPDK can accelerate container networking performance ● Container with SR-IOV for ● DPDK & SR-IOV for container user land approaching physical high throughput non DPDK application server performance ● unified management of VF ● multi-tenancy issue ● security issue Benefits High throughput Low latency Deterministic networking

Q&A Thank you! ope openstack @ @ O OpenStack ope openstack OpenStac ackFoundat ation