Network virtualisation using Crossbow Technology Uroš Nedić, MSc OpenSolaris Contributor
● Crossbow Technical Background ● Network Machines ● Network in a Bo x 2
Crossbow Features • Virtualization > IP Instances > NIC Virtualization - VNICs > Service Virtualization - Flows • Resource partitioning > QoS/Diffserv (without performance penalties) > SLA on a per connection basis > Class of Service Support • Performance & Observability > Dynamic polling > H/W and S/W fanout to multiple cores > Real Time Usage and History Better Defense against DDOS attacks 3
Virtual Stacks Services and Protocols Compute Resources CPU 1 CPU 2 CPU 'n' VIRTUAL VIRTUAL VIRTUAL SQUEUE SQUEUE SQUEUE CPU 1 Virtual Squeue CPU 2 Virtual Squeue VOIP HTTPS DEFAULT TCP UDP DEFAULT SQUEUE SQUEUE SQUEUE SQUEUE SQUEUE SQUEUE Virtual Virtual Virtual Virtual Virtual Virtual NIC NIC NIC NIC NIC NIC Memory Memory Memory Memory Memory Memory Partition Partition Partition Partition Partition Partition Flow Classifier Flow Classifier NIC 1 NIC 2 4
Virtual Stacks Services and Protocols Compute Resources CPU 1 CPU 2 CPU 'n' VIRTUAL VIRTUAL VIRTUAL SQUEUE SQUEUE SQUEUE CPU 1 Virtual Squeue The Squeue switches the MSI interrupt per stack between interrupt and polling mode VOIP HTTPS DEFAULT and controls the rate of packet arrival for SQUEUE SQUEUE SQUEUE the virtual stack. The VNICs are in the control path only. Virtual Virtual Virtual NIC NIC NIC The data link layer is bypassed. Memory Memory Memory Memory Memory Memory Partition Partition Partition Partition Partition Partition Flow Classifier Flow Classifier NIC 1 NIC 2 5
Crossbow Architecture ● Partition the NIC Hardware, kernel queues, and CPU and allow creation of Virtual NICs ● Use dynamic polling on Virtual NICs to schedule rate of packet arrival per VNIC ● Effect of dynamic polling Mpstat (older driver) intr ithr csw icsw migr smtx srw syscl usr sys wt idl 10818 8607 4558 1547 161 1797 289 19112 17 69 0 12 Mpstat (Crossbow based driver) intr ithr csw icsw migr smtx srw syscl usr sys wt idl 2823 1489 875 151 93 261 1 19825 15 57 0 27 ● Use Dynamic polling for B/W partitioning and isolation without any support from switches and routers 6
Parallelized Stack: Made for Cores/Threads Network NIC Kernel CPU Application VNIC C Squeue Rx/Tx Kernel Qs DMA and threads L A VNIC Dedicated Rx/Tx S Squeue Kernel Qs Lane DMA and threads S I F I VNIC E Rx/Tx Squeue Kernel Qs R DMA and threads Neptune Multi-Core Application Threads Or Crossbow 1000’s of Logical 2 x 10 Gbe Processor Solaris Containers Virtualized Network Connections and/or Or Virtual Machines Streams & VNICs 4 x 1Gbe 7
Virtual Network Containers Virtualization Solaris Zone Zone Global xb1-z1 xb1-z2 • Exclusive IP Stack Zone Virtual NICs Virtual Virtual • SQUEUE SQUEUE • Virtual Switches Exclusive IP Exclusive IP Resource Control Instance Instance • Bandwidth Limits & Gauarantees VNIC1 VNIC2 bge0 Priority • (100Mbps) (200Mbps) • Dedicated CPUs Observability Rx/Tx Rx/Tx Rx/Tx DMA DMA DMA • Watch real time usage for each VNIC Flow Classifier NIC • Snmp and Kstat per VNIC History at no cost • Client Client xb2 xb3 8
Virtual Machines Solaris Guest OS 2 Solaris Guest OS 1 Solaris Host OS NIC Virtualization Engine NIC Virtualization Engine NIC Virtualization Engine Guest OS 2 Guest OS 1 VIRTUAL SQUEUE Host OS VIRTUAL SQUEUE VIRTUAL SQUEUE All Traffic All Traffic HTTP HTTPS DEFAULT SQUEUE SQUEUE SQUEUE Guest OS 2 VNIC Virtual Virtual Virtual Host OS VNIC NIC NIC NIC Guest Guest Guest Guest Host OS OS 1 OS 1 OS 1 OS 2 All traffic HTTP HTTPS DEFAULT All Traffic H/W Flow Classifier NIC 9
Network Machines Solaris DMZ Apache Router/Firewall DNS Server TCP/ TCP/ TCP/ APIs UDP UDP UDP for ISVs IP IP IP at Dedicated each CPUs layer Virtual Virtual Virtual Virtual NIC A NIC A NIC B NIC B N2/NIU Rx/Tx Rx/Tx Rx/Tx Rx/Tx Rx/Tx Rx/Tx DMA DMA DMA DMA DMA DMA Flow Classifier Flow Classifier NIC A NIC B Internet Intranet 10
Anatomy of a Typical Router Major cost paid on bringing the packet in and route lookup etc. PORT 0 Xmit Q 1 Xmit Port 0 PORT 1 Xmit Q 2 Xmit Port 1 Route Lookup Xmit Q 3 Xmit Port 2 PORT 2 Engine Xmit Q m Xmit Port m PORT n Packets are dropped on Xmit Queues 11
Anatomy of Crossbow Router Network OpenSolaris Compute Interface Network Stack Interface DMA 0 VNIC HTTPS T/C DMA 1 VNIC Data T/C Packet Classifier DMA 2 VNIC Voice T/C DMA 16 VNIC Video T/C Policy-based packet drop on ingress—zero cost drop. Explicit mapping of Compute Threads or Cores to Network Threads creating virtual • “express lanes” Dedicated BW, Priority, and CPU/Threads assigned to each lane • • High performance packet classification 12
Physical Network Router Host 1 Host 2 Client Port6 Port9 Port3 Port1 Port2 20.0.0.3 20.0.0.1 10.0.0.3 10.0.0.1 10.0.0.2 1 Gbps 1 Gbps 1 Gbps 100Mbps 1 Gbps Switch 3 Switch 1 Crossbow Network in a Box 1 �
Example VNIC Usage • Done using dladm(1M) , as with other data-link interface administration # dladm create-vnic -d bge1 vnic1 # dladm create-vnic -d bge1 -m random -p maxbw=100M -p cpus=4,5,6 vnic2 # dladm create-etherstub vswitch1 # dladm show-etherstub LINK vswitch1 # dladm create-vnic -d vswitch1 -p maxbw=1000M vnic3 # dladm show-vnic LINK OVER MACTYPE MACVALUE BANDWIDTH CPUS vnic1 bge1 factory 0:1:2:3:4:5 - - vnic2 bge1 random 2:5:6:7:8:9 max=100M 4,5,6 vnic3 vswitch1 random 4:3:4:7:0:1 max=1000M - 1 �
Join Us. ... • Our communities and projects are open on OpenSolaris.org: > CrossBow: http://opensolaris.org/os/project/crossbow > VNM: http://opensolaris.org/os/project/vnm > Networking: http://opensolaris.org/os/community/networking • Where you will find: > Lively discussions, design docs, FAQs, source code drops, binary releases, etc... 1 �
Recommend
More recommend
