Quality of Service on Gigabit Ethernet for Event Builder Y.Yasu (1) - PowerPoint PPT Presentation

DAQ2000, Lyon ATLAS-Japan DAQ Group Quality of Service on Gigabit Ethernet for Event Builder Y.Yasu (1) , Y.Nagasaka (2) , H.Hasegawa (3) , A.Manabe (1) , M.Nomachi (4) , H.Fujii (1) , Y.Watase (1) (1)High Energy Accelerator Research Organization (KEK) Oho 1-1, Tsukuba, Ibaraki 305 Japan (2)Department of Electrical Engineering Nagasaki Institute of Applied Science (NIAS) 536 Aba-machi, Nagasaki 851-01 Nagasaki Japan (3)Faculty of Science Shinshu University Asahi 3-1-1, Matsumoto, 390-8621 Nagano Japan (4)Graduate School of Science Osaka University Machikaneyama 1-1, Toyanaka-city, 560-0043 Osaka Japan October 20, 2000 Y.Yasu 1

DAQ2000, Lyon ATLAS-Japan DAQ Group Contents Gigabit Ethernet and Quality of Service(QoS) for Event Builder Basic performance of Gigabit Ethernet QoS in Linux QoS performance on Gigabit Ethernet October 20, 2000 Y.Yasu 2

DAQ2000, Lyon ATLAS-Japan DAQ Group Gigabit Ethernet and Quality of Service for Event Builder Congestion Avoidance by Global Traffic Control(Fermilab, KEK) Special Hardware with Barrel shifter etc. by traffic shaping for each node(RD31/ CERN) ATM including QoS(Constant Bit Rate) How can we use Gigabit Ethernet with TCP/IP ? October 20, 2000 Y.Yasu 3

DAQ2000, Lyon ATLAS-Japan DAQ Group Basic performance of Gigabit Ethernet Hardware configuration and setup Performance measurement of memory copy Tuning network driver Performance measurement of Gigabit Ethernet October 20, 2000 Y.Yasu 4

DAQ2000, Lyon ATLAS-Japan DAQ Group Hardware configuration System PC500 PC733 PC800 CPU type PentiumIII PentiumIII PentiumIII CPU Freq. 500MHz 733MHz 800MHz Cache 512KB 256KB 256KB SuperWorks Chipset 440GX 440BX ServerSet III LE Bus speed 100MHz 133MHz 100MHz Memory type SDRAM SDRAM SDRAM Memory size 256 128 256 PCI(bus width) 32-bit 64-bit/32-bit 32-bit OS version 2.2.14-12 2.2.14-12 2.2.14-5 gcc version egcs-2.91.66 egcs-2.91.66 egcs-2.91.66 NIC AceNIC(1MB) AceNIC(1MB) AceNIC(1MB) Network Driver v0.47 v0.47 v0.47 version October 20, 2000 Y.Yasu 5

DAQ2000, Lyon ATLAS-Japan DAQ Group Performance measurement of memory copy 500 Memory-copy speed in MB/s 450 400 350 PentiumIII- 300 733MHz/SuperWorks PentiumIII-500MHz/440GX 250 PentiumIII-800MHz/440BX 200 150 100 50 by memcpy system call 0 0.1 1 10 100 Memory size in MB October 20, 2000 Y.Yasu 6

DAQ2000, Lyon ATLAS-Japan DAQ Group Hardware setup Alteon Switch 180 PC800 PC733 PC500 PC500 PC500 PC500 PC733 PC800 For measurement of QoS performance For measurement of and measurement of basic performance basic performance October 20, 2000 Y.Yasu 7

DAQ2000, Lyon ATLAS-Japan DAQ Group Tuning network driver Normal frame Jumbo frame 1000 1000 Transfer speed in Mbit/s 900 Transfer speed in Mbit/s 900 800 800 700 700 600 tx=20:rx=30 600 tx=400:rx=120 tx=200:rx=300 500 tx=996:rx=1000 tx=400:rx=600 500 tx=2000:rx=2000 tx=2:rx=3 400 tx=40:rx=12 400 300 300 200 200 100 100 0 0 0.01 1 100 10000 0.01 1 100 10000 Message size in KB Message size in KB Tuned parameters of Network driver : Normal frame : tx_coal=996 : rx_coal=1000 Jumbo frame : tx_coal=200 : rx_coal=300 /proc/sys/net/core/rmem_max 4000000 Other parameters : default /proc/sys/net/core/wmem_max 4000000 Used systems : PC500s TCP buffer size : 524288 bytes Nagle algorithm is enable. Measuring tool : Netperf October 20, 2000 Y.Yasu 8

DAQ2000, Lyon ATLAS-Japan DAQ Group Performance measurement of Gigabit Ethernet Transfer speed in normal frame Transfer speed in jumbo frame Bus width dependency(64-bit PCI vs. 32-bit PCI) October 20, 2000 Y.Yasu 9

DAQ2000, Lyon ATLAS-Japan DAQ Group Transfer speed in normal frame 1000 900 Transfer speed in Mbit/s 800 700 600 Pentium500MHz 500 Pentium733MHz Pentium800MHz 400 300 200 100 0 0.01 0.1 1 10 100 1000 Message size in KB Transfer speed does not depend on CPU speed in normal frame October 20, 2000 Y.Yasu 10

DAQ2000, Lyon ATLAS-Japan DAQ Group Transfer speed in jumbo frame 1000 900 Transfer speed in Mbit/s 800 700 600 Pentium500MHz Pentium733MHz 500 Pentium800MHz 400 300 200 100 0 0.01 1 100 10000 Message size in KB Transfer speed depends on Chipset performance in jumbo frame. October 20, 2000 Y.Yasu 11

DAQ2000, Lyon ATLAS-Japan DAQ Group Bus width dependency(64-bit PCI vs. 32-bit PCI) 1000 900 Transfer speed in Mbit/s 800 700 600 32-bit PCI(mtu1500) 64-bit PCI(mtu1500) 500 32-bit PCI(mtu9000) 64-bit PCI(mtu9000) 400 300 200 100 On PC733s 0 0.01 1 100 10000 Message size in KB Bus width does not affect transfer speed in normal frame. 64-bit PCI improves transfer speed in jumbo frame, up to 990Mbit/s. October 20, 2000 Y.Yasu 12

DAQ2000, Lyon ATLAS-Japan DAQ Group QoS in Linux Class-Based Queueing Token Bucket Filter Traffic control in TCP/IP Queueing discipline for Linux QoS performance on Gigabit Ethernet 1x1 system 3x1 system 1x3 system October 20, 2000 Y.Yasu 13

DAQ2000, Lyon ATLAS-Japan DAQ Group Class-Based Queueing (CBQ) Class 1 Set overlimit estimator packet Class 2 Packet scheduler classifier Class 3 CBQ can classify incoming packets into multiple classes. CBQ can share and limit the transfer rate. October 20, 2000 Y.Yasu 14

DAQ2000, Lyon ATLAS-Japan DAQ Group Token Bucket Filter (TBF) rate Token buffer Bucket size Data packet Data Buffer TBF can limit the transfer rate according to the token rate. October 20, 2000 Y.Yasu 15

DAQ2000, Lyon ATLAS-Japan DAQ Group QoS in Linux QoS in Linux kernel 2.2.x and Iproute2 for Linux supplies the following functions to control IP packets; For IP packets of inbound and outbound Classifying Prioritizing Sharing Limiting QoS in Linux kernel 2.2.x : in kernel standard distribution Iproute2 : ftp://ftp.inr.ac.ru/ip-routing/iproute2-current.tar.gz Iproute2 supports a typical command called tc. Can we use the QoS such as Class-Based Queueing and Token Bucket Filter for shaping traffic on Event Builder? October 20, 2000 Y.Yasu 16

DAQ2000, Lyon ATLAS-Japan DAQ Group Traffic control in TCP/IP Upper layers(TCP,UDP,...) Traffic Control Input de-multiplexing Forwarding Output queueing Traffic control is done only at output queueing. October 20, 2000 Y.Yasu 17

DAQ2000, Lyon ATLAS-Japan DAQ Group Queueing discipline for Linux Classifier Qdisc (TBF) u32: Rate:100Mbit Specified IP Address Class CBQ 1:1 rate:1000Mbit Ῡ ̰ Qdisc CBQ 1: bandwidth:1000Mbit Ῡ ̰ U32 is a classifier. TBF limits the transfer rate to 100Mbit/s at this example. The configuration was used at 1x1 system for the evaluation. October 20, 2000 Y.Yasu 18

DAQ2000, Lyon ATLAS-Japan DAQ Group QoS performance on 1x1 system 600 Measured rate in Mbit/s No QoS 500 400 Ideal rate 300 Measured rate 200 100 On PC500s 0 0 100 200 300 400 500 600 Assigned rate in Mbit/s Bandwidth assignment worked well. The overhead of QoS was very small and nearly 1 %. October 20, 2000 Y.Yasu 19

DAQ2000, Lyon ATLAS-Japan DAQ Group QoS performance on 3x1 system 250 Measured rate in Mbit/s 200 No QoS Ideal rate 150 Measured rate on node1 Measured rate on node2 100 Measured rate on node3 50 On PC500s 0 0 50 100 150 200 250 Assigned rate in Mbit/s Bandwidth assignment worked well October 20, 2000 Y.Yasu 20

DAQ2000, Lyon ATLAS-Japan DAQ Group QoS performance on 1x3 system 250 Measured rate in Mbit/s No QoS 200 150 Ideal rate Measured rate on node1 Measured rate on node2 Measured rate on node3 100 50 On PC500s 0 0 50 100 150 200 250 Assigned rate in Mbit/s Bandwidth assignment worked well at small rate, but did not work at full rate. Packet sharing performance without QoS was better than that with QoS. October 20, 2000 Y.Yasu 21

DAQ2000, Lyon ATLAS-Japan DAQ Group Conclusion * Traffic management of event data flow is necessary for event builder. * IP-based QoS with Gigabit Ethernet on PC/Linux was investigated. * The transfer speed did not depend on CPU speed in normal frame. * High performance chipset and 64-bit PCI improved the transfer speed in jumbo frame, up to 990Mbit/s. * Assignment of transfer rate was possible by using Linux/TC command, but it is not clear that the QoS on Gigabit Ethernet is effective for event builder. Thus, more investigation is necessary. October 20, 2000 Y.Yasu 22