Quality of Service on Linux for JPS ATLAS the Atlas TDAQ Event Building Network Okinawa TDAQ Y.Yasu (1) , A.Manabe (1) , Y.Nagasaka (2) , H.Hasegawa (3) , M.Shimojima (4) , M.Nomachi (5) , H.Fujii (1) and Y.Watase (1) on behalf of the Atlas Trigger/DAQ group (1)High Energy Accelerator Research Organization (KEK) Oho 1-1, Tsukuba, Ibaraki 305 Japan (2) Department of Information and Intellectual Systems Engineering Faculty of Engineering Hiroshima Institute of Technology 2-1-1 Miyake, Saeki-ku, Hiroshima 731-5193 Japan (3)Faculty of Science, Shinshu University Asahi 3-1-1, Matsumoto, 390-8621 Nagano Japan (4)Department of Electrical Engineering Nagasaki Institute of Applied Science (NIAS) 536 Aba-machi, Nagasaki 851-01 Japan (5)Graduate School of Science, Osaka University Machikaneyama 1-1, Toyanaka-city, 560-0043 Osaka Japan September 24, 2001 Y.Yasu 1
Contents JPS ATLAS Okinawa TDAQ Why QoS is necessary What QoS is Performance Measurements Packet loss on UDP/IP multicast transfer QoS software overhead on TCP/IP transfer Packet distribution on TCP/IP transfer Conclusion September 24, 2001 Y.Yasu 2
JPS ATLAS ATLAS Event Builder System Okinawa TDAQ September 24, 2001 Y.Yasu 3
Why QoS is necessary JPS ATLAS Okinawa TDAQ > UDP/IP multicast transfer is attractive for broadcasting messages to Event Builder Sources from scalability point of view, but UDP/IP may lose the messages(packets). Event Builder Data Flow Sources Manager > Constant Bit Rate of ATM network can eliminate the packet loss by managing the bandwidth. > QoS in Linux kernel can manage the bandwidth. Can the QoS eliminate the packet loss? September 24, 2001 Y.Yasu 4
JPS ATLAS What QoS is: Class-Based Queueing (CBQ) Okinawa TDAQ Class 1 Set over-limit estimator packet Class 2 Packet scheduler classifier Class 3 CBQ can classify incoming packets into multiple classes. CBQ can share and limit the transfer rate. September 24, 2001 Y.Yasu 5
JPS ATLAS What QoS is : Token Bucket Filter (TBF) Okinawa TDAQ rate Token buffer Bucket size Data packet Data Buffer TBF can limit the transfer rate according to the token rate September 24, 2001 Y.Yasu 6
JPS ATLAS What QoS is : Traffic control in Linux kernel Okinawa TDAQ Upper layers(TCP,UDP,...) Traffic Control Input de-multiplexing Forwarding Output queueing Traffic control is done only at output queueing. Outgoing packet is only controlled. September 24, 2001 Y.Yasu 7
JPS ATLAS What QoS is : Queueing discipline for Linux Okinawa TDAQ Classifier Qdisc (TBF) u32: bandwidth:100Mbit/s Specified IP Address Class(CBQ) 1:1 bandwidth:1000Mbit/s Root Qdisc(CBQ) 1: bandwidth :1000Mbit/s U32 can specify IP address to manage the bandwidth. TBF limits the bandwidth to 100Mbit/s on 1000Mbit/s network at this example. September 24, 2001 Y.Yasu 8
JPS ATLAS Performance measurement Okinawa TDAQ Packet loss measurement Measurement of QoS software overhead Packet distribution measurement September 24, 2001 Y.Yasu 9
Testbed JPS ATLAS Okinawa TDAQ 3Com Super StackII Switch 9300 12 ports PC500 PC500 PC500 PC500 Configuration of PC500 system CPU: PentiumIII/500MHz Linux kernel version : 2.4.5 Chipset 440GX gcc version: egcs-2.91-66 Memory: 100MHz/SDRAM/256MB acenic driver version : 0.8 PCIbus: 32-bit/33MHz NIC: AceNIC(1MB) September 24, 2001 Y.Yasu 10
Packet loss on JPS ATLAS UDP/IP multicast transfer Okinawa TDAQ 3Com Super StackII Switch 9300 12 ports PC500 PC500 PC500 PC500 Sender Receiver Receiver Receiver Sender sends message with 10 bytes to receivers in best effort. September 24, 2001 Y.Yasu 11
Packet loss on JPS ATLAS UDP/IP multicast transfer (Cont.) Okinawa TDAQ No QoS Transfer speed in Mbit/s Packet loss in percentage 1200 100 90 1000 80 70 800 60 600 50 40 400 30 20 200 10 0 0 0 300 600 900 1200 1500 Message size in bytes packet loss in any size September 24, 2001 Y.Yasu 12
Packet loss on JPS ATLAS UDP/IP multicast transfer(Cont.) Okinawa TDAQ Assigned bandwidth = 10 Mbit/s Transfer speed in Mbit/s Packet loss in percentage 12 100 10 80 8 60 6 40 4 20 2 0 0 0 300 600 900 1200 1500 Message size in bytes No packet loss in any size September 24, 2001 Y.Yasu 13
Packet loss on JPS ATLAS UDP/IP multicast transfer(Cont.) Okinawa TDAQ Transfer speed in Mbit/s Assigned bandwidth = 50 Mbit/s Packet loss in percentage 50 100 40 80 30 60 20 40 10 20 0 0 0 20 40 60 80 100 Message size in bytes packet loss in 60-byte size September 24, 2001 Y.Yasu 14
Packet loss on JPS ATLAS UDP/IP multicast transfer(Cont.) Okinawa TDAQ Assigned bandwidth = 100 Mbit/s Transfer speed in Mbit/s Packet loss in percentage 120 100 100 80 80 60 60 40 40 20 20 0 0 0 300 600 900 1200 1500 Message size in bytes No packet loss out of 200-byte size September 24, 2001 Y.Yasu 15
QoS Software Overhead JPS ATLAS Okinawa TDAQ 3Com Super StackII Switch 9300 12 ports PC500 PC500 PC500 PC500 Sender Receiver Assigned bandwidth is 1000Mbit/s. Sender sends message with 10 bytes to a receiver in best effort. This bandwidth does not limit the transfer, but it consumes CPU time. September 24, 2001 Y.Yasu 16
JPS ATLAS QoS Software Overhead: TCP/IP transfer speed Okinawa TDAQ Transfer speed in Mbit/sec. 700 600 500 400 No QoS QoS 300 200 100 0 10 100 1000 10000 Message size in bytes Peak speed was same. September 24, 2001 Y.Yasu 17
JPS ATLAS QoS Software Overhead: the CPU Usage Okinawa TDAQ CPU Usage (%) 110 100 90 sender cpu usage(No QoS) sender cpu 80 usage(QoS) 70 In small size of message, QoS was 10 % worse. 60 In large size of message, 10 100 1000 10000 the percentage increases. Message size in bytes September 24, 2001 Y.Yasu 18
TCP/IP Packet Distribution JPS ATLAS Okinawa TDAQ 3Com Super StackII Switch 9300 12 ports PC500 PC500 PC500 PC500 Sender Receiver Assigned bandwidth is 10Mbit/s. Sender sends message with 10 bytes to a receiver. tcpdump utility is used for capturing packets September 24, 2001 Y.Yasu 19
TCP/IP Packet Distribution(Cont.) JPS ATLAS Okinawa TDAQ QoS No QoS Horizontal axis : sending packet time interval With QoS, some packets were sent at 10 msec scheduling time. Without QoS, packets are mostly sent in several 10 usec. September 24, 2001 Y.Yasu 20
TCP/IP Packet Distribution(Cont.) JPS ATLAS Okinawa TDAQ QoS with 1kHz trigger rate No QoS with 1kHz trigger rate With 1kHz trigger, packets are mostly sent around 1msec. There are no difference between QoS and NoQoS. September 24, 2001 Y.Yasu 21
Conclusion JPS ATLAS Okinawa TDAQ We measured with/without QoS in Linux kernel: 1) Packet loss on UDP/IP multicast transfer 2) QoS software overhead on TCP/IP transfer 3) Packet distribution on TCP/IP transfer We concluded: 1) QoS could eliminate packet loss on UDP/IP multicast transfer. This shows the feasibility to solve the scalability issue of ATLAS TDAQ Event Builder. 2) QoS was efficiently performed on TCP/IP transfer. CPU usage of QoS on the transfer was small. 3) With QoS, some packets were sent at 10 msec scheduling time. September 24, 2001 Y.Yasu 22
Recommend
More recommend
