What w e have learned from developing and running ABw E Jiri Navratil, Les R.Cottrell (SLAC)
Why E2E tools are needed • The scientific community is increasingly dependent on networking as international cooperation grows. HEP users (needs transfer huge amount of data between experimental sites as SLAC, FNAL, CERN , etc. (where data is created) and home institutes spread over the world) • What ISPs (as Abilene,Esnet,Geant..) can offer to the users for getting information? (Not too much because they are only in the middle of the path and they don’t cover all parts of connections)
RAL RAL LAN LAN DL DL FZU LAN FZU LAN LAN LAN JANET JANET CESNET CESNET MichNET MichNET GEANT GEANT INFN ABILENE INFN ABILENE FNAL-LAN FNAL-LAN MIB MIB LAN LAN ESNET ESNET RENATER RENATER NERSC- NERSC- LAN LAN CERN CERN LAN LAN IN2P3 IN2P3 CALREN LAN CALREN LAN SLAC SLAC LAN LAN Data sources Users
RAL RAL LAN LAN DL DL FZU LAN FZU LAN LAN LAN JANET JANET CESNET CESNET MichNET MichNET GEANT GEANT INFN ABILENE INFN ABILENE FNAL-LAN FNAL-LAN MIB MIB LAN LAN ESNET ESNET RENATER RENATER NERSC- NERSC- LAN LAN CERN CERN LAN LAN IN2P3 IN2P3 CALREN LAN CALREN LAN SLAC SLAC LAN LAN Data sources Users
There must be always somebody who gives complex • information to the users of the community or the users have to have a tool which give them such information How fast I can transfer 20 GB from my experimental site (SLAC,CERN) to • my home institute? • Can I run graphical 3D visualization program with data located 1000 miles away? • How stable is line ? (Can I use it in the same conditions for 5 minutes or 2 hours or whole day ?) All such questions must be replied in few seconds doesn’t matter if for individual user or for Grid brokers • Global science has no day and night. To reply this we needed the tools that could be used in continuous mode 24 hours a day 7 days a week which can non intrusively detect changes on multiple path or on demand by any user
ABwE:Basic terminology: • Generally: Available bandwidth = Capacity – Load • ABwE measure Td – Time dispersion P1-P2 (20x PP) We are trying to distinguish two basic states in our results: f - “ Dominate (free) ” – when Td ~= const -“ loaded ” with Td = other value Td Td results from “ Dominate ” state are used to estimate DBC - Dynamic Bottleneck Capacity Td measured during the “ loaded ” state is used to estimate the level of XTR (cross traffic) ABw = DBC – XTR
Abing: Estimation principles: Td Tp (pairs) Tx (cross traffic) Td T n f Examples Td from different paths ”Dominating state” ”Dominating state” “Load state” “Load state” (when sustained load or no load) (when load is changing) Td i = Td i+1 = .. Td i+n q = Tx/ Tn (Tx=Td –Tp) Tx – busy time (transmit time for cross trafic) f Tn – transmit time for average packet q – relative queue increment (QDF) Td domin during decision interval Td (h-1) Td u = q/(q+1) Dbc = L pp /T d domin CT=u*Dbc Abw= Dbc -CT
What is DBC DBC • DBC DBC characterize instant high capacity bottleneck that DOMINATE on the path • It covers situations when routers in the path are overloaded and sending packets back to back with its maximal rates • We discovered that in most cases only one node dominates in the instant of our measurements (in our decision interval )
ABw E: Example of narrow link in the path narrow link in the path (Pipes analogy w ith different diameter and aperture) No impact (in t1) No impact (in t1) load load 1000 1000 622 622 622 100 Empty pipes Light beam Light source 622 622 622 ABW DBC DBC ABW monitor SLAC to UFL ABW monitor SLAC to UFL Ab Abw = DB DBC C – XTR XTR link that has domination effect on bandwidth DBC ABW
Example of heavy loaded link in the path heavy loaded link in the path (Pipes analogy w ith different diameter and aperture) strong XTraffic -> Impact (in t1) No impact (in t1) load load 1000 1000 622 622 622 415 Empty links (pipes) Light beam Light source 622 622 622 DBC DBC ABW monitor SLAC to UFL ABW monitor SLAC to UFL Heavy load (strong cross traffic) appeared in the path Abw Abw = DBC DBC – XT XTR It shows new DBC in the path because this load dominates in whole path ! Normal situation DBC~ 400 Mbits/s strong XTR Abilene MRTG graph Available bandwidth (cross traffic) ATLA to UFL
ABw E ABw E / MRTG match: / MRTG match: TCP test to TCP test to UFL UFL CALREN shows sending traffic 600 Mbits/s UFL UFL IPLS shows traffic 800-900 Mbits/s Heavy load (xtraffic) appeared in the path (defined new DBC in the path) Normal situation
Confront ABw E results w ith other tools Iperf,Pathload,Pathchirp
SLAC-DataTAG-CERN Probe Pr obe Recei eceiver ver NIC-1000Mbps test environment GbE Chicago, Il Chicago, Il User traffic (4 workstations with NIC1000Mbis/s + OC-12 ES.net path) To CERN (Ch) 2.5 Gbits/s DataTag GbE ES.net ES.net path ES.net path Menlo Park, Ca XT rec. (622 Mbits/s) (622 Mbits/s) NIC-1000Mbps Probe Probe Sender Sender ES.net NIC-1000Mbps GbE Experimental path SLAC 1 rtr-gsr-test 0.169 ms 0.176 ms 0.121 ms 2 rtr-dmz1-ger 0.318 ms 0.321 ms 0.340 ms User traffic 3 slac-rt4.es.net 0.339 ms 0.325 ms 0.345 ms 4 snv-pos-slac.es.net 0.685 ms 0.687 ms 0.693 ms GbE 5 chicr1-oc192-snvcr1.es.net 48.777 ms 48.758 ms 48.766 ms XT gen. 6 chirt1-ge0-chicr1.es.net 48.878 ms 48.778 ms 48.774 ms NIC-1000Mbps 7 chi-esnet.abilene.iu.edu 58.864 ms 58.851 ms 59.002 ms 8 r04chi-v-187.caltech.datatag.org 59.045 ms 59.060 ms 59.041 ms Probing packets Injected Cross traffic User traffic (background)
The match of the cross traffic (ABW – XT compare to injection traffic generated by Iperf) Level of background traffic Zoom DBC (OC-12 ) Available bandwidth Measured xt ( cross-traffic) Injected CT (cross traffic by Iperf) Conlusion: Iperf measure own performance which can approach DBC (in best case)
What w e learned from CAIDA testbed
Internet H Internet HOP/HOPS vers. Testbed P/HOPS vers. Testbed I n t e r n e t P a t h PP I-HOP Decision interval is changing (growing) Internet cross traffic .. 20 x 2. Packet Pair 1. Packet Pair Packet Length ~ MTU 2 1 2 1 Probes Not relevant packets CT1 Cross traffic sources 2 1 CT2 CT3 25 ms PP TBED Relevant packets cause a dispersion Decision interval (12 µ s for Oc12) If CT < 30% abw had detection problem ! Simul. cross traffic N o t r e l e v a n TBed CT t p a c k e t s Initial decision interval
How to improve “detection effectiveness” How to improve “detection effectiveness” Solution X .. 20 x .. 100 x Packet Length ~ MTU 1. Packet Pair 2. Packet Pair 2 1 2 1 decision interval CT 2 2 1 1 Solution LP CT 6 5 4 3 2 1 Solution nP CT New initial decision interval 25 ms Measurement time Solution LP – Long packets (9k) Relevant packets cause a dispersion 0.5 s to 2.5 s (creates micro-bottlenecks) Solution nP – n dummy Packets (mini-train)
S10 (Mini-train with S2 (PP-Packet Pair) 8 dummy packets) PP versus PP versus TRAIN: TRAIN: ABW and DBC merge merge in TRAIN samples (SLAC-CALTECH path)
s10 s2 s3 s4 s7 s5 PP versus PP versus TRAIN: TRAIN: ABW and DBC merge merge in TRAIN samples (SLAC-CALTECH path)
Compare long term Bandw idth statistics on real paths ESNET, Abilene, Europe
SLAC - Mib.infn.it SLAC - Rice.edu IEPM (achievable throughput via Iperf) (red bars) ABW: Available bandwidth (blue lines) SLAC - Man.ac.uk SLAC - ANL.gov IEPM (achievable throughput via Iperf) (red bars) ABW: Available bandwidth (blue lines) IEPM-Iperf vers. ABW (24 hours match)
Scatter plot graphs Achievable throughput via Iperf versus ABw on different paths (range 20–800 Mbits/s) (28 days history)
ABw data New CENIC path 1000 Mbits/s back to new CENIC path Iperf data to 100 Mbits/s Drop to 622 Mbits/s path by error In all cases the match of results from Iperf and ABw is evident 28 days bandw idth history During this time w e can see several different situations caused by different routing from SLAC to CALTECH
What we can detect with continues bandwidth monitoring • Immediate bandwidth on the path • Automatic routing changes when line is broken (move to backup lines) • Unexpected Network changes (Routing changes between networks, etc.) • Line updates (155 -> 1Giga, etc.) • Extreme heavy load
ABw as ABw s Troubleshooting tool Troubleshooting tool ( Discovering Routing problems and initiate alarming ) ( Discovering Routing problems and initiate alarming ) Results of traceroute analysis BW problem resolved (17:00) Routing back on standard path Via Abilene Original path via CALREN/CENIC DBC Problematic link discovered Available bandwidth Bandwidth problem discovered (14:00) User traffic Send alarm Standard routing via CALREN/CENIC (Example from SLAC – CENIC path)
SLAC SLAC – CENIC path upgrade from 1 to 10 Gigabit CENIC path upgrade from 1 to 10 Gigabit (Current monitoring machines allow monitor traffic in range 1 < 1000 Mbits only) Skip to new 10GBits/s link (our monitor is on 1GbE) To backup Router (degrading line for while)
Upgrade 155Mbits/s line to 1000Mbits/s at dl.uk
Recommend
More recommend
