Network Topology-aware Traffic Scheduling Emin Gabrielyan cole - PDF document

6th SOS Workshop on Distributed Supercomputing: Data Intensive Computing March 4-6, 2002, Badehotel Bristol, Leukerbad, Valais, Switzerland Network Topology-aware Traffic Scheduling Emin Gabrielyan École Polytechnique Fédérale de Lausanne, Switzerland Emin.Gabrielyan@epfl.ch

25-transfer data exchange T1 T2 T3 T4 T5 l 1 l 2 l 3 l 4 l 5 l 11 l 12 l 6 l 7 l 8 l 9 l 10 R1 R2 R3 R4 R5 T1 T2 T3 T4 T5 ... R1 R2 R3 R4 R5

Round-robin schedule T1 T2 T3 T4 T5 T1 T2 T3 T4 T5 R1 R2 R3 R4 R5 R1 R2 R3 R4 R5 T1 T2 T3 T4 T5 R1 R2 R3 R4 R5 T1 T2 T3 T4 T5 T1 T2 T3 T4 T5 R1 R2 R3 R4 R5 R1 R2 R3 R4 R5

Round-robin Throughput 1 2 3 .1 3 .2 4 .2 4 .1 connection throughput 5 mean number of connections per timeframe { { ⁄ ⋅ ⁄ ⁄ T roundrobin 100 MB s 357 MB s = 25 7 = total throughput number of transfers number of timeframes

Liquid Schedule step 1 step 2 step 3 step 4 step 5 step 6 ⁄ ⋅ ⁄ ⁄ T liquid 100 MB s 416 MB s = 25 6 = { mean number of connections per step

Load of Links and Transfers T1 T2 T3 T4 T5 ... X = R1 R2 R3 R4 R5 The 25 transfer traffic T1 T2 T3 T4 T5 5 5 l 2 5 l 3 5 l 4 l 5 5 l 11 l 1 6 bottlenecks 5 6 l 12 l 6 5 5 5 5 l 8 l 7 l 10 l 9 R1 R2 R3 R4 R5 λ l 1 X ( , ) , …λ l 12 X ( , ) = 5 = 6 { , } … l 1 l 12 l 6 , { , , } … , l 1 l 6 Transfers:

Duration of the Traffic l 1 l 2 l 3 l 4 l 5 l 11 l 12 l 6 l 7 l 8 l 9 l 10 {l 1 , l 6 }, {l 1 , l 7 }, {l 1 , l 8 }, {l 1 , l 12 , l 9 }, {l 1 , l 12 , l 10 }, {l 2 , l 6 }, {l 2 , l 7 }, {l 2 , l 8 }, {l 2 , l 12 , l 9 }, {l 2 , l 12 , l 10 }, X= {l 3 , l 6 }, {l 3 , l 7 }, {l 3 , l 8 }, {l 3 , l 12 , l 9 }, {l 3 , l 12 , l 10 }, {l 4 , l 11 , l 6 }, {l 4 , l 11 , l 7 }, {l 4 , l 11 , l 8 }, {l 4 , l 9 }, {l 4 , l 10 }, {l 5 , l 11 , l 6 }, {l 5 , l 11 , l 7 }, {l 5 , l 11 , l 8 }, {l 5 , l 9 }, {l 5 , l 10 } , ,... λ l 1 X ( , ) λ l 2 X ( , ) = 5 = 5 , λ l 11 X ( , ) λ l 12 X ( , ) = 6 = 6 Λ X ( ) = 6

Liquid Throughput {l 1 , l 6 }, {l 1 , l 7 }, {l 1 , l 8 }, {l 1 , l 12 , l 9 }, {l 1 , l 12 , l 10 }, {l 2 , l 6 }, {l 2 , l 7 }, {l 2 , l 8 }, {l 2 , l 12 , l 9 }, {l 2 , l 12 , l 10 }, X= {l 3 , l 6 }, {l 3 , l 7 }, {l 3 , l 8 }, {l 3 , l 12 , l 9 }, {l 3 , l 12 , l 10 }, {l 4 , l 11 , l 6 }, {l 4 , l 11 , l 7 }, {l 4 , l 11 , l 8 }, {l 4 , l 9 }, {l 4 , l 10 }, {l 5 , l 11 , l 6 }, {l 5 , l 11 , l 7 }, {l 5 , l 11 , l 8 }, {l 5 , l 9 }, {l 5 , l 10 } the throughput of a single link total number of transfers ( ) # X ⋅ T liquid - T link = - - - - - - - - - - - = Λ X ( ) the duration of the traffic (the load of its bottlenecks) = 25 ⋅ ⁄ ⁄ - 100 MB s 417 MB s - - - - = 6

No liquid schedule T R l 4 l 1 the 5 trans- fers block the access to l 9 bottlenecks l 7 R T l 2 l 6 l 8 l 3 l 5 T R {l 1 , l 7 , l 8 , l 6 }, ( ) # X = 3 X {l 2 , l 8 , l 9 , l 4 }, = Λ X ( ) = 2 {l 3 , l 9 , l 7 , l 5 } ( ) # X ⋅ T liquid - T link = - - - - - - - - - - - = Λ X ( ) ⁄ ⋅ ⁄ ⁄ 100 MB s 150 MB s = 3 2 =

Swiss-T1 Cluster N07 N08 N06 N09 PR15 PR16 PR14 PR17 N05 PR13 PR18 N10 P P R R PR20 PR11 1 N04 1 P N P 2 9 1 R R 1 PR22 PR09 2 1 1 0 PR08 PR23 3 N12 0 PR07 PR24 N PR06 PR25 1 2 N13 N02 P P R R 0 2 5 6 PR27 PR04 N01 P P N14 R R 2 0 3 8 PR02 PR29 3 0 PR30 PR01 N00 N15 PR31 PR00 PR32 PR63 1 N16 3 PR33 PR62 N PR34 PR61 4 7 N17 N30 5 0 3 6 R R P P PR36 PR59 7 8 3 5 N29 R N18 R P P PR38 PR57 6 5 PR39 PR56 PR40 N N28 PR55 1 PR41 PR54 9 2 3 5 4 PR52 PR43 N20 R N27 R 1 4 P P PR45 5 PR50 4 PR46 PR49 PR48 PR47 R R N21 6 P P 2 N N22 N25 N24 N23 N00 PR01 Receiving Processor Routing information Node 0 PR00 Sending Processor Network link Switch

363 Test Traffics 1800 1600 Liquid throughput (MB/s) 1400 Upper bound 1200 1000 Lower bound 800 600 400 200 0 0 4 8 12 16 20 24 28 32 Number of contributing nodes

363-Topology Test-bed 2800 Aggregate throughput (MB/s) 2400 t u p h g u o r 2000 h t r a b s s o r C 1600 1200 t u p h g 800 u o r h t d i u q i 400 L 0 ) ) 0 ) ) ) ) ) ) ) ) ) ) ) 9 1 2 4 5 8 0 0 6 9 2 4 ( ( 1 1 1 1 1 1 1 2 2 2 3 0 0 ( ( ( ( ( ( ( ( ( ( ( 3 0 0 0 0 0 0 0 0 0 0 0 6 9 2 5 8 1 4 7 0 3 6 1 1 1 2 2 2 3 3 3

Round-robin throughput t u p h g u o r h 1800 t d i u q i l 1600 T1 Cluster 1400 1200 1000 800 600 n i b o r - d n u o 400 r d e r u s a e m 200 0 0 9 1 3 4 5 6 8 9 1 3 6 1 1 1 1 1 1 1 2 2 2 Numbers of nodes for the 363 sub-topologies

Team: set of non-congesting transfers using all bottlenecks {l 1 , l 6 }, {l 1 , l 7 }, {l 1 , l 8 }, {l 1 , l 12 , l 9 }, {l 1 , l 12 , l 10 }, {l 2 , l 6 }, {l 2 , l 7 }, {l 2 , l 8 }, {l 2 , l 12 , l 9 }, {l 2 , l 12 , l 10 }, X = {l 3 , l 6 }, {l 3 , l 7 }, {l 3 , l 8 }, {l 3 , l 12 , l 9 }, {l 3 , l 12 , l 10 }, {l 4 , l 11 , l 6 }, {l 4 , l 11 , l 7 }, {l 4 , l 11 , l 8 }, {l 4 , l 9 }, {l 4 , l 10 }, {l 5 , l 11 , l 6 }, {l 5 , l 11 , l 7 }, {l 5 , l 11 , l 8 }, {l 5 , l 9 }, {l 5 , l 10 } {l 1 , l 8 }, {l 1 , l 12 , l 9 }, {l 2 , l 12 , l 9 }, {l 1 , l 12 , l 10 }, {l 2 , l 7 }, { l 3 , l 6 }, {l 2 , l 6 }, {l 3 , l 8 }, {l 4 , l 10 }, {l 4 , l 11 , l 6 }, {l 4 , l 11 , l 7 }, , , , {l 5 , l 11 , l 7 } {l 5 , l 10 } {l 5 , l 9 } α = {l 1 , l 7 }, {l 1 , l 6 }, { { l 3 , l 12 , l 10 }, { l 5 , l 11 , l 8 } } {l 2 , l 8 }, {l 2 , l 12 , l 10 }, { l 4 , l 9 }, { l 3 , l 12 , l 9 }, { l 3 , l 7 }, , , {l 5 , l 11 , l 6 } { l 4 , l 11 , l 8 } schedule α is liquid ⇔ number of steps load of the bottlenecks ⇔ # α ( ) Λ X ( ) ⇔ = ⇔ ∀ ( ∈ α ) A is a team of X A

Traffic without a team l 4 l 1 l 9 l 7 l 2 l 6 l 8 l 3 l 5 {l 1 , l 7 , l 8 , l 6 }, X {l 2 , l 8 , l 9 , l 4 }, = {l 3 , l 9 , l 7 , l 5 }

Liquid schedule search tree → ( ) { A 1 A 2 A 3 … A n , , } X Choice X = → ( ) { , , … } X 1 X A 1 Choice X 1 A 1 1 A 1 2 = – = , X 1 1 X 1 A 1 1 = – , , X 1 2 X 1 A 1 2 = – , , ... → ( ) { , , … } X 2 X A 2 Choice X 2 A 2 1 A 2 2 = – = , X 2 1 X 2 A 2 1 = – , , X 2 2 X 2 A 2 2 = – , , ... → ( ) { , , … } X 3 X A 3 Choice X 3 A 3 1 A 3 2 = – = , X 3 1 X 3 A 3 1 = – , , possible steps to the next layer ... ( ) { ∈ ℑ X ( ) ⊂ } Choice X i 1 i 1 … i n A A X i 1 i 1 … i n = , , set of all possible teams of X

Additional bottlenecks 2 bottlenecks A 1 A( X )=6 2 A 1,1 bottlenecks ( X 1 )=5 A X (25 transfers) 4 A 1,1,1 X 1 = X - A 1 (20 transfers) bottlenecks ( X 1,1 )=4 A X 1,1 = X 1 - A 1,1 (16 transfers) 4 bottlenecks ( X 1,1,1 )=3 A 6 bottlenecks 8 bottlenecks

Prediction of Dead-ends 2 bottlenecks A 1 A( X )=6 2 A 1,1 bottlenecks ( X 1 )=5 A X (25 transfers) X 1 = X - A 1 (20 transfers) A 1,1,1 4 bottlenecks ( X 1,1 )=4 X 1,1 = X 1 - A 1,1 (16 transfers) A load is 4 16-transfer traffic load is 4

Liquid schedule search optimization Y : reduced traffic original traffic’s ℑ Y ( ) ⊂ { ∈ ℑ X ( ) ⊂ } A A Y teams formed { from the re- duced traffic teams of the reduced traffic → ( ) { A 1 A 2 A 3 … A n , , } X Choice X = → ( ) { , , … } X 1 X A 1 Choice X 1 A 1 1 A 1 2 = – = , X 1 1 X 1 A 1 1 = – , , X 1 2 X 1 A 1 2 = – , , ... → ( ) { , , … } X 2 X A 2 Choice X 2 A 2 1 A 2 2 = – = , ... ( ) { ∈ ℑ X ( ) ⊂ } Choice Y A A Y = decrease of the search space without affect- ing the solution ( ) ℑ Y ( ) Choice Y = space

Liquid schedules construction ℑ full Y ( ) ⊂ ℑ Y ( ) { full teams of the reduced traffic ( ) ℑ Y ( ) Choice Y = decrease of the search space without affecting the ℑ full Y ( ) ( ) Choice Y solution space = • For more than 90% of the test-bed topolo- gies the search of liquid schedules took less than 0.1s on a single 500MHz processor. • For 8 topologies out of 363 solution was not found within 24 hours.

Results 2000 1800 All-to-all throughput (MB/s) 1600 1400 1200 1000 800 600 400 200 0 0 8 0 1 2 3 4 5 5 6 7 8 9 0 1 2 4 5 0 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 3 Number of nodes for the 363 sub-topologies

Conclusion • Data exchanges relying on the liquid schedules may be carried out several times faster com- pared with topology-unaware schedules. • Our method may be applied to applications requiring high network efficiency, such as video or voice traffic management, high energy phys- ics data acquisition and event assembling. • At the present we consider only static routing scheme. Dynamic routing could possibly be also combined in the algorithms. • Fixed packet size transfers are considered. • The network latency are neglected in compari- son with the transfer times. Thank You! Contact: Emin.Gabrielyan@epfl.ch