Scheduling the I/O of HPC applications under congestion Ana - - PowerPoint PPT Presentation

▶
scheduling the i o of hpc applications under congestion
SMART_READER_LITE
LIVE PREVIEW

Scheduling the I/O of HPC applications under congestion Ana - - PowerPoint PPT Presentation

Feb 04, 2024 145 likes •628 views

Scheduling the I/O of HPC applications under congestion Ana Gainaru, Guillaume Aupy, Anne Benoit, Yves Robert, Franck Cappello & Marc Snir JLPC Sophia-Antipolis - June 2014 I/O scheduling 1 Motivation G. Aupy Motivation 2 Model Model

slide-1
SLIDE 1

Scheduling the I/O of HPC applications under congestion

Ana Gainaru, Guillaume Aupy, Anne Benoit, Yves Robert, Franck Cappello & Marc Snir JLPC Sophia-Antipolis - June 2014

slide-2
SLIDE 2

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

1.0

1 Motivation 2 Model

Platform Applications Objectives

3 Algorithms 4 Simulations

Applications Assessment of heuristics

5 Experiments 6 Conclusion

slide-3
SLIDE 3

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

2.0

Interconnect technologies: A major challenge

Without efficient interconnect technology, exascale systems would be more like data-centers

The challenge:

Flops are “free”, we need to optimize data-movement!

slide-4
SLIDE 4

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

2.0

Interconnect technologies: A major challenge

Analysis of the Intrepid system @Argonne: I/O throughput decrease (percentage per application, over 400 applications).

slide-5
SLIDE 5

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

3.0

1 Motivation 2 Model

Platform Applications Objectives

3 Algorithms 4 Simulations

Applications Assessment of heuristics

5 Experiments 6 Conclusion

slide-6
SLIDE 6

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

4.0

Platform

  • N unit-speed processors, equipped with an I/O card of

bandwidth b

  • Centralized I/O system with total bandwidth B

b=0.1Gb/s/Node

=B

Model instantiation for the Intrepid platform.

slide-7
SLIDE 7

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

5.0

Applications

K applications competing for I/O. For application App(k):

  • Released at time rk;
  • Executed on β(k) procs;
  • n(k)

tot instances: I(k) i

consists of w(k,i) units of computation followed by the transfer of a volume vol(k,i)

io

;

  • The minimum time to execute vol(k,i)

io

is: time(k,i)

io

= vol(k,i)

io

min(β(k)b, B);

  • Last instance finishes at time dk.
slide-8
SLIDE 8

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

5.0

Applications

App(1) App(2) App(3) bw Time B

slide-9
SLIDE 9

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

5.0

Applications

App(1) App(2) App(3) w(1,1) w(2,1) w(3,1) bw Time B

slide-10
SLIDE 10

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

5.0

Applications

App(1) App(2) App(3) w(1,1) w(2,1) w(3,1) bw Time B

slide-11
SLIDE 11

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

5.0

Applications

App(1) App(2) App(3) w(1,1) w(2,1) w(3,1) bw Time B

slide-12
SLIDE 12

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

5.0

Applications

App(1) App(2) App(3) w(1,1) w(2,1) w(3,1) bw Time B

slide-13
SLIDE 13

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

5.0

Applications

App(1) App(2) App(3) w(1,1) w(2,1) w(3,1) bw Time B

slide-14
SLIDE 14

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

5.0

Applications

App(1) App(2) App(3) w(1,1) w(2,1) w(3,1) w(1,2) bw Time B

slide-15
SLIDE 15

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

5.0

Applications

App(1) App(2) App(3) w(1,1) w(2,1) w(3,1) w(1,2) bw Time B

slide-16
SLIDE 16

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

5.0

Applications

App(1) App(2) App(3) w(1,1) w(2,1) w(3,1) w(1,2) w(3,2) bw Time B

slide-17
SLIDE 17

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

5.0

Applications

App(1) App(2) App(3) w(1,1) w(2,1) w(3,1) w(1,2) w(3,2) w(2,2) bw Time B

slide-18
SLIDE 18

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

5.0

Applications

App(1) App(2) App(3) w(1,1) w(2,1) w(3,1) w(1,2) w(3,2) w(2,2) w(1,3) w(2,3) w(3,3) bw Time B

slide-19
SLIDE 19

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

6.0

Objectives

Definition (Application efficiency)

˜ ρ(k)(t) =

  • i≤n(k)(t) w(k,i)

t − rk , where n(k)(t) is the number of instances of App(k) executed at time t.

slide-20
SLIDE 20

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

6.0

Objectives

Definition (Application efficiency)

˜ ρ(k)(t) =

  • i≤n(k)(t) w(k,i)

t − rk , where n(k)(t) is the number of instances of App(k) executed at time t. Obviously: t − rk ≥

i≤n(k)(t)

  • w(k,i) + time(k,i)

io

  • .

Hence: ˜ ρ(k)(t) ≤ ρ(k)(t) =

  • i≤n(k)(t) w(k,i)
  • i≤n(k)(t)
  • w(k,i) + time(k,i)

io

.

slide-21
SLIDE 21

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

6.0

Objectives

  • SysEfficiency:

maximize 1 N

K

  • k=1

β(k)˜ ρ(k)(dk).

  • Dilation:

minimize max

k=1..K

ρ(k)(dk) ˜ ρ(k)(dk).

slide-22
SLIDE 22

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

7.0

1 Motivation 2 Model

Platform Applications Objectives

3 Algorithms 4 Simulations

Applications Assessment of heuristics

5 Experiments 6 Conclusion

slide-23
SLIDE 23

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

8.0

Scheduler

The scheduler monitors the stream of I/O calls; decides on the fly which applications can perform I/O.

  • At each time step, it has access to the state of the system

(each application efficiency, ˜ ρ(k)).

  • Based on a given strategy, chooses a subset of applications

that are allowed to perform I/O.

slide-24
SLIDE 24

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

8.0

Scheduler

The scheduler monitors the stream of I/O calls; decides on the fly which applications can perform I/O.

  • At each time step, it has access to the state of the system

(each application efficiency, ˜ ρ(k)).

  • Based on a given strategy, chooses a subset of applications

that are allowed to perform I/O. When a strategy favors App(k), it means that App(k) is executed as fast as possible (min

  • bβ(k), bwavail
  • ).
slide-25
SLIDE 25

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

9.0

Different strategies

  • RoundRobin: Similar to the current scheduler in HPC
  • systems. Applications are served following the

“First-Come, First Served” principle.

slide-26
SLIDE 26

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

9.0

Different strategies

  • RoundRobin: Similar to the current scheduler in HPC
  • systems. Applications are served following the

“First-Come, First Served” principle.

  • MinDilation: favors applications with high values of

ρ(k)(t) ˜ ρ(k)(t).

slide-27
SLIDE 27

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

9.0

Different strategies

  • RoundRobin: Similar to the current scheduler in HPC
  • systems. Applications are served following the

“First-Come, First Served” principle.

  • MinDilation: favors applications with high values of

ρ(k)(t) ˜ ρ(k)(t).

  • MaxSysEff: favors applications with low values of

β(k)˜ ρ(k)(t).

slide-28
SLIDE 28

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

9.0

Different strategies

  • RoundRobin: Similar to the current scheduler in HPC
  • systems. Applications are served following the

“First-Come, First Served” principle.

  • MinDilation: favors applications with high values of

ρ(k)(t) ˜ ρ(k)(t).

  • MaxSysEff: favors applications with low values of

β(k)˜ ρ(k)(t).

  • MinMax: same as MaxSysEff, unless there exists an

applications with ˜

ρ(k)(t) ρ(k)(t) below a threshold γ. In that case,

switches to MinDilation.

slide-29
SLIDE 29

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

9.0

Different strategies

  • RoundRobin: Similar to the current scheduler in HPC
  • systems. Applications are served following the

“First-Come, First Served” principle.

  • MinDilation: favors applications with high values of

ρ(k)(t) ˜ ρ(k)(t).

  • MaxSysEff: favors applications with low values of

β(k)˜ ρ(k)(t).

  • MinMax: same as MaxSysEff, unless there exists an

applications with ˜

ρ(k)(t) ρ(k)(t) below a threshold γ. In that case,

switches to MinDilation. Priority variant: if an application has started to do some I/O, then it is prioritized.

slide-30
SLIDE 30

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

10.0

1 Motivation 2 Model

Platform Applications Objectives

3 Algorithms 4 Simulations

Applications Assessment of heuristics

5 Experiments 6 Conclusion

slide-31
SLIDE 31

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

11.0

Applications

(≤ 1, 284 nodes) (≥ 1, 285 nodes) (≥ 4, 584 nodes)

Percentage time spent doing I/O per application type.

We use Darshan to capture the behavior of applications that ran on Intrepid (2013).

slide-32
SLIDE 32

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

11.0

Applications

System usage per day for each application type

We use Darshan to capture the behavior of applications that ran on Intrepid (2013).

slide-33
SLIDE 33

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

12.0

SysEfficiency Dilation 20 40 60 2 4 6 8

(a) 10 large applications, ratio of 20% Objectives for different mixes of applications and I/O computation ratios.

RoundRobin Priority-RoundRobin MinDilation Priority-MinDilation MaxSysEff Priority-MaxSysEff MinMax Priority-MinMax

slide-34
SLIDE 34

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

12.0

SysEfficiency Dilation 20 40 60 2 4 6 8 10 12 14 16

(b) 50 small and 5 large applications, ratio of 20% Objectives for different mixes of applications and I/O computation ratios.

RoundRobin Priority-RoundRobin MinDilation Priority-MinDilation MaxSysEff Priority-MaxSysEff MinMax Priority-MinMax

slide-35
SLIDE 35

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

12.0

SysEfficiency Dilation 20 40 2 4 6 8

(c) 50 small and 5 large applications, ratio of 35% Objectives for different mixes of applications and I/O computation ratios.

RoundRobin Priority-RoundRobin MinDilation Priority-MinDilation MaxSysEff Priority-MaxSysEff MinMax Priority-MinMax

slide-36
SLIDE 36

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

13.0

Comparison of the heuristics on current platforms

We then compared our results with the Intrepid and Mira scheduler when congestion occurs. We report here only the MinMax heuristic and its Priority variant. Note that Intrepid and Mira use an architectural enhancement to improve the behavior of applications with large bursts of I/O: Burst Buffers.

slide-37
SLIDE 37

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

13.0

Comparison of the heuristics on current platforms

Intrepid Upper Limit 2 4 6 8 10 12 14 16 18 20 22 24 26 28 2 4 6 8 10 12 Dilation 2 4 6 8 10 12 14 16 18 20 22 24 26 28 40 60 80 100 SysEfficiency

slide-38
SLIDE 38

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

13.0

Comparison of the heuristics on current platforms

Intrepid MinMax Upper Limit 2 4 6 8 10 12 14 16 18 20 22 24 26 28 2 4 6 8 10 12 Dilation 2 4 6 8 10 12 14 16 18 20 22 24 26 28 40 60 80 100 SysEfficiency

slide-39
SLIDE 39

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

13.0

Comparison of the heuristics on current platforms

Intrepid MinMax Priority Upper Limit 2 4 6 8 10 12 14 16 18 20 22 24 26 28 2 4 6 8 10 12 Dilation 2 4 6 8 10 12 14 16 18 20 22 24 26 28 40 60 80 100 SysEfficiency

slide-40
SLIDE 40

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

13.0

Comparison of the heuristics on current platforms

Mira MinMax Priority Upper Limit 2 4 6 8 10 40 60 80 100 SysEfficiency

slide-41
SLIDE 41

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

13.0

Comparison of the heuristics on current platforms

Mira MinMax Priority BurstBuffers Upper Limit 2 4 6 8 10 40 60 80 100 SysEfficiency

slide-42
SLIDE 42

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

14.0

1 Motivation 2 Model

Platform Applications Objectives

3 Algorithms 4 Simulations

Applications Assessment of heuristics

5 Experiments 6 Conclusion

slide-43
SLIDE 43

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

15.0

  • Experiments on Vesta (development platform for Mira)
  • Vesta is using hard disks and is affected by locality: we
  • nly used the Priority variant of heuristics
  • We implemented the heuristics as an additional layer on

top of Vesta I/O scheduler

slide-44
SLIDE 44

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

15.0

  • Experiments on Vesta (development platform for Mira)
  • Vesta is using hard disks and is affected by locality: we
  • nly used the Priority variant of heuristics
  • We implemented the heuristics as an additional layer on

top of Vesta I/O scheduler

Execution time overhead of our implementation of the IOR benchmark.

slide-45
SLIDE 45

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

16.0

SysEfficiency (above) and Dilation (below) for different scenarios on Vesta.

slide-46
SLIDE 46

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

17.0

Dilation values for the applications from 512/256/256/32 scenario.

slide-47
SLIDE 47

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

18.0

1 Motivation 2 Model

Platform Applications Objectives

3 Algorithms 4 Simulations

Applications Assessment of heuristics

5 Experiments 6 Conclusion

slide-48
SLIDE 48

I/O scheduling

  • G. Aupy

Motivation Model

Platform Applications Objectives

Algorithms Simulations

Applications Assessment of heuristics

Experiments Conclusion

Conclusion

  • New I/O scheduler taking global view of system into

account

  • Outperforms current scheduler
  • More experiments needed on larger application sets
  • Window-based schedules for periodic applications?