Introductory Queueing Theory Tutorial Mor Harchol-Balter Computer - PowerPoint PPT Presentation

Introductory Queueing Theory Tutorial Mor Harchol-Balter Computer Science Dept, Carnegie Mellon Univ. 1

Outline I. Basic Vocabulary Avg arrival rate, l Response time, T o o Little’s Law Avg service rate, µ o o Exponential vs. Pareto/Heavy-tailed Avg load, r o o Poisson Process o Avg throughput, X o II.Single-server queues o Scheduling: FCFS, PS, SJF, LAS, SRPT M/G/1 response time o o Scheduling: Priority Classes Inspection Paradox o o Scheduling: SOAP Framework (New) Effect of job size variability o Effect of load o III.Multi-server queues Single shared queue, M/G/k Replication of jobs (New) o o Load balancing across queues Multi-task jobs and fork-join (New) o o Cycle stealing Networks of queues o o 2

Vocabulary l < µ Avg. Avg. FCFS service rate throughout arrival rate µ jobs l jobs sec sec ! = job size = service requirement 2 ! = 3 4 sec sec Avg service rate Example: jobs µ = 3 sec On average, job needs 3x10 6 cycles Avg size of job o Server executes 9x10 6 cycles/sec on this server: o E S = [ ] 1 sec. 3 3

Vocabulary: Load FCFS l µ jobs jobs sec sec 1 = E S [ ] S : job size µ l r = = = l = Load (utilization) Frac. time server busy E S [ ] µ Example: jobs l = arrive 2 2 o sec r = 3 E S = [ ] Each job requires sec on avg 1 o 3

Vocabulary: Throughput Defn: Throughput X is the average rate at which jobs complete (jobs/sec) QUESTION: Assume l < µ Which has higher throughput, C ? l µ jobs jobs sec sec l 2µ 2µ jobs jobs sec sec 5

Vocabulary: Throughput avg rate at which l µ C : jobs jobs sec jobs sec complete = l X (assuming no jobs dropped) 6

Vocabulary: Response Time job size S : T T Q 1 = E S [ ] l µ jobs jobs µ sec sec l r = l = E S [ ] µ T = response time T = queueing time (waiting time) Q Number jobs in system ! = # $ = #[!] Little’s Law: ' 7

Vocabulary: Response Time job size S : T T Q 1 = E S [ ] l µ jobs jobs µ sec sec l r = l = E S [ ] µ T = response time T = queueing time (waiting time) Q Q: Given that l < µ , what causes wait? < µ A: Variability in the arrival process & service requirements 8

Vocabulary: Response Time job size S : T T Q 1 = E S [ ] l µ jobs jobs µ sec sec l r = l = E S [ ] µ Variability Variability in arrival in job size, S process 9

Job Size Distributions “Most jobs are small; few jobs are large” µ a S ~ Exp( ) S ~ Pareto( ) 1 1 1 > = Pr{ S x } > = e µ - x Pr{ S x } x a heavy ½ ½ tail ¼ ¼ x x 0 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 10

Job Size Distributions S is time until coin with µ d µ d µ d µ d µ d µ d µ d µ d prob µ d comes up heads time d 2 d 3 d 4 d 5 d 6 d 7 d 8 d S µ a = S ~ Exp( ) S ~ Pareto( 1) 1 1 1 > = Pr{ S x } e µ - > = x Pr{ S x } x ½ ½ ¼ ¼ x x 0 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 11

Job Size Distributions • “Memoryless” • Decreasing hazard rate • Lower variability • Infinite variance • Light-tail: • Heavy-tail: top 1% of jobs top 1% of jobs comprise 5% load. comprise 50% load. µ a = S ~ Exp( ) S ~ Pareto( 1) 1 1 1 > = Pr{ S x } e µ - > = x Pr{ S x } x ½ ½ ¼ ¼ x x 0 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 12

Job Size Distributions • “Memoryless” Representative of: -- UNIX job sizes sizes • Lower variability -- Supercomputing job sizes • Light-tail: -- File sizes top 1% of jobs -- Human wealth comprise 5% load. -- Damage due to forest fires, earthquakes, etc. µ a = S ~ Exp( ) S ~ Pareto( 1) 1 1 1 > = Pr{ S x } e µ - > = x Pr{ S x } x ½ ½ ¼ ¼ x x 0 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 13

Variability T job size S : T Q 1 = E S [ ] l µ jobs jobs µ sec sec l r = l = E S [ ] µ Variability Variability in arrival in job size, S process 14

Vocabulary: Poisson Process with rate l Arrival Arrival Arrival d 2 d 3 d 4 d 5 d 6 d 7 d 8 d 9 d time l l l S ~ Exp( ) S ~ Exp( ) S ~ Exp( ) (Poisson process comes up when aggregating many users) 15

Outline I. Basic Vocabulary Response time, T o Avg arrival rate, l o Little’s Law o Avg service rate, µ o Exponential vs. Pareto/Heavy-tailed o Avg load, r o Poisson Process o Avg throughput, X o II.Single-server queues o Scheduling: FCFS, PS, SJF, LAS, SRPT M/G/1 response time o Inspection Paradox o Scheduling: Priority Classes o Effect of job size variability o Scheduling: SOAP Framework (New) o Effect of load o III.Multi-server queues Single shared queue, M/G/k Replication of jobs (New) o o Load balancing across queues Multi-task jobs and fork-join (New) o o Cycle stealing Networks of queues o o 16

Single-Server Queue T S : job size T Q 1 = E S [ ] l µ jobs jobs µ sec sec l r = l = E S [ ] µ M/G/1 1 server General Exponential i.i.d. inter-arrival service times (M = memoryless) times r ? Q: Does low è low E T [ ] Q 17

Single-Server Queue T S : job size T Q 1 = E S [ ] l µ jobs jobs µ sec sec l r = l = E S [ ] µ M/G/1 1 server General Exponential i.i.d. inter-arrival service times (M = memoryless) times A: low load does NOT ensure low wait 18

M/G/1 r 2 E S [ ] = × E [ Q T ] - r 1 2 [ ] E S Where is this coming from? A: low load does NOT ensure low wait 19

Waiting for the bus 20

Waiting for the bus S : time between buses E S = [ ] 1 m n 0 i time S S S QUESTION: On average, how long do I have to wait for a bus? (a) < 5 min (b) 5 min (c) 10 min (d) >10 min

Waiting for the bus S : time between buses Wait time S S S 2 E S [ ] = >> E [Wait] E S [ ] 2 [ ] E S “Inspection Paradox” 22

M/G/1 r 2 E S [ ] = × E [ Q T ] - r 1 2 [ ] E S High load High job size leads to variability leads to high wait high wait To drop load, we can increase server speed. Q : What can we do to combat job size variability? A : Smarter scheduling! 23

Scheduling in M/G/1 l µ jobs jobs sec sec Well-studied scheduling policies : FCFS (First-Come-First-Served, non-preemptive) PS (Processor-Sharing, preemptive) SJF (Shortest-Job-First, non-preemptive) SRPT (Shortest-Remaining-Processing-Time, preemptive) LAS (Least-Attained-Service First, preemptive)

Scheduling in M/G/1 E [ T ] FCFS SJF PS 9 Under high job size 7 variability 5 LAS 3 SRPT 1 r 0 0.2 0.4 0.6 0.8 1.0 FCFS (First-Come-First-Served, non-preemptive) PS (Processor-Sharing, preemptive) SJF (Shortest-Job-First, non-preemptive) SRPT (Shortest-Remaining-Processing-Time, preemptive) LAS (Least-Attained-Service First, preemptive)

Priority Classes ! " jobs 1 st sec 2 nd ! # jobs sec According to Ruth Williams (genetic networks): • Jobs à molecules • Server à enzyme • Classes à protein species • Reneging à dilution • Class 1’s load and variability can really affect class 2 26

Big Scheduling Breakthrough [Scully, Harchol-Balter, Scheller-Wolf SIGMETRICS 2018] The SOAP framework: Enables first analysis of many previously intractable policies: SERPT : Prioritize jobs by Expected Remaining Size Gittins : Prioritize jobs by their Gittins Index Discretized Policies : Preemptions only at specific ages Mixed Priority Classes : Priority classes, where each class can have its own scheduling policy. ---

Outline I. Basic Vocabulary Response time, T o Avg arrival rate, l o Waiting time, T Q o Avg service rate, µ o Exponential vs. Pareto/Heavy-tailed o Avg load, r o Poisson Process o Avg throughput, X o II.Single-server queues o Scheduling: FCFS, PS, SJF, LAS, SRPT M/G/1 response time o Inspection Paradox o Scheduling: Priority Classes o Effect of job size variability o Scheduling: SOAP Framework (New) o Effect of load o III.Multi-server queues Single shared queue, M/G/k Replication of jobs (New) o o Load balancing across queues Multi-task jobs and fork-join (New) o o Cycle stealing Networks of queues o o 28

M/G/k Model When server frees up, it grabs next available job k servers Q: How does M/G/k compare with M/G/1 at k-speed? A: Both worse and better! 29

Load Balancing Model Probabilistically split into independent queues. p 1 p 3 p 2 30

Load Balancing Model Round-Robin Join-Shortest-Queue L.B. Least-Work-Left Size-Interval Assignment Smart Load Balancing è Much reduced mean response time 31

Cycle Stealing Model (N-model) A’s B’s 2D-inf Markov Chain L.B. B’s have priority, OnlyA’s. but if idle, then work on A’s. 32

Replication Model [Gardner, Harchol-Balter, Scheller-Wolf Transactions on Networking 2017] [Gardner, Harchol-Balter, Scheller-Wolf Operations Research 2017] 33

Replication Model [Gardner, Harchol-Balter, Scheller-Wolf Transactions on Networking 2017] [Gardner, Harchol-Balter, Scheller-Wolf Operations Research 2017] Same job goes to multiple queues. Job is “done” as soon as first copy completes. 34