A Comparison of Two Parallel Ranking and Selection Procedures Eric C. Ni Shane G. Henderson School of Operations Research and Information Engineering, Cornell University Susan R. Hunter School of Industrial Engineering, Purdue University Dragos Florin Ciocan INSEAD Winter Simulation Conference, Savannah, GA December 8, 2014
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures 1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 1/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures 1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 2/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures Ranking and Selection (R&S) max µ i = E [ Y ( i ; ξ )] i ∈S • Optimize a function through a stochastic simulation. • Feasible region is finite: k = |S| < ∞ . • User-specified parameter δ . • Types of statistical guarantee (assuming µ 1 ≤ . . . ≤ µ k ): ◮ Correct selection (CS): P[select system k | µ k ≥ µ k − 1 + δ ] ≥ 1 − α ; ◮ Good selection (GS): P[select system i : µ i ≥ µ k − δ ] ≥ 1 − α. Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 3/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures Paths to GS guarantee GS procedures: • Multiple comparisons with the best (Rinott, 1978; Nelson and Matejcik, 1995; Nelson et al., 2001) ◮ Simultaneous confidence intervals on µ i − max j � = i µ j of width δ for all i . • Best-arm selection algorithms (Jamieson et al., 2013; Jamieson and Nowak, 2014) ◮ Guarantees to find the true best system if it is unique. CS procedures: • Sequential screening (Paulson, 1964; Fabian, 1974; Kim and Nelson, 2001, 2006; Hong, 2006). ◮ The assumption µ k ≥ µ k − 1 + δ is essential in proving statistical validity. Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 4/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures 1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 5/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures Previous work on parallel R&S • Web services-based parallel simulation optimization (Yoo et al., 2009; Luo et al., 2000) • Parallel version of a screening-based procedure, with asymptotic CS guarantee (Luo et al., 2013) Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 5/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures A parallel R&S procedure NHH Ni et al. (2013) • Stage 0: Simulate all systems to estimate simulation completion times. • Stage 1: (If variances need to be estimated) Independently of Stage 0, systems are simulated in parallel to obtain variance estimates. • Stage 2: Remaining systems are iteratively simulated and screened ( both in parallel ) until one system remains. Highly scalable. Inherits CS guarantee from its sequential predecessor (Hong, 2006). Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 6/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures (Parallel) Procedure NSGS Two-stage procedure proposed by Nelson et al. (2001). • Stage 0: We simulate all systems to estimate simulation completion times. • Stage 1: (If variances need to be estimated) Independently of Stage 0, systems are simulated in parallel to obtain variance estimates. Perform a round of screening (in parallel) with Stage 1 output . • Stage 2: (The Rinott step) Simulate up to ⌈ ( hS i /δ ) 2 ⌉ replications of each system i and choose the system with the highest sample mean . Parallel implementation inherits GS guarantee from Nelson et al. (2001), as the Rinott step leads to MCB confidence intervals. Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 7/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures Procedure complexity Let ∆ i := µ k − µ i . The expected number of replications required to eliminate a system i by system k is approximately • O ( σ 2 i ∆ − 2 log(∆ − 2 i )) using best-arm algorithms (Jamieson i et al. (2013)) • O (( σ 2 i + σ i σ k )[max(∆ i , δ )] − 1 δ − 1 ) using NHH • O ( σ 2 i δ − 2 ) using NSGS (and its parallel version) Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 8/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures 1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 9/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures Numerical example Our parallel procedures are applied to a throughput-maximization problem ( SimOpt.org ). Table 1: Summary of three instances of the test problem Number of Highest Num. of systems in [ µ k − δ, µ k ] Instance systems k mean µ k δ = 0 . 01 δ = 0 . 1 δ = 1 1 3249 5.78 6 21 256 2 57624 15.70 12 43 552 3 1016127 41.66 28 97 866 Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 9/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures Numerical example Instance 1: 3249 systems Instance 2: 57624 systems 1.0 3.0 Standard Deviation σ Standard Deviation σ 2.5 0.8 2.0 0.6 1.5 0.4 1.0 0.2 0.5 0.0 0.0 0 1 2 3 4 5 6 7 0 5 10 15 20 Mean µ Mean µ Figure 1: Mean-standard deviation profiles of two instances Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 10/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures Procedure Performance Table 2: Summary of procedure costs on 3 instances of the throughput maximization problem with α = 0 . 05, n 0 = 20 Total simulation Total Per Number of replications running replication ( × 10 6 ) systems k n 1 δ Procedure time (s) time ( µ s) 3249 60 0.01 NHH 3.9 23 375 NSGS 15.0 182 758 0.1 NHH 1.1 7.0 396 NSGS 0.4 2.9 480 57624 80 0.01 NHH 92.0 536 365 NSGS 1,600.0 10,327 403 0.1 NHH 30.0 179 371 NSGS 22.0 158 455 1016127 100 0.01 NHH 3,600.0 1,901 726 NSGS N/A (too costly) 0.1 NHH 720.0 324 572 NSGS 2,300.0 1,724 564 Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 11/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures NHH procedure NSGS procedure Wallclock time (s) Wallclock time (s) 10 3 10 3 10 2 10 2 Perfect scaling Perfect scaling 10 1 Actual performance 10 1 Actual performance 3 15 63 255 1023 3 15 63 255 1023 Number of workers Number of workers Figure 2: Scaling result on 57,624 systems, δ = 0 . 1 Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 12/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures 1 Introduction 2 Parallel Procedures 3 Numerical Experiments 4 New PGS Procedure 5 Conclusion Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 13/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures A parallel R&S procedure with PGS Two main stages: • Stage 1: ( In parallel ) Simulate and periodically screen systems until one system remains, or a pre-specified termination criterion is met. ◮ The screening method and termination criterion are jointly chosen such that P [System k survives Stage 1] is sufficiently large, say 1 − α/ 2. • Stage 2: (The Rinott step) Simulate up to ⌈ ( hS i /δ ) 2 ⌉ replications of each system i and choose the system with the highest sample mean . ◮ Stage 2 guarantees good selection amongst those surviving Stage 1. Theorem 1 The new procedure provides a good selection guarantee P[select system i : µ i ≥ µ k − δ ] ≥ 1 − α if simulation outcomes Y ( i ; ξ ) ∼ Normal( µ i , σ 2 i ). Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 13/18
Ni, Henderson, Hunter and Ciocan A Comparison of Two Parallel Ranking and Selection Procedures Implementing the PGS procedure MapReduce • Standard distributed programming model • Map() procedure to process data in parallel ◮ Simulate surviving systems • Reduce() procedure to summarize ◮ Screen using the additional statistics • Easy to program but incurs higher overhead • Runs on the cloud (e.g. Amazon EC2) Introduction Parallel Procedures Numerical Experiments New PGS Procedure Conclusion 14/18
Recommend
More recommend
