Getting the most out of your planner(s): from static to dynamic algorithm configuration Frank Hutter University of Freiburg & Bosch Center for AI fh@cs.uni-freiburg.de @FrankRHutter @AutoMLFreiburg These slides are available at www.automl.org/talks -- all references are hyperlinks
Motivation • Algorithm configuration (AC) finds good settings of your parameters – But it is limited: the parameter setting is fixed • We propose dynamic algorithm configuration (DAC) – This can change parameters based on the instance at hand, search progress, time, etc. parameter State setting features DAC Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 1
Outline • Part 1: an overview of previous meta-algorithmic approaches – Algorithm Configuration – Algorithm Portfolios Holger Kevin Leyton-Brown Hoos • Part 2: Dynamic Algorithm Configuration Theresa Gresa André Marius David Noor Steven Robert Eimer Shala Biedenkapp Lindauer Speck Mattmüller Awad Adriaensen Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 2
The Algorithm Configuration (AC) Problem Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 3
What Can be Parameters in Planning? In general Examples • Heuristics • Any design decision for which – Which heuristics to use you have more than 1 alternative – Subparameters of each heuristic – How to combine the heuristics • Parameter types – Boolean, categorical, • Search strategy integer, continuous – Conditional: only active dependent – Global / local search on setting of other parameters – Randomization – How to combine them • Often, parameters give rise to a high-dimensional structured space • Problem encoding – E.g., LPG: 62 parameters, – Domain model 6.5 × 10 17 configurations – Problem model Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 4
AC is a Useful Abstraction: Improvements in Many Areas [Hutter et al, FMCAD 2007] [Hutter et al, CPAIOR 2010] [Hutter et al, AAAI 2007] [Fawcett et al, TR 2009] [Fawcett et al, ICAPS-PAL 2011] [Vallati et al, SOCS 2013] [Vallati et al, IJCAI 2015] [Vallati & Serina, ICAPS 2018] AC is also a key enabling technology in automated machine learning (AutoML), e.g.: – Auto-WEKA [Thornton et al, KDD 2013] – Auto-sklearn [Feurer et al, NeurIPS 2015] – Auto-PyTorch [Zimmer et al, arXiv 2020] Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 5
AC is a Useful Abstraction: Increasingly Popular AC is increasingly popular (citation numbers from Google scholar) • Iterated F-Race Sampling based – • GGA/GGA++ Genetic algorithm – • ParamILS Local search – • SMAC Bayesian optimization – • All these algorithms are available through a unified interface in AClib Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 6
Empirical Evaluation of AC Methods in AClib [Hutter et al, 2020] Mixed integer programming AI planning Boolean satisfiability solving (SAT) TSP Answer set programming (ASP) Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 7
Sequential Model-based AC (SMAC): high-level overview [Hutter et al, LION 2011] Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 8
Bayesian Optimization Visualized f( θ ) θ Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 9
Sequential Model-based AC (SMAC): high-level overview [Hutter et al, LION 2011] Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 10
Saving time: aggressive racing [Hutter et al, JAIR 2009] Using a fixed number of N instances is suboptimal – Large N: too slow – Small N: too noisy, overfitting Adaptive choice of N (in FocusedILS & SMAC) – Start with N=1, reject aggressively – Increase only for good configurations ParamILS on a single QWH instance (test performance with 1000 new seeds) Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 11
Saving More Time: Adaptive Capping [Hutter et al, JAIR 2009] • Poor configurations often take a very long time (e.g., 1h vs. 1s) • We can cap their evaluation when we know them to be worse than the incumbent Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 12
Sequential Model-based AC (SMAC): high-level overview [Hutter et al, LION 2011] Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 13
All of these components matter for performance [Hutter et al, 2020] Example: optimizing CPLEX on combinatorial auctions (Regions-100) Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 14
AC Application #1: FD-AutoTune [Fawcett et al, ICAPS-PAL 2011] • Parameter space for Fast Downward – Choice of heuristics & subparameters slowdowns • h lm ( × 12) • h ff ( × 3) • h lmcut ( × 2) • h goal_count ( × 3) • h add ( × 3) • h mas ( × 4) • h cg ( × 3 ) • h hm ( × 2) • h cea ( × 3) speedups • h blind • h max – Search • 8 additional parameters – In total: 45 params, 2.99 × 10 13 configs Result: over 10x speedup on average • Domain-wise configuration with Per domain: 1x – 23x speedup FocusedILS Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 15
AC Application #2: Configuration of LPG [Vallati et al, SOCS 2013] • Parameter space for LPG (local search on linear action graph) • Preprocessing ( × 6) • Reachability • Search strategy ( × 15) information ( × 7) • Flaw selection strategy ( × 8) Search • • Search neighbourhood ( × 6) randomization ( × 3) • Heuristic funcion ( × 17) In total: 62 params, 6.5 × 10 17 configs • Domain-wise configuration with FocusedILS Configuration can also improve quality Result: over 10x speedup on average Per domain: 3x – 118x speedup Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 16
AC Application #3: Domain Model Configuration [Vallati et al, IJCAI 2015] • Parameter space for any planner, for how to rewrite the PDDL file – Order of domain predicates – Order of operators – Within each operator: • Order of preconditions • Order of postconditions – Up to 109 continuous parameters configured with SMAC • Analysis can provide useful information to effectively engineer domain models – fANOVA parameter importance suggests: Yahsp on Depots • First list operators that are used most/early Per domain: 1x – 339x speedup • First list preconditions unlikely to be satisfied Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 17
AC Application #4: Problem Model Configuration [Vallati & Serina, ICAPS 2018] • For any planner, how to rewrite the problem model file • Need a domain-specific heuristic that applies for all problems in the domain – Construct a parameterized heuristic using features of facts in the planning encoding graph (PEG) – Configure the heuristic‘s 26 parameters by SMAC • Per domain: 1x – 39x speedup • Analysis can provide useful information to effectively engineer problem models – fANOVA parameter importance suggests: • Initial and goal states‘ ordering should be aligned • First list propositional facts that often occur in preconditions & often occur positively • First list propositional facts that are most connected in the PEG Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 18
Outline • Part 1: an overview of previous meta-algorithmic approaches – Algorithm Configuration – Algorithm Portfolios • Part 2: Dynamic Algorithm Configuration Frank Hutter: Getting the most out of your planner(s) Slides available at http://www.automl.org/talks 19
Recommend
More recommend
