Meta-algorithmic techniques in SAT solving: Automated configuration, selection and beyond Holger H. Hoos BETA Lab Department of Computer Science University of British Columbia Canada SAT/SMT Summer School Trento, Italy, 2012/06/12
What fuels progress in SAT solving? ◮ Insights into SAT (theory) ◮ Creativity of algorithm designers � heuristics ◮ Advanced debugging techniques � fuzz testing, delta-debugging ◮ Principled experimentation � statistical techniques, experimentation platforms ◮ SAT competitions Holger Hoos: Meta-algorithmic techniques in SAT solving 2
2009: 5 of 27 medals 2011: 28 of 54 medals Holger Hoos: Meta-algorithmic techniques in SAT solving 3
Meta-algorithmic techniques ◮ algorithms that operate upon other algorithms (SAT solvers) � = meta-heuristics ◮ here: algorithms whose inputs include one or more SAT solvers ◮ configurators ( e.g. , ParamILS, GGA, SMAC) ◮ selectors ( e.g. , SATzilla, 3S) ◮ schedulers ( e.g. , aspeed; also: 3S, SATzilla) ◮ (parallel) portfolios ( e.g. , ManySAT, ppfolio) ◮ run-time predictors ◮ experimentation platforms ( e.g. , EDACC, HAL) Holger Hoos: Meta-algorithmic techniques in SAT solving 4
Why are meta-algorithmic techniques important? ◮ no one knows how to best solve SAT (or any other NP-hard problem) � no single dominant solver ◮ state-of-the-art performance often achieved by combinations of various heuristic choices ( e.g. , pre-processing; variable/value selection heuristic; restart rules; data structures; . . . ) � complex interactions, unexpected behaviour ◮ performance can be tricky to assess due to ◮ differences in behaviour across problem instances ◮ stochasticity � potential for suboptimal choices in solver development, applications Holger Hoos: Meta-algorithmic techniques in SAT solving 5
Why are meta-algorithmic techniques important? ◮ human intuitions can be misleading, abilities are limited � substantial benefit from augmentation with computational techniques ◮ use of fully specified procedures (rather than intuition / ad hoc choices) can improve reproducibility, facilitate scientific analysis / understanding Holger Hoos: Meta-algorithmic techniques in SAT solving 6
SAT is a hotbed for meta-algorithmic work! ◮ Drosophila problem for computing science (and beyond) ◮ prototypical NP-hard problem ◮ prominent in various areas of CS and beyond ◮ important applications ◮ conceptual simplicity aids solver design / development ◮ active and diverse community ◮ SAT competitions Holger Hoos: Meta-algorithmic techniques in SAT solving 7
Outline 1. Introduction 2. Algorithm configuration 3. Algorithm selection 4. Algorithm scheduling 5. Parallel algorithm portfolios 6. Programming by Optimisation (PbO) Holger Hoos: Meta-algorithmic techniques in SAT solving 8
Traditional algorithm design approach: ◮ iterative, manual process ◮ designer gradually introduces/modifies components or mechanisms ◮ performance is tested on benchmark instances ◮ design often starts from generic or broadly applicable problem solving method ( e.g. , evolutionary algorithm) Holger Hoos: Meta-algorithmic techniques in SAT solving 9
Note: ◮ During the design process, many decisions are made. ◮ Some choices take the form of parameters, others are hard-coded. ◮ Design decisions interact in complex ways. Holger Hoos: Meta-algorithmic techniques in SAT solving 10
Problems: ◮ Design process is labour-intensive. ◮ Design decisions often made in ad-hoc fasion, based on limited experimentation and intuition. ◮ Human designers typically over-generalise observations, explore few designs. ◮ Implicit assumptions of independence, monotonicity are often incorrect. ◮ Number of components and mechanisms tends to grow in each stage of design process. � complicated designs, unfulfilled performance potential Holger Hoos: Meta-algorithmic techniques in SAT solving 11
Solution: Automated Algorithm Configuration ◮ Key idea: expose design choices as parameters, Key idea: use automated procedure to effectively Key idea: explore resulting configuration space ◮ Given: algorithm A with parameters ¯ p and configuration space C ; set Π of benchmark instances, performance metric m ◮ Objective: find configuration c ∗ ∈ C for which A performs best on Π according to metric m Holger Hoos: Meta-algorithmic techniques in SAT solving 12
Algorithm configuration Lo Hi Holger Hoos: Meta-algorithmic techniques in SAT solving 13
Example: SAT-based software verification Hutter, Babic, HH, Hu (2007) ◮ Goal: Solve suite of SAT-encoded software verification Goal: instances as fast as possible ◮ new DPLL-style SAT solver Spear (by Domagoj Babic) = highly parameterised heuristic algorithm = (26 parameters, ≈ 8 . 3 × 10 17 configurations) ◮ manual configuration by algorithm designer ◮ automated configuration using ParamILS, a generic algorithm configuration procedure Hutter, HH, St¨ utzle (2007) Holger Hoos: Meta-algorithmic techniques in SAT solving 14
Spear : Empirical results on software verification benchmarks solver num. solved mean run-time MiniSAT 2.0 302/302 161.3 CPU sec Spear original 298/302 787.1 CPU sec Spear generic. opt. config. 302/302 35.9 CPU sec Spear specific. opt. config. 302/302 1.5 CPU sec ◮ ≈ 500-fold speedup through use automated algorithm configuration procedure (ParamILS) ◮ new state of the art (winner of 2007 SMT Competition, QF BV category) Holger Hoos: Meta-algorithmic techniques in SAT solving 15
SPEAR on software verification 4 10 3 10 auto-config. [CPU sec] 2 10 1 10 0 10 −1 10 −2 10 −2 10 −1 10 0 10 1 10 2 10 3 10 4 10 default config. [CPU sec] Holger Hoos: Meta-algorithmic techniques in SAT solving 16
Advantages of using automated configurators: ◮ enables better exploration of larger design spaces � better performance ◮ lets human designer focus on higher-level issues � more effective use of human expertise / creativity ◮ uses principled, fully formalised methods to find good configurations � better reproducibility, fairer comparisons, insights ◮ can be used to customise algorithms for use in specific application context with minimal human effort � expanded range of successful applications Holger Hoos: Meta-algorithmic techniques in SAT solving 17
Approaches to automated algorithm configuration ◮ Standard optimisation techniques ( e.g. , CMA-ES – Hansen & Ostermeier 2001; MADS – Audet & Orban 2006) ◮ Advanced sampling methods ( e.g. , REVAC – Nannen & Eiben 2006–9) ◮ Racing ( e.g. , F-Race – Birattari, St¨ utzle, Paquete, Varrentrapp 2002; Iterative F-Race – Balaprakash, Birattari, St¨ utzle 2007) ◮ Model-free search ( e.g. , ParamILS – Hutter, HH, St¨ utzle 2007; Hutter, HH, Leyton-Brown, St¨ utzle 2009; GGA – Ans´ otegui, Sellmann, Tierney 2009) ◮ Sequential model-based optimisation ( e.g. , SPO – Bartz-Beielstein 2006; SMAC – Hutter, HH, Leyton-Brown 2011–12) Holger Hoos: Meta-algorithmic techniques in SAT solving 18
Iterated Local Search (Initialisation) Holger Hoos: Meta-algorithmic techniques in SAT solving 19
Iterated Local Search (Initialisation) Holger Hoos: Meta-algorithmic techniques in SAT solving 19
Iterated Local Search (Local Search) Holger Hoos: Meta-algorithmic techniques in SAT solving 19
Iterated Local Search (Perturbation) Holger Hoos: Meta-algorithmic techniques in SAT solving 19
Iterated Local Search (Local Search) Holger Hoos: Meta-algorithmic techniques in SAT solving 19
Iterated Local Search ? Selection (using Acceptance Criterion) Holger Hoos: Meta-algorithmic techniques in SAT solving 19
Iterated Local Search (Perturbation) Holger Hoos: Meta-algorithmic techniques in SAT solving 19
ParamILS ◮ iterated local search in configuration space ◮ initialisation: pick best of default + R random configurations ◮ subsidiary local search: iterative first improvement, change one parameter in each step ◮ perturbation: change s randomly chosen parameters ◮ acceptance criterion: always select better configuration ◮ number of runs per configuration increases over time; ensure that incumbent always has same number of runs as challengers Holger Hoos: Meta-algorithmic techniques in SAT solving 20
SATenstein: Automatically Building Local Search Solvers for SAT KhudaBukhsh, Xu, HH, Leyton-Brown (2009) Frankenstein: create perfect human being from scavenged body parts SATenstein: create perfect SAT solvers using components scavenged from existing solvers Geneneral approach: ◮ components from GSAT, WalkSAT, dynamic local search and G2WSAT algorithms ◮ flexible SLS framework (derived from UBCSAT) ◮ find performance-optimising instantiations using ParamILS Holger Hoos: Meta-algorithmic techniques in SAT solving 21
Challenge: ◮ 41 parameters (mostly categorical) ◮ over 2 · 10 11 configurations ◮ 6 well-known distributions of SAT instances (QCP, SW-GCP, R3SAT, HGEN, FAC, CBMC-SE) ◮ 11 challenger algorithms (includes all winning SLS solvers from SAT competitions 2003–2008) Holger Hoos: Meta-algorithmic techniques in SAT solving 22
Result: ◮ factor 70–1000 performance improvements over best challengers on QCP, HGEN, CBMC-SE ◮ factor 1.4–2 performance improvement over best challengers on SW-GCP, R3SAT, FAC Holger Hoos: Meta-algorithmic techniques in SAT solving 23
Recommend
More recommend
