SMAC and XGBoost your Theorem Prover Edvard K. Holden Konstantin - PowerPoint PPT Presentation

SMAC and XGBoost your Theorem Prover Edvard K. Holden Konstantin Korovin The University of Manchester 1

Theorem Proving in First-Order Logic Proof Axioms + Conjecture iProver | Counter model | Timeout 2

Heuristics - The Key to Success ● Controls the proving process ● Crucial for performance ● No single optimal heuristic ● Manual exploration is infeasible 3

Heuristics - iProver ~100 Options ... --instantiation_flag true --inst_lit_sel [+prop;+sign;+ground;-num_var;-num_symb] --inst_lit_sel_side num_symb --inst_solver_per_active 1400 --inst_passive_queues [[-conj_dist;+conj_symb;-num_var];[+age;-num_symb]] --inst_passive_queues_freq [25;2] … --res_passive_queues [[+conj_symb;-num_symb];[+age;-num_symb]] --res_passive_queues_freq [15;5] --res_forward_subs full … 4

Proving Problems iProver 5

Proving Problems iProver 6

Proving Problems Solved Black 3 / 3 iProver Blue 1 / 2 Red 0 / 3 7

Proving Problems Heuristic 1 iProver Heuristic 2 Heuristic 3 8

Proving Problems Heuristic 1 iProver Heuristic 2 Heuristic 3 9

Proving Problems Heuristic 1 iProver Heuristic 2 Heuristic 3 ∴ All Problems Solved 10

Proving Problems What are the heuristics? How to group? Heuristic 1 How to map? iProver Heuristic 2 Heuristic 3 11

Heuristic Challenges Phase 1 ● Discover good heuristics Phase 2 ● Select the right heuristic 12

Phase 1 Learning and discovering efficient heuristics 13

Heuristic Learning - Optimisation [Heuristic] iProver Optimiser Feedback:= #Problems Solved 14

Heuristic Learning - SMAC Sequential Model-Based Algorithm Configuration - Construct the heuristics - Optimisation Parameters: ordinal, categorical, real - Optimise with Random Forest - Maximise number of solved problems 15

Heuristic Learning - Optimisation & Clustering [Heuristic] Optimiser iProver [Feedback] [Heuristic] Optimiser iProver [Feedback] [Heuristic] Optimiser iProver [Feedback]

Heuristic Learning - Clustering Problems (Problem Properties) Clustering (Heuristic Evaluation) 17

Heuristic Learning - Overview Features Opt Loop Heuristic 1 (Re) Cluster . . . Opt Loop Heuristic n Opt Loop [Evaluation Features] Heuristic Evaluation 18

Heuristic Learning - Results 500 CASC FOF Problems ● Default solves 207 ● ● Optimise ~2 days ● Optimise instantiation options 19

Phase 2 Selecting the best heuristic 20

Heuristic Mapping - Supervised Learning Features Model Label 21

Heuristic Mapping - Overview Features ML Model Label Problem XGBoost Heuristic iProver 22 22

Heuristic Mapping: Labelling Optimal Time Mapping Temporal Property Mapping AVG Label Time: 27 s AVG Label Time: 42 s 23

Heuristic Mapping - Model Results 10-Fold-Cross-Validation Test Accuracy 86% ± 2% Ratio of solved problems 88% ± 2% 24

Heuristic Mapping - Prover Results Default Best Optimised Heuristic *Trained with 30-70 split Heuristic Heuristic Mapping* Solved: 207 217 248 AVG Time in 27.9 28.7 26.0 intersection: 25

Conclusion Heuristic evaluation to learn heuristics Multi-class heuristic selection - Solves 24% more problems - Specialised and diverse heuristics - Reduces solving times by 60% - Solves nearly all solvable problems - 16.3% speed improvement over default heuristic 26