Learning-Assisted Reasoning within Interactive Theorem Provers - PowerPoint PPT Presentation

Learning-Assisted Reasoning within Interactive Theorem Provers Thibault Gauthier May 17, 2019 1 / 44

conjecture black box proof 2 / 44

library conjecture black box proof 2 / 44

white, spherical, many petals red, star-shaped, five petals Object Properties 3 / 44

Proof Assistant Theorems Constants Mizar 51086 9172 Coq 23320 4841 HOL4 16476 2247 HOL Light 16191 820 Isabelle/HOL 14814 1076 Matita 1712 629 4 / 44

= + x x 0 x ∀ . logical operators constants variables 5 / 44

library conjecture proof 6 / 44

Demo 7 / 44

library conjecture provers proof 8 / 44

library conjecture translation provers proof 8 / 44

library conjecture theorem prediction translation provers proof 8 / 44

Formula Syntactic features ∀ x , y . ( x + y ) × ( x − y ) = x 2 − y 2 Conjecture ∀ x , y , z . x × ( y + z ) = x × y + x × z ∀ x , y . x + y = y + x Library ∀ x , y . x × y = y × x e i π +1 = 0 ( x 2 ) ′ = 2 × x 9 / 44

Formula Syntactic features ∀ x , y . ( x + y ) × ( x − y ) = x 2 − y 2 2 Conjecture + , × , ∀ x , y , z . x × ( y + z ) = x × y + x × z × , + ∀ x , y . x + y = y + x + Library ∀ x , y . x × y = y × x × e i π +1 = 0 e , i , × , π , + , 1 , 0 ( x 2 ) ′ = 2 × x 2 ′ , 2 , × , 9 / 44

49 12 71 85 conjecture theorem → rule lemma 10 / 44

49 12 71 72 12 71 85 conjecture theorem → rule lemma 10 / 44

49 12 71 72 12 71 85 49 12 71 conjecture theorem → rule lemma 10 / 44

49 12 71 72 12 71 85 49 12 71 Theorem Dependencies conjecture theorem 85 49 12 71 → rule 102 51 45 86 12 lemma . . . . . . 10 / 44

Re-proving Tested library Benchmark Success standard library 40% judgement day 77% flyspeck 39% standard library 50% standard library 41% 11 / 44

Demo 12 / 44

library conjecture proof 13 / 44

library conjecture tactic prediction proof search proof 14 / 44

Tactics Useful for Solvers linear system, differential equations Simplifiers irreducible fraction, differentiation Induction natural numbers, lists, trees 15 / 44

Tactic Prediction Proof search Formal library Conjecture Search tree Proof recording Tactic policy Knowledge base Tactic value Training Proof minimization Predictors Proof 16 / 44

Tactic Prediction Proof search Formal library Conjecture Search tree Proof recording Tactic policy Knowledge base Tactic value Training Proof minimization Predictors Proof 16 / 44

axiom conjecture → rule lemma 17 / 44

axiom conjecture → rule lemma 17 / 44

axiom conjecture → rule lemma 17 / 44

axiom conjecture → tactic goal 18 / 44

axiom conjecture → tactic goal 18 / 44

axiom conjecture → tactic goal 18 / 44

REWRITE TAC INDUCT TAC METIS TAC 19 / 44

THENL tactical composes the effect of tactics. 20 / 44

THENL tactical composes the effect of tactics. INDUCT TAC 20 / 44

THENL tactical composes the effect of tactics. METIS TAC REWRITE TAC INDUCT TAC 20 / 44

THENL tactical composes the effect of tactics. METIS TAC REWRITE TAC INDUCT TAC 20 / 44

THENL tactical composes the effect of tactics. METIS TAC REWRITE TAC INDUCT TAC THENL [REWRITE TAC,METIS TAC] INDUCT TAC 20 / 44

Demo 21 / 44

Tactic Prediction Proof search Formal library Conjecture Search tree Proof recording Tactic policy Knowledge base Tactic value Training Proof minimization Predictors Proof 22 / 44

Proof recording Original proof: THENL [ REWRITE TAC, METIS TAC ] INDUCT TAC Modified proof: (R numLib.INDUCT TAC) THENL [ R boolLib.REWRITE TAC, R metisLib.METIS TAC ] Database of tactics: R (f n) (f (SUC n)) ⇒ transitive R: INDUCT TAC n ∗ m ≤ n ∗ p ⇒ (n = 0) ∨ m ≤ p : REWRITE TAC INJ f U(:num) s ⇒ INFINITE s : METIS TAC ... 23 / 44

Tactic Prediction Proof search Formal library Conjecture Search tree Proof recording Tactic policy Knowledge base Tactic evaluation Training Proof minimization Predictors Proof 24 / 44

Prediction algorithm Algorithm: Nearest neighbor weighted by TF-IDF heuristics Effect: Order goals from the database according to their distance to a target goal. Remark: This is algorithm performs premise selection. How do we adapt it to predict tactics? 25 / 44

Policy Database of tactics is a map from goals to tactics. R (f n) (f (SUC n)) ⇒ transitive R: INDUCT TAC n ∗ m ≤ n ∗ p ⇒ (n = 0) ∨ m ≤ p : REWRITE TAC INJ f U(:num) s ⇒ INFINITE s : METIS TAC ... An order on goals induces an order on tactics. New goal appearing during proof search: LENGTH (MAP f l) = LENGTH l Policy for the new goal: Rank Tactic Policy 1 REWRITE TAC 0.5 2 METIS TAC 0.25 ... 4 INDUCT TAC 0.0625 ... 26 / 44

Value Database of lists of goals: ◮ Positive examples: appears in human proofs. ◮ Negative examples: produced during TacticToe search but do not appear in the final proof. 27 / 44

Plan Tactic Prediction Proof search Formal library Conjecture Search tree Proof recording Tactic policy Knowledge base Tactic value Training Proof minimization Predictors Proof 28 / 44

Training Improve recorded data to create better predictions during search. 29 / 44

Training: orthogonalization Issue: Many tactics are doing the same job on a goal g . Solution: Competition for g where the most popular tactic wins. 30 / 44

Training: orthogonalization Recorded goal-tactic pair: LENGTH (MAP f l) = LENGTH l: INDUCT TAC Competition: Progress Coverage INDUCT TAC Yes 136 REWRITE TAC No 2567 METIS TAC Yes 694 Added to the database: LENGTH (MAP f l) = LENGTH l: METIS TAC Result: 6 % improvement. 31 / 44

Training: abstraction Issue: Some theorems are never used inside tactics. Solution: Abstract all lists of theorems in a tactic and instantiate them depending on the target goal. 32 / 44

Training: abstraction Abstraction algorithm: : REWRITE TAC [ T1,T2 ] Original Abstraction : REWRITE TAC X Instantiation: REWRITE TAC [ T67, T1, T43, ... ] Question: Dow we keep the original or the abstraction ? Answer: Let them compete during orthogonalization. Result: 15% improvement 33 / 44

Training: preselection Issue: Predictions are too slow during proof search. Solution: Preselect 1000 suitable tactics by importing proofs (many tactics) from related goals. 34 / 44

Tactic Prediction Proof search Formal library Conjecture Search tree Proof recording Tactic policy Knowledge base Tactic evaluation Training Proof minimization Predictors Proof 35 / 44

Proof search: search tree ... ... a j a 1 t j t m t 1 ... ... a 0 g 0 g i g n 36 / 44

Proof search: advanced tree search 0.5 0.25 0.125 37 / 44

Proof search: advanced tree search 0.25 0.5 0.125 0.7 0.25 0.125 0.5 0.7 37 / 44

Proof search: advanced tree search 0.3 0.25 0.125 0.5 0.7,0.3 0.25 0.125 0.5 0.7,0.3 37 / 44

Proof search: advanced tree search 0.3 0.25 0.125 0.5 0.7,0.3 0.8 0.25 0.125 0.5 0.7,0.3,0.8 37 / 44

Re-proving Tested library Proof automation Success 50% 66% 38 / 44

Re-proving: HOL4 proofs found in less than x seconds 4 , 000 3 , 000 2 , 000 TacticToe 1 , 000 E prover 0 0 10 20 30 40 50 60 39 / 44

Re-proving: percentage of solved HOL4 proof of size x 100 TacticToe 80 E prover 60 40 20 0 0 2 4 6 8 10 12 14 16 18 20 40 / 44

Tactic Prediction Proof search Formal library Conjecture Search tree Proof recording Tactic policy Knowledge base Tactic value Training Proof minimization Predictors Proof 41 / 44

Before: boolLib.REWRITE TAC [ DB.fetch "list" "EVERY_CONJ" ,... ] THEN BasicProvers.Induct on [ HolKernel.QUOTE "l"] THENL [ BasicProvers.SRW TAC [] [] , simpLib.ASM SIMP TAC (BasicProvers.srw ss ()) [ boolLib.DISJ IMP THM, DB.fetch "list" "MAP" , DB.fetch "list" "CONS_11" , boolLib.FORALL AND THM ]] After: Induct on ‘l‘ THENL [ SRW TAC [] [] , ASM SIMP TAC (srw ss ()) [ DISJ IMP THM, FORALL AND THM ]] 42 / 44

Summary: TacticToe learns from human proofs to solve new goals. Advantages over ATPs (E prover) for ITP (HOL4) users: ◮ Includes domain specific automation found in the ITP. ◮ Generated proofs are human-level proofs. ◮ No translation or reconstruction needed. 43 / 44

Demo 44 / 44