Exploiting Resolution Proofs to Speed Up LTL Vacuity Detection for - PowerPoint PPT Presentation

Bounded Model Checking (BMC) Check if property p holds up to k steps on model M: M | = k p i.e., can we reach a state in k steps that satisfies ¬ p? b c b c Model Model Model Property Property Property {b,c} {b,c} {c} {c} {a,b} {a,b} "unroll" p holds "unroll" "unroll" p holds p holds transition relation at some step transition relation transition relation at some step at some step ( a ), ( b ), ( c ) ( b ), ( c ) 0 0 0 0 0 k = 0 SAT solver SAT solver SAT solver SAT SAT SAT UNSAT UNSAT UNSAT M p M p M p M p M p M p k k k k k k Output: counter− Output: counter− Output: counter− Output: resolution Output: resolution Output: resolution example example example proof proof proof 6 / 25

Bounded Model Checking (BMC) Check if property p holds up to k steps on model M: M | = k p i.e., can we reach a state in k steps that satisfies ¬ p? b c b c b c Model Model Model Model Property Property Property Property {b,c} {b,c} {b,c} {c} {c} {c} {a,b} {a,b} {a,b} "unroll" "unroll" p holds p holds "unroll" "unroll" p holds p holds transition relation transition relation at some step at some step transition relation transition relation at some step at some step ( a ), ( b ), ( c ) ( a ), ( b ), ( c ) ( b ), ( c ) ( b ), ( c ) 0 0 0 0 0 0 0 0 0 0 k = 0 SAT solver SAT solver SAT solver SAT solver ( c ) ( c ) 0 0 SAT SAT SAT SAT UNSAT UNSAT UNSAT UNSAT M p M p M p M p M p M p M p M p ( ) k k k k k k k k Output: counter− Output: counter− Output: counter− Output: counter− Output: resolution Output: resolution Output: resolution Output: resolution example example example example proof proof proof proof 6 / 25

Bounded Model Checking (BMC) Check if property p holds up to k steps on model M: M | = k p i.e., can we reach a state in k steps that satisfies ¬ p? b c b c b c Model Model Model Model Property Property Property Property {b,c} {b,c} {b,c} {c} {c} {c} {a,b} {a,b} {a,b} "unroll" "unroll" p holds p holds "unroll" "unroll" p holds p holds transition relation transition relation at some step at some step G OAL : use resolution proof for vacuity detection transition relation transition relation at some step at some step ( a ), ( b ), ( c ) ( a ), ( b ), ( c ) ( b ), ( c ) ( b ), ( c ) 0 0 0 0 0 0 0 0 0 0 focus on variable vacuity k = 0 use naive detection as baseline for comparison SAT solver SAT solver SAT solver SAT solver ( c ) ( c ) 0 0 SAT SAT SAT SAT UNSAT UNSAT UNSAT UNSAT M p M p M p M p M p M p M p M p ( ) k k k k k k k k Output: counter− Output: counter− Output: counter− Output: counter− Output: resolution Output: resolution Output: resolution Output: resolution example example example example proof proof proof proof 6 / 25

Outline Model Checking Sanity Checks Naive Vacuity Detection Brief Overview of Vacuity Detection Bounded Model Checking New methods: Irrelevance Local Irrelevance Peripherality Implementation: V AQ T REE Experiments Conclusions and Future Work 7 / 25

Algorithm 1 - Irrelevance ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property 8 / 25

Algorithm 1 - Irrelevance ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () 8 / 25

Algorithm 1 - Irrelevance ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () Variables in the property but not in the UNSAT core are irrelevant 8 / 25

Algorithm 1 - Irrelevance ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () Variables in the property but not in the UNSAT core are irrelevant 8 / 25

Algorithm 1 - Irrelevance ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () Variables in the property but not in the UNSAT core are irrelevant 8 / 25

Algorithm 1 - Irrelevance ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () Variables in the property but not in the UNSAT core are irrelevant 8 / 25

Algorithm 1 - Irrelevance ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () Variables in the property but not in the UNSAT core are irrelevant 8 / 25

Algorithm 1 - Irrelevance ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () Variables in the property but not in the UNSAT core are irrelevant V ACUITY : d, e, f not in UNSAT core ⇒ irrelevant ⇒ vacuous 8 / 25

Algorithm 1 - Irrelevance ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () Variables in the property but not in the UNSAT core are irrelevant V ACUITY : d, e, f not in UNSAT core ⇒ irrelevant ⇒ vacuous Linear in size of UNSAT core 8 / 25

Algorithm 1 - Irrelevance ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () Variables in the property but not in the UNSAT core are irrelevant V ACUITY : d, e, f not in UNSAT core ⇒ irrelevant ⇒ vacuous Linear in size of UNSAT core Very incomplete 8 / 25

Algorithm 2 - Local Irrelevance Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () 9 / 25

Algorithm 2 - Local Irrelevance Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () Variables that only appear in the property part of the UNSAT core are locally irrelevant 9 / 25

Algorithm 2 - Local Irrelevance Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () s Variables that only appear in the property part of the UNSAT core are locally irrelevant 9 / 25

Algorithm 2 - Local Irrelevance Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () ss Variables that only appear in the property part of the UNSAT core are locally irrelevant 9 / 25

Algorithm 2 - Local Irrelevance Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () sss Variables that only appear in the property part of the UNSAT core are locally irrelevant 9 / 25

Algorithm 2 - Local Irrelevance Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () sss Variables that only appear in the property part of the UNSAT core are locally irrelevant V ACUITY : a only in Property part of the UNSAT core ⇒ locally irrelevant ⇒ vacuous 9 / 25

Algorithm 2 - Local Irrelevance Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () sss Variables that only appear in the property part of the UNSAT core are locally irrelevant V ACUITY : a only in Property part of the UNSAT core ⇒ locally irrelevant ⇒ vacuous Linear in size of UNSAT core 9 / 25

Algorithm 2 - Local Irrelevance Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () sss Variables that only appear in the property part of the UNSAT core are locally irrelevant V ACUITY : a only in Property part of the UNSAT core ⇒ locally irrelevant ⇒ vacuous Linear in size of UNSAT core More precise than Irrelevance 9 / 25

Algorithm 2 - Local Irrelevance Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () sss Variables that only appear in the property part of the UNSAT core are locally irrelevant V ACUITY : a only in Property part of the UNSAT core ⇒ locally irrelevant ⇒ vacuous Linear in size of UNSAT core More precise than Irrelevance Still very incomplete 9 / 25

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) Variables that are not central to the proof are peripheral

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) Variables that are not central to the proof are peripheral

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) Variables that are not central to the proof are peripheral

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) Variables that are not central to the proof are peripheral

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () Variables that are not central to the proof are peripheral

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () Variables that are not central to the proof are peripheral

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) () Variables that are not central to the proof are peripheral

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) Resolution on b () occurs in Property Variables that are not central to the proof are peripheral

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ b ) ( ¬ b ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( b ) ( c ) ( ¬ b ) Resolution on b () occurs in Property Variables that are not central to the proof are peripheral

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ x ) ( ¬ x ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( x ) ( c ) ( ¬ b ) Resolution on b () occurs in Property Can replace b by x in Property Variables that are not central to the proof are peripheral

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ x ) ( ¬ x ∨ c ) ( ¬ b ∨ ¬ c ) ( b ) ( x ) ( c ) ( ¬ b ) () Variables that are not central to the proof are peripheral

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ x ) ( ¬ x ∨ y ) ( ¬ b ∨ ¬ c ) ( b ) ( x ) ( y ) ( ¬ b ) Cannot replace c () by y in this proof Variables that are not central to the proof are peripheral 10 / 25

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ x ) ( ¬ x ∨ y ) ( ¬ b ∨ ¬ c ) ( b ) ( x ) ( y ) ( ¬ b ) Cannot replace c () by y in this proof Variables that are not central to the proof are peripheral 10 / 25

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ x ) ( ¬ x ∨ y ) ( ¬ b ∨ ¬ c ) ( b ) ( x ) ( y ) ( ¬ b ) Cannot replace c () by y in this proof Variables that are not central to the proof are peripheral V ACUITY : replaced b by x in Property without changing proof ⇒ peripheral ⇒ vacuous 10 / 25

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ x ) ( ¬ x ∨ y ) ( ¬ b ∨ ¬ c ) ( b ) ( x ) ( y ) ( ¬ b ) Cannot replace c () by y in this proof Variables that are not central to the proof are peripheral V ACUITY : replaced b by x in Property without changing proof ⇒ peripheral ⇒ vacuous Linear in size of resolution proof 10 / 25

Algorithm 3 - Peripherality Property Model ( ¬ a ) ( a ∨ x ) ( ¬ x ∨ y ) ( ¬ b ∨ ¬ c ) ( b ) ( x ) ( y ) ( ¬ b ) Cannot replace c () by y in this proof Variables that are not central to the proof are peripheral V ACUITY : replaced b by x in Property without changing proof ⇒ peripheral ⇒ vacuous Linear in size of resolution proof If p is vacuous, there exists a resolution proof s.t. p is peripheral 10 / 25

Complete Analysis G OAL : complete analysis using Naive Detection for leftover variables E XAMPLE : ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property 11 / 25

Complete Analysis G OAL : complete analysis using Naive Detection for leftover variables E XAMPLE : ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property I RRELEVANCE ALGORITHM 11 / 25

Complete Analysis G OAL : complete analysis using Naive Detection for leftover variables E XAMPLE : ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property I RRELEVANCE ALGORITHM d,e,f are vacuous 11 / 25

Complete Analysis G OAL : complete analysis using Naive Detection for leftover variables E XAMPLE : ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property I RRELEVANCE ALGORITHM d,e,f are vacuous C OMPLETING STEP 11 / 25

Complete Analysis G OAL : complete analysis using Naive Detection for leftover variables E XAMPLE : ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property I RRELEVANCE ALGORITHM d,e,f are vacuous C OMPLETING STEP 6 extra model checking runs 11 / 25

Complete Analysis G OAL : complete analysis using Naive Detection for leftover variables E XAMPLE : ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property I RRELEVANCE ALGORITHM d,e,f are vacuous C OMPLETING STEP 6 extra model checking runs M | = p [ a ← true ] ? M | = p [ a ← false ] ? 11 / 25

Complete Analysis G OAL : complete analysis using Naive Detection for leftover variables E XAMPLE : ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property I RRELEVANCE ALGORITHM d,e,f are vacuous C OMPLETING STEP 6 extra model checking runs M | = p [ a ← true ] ? p is vacuous w.r.t. a iff = p [ a ← true ] = M | M | = p [ a ← false ] ? M | = p [ a ← false ] 11 / 25

Complete Analysis G OAL : complete analysis using Naive Detection for leftover variables E XAMPLE : ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property I RRELEVANCE ALGORITHM d,e,f are vacuous C OMPLETING STEP 6 extra model checking runs M | = p [ a ← true ] ? p is vacuous w.r.t. a iff = p [ a ← true ] = M | M | = p [ a ← false ] ? M | = p [ a ← false ] Similar for b,c 11 / 25

Complete Analysis G OAL : complete analysis using Naive Detection for leftover variables E XAMPLE : ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property I RRELEVANCE ALGORITHM d,e,f are vacuous C OMPLETING STEP 6 extra model checking runs M | = p [ a ← true ] ? p is vacuous w.r.t. a iff = p [ a ← true ] = M | M | = p [ a ← false ] ? M | = p [ a ← false ] Similar for b,c I RRELEVANCE METHOD : Irrelevance algorithm + completing step 11 / 25

Complete Analysis G OAL : complete analysis using Naive Detection for leftover variables E XAMPLE : ( ¬ b ∨ ¬ c ) , ( b ) , ( ¬ e ) , ( d ∨ f ) Model ( ¬ a ) , ( a ∨ b ) , ( ¬ b ∨ c ) , ( d ∨ e ∨ f ) , ( a ∨ ¬ c ∨ d ) Property I RRELEVANCE ALGORITHM d,e,f are vacuous C OMPLETING STEP 6 extra model checking runs M | = p [ a ← true ] ? p is vacuous w.r.t. a iff = p [ a ← true ] = M | M | = p [ a ← false ] ? M | = p [ a ← false ] Similar for b,c I RRELEVANCE METHOD : Irrelevance algorithm + completing step Local Irrelevance and Peripherality are also extended in this manner 11 / 25

V AQ T REE : Vacuity Detection Framework Model property To our knowledge, V AQ T REE is the first vacuity detection tool for BMC A: SMV −> CNF [A] N U SMV v. 2.3.1, modified to Translator identify model/property clauses [B] M INI S AT -p v. 1.14, modified to B: Proof−outputting SAT solver output XML proof [C] New component (Java) C: Proof Analyzer proof analysis done in memory 700 MB of RAM ≈ 2.5 million D: Completing Step resolutions [D] New component (Perl) VaqTree Vacuity Results 12 / 25

Evaluation G OALS : Compare effectiveness of the three algorithms how many vacuous variables can each algorithm detect? Evaluate the performance of the three methods, using Naive Detection as a baseline are any of our methods faster than Naive Detection? B ENCHMARKS : Models and properties from the N U SMV distribution Models and properties from the IBM Formal Verification Benchmarks Library 13 / 25

Benchmark 1 S ETUP Models and properties: N U SMV distribution 121 properties: 99 present vacuity 2 - 4 temporal operators per property, from { G, F , U, X } 6 variables on average, 26 max., 1 min. Largest proof: 2.5 million resolutions 14 / 25

Interpreting Performance Graphs "Algorithm" (s) Naive detection (s) 15 / 25

Interpreting Performance Graphs Plotting execution times � "Algorithm" (s) � � � � � � � � � � � � � � � � � � �� �� � � � � � � � � � �� �� � � � � Naive detection (s) 15 / 25

Interpreting Performance Graphs � � "Algorithm" (s) "Algorithm" (s) � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� �� �� �� � � � � � � � � � � � � � � � � � � �� �� �� �� Naive � � � � Detection � � � � is faster here Naive detection (s) Naive detection (s) 15 / 25

Interpreting Performance Graphs “Method” is faster here � � � "Algorithm" (s) "Algorithm" (s) "Algorithm" (s) � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� �� �� �� �� �� � � � � � � � � � � � � � � � � � � � � � � � � � � � �� �� �� �� �� �� � � � � � � � � � � � � Naive detection (s) Naive detection (s) Naive detection (s) 15 / 25

Interpreting Performance Graphs � � � � "Algorithm" (s) "Algorithm" (s) "Algorithm" (s) "Algorithm" (s) � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� �� �� �� �� �� �� �� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� �� �� �� �� �� �� �� “Method” is � � � � � � � � faster by an � � � � � � � � order of magnitude Naive detection (s) Naive detection (s) Naive detection (s) Naive detection (s) here 15 / 25

Benchmark 1: Performance A B Local Irrelevance (s) Local Irrelevance (s) Irrelevance (s) Irrelevance (s) Naive detection (s) Naive detection (s) Naive detection (s) Naive detection (s) Execution times measured C for complete methods Peripherality (s) Peripherality (s) Naive detection (s) Naive detection (s) 16 / 25

Benchmark 1: Performance A B Local Irrelevance (s) Local Irrelevance (s) Irrelevance (s) Irrelevance (s) Naive detection (s) Naive detection (s) Naive detection (s) Naive detection (s) Execution times measured C for complete methods Peripherality (s) Peripherality (s) Peripherality is much slower in some cases Naive detection (s) Naive detection (s) 16 / 25

Why is Peripherality much slower in some cases? Naive Detection Peripherality Φ 1 = M | = p 1 Φ = M | = p Φ 2 = M | = p 2 . . . Φ n = M | = p n

Why is Peripherality much slower in some cases? Low clause/variable ratio Peripherality (s) Peripherality (s) No vacuous variables Large resolution proofs Naive detection (s) Naive detection (s) Naive Detection Peripherality Φ 1 = M | = p 1 Φ = M | = p Φ 2 = M | = p 2 . . . Φ n = M | = p n

Why is Peripherality much slower in some cases? Low clause/variable ratio Peripherality (s) Peripherality (s) No vacuous variables Large resolution proofs Naive detection (s) Naive detection (s) Naive Detection Peripherality Φ 1 = M | = p 1 Φ = M | = p UNSAT Φ 2 = M | = p 2 SAT . . . Φ n = M | = p n 17 / 25

Why is Peripherality much slower in some cases? Low clause/variable ratio Peripherality (s) Peripherality (s) No vacuous variables Large resolution proofs Naive detection (s) Naive detection (s) Naive Detection Peripherality Φ 1 = M | = p 1 Φ = M | = p UNSAT Φ 2 = M | = p 2 SAT . . . Φ n = M | = p n T time: τ 1 , τ 2 , . . . τ n to find sat. assignment periph. analysis 17 / 25

Why is Peripherality much slower in some cases? Low clause/variable ratio Peripherality (s) Peripherality (s) No vacuous variables Large resolution proofs Naive detection (s) Naive detection (s) Naive Detection Peripherality Φ 1 = M | = p 1 Φ = M | = p UNSAT Φ 2 = M | = p 2 SAT . . . Φ n = M | = p n T time: τ 1 , τ 2 , . . . τ n to find sat. assignment periph. analysis τ i <<< T 17 / 25

Why is Peripherality much slower in some cases? Low clause/variable ratio Peripherality (s) Peripherality (s) No vacuous variables Large resolution proofs Naive detection (s) Naive detection (s) Naive Detection Peripherality Φ 1 = M | = p 1 Φ = M | = p UNSAT Φ 2 = M | = p 2 SAT . . . Φ n = M | = p n T time: τ 1 , τ 2 , . . . τ n to find sat. assignment periph. analysis τ i <<< T � τ i <<< T 17 / 25

Interpreting Effectiveness Graphs Less precise algorithm �� �� � �� �� �� �� � � � � � �� �� �� �� � �� �� �� �� � � �� �� �� �� � � � � More precise algorithm 18 / 25

Interpreting Effectiveness Graphs # vacuous variables found: ( x , y ) x = found by X-axis algorithm y = found by Y-axis algorithm Less precise algorithm �� �� � �� �� �� �� � � � � � �� �� �� �� � �� �� �� �� � � �� �� �� �� � � � � More precise algorithm 18 / 25

Interpreting Effectiveness Graphs # vacuous variables found: ( x , y ) x = found by X-axis algorithm y = found by Y-axis algorithm Less precise algorithm �� �� � �� �� �� �� X-axis algorithm is more precise, � � � so x ≥ y always � � �� �� �� �� � �� �� �� �� � � �� �� �� �� � � � � More precise algorithm 18 / 25

Interpreting Effectiveness Graphs # vacuous variables found: ( x , y ) x = found by X-axis algorithm y = found by Y-axis algorithm Less precise algorithm �� �� � �� �� �� �� X-axis algorithm is more precise, � � � so x ≥ y always � � �� �� �� �� � �� �� �� �� � � �� �� �� �� � � � � Larger point = more test cases More precise algorithm 18 / 25

Benchmark 1: Effectiveness A B C Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Irrelevance Irrelevance Irrelevance Irrelevance Irrelevance Irrelevance Irrelevance Irrelevance ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Naive detection Naive detection Naive detection Naive detection Naive detection Naive detection Naive detection Naive detection Naive detection 19 / 25

Benchmark 1: Effectiveness A B C Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Irrelevance Irrelevance Irrelevance Irrelevance Irrelevance Irrelevance Irrelevance Irrelevance ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Naive detection Naive detection Naive detection Naive detection Naive detection Naive detection Naive detection Naive detection Naive detection Reduced # of extra model checking runs: ≥ 40% reduction in 54% of cases with vacuity 19 / 25

Benchmark 1: Effectiveness A B C Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Irrelevance Irrelevance Irrelevance Irrelevance Irrelevance Irrelevance Irrelevance Irrelevance ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% ≥ 40% Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Local Irrelevance Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Peripherality Naive detection Naive detection Naive detection Naive detection Naive detection Naive detection Naive detection Naive detection Naive detection Reduced # of extra model checking runs: ≥ 40% reduction in 54% of cases with vacuity Local Irrelevance (s) Local Irrelevance (s) Local Irrelevance is faster than Naive Detection in 70 cases (59%): Twice as fast in 40% of these cases Order of magnitud faster in 30% of these cases Naive detection (s) Naive detection (s) 19 / 25

Benchmark 2 G OAL : evaluate scalability of our tool to industrial models S ETUP Models and properties: IBM Formal Verification Benchmarks Library 18 properties: 12 present vacuity 1 temporal operator, from { G, F } 4 variables on average, 17 max., 1 min. Picked k-depth in line with bounds used in Benchmark 1 Largest proof: 500k resolutions 20 / 25

Benchmark 2 G OAL : evaluate scalability of our tool to industrial models S ETUP Models and properties: IBM Formal Verification Benchmarks Library 18 properties: 12 present vacuity 1 temporal operator, from { G, F } 4 variables on average, 17 max., 1 min. Picked k-depth in line with bounds used in Benchmark 1 Largest proof: 500k resolutions Proof sizes are in same range as those for Benchmark 1 new models are more complex but properties are simpler 20 / 25

Benchmark 2: Scalability A B C Local Irrelevance (s) Local Irrelevance (s) Peripherality (s) Peripherality (s) Irrelevance (s) Irrelevance (s) Naive detection (s) Naive detection (s) Naive detection (s) Naive detection (s) Naive detection (s) Naive detection (s) Reasonable execution times No noticeable spike in peripherality execution times models with low clause/variable ratio present vacuity proofs for these models are medium-sized Little vacuity in this suite, yet algorithms detect some vacuity 21 / 25

Experimental Conclusions Benchmark 1 Benchmark 2 Models Simple Complex Properties Complex Simple Irrelevance Very fast Very fast Local Irrelevance Fastest Fastest Peripherality Slow in certain cases Very fast Our algorithms: discover vacuous variables . . . via relatively inexpensive analyses of BMC artifacts Our methods are complete and generally faster than Naive Detection 22 / 25

Summary Vacuity detection for BMC we analyze BMC artifacts like UNSAT cores and resolution proofs Proposed and implemented a vacuity detection tool, V AQ T REE 23 / 25

Summary Vacuity detection for BMC we analyze BMC artifacts like UNSAT cores and resolution proofs Proposed and implemented a vacuity detection tool, V AQ T REE Step towards making vacuity detection part of complete process 23 / 25

Future Work When do our algorithms apply? heuristics based on clause/variable ratio and proof size Increase scalability of our tool implement on-the-fly proof analysis Use interpolants for vacuity detection Use results of previous depths for vacuity detection 24 / 25