GuidedSampler: Coverage-guided Sampling of SMT Solutions Rafael - PowerPoint PPT Presentation

GuidedSampler: Coverage-guided Sampling of SMT Solutions Rafael Dutra, Jonathan Bachrach, Koushik Sen EECS Department UC Berkeley Formal Methods in Computer-Aided Design October 25, 2019

Constraint Sampling Input: SMT formula φ A SMT solver can generate one solution: ∧ (x + y = 4 ∧ x ≥ 0 ∧ x < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), mem[0] mem[1] where σ 0 1 0 0 0 1 0 0 0 x = mem[0], y = mem[1], mem’ = store(mem, mem[0], -1 * mem[mem[0]]) 2

Constraint Sampling Input: SMT formula φ Goal: Generate many solutions to φ ∧ (x + y = 4 ∧ x ≥ 0 ∧ x < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), mem[0] mem[1] where σ 0 1 0 0 0 1 0 0 0 x = mem[0], σ 1 0 0 0 0 1 0 1 0 y = mem[1], mem’ = store(mem, mem[0], -1 * mem[mem[0]]) σ 2 1 1 0 0 1 0 0 1 σ 3 1 0 0 0 0 1 0 0 σ 4 0 1 0 0 1 0 1 1 σ 5 0 0 0 0 0 1 1 0 3

Motivation: Sampling Solutions ● Thoroughly exercising some target functionality ● Synthesis ● Constrained-Random Verification ● Symbolic execution int4 x, y, z, w; i < 4 int4 mem[4] = {x, y, z, w}; mem[0] < 0 for (int4 i = 0; i < 4; ++i) { ∨ mem[0] ≥ 4 mem[mem[i]] *= -1; } 4

SMT: Satisfiability Modulo Theories SMT formula φ ∧ (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1 * mem[mem[0]]) mem ∈ Array(BV[4], BV[4]) 5

SMT: Satisfiability Modulo Theories SMT formula φ Bit-vector ∧ (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1 * mem[mem[0]]) mem ∈ Array(BV[4], BV[4]) 6

SMT: Satisfiability Modulo Theories SMT formula φ Bit-vector ∧ (mem[0] ≥ 0 ∧ mem[0] < 4) Array ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1 * mem[mem[0]]) mem ∈ Array(BV[4], BV[4]) 7

State of the art ● SMTSampler (our prior work) ○ Efficient generation of solutions for SMT formulas Markov Chain Monte Carlo (MCMC) ● ○ Works for linear constraints and can generate biased solutions Constraint solver heuristics ● ○ Can be expensive, requiring one solver call per solution ● Universal hashing Expensive, but can guarantee exact distribution of solution ○ ● Weighted Sampling ○ Literal-weighted distributions: WAPS 8

SMTSampler Input: SMT formula φ Goal: Generate solutions to φ ∧ (x + y = 4 ∧ x ≥ 0 ∧ x < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), mem[0] mem[1] where σ 0 1 0 0 0 1 0 0 0 x = mem[0], σ 1 0 0 0 0 1 0 1 0 y = mem[1], mem’ = store(mem, mem[0], -1 * mem[mem[0]]) σ 2 1 1 0 0 1 0 0 1 σ 3 1 0 0 0 0 1 0 0 σ 4 0 1 0 0 1 0 1 1 σ 5 0 0 0 0 0 1 1 0 9

Coverage-guided Sampling Input: SMT formula φ Goal: Generate solutions to φ ∧ (x + y = 4 ∧ x ≥ 0 ∧ x < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), mem[0] mem[1] where σ 0 1 0 0 0 1 0 0 0 x = mem[0], σ 1 0 0 0 0 1 0 1 0 y = mem[1], mem’ = store(mem, mem[0], -1 * mem[mem[0]]) σ 2 1 1 0 0 1 0 0 1 Input: Coverage predicates σ 3 1 0 0 0 0 1 0 0 mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 σ 4 0 1 0 0 1 0 1 1 ψ 1 ψ 2 ψ 3 σ 5 0 0 0 0 0 1 1 0 10

Coverage-guided Sampling Input: SMT formula φ Goal: Generate solutions to φ such that the predicates ψ 1 , ψ 2 , …, ψ n are covered uniformly ∧ (x + y = 4 ∧ x ≥ 0 ∧ x < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), mem[0] mem[1] ψ 1 ψ 2 ψ 3 where σ 0 1 0 0 0 1 0 0 0 0 1 0 x = mem[0], σ 1 0 0 0 0 1 0 1 0 0 1 0 y = mem[1], mem’ = store(mem, mem[0], -1 * mem[mem[0]]) σ 2 1 1 0 0 1 0 0 1 0 0 1 Input: Coverage predicates σ 3 1 0 0 0 0 1 0 0 1 1 0 mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 σ 4 0 1 0 0 1 0 1 1 0 1 0 ψ 1 ψ 2 ψ 3 σ 5 0 0 0 0 0 1 1 0 1 1 0 11

Coverage-guided Sampling Input: SMT formula φ Goal: Generate solutions to φ such that the predicates ψ 1 , ψ 2 , …, ψ n are covered uniformly ∧ (x + y = 4 ∧ x ≥ 0 ∧ x < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), mem[0] mem[1] ψ 1 ψ 2 ψ 3 where σ 0 1 0 0 0 1 0 0 0 0 1 0 x = mem[0], σ 1 0 0 0 0 1 0 1 0 0 1 0 y = mem[1], mem’ = store(mem, mem[0], -1 * mem[mem[0]]) σ 2 1 1 0 0 1 0 0 1 0 0 1 Input: Coverage predicates σ 3 1 0 0 0 0 1 0 0 1 1 0 mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 σ 4 0 1 0 0 1 0 1 1 0 1 0 ψ 1 ψ 2 ψ 3 σ 5 0 0 0 0 0 1 1 0 1 1 0 12

Uniformity over Coverage Classes 13

Uniformity over Coverage Classes 14

Uniformity over Coverage Classes 15

Challenges ● Coverage of the formula might still not be ideal even using state-of-the-art approaches, such as SMTSampler User might be interested in a specific notion of coverage for the produced ● solutions 16

GuidedSampler 17

GuidedSampler Our goals: Our approach: ● Sample solutions from a formula φ , but ● Compute simple mutations that can be have the distribution determined by the applied to one solution to generate coverage predicates ψ 1 , ψ 2 , …, ψ n another solution from a different class Uniformly sample solutions from the Combine those mutations together to ● ● different coverage classes generate a large number of new ● Uniformly sample within each solutions coverage class 18

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) Formula φ Coverage Predicates ψ 1 mem’[1] < 0 ψ 2 mem’[1] ≥ 4 ψ 3 mem’[0] < 0 ... 19

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) x0 x1 x2 x3 y0 y1 y2 y3 Formula φ Coverage Predicates x = mem[0] Random assignment σ’ 1 0 1 0 1 0 0 0 ψ 1 y = mem[1] mem’[1] < 0 ψ 2 mem’[1] ≥ 4 ψ 3 mem’[0] < 0 ... 20

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) ψ 1 ψ 2 ψ 3 x0 x1 x2 x3 y0 y1 y2 y3 Formula φ Coverage Predicates x = mem[0] Random assignment σ’ 1 0 1 0 1 0 0 0 0 1 1 Random ψ 1 y = mem[1] mem’[1] < 0 Class ψ 2 mem’[1] ≥ 4 ψ 3 mem’[0] < 0 ... 21

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) ψ 1 ψ 2 ψ 3 x0 x1 x2 x3 y0 y1 y2 y3 Formula φ Coverage Predicates x = mem[0] Random assignment σ’ 1 0 1 0 1 0 0 0 0 1 1 Random ψ 1 y = mem[1] mem’[1] < 0 MAX-SMT ψ 1 ψ 2 ψ 3 Class ψ 2 mem’[1] ≥ 4 Solution σ 1 0 0 0 1 0 0 0 0 1 0 ψ 3 mem’[0] < 0 ... 22

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) x0 x1 x2 x3 y0 y1 y2 y3 Formula φ Coverage Predicates x = mem[0] Random assignment σ’ 1 0 1 0 1 0 0 0 ψ 1 y = mem[1] mem’[1] < 0 MAX-SMT ψ 1 ψ 2 ψ 3 ψ 2 mem’[1] ≥ 4 Solution σ 1 0 0 0 1 0 0 0 0 1 0 ψ 3 mem’[0] < 0 ... MAX-SMT 23

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) x0 x1 x2 x3 y0 y1 y2 y3 Formula φ Coverage Predicates x = mem[0] Random assignment σ’ 1 0 1 0 1 0 0 0 ψ 1 y = mem[1] mem’[1] < 0 MAX-SMT ψ 1 ψ 2 ψ 3 ψ 2 mem’[1] ≥ 4 Solution σ 1 0 0 0 1 0 0 0 0 1 0 ψ 3 mem’[0] < 0 ... MAX-SMT Soft constraints Hard constraints ● ψ 2 = 1 φ ● ψ 3 = 0 ● ● ψ 1 ≠ 0 ● x1 = 0 ● x2 = 0 ● x3 = 0 ● y0 = 1 ● y1 = 0 ● y2 = 0 ● y3 = 0 24

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) x0 x1 x2 x3 y0 y1 y2 y3 Formula φ Coverage Predicates x = mem[0] Random assignment σ’ 1 0 1 0 1 0 0 0 ψ 1 y = mem[1] mem’[1] < 0 MAX-SMT ψ 1 ψ 2 ψ 3 ψ 2 mem’[1] ≥ 4 Solution σ 1 0 0 0 1 0 0 0 0 1 0 ψ 3 mem’[0] < 0 ... MAX-SMT 0 0 0 0 1 0 1 0 1 1 0 σ 1 25

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) x0 x1 x2 x3 y0 y1 y2 y3 Formula φ Coverage Predicates x = mem[0] Random assignment σ’ 1 0 1 0 1 0 0 0 ψ 1 y = mem[1] mem’[1] < 0 MAX-SMT ψ 1 ψ 2 ψ 3 ψ 2 mem’[1] ≥ 4 Solution σ 1 0 0 0 1 0 0 0 0 1 0 ψ 3 mem’[0] < 0 ... MAX-SMT 0 0 0 0 1 0 1 0 σ 1 26

(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) x0 x1 x2 x3 y0 y1 y2 y3 Formula φ Coverage Predicates x = mem[0] Random assignment σ’ 1 0 1 0 1 0 0 0 ψ 1 y = mem[1] mem’[1] < 0 MAX-SMT ψ 1 ψ 2 ψ 3 ψ 2 mem’[1] ≥ 4 Solution σ 1 0 0 0 1 0 0 0 0 1 0 ψ 3 mem’[0] < 0 ... MAX-SMT 0 0 0 0 1 0 1 0 σ 1 27