guarantees in in program synthesis
play

Guarantees in in Program Synthesis Qinheping Hu , Jason Breck , - PowerPoint PPT Presentation

Guarantees in in Program Synthesis Qinheping Hu , Jason Breck , John Cyphert , Loris D'Antoni , Thomas Reps Program Synthesis Specification Solution Program Synthesizer Search space Program Synthesis is Un Unpredicta edictable ble


  1. Guarantees in in Program Synthesis Qinheping Hu , Jason Breck , John Cyphert , Loris D'Antoni , Thomas Reps

  2. Program Synthesis Specification Solution Program Synthesizer Search space Program Synthesis is Un Unpredicta edictable ble

  3. Program Synthesis is Un Unpredictable edictable Search space Solution space

  4. Program Synthesis is Un Unpredictable edictable Specification Solution Program Synthesizer Search space Ability to prefer a solution when there are multiple solution

  5. Program Synthesis is Un Unpredictable edictable Specification: 𝑔 𝑦 = 𝑦 Solution space Search space constant programs: 1,2,3,…

  6. Program Synthesis is Un Unpredictable edictable Solution Program ? Search-based synthesizer timeout Solution space Search space

  7. Gu Guarantees ntees in Program Synthesis Specification Solution Program Synthesizer Search space Ability to answer no solutions Ability to prefer a solution when there are multiple solution Make program synthesis predictable

  8. Syntax-Guided Synthesis with Quantitative Objectives [CAV18]

  9. Syntax-Guided Synthesis (SyGuS) πœ’ 𝑛𝑏𝑦(𝑦, 𝑧), 𝑦, 𝑧 : 𝑛𝑏𝑦 𝑦, 𝑧 β‰₯ 𝑦 ∧ 𝑛𝑏𝑦 𝑦, 𝑧 β‰₯ 𝑧 ∧ (𝑛𝑏𝑦 𝑦, 𝑧 = 𝑦 ∨ 𝑛𝑏𝑦 𝑦, 𝑧 = 𝑧) Specification Solution Program Synthesizer Search space 𝑓 ∈ 𝑀(𝐻) such that βˆ€π‘¦, 𝑧. πœ’ 𝑓, 𝑦, 𝑧 Start = +(Start, Start) | π½π‘ˆπΉ(BExpr, Start, Start) 𝑦 𝑧 0 1 max 𝑦, 𝑧 = π½π‘ˆπΉ(> (𝑦, 𝑧), 𝑦, π½π‘ˆπΉ(< 𝑦, 𝑧 , 𝑧, 𝑦) BExpr = 𝑂𝑝𝑒(BExpr ) | > (Start, Start) |π΅π‘œπ‘’(BExpr, BExpr)

  10. A limitation of SyGuS What we expected Outp tput of f th the CVC4 solv lver

  11. Features you want Optim imization Obje jectives Readable Smallest size Efficient Least number of multiplication Most likely Largest probability

  12. Gu Guarantees ntees in SyGuS Specification Solution Program SyGuS solver Search space Quantitative objectives

  13. Adding Quantitative Objective to SyGuS Start = Start + Start BExpr = Start > Start | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start | π‘œπ‘π‘’ BExpr x 𝑧 0 1 | BExpr π‘π‘œπ‘’ BExpr

  14. Adding Quantitative Objective to SyGuS Start = Start + Start BExpr = Start > Start | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start | π‘œπ‘π‘’ BExpr x 𝑧 0 1 | BExpr π‘π‘œπ‘’ BExpr Quantitative objective: Minimize number of if-statement

  15. Adding Quantitative Objective to SyGuS Start = Start + Start /0 BExpr = Start > Start /0 | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start /1 | π‘œπ‘π‘’ BExpr /0 x/0 𝑧/0 0/0 1/0 | BExpr π‘π‘œπ‘’ BExpr /0 Quantitative objective: Minimize number of if-statement

  16. ITE:1 If(x>y) ITE:1 >:0 Y:0 then = if(x > x) then 0 else x Else y X:0 Y:0 X:0 >:0 0:0 X:0 X:0 Weight = 2

  17. Adding Quantitative Objective to SyGuS Weighted grammar Start = Start + Start /0 BExpr = Start > Start /0 | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start /1 | π‘œπ‘π‘’ BExpr /0 x/0 𝑧/0 0/0 1/0 | BExpr π‘π‘œπ‘’ BExpr /0 Quantitative objective: Minimize number of if-statement

  18. Adding Quantitative Objective to SyGuS Weighted grammar Start = Start + Start /0 BExpr = Start > Start /0 | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start /1 | π‘œπ‘π‘’ BExpr /0 x/0 𝑧/0 0/0 1/0 | BExpr π‘π‘œπ‘’ BExpr /0 Quantitative objective: Minimize number of if-statement Minimize weight QSyGuS: (specification,weighted grammar, Quantitative objective)

  19. Solving QSyGuS

  20. QSyGuS WTG 𝑋 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’

  21. SyGuS QSyGuS ignore WTG 𝑋 weight CFG 𝐻 Constraint 𝜚 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’

  22. SyGuS QSyGuS Solution’s WTG 𝑋 weight 𝑑 1 CFG 𝐻 CFG 𝐻 <𝑑 1 Constraint 𝜚 Constraint 𝜚 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’

  23. SyGuS QSyGuS Solution’s WTG 𝑋 weight 𝑑 2 CFG 𝐻 CFG 𝐻 <𝑑 1 CFG 𝐻 <𝑑 2 Constraint 𝜚 Constraint 𝜚 Constraint 𝜚 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’

  24. SyGuS QSyGuS Solution’s WTG 𝑋 weight 𝑑 2 CFG 𝐻 CFG 𝐻 <𝑑 1 CFG 𝐻 <𝑑 2 β‹― Constraint 𝜚 Constraint 𝜚 Constraint 𝜚 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’

  25. SyGuS QSyGuS Solution’s WTG 𝑋 weight 2 CFG 𝐻 CFG 𝐻 <2 Constraint 𝜚 Constraint 𝜚 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’ CFG 𝐻 <2 WTG 𝑋 Program 𝑄 has π‘₯π‘“π‘—π‘•β„Žπ‘’ < 2 iff 𝐻 <2 accept 𝑄

  26. WTG 𝑋 CFG 𝐻 <2 Grammar Reduction Idea: keep tr Id track of f th the weight in in th the non-terminals Start = Start + Start /0 BExpr = Start > Start /0 | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start /1 | π‘œπ‘π‘’ BExpr /0 x/0 𝑧/0 0/0 1/0 | BExpr π‘π‘œπ‘’ BExpr /0 π‘₯π‘“π‘—π‘•β„Žπ‘’ < 2

  27. WTG 𝑋 CFG 𝐻 <2 Grammar Reduction Id Idea: keep tr track of f th the weight in in th the non-terminals Start = Start + Start /0 BExpr = Start > Start /0 | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start /1 | π‘œπ‘π‘’ BExpr /0 x/0 𝑧/0 0/0 1/0 | BExpr π‘π‘œπ‘’ BExpr /0 π‘₯π‘“π‘—π‘•β„Žπ‘’ < 2 (Start, < 2) = Start, 0 | (Start, 1)

  28. WTG 𝑋 CFG 𝐻 <2 Grammar Reduction Idea: keep tr Id track of f th the weight in in th the non-terminals Start = Start + Start /0 BExpr = Start > Start /0 | 𝑗𝑔 BExpr π‘’β„Žπ‘“π‘œ Start π‘“π‘šπ‘‘π‘“ Start /1 | π‘œπ‘π‘’ BExpr /0 x/0 𝑧/0 0/0 1/0 | BExpr π‘π‘œπ‘’ BExpr /0 π‘₯π‘“π‘—π‘•β„Žπ‘’ < 2 (Start, < 2) = Start, 0 | (Start, 1) Start, 1 = Start, 0 + Start, 1 | Start, 1 + (Start, 0) | 𝑗𝑔 BExpr, 0 π‘’β„Žπ‘“π‘œ (Start, 0) π‘“π‘šπ‘‘π‘“ (Start, 0) Start, 0 = Start, 0 + Start, 0 𝑦 𝑧 0 1 β‹―

  29. Handling complex weight constraints Tree grammars are closed under Boolean operations Minimization Minimization Linear search 3 < π‘₯π‘“π‘—π‘•β„Žπ‘’ Complement of 𝐻 <4 𝐻 <5 ∩ 𝐻 >2 2 < π‘₯π‘“π‘—π‘•β„Žπ‘’ < 5 𝐻 π‘₯π‘“π‘—π‘•β„Žπ‘’ 1 >3 ∩ 𝐻 π‘₯π‘“π‘—π‘•β„Žπ‘’ 2 <0.5 3 < π‘₯π‘“π‘—π‘•β„Žπ‘’ 1 and π‘₯π‘“π‘—π‘•β„Žπ‘’ 2 < 0.5

  30. Evaluation

  31. Evaluation SyGuS QSyGuS WTG 𝑋 CFG 𝐻 CFG 𝐻 <𝑑 1 CFG 𝐻 <𝑑 2 β‹― Constraint 𝜚 Constraint 𝜚 Constraint 𝜚 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’ SyGuS solvers ESolver CVC4

  32. Evaluation 26 Benchmarks taken from SyGuS 1.minimize number of specified operator, minimize solution size 2.maximize solution probability 3.find sorted optimal for (# of specified operators, size) 4.find Pareto optimal for (# of specified operators, size) Find solution with better cost for 16/26 SyGuS benchmarks Find optimal in 14/26 (couldn’t prove optimality for 2 benchmarks) Average time 3.1x Compared to SyGuS

  33. Conclusion SyGuS QSyGuS Solution’s WTG 𝑋 weight 𝐷 2 CFG 𝐻 CFG 𝐻 <𝑑 1 CFG 𝐻 <𝑑 2 β‹― Constraint 𝜚 Constraint 𝜚 Constraint 𝜚 Constraint 𝜚 Minimize π‘₯π‘“π‘—π‘•β„Žπ‘’ optimal unrealizable

  34. Gu Guarantees ntees in SyGuS Specification Solution Program SyGuS solver Search space Ability to answer no solutions Answering unrealizable Ability to prefer a solution when there are multiple solution Quantitative objectives

  35. Tue 12:10 come to my CAV talk Proving Unrealizability for Syntax-Guided Synthesis [CAV19]

  36. A Syntax-Guided Synthesis (SyGuS) is Specification Search space G: πœ’ 𝑔(𝑦, 𝑧), 𝑦, 𝑧 : Start = +(Start, Start) 𝑔 𝑦, 𝑧 β‰₯ 𝑦 ∧ | π½π‘ˆπΉ(BExpr, Start, Start) 𝑔 𝑦, 𝑧 β‰₯ 𝑧 ∧ 𝑦 𝑧 0 1 (𝑔 𝑦, 𝑧 = 𝑦 ∨ 𝑔 𝑦, 𝑧 = 𝑧) BExpr = 𝑂𝑝𝑒(BExpr ) | > (Start, Start) |π΅π‘œπ‘’(BExpr, BExpr) Goal: find a program 𝑓 ∈ 𝑀(𝐻) such that βˆ€π‘¦, 𝑧. πœ’ 𝑓, 𝑦, 𝑧 max 𝑦, 𝑧 = π½π‘ˆπΉ(> (𝑦, 𝑧), 𝑦, 𝑧)

  37. Unrealizable SyGuS Problems Start = +(Start, Start) 𝑦 𝑧 0 1 βˆ€π‘¦, 𝑧. max 𝑦, 𝑧 β‰₯ 𝑦 ∧ max 𝑦, 𝑧 β‰₯ 𝑧 ∧ (max 𝑦, 𝑧 = 𝑦 ∨ max 𝑦, 𝑧 = 𝑧) No Solution

  38. CEGIS-based Frameowrk Nope ESolver Example Set 𝑄 𝐹 UNSAT SAT 𝑄 is a solution Verifier Z3 new example e

  39. CEGIS-based Frameowrk Nope 𝑆𝑓 𝐹 Unreachable Seahorn Reduction UNREALIZABLE ESolver Example Set 𝑄 𝐹 UNSAT SAT 𝑄 is a solution Verifier Z3 new example e

  40. SyGuS is unrealizable ↔ reachability problem 𝑆𝑓 𝐹 is unsatisfiable Reachability Verifier 𝑆𝑓 𝐹 Unreachable Seahorn Reduction UNREALIZABLE

  41. Gu Guarantees ntees in Program Synthesis Specification Solution Program Synthesizer Search space Ability to answer no solutions Ability to prefer a solution Answering unrealizable when there are multiple solution 1. Tue 12:10 come to my talk More quantitative objectives 2. Beyond SyGuS 1.Semantic quantitative objectives 2.Resource bounded synthesis

Recommend


More recommend