Programming Not Only by Example Hila Peleg September 2017 Joint - PowerPoint PPT Presentation

Programming Not Only by Example Hila Peleg September 2017 Joint work with Sharon Shoham and Eran Yahav The research leading to these results has received funding from the European Union's - Seventh Framework Programme (FP7) under grant agreement n° 615688 – ERC- COG-PRIME. Art from xkcd.com by Randall Munroe, licensed under Creative Commons

Program Synthesis • The problem of producing code that will I need code satisfy a (usually) partial specifications that will solve • Many existing works are aimed at end-users my problem performing simple repetitive tasks • Others produce code meant to be executed • These all require human-synthesizer interaction, often iteratively

Programming by Example An example of the Find the most desired behavior: frequent bigram Synthesize! in a string • Input: "a bd fibfcfde bd fd e bd ihgfkjfde bd " • Output: "bd"

Many examples are inherently ambiguous • PBE aims for Input: Wait, that’s "a bd fibfcfde bd fd consistency with not what I e bd ihgfkjfde bd " examples meant! • Examples don’t Output: "bd" convey intent uniquely input • In other words: .takeRight(2) overfitting

Problem I: Finding a differentiating example is hard • Both a counterexample and an Drat! Still consistent example with the bad program! • Implicitly convey explicit knowledge • Usually takes a RTE and some time

Problem II: Examples are terrible at directing search input I want it to never use Input: "abbba" min again //ab,bd,df, … Output: "bb" Impossible. .sliding(2) Even if we give an example //bd,df,fi ,… input//abbba where it’s the .sliding(2)//ab,bb ,… maximum? .drop(1) .drop(1)//bb,bb,ba //bd .dropRight(1)//bb,bb .min .min //bb

Problem II: An impossibility result • Claim: If f is a function in the vocabulary, f cannot be eliminated from the search space Aargh! I just need by examples alone it to not use that • And in practice, undesirable elements are all anymore! over the search space – they come back • As often as half the session! • The interaction model is too lean: • You can see the problem • But you can’t communicate the problem • Longer search • But also frustration

Granularity of the interaction • Programmers can interact on a lower This part looks fine… but that’s level clearly wrong. • They understand sub-problems • They understand the code • (They can be helped to understand) • They should be given the power

The Granular Interaction Model (GIM) • A programmer can talk at input //ab,bd,df, … the level of the program That looks right .sliding(2) • Read debug info //bd,df,fi ,… • Reason about subtrees or .drop(1) sequences of methods • Or intermediate states //bd Those are wrong .min • But also examples, if those happen to be easier

Back to our example Input: Find the most The user can "a bd fibfcfde bd fd frequent bigram answer locally: e bd ihgfkjfde bd " in a string exclude takeRight(2) Output: "bd" input .takeRight(2)

Another step The synthesizer prunes the search space and produces another answer input //"abdfibfcfdebdfdebdihgfkjfdebd" .drop(1) //"bdfibfcfdebdfdebdihgfkjfdebd" .take(2) //"bd" Answer: exclude drop(1 )∙take( 2)

The synthesizer answers input//"abdfibfcfdebdfdebdihgfkjfdebd" .zip(input.drop(1))//List((a,b),(b,d),(d,f),(f,i),...) .take(2)//List((a,b),(b,d)) .map(p => p._1.toString + p._2)//List("ab","bd") .max//"bd" User provides a compound answer: retain zip(input.drop(1)) exclude take(2) And possibly even retain map(p => p._1.toString + p._2)

Until finally input//"abdfibfcfdebdfdebdihgfkjfdebd" .zip(input.drop(1))//List((a,b),(b,d),(d,f),(f,i ),… .map(p => p._1.toString + p._2)//List("ab","bd",... .groupBy(x => x)//Map("bf"->List("bf"),"ib"-> List("ib"),... .map(kv => kv._1 -> kv._2.length)//Map("bf"->1, "ib"->1,... .maxBy(_._2)//("bd",4) ._1//"bd"

Can this also help the synthesizer? • “Well - suited” feedback Dammit, GIM, I’m • To the domain a programmer, • And to the synthesizer not a gardener! • For example: functional concatenations are trees • Help prune the state

The usability of GIM • We studied GIM in the wild (32 users, Can your data industry/academia) fully reflect my • Users loved the debug information frustration? • Total synthesis session time is the same Well… • But each iteration was shorter than PBE • Users spent less time fighting “distracting” sub-programs

Users like examples (but not that much) 12 100% Reached correct answer (no. users) 90% 10 Portion of examples (%) 80% 70% 8 60% 50% 6 40% 4 30% 20% 2 10% 0% 0 histogram no. lines most histogram no. lines most histogram no. lines most with text frequent with text frequent with text frequent PBE Syntax GIM PBE Syntax GIM PBE Syntax GIM word word word histogram no. lines with text most frequent word all users familiar with Scala not familiar with Scala Reached target answer