learning to format coq code using language models
play

Learning to Format Coq Code Using Language Models Pengyu Nie 1 , - PowerPoint PPT Presentation

Dec 04, 2022 •228 likes •487 views

Learning to Format Coq Code Using Language Models Pengyu Nie 1 , Karl Palmskog 2 , Junyi Jessy Li 1 , and Milos Gligoric 1 The Coq Workshop 2020 1 The University of Texas at Austin 2 KTH Royal Institute of Technology Background: Coq is a

  1. Learning to Format Coq Code Using Language Models Pengyu Nie 1 , Karl Palmskog 2 , Junyi Jessy Li 1 , and Milos Gligoric 1 The Coq Workshop 2020 1 The University of Texas at Austin 2 KTH Royal Institute of Technology

  2. Background: Coq is a Language Platform Coq extensibility has provided us with a linguistic zoo: libraries: MathComp, Stdpp, TLC, Stdlib, ... tactic and proof languages: Ltac, Ltac2, Mtac2, SSReflect, ... embedded languages: Verifiable C, RustBelt, MetaCoq, ... 2 / 24

  3. Example: Coq/SSReflect/MathComp Lemma totient_coprime m n : coprime m n -> totient (m * n) = totient m * totient n. Proof. move=> co_mn; have [-> //| m_gt0] := posnP m. have [->|n_gt0] := posnP n; first by rewrite !muln0. rewrite !totientE ?muln_gt0 ?m_gt0 //. have /(perm_big _)->: perm_eq (primes (m * n)) (primes m ++ primes n). apply: uniq_perm => [||p]; first exact: primes_uniq. by rewrite cat_uniq !primes_uniq -coprime_has_primes // co_mn. by rewrite mem_cat primes_mul. rewrite big_cat /= !big_seq. congr (_ * _); apply: eq_bigr => p; rewrite mem_primes => /and3P[_ _ dvp]. rewrite (mulnC m) logn_Gauss //; move: co_mn. by rewrite -(divnK dvp) coprime_mull => /andP[]. rewrite logn_Gauss //; move: co_mn. by rewrite coprime_sym -(divnK dvp) coprime_mull => /andP[]. Qed. 3 / 24

  4. Example: Coq/Ltac/Stdpp Lemma list_find_app_Some l1 l2 i x : list_find P (l1 ++ l2) = Some (i,x) ↔ list_find P l1 = Some (i,x) ∨ length l1 ≤ i ∧ list_find P l1 = None ∧ list_find P l2 = Some (i - length l1,x). Proof. split. - intros ([?|[??]]%lookup_app_Some&?&Hleast)%list_find_Some. + left. apply list_find_Some; eauto using lookup_app_l_Some. + right. split; [lia|]. split. { apply list_find_None, Forall_lookup. intros j z ??. assert (j < length l1) by eauto using lookup_lt_Some. naive_solver eauto using lookup_app_l_Some with lia. } apply list_find_Some. split_and!; [done..|]. intros j z ??. eapply (Hleast (length l1 + j)); [|lia]. by rewrite lookup_app_r, minus_plus by lia. - intros [(?&?&Hleast)%list_find_Some|(?&Hl1&(?&?&Hleast)%list_find_Some)]. + apply list_find_Some. split_and!; [by auto using lookup_app_l_Some..|]. assert (i < length l1) by eauto using lookup_lt_Some. intros j y ?%lookup_app_Some; naive_solver eauto with lia. + rewrite list_find_Some, lookup_app_Some. split_and!; [by auto..|]. intros j y [?|?]%lookup_app_Some ?; [|naive_solver auto with lia]. by eapply (Forall_lookup_1 (not o P) l1); [by apply list_find_None|..]. Qed. 4 / 24

  5. Example: Coq/Ltac/Stdlib Lemma sec_left_sum_tree (X Y:Set) (p : WFT X): forall (A : X -> X -> Prop), SecureBy A p -> SecureBy (left_sum_lift A) (left_sum_tree Y p). induction p. intros A Zsec. simpl in *. intros v w x y z. destruct x; (repeat (auto; firstorder)). destruct v; (repeat (auto; firstorder)). destruct w; (repeat (auto; firstorder)). destruct v; (repeat (auto; firstorder)). destruct w; (repeat (auto; firstorder)). intros. simpl. intro x. destruct x; repeat auto. eapply sec_strengthen. Focus 2. apply H. apply H0. intros. destruct x0; repeat (auto; firstorder). destruct y; repeat (auto; firstorder). simpl in *. intro x. destruct x; repeat (auto; firstorder). eapply sec_strengthen. Focus 2. apply H. apply H0. intros. destruct x0; repeat (auto;firstorder). destruct y0; repeat (auto;firstorder). Defined. 5 / 24

  6. Problem: Users Need Help to Follow Coding Conventions coding conventions are important in large/medium sized Coq projects but, writing fully idiomatic Coq/SSReflect takes months of training ... ... and doesn’t generalize to projects using Stdpp or CompCert reading contribution guidelines is no substitute for expert feedback! 6 / 24

  7. Enforcing Conventions: Coq’s Beautifier ( make beautify ) Lemma sec_left_sum_tree (X Y : Set) (p : WFT X) : forall A : X -> X -> Prop, SecureBy A p -> SecureBy (left_sum_lift A) (left_sum_tree Y p).(induction p).( intros A Zsec).( simpl in *).( intros v w x y z).(destruct x; repeat (auto; firstorder)).(destruct v; repeat (auto; firstorder)).(destruct w; repeat (auto; firstorder)).(destruct v; repeat (auto; firstorder)).(destruct w; repeat (auto; firstorder)).( intros).( simpl). intro x.(destruct x; repeat auto).(eapply sec_strengthen).Focus 2.(apply H).(apply H0).( intros).(destruct x0; repeat (auto; firstorder)).(destruct y; repeat (auto; firstorder)).( simpl in *). intro x.(destruct x; repeat (auto; firstorder)).(eapply sec_strengthen).Focus 2.(apply H).(apply H0).( intros).(destruct x0; repeat (auto; firstorder)).(destruct y0; repeat (auto; firstorder)).Defined. 7 / 24

  8. Enforcing Conventions: SerAPI’s Pretty-Printer Lemma sec_left_sum_tree (X Y : Set) (p : WFT X) : forall A : X -> X -> Prop, SecureBy A p -> SecureBy (left_sum_lift A) (left_sum_tree Y p). (induction p). (intros A Zsec). (simpl in *). (intros v w x y z). (destruct x; repeat (auto; firstorder)). (destruct v; repeat (auto; firstorder)). (destruct w; repeat (auto; firstorder)). (destruct v; repeat (auto; firstorder)). (destruct w; repeat (auto; firstorder)). (intros). (simpl). intro x. (destruct x; repeat auto). (eapply sec_strengthen). Focus 2. (apply H). (apply H0). (intros). (destruct x0; repeat (auto; firstorder)). (destruct y; repeat (auto; firstorder)). (* ... more of the same ... *) Defined. 8 / 24

  9. Pros and Cons of Rule-Based Linting + simple and fast + easy to integrate into development process - addresses small subset of all conventions - tedious to define new rules - will never support all Coq languages 9 / 24

  10. A Flexible Alternative: Naturalness and Language Models Coq code has high naturalness , i.e., repetitions and patterns naturalness of code can be exploited in language models language models summarize statistical properties of code there are already Java formatters/analyzers using naturalness 10 / 24

  11. Our Message to the Coq Community rule-based linters will always lag behind prevailing conventions language models are the right way to handle conventions: 1 pick a trained language model based on preferred library/style 2 refine the model by training it on your own code 3 use refined model to suggest conventions in all code rule-based linters still useful as rerankers of suggestions 11 / 24

  12. Our Contributions two initial language models to learn and suggest space formatting in Coq files: baseline and advanced implementation of the language models in a toolchain based on Coq 8.10 and SerAPI 0.7.1 preliminary evaluation using a MathComp 1.9.0 based corpus machine readable representations as S-expressions via SerAPI 100k+ proof script lines, 63k+ lines of Gallina 2.2M+ Coq lexer tokens this is part of an umbrella project to suggest coding conventions for Coq using machine learning techniques https://github.com/EngineeringSoftware/roosterize 12 / 24

  13. Running Example From the RegLang Project Lemma mg_eq_proof L1 L2 (N1 : mgClassifier L1) : L1 =i L2 -> nerode L2 N1. Proof. move => H0 u v. split => [/nerodeP H1 w|H1]. - by rewrite -!H0. - apply/nerodeP => w. by rewrite !H0. Qed. 13 / 24

  14. Machine Learning Approach Task: predict spacing between tokens obtained from Coq’s lexer 1 obtain tokens and spacing via SerAPI’s sertok program 2 train model on spacing between tokens in lots of Coq code 3 use model to predict spacing between two given Coq tokens 14 / 24

  15. Feature Extraction Lemma mg_eq_proof L1 L2 (N1 : mgClassifier L1) : L1 =i L2 -> nerode L2 N1. (Sentence((IDENT Lemma)(IDENT mg_eq_proof)(IDENT L1)(IDENT L2) (KEYWORD"(")(IDENT N1)(KEYWORD :)(IDENT mgClassifier) (IDENT L1)(KEYWORD")")(KEYWORD :)(IDENT L1)(KEYWORD =i)(IDENT L2) (KEYWORD ->)(IDENT nerode)(IDENT L2)(IDENT N1)(KEYWORD .))) ( Content , Kind , # Newlines , # Spaces ) [( null , BOS , 0 , 0) , ( Lemma , IDENT , 2 , 0) , ( mg eq proof , IDENT , 0 , 1) , . . . ] 15 / 24

  16. Language Models: n-gram and Neural n-gram Model (Baseline) inserts spacing as special tokens before each token predicts next token after observing the n − 1 previous ones by statistical way (finding the most frequent token appearing after the n − 1 previous tokens in the training set) Neural Model (Advanced) embeds Coq tokens and spacing information into vectors predicts spacing using embedding vectors captures deeper formatting rules than statistical approach 16 / 24

  17. Corpus Based on MathComp 1.9.0 LOC Project SHA #Files #Lemmas #Toks Spec. Proof finmap 4 940 78,449 4,260 2,191 27642a8 fourcolor 0851d49 60 1,157 560,682 9,175 27,963 math-comp 89 8,802 1,076,096 38,243 46,470 748d716 odd-order ca602a4 34 367 519,855 11,882 24,243 Avg. N/A 46.75 2,816.50 558,770.50 15,890.00 25,216.75 Σ N/A 187 11,266 2,235,082 63,560 100,867 17 / 24

  18. Evaluation Setup 1 Randomly split corpus files into training, validation and testing sets which contain 80%, 10%, 10% of the files, respectively 2 Train model using training and validation sets 3 Apply model on testing set, and evaluate suggested spacing against existing spacing 18 / 24

  19. Results Model Top-1 Accuracy Top-3 Accuracy Neural 96.8% 99.7% n-gram 93.4% 98.9% Caveats: top-k accuracy assumes all errors are equally important but, subjective severity of spacing errors can differ greatly 19 / 24

Recommend

Learning to Format Coq Code Using Language Models Pengyu Nie 1 , Karl Palmskog 2 , Junyi Jessy Li

Learning to Format Coq Code Using Language Models Pengyu Nie 1 , Karl Palmskog 2 , Junyi Jessy Li

Learning to Format Coq Code Using Language Models Pengyu Nie 1 , Karl Palmskog 2 , Junyi Jessy Li 1 , and Milos Gligoric 1 The Coq Workshop 2020 1 The University of Texas at Austin 2 KTH Royal Institute of Technology Background: Coq is a

630 views • 25 slides
Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and

Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and

Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and Computation, U. of Amsterdam MSc Brain &amp; Cognitive Science, Artificial Intelligence, Logic MoL Guest Lecture Computational Models of Language

945 views • 58 slides
Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution ? What is language? never start with a dictionary definition!! the language of Google search query completion A language is a dialect with

1.13k views • 30 slides
Transfer learning with neural language models CS 685, Spring 2020 Advanced Natural Language

Transfer learning with neural language models CS 685, Spring 2020 Advanced Natural Language

Transfer learning with neural language models CS 685, Spring 2020 Advanced Natural Language Processing Mohit Iyyer College of Information and Computer Sciences University of Massachusetts Amherst many slides from Jacob Devlin &amp; Matt Peters

869 views • 49 slides
Character-level Language Models With Word-level Learning Arvid Frydenlund March 16, 2018

Character-level Language Models With Word-level Learning Arvid Frydenlund March 16, 2018

Character-level Language Models With Word-level Learning Arvid Frydenlund March 16, 2018 Character-level Language models Want language models with an open vocabulary Character-level models give this for free Treat the probability of

1.07k views • 5 slides
4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language models: log-linear language models [9, 4], which take a very di ff erent approach than the count-based n -grams described above. 11 4.1 Model

501 views • 8 slides
Language Modeling CS 6956: Deep Learning for NLP Overview What is a language model? How

Language Modeling CS 6956: Deep Learning for NLP Overview What is a language model? How

Language Modeling CS 6956: Deep Learning for NLP Overview What is a language model? How do we evaluate language models? Traditional language models Feedforward neural networks for language modeling Recurrent neural networks for

1.58k views • 58 slides
SQL-PL4OCL: an automatic code generator from OCL to SQL procedural language Marina Egea and

SQL-PL4OCL: an automatic code generator from OCL to SQL procedural language Marina Egea and

SQL-PL4OCL: an automatic code generator from OCL to SQL procedural language Marina Egea and Carolina Dania Sep. 19, 2017 MoDELS. Austin, Texas Outline Motivation Background Mapping OCL to SQL-PL How to map data models

402 views • 38 slides
Statistical Natural Language Processing Statistical models: learning, inference, estimation,

Statistical Natural Language Processing Statistical models: learning, inference, estimation,

Statistical Natural Language Processing Statistical models: learning, inference, estimation, prediction ar ltekin University of Tbingen Seminar fr Sprachwissenschaft Summer Semester 2017 Statistical models: learning, inference,

497 views • 25 slides
Code Games Or How I Learned to Stop Worrying and Love to Code Jacob Wilkins Topics Learning New

Code Games Or How I Learned to Stop Worrying and Love to Code Jacob Wilkins Topics Learning New

Code Games Or How I Learned to Stop Worrying and Love to Code Jacob Wilkins Topics Learning New Languages Esolangs Code Puzzles Code Golf Actual Computer Games Code Games | May 2017 | 2/19 New Languages Learning a new programming language

560 views • 19 slides
Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models answer the question: How likely is a string of English words good English? Help with reordering p lm (the house is small) &gt; p lm (small the is

1.06k views • 51 slides
Learning Discourse-level Diversity for Neural Dialog Models Using Conditional Variational

Learning Discourse-level Diversity for Neural Dialog Models Using Conditional Variational

Learning Discourse-level Diversity for Neural Dialog Models Using Conditional Variational Autoencoders Tiancheng Zhao, Ran Zhao and Maxine Eskenazi Language Technologies Institute Carnegie Mellon University Code&amp;Data:

585 views • 29 slides
Advanced Probabilistic Models for Generalized SSMs Speech and Language Nonlinear SSMs Bayesian

Advanced Probabilistic Models for Generalized SSMs Speech and Language Nonlinear SSMs Bayesian

Probabilistic Models State-Space Models Advanced Probabilistic Models for Generalized SSMs Speech and Language Nonlinear SSMs Bayesian Learning Mark Andrews Inferring Trajectories Gatsby Computational Neuroscience Unit Learning Example

424 views • 10 slides
Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models Language Models Language models are generative models of text s ~ P(x) The Malfoys! said Hermione. Harry was watching him. He looked like

1.18k views • 37 slides
Constrained Conditional Models Learning and Inference in Natural Language Understanding Dan Roth

Constrained Conditional Models Learning and Inference in Natural Language Understanding Dan Roth

Constrained Conditional Models Learning and Inference in Natural Language Understanding Dan Roth Department of Computer Science University of Illinois at Urbana-Champaign With thanks to: Collaborators: Ming-Wei Chang, Vasin Punyakanok, Lev

579 views • 53 slides
Beyond LSP Getting Your Language into Theia and VS Code Dr. Jan Khnlein @jankoehnlein The

Beyond LSP Getting Your Language into Theia and VS Code Dr. Jan Khnlein @jankoehnlein The

Beyond LSP Getting Your Language into Theia and VS Code Dr. Jan Khnlein @jankoehnlein The Goal Custom domain-specific language Rich text editor support With diagram support as VS Code Extension! 2 VS Code... Isn't this

625 views • 39 slides
(Even More) Language Modeling: Multi-Task Learning, and Building Blocks of Transformers CMSC

(Even More) Language Modeling: Multi-Task Learning, and Building Blocks of Transformers CMSC

(Even More) Language Modeling: Multi-Task Learning, and Building Blocks of Transformers CMSC 473/673 Frank Ferraro Outline Multi-Task Learning The Attention Mechanism Transformer Language Models as General Language Encoders Remember

1.33k views • 96 slides
CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan

CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan

CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan Klein, Michael Collins, Yejin Choi, Dan Jurafsky Overview The language modeling problem N-gram language models Evaluation: perplexity

530 views • 50 slides
Theories and Models of Language Change Conclusion Homeworks Session 7: Models III - Emergence of

Theories and Models of Language Change Conclusion Homeworks Session 7: Models III - Emergence of

Roland Mhlenbernd Introduction: The Evolut. Approach The Iterated Learning Model Theories and Models of Language Change Conclusion Homeworks Session 7: Models III - Emergence of Grammar Roland Mhlenbernd 2014/12/03 Review: Universal

631 views • 24 slides
MLA Format Research Papers The Modern Language Association (MLA) is an organization of students,

MLA Format Research Papers The Modern Language Association (MLA) is an organization of students,

MLA Format Research Papers The Modern Language Association (MLA) is an organization of students, writers, professors, and other teachers of English involved with the study and teaching of language and literature and writing. They discuss,

376 views • 9 slides
Code Shape More on Three-address Code Generation cs5363 1 Machine Code Translation A

Code Shape More on Three-address Code Generation cs5363 1 Machine Code Translation A

Code Shape More on Three-address Code Generation cs5363 1 Machine Code Translation A single language construct can have many implementations many-to-many mappings from high-level source language to low-level target machine language

560 views • 16 slides
3D models 3D models Output file format: PLY (Poligon File Format) with VCG extension library SW

3D models 3D models Output file format: PLY (Poligon File Format) with VCG extension library SW

Laser scanner technologies and 3D models integration: Laser scanner technologies and 3D models integration: analysis and methodological approach analysis and methodological approach Dante Abate, M.Cristina Baracca, Roberto Ciavarella, Graziano

140 views • 10 slides
Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for

Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for

Statistical language models Statistical language models in IR Discussion Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for IR Hinrich Sch utze Institute for Natural Language Processing,

1.69k views • 141 slides
Models of Language Evolution Session 10 : Iterated Learning and the Evolution of Compositionality

Models of Language Evolution Session 10 : Iterated Learning and the Evolution of Compositionality

Models of Language Evolution Session 10 : Iterated Learning and the Evolution of Compositionality &amp; Recursion Michael Franke Seminar f ur Sprachwissenschaft Eberhard Karls Universit at T ubingen Compositionality, Recursion &amp;

852 views • 30 slides

More recommend