evalb improving cky parsing hw3
play

EVALB, Improving CKY Parsing, Hw3 Evaluating parsers Hw3 - PowerPoint PPT Presentation

Sep 21, 2023 •371 likes •1.26k views

EVALB, Improving CKY Parsing, Hw3 Scott Farrar CLMA, University of Washington far- rar@u.washington.edu EVALB, Improving CKY Parsing, Hw3 Evaluating parsers Hw3 Optimization: tips and tricks Scott Farrar 1. Size of the grammar CLMA,

  1. EVALB, Improving CKY Parsing, Hw3 Scott Farrar CLMA, University of Washington far- rar@u.washington.edu EVALB, Improving CKY Parsing, Hw3 Evaluating parsers Hw3 Optimization: tips and tricks Scott Farrar 1. Size of the grammar CLMA, University of Washington 2. Limit rules added to chart 3. Sentence length farrar@u.washington.edu January 28, 2010 1/42

  2. EVALB, Improving Today’s lecture CKY Parsing, Hw3 Scott Farrar CLMA, University of Washington far- rar@u.washington.edu Evaluating parsers Evaluating parsers 1 Hw3 Optimization: tips and tricks Hw3 2 1. Size of the grammar 2. Limit rules added to chart 3. Sentence length Optimization: tips and tricks 3 1. Size of the grammar 2. Limit rules added to chart 3. Sentence length 2/42

  3. EVALB, Improving Parsing: dev/train/test paradigm CKY Parsing, Hw3 Scott Farrar The Wall Street Journal (WSJ) section of the Penn CLMA, University of Washington far- Treebank (PTB), for all its faults, provides a very useful rar@u.washington.edu resource for comparing parser performance. Evaluating parsers Hw3 Optimization: tips and tricks 1. Size of the grammar 2. Limit rules added to chart 3. Sentence length 3/42

  4. EVALB, Improving Parsing: dev/train/test paradigm CKY Parsing, Hw3 Scott Farrar The Wall Street Journal (WSJ) section of the Penn CLMA, University of Washington far- Treebank (PTB), for all its faults, provides a very useful rar@u.washington.edu resource for comparing parser performance. Evaluating parsers Hw3 In building a probabilistic parser, there are four kinds of Optimization: tips and tricks resources that are commonly used esp. in the ACL related 1. Size of the grammar literature: 2. Limit rules added to chart 3. Sentence length 3/42

  5. EVALB, Improving Parsing: dev/train/test paradigm CKY Parsing, Hw3 Scott Farrar The Wall Street Journal (WSJ) section of the Penn CLMA, University of Washington far- Treebank (PTB), for all its faults, provides a very useful rar@u.washington.edu resource for comparing parser performance. Evaluating parsers Hw3 In building a probabilistic parser, there are four kinds of Optimization: tips and tricks resources that are commonly used esp. in the ACL related 1. Size of the grammar literature: 2. Limit rules added to chart 3. Sentence length 1 training data : large number of annotated sentences (sec. 2–21 of PTB has 39,830 sentences) 3/42

  6. EVALB, Improving Parsing: dev/train/test paradigm CKY Parsing, Hw3 Scott Farrar The Wall Street Journal (WSJ) section of the Penn CLMA, University of Washington far- Treebank (PTB), for all its faults, provides a very useful rar@u.washington.edu resource for comparing parser performance. Evaluating parsers Hw3 In building a probabilistic parser, there are four kinds of Optimization: tips and tricks resources that are commonly used esp. in the ACL related 1. Size of the grammar literature: 2. Limit rules added to chart 3. Sentence length 1 training data : large number of annotated sentences (sec. 2–21 of PTB has 39,830 sentences) 2 development data : small number of annotated sentences used to “tweak” parser (sec. 22, of PTB) 3/42

  7. EVALB, Improving Parsing: dev/train/test paradigm CKY Parsing, Hw3 Scott Farrar The Wall Street Journal (WSJ) section of the Penn CLMA, University of Washington far- Treebank (PTB), for all its faults, provides a very useful rar@u.washington.edu resource for comparing parser performance. Evaluating parsers Hw3 In building a probabilistic parser, there are four kinds of Optimization: tips and tricks resources that are commonly used esp. in the ACL related 1. Size of the grammar literature: 2. Limit rules added to chart 3. Sentence length 1 training data : large number of annotated sentences (sec. 2–21 of PTB has 39,830 sentences) 2 development data : small number of annotated sentences used to “tweak” parser (sec. 22, of PTB) 3 test data : small-medium number of un-annotated sentences used as input to parser (sec. 23 of PTB has 2416 sentences, ∼ 6% of training set) 3/42

  8. EVALB, Improving Parsing: dev/train/test paradigm CKY Parsing, Hw3 Scott Farrar The Wall Street Journal (WSJ) section of the Penn CLMA, University of Washington far- Treebank (PTB), for all its faults, provides a very useful rar@u.washington.edu resource for comparing parser performance. Evaluating parsers Hw3 In building a probabilistic parser, there are four kinds of Optimization: tips and tricks resources that are commonly used esp. in the ACL related 1. Size of the grammar literature: 2. Limit rules added to chart 3. Sentence length 1 training data : large number of annotated sentences (sec. 2–21 of PTB has 39,830 sentences) 2 development data : small number of annotated sentences used to “tweak” parser (sec. 22, of PTB) 3 test data : small-medium number of un-annotated sentences used as input to parser (sec. 23 of PTB has 2416 sentences, ∼ 6% of training set) 4 gold standard : annotated version of test data, with no errors (hidden till parser is developed) 3/42

  9. EVALB, Improving Recall our discussion first day of class CKY Parsing, Hw3 Scott Farrar CLMA, University of Washington far- rar@u.washington.edu Definition Evaluating parsers Hw3 objective criterion : that which a parser tries to maximize. Optimization: tips and tricks 1. Size of the grammar 2. Limit rules added to chart 3. Sentence length 4/42

  10. EVALB, Improving Recall our discussion first day of class CKY Parsing, Hw3 Scott Farrar CLMA, University of Washington far- rar@u.washington.edu Definition Evaluating parsers Hw3 objective criterion : that which a parser tries to maximize. Optimization: tips and tricks 1. Size of the Definition grammar 2. Limit rules added to chart tree accuracy : (harsh) exact match criterion; 1 for perfect 3. Sentence length match, otherwise 0. 4/42

  11. EVALB, Improving Recall our discussion first day of class CKY Parsing, Hw3 Scott Farrar CLMA, University of Washington far- rar@u.washington.edu Definition Evaluating parsers Hw3 objective criterion : that which a parser tries to maximize. Optimization: tips and tricks 1. Size of the Definition grammar 2. Limit rules added to chart tree accuracy : (harsh) exact match criterion; 1 for perfect 3. Sentence length match, otherwise 0. Non-exact matches can be very useful for some tasks: named entity extraction, information retrieval, document clustering 4/42

  12. EVALB, Improving PARSEVAL CKY Parsing, Hw3 Scott Farrar CLMA, University of Washington far- rar@u.washington.edu Definition Evaluating parsers PARSEVAL measures : standard metrics for evaluation Hw3 using the component pieces of a parse; a way to give partial Optimization: tips credit. and tricks 1. Size of the grammar 2. Limit rules added evalb is an implementation of the PARSEVAL measures to chart 3. Sentence length The evalb program uses several PARSEVAL measures: labeled precision (LP) labeled recall (LR) F-measure cross bracketing 5/42

  13. EVALB, Improving PARSEVAL: Labeled precision CKY Parsing, Hw3 Scott Farrar CLMA, University of Washington far- rar@u.washington.edu Definition Evaluating parsers Labeled Precision (LP) : the average of how many brackets Hw3 in the resulting parse tree match those in the gold standard Optimization: tips and tricks (same span). Focusing in on specific problems can increase 1. Size of the grammar precision. Broadening your methodology can decrease 2. Limit rules added to chart 3. Sentence length precision. Labeled precision includes the node label as well. LP = # of correct constituents in candidate parse of s # of total constituents in candidate parse of s 6/42

  14. EVALB, Improving PARSEVAL: Labeled recall CKY Parsing, Hw3 Scott Farrar CLMA, University of Washington far- rar@u.washington.edu Definition Evaluating parsers Labeled Recall (LR) : the average of how many brackets in Hw3 the gold standard are in the resulting parse. Did you get Optimization: tips and tricks them all? Coverage . Focusing in on specific problems can 1. Size of the grammar decrease recall, because other problems may get ignored. 2. Limit rules added to chart 3. Sentence length Labeled recall includes the node label as well. LR = # of correct constituents in candidate parse of s # of correct constituents in reference parse of s 7/42

  15. EVALB, Improving P, R errors CKY Parsing, Hw3 Scott Farrar CLMA, University Example of Washington far- rar@u.washington.edu PP attachment error Evaluating parsers Hw3 (S (NP (A a)) (VP(B b) (PP (C c))) ) gold (S (NP (A a)) (VP(B b) ) (PP (C c))) Optimization: tips and tricks 1. Size of the grammar 2. Limit rules added to chart 3. Sentence length 8/42

  16. EVALB, Improving P, R errors CKY Parsing, Hw3 Scott Farrar CLMA, University Example of Washington far- rar@u.washington.edu PP attachment error Evaluating parsers Hw3 (S (NP (A a)) (VP(B b) (PP (C c))) ) gold (S (NP (A a)) (VP(B b) ) (PP (C c))) Optimization: tips and tricks 1. Size of the Constituents in gold: S (0 , 3), NP (0 , 1), VP (1 , 3), PP (2 , 3) grammar 2. Limit rules added to chart 3. Sentence length 8/42

Recommend

Improving Domain Independent Question Parsing with Synthetic Treebanks COLING 2018: LAW-MWE-CxG

Improving Domain Independent Question Parsing with Synthetic Treebanks COLING 2018: LAW-MWE-CxG

Improving Domain Independent Question Parsing with Synthetic Treebanks COLING 2018: LAW-MWE-CxG Halim-Antoine Boukaram , Nizar Habash, Micheline Ziadee, and Majd Sakr American University of Science and Technology, Lebanon New York

294 views • 25 slides
CSC 4181 Compiler Construction Parsing 1 1 Outline Top-down v.s. Bottom-up Top-down parsing

CSC 4181 Compiler Construction Parsing 1 1 Outline Top-down v.s. Bottom-up Top-down parsing

CSC 4181 Compiler Construction Parsing 1 1 Outline Top-down v.s. Bottom-up Top-down parsing Bottom-up parsing Recursive-descent Shift-reduce parsers parsing LR(0) parsing LL(1) parsing LR(0) items LL(1) parsing

429 views • 29 slides
Introduction to Bottom-Up Parsing Shift-reduce parsing The LR parsing algorithm

Introduction to Bottom-Up Parsing Shift-reduce parsing The LR parsing algorithm

Outline Review LL parsing Introduction to Bottom-Up Parsing Shift-reduce parsing The LR parsing algorithm Constructing LR parsing tables 2 Top-Down Parsing: Review Top-Down Parsing: Review Top-down parsing expands a

470 views • 13 slides
Deep Neural Networks for Improving Computer-Aided Diagnosis, Segmentation and Text/Image Parsing in

Deep Neural Networks for Improving Computer-Aided Diagnosis, Segmentation and Text/Image Parsing in

Deep Neural Networks for Improving Computer-Aided Diagnosis, Segmentation and Text/Image Parsing in Radiology Le Le Lu Lu, Ph.D. .D. Joint work with Holge ger r R. Roth, h, Hoo Hoo-chan chang Shin, n, Ari i Seff, Xiaoso aosong g Wa

643 views • 60 slides
Top-Down Parsing Slides modified from Louden Book and Dr. Scherger Top Down Parsing A

Top-Down Parsing Slides modified from Louden Book and Dr. Scherger Top Down Parsing A

Compiler Design and Construction Top-Down Parsing Slides modified from Louden Book and Dr. Scherger Top Down Parsing A top-down parsing algorithm parses an input string of tokens by tracing out the steps in a leftmost derivation. Such an

1.12k views • 96 slides
Statistical Dependency Parsing in Korean: From Corpus Generation To Automatic Parsing Workshop on

Statistical Dependency Parsing in Korean: From Corpus Generation To Automatic Parsing Workshop on

Statistical Dependency Parsing in Korean: From Corpus Generation To Automatic Parsing Workshop on Statistical Parsing of Morphologically-Rich Languages 12th International Conference on Parsing Technologies Jinho D. Choi & Martha Palmer

199 views • 19 slides
Basic Parsing Algorithms Chart Parsing Seminar Recent Advances in Parsing Technology WS

Basic Parsing Algorithms Chart Parsing Seminar Recent Advances in Parsing Technology WS

Basic Parsing Algorithms Chart Parsing Seminar Recent Advances in Parsing Technology WS 2011/2012 Anna Schmidt Talk Outline Chart Parsing Basics Chart Parsing Algorithms Earley Algorithm CKY Algorithm Basics

1.12k views • 73 slides
Overview Introduction Lexicalized TAG, Advantages of parsing with LTAG Parsing LTAGs

Overview Introduction Lexicalized TAG, Advantages of parsing with LTAG Parsing LTAGs

Parsing with Lexicalized TAG (1) Extracting and comparing LTAG (2) Presentation by Philip John Gorinski Seminar Recent Advances in Parsing Technology Saarland University, Winter Term 2011/12 (1) Yves Shabes, Aravind K. Joshi, 1990 (2)

514 views • 36 slides
* 07/16/96 Plan for Today Shift-reduce parsing The problem with predictive top down parsing

* 07/16/96 Plan for Today Shift-reduce parsing The problem with predictive top down parsing

* 07/16/96 Plan for Today Shift-reduce parsing The problem with predictive top down parsing CS453 : Shift Reduce Parsing LR parsing: bottom up Unambiguous Grammars LR(0) and SLR Parse Tables Performs a right-most derivation in

316 views • 6 slides
1 Determinism and Parsing The parsing problem is, given a string w and a context-free grammar G ,

1 Determinism and Parsing The parsing problem is, given a string w and a context-free grammar G ,

1 Determinism and Parsing The parsing problem is, given a string w and a context-free grammar G , to decide if w L ( G ), and if so, to construct a derivation or a parse tree for w . Parsing is studied in courses in compilers. To be efficient

519 views • 12 slides
Bottom-up parsing LR parsing Construct parse tree for input from leaves up LR( k ) parsing

Bottom-up parsing LR parsing Construct parse tree for input from leaves up LR( k ) parsing

Bottom-up parsing LR parsing Construct parse tree for input from leaves up LR( k ) parsing reducing a string of tokens to single start symbol L eft-to-right scan of input, R ightmost derivation (inverse of deriving a string of tokens from

413 views • 5 slides
Bottom Up Parsing Also known as Shift-Reduce parsing More powerful than top down Dont

Bottom Up Parsing Also known as Shift-Reduce parsing More powerful than top down Dont

9/26/2012 Bottom Up Parsing Also known as Shift-Reduce parsing More powerful than top down Dont need left factored grammars Can handle left recursion Bottom Up Parsing Attempt to construct parse tree from an input string

654 views • 6 slides
Dependency Parsing & Feature-based Parsing Ling571 Deep Processing Techniques for NLP

Dependency Parsing & Feature-based Parsing Ling571 Deep Processing Techniques for NLP

Dependency Parsing & Feature-based Parsing Ling571 Deep Processing Techniques for NLP February 2, 2015 Roadmap Dependency parsing Graph-based dependency parsing Maximum spanning tree CLE Algorithm Learning

899 views • 87 slides
Statistical Parsing Parsing context-free languages ar ltekin University of Tbingen

Statistical Parsing Parsing context-free languages ar ltekin University of Tbingen

Statistical Parsing Parsing context-free languages ar ltekin University of Tbingen Seminar fr Sprachwissenschaft November 8, 2016 Recap Parsing basics CKY Earley Closing Ingredients of a (natural language) parser .

1.17k views • 51 slides
Generalised Parsing and Combinator Parsing A Happy Marriage? L. Thomas van Binsbergen

Generalised Parsing and Combinator Parsing A Happy Marriage? L. Thomas van Binsbergen

Conventional Parser Combinators Aspects of Generalised Parsing Combinators for Generalised Parsing Generalised Parsing and Combinator Parsing A Happy Marriage? L. Thomas van Binsbergen ltvanbinsbergen@acm.org Royal Holloway, University of

452 views • 30 slides
Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs

Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs

Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs Semantic Parsing Dependency parsing models the syntactic structure between words in a sentence. Dependency Parsing vs Semantic Parsing

637 views • 31 slides
Statistical Parsing Dependency parsing ar ltekin University of Tbingen Seminar fr

Statistical Parsing Dependency parsing ar ltekin University of Tbingen Seminar fr

Statistical Parsing Dependency parsing ar ltekin University of Tbingen Seminar fr Sprachwissenschaft November 2016 Recap/background Dependency grammar Dependency parsing Evaluation Summary Ingredients of a parser

1.44k views • 89 slides
Plan for Today Predictive parsing as a specific subclass of recursive descent parsing

Plan for Today Predictive parsing as a specific subclass of recursive descent parsing

Plan for Today Predictive parsing as a specific subclass of recursive descent parsing complexity comparisons with general parsing might have to left-factor necessary to remove left-recursion (MiniSVG modification as example) Working

346 views • 5 slides
From Dependency Parsing to Imitation Learning CMSC 723 / LING 723 / INST 725 Marine Carpuat Fig

From Dependency Parsing to Imitation Learning CMSC 723 / LING 723 / INST 725 Marine Carpuat Fig

From Dependency Parsing to Imitation Learning CMSC 723 / LING 723 / INST 725 Marine Carpuat Fig credits: Joakim Nivre, Yoav Goldberg, Hal Daume III T odays topics: Addressing compounding error Improving on gold parse oracle

687 views • 22 slides
Lectures 7 and 8. Parsing (syntax analysis) Wei Le 2015.9 Bottom Up Parsing Recognize many

Lectures 7 and 8. Parsing (syntax analysis) Wei Le 2015.9 Bottom Up Parsing Recognize many

Lectures 7 and 8. Parsing (syntax analysis) Wei Le 2015.9 Bottom Up Parsing Recognize many programming languages, used in practice More general, more powerful parser: less restrictions on grammars, e.g., can handle left-recursive grammar

680 views • 64 slides
LL1 Parsing a b $ S 1 A 4 B 3 2 a b b b 1. S --> a B

LL1 Parsing a b $ S 1 A 4 B 3 2 a b b b 1. S --> a B

LL1 Parsing a b $ S 1 A 4 B 3 2 a b b b 1. S --> a B Input 2. B --> 3. B --> A S 4. A --> b B Parse Stack Compare Input and Stacktop 1 LL1 Parsing a b $ S 1 A

694 views • 17 slides
Programming Languages: Parsing Onur Tolga S ehito glu Computer Engineering,METU 27 May

Programming Languages: Parsing Onur Tolga S ehito glu Computer Engineering,METU 27 May

Programming Languages:Parsing Programming Languages: Parsing Onur Tolga S ehito glu Computer Engineering,METU 27 May 2009 Programming Languages:Parsing Outline 1 Parsing Top-down Parsing Recursive Descent Parser LL Parsers

215 views • 8 slides
Models of Human Parsing Experimental Data 2 Informatics 2A: Lecture 22 Eye-tracking Reading

Models of Human Parsing Experimental Data 2 Informatics 2A: Lecture 22 Eye-tracking Reading

Human Parsing Human Parsing Experimental Data Experimental Data Bottom-Up Parser Bottom-Up Parser Left Corner Parser Left Corner Parser Human Parsing 1 Cognitive Constraints Garden Paths Dimensions of Parsing Models of Human Parsing

537 views • 9 slides
Top-Down Parsing Top-Down Parsing #1 Extra Credit Question Given this grammar G: E

Top-Down Parsing Top-Down Parsing #1 Extra Credit Question Given this grammar G: E

Top-Down Parsing Top-Down Parsing #1 Extra Credit Question Given this grammar G: E E + T E T T T * int T int T ( E ) Is the string int * (int + int) in L(G)? Give a derivation or prove that it

1.08k views • 61 slides

More recommend