Models for Sentence Compression A Comparison across Domains, Training Requirements and Evaluation Measures James Clarke and Mirella Lapata School of Informatics University of Edinburgh July 2006 ACL 2006 James Clarke and Mirella Lapata 1
Introduction What is Sentence Compression? Sentence Compression Can be viewed as producing a summary of a single sentence. James Clarke and Mirella Lapata 2
Introduction What is Sentence Compression? Sentence Compression Can be viewed as producing a summary of a single sentence. More formally A compressed sentence should: Use less words than the original sentence. Preserve the most important information. Remain grammatical. James Clarke and Mirella Lapata 3
Introduction Simplification Sentence compression can involve... word reordering word deletion word substitution word insertion James Clarke and Mirella Lapata 4
Introduction Simplification Sentence compression can involve... word reordering word deletion word substitution word insertion Ideally we want to exploit all of these operations but let’s start simple: Knight and Marcu (2002) Given an input sentence of words W = w 1 , w 2 , . . . , w n , a compression is formed by dropping any subset of these words. James Clarke and Mirella Lapata 5
Introduction Example Compression Original Prime Minister Tony Blair today insisted the case for holding terrorism suspects without trial was “absolutely compelling” as the government published new legislation allowing detention for 90 days without charge. James Clarke and Mirella Lapata 6
Introduction Example Compression Original Prime Minister Tony Blair today insisted the case for holding terrorism suspects without trial was “absolutely compelling” as the government published new legislation allowing detention for 90 days without charge. Compression Tony Blair insisted the case for holding terrorism suspects without trial was “compelling”. James Clarke and Mirella Lapata 7
Introduction Example Compression Original Prime Minister Tony Blair today insisted the case for holding terrorism suspects without trial was “absolutely compelling” as the government published new legislation allowing detention for 90 days without charge. Compression Tony Blair insisted the case for holding terrorism suspects without trial was “compelling”. James Clarke and Mirella Lapata 8
Introduction Outline Sentence Compression 1 Motivation Previous Work Our Work 2 How do humans compress sentences? Do existing methods port well across domains? What about automatic evaluation measures? Discussion 3 James Clarke and Mirella Lapata 9
Sentence Compression Motivation Outline Sentence Compression 1 Motivation Previous Work Our Work 2 How do humans compress sentences? Do existing methods port well across domains? What about automatic evaluation measures? Discussion 3 James Clarke and Mirella Lapata 10
Sentence Compression Motivation Applications Within summarisation: Current systems contain manually written rules for sentence compression. Other Applications include: Subtitle generation. Text compression for display on small screens. Audio scanning devices for the blind. James Clarke and Mirella Lapata 11
Sentence Compression Previous Work Outline Sentence Compression 1 Motivation Previous Work Our Work 2 How do humans compress sentences? Do existing methods port well across domains? What about automatic evaluation measures? Discussion 3 James Clarke and Mirella Lapata 12
Sentence Compression Previous Work Previous Work Methods ✟ ❍ ✟✟✟✟✟✟✟✟ ❍ ❍ ❍ ❍ ❍ ❍ ❍ ❍ Supervised Unsupervised ✟✟✟✟✟✟ ❍ ❍ ❍ Hori & Furui (2004) ❍ ❍ Charniak & Turner (2005) ❍ Generative Discriminative Knight & Marcu (2002) Knight & Marcu (2002) Turner & Charniak (2005) Riezler et al. (2003) Nguyen et al. (2004) McDonald (2006) James Clarke and Mirella Lapata 13
Sentence Compression Previous Work Data Requirements Parallel Corpora Most approaches rely on a parallel corpus. Automatically produced Ziff-Davis (Knight and Marcu, 2002). Domain: newspaper articles. There is no ‘natural’ resource of original-compressed sentences. James Clarke and Mirella Lapata 14
Sentence Compression Previous Work Data Requirements Parallel Corpora Most approaches rely on a parallel corpus. Automatically produced Ziff-Davis (Knight and Marcu, 2002). Domain: newspaper articles. There is no ‘natural’ resource of original-compressed sentences. Document . . . blah blah blah. The Abstract documentation is excellent – it Blah blah blah. The is clearly written with numerous documentation is excellent. drawings, cautions and tips, Blah blah blah . . . and includes an entire section on troubleshooting. Blah . . . James Clarke and Mirella Lapata 15
Sentence Compression Previous Work Data Requirements Parallel Corpora Most approaches rely on a parallel corpus. Automatically produced Ziff-Davis (Knight and Marcu, 2002). Domain: newspaper articles. There is no ‘natural’ resource of original-compressed sentences. Document . . . blah blah blah. The Abstract documentation is excellent – it Blah blah blah. The is clearly written with numerous documentation is excellent. drawings, cautions and tips, Blah blah blah . . . and includes an entire section on troubleshooting. Blah . . . James Clarke and Mirella Lapata 16
Sentence Compression Previous Work Evaluation Methodology Algorithms are evaluated on small sample (32 sentences). Humans are asked to assess grammaticality and information content. Typically four participants are used. Unlike machine translation, no established automatic measure. Comparisons across systems and system-configurations? James Clarke and Mirella Lapata 17
Our Work How do humans compress sentences? Outline Sentence Compression 1 Motivation Previous Work Our Work 2 How do humans compress sentences? Do existing methods port well across domains? What about automatic evaluation measures? Discussion 3 James Clarke and Mirella Lapata 18
Our Work How do humans compress sentences? Human-authored Compression Corpus Spoken Text Natural domain for compression applications. Speech is challenging (ungrammatical, incomplete). No naturally occurring compression corpora. James Clarke and Mirella Lapata 19
Our Work How do humans compress sentences? Human-authored Compression Corpus Spoken Text Natural domain for compression applications. Speech is challenging (ungrammatical, incomplete). No naturally occurring compression corpora. Methodology 50 Broadcast news documents. 3 annotators remove tokens from original transcript: preserve most important information in original sentence. preserve grammaticality of the compressed sentence. Could also leave a sentence uncompressed. James Clarke and Mirella Lapata 20
Our Work How do humans compress sentences? Example Human Compressions Original President Boris Yeltsin has won the most votes in Russia ’s hotly contested presidential election , one watched around the world . Compressions Boris Yeltsin has the most votes in Russia ’s presidential election . 1 Boris Yeltsin has won the most votes in Russia ’s presidential 2 election , watched around the world . Boris Yeltsin has won the most votes in Russia ’s presidential 3 election . James Clarke and Mirella Lapata 21
Our Work How do humans compress sentences? Analysis: Compression Rate A1 A2 A3 Av Ziff-Davis % compressed 88 79 87 84.4 97 CompRate 73.1 79.0 70.0 73.03 47 James Clarke and Mirella Lapata 22
Our Work How do humans compress sentences? Analysis: Compression Rate A1 A2 A3 Av Ziff-Davis % compressed 88 79 87 84.4 97 CompRate 73.1 79.0 70.0 73.03 47 Similar compression rates for annotators. James Clarke and Mirella Lapata 23
Our Work How do humans compress sentences? Analysis: Compression Rate A1 A2 A3 Av Ziff-Davis % compressed 88 79 87 84.4 97 CompRate 73.1 79.0 70.0 73.03 47 Similar compression rates for annotators. Ziff-Davis corpus is compressed much more aggressively. James Clarke and Mirella Lapata 24
Our Work How do humans compress sentences? Analysis: Compression Rate A1 A2 A3 Av Ziff-Davis % compressed 88 79 87 84.4 97 CompRate 73.1 79.0 70.0 73.03 47 Similar compression rates for annotators. Ziff-Davis corpus is compressed much more aggressively. Ziff-Davis corpus may not be comparable with human performance. James Clarke and Mirella Lapata 25
Our Work How do humans compress sentences? Analysis: Spans Distribution of lengths of words spans dropped Ann1 Ann2 0.45 Ann3 Ziff-Davis 0.375 Relative number of drops 0.3 0.225 0.15 0.075 0 1 2 3 4 5 6 7 8 9 10+ Length of word spans dropped James Clarke and Mirella Lapata 26
Our Work Do existing methods port well across domains? Outline Sentence Compression 1 Motivation Previous Work Our Work 2 How do humans compress sentences? Do existing methods port well across domains? What about automatic evaluation measures? Discussion 3 James Clarke and Mirella Lapata 27
Recommend
More recommend
