learning to generate concept to text generation using
play

Learning to generate: Concept-to-text generation using machine - PowerPoint PPT Presentation

Feb 21, 2023 •382 likes •1.83k views

Learning to generate: Concept-to-text generation using machine learning Ioannis Konstas Institute for Language, Cognition and Computation University of Edinburgh Aberdeen, NLG Summer School 21 July 2015 Konstas (ILCC) Concept-to-Text

  1. Problem Formulation Key Idea Key Idea Temperature Cloud Sky Cover Cloudy, with a low around 10. South wind between 15 and 30 Time Min Mean Max Time Percent (%) mph. 06:00-21:00 9 15 21 06:00-09:00 25-50 09:00-12:00 50-75 Partly cloudy, with a low Wind Speed Wind Direction around 9. Time Min Mean Max Time Mode Breezy, with a south wind be- 06:00-21:00 S 06:00-21:00 15 20 30 tween 15 and 30 mph. Konstas (ILCC) Concept-to-Text Generation 21 July 2015 8 / 56

  2. Problem Formulation Key Idea Key Idea Temperature Cloud Sky Cover Cloudy, with a low around 10. South wind between 15 and 30 Time Min Mean Max Time Percent (%) mph. 06:00-21:00 9 15 21 06:00-09:00 25-50 09:00-12:00 50-75 Partly cloudy, with a low Wind Speed Wind Direction around 9. Time Min Mean Max Time Mode Breezy, with a south wind be- 06:00-21:00 S 06:00-21:00 15 20 30 tween 15 and 30 mph. Konstas (ILCC) Concept-to-Text Generation 21 July 2015 8 / 56

  3. Problem Formulation Key Idea Traditional NLG Pipeline Communicative Goal Input Data Content Planning Sentence Planning Surface Realisation Text Konstas (ILCC) Concept-to-Text Generation 21 July 2015 9 / 56

  4. Problem Formulation Key Idea Traditional NLG Pipeline Communicative Goal Input Data Content Planning Liang et al. (2009) Content Selection Document Planning Sentence Planning Surface Realisation Text Konstas (ILCC) Concept-to-Text Generation 21 July 2015 9 / 56

  5. Learning Alignments Liang et al. 2009 Liang et al., ACL 2009 Learning Semantic Correspondences with Less Supervision Konstas (ILCC) Concept-to-Text Generation 21 July 2015 10 / 56

  6. Learning Alignments Liang et al. 2009 Alignment Task Wind Chill Temperature Time Min Mean Max Time Min Mean Max 06-21 0 0 0 06-21 52 61 70 Wind Speed Wind Direction Time Mode Time Min Mean Max 06-21 S 06-21 11 22 29 Gust Precipitation Potential Time Min Mean Max Time Min Mean Max Showers and thunderstorms. 06-21 0 20 39 06-21 26 81 100 High near 70. Cloudy, Sky Cover Rain Chance with a south wind around 20mph, Time Mode Time Percent (%) 06-21 Def 06-21 75-100 with gusts as high as 40 mph. 06-09 Lkly 06-09 75-100 06-13 Def 06-13 50-75 Chance of precipitation is 100%. 09-21 Def 09-21 75-100 13-21 Def 13-21 75-100 Snow Chance Thunder Chance Freezing Rain Chance Sleet Chance Time Mode Time Mode Time Mode Time Mode 06-21 – 06-21 Def 06-21 – 06-21 – 06-09 – 06-09 Lkly 06-09 – 06-09 – 06-13 – 06-13 Chc 06-13 – 06-13 – 09-21 – 09-21 Def 09-21 – 09-21 – 13-21 – 13-21 Def 13-21 – 13-21 – Konstas (ILCC) Concept-to-Text Generation 21 July 2015 11 / 56

  7. Learning Alignments Liang et al. 2009 Generative Story Record choice: choose a sequence of records r = � � 1 r 1 , . . . , r | r | | r | 1 � p ( r | d ) = p ( r i . t | r i − 1 . t ) | s ( r i . t ) | i p ( r , f , c , w | d ) = p ( r | d ) p ( f | r ) p ( c , w | r , f , d ) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 12 / 56

  8. Learning Alignments Liang et al. 2009 Generative Story Record choice: choose a sequence of records r = � � 1 r 1 , . . . , r | r | | r | 1 � p ( r | d ) = p ( r i . t | r i − 1 . t ) | s ( r i . t ) | i Field choice: for each chosen record r i , select a sequence of fields 2 f i = � � f i 1 , . . . , f i | f i | | r i . f | � p ( f | r i . t ) = p ( r i . f k | r i . f k − 1 ) k p ( r , f , c , w | d ) = p ( r | d ) p ( f | r ) p ( c , w | r , f , d ) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 12 / 56

  9. Learning Alignments Liang et al. 2009 Generative Story Record choice: choose a sequence of records r = � � 1 r 1 , . . . , r | r | | r | 1 � p ( r | d ) = p ( r i . t | r i − 1 . t ) | s ( r i . t ) | i Field choice: for each chosen record r i , select a sequence of fields 2 f i = � � f i 1 , . . . , f i | f i | | r i . f | � p ( f | r i . t ) = p ( r i . f k | r i . f k − 1 ) k Word choice: for each chosen field f ik , choose a number c ik > 0 uniformly, and 3 generate a sequence of c ik words. | w | � p ( w | r i , r i . f k , r i . f k . t , c ik ) = p ( w j | r i . t , r i . f k . v ) j p ( r , f , c , w | d ) = p ( r | d ) p ( f | r ) p ( c , w | r , f , d ) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 12 / 56

  10. Learning Alignments Liang et al. 2009 Hierarchical Semi-Markov Model (HSMM) d r | r | . . . . . . r 1 r i . . . . . . r | r | . f | f | . . . r i . f | f | r 1 . f 1 r i . f 1 . . . . . . . . . . . . w 1 w N w w w w w w EM Training: dynamic program similar to the inside-outside algorithm Konstas (ILCC) Concept-to-Text Generation 21 July 2015 13 / 56

  11. Learning Alignments Liang et al. 2009 Aligned Output temperature 1 Records: k skyCover 1 Fields: max= 70 percent= 75-100 N Text: High near 70 . Cloudy , Records: k windDir 1 k windSpeed 1 Fields: N mode= S N N mean= 20 Text: with ag southg windg aroundg 20 mph . Konstas (ILCC) Concept-to-Text Generation 21 July 2015 14 / 56

  12. Pipeline Approaches Outline Outline Problem Formulation Learning Alignments Pipeline Approach Joint Approaches Konstas (ILCC) Concept-to-Text Generation 21 July 2015 15 / 56

  13. Pipeline Approaches History-based Generation Traditional NLG Pipeline Communicative Goal Input Data Content Planning Kim and Mooney (2010) Content Selection Angeli et al. (2010) Document Planning Sentence Planning Surface Realisation Text Konstas (ILCC) Concept-to-Text Generation 21 July 2015 16 / 56

  14. Pipeline Approaches History-based Generation Angeli et al., EMNLP 2010 A Simple Domain-Independent Probabilistic Approach to Generation Konstas (ILCC) Concept-to-Text Generation 21 July 2015 17 / 56

  15. Pipeline Approaches History-based Generation Generative Story for i = 1, 2, . . . : 1 choose a record r i ∈ d Konstas (ILCC) Concept-to-Text Generation 21 July 2015 18 / 56

  16. Pipeline Approaches History-based Generation Generative Story for i = 1, 2, . . . : 1 choose a record r i ∈ d 2 if r i = stop : return Konstas (ILCC) Concept-to-Text Generation 21 July 2015 18 / 56

  17. Pipeline Approaches History-based Generation Generative Story for i = 1, 2, . . . : 1 choose a record r i ∈ d 2 if r i = stop : return 3 choose a field f j ∈ r i . t . f Konstas (ILCC) Concept-to-Text Generation 21 July 2015 18 / 56

  18. Pipeline Approaches History-based Generation Generative Story for i = 1, 2, . . . : 1 choose a record r i ∈ d 2 if r i = stop : return 3 choose a field f j ∈ r i . t . f 4 choose a template T k ∈ r i . t . f j . T Konstas (ILCC) Concept-to-Text Generation 21 July 2015 18 / 56

  19. Pipeline Approaches History-based Generation Generative Story for i = 1, 2, . . . : 1 choose a record r i ∈ d 2 if r i = stop : return 3 choose a field f j ∈ r i . t . f 4 choose a template T k ∈ r i . t . f j . T Konstas (ILCC) Concept-to-Text Generation 21 July 2015 18 / 56

  20. Pipeline Approaches History-based Generation Generative Story for i = 1, 2, . . . : 1 choose a record r i ∈ d 2 if r i = stop : return 3 choose a field f j ∈ r i . t . f 4 choose a template T k ∈ r i . t . f j . T Each decision is governed by a set of feature templates Konstas (ILCC) Concept-to-Text Generation 21 July 2015 18 / 56

  21. Pipeline Approaches History-based Generation Feature Templates Record R1 list of k = 1 , 2 record types r 2 . t =temp ∧ ( r 1 . t , r 0 . t )=(skyCover, start ) R2 set of prev record types r 2 . t =temp ∧ { r 1 . t }={skyCover} R3 record type already gen r 2 . t =temp ∧ r j . t � = temp, ∀ j < 2 R4 field values r 2 . t =temp ∧ r 2 .v[min]=10, r 2 .v[max]=20 � stop | degrees . � R5 stop under LM r 3 .t= stop × p LM Konstas (ILCC) Concept-to-Text Generation 21 July 2015 19 / 56

  22. Pipeline Approaches History-based Generation Feature Templates Record R1 list of k = 1 , 2 record types r 2 . t =temp ∧ ( r 1 . t , r 0 . t )=(skyCover, start ) R2 set of prev record types r 2 . t =temp ∧ { r 1 . t }={skyCover} R3 record type already gen r 2 . t =temp ∧ r j . t � = temp, ∀ j < 2 R4 field values r 2 . t =temp ∧ r 2 .v[min]=10, r 2 .v[max]=20 � stop | degrees . � R5 stop under LM r 3 .t= stop × p LM Field F1 field set f 2 = {time, min, mean, max} F2 field values f 2 = {min, max} ∧ f 2 .v[min]=10, . . . Konstas (ILCC) Concept-to-Text Generation 21 July 2015 19 / 56

  23. Pipeline Approaches History-based Generation Feature Templates Record R1 list of k = 1 , 2 record types r 2 . t =temp ∧ ( r 1 . t , r 0 . t )=(skyCover, start ) R2 set of prev record types r 2 . t =temp ∧ { r 1 . t }={skyCover} R3 record type already gen r 2 . t =temp ∧ r j . t � = temp, ∀ j < 2 R4 field values r 2 . t =temp ∧ r 2 .v[min]=10, r 2 .v[max]=20 � stop | degrees . � R5 stop under LM r 3 .t= stop × p LM Field F1 field set f 2 = {time, min, mean, max} F2 field values f 2 = {min, max} ∧ f 2 .v[min]=10, . . . Template W1 base/coarse B(T 2 ) = � with a low around [min] � C(T 2 ) = � with a [time] around [min] � W2 field values W3 1 st word of T under LM p LM ( with | cloudy , ) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 19 / 56

  24. Pipeline Approaches History-based Generation Feature Templates Record R1 list of k = 1 , 2 record types r 2 . t =temp ∧ ( r 1 . t , r 0 . t )=(skyCover, start ) R2 set of prev record types r 2 . t =temp ∧ { r 1 . t }={skyCover} R3 record type already gen r 2 . t =temp ∧ r j . t � = temp, ∀ j < 2 R4 field values r 2 . t =temp ∧ r 2 .v[min]=10, r 2 .v[max]=20 � stop | degrees . � R5 stop under LM r 3 .t= stop × p LM Field F1 field set f 2 = {time, min, mean, max} F2 field values f 2 = {min, max} ∧ f 2 .v[min]=10, . . . Template W1 base/coarse B(T 2 ) = � with a low around [min] � C(T 2 ) = � with a [time] around [min] � W2 field values W3 1 st word of T under LM p LM ( with | cloudy , ) | c | � � c j | c < j ; θ � p ( c | d ; θ ) = p j = 1 Konstas (ILCC) Concept-to-Text Generation 21 July 2015 19 / 56

  25. Pipeline Approaches History-based Generation Feature Templates Record R1 list of k = 1 , 2 record types r 2 . t =temp ∧ ( r 1 . t , r 0 . t )=(skyCover, start ) R2 set of prev record types r 2 . t =temp ∧ { r 1 . t }={skyCover} R3 record type already gen r 2 . t =temp ∧ r j . t � = temp, ∀ j < 2 R4 field values r 2 . t =temp ∧ r 2 .v[min]=10, r 2 .v[max]=20 � stop | degrees . � R5 stop under LM r 3 .t= stop × p LM Field F1 field set f 2 = {time, min, mean, max} F2 field values f 2 = {min, max} ∧ f 2 .v[min]=10, . . . Template W1 base/coarse B(T 2 ) = � with a low around [min] � C(T 2 ) = � with a [time] around [min] � W2 field values W3 1 st word of T under LM p LM ( with | cloudy , ) | c | � � c j | c < j ; θ � p ( c | d ; θ ) = p j = 1 L-BFGS learning: Use Liang et al. (2009) alignments to compute features Konstas (ILCC) Concept-to-Text Generation 21 July 2015 19 / 56

  26. Pipeline Approaches History-based Generation Decoding � c j | c < j ; θ � c j = arg max ˆ p c j Greedy search: choose the best decision ˆ c j until the stop record is drawn Konstas (ILCC) Concept-to-Text Generation 21 July 2015 20 / 56

  27. Pipeline Approaches History-based Generation Decoding � c j | c < j ; θ � c j = arg max ˆ p c j Greedy search: choose the best decision ˆ c j until the stop record is drawn Alternatively, sample from the distribution p � c j | c < j ; θ � ; Konstas (ILCC) Concept-to-Text Generation 21 July 2015 20 / 56

  28. Pipeline Approaches History-based Generation Decoding � c j | c < j ; θ � c j = arg max ˆ p c j Greedy search: choose the best decision ˆ c j until the stop record is drawn Alternatively, sample from the distribution p � c j | c < j ; θ � ; � c j | d ; θ � Viterbi search over arg max c j p Konstas (ILCC) Concept-to-Text Generation 21 July 2015 20 / 56

  29. Pipeline Approaches History-based Generation Conclusions Generation recast into a generative story Ensemble of local decisions Discriminatively trained end-to-end generation system Konstas (ILCC) Concept-to-Text Generation 21 July 2015 21 / 56

  30. Pipeline Approaches History-based Generation Conclusions Generation recast into a generative story Ensemble of local decisions Discriminatively trained end-to-end generation system How about we model generation jointly and learn without supervision? Konstas (ILCC) Concept-to-Text Generation 21 July 2015 21 / 56

  31. Pipeline Approaches Outline Outline Problem Formulation Learning Alignments Pipeline Approach Joint Approaches Konstas (ILCC) Concept-to-Text Generation 21 July 2015 22 / 56

  32. Joint Approaches Grammar-based Generation Traditional NLG Pipeline Communicative Goal Input Data Content Planning Kim and Mooney (2010) Content Selection Angeli et al. (2010) Document Planning Konstas and Lapata (2012a, Sentence Planning 2012b, 2013b) Surface Realisation Text Konstas (ILCC) Concept-to-Text Generation 21 July 2015 23 / 56

  33. Joint Approaches Grammar-based Generation Konstas and Lapata, NAACL 2012 Unsupervised Concept-to-text Generation with Hypergraphs Konstas and Lapata, JAIR 2013 A Global Model for Concept-to-Text Generation Konstas (ILCC) Concept-to-Text Generation 21 July 2015 24 / 56

  34. Joint Approaches Grammar-based Generation Grammar Konstas (ILCC) Concept-to-Text Generation 21 July 2015 25 / 56

  35. Joint Approaches Grammar-based Generation Grammar 1 S → R ( start ) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 25 / 56

  36. Joint Approaches Grammar-based Generation Grammar 1 S → R ( start ) 2 R ( r i . t ) → FS ( r j , start ) R ( r j . t ) | FS ( r j , start ) R ( skyCover 1 . t ) → FS ( temperature 1 , start ) R ( temperature 1 . t ) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 25 / 56

  37. Joint Approaches Grammar-based Generation Grammar Rain Chance Thunder Chance Temperature Sky Cover Wind Direction Wind Speed Gust Precipitation Potential Time Mode Time Mode Time Mode Time Min Mean Max Time Min Mean Max Time Percent (%) Time Min Mean Max Time Min Mean Max 06-21 Def 06-21 Def 06-21 S 06-21 0 20 39 06-21 52 61 70 06-21 75-100 06-21 11 22 29 06-21 26 81 100 06-09 Lkly 06-09 Lkly 06-09 75-100 06-13 Def 06-13 Chc 06-13 50-75 09-21 Def 09-21 Def 09-21 75-100 13-21 Def 13-21 Def 13-21 75-100 1 S → R ( start ) 2 R ( r i . t ) → FS ( r j , start ) R ( r j . t ) | FS ( r j , start ) R ( skyCover 1 . t ) → FS ( temperature 1 , start ) R ( temperature 1 . t ) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 25 / 56

  38. Joint Approaches Grammar-based Generation Grammar Rain Chance Thunder Chance Temperature Sky Cover Wind Direction Wind Speed Gust Precipitation Potential Time Mode Time Mode Time Mode Time Min Mean Max Time Min Mean Max Time Percent (%) Time Min Mean Max Time Min Mean Max 06-21 Def 06-21 Def 06-21 S 06-21 0 20 39 06-21 52 61 70 06-21 75-100 06-21 11 22 29 06-21 26 81 100 06-09 Lkly 06-09 Lkly 06-09 75-100 06-13 Def 06-13 Chc 06-13 50-75 09-21 Def 09-21 Def 09-21 75-100 13-21 Def 13-21 Def 13-21 75-100 1 S → R ( start ) 2 R ( r i . t ) → FS ( r j , start ) R ( r j . t ) | FS ( r j , start ) 3 FS ( r , r . f i ) → F ( r , r . f j ) FS ( r , r . f j ) | F ( r , r . f j ) FS ( wSpeed 1 , min ) → F ( wSpeed 1 , max ) FS ( wSpeed 1 , max ) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 25 / 56

  39. Joint Approaches Grammar-based Generation Grammar Rain Chance Thunder Chance Temperature Sky Cover Wind Direction Wind Speed Gust Precipitation Potential Time Mode Time Mode Time Mode Time Min Mean Max Time Min Mean Max Time Percent (%) Time Min Mean Max Time Min Mean Max 06-21 Def 06-21 Def 06-21 S 06-21 0 20 39 06-21 52 61 70 06-21 75-100 06-21 11 22 29 06-21 26 81 100 06-09 Lkly 06-09 Lkly 06-09 75-100 06-13 Def 06-13 Chc 06-13 50-75 09-21 Def 09-21 Def 09-21 75-100 13-21 Def 13-21 Def 13-21 75-100 1 S → R ( start ) 2 R ( r i . t ) → FS ( r j , start ) R ( r j . t ) | FS ( r j , start ) 3 FS ( r , r . f i ) → F ( r , r . f j ) FS ( r , r . f j ) | F ( r , r . f j ) 4 F ( r , r . f ) → W ( r , r . f ) F ( r , r . f ) | W ( r , r . f ) F ( gust 1 , min ) → W ( gust 1 , mean ) F ( gust 1 , mean ) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 25 / 56

  40. Joint Approaches Grammar-based Generation Grammar Rain Chance Thunder Chance Temperature Sky Cover Wind Direction Wind Speed Gust Precipitation Potential Time Mode Time Mode Time Min Mean Max Time Percent (%) Time Mode Time Min Mean Max Time Min Mean Max Time Min Mean Max 06-21 Def 06-21 Def 06-21 52 61 70 06-21 75-100 06-21 S 06-21 11 22 29 06-21 0 20 39 06-21 26 81 100 06-09 Lkly 06-09 Lkly 06-09 75-100 06-13 Def 06-13 Chc 06-13 50-75 09-21 Def 09-21 Def 09-21 75-100 13-21 Def 13-21 Def 13-21 75-100 1 S → R ( start ) 2 R ( r i . t ) → FS ( r j , start ) R ( r j . t ) | FS ( r j , start ) 3 FS ( r , r . f i ) → F ( r , r . f j ) FS ( r , r . f j ) | F ( r , r . f j ) 4 F ( r , r . f ) → W ( r , r . f ) F ( r , r . f ) | W ( r , r . f ) 5 W ( r , r . f ) → α | g( f . v ) W ( skyCover 1 , %) → cloudy [ % . v = ‘75-100’ ] Konstas (ILCC) Concept-to-Text Generation 21 July 2015 25 / 56

  41. Joint Approaches Grammar-based Generation Grammar 1 S → R ( start ) 2 R ( r i . t ) → FS ( r j , start ) R ( r j . t ) | FS ( r j , start ) 3 FS ( r , r . f i ) → F ( r , r . f j ) FS ( r , r . f j ) | F ( r , r . f j ) 4 F ( r , r . f ) → W ( r , r . f ) F ( r , r . f ) | W ( r , r . f ) 5 W ( r , r . f ) → α | g( f . v ) EM Training: dynamic program similar to the inside-outside algorithm Konstas (ILCC) Concept-to-Text Generation 21 July 2015 25 / 56

  42. Joint Approaches Grammar-based Generation Decoding � � g = f ˆ arg max g , h p ( g ) · p ( g , h | d ) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 26 / 56

  43. Joint Approaches Grammar-based Generation Decoding � � g = f ˆ arg max g , h p ( g ) · p ( g , h | d ) Bottom-up Viterbi search Keep k-best derivations at each node, cube pruning (Chiang, 2007) p ( g ) rescores derivations by linearly interpolating: n-gram language model dependency model (DMV; Klein and Manning, 2004) Implement using hypergraphs (Klein and Manning, 2001) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 26 / 56

  44. Joint Approaches Grammar-based Generation Decoding Leaf nodes ǫ emit a k-best list of words W 0 , 1 (skyCover 1 . t ,%) ǫ  mostly ; RB  cloudy ; JJ   sunny ; JJ · · · Konstas (ILCC) Concept-to-Text Generation 21 July 2015 27 / 56

  45. Joint Approaches Grammar-based Generation Decoding  mostly cloudy ⋆ the morning ; JJ  11 am ; JJ mostly cloudy ⋆ after FS 0 , 5 (skyCover 1 . t ,start)   mostly cloudy ⋆ then becoming ; JJ · · ·  mostly cloudy ; RB  F 0 , 2 (skyCover 1 . t ,%) W 4 , 5 (skyCover 1 . t ,time) mostly clouds ; NNS  , ; JJ  cloudy  morning ; NN  · · · 11 am ; NN   after ; PREP W 0 , 1 (skyCover 1 . t ,%) W 1 , 2 (skyCover 1 . t ,%) · · ·  mostly ; RB   mostly ; RB  cloudy ; JJ cloudy ; JJ     sunny ; JJ sunny ; JJ · · · · · · Konstas (ILCC) Concept-to-Text Generation 21 July 2015 28 / 56

  46. Joint Approaches Grammar-based Generation Decoding  mostly cloudy ⋆ the morning ; JJ  11 am ; JJ mostly cloudy ⋆ after FS 0 , 5 (skyCover 1 . t ,start)   mostly cloudy ⋆ then becoming ; JJ · · ·  mostly cloudy ; RB  F 0 , 2 (skyCover 1 . t ,%) W 4 , 5 (skyCover 1 . t ,time) mostly clouds ; NNS  , ; JJ  cloudy  morning ; NN  · · · 11 am ; NN   after ; PREP W 0 , 1 (skyCover 1 . t ,%) W 1 , 2 (skyCover 1 . t ,%) · · ·  mostly ; RB   mostly ; RB  cloudy ; JJ cloudy ; JJ     sunny ; JJ sunny ; JJ · · · · · · Konstas (ILCC) Concept-to-Text Generation 21 July 2015 28 / 56

  47. Joint Approaches Grammar-based Generation Decoding  mostly cloudy ⋆ the morning ; JJ  11 am ; JJ mostly cloudy ⋆ after FS 0 , 5 (skyCover 1 . t ,start)   mostly cloudy ⋆ then becoming ; JJ · · ·  mostly cloudy ; RB  F 0 , 2 (skyCover 1 . t ,%) W 4 , 5 (skyCover 1 . t ,time) mostly clouds ; NNS  , ; JJ  cloudy  morning ; NN  · · · 11 am ; NN   after ; PREP W 0 , 1 (skyCover 1 . t ,%) W 1 , 2 (skyCover 1 . t ,%) · · ·  mostly ; RB   mostly ; RB  cloudy ; JJ cloudy ; JJ     sunny ; JJ sunny ; JJ · · · · · · Konstas (ILCC) Concept-to-Text Generation 21 July 2015 28 / 56

  48. Joint Approaches Results Experimental Setup Data RoboCup : simulated sportscasting [214 words] (Chen and Mooney, 2008) WeatherGov : weather reports [4 sents, 345 words] (Liang et al., 2009) Atis : flight booking [1 sent, 927 words] (Zettlemoyer and Collins, 2007) WinHelp : troubleshooting guides [4.3 sents, 629 words] (Branavan et al., 2009) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 29 / 56

  49. Joint Approaches Results Experimental Setup Data RoboCup : simulated sportscasting [214 words] (Chen and Mooney, 2008) WeatherGov : weather reports [4 sents, 345 words] (Liang et al., 2009) Atis : flight booking [1 sent, 927 words] (Zettlemoyer and Collins, 2007) WinHelp : troubleshooting guides [4.3 sents, 629 words] (Branavan et al., 2009) Evaluation Automatic evaluation: BLEU-4 Human evaluation: Fluency, Semantic Correctness Konstas (ILCC) Concept-to-Text Generation 21 July 2015 29 / 56

  50. Joint Approaches Results Experimental Setup Data RoboCup : simulated sportscasting [214 words] (Chen and Mooney, 2008) WeatherGov : weather reports [4 sents, 345 words] (Liang et al., 2009) Atis : flight booking [1 sent, 927 words] (Zettlemoyer and Collins, 2007) WinHelp : troubleshooting guides [4.3 sents, 629 words] (Branavan et al., 2009) Evaluation Automatic evaluation: BLEU-4 Human evaluation: Fluency, Semantic Correctness System Comparison 1 − best, k -Best-lm , k -Best-lm-dmv Angeli et al. (2010) Konstas (ILCC) Concept-to-Text Generation 21 July 2015 29 / 56

  51. Joint Approaches Results Results: Automatic Evaluation RoboCup WeatherGov 40 40 38 . 4 34 . 18 33 . 7 30 . 9 29 . 73 28 . 7 30 30 BLEU-4 BLEU-4 20 20 10 . 79 8 . 64 10 10 Base Angeli k -lm k -lm-dmv Base Angeli k -lm k -lm-dmv Atis WinHelp 39 . 03 40 40 38 . 26 32 . 21 30 . 37 29 . 3 30 30 26 . 77 BLEU-4 BLEU-4 20 20 16 . 02 11 . 85 10 10 Base Angeli k -lm k -lm-dmv Base Angeli k -lm k -lm-dmv Konstas (ILCC) Concept-to-Text Generation 21 July 2015 30 / 56

  52. Joint Approaches Results Results: Human Evaluation (Fluency) RoboCup WeatherGov 5 5 4 . 61 4 . 47 4 . 31 4 . 26 4 . 03 3 . 92 4 4 3 3 Fluency Fluency 2 . 47 1 . 82 2 2 1 1 0 0 Base k -lm-dmvAngeli Human Base k -lm-dmvAngeli Human Atis WinHelp 5 5 4 . 15 4 . 1 4 . 01 4 4 3 . 56 3 . 57 3 . 41 3 3 Fluency Fluency 2 . 57 2 . 4 2 2 1 1 0 0 Base k -lm-dmvAngeli Human Base k -lm-dmvAngeli Human Konstas (ILCC) Concept-to-Text Generation 21 July 2015 31 / 56

  53. Joint Approaches Results Output WeatherGov Temperature Cloud Sky Cover Chance of Rain Time Mode Time Min Mean Max Time Percent (%) 06:00-11:00 Slight Chance 06:00-21:00 30 38 44 06:00-21:00 75-100 Wind Speed Precipitation Potential (%) Wind Direction Time Mode Time Min Mean Max Time Min Mean Max 06:00-21:00 ENE 06:00-21:00 6 6 7 06:00-21:00 9 20 35 k - Best : A chance of rain showers before 11am. Mostly cloudy, with a high near 44. East wind between 6 and 7 mph. Angeli : A chance of showers. Patchy fog before noon. Mostly cloudy, with a high near 44. East wind between 6 and 7 mph. Chance of precipitation is 35% Human : A 40 percent chance of showers before 10am. Mostly cloudy, with a high near 44. East northeast wind around 7 mph. Konstas (ILCC) Concept-to-Text Generation 21 July 2015 32 / 56

  54. Joint Approaches Results Output Atis Flight Day Search Input: type what from to day dep/ar/ret query flight milwaukee phoenix saturday departure What are the flights from Milwuakee to Phoenix on Saturday k - Best : Show me the flights between Milwuakee and Phoenix on Saturday Angeli : Milwuakee to Phoenix on Saturday Human : Konstas (ILCC) Concept-to-Text Generation 21 July 2015 33 / 56

  55. Joint Approaches Results Dependency Output Atis ROOT on show on me on from on Phoenix Phoenix show me the flights from Milwaukee to Phoenix on Saturday Konstas (ILCC) Concept-to-Text Generation 21 July 2015 34 / 56

  56. Joint Approaches Results Conclusions Generation as parsing problem Unsupervised end-to-end generation system Performance comparable to state-of-the-art Konstas (ILCC) Concept-to-Text Generation 21 July 2015 35 / 56

  57. Joint Approaches Results Conclusions Generation as parsing problem Unsupervised end-to-end generation system Performance comparable to state-of-the-art What about document planning? Konstas (ILCC) Concept-to-Text Generation 21 July 2015 35 / 56

  58. Joint Approaches Inducing Document Planning Traditional NLG Pipeline Communicative Goal Input Data Content Planning Kim and Mooney (2010) Content Selection Angeli et al. (2010) Document Planning Konstas and Lapata (2012a, Sentence Planning 2012b, 2013a) Konstas and Lapata (2013a) Surface Realisation Text Konstas (ILCC) Concept-to-Text Generation 21 July 2015 36 / 56

  59. Joint Approaches Inducing Document Planning Traditional NLG Pipeline Communicative Goal Input Data Content Planning Kim and Mooney (2010) Content Selection Angeli et al. (2010) Document Planning Konstas and Lapata (2012a, Sentence Planning 2012b, 2013a) Konstas and Lapata (2013a) Surface Realisation Text Konstas (ILCC) Concept-to-Text Generation 21 July 2015 36 / 56

  60. Joint Approaches Inducing Document Planning Konstas and Lapata, EMNLP 2013 Inducing Document Plans for Concept-to-text Generation, EMNLP 2013 Konstas (ILCC) Concept-to-Text Generation 21 July 2015 37 / 56

  61. Joint Approaches Key Idea Key Idea Desktop Start Location Start Target Cmd Name Type Name Type Cmd Name Type Cmd Name Type left-click settings button start menu button left-click start button left-click control panel button control panel window Navigate Window Context Menu Action Context Menu Window Target Cmd Name Type Cmd Name Type Cmd Name Type Cmd Name Type left-click advanced tab left-click advanced button left-click accounts and users window double-click users and passwords item Click start, point to settings, and then click control panel. Double-click users and passwords. On the advanced tab, click advanced. Konstas (ILCC) Concept-to-Text Generation 21 July 2015 38 / 56

  62. Joint Approaches Key Idea Key Idea Desktop Start Location Start Target Cmd Name Type Name Type Cmd Name Type Cmd Name Type left-click settings button start menu button left-click start button left-click control panel button control panel window Navigate Window Context Menu Action Context Menu Window Target Cmd Name Type Cmd Name Type Cmd Name Type Cmd Name Type left-click advanced tab left-click advanced button left-click accounts and users window double-click users and passwords item Click start, point to settings, and then click control panel. Double-click users and passwords. On the advanced tab, click advanced. Konstas (ILCC) Concept-to-Text Generation 21 July 2015 38 / 56

  63. Joint Approaches Key Idea Key Idea Desktop Start Start Target Cmd Name Type Cmd Name Type Cmd Name Type left-click settings button left-click start button left-click control panel button Window Target Cmd Name Type double-click users and passwords item Context Menu Action Context Menu Cmd Name Type Cmd Name Type left-click advanced tab left-click advanced button Click start, point to settings, and then click control panel. Double-click users and passwords. On the advanced tab, click advanced. Konstas (ILCC) Concept-to-Text Generation 21 July 2015 38 / 56

  64. Joint Approaches Key Idea Key Idea Desktop Start Start Target Cmd Name Type Cmd Name Type Cmd Name Type left-click settings button left-click start button left-click control panel button Window Target Cmd Name Type double-click users and passwords item Context Menu Action Context Menu Cmd Name Type Cmd Name Type left-click advanced tab left-click advanced button Click start, point to settings, and then click control panel. Double-click users and passwords. On the advanced tab, click advanced. Konstas (ILCC) Concept-to-Text Generation 21 July 2015 38 / 56

  65. Joint Approaches Key Idea Key Idea Desktop Start Start Target Cmd Name Type Cmd Name Type Cmd Name Type left-click settings button left-click start button left-click control panel button Window Target Cmd Name Type double-click users and passwords item Context Menu Action Context Menu Cmd Name Type Cmd Name Type left-click advanced tab left-click advanced button Click start, point to settings, and then click control panel. Double-click users and passwords. On the advanced tab, click advanced. Konstas (ILCC) Concept-to-Text Generation 21 July 2015 38 / 56

  66. Joint Approaches Key Idea Key Idea Key Idea: Grammar-based document plans Konstas (ILCC) Concept-to-Text Generation 21 July 2015 39 / 56

  67. Joint Approaches Key Idea Key Idea Key Idea: Grammar-based document plans Re-use the generation model based on a PCFG grammar of input Konstas (ILCC) Concept-to-Text Generation 21 July 2015 39 / 56

  68. Joint Approaches Key Idea Key Idea Key Idea: Grammar-based document plans Re-use the generation model based on a PCFG grammar of input Replace existing locally coherent Content Selection model and incorporate global Document Planning (explore two solutions): Konstas (ILCC) Concept-to-Text Generation 21 July 2015 39 / 56

  69. Joint Approaches Key Idea Key Idea Key Idea: Grammar-based document plans Re-use the generation model based on a PCFG grammar of input Replace existing locally coherent Content Selection model and incorporate global Document Planning (explore two solutions): Konstas (ILCC) Concept-to-Text Generation 21 July 2015 39 / 56

  70. Joint Approaches Key Idea Key Idea Key Idea: Grammar-based document plans Re-use the generation model based on a PCFG grammar of input Replace existing locally coherent Content Selection model and incorporate global Document Planning (explore two solutions): Patterns of record sequences within a sentence and among sentences Rhetorical Structure Theory (Mann and Thompson, 1988) inspired plans Konstas (ILCC) Concept-to-Text Generation 21 July 2015 39 / 56

  71. Joint Approaches Planning with Record Sequences Planning with Record Sequences Key idea: Grammar on sequences of record types Konstas (ILCC) Concept-to-Text Generation 21 July 2015 40 / 56

  72. Joint Approaches Planning with Record Sequences Planning with Record Sequences Key idea: Grammar on sequences of record types 1 Click start, point to settings, and then click control panel. � Double-click users and passwords. � On the advanced tab, click advanced. � Split a document into sentences, each terminated by a full-stop. Konstas (ILCC) Concept-to-Text Generation 21 July 2015 40 / 56

  73. Joint Approaches Planning with Record Sequences Planning with Record Sequences Key idea: Grammar on sequences of record types 1 Click start, point to settings, and then click control panel. � Double-click users and passwords. � On the advanced tab, click advanced. � Split a document into sentences, each terminated by a full-stop. desktop | start | start-target Click start, point to settings, and then click control panel. � 2 window-target contextMenu | action-contextMenu Double-click users and passwords. � On the advanced tab, click advanced. � Then split a sentence further into a sequence of record types. Konstas (ILCC) Concept-to-Text Generation 21 July 2015 40 / 56

  74. Joint Approaches Planning with Record Sequences Planning with Record Sequences Key idea: Grammar on sequences of record types 1 Click start, point to settings, and then click control panel. � Double-click users and passwords. � On the advanced tab, click advanced. � Split a document into sentences, each terminated by a full-stop. desktop | start | start-target Click start, point to settings, and then click control panel. � 2 window-target contextMenu | action-contextMenu Double-click users and passwords. � On the advanced tab, click advanced. � Then split a sentence further into a sequence of record types. 3 Goal: Learn patterns of record type sequences within and among sentences Konstas (ILCC) Concept-to-Text Generation 21 July 2015 40 / 56

Recommend

GANocracy Outline Background: Text Generation Latent-Variable Generation Learning

GANocracy Outline Background: Text Generation Latent-Variable Generation Learning

Controlling Text Generation Alexander Rush (and Sam Wiseman) Harvard / Cornell Tech GANocracy Outline Background: Text Generation Latent-Variable Generation Learning Neural Templates Machine Learning for Text Generation y 1: T =

710 views • 55 slides
Unsupervised Machine Translation Sachin Kumar Conditional Text Generation Generate text

Unsupervised Machine Translation Sachin Kumar Conditional Text Generation Generate text

CMU CS11-737: Multilingual NLP (Fall 2020) Unsupervised Machine Translation Sachin Kumar Conditional Text Generation Generate text according to a specification: P(Y|X) Input X Output Y (Text) Task English Hindi Machine Translation

743 views • 35 slides
Unsupervised Concept-to-text Generation with Hypergraphs Ioannis Konstas, Mirella Lapata

Unsupervised Concept-to-text Generation with Hypergraphs Ioannis Konstas, Mirella Lapata

Unsupervised Concept-to-text Generation with Hypergraphs Ioannis Konstas, Mirella Lapata Institute for Language, Cognition and Computation University of Edinburgh NAACL 2012, Montral Konstas, Lapata (ILCC) Unsupervised Concept-to-text

1.47k views • 62 slides
Speech Generation From Concept and from Text Martin Jansche CS 6998 2004-02-11 Components of

Speech Generation From Concept and from Text Martin Jansche CS 6998 2004-02-11 Components of

Speech Generation From Concept and from Text Martin Jansche CS 6998 2004-02-11 Components of spoken output systems Front end: From input to control parameters. From naturally occurring text; or From constrained mark-up language; or

420 views • 27 slides
Statistical Perspectives on Text-to-Text Generation Noah Smith Language Technologies Institute

Statistical Perspectives on Text-to-Text Generation Noah Smith Language Technologies Institute

Statistical Perspectives on Text-to-Text Generation Noah Smith Language Technologies Institute Machine Learning Department School of Computer Science Carnegie Mellon University nasmith@cs.cmu.edu Im A Learning Guy I use statistics for

1.2k views • 82 slides
Concept-to-text Generation via Discriminative Reranking Ioannis Konstas and Mirella Lapata School

Concept-to-text Generation via Discriminative Reranking Ioannis Konstas and Mirella Lapata School

Concept-to-text Generation via Discriminative Reranking Ioannis Konstas and Mirella Lapata School of Informatics Institute for Language, Cognition and Computation University of Edinburgh ACL 2012, Jeju Island Konstas, Lapata (ILCC)

1.17k views • 69 slides
Text-to-Text Generation Katja Filippova katjaf@google.com Friday, August 19, 2011 1 This

Text-to-Text Generation Katja Filippova katjaf@google.com Friday, August 19, 2011 1 This

Text-to-Text Generation Katja Filippova katjaf@google.com Friday, August 19, 2011 1 This course A quick overview of a number of topics under the umbrella term text-to-text generation. Research problems - what is being done and

1.95k views • 119 slides
Newburgh Community Trust Wind Power Project Concept Generate

Newburgh Community Trust Wind Power Project Concept Generate

Newburgh Community Trust Wind Power Project Concept Generate the Go Carbon areas electricity Neutral requirements. Generate as Cover domes@c much

493 views • 11 slides
Neural AMR : Sequence-to-Sequence Models for Parsing and Generation annis Konstas joint work

Neural AMR : Sequence-to-Sequence Models for Parsing and Generation annis Konstas joint work

Neural AMR : Sequence-to-Sequence Models for Parsing and Generation annis Konstas joint work with Srinivasan Iyer, Mark Yatskar, Yejin Choi, Luke Zettlemoyer AMR graph Generate from AMR graph text Decoder Encoder text Attention AMR

1.32k views • 116 slides
Concept and text by

Concept and text by

Mozan as Urkesh: Archaeology in the Making Concept and text by Giorgio Buccellati Arabic translation by Rasha Endari Lojain Hatahet Yasmine Mahmud 2008 B y definition,

551 views • 24 slides
Language to Image Generation Generate a bird with Generate a bird with Generate a bird

Language to Image Generation Generate a bird with Generate a bird with Generate a bird

Language to Image Generation Generate a bird with Generate a bird with Generate a bird with wings that are blue and wings that are black wings that are red and and a white a yellow a red be red belly lly white belly belly

667 views • 32 slides
Contents Text Mining Concept Tasks Twitter Data Analysis with R Twitter Extracting Tweets

Contents Text Mining Concept Tasks Twitter Data Analysis with R Twitter Extracting Tweets

Text Mining with R Yanchang Zhao http://www.RDataMining.com Tutorial on Machine Learning with R The Melbourne Data Science Week 2017 1 June 2017 Chapter 10: Text Mining, in R and Data Mining: Examples and Case Studies .

733 views • 57 slides
11-830 Computational Ethics for NLP Ethical Concerns on OpenAI Text Generation System Discussion

11-830 Computational Ethics for NLP Ethical Concerns on OpenAI Text Generation System Discussion

11-830 Computational Ethics for NLP Ethical Concerns on OpenAI Text Generation System Discussion OpenAI Text Generation System CNN: This AI is so good at writing that its creators wont let you use it Takes in a text prompt and

336 views • 8 slides
Text-to-Image Generation Yu Cheng Text-to-Image Synthesis Text-to-Image Synthesis

Text-to-Image Generation Yu Cheng Text-to-Image Synthesis Text-to-Image Synthesis

Text-to-Image Generation Yu Cheng Text-to-Image Synthesis Text-to-Image Synthesis StackGAN, AttnGAN, TAGAN, ObjGAN Text-to-Video Synthesis GAN-based methods, VAE-based methods, StoryGAN Dialogue-based Image Synthesis

827 views • 45 slides
From Text Message to Research Paper: Teaching the Text Message Generation Heidi Wright Ohio

From Text Message to Research Paper: Teaching the Text Message Generation Heidi Wright Ohio

From Text Message to Research Paper: Teaching the Text Message Generation Heidi Wright Ohio State University Introduction and Organization Origins- SIUC 2006 Increased number of tasks and shorter time frame Insufficient background

241 views • 22 slides
Feat u re Generation FE ATU R E E N G IN E E R IN G W ITH P YSPAR K John Hog u e Lead Data

Feat u re Generation FE ATU R E E N G IN E E R IN G W ITH P YSPAR K John Hog u e Lead Data

Feat u re Generation FE ATU R E E N G IN E E R IN G W ITH P YSPAR K John Hog u e Lead Data Scientist Wh y generate ne w feat u res ? M u ltipl y ing S u mming Di erencing Di v iding FEATURE ENGINEERING WITH PYSPARK Wh y generate ne w feat

704 views • 34 slides
SAE AERO MICRO PRESENTATION 2: CONCEPT GENERATION AND EVALUATION NAU CAPSTONE 2019-2020: THE PROP

SAE AERO MICRO PRESENTATION 2: CONCEPT GENERATION AND EVALUATION NAU CAPSTONE 2019-2020: THE PROP

SAE AERO MICRO PRESENTATION 2: CONCEPT GENERATION AND EVALUATION NAU CAPSTONE 2019-2020: THE PROP DOGS CORBIN MILLER, ELI PERLEBERG, AND ZACH SIMMONS 10/8/19 Zach PROJECT REVIEW &amp; DESCRIPTION SELF-LEARNING RADIO CONTROL SYSTEMS

479 views • 19 slides
Parsing Eric McCreath Overview In this lecture we will look at: structured text, generation,

Parsing Eric McCreath Overview In this lecture we will look at: structured text, generation,

Parsing Eric McCreath Overview In this lecture we will look at: structured text, generation, parsing, tokens, grammars, and writing a simple parser. 2 Structured Text Information is often stored in text files. Also we often provide

522 views • 13 slides
Non-Monotonic Sequential Text Generation Sean Welleck, Kiant Brantley, Hal Daum III,

Non-Monotonic Sequential Text Generation Sean Welleck, Kiant Brantley, Hal Daum III,

Non-Monotonic Sequential Text Generation Sean Welleck, Kiant Brantley, Hal Daum III, Kyunghyun Cho Sequential Text Generation Y = ( y 1 , y 2 , , y N ) ( hi , how , are , you , ? ) Sequential Text Generation Unconditional Y ( hi ,

459 views • 24 slides
How to Generate Insight Using Text Network Analysis Dmitry Paranyushkin 1. Cognitive Network

How to Generate Insight Using Text Network Analysis Dmitry Paranyushkin 1. Cognitive Network

How to Generate Insight Using Text Network Analysis Dmitry Paranyushkin 1. Cognitive Network Protocol (draft proposal) data packet the moon is white timestamp sender receiver type: RELATED_TO (:TO) timestamp: ymdhmsms

618 views • 25 slides
What can Statistical Machine Translation teach Neural Text Generation about Optimization? Graham

What can Statistical Machine Translation teach Neural Text Generation about Optimization? Graham

What can Statistical Machine Translation teach Neural Text Generation about Optimization? Graham Neubig @ NAACL Workshop on Methods for Optimizing and Evaluating Neural Language Generation 6/6/2019 or How to Optimize your Neural Generation

638 views • 34 slides
+ How to generate security cameras: Towards defence generation for socio-technical systems Olga

+ How to generate security cameras: Towards defence generation for socio-technical systems Olga

+ How to generate security cameras: Towards defence generation for socio-technical systems Olga Gadyatskaya (SnT, University of Luxembourg) olga.gadyatskaya@uni.lu ADT2P + Agenda 2 n Socio-technical models and attack generation n

778 views • 28 slides
Text Generation with Exemplar-based Adaptive Decoding Hao Peng, Ankur Parikh, Manaal Faruqui,

Text Generation with Exemplar-based Adaptive Decoding Hao Peng, Ankur Parikh, Manaal Faruqui,

Text Generation with Exemplar-based Adaptive Decoding Hao Peng, Ankur Parikh, Manaal Faruqui, Bhuwan Dhingra, Dipanjan Das @NAACL June 4, 2019 Outline Background and Overview Adaptive Decoding Experiments Conditioned Text Generation

652 views • 49 slides
I have nothing to disclose Learning objectives: in 1 hour you will be able to 1. Generate a

I have nothing to disclose Learning objectives: in 1 hour you will be able to 1. Generate a

Management of Common Knee Disorders: What You Kneed to Know UCSF Essentials of Womens Health July 8, 2015 Carlin Senter, M.D. I have nothing to disclose Learning objectives: in 1 hour you will be able to 1. Generate a differential

438 views • 32 slides

More recommend