spell once summon anywhere a two level open vocabulary
play

Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language - PowerPoint PPT Presentation

Jan 12, 2023 •460 likes •1.53k views

Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model AAAI 2019 Technical Track Sabrina J. Mielke and Jason Eisner sjmielke@jhu.edu , jason@cs.jhu.edu Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA

  1. Language modeling: a generative story of text p ( the cat chased the ) = p ( the ) · p ( cat | the ) · p ( chased | the cat ) · p ( the | the cat chased ) Lexicon / vocabulary Text generation with an RNN type spelling embedding σ ( w ) e ( w ) RNN RNN RNN RNN w cell cell cell cell � [ 0.2, ··· ,0.0 ] the 1 � [ 0.4, ··· ,0.5 ] the cat c a g e d the cat 2 UNK � [ − 0.1, ··· ,0.2 ] chased 3 � [ 0.3, ··· ,0.1 ] UNK 4 ...but what is the word? Pure character-level model as the solution? Ugh, spelling the again ... RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e t h e c a t c h a s e d 2

  2. Language modeling: a generative story of text p ( the cat chased the ) = p ( the ) · p ( cat | the ) · p ( chased | the cat ) · p ( the | the cat chased ) Lexicon / vocabulary Text generation with an RNN type spelling embedding σ ( w ) e ( w ) RNN RNN RNN RNN w cell cell cell cell � [ 0.2, ··· ,0.0 ] the 1 � [ 0.4, ··· ,0.5 ] the cat c a g e d the cat 2 UNK � [ − 0.1, ··· ,0.2 ] chased 3 � [ 0.3, ··· ,0.1 ] UNK 4 ...but what is the word? Pure character-level model as the solution? Ugh, spelling the again ... RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell ...can’t we memorize it? t h e t h e c a t c h a s e d 2

  3. Our model: Spell once, summon anywhere 3

  4. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 � ) e ( 2 � ) e ( 3 3

  5. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN cell t � ) e ( 2 � ) e ( 3 3

  6. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN RNN cell cell t h � ) e ( 2 � ) e ( 3 3

  7. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) e ( 2 � ) e ( 3 3

  8. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 � ) e ( 2 � ) e ( 3 3

  9. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 � ) e ( 3 3

  10. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 � ) e ( 3 RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  11. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 � ) e ( 3 RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  12. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: � h 1 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 � ) e ( 3 RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  13. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: � h 1 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 embeddings � = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 � ) e ( 3 RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  14. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: � h 1 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 embeddings � = 1 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the � ) e ( 3 σ ( w 1 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  15. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN � � h 1 h 2 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 embeddings � = 1 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the � ) e ( 3 σ ( w 1 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  16. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN � � h 1 h 2 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 embeddings � � = 1 = 2 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the � ) e ( 3 σ ( w 1 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  17. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN � � h 1 h 2 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 embeddings � � = 1 = 2 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat � ) e ( 3 σ ( w 1 ) σ ( w 2 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  18. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN RNN � � � h 1 h 2 h 3 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 embeddings � � = 1 = 2 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat � ) e ( 3 σ ( w 1 ) σ ( w 2 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  19. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN RNN � � � h 1 h 2 h 3 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 w 3 embeddings � � � = 1 = 2 = 3 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat � ) e ( 3 σ ( w 1 ) σ ( w 2 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  20. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN RNN � � � h 1 h 2 h 3 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 w 3 embeddings � � � = 1 = 2 = 3 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat chased � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 3 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  21. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN RNN RNN � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 w 3 embeddings � � � = 1 = 2 = 3 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat chased � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 3 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  22. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN RNN RNN � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 w 3 w 4 embeddings � � � � = 1 = 2 = 3 = 1 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat chased � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 3 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  23. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN RNN RNN � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 w 3 w 4 embeddings � � � � = 1 = 2 = 3 = 1 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat chased the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 3 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  24. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 4 ? � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  25. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 4 ? � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  26. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  27. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. Unknown words are spelled out “on-demand” using the same character-level model. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t RNN cell � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  28. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. Unknown words are spelled out “on-demand” using the same character-level model. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t RNN RNN cell cell � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  29. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. Unknown words are spelled out “on-demand” using the same character-level model. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t RNN RNN RNN cell cell cell � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  30. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. Unknown words are spelled out “on-demand” using the same character-level model. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t RNN RNN RNN RNN cell cell cell cell � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  31. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. Unknown words are spelled out “on-demand” using the same character-level model. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t RNN RNN RNN RNN RNN cell cell cell cell cell � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  32. Samples from the model Sampled text from our model: Following the death of Edward Mc- Cartney in 1060 , the new defini- tion was transferred to the WDIC of Fullett . 5

  33. Samples from the model Sampled text from our model: Following the death of Edward Mc- Cartney in 1060 , the new defini- tion was transferred to the WDIC of Fullett . novel word with contextually appropriate spelling 5

  34. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding Following the death of Edward Mc- Cartney in 1060 , the new defini- tion was transferred to the WDIC of Fullett . novel word with contextually appropriate spelling 5

  35. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- tion was transferred to the WDIC of Fullett . novel word with contextually appropriate spelling 5

  36. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC of Fullett . novel word with contextually appropriate spelling 5

  37. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC Clive Dickey ❀ of Fullett . novel word with contextually appropriate spelling 5

  38. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC Clive Dickey ❀ of Fullett . Southport Strigger ❀ novel word with contextually appropriate spelling 5

  39. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC Clive Dickey ❀ of Fullett . Southport Strigger ❀ novel word Carl Wuly ❀ with contextually appropriate spelling 5

  40. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC Clive Dickey ❀ of Fullett . Southport Strigger ❀ novel word Carl Wuly ❀ with contextually Chants Tranquels ❀ appropriate spelling 5

  41. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC Clive Dickey ❀ of Fullett . Southport Strigger ❀ novel word Carl Wuly ❀ with contextually Chants Tranquels ❀ appropriate spelling valuables migrations ❀ 5

  42. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC Clive Dickey ❀ of Fullett . Southport Strigger ❀ novel word Carl Wuly ❀ with contextually Chants Tranquels ❀ appropriate spelling valuables migrations ❀ So why is this a good way of modeling language? 5

  43. Linguistic notions: duality of patterning The meaningful elements in any language—"words" in everyday parlance [ ... ] — [ ... ] are represented by [ a ] small stock of distinguishable sounds which are in characters themselves wholly meaningless. – Hockett, 1960 6

  44. Linguistic notions: duality of patterning The meaningful elements in any language—"words" in everyday parlance [ ... ] — [ ... ] are represented by [ a ] small stock of distinguishable sounds which are in characters themselves wholly meaningless. – Hockett, 1960 “ Meaningless ” character composition should be separate from “ meaningful ” word composition! 6

  45. Linguistic notions: duality of patterning The meaningful elements in any language—"words" in everyday parlance [ ... ] — [ ... ] are represented by [ a ] small stock of distinguishable sounds which are in characters themselves wholly meaningless. – Hockett, 1960 “ Meaningless ” character composition should be separate from “ meaningful ” word composition! We should need a word’s spelling only to define it – not to later use it. 6

  46. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? 7

  47. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? • Irregular words have uncommon spellings children ...yet we use them like regular words! 7

  48. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? • Irregular words have uncommon spellings children ...yet we use them like regular words! • Function words have uncommon spellings the , of ...yet we use them all the time without feeling weird! 7

  49. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? • Irregular words have uncommon spellings children ...yet we use them like regular words! • Function words have uncommon spellings the , of ...yet we use them all the time without feeling weird! We should need a word’s spelling only to define it – not to later use it. Recall: 7

  50. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? • Irregular words have uncommon spellings children ...yet we use them like regular words! • Function words have uncommon spellings the , of ...yet we use them all the time without feeling weird! We should need a word’s spelling only to define it – not to later use it. Recall: i.e. character-level models do it wrong! 7

  51. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? • Irregular words have uncommon spellings children ...yet we use them like regular words! • Function words have uncommon spellings the , of ...yet we use them all the time without feeling weird! We should need a word’s spelling only to define it – not to later use it. Recall: i.e. character-level models do it wrong! ...and they’re slow as hell... 7

  52. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? • Irregular words have uncommon spellings children ...yet we use them like regular words! • Function words have uncommon spellings the , of ...yet we use them all the time without feeling weird! We should need a word’s spelling only to define it – not to later use it. Recall: i.e. character-level models do it wrong! ...and they’re slow as hell... usage ⊥ spelling | embedding 7

  53. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? 8

  54. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. – de Saussure, 1916, translated 8

  55. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated 8

  56. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated Meaning is not fully predictable from spellings. 8

  57. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated Meaning is not fully predictable from spellings. Example: neither silly nor folly is an adverb, even though they both end in - ly ! 8

  58. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated Meaning is not fully predictable from spellings. Example: neither silly nor folly is an adverb, even though they both end in - ly ! “construction” models like e ( caged ) : = CNN ( c a g e d ) ignore this! 8

  59. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated Meaning is not fully predictable from spellings. Example: neither silly nor folly is an adverb, even though they both end in - ly ! “construction” models like e ( caged ) : = CNN ( c a g e d ) ignore this! ⇒ Allow any pairing a priori, but use spellings as prior / regularization! 8

  60. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated Meaning is not fully predictable from spellings. Example: neither silly nor folly is an adverb, even though they both end in - ly ! “construction” models like e ( caged ) : = CNN ( c a g e d ) ignore this! ⇒ Allow any pairing a priori, but Outliers ( children , the , ...) use spellings as prior / regularization! may have idiosyncratic embeddings! 8

  61. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated Meaning is not fully predictable from spellings. Example: neither silly nor folly is an adverb, even though they both end in - ly ! “construction” models like e ( caged ) : = CNN ( c a g e d ) ignore this! ⇒ Allow any pairing a priori, but Outliers ( children , the , ...) use spellings as prior / regularization! may have idiosyncratic embeddings! regularize embeddings, don’t construct them 8

  62. Recap: how does our model implement these ideas? Embeddings and spellings are connected on the type level, ensuring conditional independence of usage and spelling while assigning positive probability to any pairing! e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t RNN RNN RNN RNN RNN cell cell cell cell cell � ) σ ( 2 the cat c a g e d the � ) e ( 3 s ∼ p spell ( · | � σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) h 3 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 9

  63. How do we evaluate open-vocabulary language models? 1. Report likelihood p ( held-out text ) as perplexity? ( ↓ lower is better) 10

  64. How do we evaluate open-vocabulary language models? bits per character 1. Report likelihood p ( held-out text ) as perplexity ( ↓ lower is better) 10

  65. How do we evaluate open-vocabulary language models? bits per character 1. Report likelihood p ( held-out text ) as perplexity ( ↓ lower is better) 2. no UNKing allowed! 10

  66. How do we evaluate open-vocabulary language models? bits per character 1. Report likelihood p ( held-out text ) as perplexity ( ↓ lower is better) no UNKing allowed! ∗ 2. 10

  67. How do we evaluate open-vocabulary language models? bits per character 1. Report likelihood p ( held-out text ) as perplexity ( ↓ lower is better) no UNKing allowed! ∗ 2. ______________ ∗ Yes, we call some words “UNK” temporarily , but we still generate them fully ! 10

  68. How do we evaluate open-vocabulary language models? bits per character 1. Report likelihood p ( held-out text ) as perplexity ( ↓ lower is better) no UNKing allowed! ∗ 2. → we must predict every character of the text, regardless of vocabulary size ______________ ∗ Yes, we call some words “UNK” temporarily , but we still generate them fully ! 10

  69. How do we evaluate open-vocabulary language models? bits per character 1. Report likelihood p ( held-out text ) as perplexity ( ↓ lower is better) no UNKing allowed! ∗ 2. → we must predict every character of the text, regardless of vocabulary size ______________ ∗ Yes, we call some words “UNK” temporarily , but we still generate them fully ! ⇒ A tunable “vocabulary size” hyperparameter decides what is temporary-UNK. 10

  70. Results test WikiText-2 (Merity et al., 2017) ← 1.8 1.4 � 2.5 million tokenized words from the English Wikipedia 11

  71. Results test WikiText-2 (Merity et al., 2017) ← 1.8 1.4 � 2.5 million tokenized words from the English Wikipedia RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN � 1.775 cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e c a t c h a s e d 11

  72. Results test WikiText-2 (Merity et al., 2017) ← 1.8 1.4 � 2.5 million tokenized words from the English Wikipedia RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN � 1.775 cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e c a t c h a s e d HCLM + cache � 1.500 previous SOTA (Kawakami et al., 2017) 11

  73. Results test WikiText-2 (Merity et al., 2017) ← 1.8 1.4 � 2.5 million tokenized words from the English Wikipedia RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN � 1.775 cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e c a t c h a s e d HCLM + cache � 1.500 previous SOTA (Kawakami et al., 2017) RNN RNN RNN RNN RNN � BPE: cell cell cell cell cell 1.468 the ca@ @ t cha@ @ sed 11

  74. Results test WikiText-2 (Merity et al., 2017) ← 1.8 1.4 � 2.5 million tokenized words from the English Wikipedia RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN � 1.775 cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e c a t c h a s e d HCLM + cache � 1.500 previous SOTA (Kawakami et al., 2017) RNN RNN RNN RNN RNN � BPE: cell cell cell cell cell 1.468 the ca@ @ t cha@ @ sed 11

  75. Results test WikiText-2 (Merity et al., 2017) ← 1.8 1.4 � 2.5 million tokenized words from the English Wikipedia RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN � 1.775 cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e c a t c h a s e d HCLM + cache � 1.500 previous SOTA (Kawakami et al., 2017) RNN RNN RNN RNN RNN � BPE: cell cell cell cell cell 1.468 the ca@ @ t cha@ @ sed � our full model: Spell Once, Summon Anywhere 1.455 11

  76. Results on dev data test WikiText-2 (Merity et al., 2017) rare novel frequent 2.5 million tokenized words from the English Wikipedia all words words words RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN 3.89 2.08 1.38 1.775 cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e c a t c h a s e d HCLM + cache – – – 1.500 previous SOTA (Kawakami et al., 2017) RNN RNN RNN RNN RNN BPE: cell cell cell cell cell 4.01 1.70 1.08 1.468 the ca@ @ t cha@ @ sed our full model: Spell Once, Summon Anywhere 4.00 1.64 1.10 1.455 11

Recommend

Summon Implementation at IMSA: Illinois Mathematics and Science Academy Paula Garrett The

Summon Implementation at IMSA: Illinois Mathematics and Science Academy Paula Garrett The

4/24/13 Summon Implementation at IMSA: Illinois Mathematics and Science Academy Paula Garrett The Summon Decision IMSA environment High school; college-level research 70 databases (35 vendors) Federated search vs.

240 views • 4 slides
Nuspell: version 3 of the new spell checker FOSS spell checker implemented in C++17 with aid of

Nuspell: version 3 of the new spell checker FOSS spell checker implemented in C++17 with aid of

Nuspell: version 3 of the new spell checker FOSS spell checker implemented in C++17 with aid of Mozilla Sander van Geloven FOSDEM, Brussels February 1, 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

611 views • 39 slides
Nuspell: the new spell checker FOSS spell checker implemented in C++14 with aid of Mozilla.

Nuspell: the new spell checker FOSS spell checker implemented in C++14 with aid of Mozilla.

Nuspell: the new spell checker FOSS spell checker implemented in C++14 with aid of Mozilla. Sander van Geloven FOSDEM, Brussels February 2, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

327 views • 21 slides
Open Vocabulary Learning on Source Code with a Graph-Structured Cache Milan Cvitkovic Badal

Open Vocabulary Learning on Source Code with a Graph-Structured Cache Milan Cvitkovic Badal

Open Vocabulary Learning on Source Code with a Graph-Structured Cache Milan Cvitkovic Badal Singh Anima Anandkumar Caltech, Amazon Web Services Amazon Web Services Caltech ICML, 2019-6-12 Open Vocabulary Learning Goal: Models that can

137 views • 10 slides
Spell Checking: Edit Distance VSM, session 8 CS6200: Information Retrieval Slides by: Jesse

Spell Checking: Edit Distance VSM, session 8 CS6200: Information Retrieval Slides by: Jesse

Spell Checking: Edit Distance VSM, session 8 CS6200: Information Retrieval Slides by: Jesse Anderton Spell Checking poiner sisters 10-15% of all queries contain spelling brimingham news errors, so spell checking can help a catamarn sailing

258 views • 7 slides
LEVEL OPEN CONDITION:- 1 ID DIRECT 10000 = 1 LEVEL 1 ID DIRECT 25000 = 3 LEVEL 1 ID

LEVEL OPEN CONDITION:- 1 ID DIRECT 10000 = 1 LEVEL 1 ID DIRECT 25000 = 3 LEVEL 1 ID

INCENTIVE INCOME o o o o o o o Darwin Platform Group of Companies 21 level5% LEVEL OPEN CONDITION:- 1 ID DIRECT 10000 = 1 LEVEL 1 ID DIRECT 25000 = 3 LEVEL 1 ID DIRECT 50000 = 5 LEVEL 1 ID DIRECT 100000 = 7 LEVEL o

730 views • 42 slides
Character-level Language Models With Word-level Learning Arvid Frydenlund March 16, 2018

Character-level Language Models With Word-level Learning Arvid Frydenlund March 16, 2018

Character-level Language Models With Word-level Learning Arvid Frydenlund March 16, 2018 Character-level Language models Want language models with an open vocabulary Character-level models give this for free Treat the probability of

1.07k views • 5 slides
+ A Quantitative Survey on the Use of the Cube Vocabulary in the Linked Open Data Cloud Karin

+ A Quantitative Survey on the Use of the Cube Vocabulary in the Linked Open Data Cloud Karin

+ A Quantitative Survey on the Use of the Cube Vocabulary in the Linked Open Data Cloud Karin Becker Instituto de Informtica - Federal University of Rio Grande do Sul, Brazil Shiva Jahangiri , Craig A. Knoblock Information Sciences

472 views • 32 slides
Teaching Vocabulary Pre-Teaching Vocabulary + Pre-Teaching Vocabulary: An Example for 2 nd -5 th

Teaching Vocabulary Pre-Teaching Vocabulary + Pre-Teaching Vocabulary: An Example for 2 nd -5 th

Teaching Vocabulary Pre-Teaching Vocabulary + Pre-Teaching Vocabulary: An Example for 2 nd -5 th Grades Teacher says the word and asks students to repeat the 1. word three times. Teacher states the word in context from the text. 2. Teacher

524 views • 31 slides
Generating Fine-Grained Open Vocabulary Entity Type Descriptions Rajarshi Bhowmik and Gerard de

Generating Fine-Grained Open Vocabulary Entity Type Descriptions Rajarshi Bhowmik and Gerard de

Generating Fine-Grained Open Vocabulary Entity Type Descriptions Rajarshi Bhowmik and Gerard de Melo Introduction Knowledge Graph Vast repository of structured facts Why short textual description? Can succinctly characterize

457 views • 20 slides
VOCABULARY ATI TEAS ENGLISH AND LANGUAGE USAGE VOCABULARY Vocabulary questions on this part of

VOCABULARY ATI TEAS ENGLISH AND LANGUAGE USAGE VOCABULARY Vocabulary questions on this part of

ATI TEAS ENGLISH AND LANGUAGE USAGE REVIEW VOCABULARY ATI TEAS ENGLISH AND LANGUAGE USAGE VOCABULARY Vocabulary questions on this part of the TEAS deal with figuring out the meaning of the word based on synonyms, context clues, tone, and your

411 views • 6 slides
VOCABULARY ATI TEAS ENGLISH AND LANGUAGE USAGE VOCABULARY Vocabulary questions on this part of

VOCABULARY ATI TEAS ENGLISH AND LANGUAGE USAGE VOCABULARY Vocabulary questions on this part of

ATI TEAS ENGLISH AND LANGUAGE USAGE REVIEW VOCABULARY ATI TEAS ENGLISH AND LANGUAGE USAGE VOCABULARY Vocabulary questions on this part of the TEAS deal with figuring out the meaning of the word based on synonyms, context clues, tone, and your

336 views • 4 slides
Building Science Vocabulary: Seeds of Science Roots of Reading Goal Review our model for

Building Science Vocabulary: Seeds of Science Roots of Reading Goal Review our model for

Building Science Vocabulary: Seeds of Science Roots of Reading Goal Review our model for vocabulary development Share some vocabulary instructional routines Approach to ELL vocabulary development Vocabulary Research Low

303 views • 28 slides
CLOSE THE VOCABULARY GAP! PROVEN STRATEGIES ECE ADMINISTRATORS CAN USE TO HELP TEACHERS ENHANCE

CLOSE THE VOCABULARY GAP! PROVEN STRATEGIES ECE ADMINISTRATORS CAN USE TO HELP TEACHERS ENHANCE

CLOSE THE VOCABULARY GAP! PROVEN STRATEGIES ECE ADMINISTRATORS CAN USE TO HELP TEACHERS ENHANCE VOCABULARY DEVELOPMENT IN THEIR CLASSROOMS Julie M. Wood, Ed.D. www.juliemwood.com Is Developing Young Childrens Vocabulary Knowledge Really

156 views • 15 slides
7 6 4-5 2-3 0-1 4. Fluency Speech sustained throughout Speech sustained most of Some

7 6 4-5 2-3 0-1 4. Fluency Speech sustained throughout Speech sustained most of Some

300 Listening/Speaking Oral Presentation Evaluation Rubric Maximum points = 49 7 6 4-5 2-3 0-1 1.Vocabulary Vocabulary is consistently Uses general vocabulary Some hesitation and Limited and/or inaccurate Consistently inadequate

161 views • 3 slides
Duration Models Introduction to Single Spell Models James J. Heckman University of Chicago Econ

Duration Models Introduction to Single Spell Models James J. Heckman University of Chicago Econ

Duration Models Introduction to Single Spell Models James J. Heckman University of Chicago Econ 312, Spring 2019 Heckman Duration Models The hazard function gives the probability that a spell, denoted by the nonnegative random variable T

643 views • 21 slides
THE MAIN TECHNICAL AUDIOVISUAL VOCABULARY IN ENGLISH: HOW HOTELLYWOOD CAN HELP By

THE MAIN TECHNICAL AUDIOVISUAL VOCABULARY IN ENGLISH: HOW HOTELLYWOOD CAN HELP By

THE MAIN TECHNICAL AUDIOVISUAL VOCABULARY IN ENGLISH: HOW HOTELLYWOOD CAN HELP By Valentina Celona CONTENTS: The curriculum of our vocational school The use of technical vocabulary in the various courses Technical vocabulary

510 views • 11 slides
Denied, Detained, and Deported Vocabulary Sweiss | English Language Arts Vocabulary Practice 1.

Denied, Detained, and Deported Vocabulary Sweiss | English Language Arts Vocabulary Practice 1.

Denied, Detained, and Deported Vocabulary Sweiss | English Language Arts Vocabulary Practice 1. Exile 2. Nativist 3. Haven 4. Refuge The war victims found ______________________in 5. Hysteria the nearest hospital. 6. Euphemism 7.

475 views • 11 slides
What s in a Word? Academic Vocabulary Development for ELLs CCRC 2014 1 Essential

What s in a Word? Academic Vocabulary Development for ELLs CCRC 2014 1 Essential

What s in a Word? Academic Vocabulary Development for ELLs CCRC 2014 1 Essential Questions How do ELLs learn academic vocabulary? What strategies can we use to teach academic vocabulary? 2 Why does vocabulary matter?

386 views • 11 slides
Vocabulary Across the Workshops PCL Summer Institute, 2017 Colleen McCarthy and Dawn Stiegert

Vocabulary Across the Workshops PCL Summer Institute, 2017 Colleen McCarthy and Dawn Stiegert

Vocabulary Across the Workshops PCL Summer Institute, 2017 Colleen McCarthy and Dawn Stiegert School District of Belleville Purpose - What you signed up for: 1. Strategies to enhance vocabulary skills 2. Choosing vocabulary 3. Deepen

710 views • 47 slides
Assembling a Specialist Vocabulary: starting points. Jenifer Spencer Freelance EAP Consultant

Assembling a Specialist Vocabulary: starting points. Jenifer Spencer Freelance EAP Consultant

Assembling a Specialist Vocabulary: starting points. Jenifer Spencer Freelance EAP Consultant What are the situations? Course type : - level of specificity/ remit/ status / hours/time and mode of delivery Situation - degree of

373 views • 20 slides
WHA HATS I IN N A WORD RD? Explicit Vocabulary Instruction in the ABE, ASE, and CTE

WHA HATS I IN N A WORD RD? Explicit Vocabulary Instruction in the ABE, ASE, and CTE

WHA HATS I IN N A WORD RD? Explicit Vocabulary Instruction in the ABE, ASE, and CTE Classrooms TODA DAYS A AGENDA DA Welcomes + Poll Objectives for T odays Session Vocabulary, Comprehension, and Learning

765 views • 39 slides
Roadmap for Section 9.1 Introduction and Vocabulary Performance Measures General Structure of a

Roadmap for Section 9.1 Introduction and Vocabulary Performance Measures General Structure of a

Unit OS9: Real-Time and Embedded Systems 9.1. Introduction and Vocabulary Windows Operating System Internals - by David A. Solomon and Mark E. Russinovich with Andreas Polze Roadmap for Section 9.1 Introduction and Vocabulary Performance

535 views • 17 slides
Open Water Swimming Speaker: Dave Candler, STA President Qualifications STA Level 1 Award for

Open Water Swimming Speaker: Dave Candler, STA President Qualifications STA Level 1 Award for

Open Water Swimming Speaker: Dave Candler, STA President Qualifications STA Level 1 Award for Pool to Open Water Swimming Coaching STA Level 2 Award for Open Water Swimming Coaching. Current Situation of Leisure Providers Weather

212 views • 17 slides

More recommend