Weighted parsing for grammar-based language models FSMNLP 2019 - PowerPoint PPT Presentation

Language algebras 𝑒 ∈ AST (𝐻) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 … Fruit Fruit 𝛽 … VP → 𝛾( VBZ , PP ) NN → 𝜀 NP → 𝛿( NN ) S → 𝛽( NP , VP ) factors ( Fruit flies like bananas ) = { Fruit , like bananas , … } 7 / 21 ⟨𝑦 1 flies 𝑦 2 ⟩ 𝜌 𝛵 : T 𝑆 → T 𝛵 interpretation of 𝛵 as operations on the set of syntactic objects ℒ = 𝛦 ∗ (.) 𝛦 ∗ : T 𝛵 (terms) → 𝛦 ∗ (syntactic objects)

Language algebras ⟨ Fruit flies … ⟩ 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 … Fruit Fruit 𝑒 ∈ AST (𝐻) … VP → 𝛾( VBZ , PP ) NN → 𝜀 NP → 𝛿( NN ) S → 𝛽( NP , VP ) factors ( Fruit flies like bananas ) = { Fruit , like bananas , … } 7 / 21 𝜌 𝛵 : T 𝑆 → T 𝛵 interpretation of 𝛵 as operations on the set of syntactic objects ℒ = 𝛦 ∗ (.) 𝛦 ∗ : T 𝛵 (terms) → 𝛦 ∗ (syntactic objects)

Language algebras 𝛽 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) Fruit flies … 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 … 𝛾 𝜀 𝛿 𝑒 ∈ AST (𝐻) … VP → 𝛾( VBZ , PP ) NN → 𝜀 NP → 𝛿( NN ) S → 𝛽( NP , VP ) factors ( Fruit flies like bananas ) = { Fruit , like bananas , … } 7 / 21 𝜌 𝛵 : T 𝑆 → T 𝛵 (.) 𝛦 ∗ : T 𝛵 → 𝛦 ∗ interpretation of 𝛵 as operations on the set of syntactic objects ℒ = 𝛦 ∗ (.) 𝛦 ∗ : T 𝛵 (terms) → 𝛦 ∗ (syntactic objects)

Semirings Algebraic structure ( 𝕃 , ⊕ , ⊗ , 𝟙 , 𝟚 ) ⊗ is used to evaluate an AST to a weight ⊕ accumulates the weights of several ASTs Examples ( 𝔺 , ∨, ∧, false , true ) the Boolean semiring with 𝔺 = { false , true } ( ℕ ∞ , +, ⋅, 0, 1) the semiring of natural numbers ( ℕ ∞ , min , +, ∞, 0) the tropical semiring ( ℝ 1 0 , max , ⋅, 0, 1) the Viterbi semiring Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 8 / 21

𝛻 ⊗ = { mul 𝕝 ( 𝕝 1 , … , 𝕝 𝑛 ) = 𝕝 ⊗ 𝕝 1 ⊗ ⋯ ⊗ 𝕝 𝑛 with 𝛻 med = { del , ins , rep = , rep ≠ , nil } Multioperator monoids (M-monoids) mul 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) Minimum edit distance M-monoid ({{𝑜} ∣ 𝑜 ∈ ℕ }, min ∘ ∪, ∅, 𝛻 med ) 0 , max , 0, 𝛻 mul ) Viterbi M-monoid ( ℝ 1 Examples (𝑛) 𝕝 ∣ 𝕝 ∈ 𝕃 , 𝑛 ∈ ℕ } Generalization of semirings (𝑛) M-monoid ( 𝕃 , ⊕ , 𝟙 , 𝛻 ⊗ ) where Semiring ( 𝕃 , ⊕ , ⊗ , 𝟙 , 𝟚 ) ⟶ ( 𝕃 , ⊕ , 𝟙 , 𝛻) ⟶ ( 𝕃 , ⊕ , ⊗ , 𝟙 , 𝟚 ) 9 / 21 binary ⊗ set of 𝑛 -ary operations 𝛻 (here: distributive)

with 𝛻 med = { del , ins , rep = , rep ≠ , nil } Multioperator monoids (M-monoids) Generalization of semirings 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) Minimum edit distance M-monoid ({{𝑜} ∣ 𝑜 ∈ ℕ }, min ∘ ∪, ∅, 𝛻 med ) 0 , max , 0, 𝛻 mul ) Viterbi M-monoid ( ℝ 1 Examples (𝑛) mul ∣ 𝕝 ∈ 𝕃 , 𝑛 ∈ ℕ } 𝕝 (𝑛) ⟶ ( 𝕃 , ⊕ , 𝟙 , 𝛻) ⟶ ( 𝕃 , ⊕ , ⊗ , 𝟙 , 𝟚 ) 9 / 21 binary ⊗ set of 𝑛 -ary operations 𝛻 (here: distributive) Semiring ( 𝕃 , ⊕ , ⊗ , 𝟙 , 𝟚 ) ⇝ M-monoid ( 𝕃 , ⊕ , 𝟙 , 𝛻 ⊗ ) where 𝛻 ⊗ = { mul 𝕝 ( 𝕝 1 , … , 𝕝 𝑛 ) = 𝕝 ⊗ 𝕝 1 ⊗ ⋯ ⊗ 𝕝 𝑛

Multioperator monoids (M-monoids) Generalization of semirings 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) Minimum edit distance M-monoid ({{𝑜} ∣ 𝑜 ∈ ℕ }, min ∘ ∪, ∅, 𝛻 med ) 0 , max , 0, 𝛻 mul ) Viterbi M-monoid ( ℝ 1 Examples (𝑛) mul ∣ 𝕝 ∈ 𝕃 , 𝑛 ∈ ℕ } 𝕝 (𝑛) ⟶ ( 𝕃 , ⊕ , 𝟙 , 𝛻) ⟶ ( 𝕃 , ⊕ , ⊗ , 𝟙 , 𝟚 ) 9 / 21 binary ⊗ set of 𝑛 -ary operations 𝛻 (here: distributive) Semiring ( 𝕃 , ⊕ , ⊗ , 𝟙 , 𝟚 ) ⇝ M-monoid ( 𝕃 , ⊕ , 𝟙 , 𝛻 ⊗ ) where 𝛻 ⊗ = { mul 𝕝 ( 𝕝 1 , … , 𝕝 𝑛 ) = 𝕝 ⊗ 𝕝 1 ⊗ ⋯ ⊗ 𝕝 𝑛 with 𝛻 med = { del , ins , rep = , rep ≠ , nil }

1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2 1 : T 𝛻 (terms) → ℝ 1 0 (weight algebra) 0.2 ⋅ 𝕝 1 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) (.) ℝ 0 0 , max , 0, 𝛻 mul ) ( ℝ 1 wt : 𝑆 (set of rules) → 𝛻 (set of operations) wt (𝑒) ∈ T 𝛻 wt … 1.0 Weight algebras S → 𝛽( NP , VP ) (.) 𝛦 ∗ Fruit flies … 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 𝜌 𝛵 … 𝛾 𝜀 𝛿 𝛽 𝑒 ∈ AST (𝐻) … VP → 𝛾( VBZ , PP ) NN → 𝜀 NP → 𝛿( NN ) 10 / 21

1 : T 𝛻 (terms) → ℝ 1 0 (weight algebra) Weight algebras S → 𝛽( NP , VP ) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) (.) ℝ 0 0 , max , 0, 𝛻 mul ) ( ℝ 1 wt : 𝑆 (set of rules) → 𝛻 (set of operations) wt (𝑒) ∈ T 𝛻 wt … 1.0 0.2 ⋅ 𝕝 1 10 / 21 (.) 𝛦 ∗ Fruit flies … 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 𝜌 𝛵 … 𝛾 𝜀 𝛿 𝛽 𝑒 ∈ AST (𝐻) … VP → 𝛾( VBZ , PP ) NN → 𝜀 NP → 𝛿( NN ) 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2

Weight algebras (.) 𝛦 ∗ 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) (.) ℝ 0 0 , max , 0, 𝛻 mul ) ( ℝ 1 wt : 𝑆 (set of rules) → 𝛻 (set of operations) wt (𝑒) ∈ T 𝛻 wt … 1.0 0.2 ⋅ 𝕝 1 S → 𝛽( NP , VP ) 10 / 21 Fruit flies … 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 NP → 𝛿( NN ) NN → 𝜀 VP → 𝛾( VBZ , PP ) … 𝑒 ∈ AST (𝐻) 𝛽 𝛿 𝜀 𝛾 … 𝜌 𝛵 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2 1 : T 𝛻 (terms) → ℝ 1 0 (weight algebra)

Weight algebras (.) 𝛦 ∗ 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) (.) ℝ 0 0 , max , 0, 𝛻 mul ) ( ℝ 1 wt : 𝑆 (set of rules) → 𝛻 (set of operations) wt (𝑒) ∈ T 𝛻 wt … 1.0 0.2 S → 𝛽( NP , VP ) 10 / 21 Fruit flies … 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 NP → 𝛿( NN ) NN → 𝜀 VP → 𝛾( VBZ , PP ) … 𝑒 ∈ AST (𝐻) 𝛽 𝛿 𝜀 𝛾 … 𝜌 𝛵 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2 1 : T 𝛻 (terms) → ℝ 1 0 (weight algebra)

Weight algebras S → 𝛽( NP , VP ) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) (.) ℝ 0 0 , max , 0, 𝛻 mul ) ( ℝ 1 wt : 𝑆 (set of rules) → 𝛻 (set of operations) wt (𝑒) ∈ T 𝛻 wt 0.6 ⋅ … 1.0 0.2 (.) 𝛦 ∗ Fruit flies … 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 NP → 𝛿( NN ) NN → 𝜀 VP → 𝛾( VBZ , PP ) … 𝑒 ∈ AST (𝐻) 𝛽 𝛿 𝜀 𝛾 … 𝜌 𝛵 10 / 21 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 1 : T 𝛻 (terms) → ℝ 1 0 (weight algebra)

Weight algebras S → 𝛽( NP , VP ) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) (.) ℝ 0 0 , max , 0, 𝛻 mul ) ( ℝ 1 wt : 𝑆 (set of rules) → 𝛻 (set of operations) wt (𝑒) ∈ T 𝛻 wt 0.6 ⋅ … 1.0 0.2 0.12 ⋅ … (.) 𝛦 ∗ Fruit flies … 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 𝜌 𝛵 … 𝛾 𝜀 𝛿 𝛽 𝑒 ∈ AST (𝐻) … VP → 𝛾( VBZ , PP ) NN → 𝜀 NP → 𝛿( NN ) 10 / 21 1 : T 𝛻 (terms) → ℝ 1 0 (weight algebra)

Weight algebras S → 𝛽( NP , VP ) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) (.) ℝ 0 0 , max , 0, 𝛻 mul ) ( ℝ 1 wt : 𝑆 (set of rules) → 𝛻 (set of operations) 0 (.) ℝ 1 0.12 ⋅ … wt (𝑒) ∈ T 𝛻 wt … 1.0 0.2 ⋅ 𝕝 1 10 / 21 (.) 𝛦 ∗ NP → 𝛿( NN ) NN → 𝜀 VP → 𝛾( VBZ , PP ) … 𝑒 ∈ AST (𝐻) 𝛽 𝛿 𝜀 𝛾 … 𝜌 𝛵 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 Fruit flies … 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2 1 : T 𝛻 (terms) → ℝ 1 0 (weight algebra)

ℒ A wRTG-LM is a tuple ( (𝐻 = (𝑂, 𝛵, 𝐵 0 , 𝑆)) Weighted RTG-based language models ⏟ ⏟ ⏟ ⏟ ⏟ ⏟ ⏟ RTG , ⏟ language algebra ), ( 𝕃 , ⊕ , 𝟙 , 𝛻) ⏟ M-monoid , wt ⏟ wt : 𝑆 → 𝛻 ) . Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 ⏟⏟ Defjnition (weighted RTG-based language model) S → 𝛽( NP , VP ) 𝜌 𝛵 NP → 𝛿( NN ) NN → 𝜀 VP → 𝛾( VBZ , PP ) … 𝑒 ∈ AST (𝐻) 𝛽 𝛿 𝜀 𝛾 … 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 (.) 𝛦 ∗ 0.2 ⋅ 𝕝 1 1.0 … wt wt (𝑒) ∈ T 𝛻 0.12 ⋅ … (.) ℝ 1 0 Fruit flies … 11 / 21 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2

Weighted RTG-based language models language algebra ⏟ ⏟ ⏟ ⏟ ⏟ ⏟ ⏟ RTG , ⏟ ), S → 𝛽( NP , VP ) ( 𝕃 , ⊕ , 𝟙 , 𝛻) ⏟ M-monoid , wt ⏟ wt : 𝑆 → 𝛻 ) . Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 ⏟⏟ Defjnition (weighted RTG-based language model) (.) 𝛦 ∗ 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 NP → 𝛿( NN ) NN → 𝜀 VP → 𝛾( VBZ , PP ) … 𝑒 ∈ AST (𝐻) 𝛽 𝛿 𝜀 𝛾 … 𝜌 𝛵 11 / 21 0.2 ⋅ 𝕝 1 1.0 … wt wt (𝑒) ∈ T 𝛻 0.12 ⋅ … (.) ℝ 1 0 Fruit flies … 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2 ℒ A wRTG-LM is a tuple ( (𝐻 = (𝑂, 𝛵, 𝐵 0 , 𝑆))

Outline 1 Weighted RTG-based language models 2 The weighted parsing problem 3 The weighted parsing algorithm Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 12 / 21

The weighted parsing problem 0.2 ⋅ 𝕝 1 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) (.) 𝛦 ∗ Fruit flies … 0 (.) ℝ 1 0.12 ⋅ … wt (𝑒) ∈ T 𝛻 wt … 1.0 13 / 21 S → 𝛽( NP , VP ) 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 𝜌 𝛵 … 𝛾 𝜀 𝛿 𝛽 𝑒 ∈ AST (𝐻) … VP → 𝛾( VBZ , PP ) NN → 𝜀 NP → 𝛿( NN ) 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2

The weighted parsing problem S → 𝛽( NP , VP ) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) (.) 𝛦 ∗ 𝜌 𝛵 (.) 𝛦 ∗ Fruit flies … 0 (.) ℝ 1 0.12 ⋅ … wt (𝑒) ∈ T 𝛻 wt … 1.0 0.2 ⋅ 𝕝 1 𝛿 NP → 𝛿( NN ) NN → 𝜀 VP → 𝛾( VBZ , PP ) … 𝑒 ∈ AST (𝐻) 𝛽 𝜀 𝛾 … 𝜌 𝛵 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 13 / 21 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2 𝑢 ′ ∈ T 𝛵 𝑒 ′ ∈ AST (𝐻)

The weighted parsing problem (.) 𝛦 ∗ 0.12 ⋅ … (.) ℝ 1 0 Fruit flies … (.) 𝛦 ∗ 𝜌 𝛵 wt (𝑒 ′ ) ∈ T 𝛻 S → 𝛽( NP , VP ) wt 0.0144 (.) ℝ 1 0 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 wt (𝑒) ∈ T 𝛻 wt … 𝜀 NP → 𝛿( NN ) NN → 𝜀 VP → 𝛾( VBZ , PP ) … 𝑒 ∈ AST (𝐻) 𝛽 𝛿 𝛾 … 𝜌 𝛵 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 0.2 ⋅ 𝕝 1 1.0 13 / 21 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2 𝑢 ′ ∈ T 𝛵 𝑒 ′ ∈ AST (𝐻)

The weighted parsing problem wt (𝑒 ′ ) ∈ T 𝛻 (.) ℝ 1 0 Fruit flies … (.) 𝛦 ∗ 𝜌 𝛵 (.) 𝛦 ∗ wt S → 𝛽( NP , VP ) 0.0144 (.) ℝ 1 0 ( ℝ 1 0 , max , 0, 𝛻 mul ) Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 0.12 ⋅ … wt (𝑒) ∈ T 𝛻 wt 𝛾 NP → 𝛿( NN ) NN → 𝜀 VP → 𝛾( VBZ , PP ) … 𝑒 ∈ AST (𝐻) 𝛽 𝛿 𝜀 … … 𝜌 𝛵 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 0.2 ⋅ 𝕝 1 1.0 13 / 21 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2 𝑢 ′ ∈ T 𝛵 𝑒 ′ ∈ AST (𝐻)

The weighted parsing problem wt (𝑒 ′ ) ∈ T 𝛻 (.) ℝ 1 0 Fruit flies … (.) 𝛦 ∗ 𝜌 𝛵 (.) 𝛦 ∗ wt S → 𝛽( NP , VP ) 0.0144 (.) ℝ 1 0 ( ℝ 1 0 , max , 0, 𝛻 mul ) Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 0.12 ⋅ … wt (𝑒) ∈ T 𝛻 wt … NP → 𝛿( NN ) NN → 𝜀 VP → 𝛾( VBZ , PP ) … 𝑒 ∈ AST (𝐻) 𝛽 𝛿 𝜀 𝛾 … 𝜌 𝛵 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 0.2 ⋅ 𝕝 1 1.0 13 / 21 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2 max { 𝑢 ′ ∈ T 𝛵 𝑒 ′ ∈ AST (𝐻)

The weighted parsing problem wt (.) ℝ 1 0 Fruit flies … (.) 𝛦 ∗ 𝜌 𝛵 (.) 𝛦 ∗ wt (𝑒 ′ ) ∈ T 𝛻 0.0144 wt (𝑒) ∈ T 𝛻 (.) ℝ 1 0 ( ℝ 1 0 , max , 0, 𝛻 mul ) parse Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 S → 𝛽( NP , VP ) 0.12 ⋅ … wt … NP → 𝛿( NN ) NN → 𝜀 VP → 𝛾( VBZ , PP ) … 𝑒 ∈ AST (𝐻) 𝛽 𝛿 𝜀 𝛾 … 𝜌 𝛵 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 0.2 ⋅ 𝕝 1 1.0 13 / 21 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2 max { 𝑢 ′ ∈ T 𝛵 𝑒 ′ ∈ AST (𝐻)

The weighted parsing problem (.) ℝ 1 Fruit flies … (.) 𝛦 ∗ 𝜌 𝛵 (.) 𝛦 ∗ wt (𝑒 ′ ) ∈ T 𝛻 wt 0.0144 0 S → 𝛽( NP , VP ) ( ℝ 1 0 , max , 0, 𝛻 mul ) parse parse (𝑏) = ∑ ⊕ 𝑒∈ AST (𝐻,𝑏) wt (𝑒) Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 0 (.) ℝ 1 0.12 ⋅ … wt (𝑒) ∈ T 𝛻 NP → 𝛿( NN ) NN → 𝜀 VP → 𝛾( VBZ , PP ) … 𝑒 ∈ AST (𝐻) 𝛽 𝛿 𝜀 𝛾 … 𝜌 𝛵 𝑢 ∈ 𝑀(𝐻) ⊆ T 𝛵 0.2 ⋅ 𝕝 1 1.0 … wt 13 / 21 1.0 ⋅ 𝕝 1 ⋅ 𝕝 2 0.6 ⋅ 𝕝 1 ⋅ 𝕝 2 max { 𝑢 ′ ∈ T 𝛵 𝑒 ′ ∈ AST (𝐻)

The weighted parsing problem Algebraic dynamic programming (Giegerich, Meyer, and Stefgen 2004) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) Shamir 1961) Reduct of a grammar and a syntactic object (cf. Bar-Hillel, Perles, and matrix chain multiplication minimum edit distance Parsing with superior grammars (Knuth 1977; Nederhof 2003) Examples 𝑜 best derivation(s) best derivation(s) derivation forest probability of best derivation string probability recognition Semiring parsing (Goodman 1999) 14 / 21

Outline 1 Weighted RTG-based language models 2 The weighted parsing problem 3 The weighted parsing algorithm Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 15 / 21

((𝐻, ℒ ), 𝕃 , wt ) - 𝑏 ∈ ℒ Weighted parsing algorithm Two-phase pipeline (Goodman 1999; Nederhof 2003) - wRTG-LM parse (𝑏) = ∑ ⊕ 𝑒∈ AST (𝐻,𝑏) wt (𝑒) Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 16 / 21

Weighted parsing algorithm Two-phase pipeline (Goodman 1999; Nederhof 2003) - wRTG-LM parse (𝑏) = ∑ ⊕ 𝑒∈ AST (𝐻,𝑏) wt (𝑒) Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 16 / 21 ((𝐻, ℒ ), 𝕃 , wt ) - 𝑏 ∈ ℒ

Weighted parsing algorithm Two-phase pipeline (Goodman 1999; Nederhof 2003) - wRTG-LM parse (𝑏) = ∑ ⊕ 𝑒∈ AST (𝐻,𝑏) wt (𝑒) canonical weighted deduction system Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 2019-09-25 16 / 21 ((𝐻, ℒ ), 𝕃 , wt ) - 𝑏 ∈ ℒ

Weighted parsing algorithm Two-phase pipeline (Goodman 1999; Nederhof 2003) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) wRTG-LM system deduction weighted canonical wt (𝑒) 𝑒∈ AST (𝐻,𝑏) parse (𝑏) = ∑ ⊕ - wRTG-LM 16 / 21 ((𝐻, ℒ ), 𝕃 , wt ) ((𝐻 ′ , 𝒟ℱ 𝒣 ∅ ), 𝕃 , wt ′ ) - 𝑏 ∈ ℒ

Weighted parsing algorithm Two-phase pipeline (Goodman 1999; Nederhof 2003) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) ? = wt ′ (𝑒) 𝑒∈ AST (𝐻 ′ ) ∑ ⊕ wRTG-LM system deduction weighted canonical wt (𝑒) 𝑒∈ AST (𝐻,𝑏) parse (𝑏) = ∑ ⊕ - wRTG-LM 16 / 21 ((𝐻, ℒ ), 𝕃 , wt ) ((𝐻 ′ , 𝒟ℱ 𝒣 ∅ ), 𝕃 , wt ′ ) - 𝑏 ∈ ℒ

Weighted parsing algorithm Two-phase pipeline (Goodman 1999; Nederhof 2003) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) algorithm computation value ? = wt ′ (𝑒) 𝑒∈ AST (𝐻 ′ ) ∑ ⊕ 16 / 21 wRTG-LM system deduction weighted canonical wt (𝑒) 𝑒∈ AST (𝐻,𝑏) parse (𝑏) = ∑ ⊕ - wRTG-LM ((𝐻, ℒ ), 𝕃 , wt ) ((𝐻 ′ , 𝒟ℱ 𝒣 ∅ ), 𝕃 , wt ′ ) - 𝑏 ∈ ℒ

Weighted parsing algorithm 𝑒∈ AST (𝐻 ′ ) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) ? = 0 ) 𝑊(𝐵 ′ algorithm computation value ? = wt ′ (𝑒) ∑ ⊕ Two-phase pipeline (Goodman 1999; Nederhof 2003) wRTG-LM system deduction weighted canonical wt (𝑒) 𝑒∈ AST (𝐻,𝑏) parse (𝑏) = ∑ ⊕ - wRTG-LM 16 / 21 ((𝐻, ℒ ), 𝕃 , wt ) ((𝐻 ′ , 𝒟ℱ 𝒣 ∅ ), 𝕃 , wt ′ ) - 𝑏 ∈ ℒ

Weighted parsing algorithm 𝑒∈ AST (𝐻 ′ ) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) weighted parsing algorithm ? = 0 ) 𝑊(𝐵 ′ algorithm computation value ? = wt ′ (𝑒) ∑ ⊕ Two-phase pipeline (Goodman 1999; Nederhof 2003) wRTG-LM system deduction weighted canonical wt (𝑒) 𝑒∈ AST (𝐻,𝑏) parse (𝑏) = ∑ ⊕ - wRTG-LM 16 / 21 ((𝐻, ℒ ), 𝕃 , wt ) ((𝐻 ′ , 𝒟ℱ 𝒣 ∅ ), 𝕃 , wt ′ ) - 𝑏 ∈ ℒ

Canonical weighted deduction system Weight preserving 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) wt (𝑒) = wt ′ (𝜔(𝑒)) for every 𝑒 ∈ AST (𝐻, 𝑏) 2 Bijection 𝜔: AST (𝐻, 𝑏) → AST (𝐻 ′ ) 1 𝐵 → 𝜏(𝐵 1 , … , 𝐵 𝑛 ) is a rule { [𝐵 , 𝑏 0 ] Parsing as deduction (Shieber, Schabes, and Pereira 1995) cwds 17 / 21 - wRTG-LM ((𝐻, ℒ ), 𝕃 , wt ) wRTG-LM ((𝐻 ′ , 𝒟ℱ 𝒣 ∅ ), 𝕃 , wt ′ ) - 𝑏 ∈ ℒ [𝐵 1 , 𝑏 1 ] … [𝐵 𝑛 , 𝑏 𝑛 ] 𝑏 0 , 𝑏 1 , … , 𝑏 𝑛 ∈ factors (𝑏) 𝑏 0 = 𝜏(𝑏 1 , … , 𝑏 𝑛 )

Canonical weighted deduction system Weight preserving 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) wt (𝑒) = wt ′ (𝜔(𝑒)) for every 𝑒 ∈ AST (𝐻, 𝑏) 2 Bijection 𝜔: AST (𝐻, 𝑏) → AST (𝐻 ′ ) 1 17 / 21 { [𝐵 , 𝑏 0 ] Parsing as deduction (Shieber, Schabes, and Pereira 1995) cwds - wRTG-LM ((𝐻, ℒ ), 𝕃 , wt ) wRTG-LM ((𝐻 ′ , 𝒟ℱ 𝒣 ∅ ), 𝕃 , wt ′ ) - 𝑏 ∈ ℒ 𝐵 → 𝜏(𝐵 1 , … , 𝐵 𝑛 ) is a rule [𝐵 1 , 𝑏 1 ] … [𝐵 𝑛 , 𝑏 𝑛 ] 𝑏 0 , 𝑏 1 , … , 𝑏 𝑛 ∈ factors (𝑏) 𝑏 0 = 𝜏(𝑏 1 , … , 𝑏 𝑛 )

Canonical weighted deduction system Weight preserving 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) wt (𝑒) = wt ′ (𝜔(𝑒)) for every 𝑒 ∈ AST (𝐻, 𝑏) 2 Bijection 𝜔: AST (𝐻, 𝑏) → AST (𝐻 ′ ) 1 𝐵 → 𝜏(𝐵 1 , … , 𝐵 𝑛 ) is a rule { [𝐵 , 𝑏 0 ] Parsing as deduction (Shieber, Schabes, and Pereira 1995) cwds 17 / 21 - wRTG-LM ((𝐻, ℒ ), 𝕃 , wt ) wRTG-LM ((𝐻 ′ , 𝒟ℱ 𝒣 ∅ ), 𝕃 , wt ′ ) - 𝑏 ∈ ℒ [𝐵 1 , 𝑏 1 ] … [𝐵 𝑛 , 𝑏 𝑛 ] 𝑏 0 , 𝑏 1 , … , 𝑏 𝑛 ∈ factors (𝑏) 𝑏 0 = 𝜏(𝑏 1 , … , 𝑏 𝑛 )

Weighted parsing algorithm 𝑒∈ AST (𝐻 ′ ) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) weighted parsing algorithm ? = 0 ) 𝑊(𝐵 ′ algorithm computation value ? = wt ′ (𝑒) ∑ ⊕ Two-phase pipeline (Goodman 1999; Nederhof 2003) wRTG-LM system deduction weighted canonical wt (𝑒) 𝑒∈ AST (𝐻,𝑏) parse (𝑏) = ∑ ⊕ - wRTG-LM 18 / 21 ((𝐻, ℒ ), 𝕃 , wt ) ((𝐻 ′ , 𝒟ℱ 𝒣 ∅ ), 𝕃 , wt ′ ) - 𝑏 ∈ ℒ

Value computation algorithm 5: 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 12: until changed = false 𝑊(𝐵) ← 𝑊 new 11: changed ← true 10: 9: 8: 7: 6: changed ← false 𝑊(𝐵) ← 𝟙 0 , 𝑆 ′ ) Output: 𝑊(𝐵 ′ 0 ) 2: 19 / 21 3: repeat 4: Input: a wRTG-LM ((𝐻 ′ , 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , ⊕ , 𝟙 , 𝛻), wt ′ ) with 𝐻 ′ = (𝑂 ′ , 𝛵 ′ , 𝐵 ′ Variables: 𝑊: 𝑂 ′ → 𝕃 , 𝑊 new ∈ 𝕃 , changed ∈ 𝔺 1: for each 𝐵 ∈ 𝑂 ′ do for each 𝐵 ∈ 𝑂 ′ do 𝑊 new ← 𝟙 for each 𝑠 = (𝐵 → ⟨𝑦 1 … 𝑦 𝑛 ⟩(𝐵 1 , … , 𝐵 𝑛 )) in 𝑆 ′ do 𝑊 new ← 𝑊 new ⊕ wt ′ (𝑠)(𝑊(𝐵 1 ), … , 𝑊(𝐵 𝑛 )) if 𝑊(𝐵) ≠ 𝑊 new then

⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ … 𝜀 𝟙 𝟙 ( B A S 𝜏 𝛾 𝛿 S 𝛽 B A → 𝛾 − − − 𝟙 Value computation algorithm (example) 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 𝜕 3 () 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 − 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () − − − → 𝛿( A ) − − − − − − 𝜕 2 S − → 𝜀( B ) − − − − − − 𝜕 1 S − A 𝜕 3 − 𝜕 5 B → 𝜏( A , S ) − − − − − − 𝜕 4 B → 𝛽 − − − − − − − 20 / 21 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt )

⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ … 𝜏 𝟙 𝟙 𝟙 ( B A S S 𝜀 𝛿 𝛾 𝛽 B A → 𝛾 − − ) Value computation algorithm (example) 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 𝜕 3 () 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 − 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () − − − S − − − − − − − 𝜕 2 → 𝛿( A ) A − − − − − − − 𝜕 1 S → 𝜀( B ) 𝜕 3 − − 𝜕 5 B → 𝜏( A , S ) − − − − − − − 𝜕 4 B → 𝛽 − − − − − − 20 / 21 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt )

⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) = ( ) ↦ ( ) = ( ) ↦ … 𝜀 𝟙 𝟙 ( B A S 𝜏 𝛾 𝛿 S 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) 𝛽 B A → 𝛾 − − − 𝟙 Value computation algorithm (example) 𝜕 3 () 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 − ) − − − S − − − − − − − 𝜕 2 → 𝛿( A ) A − − − − − − − 𝜕 1 S → 𝜀( B ) 𝜕 3 20 / 21 − 𝜕 5 B → 𝜏( A , S ) − − − − − − − 𝜕 4 B → 𝛽 − − − − − − ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ) ↦ (

⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ↦ ( ) = ( ) ↦ … Value computation algorithm (example) 𝛿 𝟙 ( B A S 𝜏 𝜀 𝛾 S 𝟙 𝛽 B A → 𝛾 − − − 𝟙 𝕝 1 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 𝜕 3 () 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) ) 𝕝 3 𝕝 2 − 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () − − − S − − − − − − − 𝜕 2 → 𝛿( A ) 𝜕 3 − − − − − − − 𝜕 1 S → 𝜀( B ) A 20 / 21 − 𝜕 5 B → 𝜏( A , S ) − − − − − − 𝜕 4 B → 𝛽 − − − − − − − ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ) ↦ ( ) = (

⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) = ( ) ↦ … Value computation algorithm (example) S 𝟙 ( B A S 𝜏 𝜀 𝛿 𝛽 𝛾 𝟙 B A → 𝛾 − − − − 𝟙 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) ) 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 1 ) ⊕ 𝜕 5 () 𝜕 3 () 2 , 𝕝 ′ 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 − − − 20 / 21 S − − − − − − − 𝜕 2 → 𝛿( A ) A − − − − − − − 𝜕 1 S 𝜕 5 → 𝜀( B ) B → 𝛽 − − − − − 𝜕 4 𝜕 3 − − − − → 𝜏( A , S ) − − − B − − ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ) ↦ ( ) = ( ) ↦ (

⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) Value computation algorithm (example) S 𝟙 ( B A S 𝜏 𝜀 𝛿 𝛽 𝛾 𝟙 B A → 𝛾 − − − − 𝟙 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) 1 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) ↦ … ) 3 𝕝 ′ 2 𝕝 ′ 𝕝 ′ 𝜕 3 () 1 ) ⊕ 𝜕 5 () 2 , 𝕝 ′ 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 − − − 20 / 21 → 𝛿( A ) − − − − − − 𝜕 2 S − 𝜕 5 − − − − − − 𝜕 1 S − → 𝜀( B ) A 𝜕 4 B → 𝜏( A , S ) − − − − − − − B → 𝛽 − − − − − − − 𝜕 3 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ) ↦ ( ) = ( ) ↦ ( ) = (

⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) Value computation algorithm (example) 𝛾 ( B A S 𝜏 𝜀 𝛿 S B 𝛽 𝟙 A → 𝛾 − − − − − 𝟙 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) 𝟙 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 𝜕 5 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 𝜕 3 () − − B → 𝛿( A ) − − − − − − 𝜕 2 S − → 𝜀( B ) − − − − − − 𝜕 1 S − 20 / 21 A 𝜕 3 → 𝜏( A , S ) − − − − − − − 𝜕 4 B → 𝛽 − − − − − − − ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) 𝛽 B A S 𝜏 𝜀 𝛿 S 𝛾 B 𝟙 A → 𝛾 − − − − − − ( 𝟙 B 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 𝟙 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 𝜕 3 () 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) 𝜕 5 − → 𝜏( A , S ) → 𝛿( A ) − − − − − − 𝜕 2 S − − − − − − − − 𝜕 1 S − 20 / 21 → 𝜀( B ) → 𝛽 − − − − − − A B 𝜕 4 − − − − − − − 𝜕 3 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) 𝛽 B A S 𝜏 𝜀 𝛿 S 𝛾 B 𝟙 A → 𝛾 − − − − − − ( 𝟙 B 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 𝟙 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 𝜕 3 () 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) 𝜕 5 − → 𝜏( A , S ) → 𝛿( A ) − − − − − − 𝜕 2 S − − − − − − − − 0.8⋅ 𝕝 1 S − 20 / 21 → 𝜀( B ) → 𝛽 − − − − − − A B 𝜕 4 − − − − − − − 𝜕 3 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) 𝛽 B A S 𝜏 𝜀 𝛿 S 𝛾 B 𝟙 A → 𝛾 − − − − − − ( 𝟙 B 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 𝟙 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 𝜕 3 () 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) 𝜕 5 − → 𝜏( A , S ) → 𝛿( A ) − − − − − − 0.1⋅ 𝕝 1 S − − − − − − − − 0.8⋅ 𝕝 1 S − 20 / 21 → 𝜀( B ) → 𝛽 − − − − − − A B 𝜕 4 − − − − − − − 𝜕 3 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) 𝛽 B A S 𝜏 𝜀 𝛿 S 𝛾 B 𝟙 A → 𝛾 − − − − − − ( 𝟙 B 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 𝟙 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 𝜕 3 () 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) 𝜕 5 − → 𝜏( A , S ) → 𝛿( A ) − − − − − − 0.1⋅ 𝕝 1 S − − − − − − − − 0.8⋅ 𝕝 1 S − 20 / 21 → 𝜀( B ) → 𝛽 − − − − − − A B 𝜕 4 − − − − − − − 0.5 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) 𝛽 B A S 𝜏 𝜀 𝛿 S 𝛾 B 𝟙 A → 𝛾 − − − − − − ( 𝟙 B 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 𝟙 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 𝜕 3 () 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) 0.1 − → 𝜏( A , S ) → 𝛿( A ) − − − − − − 0.1⋅ 𝕝 1 S − − − − − − − − 0.8⋅ 𝕝 1 S − 20 / 21 → 𝜀( B ) → 𝛽 − − − − − − A B 0.7⋅ 𝕝 1 ⋅ 𝕝 2 − − − − − − − 0.5 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) 𝛽 B A S 𝜏 𝜀 𝛿 S 𝛾 B 0 A → 𝛾 − − − − − − ( 0 B 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 0 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 𝜕 3 () 𝜕 1 ( 𝟙 ) ⊕ 𝜕 2 ( 𝟙 ) 0.1 − → 𝜏( A , S ) → 𝛿( A ) − − − − − − 0.1⋅ 𝕝 1 S − − − − − − − − 0.8⋅ 𝕝 1 S − 20 / 21 → 𝜀( B ) → 𝛽 − − − − − − A B 0.7⋅ 𝕝 1 ⋅ 𝕝 2 − − − − − − − 0.5 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) 𝛽 B A S 𝜏 𝜀 𝛿 S 𝛾 B 0 A → 𝛾 − − − − − − ( 0 B 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 0 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 𝜕 3 () 0.8 ⋅ 0 max 0.1 ⋅ 0 0.1 − → 𝜏( A , S ) → 𝛿( A ) − − − − − − 0.1⋅ 𝕝 1 S − − − − − − − − 0.8⋅ 𝕝 1 S − 20 / 21 → 𝜀( B ) → 𝛽 − − − − − − A B 0.7⋅ 𝕝 1 ⋅ 𝕝 2 − − − − − − − 0.5 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) 𝛽 B A S 𝜏 𝜀 𝛿 S 𝛾 B 0 A → 𝛾 − − − − − − ( 0 B 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 0 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 𝜕 3 () 0 0.1 − → 𝜏( A , S ) → 𝛿( A ) − − − − − − 0.1⋅ 𝕝 1 S − − − − − − − − 0.8⋅ 𝕝 1 S − 20 / 21 → 𝜀( B ) → 𝛽 − − − − − − A B 0.7⋅ 𝕝 1 ⋅ 𝕝 2 − − − − − − − 0.5 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) 𝛽 B A S 𝜏 𝜀 𝛿 S 𝛾 B 0 A → 𝛾 − − − − − − ( 0 B 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 0 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 𝜕 4 ( 𝕝 2 , 𝕝 1 ) ⊕ 𝜕 5 () 0.5 0 0.1 − → 𝜏( A , S ) → 𝛿( A ) − − − − − − 0.1⋅ 𝕝 1 S − − − − − − − − 0.8⋅ 𝕝 1 S − 20 / 21 → 𝜀( B ) → 𝛽 − − − − − − A B 0.7⋅ 𝕝 1 ⋅ 𝕝 2 − − − − − − − 0.5 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) 𝛽 B A S 𝜏 𝜀 𝛿 S 𝛾 B 0 A → 𝛾 − − − − − − ( 0 B 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 0 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 0.7 ⋅ 0.5 ⋅ 0 max 0.1 0.5 0 0.1 − → 𝜏( A , S ) → 𝛿( A ) − − − − − − 0.1⋅ 𝕝 1 S − − − − − − − − 0.8⋅ 𝕝 1 S − 20 / 21 → 𝜀( B ) → 𝛽 − − − − − − A B 0.7⋅ 𝕝 1 ⋅ 𝕝 2 − − − − − − − 0.5 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) 𝛽 B A S 𝜏 𝜀 𝛿 S 𝛾 B 0 A → 𝛾 − − − − − − ( 0 B 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 0 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 𝕝 3 𝕝 2 𝕝 1 0.1 0.5 0 0.1 − → 𝜏( A , S ) → 𝛿( A ) − − − − − − 0.1⋅ 𝕝 1 S − − − − − − − − 0.8⋅ 𝕝 1 S − 20 / 21 → 𝜀( B ) → 𝛽 − − − − − − A B 0.7⋅ 𝕝 1 ⋅ 𝕝 2 − − − − − − − 0.5 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) = ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) B S 𝜏 𝜀 𝛿 S 𝛾 𝛽 A B → 𝛾 − − − − − − − A ( → 𝜏( A , S ) 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 0 𝜕 4 ( 𝕝 ′ 𝜕 3 () 𝜕 1 ( 𝕝 2 ) ⊕ 𝜕 2 ( 𝕝 3 ) 0.1 0.5 0 0 0 0.1 B − → 𝛿( A ) − − − − − 0.1⋅ 𝕝 1 S − − − − − − − − 0.8⋅ 𝕝 1 S − − → 𝜀( B ) − − − − − − 0.7⋅ 𝕝 1 ⋅ 𝕝 2 A → 𝛽 B − − − − − − 0.5 20 / 21 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) ↦ ( ) = ( ) ↦ …

Value computation algorithm (example) B S 𝜏 𝜀 𝛿 S 𝛾 𝛽 A B → 𝛾 − − − − − − − A ( → 𝜏( A , S ) 1 ) ⊕ 𝜕 5 () 2019-09-25 Richard Mörbitz, Heiko Vogler: Weighted parsing for grammar-based language models (FSMNLP 2019) 3 𝕝 ′ 2 𝕝 ′ 1 𝕝 ′ 2 , 𝕝 ′ 0 𝜕 4 ( 𝕝 ′ 𝜕 3 () 0.8 ⋅ 0.5 max 0.1 ⋅ 0.1 0.1 0.5 0 0 0 0.1 B − → 𝛿( A ) − − − − − 0.1⋅ 𝕝 1 S − − − − − − − − 0.8⋅ 𝕝 1 S − − → 𝜀( B ) − − − − − − 0.7⋅ 𝕝 1 ⋅ 𝕝 2 A → 𝛽 B − − − − − − 0.5 20 / 21 ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( 𝕃 , 𝟙 , ⊕ , 𝛻), wt ) ⇝ ((𝐻, 𝒟ℱ 𝒣 ∅ ), ( ℝ 1 0 , 0, max , 𝛻 mul ), wt ) ) ↦ ( ) ↦ ( ) = ( ) ↦ …

Weighted parsing for grammar-based language models FSMNLP 2019 - PowerPoint PPT Presentation

Weighted parsing for grammar-based language models FSMNLP 2019 Richard Mrbitz Heiko Vogler 2019-09-25 The weighted parsing problem language e.g., English sentences ( ) syntactic object e.g., Fruit fmies like bananas ?

Introduction to Bottom-Up Parsing Shift-reduce parsing The LR parsing algorithm

Working Together What does his future hold? Carres Grammar School Carres Grammar School

CSC 4181 Compiler Construction Parsing 1 1 Outline Top-down v.s. Bottom-up Top-down parsing

Grammar and word order Grammar and word order Grammar Grammar Includes morphology and syntax

Weighted graphs Weighted graphs Weighted graphs Weighted graphs Graphs with numbers, called

Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs

General Context-Free Grammar Parsing: Application of grammar rewrite rules A phrase structure

General Context-Free Grammar Parsing Application of grammar rewrite rules A phrase structure

Basic Parsing Algorithms Chart Parsing Seminar Recent Advances in Parsing Technology WS

Graph-Based Parsing Joakim Nivre Uppsala University Department of Linguistics and Philology

Parsing beyond context-free grammar: necessarily adjacent. Range Concatenation Grammar Parsing

1 Determinism and Parsing The parsing problem is, given a string w and a context-free grammar G ,

Parsing with PCFGs Joakim Nivre Uppsala University Department of Linguistics and Philology

Objectives LL Parsing The topic for this lecture is a kind of grammar that works well with

Natural Language Processing Lecture 13: More on CFG Parsing Probabilistjc/Weighted Parsing

Introduction to English Linguistics 4: Grammar and Syntax I Grammar and Syntax Grammar The

INTERNATIONAL SEM INAR ON HORTICULTURE TO SUPPORT FOOD SECURITY 2010 SCHEDULE of PARALLEL SESSION

The Digital Revolution: The move from Teaching to Learning Revolution Compare Education Values

PHP Angel R. Roman Keyla Perez Velez Michael Voss Julian Richen Group 12 Questions to be

Evergreen Inventory From Start to Finish Introduction Presenters CUTPL Information

PostGIS Feature Frenzy!!! Wednesday, September 16, 15 frenzy because state of postgis

HANNAH HOWARD #ABOUTME Personal hannah@carbon Anecdote: I have @techgirlwonder ve.com a

GMO and its impact on our modern society By Jaeseung Shin, Jevon Suharnoko What is GMO?

2016 MERGER BY INCORPORATION of Centrale del Latte di Firenze, Pistoia e Livorno S,p,A, into

Sambuz

Useful Links

Newsletter

Mail Us