Lagrangian Based Approaches for Lexicalized Tree Adjoining Grammar - PowerPoint PPT Presentation

Lexicalized Tree Adjoining Grammar (LTAG) Motivations • Mildly context-sensitive formalism • Linguistically plausible • Semantics Elementary tree Extended part-of-speech tags with structural constraints e.g. A verb with a subject on its left-side S Constituents NP ↓ Substitution site VP VB Part-of-speech tag walks Lexical leaf 1. Lexicalized Tree Adjoining Grammar Parsing 10 / 51

Example VB walks (verb) 1. Lexicalized Tree Adjoining Grammar Parsing 11 / 51

Example S S (sbj) NP ↓ (sbj) NP ↓ VP VP NP ↓ (obj) VB VB VB walks walks walks (verb) (intransitive verb) (transitive verb) 1. Lexicalized Tree Adjoining Grammar Parsing 11 / 51

Elementary tree combination S S NP NP ↓ VP ⇒ NP VP PRP VB PRP VB She walks She walks Substitution 1. Lexicalized Tree Adjoining Grammar Parsing 12 / 51

Elementary tree combination S S NP NP ↓ VP ⇒ NP VP PRP VB PRP VB She walks She walks Substitution S NP ↓ S VP NP ↓ ⇒ VP VP VP VP ∗ RB VB RB VB deliberately walks deliberately walks Adjunction 1. Lexicalized Tree Adjoining Grammar Parsing 12 / 51

LTAG derivation tree deliberately dog She walks the Bottom-up construction of the syntactic phrase structure 1. Lexicalized Tree Adjoining Grammar Parsing 13 / 51

LTAG derivation tree S VP NP ↓ NP NP NP VP VP ∗ RB NP ↓ NP ∗ NN PRP VBZ DET deliberately dog She walks the Bottom-up construction of the syntactic phrase structure 1. Lexicalized Tree Adjoining Grammar Parsing 13 / 51

LTAG derivation tree S VP NP ↓ NP NP NP VP VP ∗ RB NP ↓ NP ∗ NN PRP VBZ DET deliberately dog She walks the Bottom-up construction of the syntactic phrase structure 1. Lexicalized Tree Adjoining Grammar Parsing 13 / 51

LTAG derivation tree S VP NP ↓ NP NP NP VP VP ∗ RB NP ↓ NP ∗ NN PRP VBZ DET deliberately dog She walks the Bottom-up construction of the syntactic phrase structure v 3 v 1 v 5 v 2 v 4 deliberately dog She walks the Representation alternative as a derivation tree [Rambow et al. 1997] 1. Lexicalized Tree Adjoining Grammar Parsing 13 / 51

LTAG derivation tree S VP NP ↓ NP NP NP VP VP ∗ RB NP ↓ NP ∗ NN PRP VBZ DET deliberately dog She walks the Bottom-up construction of the syntactic phrase structure v 3 τ 3 v 1 v 5 τ 1 τ 5 v 2 v 4 τ 2 τ 4 deliberately dog She walks the Representation alternative as a derivation tree [Rambow et al. 1997] 1. Lexicalized Tree Adjoining Grammar Parsing 13 / 51

LTAG derivation tree S 1 . 1 1 . 2 1 VP NP ↓ NP NP NP VP 1 . 2 . 2 VP ∗ RB NP ↓ NP ∗ NN PRP VBZ DET deliberately dog She walks the Bottom-up construction of the syntactic phrase structure v 3 1 . 1 1 . 2 . 2 τ 3 1 . 2 v 1 v 5 1 τ 1 τ 5 v 2 v 4 τ 2 τ 4 deliberately dog She walks the Representation alternative as a derivation tree [Rambow et al. 1997] 1. Lexicalized Tree Adjoining Grammar Parsing 13 / 51

Weighted LTAG parsing Weights • Tag weights (elementary tree assignation) • Dependency weights (combination operations) Parsing goal • Compute the syntactic structure of maximum weight 1. Lexicalized Tree Adjoining Grammar Parsing 14 / 51

Weighted LTAG parsing Weights • Tag weights (elementary tree assignation) • Dependency weights (combination operations) Parsing goal • Compute the syntactic structure of maximum weight Complexity [Eisner et al. 2000] O ( n 6 max( n , g ) gt ) : n : sentence length t : maximum tree size g : maximum ambiguity ⇒ O ( n 7 ) asymptotically w.r.t. the sentence length 1. Lexicalized Tree Adjoining Grammar Parsing 14 / 51

2. Efficient parsing with Lagrangian relaxation

Integer Linear Programming Integer Linear Program (ILP) y ⊤ w max (maximize the weight of the structure y ) y s.t. Ay − b ≤ 0 (constraints on the structure) 2. Efficient parsing with Lagrangian relaxation 15 / 51

Integer Linear Programming Integer Linear Program (ILP) y ⊤ w max (maximize the weight of the structure y ) y s.t. Ay − b ≤ 0 (easy constraints) By − c ≤ 0 (difficult constraints) 2. Efficient parsing with Lagrangian relaxation 15 / 51

Integer Linear Programming Integer Linear Program (ILP) y ⊤ w max (maximize the weight of the structure y ) y s.t. Ay − b ≤ 0 (easy constraints) By − c ≤ 0 (difficult constraints) Intuition • Remove difficult constraints • Introduce them as penalties in the objective • Solve the new reparametrized problem iteratively 2. Efficient parsing with Lagrangian relaxation 15 / 51

Example: dependency parsing dog She walks the 2. Efficient parsing with Lagrangian relaxation 16 / 51

Example: dependency parsing v 0 v 1 v 2 v 3 v 4 dog * She walks the 2. Efficient parsing with Lagrangian relaxation 16 / 51

Example: dependency parsing v 0 v 1 v 2 v 3 v 4 dog * She walks the Reduction Dependency tree ⇔ v 0 -rooted spanning arborescence i.e. a connected graph such that: • v 0 : no incoming arc • v 1 . . . v 4 : exactly one incoming arc • Acyclic 2. Efficient parsing with Lagrangian relaxation 16 / 51

Example: dependency parsing w 0 , 1 v 0 v 1 v 2 v 3 v 4 dog * She walks the Graph construction 1. Add arc candidates 2. Add arc weights 2. Efficient parsing with Lagrangian relaxation 16 / 51

Example: dependency parsing w 3 , 1 w 0 , 1 v 0 v 1 v 2 v 3 v 4 dog * She walks the Graph construction 1. Add arc candidates 2. Add arc weights 2. Efficient parsing with Lagrangian relaxation 16 / 51

Example: dependency parsing v 0 v 1 v 2 v 3 v 4 dog * She walks the Graph construction 1. Add arc candidates 2. Add arc weights 2. Efficient parsing with Lagrangian relaxation 16 / 51

Example: dependency parsing v 0 v 1 v 2 v 3 v 4 dog * She walks the Graph construction 1. Add arc candidates 2. Add arc weights 3. Compute the spanning arborescence of maximum weight 2. Efficient parsing with Lagrangian relaxation 16 / 51

Example: dependency parsing ILP formulation • y : arc incidence vector ( y a = 1 iff arc a is selected) • w : arc weight vector y ⊤ w max (arc-factored model) y 2. Efficient parsing with Lagrangian relaxation 16 / 51

Example: dependency parsing ILP formulation • y : arc incidence vector ( y a = 1 iff arc a is selected) • w : arc weight vector y ⊤ w max (arc-factored model) y � s.t. y a = 0 (root) a ∈ δ in ( v 0 ) 2. Efficient parsing with Lagrangian relaxation 16 / 51

Example: dependency parsing ILP formulation • y : arc incidence vector ( y a = 1 iff arc a is selected) • w : arc weight vector y ⊤ w max (arc-factored model) y � s.t. y a = 0 (root) a ∈ δ in ( v 0 ) � ∀ v ∈ V + y a = 1 (one head/word) a ∈ δ in ( v ) 2. Efficient parsing with Lagrangian relaxation 16 / 51

Example: dependency parsing ILP formulation • y : arc incidence vector ( y a = 1 iff arc a is selected) • w : arc weight vector y ⊤ w max (arc-factored model) y � s.t. y a = 0 (root) a ∈ δ in ( v 0 ) � ∀ v ∈ V + y a = 1 (one head/word) a ∈ δ in ( v ) ∀ W ⊆ V + � y a ≥ 1 (connectedness) a ∈ δ in ( W ) 2. Efficient parsing with Lagrangian relaxation 16 / 51

Example: dependency parsing ILP formulation • y : arc incidence vector ( y a = 1 iff arc a is selected) • w : arc weight vector y ⊤ w max (arc-factored model) y � s.t. y a = 0 (root) a ∈ δ in ( v 0 ) � ∀ v ∈ V + y a = 1 (one head/word) a ∈ δ in ( v ) ∀ W ⊆ V + � y a ≥ 1 (connectedness) a ∈ δ in ( W ) y ∈ { 0 , 1 } A (integrality) 2. Efficient parsing with Lagrangian relaxation 16 / 51

Example: dependency parsing ILP formulation • y : arc incidence vector ( y a = 1 iff arc a is selected) • w : arc weight vector y ⊤ w max y s.t. y ∈ Y Efficient decoding • Generic solver: simplex, interior point method, . . . • Specialized algorithm: Maximum Spanning Arborescence O ( n 2 ) [Edmonds 1967; Schrijver 2003; McDonald et al. 2005] 2. Efficient parsing with Lagrangian relaxation 16 / 51

Lagrangian relaxation Difficult constraints Force each vertex to have at most k outgoing arc 2. Efficient parsing with Lagrangian relaxation 17 / 51

Lagrangian relaxation Difficult constraints Force each vertex to have at most k outgoing arc ILP formulation y ⊤ w max y s.t. y ∈ Y (easy constraints) 2. Efficient parsing with Lagrangian relaxation 17 / 51

Lagrangian relaxation Difficult constraints Force each vertex to have at most k outgoing arc ILP formulation y ⊤ w max y s.t. y ∈ Y (easy constraints) � y a ≤ k ∀ v ∈ V (hard constraints) a ∈ δ + ( v ) 2. Efficient parsing with Lagrangian relaxation 17 / 51

Lagrangian relaxation Difficult constraints Force each vertex to have at most k outgoing arc ILP formulation y ⊤ w max y s.t. y ∈ Y (easy constraints) � y a ≤ k ∀ v ∈ V (hard constraints) a ∈ δ + ( v ) Lagrangian relaxation 1. Relax difficult constraints as penalties in the objective λ ≥ 0: vector of Lagrangian multipliers 2. Efficient parsing with Lagrangian relaxation 17 / 51

Lagrangian relaxation Difficult constraints Force each vertex to have at most k outgoing arc Lagrangian dual y ⊤ w − � � y a − k ) max λ v ( y a ∈ δ + v v ∈ V s.t. y ∈ Y (easy constraints) Lagrangian relaxation 1. Relax difficult constraints as penalties in the objective λ ≥ 0: vector of Lagrangian multipliers ⇒ Upper bound on the original problem 2. Efficient parsing with Lagrangian relaxation 17 / 51

Lagrangian relaxation Difficult constraints Force each vertex to have at most k outgoing arc Lagrangian dual y ⊤ w ′ max (+ constant term w.r.t. y ) y s.t. y ∈ Y (easy constraints) Lagrangian relaxation 1. Relax difficult constraints as penalties in the objective λ ≥ 0: vector of Lagrangian multipliers 2. Rewrite the objective 2. Efficient parsing with Lagrangian relaxation 17 / 51

Lagrangian relaxation Difficult constraints Force each vertex to have at most k outgoing arc Lagrangian dual y ⊤ w ′ min max (+ constant term w.r.t. y ) y λ s.t. y ∈ Y (easy constraints) Lagrangian relaxation 1. Relax difficult constraints as penalties in the objective λ ≥ 0: vector of Lagrangian multipliers 2. Rewrite the objective 3. Minimize over λ 2. Efficient parsing with Lagrangian relaxation 17 / 51

Lagrangian optimization Lagrangian dual y ⊤ w ′ min max (+ constant term w.r.t. y ) y λ s.t. y ∈ Y Optimization • max y : easy (assumption) ⇒ MSA • min λ : subgradient descent ⇒ loop over the maximization 2. Efficient parsing with Lagrangian relaxation 18 / 51

Lagrangian optimization Lagrangian dual y ⊤ w ′ min max (+ constant term w.r.t. y ) y λ s.t. y ∈ Y Optimization • max y : easy (assumption) ⇒ MSA • min λ : subgradient descent ⇒ loop over the maximization Heuristic • Quality certificate • Possible optimality certificate Exhaustive search • Branch-and-bound • Exact pruning 2. Efficient parsing with Lagrangian relaxation 18 / 51

Methodology Methodology 1. Graph characterization of LTAG-derivations 2. ILP formulation of the problem 3. Lagrangian based decoder 2. Efficient parsing with Lagrangian relaxation 19 / 51

Methodology Methodology 1. Graph characterization of LTAG-derivations 2. ILP formulation of the problem 3. Lagrangian based decoder Requirements for the ILP formulation • Formulation as linear inequalities 2. Efficient parsing with Lagrangian relaxation 19 / 51

Methodology Methodology 1. Graph characterization of LTAG-derivations 2. ILP formulation of the problem 3. Lagrangian based decoder Requirements for the ILP formulation • Formulation as linear inequalities Requirements for the Lagrangian based decoder • Relaxation with "nice" objective function • Efficient algorithm that solve the relaxed problem 2. Efficient parsing with Lagrangian relaxation 19 / 51

3. A dependency-like LTAG parser

Proposed approach v 3 v 3 τ 3 1 . 1 1 . 2 . 2 1 . 2 dog ⇒ v 1 v 5 v 1 τ 1 1 v 5 τ 5 1 . v 2 v 4 v 2 τ 2 v 4 τ 4 She deliberately walks the She deliberately walks the dog A dependency-like LTAG parser 1. LTAG compatible dependency parsing [Corro et al. 2016] 2. LTAG derivation tree labeler [Corro et al. 2017b] 3. A dependency-like LTAG parser 20 / 51

Proposed approach v 3 v 3 τ 3 1 . 1 1 . 2 . 2 1 . 2 dog ⇒ v 1 v 5 v 1 τ 1 1 v 5 τ 5 1 . v 2 v 4 v 2 τ 2 v 4 τ 4 She deliberately walks the She deliberately walks the dog A dependency-like LTAG parser 1. LTAG compatible dependency parsing [Corro et al. 2016] 2. LTAG derivation tree labeler [Corro et al. 2017b] Methodology 1. Graph characterization of LTAG-derivations 2. ILP formulation of the problem 3. Lagrangian based decoder 3. A dependency-like LTAG parser 20 / 51

Dependency trees v 3 v 1 v 5 v 2 v 4 She deliberately walks the dog Structural properties of dependency structures Non-projective Projective 3. A dependency-like LTAG parser 21 / 51

Dependency trees v 3 v 1 v 5 v 2 v 4 She deliberately walks the dog Structural properties of dependency structures k-Bounded Block Degree Non-projective Projective Well-nestedness 3. A dependency-like LTAG parser 21 / 51

Dependency trees v 3 v 1 v 5 v 2 v 4 She deliberately walks the dog Structural properties of dependency structures k-Bounded Block Degree Non-projective Projective Well-nestedness LTAG derivation tree [Bodirsky et al. 2009; Kuhlmann 2010] • 2-Bounded Block Degree (2-BBD) • Well-nested (WN) 3. A dependency-like LTAG parser 21 / 51

Yield Yield of a vertex v Set of all vertices reachable from v ⇒ Required in order to defined structural properties v 0 v 2 v 4 v 1 v 3 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 22 / 51

Yield Yield of a vertex v Set of all vertices reachable from v ⇒ Required in order to defined structural properties v 0 v 0 Yield ( v 0 ) = { v 0 , v 1 , v 2 , v 3 , v 4 } v 2 v 2 v 4 v 4 v 1 v 1 v 3 v 3 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 22 / 51

Yield Yield of a vertex v Set of all vertices reachable from v ⇒ Required in order to defined structural properties v 0 Yield ( v 0 ) = { v 0 , v 1 , v 2 , v 3 , v 4 } v 2 Yield ( v 1 ) = { v 1 } v 4 v 1 v 1 v 3 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 22 / 51

Yield Yield of a vertex v Set of all vertices reachable from v ⇒ Required in order to defined structural properties v 0 Yield ( v 0 ) = { v 0 , v 1 , v 2 , v 3 , v 4 } v 2 v 2 Yield ( v 1 ) = { v 1 } v 4 v 4 Yield ( v 2 ) = { v 1 , v 2 , v 3 , v 4 } v 1 v 1 v 3 v 3 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 22 / 51

Yield Yield of a vertex v Set of all vertices reachable from v ⇒ Required in order to defined structural properties v 0 Yield ( v 0 ) = { v 0 , v 1 , v 2 , v 3 , v 4 } v 2 Yield ( v 1 ) = { v 1 } v 4 Yield ( v 2 ) = { v 1 , v 2 , v 3 , v 4 } v 1 Yield ( v 3 ) = { v 3 } v 3 v 3 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 22 / 51

Yield Yield of a vertex v Set of all vertices reachable from v ⇒ Required in order to defined structural properties v 0 Yield ( v 0 ) = { v 0 , v 1 , v 2 , v 3 , v 4 } v 2 Yield ( v 1 ) = { v 1 } v 4 v 4 Yield ( v 2 ) = { v 1 , v 2 , v 3 , v 4 } v 1 v 1 Yield ( v 3 ) = { v 3 } v 3 v 3 Yield ( v 4 ) = { v 3 , v 4 } s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 22 / 51

Contiguous yield Block degree of a vertex Minimum number of intervals needed to describe its yield Contiguous yield Yield which can be defined with a single interval v 0 v 2 v 3 v 1 v 4 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 23 / 51

Contiguous yield Block degree of a vertex Minimum number of intervals needed to describe its yield Contiguous yield Yield which can be defined with a single interval v 0 v 0 Yield ( v 0 ) = [ v 0 . . . v 4 ] BD ( v 0 ) = 1 v 2 v 2 v 3 v 3 v 1 v 1 v 4 v 4 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 23 / 51

Contiguous yield Block degree of a vertex Minimum number of intervals needed to describe its yield Contiguous yield Yield which can be defined with a single interval v 0 Yield ( v 0 ) = [ v 0 . . . v 4 ] BD ( v 0 ) = 1 v 2 Yield ( v 1 ) = [ v 1 ] ∪ [ v 4 ] BD ( v 1 ) = 2 v 3 v 1 v 1 v 4 v 4 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 23 / 51

Contiguous yield Block degree of a vertex Minimum number of intervals needed to describe its yield Contiguous yield Yield which can be defined with a single interval v 0 Yield ( v 0 ) = [ v 0 . . . v 4 ] BD ( v 0 ) = 1 v 2 v 2 Yield ( v 1 ) = [ v 1 ] ∪ [ v 4 ] BD ( v 1 ) = 2 v 3 v 3 Yield ( v 2 ) = [ v 2 . . . v 3 ] BD ( v 2 ) = 1 v 1 v 4 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 23 / 51

Contiguous yield Block degree of a vertex Minimum number of intervals needed to describe its yield Contiguous yield Yield which can be defined with a single interval v 0 Yield ( v 0 ) = [ v 0 . . . v 4 ] BD ( v 0 ) = 1 v 2 Yield ( v 1 ) = [ v 1 ] ∪ [ v 4 ] BD ( v 1 ) = 2 v 3 v 3 Yield ( v 2 ) = [ v 2 . . . v 3 ] BD ( v 2 ) = 1 v 1 Yield ( v 3 ) = [ v 3 ] BD ( v 3 ) = 1 v 4 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 23 / 51

Contiguous yield Block degree of a vertex Minimum number of intervals needed to describe its yield Contiguous yield Yield which can be defined with a single interval v 0 Yield ( v 0 ) = [ v 0 . . . v 4 ] BD ( v 0 ) = 1 v 2 Yield ( v 1 ) = [ v 1 ] ∪ [ v 4 ] BD ( v 1 ) = 2 v 3 Yield ( v 2 ) = [ v 2 . . . v 3 ] BD ( v 2 ) = 1 v 1 Yield ( v 3 ) = [ v 3 ] BD ( v 3 ) = 1 v 4 v 4 Yield ( v 4 ) = [ v 4 ] BD ( v 4 ) = 1 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 23 / 51

Contiguous yield Block degree of a vertex Minimum number of intervals needed to describe its yield Contiguous yield Yield which can be defined with a single interval v 0 Yield ( v 0 ) = [ v 0 . . . v 4 ] BD ( v 0 ) = 1 v 2 Yield ( v 1 ) = [ v 1 ] ∪ [ v 4 ] BD ( v 1 ) = 2 v 3 Yield ( v 2 ) = [ v 2 . . . v 3 ] BD ( v 2 ) = 1 v 1 v 1 Yield ( v 3 ) = [ v 3 ] BD ( v 3 ) = 1 v 4 v 4 Yield ( v 4 ) = [ v 4 ] BD ( v 4 ) = 1 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 23 / 51

Structural properties of dependency trees Projective dependency tree Arborescence with contiguous yields only v 0 v 2 v 1 v 4 v 3 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 24 / 51

Structural properties of dependency trees Projective dependency tree Arborescence with contiguous yields only v 0 v 2 v 1 v 4 v 3 s 0 s 1 s 2 s 3 s 4 Non-projective dependency tree Arborescence with at least one non-contiguous yield v 0 v 2 v 4 v 1 v 3 s 0 s 1 s 2 s 3 s 4 3. A dependency-like LTAG parser 24 / 51

Structural properties (1/2): k-BBD k-Bounded Block Degree (k-BBD) • BD of a tree: the maximal block degree of its vertices • k-BBD tree: tree with a BD less or equal to k v 0 Yield ( v 0 ) = [ v 0 . . . v 4 ] BD ( v 0 ) = 1 v 2 Yield ( v 1 ) = [ v 1 ] ∪ [ v 4 ] BD ( v 1 ) = 2 v 3 Yield ( v 2 ) = [ v 2 . . . v 3 ] BD ( v 2 ) = 1 v 1 v 1 Yield ( v 3 ) = [ v 3 ] BD ( v 3 ) = 1 v 4 v 4 BD ( v 4 ) = 1 Yield ( v 4 ) = [ v 4 ] s 0 s 1 s 2 s 3 s 4 Tree of block degree 2 3. A dependency-like LTAG parser 25 / 51

Structural properties (2/2): WN Well-nestedness (WN) • Interleaving sets I 1 ∩ I 2 = ∅ : ∃ i , j ∈ I 1 and k , l ∈ I 2 such that i < k < j < l • Well-nested tree: does not contain two vertices whose yields interleave ⇒ e.g. a yield cannot be inside and outside a gap v 0 v 0 v 2 v 2 v 3 v 4 v 1 v 1 v 4 v 3 s 0 s 1 s 2 s 3 s 4 s 0 s 1 s 2 s 3 s 4 Well-nested tree Not well-nested tree 3. A dependency-like LTAG parser 26 / 51