Reified Context Models Jacob Steinhardt Percy Liang Stanford University { jsteinhardt,pliang } @cs.stanford.edu July 8, 2015 J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 1 / 11
Structured Prediction Task input x : output y : v o l c a n i c J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 2 / 11
Contexts Are Key v o c a l J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 3 / 11
Contexts Are Key v o c a l v *o ***c DP: **l J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 3 / 11
Contexts Are Key v o c a l v *o ***c DP: **l v o c a l v vo beam search: vol volc J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 3 / 11
Contexts Are Key v o c a l v *o ***c DP: **l v o c a l v vo beam search: vol volc Key idea: contexts! ao bo *o def = co . . . J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 3 / 11
Desiderata r *o **l ***c v *a **i ***r coverage (short contexts) better uncertainty estimates (precision) stabler partially supervised learning updates J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 4 / 11
Desiderata r *o **l ***c v *a **i ***r coverage (short contexts) better uncertainty estimates (precision) stabler partially supervised learning updates J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 4 / 11
Desiderata r *o **l ***c v *a **i ***r coverage (short contexts) better uncertainty estimates (precision) stabler partially supervised learning updates J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 4 / 11
Desiderata r *o **l ***c v *a **i ***r coverage (short contexts) better uncertainty estimates (precision) stabler partially supervised learning updates r ro rol rolc v ra ral ralc expressivity (long contexts) capture complex dependencies J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 4 / 11
Desiderata r *o **l ***c v *a **i ***r coverage (short contexts) better uncertainty estimates (precision) stabler partially supervised learning updates r ro rol rolc v ra ral ralc expressivity (long contexts) capture complex dependencies J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 4 / 11
Desiderata r *o **l ***c v *a **i ***r coverage (short contexts) better uncertainty estimates (precision) stabler partially supervised learning updates r ro rol rolc v ra ral ralc expressivity (long contexts) capture complex dependencies J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 4 / 11
Desiderata r *o **l ***c v *a **i ***r coverage (short contexts) better uncertainty estimates (precision) stabler partially supervised learning updates r ro rol rolc v ra ral ralc expressivity (long contexts) capture complex dependencies r ro rol *olc v ra ral ***c ← best of both worlds y *o *ol ***r * ** *** **** J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 4 / 11
Reifying Contexts input x : J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 5 / 11
Reifying Contexts input x : output y : v o l c a n i c J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 5 / 11
Reifying Contexts input x : output y : v o l c a n i c context c : v *o *ol *olc ······ J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 5 / 11
Reifying Contexts input x : output y : v o l c a n i c context c : v *o *ol *olc ······ r ro rol *olc v ra ral ***c y *o *ol ***r * ** *** **** J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 5 / 11
Reifying Contexts input x : output y : v o l c a n i c context c : v *o *ol *olc ······ r ro rol *olc v ra ral ***c ← “context sets” y *o *ol ***r * ** *** **** C 1 C 2 C 3 C 4 J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 5 / 11
Reifying Contexts input x : output y : v o l c a n i c context c : v *o *ol *olc ······ r ro rol *olc v ra ral ***c ← “context sets” y *o *ol ***r * ** *** **** C 1 C 2 C 3 C 4 Challenge: how to trade off contexts of different lengths? J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 5 / 11
Reifying Contexts input x : output y : v o l c a n i c context c : v *o *ol *olc ······ r ro rol *olc v ra ral ***c ← “context sets” y *o *ol ***r * ** *** **** C 1 C 2 C 3 C 4 Challenge: how to trade off contexts of different lengths? = ⇒ Reify contexts as part of model! J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 5 / 11
Reified Context Models Given: context sets C 1 ,..., C L J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 6 / 11
Reified Context Models Given: context sets C 1 ,..., C L features φ i ( c i − 1 , y i ) J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 6 / 11
Reified Context Models Given: context sets C 1 ,..., C L features φ i ( c i − 1 , y i ) Define the model � � L θ ⊤ φ i ( c i − 1 , y i ) ∑ p θ ( y 1 : L , c 1 : L − 1 ) ∝ exp · κ ( y , c ) � �� � i = 1 consistency J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 6 / 11
Reified Context Models Given: context sets C 1 ,..., C L features φ i ( c i − 1 , y i ) Define the model � � L θ ⊤ φ i ( c i − 1 , y i ) ∑ p θ ( y 1 : L , c 1 : L − 1 ) ∝ exp · κ ( y , c ) � �� � i = 1 consistency Graphical model structure: C 1 C 2 C 3 C 4 κ κ κ κ κ Y 1 Y 2 Y 3 Y 4 Y 5 J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 6 / 11
Reified Context Models Given: context sets C 1 ,..., C L features φ i ( c i − 1 , y i ) Define the model � � L θ ⊤ φ i ( c i − 1 , y i ) ∑ p θ ( y 1 : L , c 1 : L − 1 ) ∝ exp · κ ( y , c ) � �� � i = 1 consistency Graphical model structure: C 1 C 2 C 3 C 4 φ 1 φ 2 φ 3 φ 4 φ 5 Y 1 Y 2 Y 3 Y 4 Y 5 J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 6 / 11
Reified Context Models Given: context sets C 1 ,..., C L features φ i ( c i − 1 , y i ) Define the model � � L θ ⊤ φ i ( c i − 1 , y i ) ∑ p θ ( y 1 : L , c 1 : L − 1 ) ∝ exp · κ ( y , c ) � �� � i = 1 consistency Graphical model structure: C 1 C 2 C 3 C 4 inference via φ 1 φ 2 φ 3 φ 4 φ 5 forward-backward! Y 1 Y 2 Y 3 Y 4 Y 5 J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 6 / 11
Adaptive Context Selection Select context sets C i during forward pass of inference J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 7 / 11
Adaptive Context Selection Select context sets C i during forward pass of inference Greedily select contexts with largest mass J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 7 / 11
Adaptive Context Selection Select context sets C i during forward pass of inference Greedily select contexts with largest mass a b c d e . . . J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 7 / 11
Adaptive Context Selection Select context sets C i during forward pass of inference Greedily select contexts with largest mass a b c c d e e . . . J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 7 / 11
Adaptive Context Selection Select context sets C i during forward pass of inference Greedily select contexts with largest mass a c b c c e d ⋆ e e . . . C 1 J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 7 / 11
Adaptive Context Selection Select context sets C i during forward pass of inference Greedily select contexts with largest mass ca a cb . . c b . c c e ea d eb ⋆ . . e e . . . ⋆ a . . . . C 1 J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 7 / 11
Adaptive Context Selection Select context sets C i during forward pass of inference Greedily select contexts with largest mass ca ca a cb . . c b . c c e ea d eb ⋆ . . e e . . . ⋆ a ⋆ a . . . . C 1 J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 7 / 11
Adaptive Context Selection Select context sets C i during forward pass of inference Greedily select contexts with largest mass ca ca a cb . . c ca b . c c e ea ⋆ a d eb ⋆ ⋆⋆ . . e e . . . ⋆ a ⋆ a . . . . C 1 C 2 J. Steinhardt & P. Liang (Stanford) Reified Context Models July 8, 2015 7 / 11
Recommend
More recommend