Learning(Distribu.ons(over(Logical(Forms(for( - PowerPoint PPT Presentation

Learning(Distribu.ons(over(Logical(Forms(for( Referring(Expression(Genera.on( Nicholas(FitzGerald(((((Yoav(Artzi(((((Luke(ZeClemoyer(

Referring(Expressions( Photo:(pwenzel(on(Flikr(

Referring(Expressions( Photo:(pwenzel(on(Flikr(

Referring(Expressions( “the(red(beans”( Photo:(pwenzel(on(Flikr(

Referring(Expressions( “the(red(beans”( “the(second(jar(from(the( leK(on(the(middle(shelf”( Photo:(pwenzel(on(Flikr(

Referring(Expressions( “the(red(beans”( “the(second(jar(from(the( leK(on(the(middle(shelf”( “the(jar(underneath(the( space(between(the(two( biggest(white(jars”( Photo:(pwenzel(on(Flikr(

Grounded(Language(Problems( Physical(Scene( Language( ``The(green( and(red( balls.’’(

Grounded(Language(Problems( Physical(Scene( Language( Logical(Form((LF)( ``The(green( ι x. ( green ( x ) ∨ red ( x )) and(red( ∧ sphere ( x ) balls.’’(

Grounded(Language(Problems( Physical(Scene( Language( Logical(Form((LF)( ``The(green( ι x. ( green ( x ) ∨ red ( x )) and(red( ∧ sphere ( x ) balls.’’(

Grounded(Language(Problems( Physical(Scene( Language( Logical(Form((LF)( ``The(green( ι x. ( green ( x ) ∨ red ( x )) and(red( ∧ sphere ( x ) spheres.’’(

Grounded(Language(Problems( Grounded(Language(Understanding(( [Matuszek(et.(al(2012],([Krishnamurthy(et.(al(2013]( ( Physical(Scene( Language( Logical(Form((LF)( ``The(green( ι x. ( green ( x ) ∨ red ( x )) and(red( ∧ sphere ( x ) spheres.’’(

Grounded(Language(Problems( Grounded(Language(Understanding(( [Matuszek(et.(al(2012],([Krishnamurthy(et.(al(2013]( ( Physical(Scene( Language( Logical(Form((LF)( ``The(green( ι x. ( green ( x ) ∨ red ( x )) and(red( ∧ sphere ( x ) spheres.’’( Realiza.on(from(LF( [White(and(Rajkumar(2009],([Lu(and(Ng(2011](

Grounded(Language(Problems( Grounded(Language(Understanding(( [Matuszek(et.(al(2012],([Krishnamurthy(et.(al(2013]( ( Physical(Scene( Language( Logical(Form((LF)( ``The(green( ι x. ( green ( x ) ∨ red ( x )) and(red( ∧ sphere ( x ) spheres.’’( Realiza.on(from(LF( LF(Genera.on( [White(and(Rajkumar(2009],([Lu(and(Ng(2011]( Focus(of(this(talk:( Generate(Distribu.on(

Goal:(Generate(Distribu.ons(over(LFs( “The(green(ball( ι x.green ( x ) ∧ sphere ( x ) and(the(red(ball.”( ∪ ι x.red ( x ) ∧ sphere ( x ) “The(green(and(red(spheres.”( ι x. ( green ( x ) ∨ red ( x )) ( ∧ sphere ( x ) “The(green(and(red(balls.”( “The(apple(and(the(pear.”( ι x.apple ( x ) ∪ ι x.pear ( x ) . .. Model(how(people(refer(to(objects( • Many(different(expressions(for(each(referent( • Some(are(more(likely(to(be(used(in(prac.ce( • We(need(to(learn(a(probability(distribu.on(

Goal:(Generate(Distribu.ons(over(LFs( P(exp)( “The(green(ball( ι x.green ( x ) ∧ sphere ( x ) and(the(red(ball.”( ∪ ι x.red ( x ) ∧ sphere ( x ) 0.1( “The(green(and(red(spheres.”( ι x. ( green ( x ) ∨ red ( x )) 0.3( ( ∧ sphere ( x ) “The(green(and(red(balls.”( 0.2( “The(apple(and(the(pear.”( ι x.apple ( x ) ∪ ι x.pear ( x ) 0.4( . .. Model(how(people(refer(to(objects( • Many(different(expressions(for(each(referent( • Some(are(more(likely(to(be(used(in(prac.ce( • We(need(to(learn(a(probability(distribu.on(

Goal:(Generate(Distribu.ons(over(LFs( Output:( Input:( ι x.green ( x ) ∧ sphere ( x ) 0.3( ∪ ι x.red ( x ) ∧ sphere ( x ) ι x. ( green ( x ) ∨ red ( x )) 0.2( ∧ sphere ( x ) 0.1( ι x.apple ( x ) ∪ ι x.pear ( x ) . . . . . . Several(advantages( • Natural(varia.on(for(genera.on( • Useful(prior(for(understanding(systems(

Learn(from(Labeled(Examples( {( }( ,( “The(green,(red,(orange(and(yellow(toys”( “The(green,(red,(yellow,(and(orange(objects”( “All(the(pieces(that(are(not(blue(or(brown”( “All(items(that(are(not(brown(or(blue”( “Everything(that(is(not(brown(or(blue”( …..( {( }( ,( “The(red(and(green(balls.”( “The(red(and(greed(spheres.”( “The(pear(and(the(apple”( “The(red(ball(and(the(green(pear”( “All(the(balls(except(the(yellow(one”( ……( ( Lots(of(varia.on(in(prac.ce( • Collected(20(sentences(per(scene( • Mean(of(6(unique(logical(forms(per(scene( • Max(of(13((

Learn(from(Labeled(Examples( {( }( ,( 0.4( ι x. ( red ( x ) ∨ green ( x ) ∨ orange ( x ) ∨ yellow ( x )) ∧ obj ( x ) 0.3( E x.obj ( x ) \ ( ι x.brown ( x ) ∧ triangle ( x ) ∪ ( ι x.blue ( x ) ∧ lego ( x ) . . . {( }( ,( 0.1( ι x.green ( x ) ∧ sphere ( x ) ∪ ι x.red ( x ) ∧ sphere ( x ) 0.3( ι x. ( green ( x ) ∨ red ( x )) ∧ sphere ( x ) 0.2( ι x.apple ( x ) ∪ ι x.pear ( x ) . . . Lots(of(varia.on(in(prac.ce( • Collected(20(sentences(per(scene( • Mean(of(6(unique(logical(forms(per(scene( • Max(of(13((

Overview( Space(of(Referring(Expressions( Probabilis.c(Model( Learning( Experiments( Results( Conclusion( ( (

Overview( Space(of(Referring(Expressions ( (Seman.c(Modeling( (Enumera.ng(Referring(Expressions ( Probabilis.c(Model( Learning( Experiments( Results( Conclusion( ( (

Seman.c(Modeling( • Simplyityped(lambda(calculus( – [Steedman(1996],([Carpenter(1997],([Steedman(2011]( • Extended(to(model(set(reference( – Capture(dis.nc.ons(present(in(data( – As(simple(as(possible(

Seman.c(Modeling( Two(Simple(Types:( :(((Sets(of(Objects( e :((([True,(False]( t

Seman.c(Modeling( ACributes( < e, t > λ x.blue ( x )

Seman.c(Modeling( ACributes( < e, t > λ x.triangle ( x )

Seman.c(Modeling( ACributes( Logical(Operators( < e, t > λ x. ¬ blue ( x )

Seman.c(Modeling( ACributes( Logical(Operators( < e, t > λ x.blue ( x ) ∧ triangle ( x )

Seman.c(Modeling( ACributes( Logical(Operators( < e, t > λ x.blue ( x ) ∨ triangle ( x )

Seman.c(Modeling( ACribute(Coordina.on( Logical(Operators( Determiners( << e, t >, e > ι x.triangle ( x )

Seman.c(Modeling( ACribute(Coordina.on( Logical(Operators( Determiners( << e, t >, e > [ ι , E , A ] E ≈ ∀ A ≈ ∃

Seman.c(Modeling( ACribute(Coordina.on( Logical(Operators( Determiners( Set(Coordina.on( << e, e >, e > ι x.triangle ( x ) ∪ ι x.apple ( x )

Seman.c(Modeling( ACribute(Coordina.on( Logical(Operators( Determiners( Set(Coordina.on( << e, e >, e > ι x.triangle ( x ) \ ι x.blue ( x ) ∧ triangle ( x )

Seman.c(Modeling( ACribute(Coordina.on( Logical(Operators( Determiners( Set(Coordina.on( Plurality( Cardinality(

Seman.c(Modeling( • Simplyityped(lambda(calculus( – [Steedman(1996],([Carpenter(1997],([Steedman(2011]( • Extended(to(model(set(reference( – Capture(dis.nc.ons(present(in(data( – As(simple(as(possible( • Two(new(contribu.ons:( – Sets(as(a(primi.ve(type((plurals)( – Coordina.on(

Enumera.ng(Logical(Forms( • Enumerate(candidate(Logical(Forms( • Problem:( – Infinite(in(general( • Goal:( – finite(set(with(good(empirical(coverage( – strategy(for(enumera.ng(

Enumera.ng(Logical(Forms( 5 4 3 2 λ x.red ( x ) λ x.blue ( x ) λ x.cube ( x ) 1 λ x.object ( x ) λ x.rect ( x ) . . .

Enumera.ng(Logical(Forms( 5 4 3 λ x. ¬ red ( x ) 2 λ x.red ( x ) λ x.blue ( x ) λ x.cube ( x ) 1 λ x.object ( x ) λ x.rect ( x ) . . .

Enumera.ng(Logical(Forms( 5 4 3 λ x. ¬ red ( x ) λ x. ¬ blue ( x ) 2 λ x.red ( x ) λ x.blue ( x ) λ x.cube ( x ) 1 λ x.object ( x ) λ x.rect ( x ) . . .

Enumera.ng(Logical(Forms( 5 4 3 ι x.red ( x ) λ x. ¬ red ( x ) λ x. ¬ blue ( x ) 2 ι x.cube ( x ) E x.object ( x ) . . . λ x.red ( x ) λ x.blue ( x ) λ x.cube ( x ) 1 λ x.object ( x ) λ x.rect ( x ) . . .

Enumera.ng(Logical(Forms( 5 4 ι x. ¬ blue ( x ) E x. ¬ red ( x ) 3 ι x.red ( x ) λ x. ¬ red ( x ) λ x. ¬ blue ( x ) 2 ι x.cube ( x ) E x.object ( x ) . . . λ x.red ( x ) λ x.blue ( x ) λ x.cube ( x ) 1 λ x.object ( x ) λ x.rect ( x ) . . .

Enumera.ng(Logical(Forms( 5 4 ι x. ¬ blue ( x ) E x. ¬ red ( x ) λ x.red ( x ) ∧ cube ( x ) 3 ι x.red ( x ) λ x. ¬ red ( x ) λ x. ¬ blue ( x ) 2 ι x.cube ( x ) E x.object ( x ) . . . λ x.red ( x ) λ x.blue ( x ) λ x.cube ( x ) 1 λ x.object ( x ) λ x.rect ( x ) . . .

Enumera.ng(Logical(Forms( 5 4 ι x. ¬ blue ( x ) E x. ¬ red ( x ) λ x.red ( x ) ∧ cube ( x ) 3 λ x.red ( x ) ∧ object ( x ) ι x.red ( x ) λ x. ¬ red ( x ) λ x. ¬ blue ( x ) 2 ι x.cube ( x ) E x.object ( x ) . . . λ x.red ( x ) λ x.blue ( x ) λ x.cube ( x ) 1 λ x.object ( x ) λ x.rect ( x ) . . .

Enumera.ng(Logical(Forms( 5 4 ι x. ¬ blue ( x ) E x. ¬ red ( x ) λ x.red ( x ) ∧ cube ( x ) 3 λ x.red ( x ) ∧ object ( x ) λ x.red ( x ) ∨ cube ( x ) . . . ι x.red ( x ) λ x. ¬ red ( x ) λ x. ¬ blue ( x ) 2 ι x.cube ( x ) E x.object ( x ) . . . λ x.red ( x ) λ x.blue ( x ) λ x.cube ( x ) 1 λ x.object ( x ) λ x.rect ( x ) . . .