Modeling Conceptual Understanding in Image Reference Games Rodolfo - PowerPoint PPT Presentation

Modeling Conceptual Understanding in Image Reference Games Rodolfo Corona * Stephan Alaniz * Zeynep Akata *Equal contribution

Yellow

Yellow

Yellow Spotted

Listener Clusters Misunderstands Misunderstands Colors Shapes 7

Speaker Listener 8

Speaker Listener Yellow feet Yellow feet Red beak Red beak Cone beak Cone beak Yellow feet Yellow feet Red beak Red beak Cone beak Cone beak Predicting attributes 9

Speaker Listener Yellow feet Yellow feet Red beak Red beak - Cone beak Cone beak Yellow feet = Cone beak Yellow feet Red beak Yellow feet Cone beak Red beak Red beak Cone beak Cone beak Selecting one attribute 10

Speaker Listener Yellow feet Yellow feet Red beak Red beak - Cone beak - Yellow feet Cone beak Yellow feet Red beak = Cone beak Yellow feet Red beak Yellow feet Cone beak Cone beak Red beak Red beak Cone beak Cone beak Listener guesses 11

Speaker Listener Reward +1 -1 “It’s image ” Yellow feet Yellow feet Red beak Red beak - Cone beak - Yellow feet Cone beak Yellow feet Red beak = Cone beak Yellow feet Red beak Yellow feet Cone beak Cone beak Red beak Red beak Cone beak Cone beak +1 match, -1 mismatch 12

Speaker Listener Reward Agent +1 -1 Embedding “It’s image ” Yellow feet Yellow feet Red beak Red beak - Cone beak - Yellow feet Cone beak Yellow feet Red beak = Cone beak Yellow feet Red beak Yellow feet Cone beak Cone beak Red beak Red beak Cone beak Cone beak Speaker incorporates game into history 13

CUB Avg. Eval Reward Number of Training Games Parameterized attribute selection policies incentivised to adapt faster to listeners.

CUB Avg. Eval Reward Number of Training Games Parameterized attribute selection policies incentivised to adapt faster to listeners.

CUB Avg. Eval Reward Number of Training Games Parameterized attribute selection policies incentivised to adapt faster to listeners.

CUB Avg. Eval Reward Number of Training Games Parameterized attribute selection policies incentivised to adapt faster to listeners.

CUB Avg. Eval Reward Number of Training Games Modeling listener crucial for maximizing game performance.

CUB VI Number of Training Games Parameterized policies embed agents with greater correspondence to ground truth clusters.

Brown back Blue underparts Rufous belly Yellow wing Discrim. Brown back Blue underparts Rufous belly Yellow wing Chosen Game 1

Brown back Blue underparts Rufous belly Yellow wing Discrim. Brown back Blue underparts Rufous belly Yellow wing Chosen Game 1 Rufous crown Yellow belly Discrim. Orange leg Yellow belly Chosen Rufous crown Solid belly pattern Spotted belly pattern Spotted back pattern Game 10

Brown back Blue underparts Rufous belly Yellow wing Discrim. Brown back Blue underparts Rufous belly Yellow wing Chosen Game 1 Rufous crown Yellow belly Discrim. Orange leg Yellow belly Chosen Rufous crown Solid belly pattern Spotted belly pattern Spotted back pattern Game 10 Orange beak Yellow belly Yellow wing White belly Discrim. Duck-like shape Has eyebrow Solid belly pattern Chosen Forked tail shape Game 100

Modeling Conceptual Understanding in East Hall B+C Poster #79 Image Reference Games Rodolfo Corona * Stephan Alaniz * Zeynep Akata *Equal contribution