Visually Grounded, Task-oriented Dialogue Elia Bruni Outline - PowerPoint PPT Presentation

Visually Grounded, Task-oriented Dialogue Elia Bruni

Outline Language grounding Visual dialogue Q A Appendix: Current and future work Q A 2

Distributional semantics We found a cute, hairy wampimuk sleeping behind the tree 3

Distributional semantics We found a cute, hairy wampimuk sleeping behind the tree wampimuk hairy 12 cute 8 tree 5 money 0 1 dangerous 4

Distributional semantics We found a cute, hairy wampimuk sleeping behind the tree wampimuk animal hairy 12 wampimuk cute 8 dog tree 5 money 0 account 1 dangerous money 5

The grounding problem Bruni et al., Distributional Semantics in Technicolor. 2012 6

Language grounding Bruni et al., Multimodal Distributional Semantics. 2014 7

Language grounding Bruni et al., Multimodal Distributional Semantics. 2014 Enrichment of pure textual vectors with complementary information coming from perceptual visual features. 8

Language grounding Bruni et al., Multimodal Distributional Semantics. 2014 Enrichment of pure textual vectors with complementary information coming from perceptual visual features. 9

Language grounding Bruni et al., Multimodal Distributional Semantics. 2014 Enrichment of pure textual vectors with complementary information coming from perceptual visual features. • Word meaning not learned in isolation but from usage 10

Language grounding Bruni et al., Multimodal Distributional Semantics. 2014 Enrichment of pure textual vectors with complementary information coming from perceptual visual features. • Word meaning not learned in isolation but from usage • Missing the interactive and dialogical aspects of language 11

Visual Dialogue 12

Visual Dialogue A man and a woman are holding umbrellas A 13

Visual Dialogue A man and a woman are holding umbrellas A What color is his umbrella? 14

Visual Dialogue A man and a woman are holding umbrellas A What color is his umbrella? 15

Visual Dialogue A man and a woman are holding umbrellas A What color is his umbrella ? 16

Visual Dialogue A man and a woman are holding umbrellas A What color is his umbrella? His umbrella is black A 17

Visual Dialogue A man and a woman are holding umbrellas A What color is his umbrella? His umbrella is black A What about hers? 18

Visual Dialogue A man and a woman are holding umbrellas A What color is his umbrella? His umbrella is black A What about hers ? 19

Visual Dialogue A What color is his umbrella ? umbrella A hers 20