visual attention in spoken hri
play

Visual Attention in Spoken HRI Maria Staudte & Matthew Crocker - PowerPoint PPT Presentation

Aug 18, 2023 •142 likes •465 views

Visual Attention in Spoken HRI Maria Staudte & Matthew Crocker Saarland University, Germany 1 Gaze in HRI Robot gaze appears active,enjoyable,friendly (e.g. Kanda et al. 01, Kuno et al. 07) Robot gaze is different

  1. Visual Attention in Spoken HRI Maria Staudte & Matthew Crocker Saarland University, Germany 1

  2. Gaze in HRI Robot gaze appears “active”,“enjoyable”,“friendly” (e.g. Kanda et al. ’01, Kuno et al. ’07) Robot gaze is different from human behaviour Can human-like robot gaze be more than “active” or “friendly”? 2

  3. Gaze in HRI Can robot gaze fulfill similar functions as human gaze? coordinating dialogue (e.g. Cassell et al. ’99, Sidner et al. ‘05) information structure (e.g. Mutlu et al. ’06) signaling and checking understanding / joint attention visual reference 3

  4. People do... ...look at an object they are about to mention (about 1 sec before mentioning) ...look at an object that someone else mentions (about 200-500 msec after mentioning) ...look at where the speaker looks 4

  5. Should robots ... ...look at an object they are about to mention? (about 1 sec before mentioning) ...look at an object that someone else mentions? (about 200-500 msec after mentioning) ...look at where the speaker looks? 5

  6. Should robots ... ...look at an object they are about to mention? (about 1 sec before mentioning) 6

  7. Hypothesis ...look at an object they are about to mention? (about 1 sec before mentioning) Cognitively motivated robot gaze behaviour is beneficial for HRI People follow robot gaze This affects the comprehension of the robot utterance 7

  8. Design - Videos showing robot - Task: Right or wrong robot statement? - Measures: • Eye-tracking • Response time (button presses) 8

  9. Example: true - congruent * Thanks to the CoSy/CogX group at DFKI for robi. 9

  10. Conditions Spoken Sentence: Statement Gaze Gaze towards: The cone is taller than the heart that’s silver. TRUE congruent <cone> <silver heart> The cone is taller than the heart that’s silver . TRUE incongruent <cone> <green heart> The cone is taller than the heart that’s silver . TRUE no gaze ------------ ----------------- The cone is taller than the heart that’s green. FALSE congruent <cone> <green heart> The cone is taller than the heart that’s green. FALSE incongruent <cone> <silver heart> The cone is taller than the heart that’s green. FALSE no gaze ------------ ----------------- 10

  11. Conditions Spoken Sentence: Statement Gaze Gaze towards: The cone is taller than the heart that’s silver . TRUE congruent <cone> <silver heart> The cone is taller than the heart that’s silver . TRUE incongruent <cone> <green heart> The cone is taller than the heart that’s silver . TRUE no gaze ------------ ----------------- The cone is taller than the heart that’s green. FALSE congruent <cone> <green heart> The cone is taller than the heart that’s green. FALSE incongruent <cone> <silver heart> The cone is taller than the heart that’s green. FALSE no gaze ------------ ----------------- 11

  12. Conditions Spoken Sentence: Statement Gaze Gaze towards: The cone is taller than the heart that’s silver. TRUE congruent <cone> <silver heart> The cone is taller than the heart that’s silver . TRUE incongruent <cone> <green heart> The cone is taller than the heart that’s silver . TRUE no gaze ------------ ----------------- The cone is taller than the heart that’s green . FALSE congruent <cone> <green heart> The cone is taller than the heart that’s green . FALSE incongruent <cone> <silver heart> The cone is taller than the heart that’s green . FALSE no gaze ------------ ----------------- 12

  13. Conditions Spoken Sentence: Statement Gaze Gaze towards: The cone is taller than the heart that’s silver . congruent TRUE <cone> <silver heart> The cone is taller than the heart that’s silver . TRUE incongruent <cone> <green heart> The cone is taller than the heart that’s silver . TRUE no gaze ------------ ----------------- The cone is taller than the heart that’s green . FALSE congruent <cone> <green heart> The cone is taller than the heart that’s green. FALSE incongruent <cone> <silver heart> The cone is taller than the heart that’s green. FALSE no gaze ------------ ----------------- 13

  14. Conditions Spoken Sentence: Statement Gaze Gaze towards: The cone is taller than the heart that’s silver. TRUE congruent <cone> <silver heart> The cone is taller than the heart that’s silver . incongruent TRUE <cone> <green heart> The cone is taller than the heart that’s silver . TRUE no gaze ------------ ----------------- The cone is taller than the heart that’s green. FALSE congruent <cone> <green heart> The cone is taller than the heart that’s green . FALSE incongruent <cone> <silver heart> The cone is taller than the heart that’s green. FALSE no gaze ------------ ----------------- 14

  15. Example: true - incongruent 15

  16. Analysis “The cone is taller than ...” Interest Recorded Areas Fixations 16

  17. Measures Response Time : Button presses for right/wrong statements Fixations : Fixated objects during robot gaze and speech IP1 IP2 IP3 “The cone is taller than the heart that is silver.” 17

  18. Design Details 48 subjects 24 items, 48 fillers 72 trials 30 min. duration Repeated-Measures ANOVA: fixation proportions response times 18

  19. Results Fixations: robot gaze - following robot speech - following Response Time: congruent < no robot gaze < incongruent 19

  20. Results True - congruent True - incongruent 20

  21. Results Fixations: robot gaze - following robot speech - following Response Time: congruent < no robot gaze < incongruent 21

  22. Results 22

  23. Summary Human behaviour in HRI is consistent with that in HHI Utterance-mediated & gaze-mediated gaze Generality of results: Speech-oriented task Mechanic robot head/eyes Errors (incongruent behaviour) 23

  24. Conclusions People rely on cognitively motivated robot gaze as visual reference Visual reference is a cue for grounding and disambiguation People associate the robot’s visual attention with what it intends to say Inconsistent robot gaze behaviour leads to comprehension difficulties (slow-down) A robot should only make “sensible” head & eye movements that are consistent with its utterance! 24

  25. Outlook Investigating perception of “intended” referents (confirming previous results and assumptions) Investigating flexibility of temporal alignment with speech Extending work to other non-verbal dimensions 25

  26. More interesting stuff... 26

  27. Results RT Experiment 27

  28. Correction Experiment Task: Correct robot’s errors (utterances) Purpose: What did people think was the intended object? 28

  29. False - No Gaze “The cone is taller than the heart that’s green.” (Predicted) Correction: The cone is shorter than the heart that’s green. The cone is taller than the heart that’s silver. 29

  30. False - Congruent “The cone is taller than the heart that’s green.” (Predicted) Correction: The cone is shorter than the heart that’s green. The cone is taller than the heart that’s silver. 30

  31. False - No Gaze “The cone is taller than the heart that’s green.” (Predicted) Correction: The cone is shorter than the heart that’s green. The cone is taller than the heart that’s silver. 31

  32. Results Green heart Silver heart Gaze to: <Green heart> ----- <Silver heart> 32

Recommend

Visual Attention FEF V4 spatial attention: simultaneous neural recordings in V4

Visual Attention FEF V4 spatial attention: simultaneous neural recordings in V4

10/29/14 Visual Attention FEF V4 spatial attention: simultaneous neural recordings in V4 &amp; Frontal Eye Fields (monkeys) Gregoriou, Gotts, Zhou, Desimone (2010) object attention: MEG &amp; fMRI in FFA (faces),

238 views • 4 slides
Attention, Transformer and BERT Prof. Kuan-Ting Lai 2020/6/16 Attention is All You Need! A.

Attention, Transformer and BERT Prof. Kuan-Ting Lai 2020/6/16 Attention is All You Need! A.

Attention, Transformer and BERT Prof. Kuan-Ting Lai 2020/6/16 Attention is All You Need! A. Waswani et al., NIPS , 2017 Google Brain &amp; University of Toronto 2 Attention Visual attention and textual attention

628 views • 21 slides
Coordinated Interplay of Scene, Utterance, and World Knowledge - Comprehension of spoken

Coordinated Interplay of Scene, Utterance, and World Knowledge - Comprehension of spoken

Coordinated Interplay of Scene, Utterance, and World Knowledge - Comprehension of spoken utterances that relate to a visual scene - Eye movements in scenes during utterance comprehension - utterance can direct attention to an object in the scene

207 views • 10 slides
Eye-tracking for capturing human visual attention Eye-tracking for capturing human visual

Eye-tracking for capturing human visual attention Eye-tracking for capturing human visual

BubbleView : an interface for crowdsourcing image importance maps and tracking visual attention Nam Wook Zoya Michelle Krzysztof Aude Fredo Hanspeter Kim* Bylinskii* Borkin Gajos Oliva Durand Pfister Eye-tracking for capturing human

830 views • 69 slides
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio Presented by Kathy Ge Motivation: Attention

567 views • 20 slides
Attention in NLP CS 6956: Deep Learning for NLP Overview What is attention Attention in

Attention in NLP CS 6956: Deep Learning for NLP Overview What is attention Attention in

Attention in NLP CS 6956: Deep Learning for NLP Overview What is attention Attention in encoder-decoder networks Various kinds of attention 2 Overview What is attention? Attention in encoder-decoder networks 3 Visual

971 views • 73 slides
Improving Image and Sentence Matching with Multimodal Attention and Visual Attributes Yan Huang

Improving Image and Sentence Matching with Multimodal Attention and Visual Attributes Yan Huang

National Lab of Institute of Automation Pattern Recognition Chinese Academy of Sciences Improving Image and Sentence Matching with Multimodal Attention and Visual Attributes

402 views • 38 slides
Cognitive Workload and Visual Attention Analyses of the Air Traffic Control Tower Flight Data

Cognitive Workload and Visual Attention Analyses of the Air Traffic Control Tower Flight Data

Cognitive Workload and Visual Attention Analyses of the Air Traffic Control Tower Flight Data Manager (TFDM) Prototype Demonstration* Kiran Lokhande &amp; Hayley J. Davison Reynolds MIT Lincoln Laboratory HFES 56 th Annual Meeting 26 October,

442 views • 25 slides
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Lei

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Lei

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, Yoshua Bengio. ICML 2015 Presented By: Sai Krishna Bollam

410 views • 18 slides
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu*, Jimmy Ba

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu*, Jimmy Ba

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu*, Jimmy Ba , Ryan Kiros , Kyunghyun Cho*, Aaron Courville*, Ruslan Salakhutdinov , Richard Zemel , Yoshua Bengio* eal*/ University of Toronto

824 views • 46 slides
Neurocognitive Dysfunction in Schizophrenia Attention/Vigilance Immediate Memory Secondary

Neurocognitive Dysfunction in Schizophrenia Attention/Vigilance Immediate Memory Secondary

Neurocognitive Dysfunction in Schizophrenia Attention/Vigilance Immediate Memory Secondary Memory Verbal Fluency, Learning Spatial Working Memory Motor Reaction Time Processing Speed Executive Functions Visual-motor Skills Early Visual

246 views • 21 slides
Analyzing the Relationship Between Head Pose and Gaze to Model Driver Visual Attention Sumit Jha

Analyzing the Relationship Between Head Pose and Gaze to Model Driver Visual Attention Sumit Jha

Analyzing the Relationship Between Head Pose and Gaze to Model Driver Visual Attention Sumit Jha and Carlos Busso Multimodal Signal Processing (MSP) Laboratory Department of Electrical Engineering, The University of Texas at Dallas,

453 views • 24 slides
A spatiotemporal model with visual attention for video classification Mo Shan and Nikolay

A spatiotemporal model with visual attention for video classification Mo Shan and Nikolay

A spatiotemporal model with visual attention for video classification Mo Shan and Nikolay Atanasov Department of Electrical and Computer Engineering July 16, 2017 Outline Motivation Proposed model Experiment Conclusion Motivation Video

161 views • 12 slides
Show, Attend, and Tell Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy

Show, Attend, and Tell Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy

Show, Attend, and Tell Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, Yoshua Bengio University of Montreal and University of

481 views • 16 slides
Spoken Language Structure Hsin-min Wang References: - X. Huang et al., Spoken Language

Spoken Language Structure Hsin-min Wang References: - X. Huang et al., Spoken Language

Spoken Language Structure Hsin-min Wang References: - X. Huang et al., Spoken Language Processing, Chapter 2 Human Speech Communication Spoken language is used to communicate information from a speaker to a listener. Speech production and

730 views • 62 slides
The Attention Economy What is the attention economy? A business model where you (as the

The Attention Economy What is the attention economy? A business model where you (as the

The Attention Economy What is the attention economy? A business model where you (as the company) want to hold the users attention as much as possible. Attention is treat like a scarce resource What are ethical issues that have emerged

170 views • 3 slides
EASM 2014 considered more informative and are supposed to attract attention (Oliva, Torralba,

EASM 2014 considered more informative and are supposed to attract attention (Oliva, Torralba,

TV SPECTATORS' VISUAL ATTENTION TO IN-STADIUM SPONSORSHIP MESSAGES Submitting author: Mrs Bartsch Katja Technische Universitt Mnchen, Department of Sport &amp; Health Management Munich, 80992 Germany All authors: Bartsch Katja (corresp),

294 views • 3 slides
MALACH : Multilingual Access to Large spoken ArCHives http://www.clsp.jhu.edu/research/malach

MALACH : Multilingual Access to Large spoken ArCHives http://www.clsp.jhu.edu/research/malach

MALACH : Multilingual Access to Large spoken ArCHives http://www.clsp.jhu.edu/research/malach (funded under NSF ITR Award 0122466) Sam Gustman Survivors of the Shoah Visual History Foundation Bhuvana Ramabhadran, Michael Picheny, Martin

886 views • 56 slides
Grounding LING 575: Spoken Dialog Systems May 12 th , 2016 1 What is Grounding? Spoken Dialog

Grounding LING 575: Spoken Dialog Systems May 12 th , 2016 1 What is Grounding? Spoken Dialog

Grounding LING 575: Spoken Dialog Systems May 12 th , 2016 1 What is Grounding? Spoken Dialog is special way of communication It is the result of a joint collaboration Achieving a common ground of mutually believed facts of what is being

1.07k views • 73 slides
THE SPOKEN BLESSING Numbers 6:22 27 Since the start of human history, the spoken blessing

THE SPOKEN BLESSING Numbers 6:22 27 Since the start of human history, the spoken blessing

THE SPOKEN BLESSING Numbers 6:22 27 Since the start of human history, the spoken blessing has played a significant role in the lives of Gods people. Gods first act after creating man and woman was to bless them. Is it possible that

737 views • 37 slides
Spoken Language Structure Berlin Chen 2003 References: - X. Huang et. al., Spoken Language

Spoken Language Structure Berlin Chen 2003 References: - X. Huang et. al., Spoken Language

Spoken Language Structure Berlin Chen 2003 References: - X. Huang et. al., Spoken Language Processing, Chapter 2 Introduction Take a button-up approach to introduce the basic concepts from sound to phonetics ( ) and phonology (

837 views • 63 slides
STANDARDS IN SPOKEN CORPORA OUTLINE (1) Case study: Spoken

STANDARDS IN SPOKEN CORPORA OUTLINE (1) Case study: Spoken

7/21/15 Thomas Schmidt, Programmbereich Mndliche Korpora STANDARDS IN SPOKEN CORPORA OUTLINE (1) Case study: Spoken corpora at the SFB 538 (2)

351 views • 22 slides
Spoken and Sign Languages Spoken and Sign Languages A Cross Modal Study Purushottam Kar Achla

Spoken and Sign Languages Spoken and Sign Languages A Cross Modal Study Purushottam Kar Achla

Spoken and Sign Languages Spoken and Sign Languages A Cross Modal Study Purushottam Kar Achla M. Raina Amitabha Mukerjee Indian Institute of Technology Kanpur 28th All India Conference of Linguists, Banaras Hindu University, 2006 1 Overview

492 views • 22 slides
Attention Eye tracking seminar 2/19/15 Presented by Tatiana Emmanouil Outline What is

Attention Eye tracking seminar 2/19/15 Presented by Tatiana Emmanouil Outline What is

Attention Eye tracking seminar 2/19/15 Presented by Tatiana Emmanouil Outline What is attention? How is attention allocated? How are eye movements related to attention? Further questions Attention Attention

331 views • 18 slides

More recommend