a framework for learning multimodal clarification
play

A Framework for Learning Multimodal Clarification Strategies Verena - PowerPoint PPT Presentation

Sep 19, 2023 •201 likes •933 views

Motivation Framework The Data Collection Performance modelling Future work Summary A Framework for Learning Multimodal Clarification Strategies Verena Rieser 1 Ivana Kruijff-Korbayov 1 Oliver Lemon 2 1 Department of Computational

  1. Motivation Framework The Data Collection Performance modelling Future work Summary A Framework for Learning Multimodal Clarification Strategies Verena Rieser 1 Ivana Kruijff-Korbayová 1 Oliver Lemon 2 1 Department of Computational Linguistics, Saarland University 2 School of Informatics, University of Edinburgh In affiliation with: TALK Project http://www.talk-project.org/

  2. Motivation Framework The Data Collection Performance modelling Future work Summary CRs in Spoken Dialogue Systems System: What city are you leaving from? User: Urbana Champaign. System: Sorry, I’m not sure I understood what you said. Where are you leaving from? User: Urbana Champaign. System: I’m still having trouble understanding you. . . . What city are you leaving from? User: Chicago. [ CMU Communicator – User-System ] → System performs badly and sounds quite artificial.

  3. Motivation Framework The Data Collection Performance modelling Future work Summary CRs in Human-Human Dialogue Cust: I guess getting a car in London will not do me much good in /uh/ Spain is that right? Agent: I’m sorry? Getting a car . . . ? Cust: Yeah I’ll need a car in Madrid. Agent: OK. Cust.: I’ll be returning on Thursday the fifth. Agent: The fifth of February? Cust.: /UHU/ [ CMU Communicator – Human-Human ] → How to convert these kinds of clarification strategies to dialogue systems?

  4. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  5. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  6. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  7. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  8. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  9. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  10. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings • Human decision making on function features was influenced by dialogue type, modality and channel quality .

  11. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings • Human decision making on function features was influenced by dialogue type, modality and channel quality .

  12. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings → We know how to generate surface forms of CRs once we have the functions • Human decision making on function features was influenced by dialogue type, modality and channel quality .

  13. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings → We know how to generate surface forms of CRs once we have the functions • Human decision making on function features was influenced by dialogue type, modality and channel quality .

  14. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings → We know how to generate surface forms of CRs once we have the functions • Human decision making on function features was influenced by dialogue type, modality and channel quality . For dialogue systems we still don’t know:

  15. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings → We know how to generate surface forms of CRs once we have the functions • Human decision making on function features was influenced by dialogue type, modality and channel quality . For dialogue systems we still don’t know: → How to set the function features?

  16. Motivation Framework The Data Collection Performance modelling Future work Summary Generating CRs in task-oriented dialogues [ Rieser and Moore ] Implications for generating clarification requests in task-oriented dialogues , ACL-05. • Form-function mappings → We know how to generate surface forms of CRs once we have the functions • Human decision making on function features was influenced by dialogue type, modality and channel quality . For dialogue systems we still don’t know: → How to set the function features? → How do these strategies perform?

  17. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  18. Motivation Framework The Data Collection Performance modelling Future work Summary Approach Assumptions • Clarification strategies involve complex decision making over a variety of contextual factors • and exhaustive planning towards maximising a desired outcome. → Apply reinforcement learning (RL) in the information state update (ISU) approach. What is RL?

  19. Motivation Framework The Data Collection Performance modelling Future work Summary Framework for learning multimodal CRs Overall approach: MDP = ( S , A , T , R ) 1. Collect data on possible strategies in WOZ experiment. → Extract { A , S , R } 2. Bootstrap an initial policy using supervised learning in the ISU approach. → Learn wizards’ decisions in context ( T ) 3. Optimise the learnt policy for dialogue systems using RL ( π * ≈ maxE [ � j ≥ i r ( d , j ) | s i , a ] ). → How can we improve online reward measures r ( d , j ) ?

  20. Motivation Framework The Data Collection Performance modelling Future work Summary Outline Motivation Previous work Framework The Learning Approach The Data Collection Experimental Setup Results form the WOZ study Performance modelling RL and Performance modelling Dialogue costs and multimodality Ambiguity and (sub-)task success Future work Policy Shaping User-centred rewards

  21. Motivation Framework The Data Collection Performance modelling Future work Summary The SAMMIE-2 1 Data Collection Figure: Multimodal Wizard-of-Oz data collection setup for an in-car music player application, using the Lane Change driving simulator. Top right: User, Top left: Wizard, Bottom: transcribers. 1 SAMMIE stands for Saarbrücken Multimodal MP3 Player Interaction Experiment (cf. for more details [ Kruijff-Korbayová et al. ] , ENLG 2005).

Recommend

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine Learning Laboratory [MultiComp Lab] 1 CMU Course 11-777: Multimodal Machine Learning 2 Lecture Objectives What is Multimodal? Multimodal:

2.48k views • 71 slides
Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine Learning Laboratory [MultiComp Lab] 1 CMU Course 11-777: Multimodal Machine Learning 2 Lecture Objectives What is Multimodal? Multimodal:

1k views • 72 slides
Multimodal Machine Learning Main Goal Define a common taxonomy for multimodal machine learning

Multimodal Machine Learning Main Goal Define a common taxonomy for multimodal machine learning

Multimodal Machine Learning Main Goal Define a common taxonomy for multimodal machine learning and provide an overview of research in this area Introduction: Preliminary Terms Modality : the way in which something happens or is experienced

575 views • 26 slides
Multimodal Deep Learning Ahmed Abdelkader Design & Innovation Lab, ADAPT Centre Talk outline

Multimodal Deep Learning Ahmed Abdelkader Design & Innovation Lab, ADAPT Centre Talk outline

Multimodal Deep Learning Ahmed Abdelkader Design & Innovation Lab, ADAPT Centre Talk outline What is multimodal learning and what are the challenges? Flickr example: joint learning of images and tags Image captioning: generating

735 views • 48 slides
Retrieving Categorical Emotions using a Probabilistic Framework to Define Preference Learning

Retrieving Categorical Emotions using a Probabilistic Framework to Define Preference Learning

Retrieving Categorical Emotions using a Probabilistic Framework to Define Preference Learning Samples R EZA L OTFIAN AND C ARLOS B USSO Rela0ve Angry label score Ha Happy Multimodal Signal Processing (MSP) lab An Angry The University of

443 views • 20 slides
Using Multimodal Learning Analytics to Identify Aspects of Collaboration in Project-based Learning

Using Multimodal Learning Analytics to Identify Aspects of Collaboration in Project-based Learning

Using Multimodal Learning Analytics to Identify Aspects of Collaboration in Project-based Learning Daniel Spikol, Malm University, Sweden Emanuele Ruffaldi, Scuola Superiore Sant'Anna,Italy Mutlu Cukurova, University College London, UK CSCL

259 views • 22 slides
Multimodal Learning for Image Captioning and Visual Question Answering Xiaodong He Deep Learning

Multimodal Learning for Image Captioning and Visual Question Answering Xiaodong He Deep Learning

Multimodal Learning for Image Captioning and Visual Question Answering Xiaodong He Deep Learning T echnology Center Microsoft Research UC Berkeley, April 7 th , 2016 Knowledge Freebase Text Vision Barack Obama is an American politician

882 views • 50 slides
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of

Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of

Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information Sudha Rao 1 and Hal Daum III 1,2 1 University of Maryland College Park 2 Microsoft Research New York CODE + DATA :

1.54k views • 111 slides
Multimodal Framework Proposal Skip Cave Chief Scientist, Intervoice Inc. 1 Workshop Goals

Multimodal Framework Proposal Skip Cave Chief Scientist, Intervoice Inc. 1 Workshop Goals

Multimodal Framework Proposal Skip Cave Chief Scientist, Intervoice Inc. 1 Workshop Goals Identify & prioritize requirements for changes, extensions, and additions to the MMI architecture to better support Speech, GUI, Ink, and

188 views • 15 slides
Concentric Cable Tender (DD6739/TX) Clarification Presentation Purpose of the clarification

Concentric Cable Tender (DD6739/TX) Clarification Presentation Purpose of the clarification

Concentric Cable Tender (DD6739/TX) Clarification Presentation Purpose of the clarification To explain the requirements of this tender. Tender extensions by suppliers must be requested 5 days prior closing date. Eskom Standard

168 views • 13 slides
An online platform for student - other learning dyad interaction, multimodal feedback and

An online platform for student - other learning dyad interaction, multimodal feedback and

An online platform for student - other learning dyad interaction, multimodal feedback and assessment. Lunch and Learn for Education Focused academics, Feb 8 th 2019 Dr Silas Taylor Convenor of Clinical Skills, UNSW Medicine

398 views • 29 slides
Speaker and Emotion Recognition of TV-Series Data Using Multimodal and Multitask Deep Learning

Speaker and Emotion Recognition of TV-Series Data Using Multimodal and Multitask Deep Learning

Speaker and Emotion Recognition of TV-Series Data Using Multimodal and Multitask Deep Learning Sashi Novitasari 1 , Quoc Truong Do 1 , Sakriani Saktj 1,3 , Dessi Lestari 2 , Satoshi Nakamura 1,3 1 Graduate School of Informatjon Science, Nara

395 views • 16 slides
Visualizing Meaning: Modeling Communication through Multimodal Simulations James Pustejovsky

Visualizing Meaning: Modeling Communication through Multimodal Simulations James Pustejovsky

Creating Situated Grounding Multimodal Simulation Situated Communication Learning by Communication Visualizing Meaning: Modeling Communication through Multimodal Simulations James Pustejovsky Brandeis University COLING 2018 Santa Fe, New

1.04k views • 76 slides
Multimodal semi-supervised learning for image classification Matthieu Guillaumin, Jakob Verbeek,

Multimodal semi-supervised learning for image classification Matthieu Guillaumin, Jakob Verbeek,

Multimodal semi-supervised learning for image classification Matthieu Guillaumin, Jakob Verbeek, Cordelia Schmid LEAR team, INRIA Grenoble, France Motivation and goal Images often come with additional textual info. Videos with scripts and

501 views • 21 slides
Unsupervised learning of multimodal image registration using domain adaptation with projected

Unsupervised learning of multimodal image registration using domain adaptation with projected

Heinrich & Hansen: Learning multimodal registration: domain adaptation & projected Earth Movers discrepancies Unsupervised learning of multimodal image registration using domain adaptation with projected Earth Movers discrepancies

679 views • 5 slides
Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning Minghai Chen*,

Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning Minghai Chen*,

Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning Minghai Chen*, Sen Wang*, Paul Pu Liang*, Tadas Baltrusaitis, Amir Zadeh, Louis-Philippe Morency Natural Computer Interaction Int Intel elligent gent Robo Ro

719 views • 27 slides
Interpretable Multimodal Deep Learning for Objective Diagnosis, Prognosis and Biomarker Discovery

Interpretable Multimodal Deep Learning for Objective Diagnosis, Prognosis and Biomarker Discovery

Interpretable Multimodal Deep Learning for Objective Diagnosis, Prognosis and Biomarker Discovery Faisal Mahmood, PhD Postdoctoral Fellow Department of Biomedical Engineering Johns Hopkins University faisalm@jhu.edu | http://faisal.ai March

1.08k views • 62 slides
Extracting Visual Knowledge from the Web with Multimodal Learning Dihong Gong, Daisy Zhe Wang

Extracting Visual Knowledge from the Web with Multimodal Learning Dihong Gong, Daisy Zhe Wang

Extracting Visual Knowledge from the Web with Multimodal Learning Dihong Gong, Daisy Zhe Wang Data Science Research Lab Department of Computer and Information Science and Engineering University of Florida In Presentation of International Joint

309 views • 13 slides
Workday Onboarding Process Clarification of Roles in Onboarding New Hires Includes clarification

Workday Onboarding Process Clarification of Roles in Onboarding New Hires Includes clarification

Workday Onboarding Process Clarification of Roles in Onboarding New Hires Includes clarification on all the guidance provided to New Hires after they are hired into a new position in Workday, and steps taken to optimize a seamless onboarding

309 views • 4 slides
Multimodal Implementation Plan Multimodal Implementation Plan OUTLINE Overview

Multimodal Implementation Plan Multimodal Implementation Plan OUTLINE Overview

Bowling Green-Warren County MPOs Multimodal Implementation Plan Multimodal Implementation Plan OUTLINE Overview Planning Process Overview Non-Motorized Latent Demand Level of T raffjc Stress Pairwise

350 views • 23 slides
Clarification meeting date: 17 September 2019 Tender Closing date: 03 October 2019 Commercial

Clarification meeting date: 17 September 2019 Tender Closing date: 03 October 2019 Commercial

CLARIFICATION MEETING FOR CORP4923 Contact Centre Telephony System Solution Implementation Clarification meeting date: 17 September 2019 Tender Closing date: 03 October 2019 Commercial clarification Eskom Representative : Violet Beetha

757 views • 28 slides
Making Sense of Multimodal Learning Analytics (MMLA) Marcelo Worsley Assistant Professor,

Making Sense of Multimodal Learning Analytics (MMLA) Marcelo Worsley Assistant Professor,

Making Sense of Multimodal Learning Analytics (MMLA) Marcelo Worsley Assistant Professor, Learning Sciences & Computer Science Agenda - What is MMLA? - Why use MMLA? - How to use MMLA? - Examples of MMLA research - On-going challenges and

721 views • 40 slides
The PAC Learning Framework Guoqing Zheng January 20, 2015 Guoqing Zheng The PAC Learning

The PAC Learning Framework Guoqing Zheng January 20, 2015 Guoqing Zheng The PAC Learning

The PAC Learning Framework Guoqing Zheng January 20, 2015 Guoqing Zheng The PAC Learning Framework Intro Questions about learning: What can be learned efficiently? (what is hard to learn?) How many examples needed? General model of

428 views • 18 slides
Samskip Multimodal Short Sea and Multimodal Business www.samskip.com 1 Samskip Group Profile

Samskip Multimodal Short Sea and Multimodal Business www.samskip.com 1 Samskip Group Profile

Samskip Multimodal Short Sea and Multimodal Business www.samskip.com 1 Samskip Group Profile Samskip is a globally active company offering transport and related services by land, sea, rail, inland waterways and air. Dutch based.

313 views • 18 slides

More recommend