Data Demonstrations Beyond Coda Developing Meeting Support Technologies: From Data to Demonstration (and Beyond) Jean Carletta The AMI Consortium and University of Edinburgh 17th Nordic Conference on Computational Linguistics May 15th 2009 Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Outline Data 1 Demonstrations 2 Beyond 3 Coda 4 Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda The AMI Consortium Goal: to develop technologies that assist people during meetings and that help them make use of recorded meeting archives. Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Outline Data 1 Demonstrations 2 Beyond 3 Coda 4 Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda The basics 100 hrs of well-recorded meetings with native and non-native speakers of English orthographically transcribed with word timings, ASR output annotated by hand and machine for many communicative behaviours system for handling the dependencies among different versions of annotations 70 hrs is groups playing roles in a fictitious design exercise Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Why role play? In real meetings... Can’t control (or even know) participants’ motivations Can’t understand domain Can’t collect enough material in one domain to build ontologies Can’t easily measure meeting outcomes Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda The roles Project Manager Industrial Designer Interface Designer Marketing Expert Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Zingers Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda IS1004d, 3:07 - 4:11 Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Best of both worlds 100 hours of small group meetings 70% role-plays of remote control design teams 30% from a variety of genres (mostly real) find out where our methods and results generalize find out where they don’t Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Hand Annotations transcription with word-level timings from forced alignment (100%) timestamping against signal (10-30%) head gestures; hand gestures for addressing and interactions with objects; location in room; gaze; emotion? discourse structure (70%) dialogue acts (some w/ addressing), named entities, topic segments, linked extractive and abstractive summaries Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Costs in person-hours per meeting hour transcription 30 topic segments + abstractive summaries 6-10 dialogue acts w/ some relations 20 addressing 12 extractive summaries linked to abstract 1 named entities 2-5 hand gestures (rough timings) 6 head gestures (rough timings) 6 head gestures (precision timings) 20 movement around room 4 Total 110-115 Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Outline Data 1 Demonstrations 2 Beyond 3 Coda 4 Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Core technologies Visual focus of attention Automatic speech recognition Disfluency detection and removal Dialogue act segmentation and recognition Who was addressed Topic segmentation Extractive summarization Video editing Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Key Infrastructure: The Hub Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda The problem with not being there If most people are face-to-face, people who ”connect” in: Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda The problem with not being there If most people are face-to-face, people who ”connect” in: Can’t tell who’s speaking Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda The problem with not being there If most people are face-to-face, people who ”connect” in: Can’t tell who’s speaking Can’t see what’s going on Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda The problem with not being there If most people are face-to-face, people who ”connect” in: Can’t tell who’s speaking Can’t see what’s going on Can’t jump in as readily Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda The problem with not being there If most people are face-to-face, people who ”connect” in: Find it (even more) socially acceptable to multi-task Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda One solution Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda One solution Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Enriched conferencing Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda What enriched conferencing needs speech recognition or keyword spotting where people are looking (”visual focus of attention”) dialogue act segmentation and labeling addressee detection We have demonstrated it in real-time using keyword spotting and showing where just one of the people in the room is looking, and off-line with everything. Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Mobile conferencing application Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Content Linking Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Content Linking Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Content Linking System controller Media server Previous Current Documents meetings meeting User interface User interface ASR KWS Spurt & speaker segmenter Summarizer Document bank Query creator and indexer aggregator Hub & middleware Web search Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Meeting Browsing Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Meeting Browsing Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Outline Data 1 Demonstrations 2 Beyond 3 Coda 4 Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Meeting Browsing Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Community of Interest Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Synthetron collaborative brainstorming on a given topic on-line, real-time, large scale discussions uses instant messaging Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Synthetron collaborative brainstorming on a given topic on-line, real-time, large scale discussions uses instant messaging requires manual analysis to identify the key themes Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Comic Book Summaries Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda COI Experiences “Being a SME, following AMI since a few years, we are very enthusiastic with the mini-project experience... We were able to translate [AMIDA’s latest research results] into a practical application with AMIDA researchers in a very short time, testing the first comics format reports with several of our end clients in less than a month, allowing pragmatic and quick cycle time.” Joanne Celens, CEO, Synthethron Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda COI Experiences “Being a SME, following AMI since a few years, we are very enthusiastic with the mini-project experience... We were able to translate [AMIDA’s latest research results] into a practical application with AMIDA researchers in a very short time, testing the first comics format reports with several of our end clients in less than a month, allowing pragmatic and quick cycle time.” Joanne Celens, CEO, Synthethron Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Outline Data 1 Demonstrations 2 Beyond 3 Coda 4 Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda Data and tools make this possible - but what kind of data, and what kind of tools? Carletta Developing Meeting Support Technologies
Data Demonstrations Beyond Coda A Toy Example Carletta Developing Meeting Support Technologies
Recommend
More recommend
