Dialogue Structure Annotation for Multi-Floor Interaction David Traum, Cassidy Henry, Stephanie Lukin, Ron Artstein, Felix Gervitz, Kimberly A. Pollard, Claire Bonial, Su Lei, Clare R. Voss, Matthew Marge, Cory J. Hayes, Susan G. Hill The work depicted here was sponsored by the U.S. Army. Statements and opinions expressed do not necessarily reflect the position or the policy of the United States Government, and no official endorsement should be inferred.
Outline 1. Conceptual Framework 3. Data ▪ Meso-level dialogue structure ▪ Doman: Human-robot collaboration ▪ Multi-floor Dialogue & multi- ▪ 2 Wizards communicators ▪ Multi-floor dialogue structure ▪ Example Annotations ▪ Corpus Statistics 2. Multi-floor Dialogue Structure Annotation scheme 4. Structure Patterns 5. Uses of data and Future work 2
Types of Dialogue Structure (Traum & Nakatani 1999) Structure Content Structure Granularity ▪ ▪ Micro – within a single turn Intentional Meso – short subdialogue ▪ ▪ Linguistic ▪ ▪ Macro – full conversation Relational/Rhetorical ▪ Attentional State ▪ Turn-taking/floor management ▪ Grounding ▪ Participant structure 3
Meso-level Dialogue Structure Annotations Structure Types Annotations ▪ Intentional: ▪ TUs: cluster of utterances Transaction Units – smallest unit of ▪ Not necessarily sequential specified and performed action, including all dialogue needed to accomplish this ▪ Relational/Rhetorical : Relations: Label 2 nd part ▪ utterance with Relations between utterances ▪ Antecedent within a transaction ▪ Relation type 4
Example: At a lunch counter ▪ Customer: I’d like a cheeseburger ▪ Waiter: one cheeseburger. ▪ Waiter: (placing burger in bag) here you go. ▪ Customer: thanks! ▪ Waiter: would you like fries with that? ▪ Customer: Sure, a large one please! ▪ Waiter: (placing fries box in bag): one large fries. 5
Example: Transaction Units (TUs) ▪ Customer: I’d like a cheeseburger ▪ Waiter: one cheeseburger. ▪ Waiter: (placing burger in bag) here you go. ▪ Customer: thanks! ▪ Waiter: would you like fries with that? ▪ Customer: Sure, a large one please! ▪ Waiter: (placing fries box in bag): one large fries. 6
Example: Relations 1. Customer: I’d like a cheeseburger 2. Waiter: one cheeseburger. Acknowledgement 3. Waiter: (placing burger in bag) here you go. Acknowledgement 4. Customer: thanks! 3 rd turn feedback 5. Waiter: would you like fries with that? Answer 6. Customer: Sure, large please! 7. Waiter: (placing fries in bag): one large fries. Acknowledgement 7
Example: TU Structures 1 Ack Ack 1. Customer: I’d like a cheeseburger 3 2 3 rd TF 2. Waiter: one cheeseburger. Acknowledgement 4 3. Waiter: (placing burger in bag) here you go. Acknowledgement 4. Customer: thanks! 3 rd turn feedback 5 Answer 5. Waiter: would you like fries with that? 6 Answer 6. Customer: Sure, large please! Ack 7 7. Waiter: (placing fries in bag): one large fries. Acknowledgement 8
Floor and Participant Structure Kinds of Interactions between Floors Participants and Floors ▪ Same purpose, distinct participants ▪ Single floor Dyadic (A,B) ▪ Co-located, observable Single floor Multiparty: (A,B,C,…) ▪ ▪ Participants play different roles for ▪ Multiple floors (with different sets different floors (e.g. active participant of participants): {(A B), (C D E)} vs overhearer) ▪ Some Shared participant(s) ▪ multi-communicating (Rentch et al) ▪ Multi-floor dialogue: ▪ Same purpose ▪ Some Multi-communicating participant(s) 9 ▪ Content flows across floors
Examples of (observable) Multi-floor dialogue Indirect Action Live Interpretation 10
Multi-floor Relation types Examples: ▪ Expansions - relate (A,B) A- >B: I’ll have a cheeseburger 1. utterances that are produced 2. (A,B) A->B: and a small coke by the same participant within the same floor. ▪ Responses - relate 1. (A,B) A->B: a small coke utterances by different 2. (A,B) B->A: no coke, pepsi participants in the same floor. (A,B) A- >B: I’ll have a cheeseburger 1. ▪ Translations - relate 2. (B,C) B->C: Cheeseburger!! utterances in different floors 11
Relations by type (1) Expansions Translation a) Continue a) Translation <from,to> b) (self-) Correction b) Partial c) Link-next c) Quotation d) Summarization d) Comment 12
Relations by type (2) Responses a. Processing: positive feedback at perception level b. acknowledgement: positive feedback of understanding c. clarification: negative feedback of understanding d. question-response e. reciprocal response: e.g. “hello” - > “hello” f. 3rd turn feedback: response to feedback g. other 13
Response sub-relations acknowledgment clarification question-response ▪ ▪ ▪ ack-done req-clar answer ▪ ▪ ▪ ack-doing clar-repair Non-Answer-Response (NAR) ▪ ▪ ack-wilco missing info ▪ ▪ ack-understand nack ▪ ▪ ack-try req-repeat ▪ ▪ ack-unsure clar-repeat ▪ ack-cant 14
Domain: Human-Robot Collaboration Human Remote reconnaissance task Commander • Unfamiliar environment • Bandwidth limitations VIEWS • User and robot not co-present • What would the human users want to say? VERBAL • Need to collect a corpus in order to train COMMANDS ROBOT and evaluate the system . (remote from • How would users naturally Commander) collaborate with this robot teammate? (Marge et al., 2016, IEEE RO-MAN) 15 15
Multi-floor data collection setting • Robot assisted by two human Human Commander “wizards” • Dialogue Manager (DM) is the VIEWS language “brain” of the robot • Robot Navigator (RN) moves robot based on instructions VERBAL DM-WIZARD RN COMMANDS MOVES ROBOT “Behind Robot Navigator the scenes” 17 17
Example Interaction 18 18
Commander Commander – Human Participant • Verbally Instructs a Robot • Sees text message responses, LIDAR map, and images sent from onboard robot 19 19
Wizard #1 – Dialogue Manager Dialogue Manager Wizard (DM-Wizard, DM) • Handles all language functions of “robot” • Responds to CMD and robot navigator (RN) via text message • Serves as mediator between RN and CMD 20 20
Wizard #2 – Robot Navigator move Robot Navigator Wizard (RN-Wizard, RN) • Handles all navigation function of “robot” • Constrained language received -> joysticks robot • Separation of wizards: • reduces cognitive load/wizard labor • removes intuition of interpreting commands 21 21
Example Interaction Proceed forward 22 22
Example Interaction How far? You can tell me to move to an object that you see, or a distance 23 23
Example Interaction Proceed forward three feet 24 24
Example Interaction Executing… 25 25
Example Interaction move forward three feet 26 26
Example Interaction move *moves robot forward 3 feet* 27 27
Example Interaction done 28 28
Example Interaction done 29 29
Data - Transcripts ▪ Time aligned Commander DM->Commander DM->RN RN (Audio Stream 1) (Chat Room 1) (Chat Room 2) (Audio Stream 2) transcripts of 4 data face the doorway on your right streams and take a picture there’s a door ▪ 2 audio streams ahead of me on the right and one just ▪ CMD and RN behind me on the right. which would ▪ 2 text streams you like me to face? the door ahead of ▪ DM->CMD, DM->RN you on the right move to face the ▪ Two conversational door ahead of you on the right, floors present image executing... image sent sent 30 30
Left floor: CMD, DM Commander DM->Commander DM->RN RN Commander (Audio Stream 1) (Chat Room 1) (Chat Room 2) (Audio Stream 2) Participant face the doorway on your right and take a picture VIEWS there’s a door ahead of me on the right and one just behind me on the right. which would you like me to face? the door ahead of VERBAL DM-WIZARD you on the right RN COMMANDS MOVES move to face the door ahead of ROBOT you on the right, image executing... image sent “ Behind sent Robot Navigator the scenes ” 32 32
Right Floor: DM, RN Commander DM->Commander DM->RN RN Commander (Audio Stream 1) (Chat Room 1) (Chat Room 2) (Audio Stream 2) Participant face the doorway on your right and take a picture VIEWS there’s a door ahead of me on the right and one just behind me on the right. which would you like me to face? the door ahead of VERBAL DM-WIZARD you on the right RN COMMANDS MOVES move to face the door ahead of ROBOT you on the right, image executing... image sent “ Behind sent Robot Navigator the scenes ” 33 33
DM translates (to) left and right Commander DM->Commander DM->RN RN Commander (Audio Stream 1) (Chat Room 1) (Chat Room 2) (Audio Stream 2) Participant face the doorway on your right and take a picture VIEWS there’s a door ahead of me on the right and one just behind me on the right. which would you like me to face? the door ahead of VERBAL DM-WIZARD you on the right RN COMMANDS MOVES move to face the door ahead of ROBOT you on the right, image executing... image sent “ Behind sent Robot Navigator the scenes ” 34 34
Recommend
More recommend
