A plan-based approach for affective sports commentary in real-time - PowerPoint PPT Presentation

Embodied Presentation Teams: A plan-based approach for affective sports commentary in real-time Author: Ivan Gregor Supervisor: Michael Kipp

Virtual Agents • Multimodal user interfaces • Entertaining, Enjoyable • Example applications – computer games – tutoring systems – commentary agents 2

GALA Challenge • Commentary on a continuous sports event – Complex behaviour – Affective – Real-time • Horse race • Tennis game GALA is a part of the IVA (Intelligent Virtual Agents) conference 3

GALA 2009 - Input • Video & ANVIL file • Timestamped events Position side Position long. Position lateral Position height server net left low receiver mid court middle middle Player events Ball events baseline right high throw shot serve cross net forehand hit_ net backhand hit _tape forehand-volley bounce Backhand volley fault smash out miss 4

Related Work - ERIC • Won GALA 2007 • Horse race reporter • Rule-based • Template-based NLG • Domain independent – horse race – tank battle game 5

Related Work - Spectators • GALA 2009 • Small set of rules • No commentary 6

Related Work - STEVE • Tutoring system • HTN planning • Virtual environment • User questions • No emotions 7

Presentation Teams (André, Rist) • Generate presentations on the fly • Choice of a presentation team • 2 ≤ distinct virtual agents • Roles (expertise) • Interest • Personality profiles • Dialogues • Opposing roles (entertaining, understanding) 8

Our system – Aims (1) • Behaviourally complex, affective commentary on a tennis game (GALA 2009) • Real-time • Presentation Team – Different roles – Attitudes to the players – Personality profiles 9

Our system – Aims (2) • Dialogue planning (HTN planning) • Interruptions • Interaction (user pre-defined questions) • Background knowledge 10

Video – Features of the system 11

Dialogue Planning INPUT: -Facts describing the current state of the tennis game -Commentators` attitudes to the players -Background facts HTN Planner (JSHOP) OUTPUT: possible plans (dialogues) 12

Dialogue Schemes (1) Dialogue Scheme Example of a Generated Dialogue A: argument for/against X A: “That serve was really phenomenal!“ B: contrary B: “Well, that is a little exaggerated! “ A: argument for/against X A: “ Blake is in great shape as usual. “ B: contrary B: “ But he already produced several unforced errors. “ A: override A: “ Still, he is the best player on the court. “ A: argue for X A: “ Excellent return by Safin .“ B: elaborate on X B: “Unreachable for Blake.“ A: background fact X A: “The brother of Blake Thomas is a well known player.“ B: evidence of X B: “His best ranking was the 141st place in 2002." A: background fact X A: “ Roddick has been 4 times injured recently.“ B: consequence of X B: “It will be hard to break through today.“ Dialogue schemes were introduced by Elisabeth Andr é and Thomas Rist 13

Dialogue Schemes (2) 14

Planning Large Dialogue Contributions 15

Planning Tree (1) • Hierarchy of dialogues • Root = goal task • Internal node = compound task • Leaf = primitive task or reference to an internal node 16

Planning Tree (2) 17

Affect • Lexical selection – Choice of a dialogue scheme • Gestures – Utterances with the gesture annotation • Facial expression – Emotion Module 18

Affect – Planning with Attitude • Lexical selection (Choice of a dialogue scheme) • Gestures (Gesture tags in utterances) Appraisal Dialogue Scheme Example of a Generated Dialogue A: positive A: argue for X A: “Outstanding ace by Blake!” B: positive B: support X B: “Blake hits blistering serve down the line!” A: positive A: argue for X A: ”Excellent forehand by Safin!” B: negative B: play down X B: “That`s a bit overstated. ” A: negative A: point out fault X A: ”Safin failed to get the ball over the net.” B: positive B: excuse X B: “Safin just overhits the serve. ” A: neutral A: convey fact X A: “The score is already 30:0.” B: negative B: consequence of X B: “ Safin and Ferrer are real losers as usual!” A: neutral A: convey fact X A: “Deuce again.” B: neutral B: elaborate on fact X B: “ Safin and Ferrer got back on board.” 19

Affect – OCC Generated Emotions (1) • Facial Expression – Emotion module (Jess) – Simulate 8 basic OCC emotions OCC Emotion Description JOY Something happened that I wanted to happen. DISTRESS Something happened that I did not want to happen. HOPE Something may happen that I really want to occur. FEAR Something may happen that I wish to never occur. RELIEF Something bad did not happen. DISAPPOINTMENT Something did not happen that I really wanted to occur. SATISFACTION Something happened that I really wanted to occur. FEAR-CONFIRMED Something bad did actually happen. OCC (Ortony, Collins, Clore) Cognitive model of emotions 20

Affect – OCC Generated Emotions (2) • Initialization: personality Personality Trait • Input: Facts + Attitudes Optimistic Choleric • Functionality Facts from the Extravert tennis game Neurotic – EEC & Social Attitudes – Initial intensity – Emotion decay EEC definitions Emotion + Initial Intensity (Jess) • Output: Vector of emotion intensities 21

Background Knowledge • CSV files with background facts about – players – tournaments Background knowledge Examples of deduced fact Player`s details A sister of a player is also a tennis professional. Ranking A player is leading the ATP score. Style A player is playing risky as usual. Injury A player has been four times injured recently. Player`s results A player won two matches in a row. Tournaments details The tournament is played in London on grass. 22

System Architecture FSM JSHOP Jess 23

Deduction of High-level facts Discourse Planner Event Manager Tennis Simulator 24

Mapping: plan → utterances Plan: a list of plan operators Operator head: Operator (variables) briskly_returned_serve ?server ?receiver ?receiver_shot Template: Template (slots) briskly_returned_serve ?server ?receiver ?receiver_shot contains 1-3 utterances: Annotated 1.) {EmotionSurprise} {Play} ?receiver generated a ?receiver_shot utterance {Look} return that was out of ?server's reach. 2.) … Command to the substitution: Avatar Engine ?server := Safin {EmotionSurprise} := $(Emotion,surprise,0.9,500,1000,3000) ?receiver := Federer {Look} := $(Motion,presentation/look/lookto_right02,400,500,0,1200,0.8) ?receiver_shot := forehand {Play} := $(Motion,interaction/bye/bye01,400,500,0,10000,1.5) 25

Video – Commentary Excerpt 26

28

Conclusion • Two virtual agents engaged in dialogues to comment on a tennis game given as GALA 2009 • Affect – Lexical selection – Facial expression – Gestures (synchronized with the speech) • Customized commentary – Commentators` attitudes to the players – User question (more engaging) • Background Knowledge 28

Future Work • EMBR (A Realtime Animation Engine for Interactive Embodied Agents) EMBR • Prosody module • Base dialogue schemes on OCC Emotions • Dynamic Replanning • Other domains – tutoring systems – tourist guides – guide for the blind 29

QUESTIONS Thankyou to the EMBOTS group, DFKI, and Charamel GmbH 30