Uncertainty in Spoken Uncertainty in Spoken Multimodal - speakers - PDF document

Baltic HLT Oct 8 2010 Natural Interaction Natural Interaction Goal oriented Uncertainty in Spoken Uncertainty in Spoken Multimodal - speakers have intentions - speech, gesture, - task Dialogue Management face, posture Dialogue Management Affective - affinity, social bonds Kristiina Jokinen Kristiina Jokinen - emotions, attitudes University of Tartu University of Tartu and and Rational agents University of Helsinki University of Helsinki - coordination of action Symbolic - cooperation - learnt conventions - grounding = build - communicative signs, words shared knowledge Constructive Dialogue Dialogue Modelling Modelling Constructive Hesitation/Uncertainty Situations � The agents monitor each other and each other The agents monitor each other and each other’ ’s actions in the s actions in the � Jokinen & Allwood (2010) communicative situation and react to the situation according their ir communicative situation and react to the situation according the beliefs, intentions and interpretation of the situation, to buil beliefs, intentions and interpretation of the situation, to build shared d shared � lack of own ability to continue knowledge and achieve a goal (Jokinen, 2009) knowledge and achieve a goal (Jokinen, 2009) � knowledge, skills � lack of permission to continue Contact Contact Hearing Hearing/seeing seeing/touching touching distance distance � situational issues Perception Perception Recognition Recognition of of meaningful meaningful symbols symbols � lack of willingness to continue Understanding Understanding Meaning Meaning creation creation for the for the symbols symbols in the in the � attitude context context Reaction Reaction Production Production of of one one’s own own behaviour behaviour Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen K. Jokinen 1

Baltic HLT Oct 8 2010 Examples Hesitation-related phenomena it’s just a… � Fillers: filled pauses, discourse markers, editing terms, parentheticals � Disfluencies � Self-corrections (repairs) Make a distance…. probably …. I don’t know � Retractions: reformulation or restart of one’s utterance � Non-verbal aspects: hand gestures, body posture Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen Multi- -level level Hybrid Hybrid Method Method Perception of hesitation signals Multi � Top Top- -down down analysis of the analysis of the collected collected data data � � Hesitation markers ( uhh, umm ) � at different meaning levels at different meaning levels � Pauses, silence • words words, , syntactic syntactic phrases phrases, , dialogue dialogue acts acts, , gestures gestures, , face face, , posture posture,… � different different tagging tagging schemes schemes: AMI, MUMIN, : AMI, MUMIN, standardisation standardisation � Slower speaking rate � manual annotation of what the human observe manual annotation of what the human observe � Higher pitch � Bottom � Bottom- -up analysis of the collected data up analysis of the collected data � Fundamental frequency F0 rises before pauses that � at different signal levels: at different signal levels: occur in major syntactic boundaries • speech, eye speech, eye-gaze, face, gesture recognition gaze, face, gesture recognition � Gesturing � different different technical technical constraints constraints: : accuracy accuracy � Compensatory pattern Carlson & Gustafson (2006) � Automatic Automatic annotation annotation of of what what ” ”happens happens” ” � F0 countours, pausing, retardation, creaky voice, syntax � Correlations � Correlations and and classifications classifications between between � the total duration increase counts rather than the contribution by � Crossing � Crossing points points syntactic or prosodic factors as such Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen K. Jokinen 2

Baltic HLT Oct 8 2010 Praat Analysis Analysis Praat Annotation – – human observations human observations Annotation - Speech, facial expressions, gestures, body posture - Using the Anvil annotation tool F0 variation pitch intensity Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen F0 for it it’ ’s s just a just a… … F0 of probably probably & & I I don don’ ’t t know know F0 for F0 of Fundamental frequency raises before pauses at major syntactic constructs, but lowers if a pause occurs in the middle Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen K. Jokinen 3

Baltic HLT Oct 8 2010 Face activity (blue peaks) correlates with Face activity (blue peaks) correlates with speech activity (green circles) and speech activity (green circles) and Conclusions Conclusions manual facial gesture manual facial gesture � Verbal expressions of hesitation and uncertainty 1 are accompanied by facial expressions and Face Activity 0.9 Facial Labels gesturing that help the partners to understand the Speech 0.8 underlying reasons for hesitation 0.7 � Situational and attitudinal hesitation seems to be accompanied by large, specific gesturing which 0.6 carries social conventions of symbolic gestures 0.5 � Hesitation and uncertainty also used in the 0.4 coordination of interaction and the activity that the 0.3 speakers are involved in 0.2 � Semantic theme of hesitation and uncertainty: non- 0.1 continuation of the current conversational topic 0 0 500 1000 1500 2000 2500 Thanks to Stefan Scherer Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen Holistic View View of of Interaction Interaction Future Work Work Holistic Future � Communicative signals are used as means � Relation between speech and gesture to manage social situation in which the parameters agents find themselves, as a reaction to � Temporal correlations conversational understanding and they form � Segmentation: what is the smallest unit communicative patterns rather than function � How the speakers learn to observe hesitation as individual signs of communication signals � What is the context (levels, activities, � Intercultural comparison: culturally accepted culture…) hesitation markers vs. interpretation of these � Cf. Compensatory pattern signals (e.g. shoulder shrug) � Deviations of the expected temporal patterns Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen K. Jokinen 4

Baltic HLT Oct 8 2010 Future Wiews Wiews Future Thank you you! ! Thank in the context context of of Baltic Baltic HLT HLT in the � � Natural Natural language language interaction interaction extended extended to the to the whole whole communication situation situation communication � Integrates Integrates language language and and speech speech technology technology � Corpus Corpus collection collection � � Speech Speech data: data: individual individual words words to to read read speech speech to to conversational conversational speech speech � What What type type of data, of data, activity activity, , what what kind kind of of equipments equipments � Analysis Analysis � � Annotation Annotation levels levels ( (tags tags for for dialogue dialogue acts acts, , gestures gestures, etc. on , etc. on pragmatic pragmatic level level) ) � Comparison Comparison of of interaction interaction strategies strategies � Possible Possible use use of of speech speech recognisers recognisers, , morphological morphological and and syntactic syntactic parsers parsers; ; also also face face and and gesture gesture recognition recognition Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen Oct 8 2010 Oct 8 2010 Baltic HLT / K.Jokinen Baltic HLT / K.Jokinen K. Jokinen 5