Social Signal Processing: Understanding Nonverbal Communication in Social Interactions Alessandro Vinciarelli 1 , 2 and Fabio Valente 2 1 University of Glasgow - Sir A.Williams Bldg, G12 8QQ (UK) 2 IDIAP Research Institute - CP592 Martigny (Switzerland) e-mail: vincia@dcs.gla.ac.uk, fvalente@idiap.ch
Outline • Part I - What is SSP? Slide 2 of 25
Outline • Part I - What is SSP? • Nonverbal behavior Slide 2 of 25
Outline • Part I - What is SSP? • Nonverbal behavior • Machine analysis of social behavior Slide 2 of 25
Outline • Part I - What is SSP? • Nonverbal behavior • Machine analysis of social behavior • Part II - SSP in Action Slide 2 of 25
Outline • Part I - What is SSP? • Nonverbal behavior • Machine analysis of social behavior • Part II - SSP in Action • Analyzing conversations Slide 2 of 25
Outline • Part I - What is SSP? • Nonverbal behavior • Machine analysis of social behavior • Part II - SSP in Action • Analyzing conversations • Roles, groups, stories and conflicts Slide 2 of 25
Outline • Part I - What is SSP? • Nonverbal behavior • Machine analysis of social behavior • Part II - SSP in Action • Analyzing conversations • Roles, groups, stories and conflicts • Part III - Future Perspectives Slide 2 of 25
Outline • Part I - What is SSP? • Nonverbal behavior • Machine analysis of social behavior • Part II - SSP in Action • Analyzing conversations • Roles, groups, stories and conflicts • Part III - Future Perspectives • The SSPNet Slide 2 of 25
Outline • Part I - What is SSP? • Nonverbal behavior • Machine analysis of social behavior • Part II - SSP in Action • Analyzing conversations • Roles, groups, stories and conflicts • Part III - Future Perspectives • The SSPNet • Challenges ahead Slide 2 of 25
Part I - What is SSP? Slide 3 of 25
Social Signals and Social Behaviour Our attention focuses on words, but we are immersed in a rich non-verbal world influencing not only the meaning of words, but also our perception of the social context. Slide 4 of 25
Social Signals and Social Behaviour mutual vocal gaze height behaviour forward posture forward posture Nonverbal Behavioural Cues interpersonal gesture distance Our attention focuses on words, but we are immersed in a rich non-verbal world influencing not only the meaning of words, but also our perception of the social context. Slide 5 of 25
Social Signals and Social Behaviour mutual vocal gaze height behaviour forward Social Signal posture forward posture Nonverbal Behavioural Cues interpersonal gesture distance Our attention focuses on words, but we are immersed in a rich non-verbal world influencing not only the meaning of words, but also our perception of the social context. Slide 6 of 25
Nonverbal Communication Nonverbal communications is based on nonverbal behavioural cues, codes, and functions. Slide 7 of 25
Nonverbal Communication Behavioural cues clothes, attractiveness somatotype, etc. self � touching postural congruence, etc. facial expression gaze behaviour, etc. prosody, pitch, rythm, etc. distamce, seating Nonverbal communications is based on nonverbal behavioural cues, codes, and functions. Slide 8 of 25
Nonverbal Communication Codes Behavioural cues Physical Appearance clothes, attractiveness somatotype, etc. Gestures self � touching Postures postural congruence, etc. Face and Eyes facial expression gaze behaviour, etc. Behaviour prosody, pitch, rythm, etc. Vocal Behaviour distamce, seating Space Environment Nonverbal communications is based on nonverbal behavioural cues, codes, and functions. Slide 9 of 25
Nonverbal Communication Codes Behavioural Functions cues Physical Appearance clothes, attractiveness forming impressions somatotype, etc. expressing emotion Gestures self � touching Postures postural congruence, etc. sending relational messages Face and Eyes facial expression gaze behaviour, etc. Behaviour managing interaction prosody, pitch, sending messages of rythm, etc. Vocal power and persasion Behaviour deceiving and distamce, seating detecting deception Space Environment Nonverbal communications is based on nonverbal behavioural cues, codes, and functions. Slide 10 of 25
Social Signal Processing Preprocessing Multimodal Data Person Behavioural Capture Detection Streams Raw Data Social Interaction Analysis Multimodal Behavioural Social Signals Social Behavioural Cues Understanding Behaviours Extraction Behavioural Streams Cues Context Understanding - A.Pentland, “ Social Signal Processing ”, IEEE Signal Processing Magazine, 24(4):108-111, 2007. - A.Vinciarelli, M.Pantic, H.Bourlard, “ Social Signal Processing: Survey of an Emerging Domain ”, Image and Vision Computing, 27(12):1743-1759, 2009. Slide 11 of 25
Part II - SSP in Action Slide 12 of 25
A Conversation Analysis Framework [...] the most widely used analytic approach is based on an analogy with the workings of the market economy. In this market there is a scarce commodity called the floor which can be defined as the right to speak. Having control of this scarce commodity is called a turn. In any situation where control is not fixed in advance, anyone can attempt to get control. This is called turn-taking. G.Yule,“ Pragmatics ”, Oxford University Press (1996) Slide 13 of 25
Turn-Taking s =a 1 s =a s 3 =a 1 s 4 =a 3 s 5 =a 2 s 6 =a 1 s 7 =a 2 1 2 3 t � 1 t � t � 3 t � 4 t � t � t � 7 t 2 5 6 The turn taking pattern can be represented with a sequence S = { ( s 1 , ∆ t 1 , ) , . . . , ( s T , ∆ t T ) } where s i ∈ A = { a 1 , . . . , a G } is a person label, and ∆ t i the length of the i th turn. • Among the most robustly detectable behavior evidences, • but how far can we go by just modeling who talks when and how much? Slide 14 of 25
Role Recognition in Broadcast Material • People play functional roles (Anchorman, Guest, etc.) • Adjacent speakers are supposed to interact • Around 85% of data time correctly labeled in terms of role A.Vinciarelli, IEEE T-Multimedia, 9(6):1215-1226 (2007) H.Salamin et al., IEEE T-Multimedia, 11(7):1373-1380, 2009 Slide 15 of 25
Role Recognition in Meetings events w 1 w 2 w 3 w 4 a 1 a a 3 2 actors s =a 1 s =a s 3 =a 1 s 4 =a 3 s 5 =a 2 s 6 =a 1 s 7 =a 2 1 2 3 t � 1 t � t � 3 t � 4 t � t � t � 7 t 2 5 6 w 1 w 2 w 3 w 4 t x 1 = (1,1,1,1) x 2 = (0,0,1,1) x 3 = (1,1,1,0) • Social networks suitable for functional roles, lexical analysis suitbale for semantic ones • Affiliation networks are suitable for small groups • Around 75% of data time correctly labeled in terms of role N.Garg et al., Proc. of ACM-Multimedia, pp. 693-696 (2008) Slide 16 of 25
Roles and Prosody s =a 1 s =a s 3 =a 1 s 4 =a 3 s 5 =a 2 s 6 =a 1 s 7 =a 2 1 2 3 t � 1 t � t � 3 t � 4 t � t � t � 7 t 2 5 6 R ∗ = arg max R ∈R p ( R | X , � α ) • Conditional Random Fields allow the combination of turn-taking and prosody • Entropy of main prosodic features • Results up to 89%, but combination does not always lead to significant improvements H.Salamin et al., Proc. of ACM-Multimedia, to appear (2010) Slide 17 of 25
Story Segmentation Story 1 Story 2 Story 3 s =a 1 s =a s 3 =a 1 s 4 =a 3 s 5 =a 2 s 6 =a 1 s 7 =a 2 1 2 3 t � 1 t � t � 3 t � 4 t � t � t � 7 t 2 5 6 The problem consists of finding the following story sequence ˆ H : ˆ H = arg max H ∈H p ( X | H ) p ( H ) (1) • The purity is around 0.75 • Longer stories are better recognized A.Vinciarelli et al., Proc. of ACM-Multimedia, pp. 261-264 (2007) Slide 18 of 25
Conflict Analysis s =a 1 s =a s 3 =a 1 s 4 =a 3 s 5 =a 2 s 6 =a 1 s 7 =a 2 1 2 3 t t t t t t t t � 1 � � 3 � 4 � � � 7 2 5 6 N ˆ � Q = arg max Q ∈Q π q 1 p ( q n | q n − 1 ) (2) n =2 where q i is one of the two groups or the moderator. • People tend to react to someone they disagree with • Groups correctly reconstructued in 66% of the cases (random grouping has an average 6 . 5% performance). A.Vinciarelli, IEEE Signal Processing Magazine, 26(5):133-136, 2009 Slide 19 of 25
Part III - Challenges Ahead Slide 20 of 25
Open Issues and Challenges • Getting psychology and engineering closer Slide 21 of 25
Open Issues and Challenges • Getting psychology and engineering closer • SSP is inherently multidisciplinary Slide 21 of 25
Open Issues and Challenges • Getting psychology and engineering closer • SSP is inherently multidisciplinary • Mutual efforts of both disciplines Slide 21 of 25
Open Issues and Challenges • Getting psychology and engineering closer • SSP is inherently multidisciplinary • Mutual efforts of both disciplines • Applying multimodal approaches Slide 21 of 25
Open Issues and Challenges • Getting psychology and engineering closer • SSP is inherently multidisciplinary • Mutual efforts of both disciplines • Applying multimodal approaches • Social signals are, by evolution, ambiguous Slide 21 of 25
Recommend
More recommend
