Sign Language Avatars Animation and Comprehensibility Michael Kipp* - PowerPoint PPT Presentation

Sign Language Avatars Animation and Comprehensibility Michael Kipp* Alexis Heloir Quan Nguyen DFKI Embodied Agents Research Group Exzellenzcluster Multimodal Computing and Interaction Universität des Saarlandes * University of Applied Sciences Augsburg research funded by:

Sign language avatars... for the internet

Deaf Sign language is primary means of communication Sign language is a real language [Stokoe 1960] Specific SL for every country (ASL, DGS, LSF, BSL ...) 500,000

what is Deaf = your name ? Sign language as Sign language is a first language primary means of communication Spoken language is a foreign language Sign language is a real language [Stokoe 1960] 80% of deaf pupils Specific SL for leave school with every country significant reading/ (ASL, DGS, LSF, writing problems BSL ...) 500,000

inexpensive expensive editable not editable possibly interactive not interactive comprehension limited comprehensible 95% commercial

Prior Work • No standard writing system for sign language Glosses : based on meaning, tool for learning ➡ YOUR NAME WHAT Notation : based on form, tool for science ➡ (Stokoe notation, HamNoSys) • Milestones ViSiCAST (2000-2003): face-to-face translation // mocap ➡ eSIGN (2002-2004): internet // procedural animation SiGML ➡ 60% comprehensibility ➡ • Recent projects More flexible notations: ➡ Zebedee (LIMSI), PDTS-SiGML (U East Anglia) Avatars for American SL (Huenerfauth et al. // DePaul Univ.), ➡ Italian SL (ATLAS project), Czech SL (U West Bohemia) ...

What's your Agent's Native Language? Greta , U Paris 8 (2006) LIMSI, SNCF, web sourds (2006) SmartBody ICT (2004) GUIDO , eSIGN, Max Televirtual (2003) Paula EMBR , DFKI Elckerlyc GeSSyCa (2009) Marc ... ...

What's your Agent's Native Language? Greta , U Paris 8 - rich form vocabulary (2006) - speech-gesture sync. - validation by - lip syncing "understanding" - locomotion Universal LIMSI, SNCF, web sourds (2006) Communi- cators SmartBody ICT (2004) GUIDO , eSIGN, Max Televirtual (2003) Paula EMBR , DFKI - control language Elckerlyc GeSSyCa (2009) Marc - validating quality ... ...

Point of Departure • Goal: Make every ECA "sign language ready" • EMBR: EMB odied agent R ealizer ➡ open source ➡ own animation language EMBRScript • Comprehensiblity?

Toward "sign language ready" • Hand shapes : 10 => 60+ (finger alphabet...)

Toward "sign language ready" • Hand shapes : 10 => 60+ (finger alphabet...) • Torso : lean/orientation, shoulder raises

Toward "sign language ready" • Hand shapes : 10 => 60+ (finger alphabet...) • Torso : lean/orientation, shoulder raises • Facial expression : higher amplitude • Mouth : sophisticated viseme set, should allow lipreading, use text-to-speech for visemes

Toward "sign language ready" • Hand shapes : 10 => 60+ (finger alphabet...) • Torso : lean/orientation, shoulder raises • Facial expression : higher amplitude • Mouth : sophisticated viseme set, should allow lipreading, use text-to-speech for visemes • Gaze : separate eye-ball from head movement

Toward "sign language ready" • Hand shapes : 10 => 60+ (finger alphabet...) • Torso : lean/orientation, shoulder raises • Facial expression : higher amplitude • Mouth : sophisticated viseme set, should allow lipreading, use text-to-speech for visemes • Gaze : separate eye-ball from head movement

Animating Sign Language: Attempt I • Video: human signer's utterance • Imitate utterance using EMBRScript • Show EMBR animation

Failed!

Reasons • Ambiguity in sign language fewer grammatical constructs ➡ • Single sign level formational manual features ➡ situational nonmanual features (almost impossible) ➡ mouthing especially important in German SL ➡ • Utterance level facial expression for sentence mode ➡ eyebrows + posture for information structure ➡ face as a visual focus point ➡ • Casual signing style makes sign harder to read human signers compensate with all of the above ➡

Consequences • Working hypothesis: Avatars with current animation methods are unable to produce understandable "spontaneous" sign language • Therefore: ➡ Overarticulate ➡ Involve Deaf experts ➡ Focus on nonmanual features ➡ Consider random facial movement

Original Overarticulated Remake Avatar

Attempt II • Overarticulated remake ➡ transcribe glosses ➡ recording • Gloss-based animation (lexicalized) ➡ compatible with EMBRScript ➡ tool support ➡ implications for HamNoSys

Video HamNoSys Animation

Video HamNoSys Animation BML

Video HamNoSys Animation BML Heloir, Kipp 2010 Kipp et al. 2010 Video BML EMBRScript Animation

Video HamNoSys Animation BML Heloir, Kipp 2010 Kipp et al. 2010 Video BML EMBRScript Animation HamNoSys

Video HamNoSys Animation BML Heloir, Kipp 2010 Kipp et al. 2010 Video EMBRScript Animation

Sample utterance: YOUR NAME WHAT gloss = sequence many pose sequence sequences pose single pose pose utterance pose pose pose pose seq. seq. seq. pose YOUR NAME WHAT

Evaluation • Corpus 11 utterances (154 glosses) from German Deaf e-learning portal ➡ quite complex sentences ➡ • Animation higher duration for remake (factor 1.8) and for animation (factor 2.3) ➡ gloss reuse factor = 1.6 (95 gloss lexemes) ➡ • Experiment 13 Deaf test subjects (6m / 7f), aged 33-55 ➡ Each session 1.5 - 2 hrs (videotaped) ➡ Pure sign language environment: Deaf assistant, use of pictograms ➡ Warm-up: 3 easy avatar sentence ➡

Delta Testing

Analysis • Analysis of videos by Deaf experts ➡ Subjects' own rating usually misleading [Huenerfauth et al. 2008] • Objective measure: count correctly recalled glosses ➡ only partial understanding? • Subjective measure: expert rates understanding for each utterance • Combine measures [Sheard et al. 2004]

Results avatar / absolute: avatar / relative: 58.4 % 41.4 %

Discussion • Comprehensibility ➡ original video = 71 % "shockingly low" ➡ overarticulated remake = 82 % ➡ avatar = 58.4 % => close to state of the art • Novel aspects: ➡ complex content ➡ direct comparison with human signers • Delta testing factors out difficulties inherent to the material (dialect, speed, bad grammar) ➡ focus on the real "delta" between avatar and human

Conclusions Thanks! • How to make an ECA sign! ➡ EMBRScript as an interface language ➡ SL synthesis research: nonmanuals and prosody ➡ Delta testing for comprehensibility • Signing avatars can profit from ECAs, and vice versa • 2nd workshop on Sign Language Translation and Avatar Technology @ACM ASSETS 2011 Dundee ! First workshop, Berlin, January 2011 Thanks to: Peter Schaar Iris König Silke Matthes Thomas Hanke