Outline 1. Introduction 1. Motivation 2. Portuguese Sign Language - PowerPoint PPT Presentation

Outline 1. Introduction 1. Motivation 2. Portuguese Sign Language 3. Goals 2. Architecture 1. Gloss Translate 2. Lookup 3. Animate 3. Implementation 4. Evaluation 5. Conclusions 6. Demo

Motivation the system would be helpful in improving communication and reducing the isolation of the deaf community higher rate of illiteracy and learning problems hard to access content, aggravated by the rise and speed of new technologies

Sign Languages and LGP

Knowledge Sources

Different Sentence Structure OSV (Object-Subject-Verb) vs. SVO (Subject-Verb-Object) strong topicalization

Different Grammar classes No distinction between nouns and adjectives Articles? (o, os,.. um, uns) inflexion: gender, number, aspect, time... spatial agreement

Gesture Subunits gestures are built from parameters hand configuration hand orientation hand placement hand movement non-manual

Writing Systems

Writing Systems

Goals basic architecture and implementation that others can build upon study existing research and list open problems

Architecture

Gloss Translate Statistical Machine Translation language model bilingual lexicon alignment model Rule-Based Machine Translation lexicon and bilingual dictionary rules

Gloss Translate Rule-Based Machine Translation pipeline

Lookup

Animate - Rig Inverse Kinematics Definition of spatial marks Facial Controls and Mouth Shapes

Animate - Procedural Gestures

Animate - Avatar variance

Animate - Procedural Gestures inter- and intra-gesture blending transitions and timing acquiring data

Implementation

Blender

Blender Python Addon Interface Rendering

Translation and Natural Language Processing Python + NLTK - Natural Language Toolkit

Lookup JSON database

Rigify

Spatial marks - mesh

Spatial marks - grid

Base configurations 57 base configurations

Action Layering and Blending

Gesture building and Transitions

Gesture building and Transitions Distinction between: regular glosses numerals fingerspelling

Evaluation 6 deaf persons / interpreters + collecting feedback, observation in informal environment

Evaluation usefulness user interface translation quality avatar perceptibility

Conclusions definitely possible, definitely a challenge Sign Languages and LGP have particularities that pose additional problems more work from a multidisciplinary team is needed more linguistic studies

Contributions 1. summary of related work and background 2. overview of pipeline and architecture 3. open implementation as proof of concept 4. list of problems and future work

Contributions

Demo