Autonomous Agents and Avatars in REVERIEs Virtual Environment Fons - PowerPoint PPT Presentation

Autonomous Agents and Avatars in REVERIE’s Virtual Environment Fons Kuijk (CWI), 
 Konstantinos C. Apostolakis, Petros Daras (CERTH), 
 Brian Ravenet (TPT), 
 Haolin Wei, David S. Monaghan (DCU) REVERIE is a Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

Siege of Haarlem (1572) Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

Teylers Museum (founded in 1784) Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

The Hiding Place (1940-1945) Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

Frans Hals Museum Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

These are facts and stories told on educational trips done in 1966-1967 Educational trips are not easy to organize In REVERIE, an ambient, content-centric Internet- based immersive environment, people can work, meet, participate in live events, socialize and share experiences as they do in real life REVERIE can be used for virtual educational trips (and for a 3D hangout and for Simon Says, and for … ) Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

An educational trip to the European Parliament Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

An educational trip to a virtual gallery Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

Objectives of REVERIE’s Virtual Environment • No need for fancy equipment. • Users free to move and look around. • Spatial audio. • Virtual humans have natural behaviour: • gaze • movements • gestures • social interaction • User driven and autonomous virtual characters plausibly interact. Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

REVERIE Architecture Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

User Analysis WHAT HOW Facial expressions for User data captured by: • webcam Puppeting (face) • Arousal & Valence • (Kinect) • Agree or Disagree • microphone • Attention • Body gestures for • Puppeting (body) • Navigation Speech for • Simple dialog Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

REVERIE Architecture Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

Reasoning Framework for Autonomy Capturing Gaze & Attention Detection Rules & User State Emotion & Affect Attitude Identi fi cation Avatar Behavior Proposer Keyword Spotting Gaze & Attention Facts from the Agent & Dialogue State Behavior Assesment World Group Affect & Social State Assessment Agent Behavior Proposer World State Obstacle Avoidance & Path Finding Legend Behavior Selection Script Proposer Planner & Realizer Scripts Analyzers Interpreters Proposers Navigation Animation Speech Realizers States Rendering Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

Configurable Clients Gaze Gaze Gaze Gaze Gaze Gaze Gaze Animation Reasoning Reasoning Reasoning Reasoning Reasoning Reasoning Animation Animation Animation Animation Animation Animation Animation Analysis Analysis Analysis Analysis Analysis Analysis Analysis Engine Framework Framework Framework Framework Framework Framework Engine Engine Engine Engine Engine Engine Engine Pub/Sub Pub/Sub Pub/Sub Pub/Sub Pub/Sub Pub/Sub Face Face Face Face Face Face Analysis Analysis Analysis Analysis Analysis Analysis Body Body Body Body Body Global Global Global Global Global Global Analysis Analysis Analysis Analysis Analysis Parameters Parameters Parameters Parameters Parameters Parameters Render Navigation Navigation Navigation Navigation Navigation Navigation Render Render Render Render Render Render Render Speech Speech Speech Speech module Server Server Server Server Server Server module module module module module module module Analysis Analysis Analysis Analysis User Client Reasoning & Reasoning & Reasoning & Reasoning & Reasoning & User Client User Client User Client User Client Navigation Server Navigation Server Navigation Server Navigation Server Navigation Server User Client User Client User Client User Client User Client User Client User Client WebSockets Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

Reasoning in REVERIE Comes in levels: Agents: fully autonomous Socio-emotional Nonverbal signals ! attitude Behaviour Planner (e.g. head upward, smile) & Realizer Intention Avatars: system adds autonomy where needed • gaze • gestures • pose based on user interaction. Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

Autonomous Gaze Attentive: Not attentive: (may lead to agent reaction) Following user camera: Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

Autonomous Gesture Request to speak Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

Options for Navigation User controlled 
 (small incremental steps): • Keyboard • Mouse • Kinect Autonomous (path planning): • Map • Follow Me Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

Conclusion • Agents react on users’ activity. • User control over avatar is configurable (the reasoning adds autonomy where needed). • Agents and avatars have human-like behavior. • Users feel being immersed. • REVERIE components • can be deployed on regular computer system; • are well suited for web-based communication. Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments

More detailed information: http://www.reveriefp7.eu/ Real and Virtual Engagement in Realistic Immersive Environments Web3D 2015 Real and Virtual Engagement in Realistic Immersive Environments