Steerable Interfaces for Steerable Interfaces for Pervasive - PowerPoint PPT Presentation

Steerable Interfaces for Steerable Interfaces for Pervasive Computing Spaces Pervasive Computing Spaces Gopal Pingali Pingali Gopal Gopal Pingali Claudio Pinhanez Pinhanez, Anthony , Anthony Levas Levas, Rick , Rick Kjeldsen Kjeldsen, Mark , Mark Claudio Claudio Pinhanez, Anthony Levas, Rick Kjeldsen, Mark Podlaseck, Han Chen, , Han Chen, Noi Sukaviriya Noi Sukaviriya Podlaseck Podlaseck, Han Chen, Noi Sukaviriya IBM T.J . Watson Research Center IBM T.J . Watson Research Center IBM T.J . Watson Research Center Hawthorne, New York Hawthorne, New York Hawthorne, New York USA USA USA

Talk Overview Talk Overview ! Motivation/definition of steerable interfaces ! Motivation/definition of steerable interfaces ! Overview of key technologies ! Overview of key technologies ! ! Demonstration of prototype smart retail environment Demonstration of prototype smart retail environment ! Conclusions and discussion ! Conclusions and discussion

Motivation Motivation Can a display appear wherever the user needs it rather than the user being enslaved to display devices (monitors, televisions etc.) ?

Steerable Interfaces for Pervasive Steerable Interfaces for Pervasive Computing Environments Computing Environments Steerable Interface: An interface to computing that can move around a physical environment to appear on ordinary objects or surfaces or even in empty space. Applications: Applications: Characteristics: Characteristics: ! Information access in public ! Information access in public ! Movable output interface ! Movable output interface spaces -- retail, transportation spaces -- retail, transportation ! Movable input interface ! Movable input interface ! Instruction and information ! Instruction and information ! Adaptation to user context ! Adaptation to user context delivery in military training delivery in military training ! Adaptation to environmental ! Adaptation to environmental centers, ships, aircraft centers, ships, aircraft context context ! Deviceless ! Deviceless interaction in interaction in ! Device-free interaction ! Device-free interaction hazardous environments hazardous environments ! Natural interaction ! Natural interaction ! Augmented reality in hospitals ! Augmented reality in hospitals ! Interactive real world games ! Interactive real world games

Steerable Interfaces -- Key Steerable Interfaces -- Key Technologies Technologies Environment modeling Steerable vision A Steerable projection P I WORLD MODEL (3D geometry, user location,display zones, Calibration calibration parameters, Steerable system state) microphones Steerable audio User Localization Geometric Reasoning

Steerable Projection: Steerable Projection: The Everywhere Display The Everywhere Display A device that creates an A device that creates an A device that creates an interactive projected interactive projected interactive projected display on any surface display on any surface display on any surface in an environment. in an environment. in an environment. Prototype Prototype Prototype bright LCD projector bright LCD projector bright LCD projector rotating mirror rotating mirror rotating mirror pan/ tilt/ zoom camera pan/ tilt/ zoom camera pan/ tilt/ zoom camera

Oblique Projection Distortion Oblique Projection Distortion display image

Correction of Oblique Projection Correction of Oblique Projection Distortion Distortion display image Goal: projected image is free of distortion when viewed by an “orthogonal” viewer

Projector/camera Calibration ! ! ! computation of homography computation of homography H/ H/Hv Hv given location of corners on a display surface given location of corners on a display surface computation of homography H/Hv given location of corners on a display surface ! ! ! need expected hand size in image and direction of entry for interaction need expected hand size in image and direction of entry for interaction need expected hand size in image and direction of entry for interaction ! ! ! manual specification of focus, pan, tilt, zoom manual specification of focus, pan, tilt, zoom manual specification of focus, pan, tilt, zoom

Integration into Office-of-the-Future: Integration into Office-of-the-Future: BlueSpace at IBM T.J at IBM T.J . Watson BlueSpace . Watson Ceiling Light s St at us Light s Task Everywhere Light s Primary Display Display Envir onment Sensors Of f iceFr ont Display Personal Air & Heat ing I nf oPanel Display Wir eless Pressur e Sensor

Scenarios in Bluespace Bluespace Scenarios in Team work support Personalized wall display Video conferencing anywhere Email notification on desk

Steerable Gesture Recognition Steerable Gesture Recognition for Input for Input Projected light changes appearance of hand! ! ! ! camera steered to projected display camera steered to projected display camera steered to projected display ! ! ! motion detected in camera image stream motion detected in camera image stream motion detected in camera image stream ! ! ! motion data matched to finger tip template motion data matched to finger tip template motion data matched to finger tip template

Surface Independent Interface Surface Independent Interface Specification Specification application space map to surface A map to surface B ! Independent specification of what and where ! Dynamic configurations of widgets; ! Map configuration to surface at run time; image space

Widget Architecture Widgets: Image Region Definition (IRD) Touch Button Tracking Area Change Event Generation (EG) Detection Surface Transformation (ST) Motion Detection Image IRD Touch Motion EG Detector ST Fingertip Tracking Return Value Component types: Transformation Image Processing In-Region IRD Motion Analysis Detection EG System Functions Instance Data ST Good high level abstraction - Efficient widget combination - Flexible widget definition

First Prototype of Steerable First Prototype of Steerable Interfaces (Siggraph Siggraph, Aug 2001) , Aug 2001) Interfaces ( Completed Interactive picture cans with M&Ms Color selection Invitation board at entrance Picture in progress: placement, finger painting

User Experience at Siggraph Siggraph 2001 2001 User Experience at SIGGRAPH01.avi

Interactive Applications Anywhere Interactive Applications Anywhere Various interactive applications can be dynamically moved to Various interactive applications can be dynamically moved to different surfaces e.g.. MediaPlayer, PowerPoint presentations, different surfaces e.g.. MediaPlayer, PowerPoint presentations, Web Browser applications etc. Web Browser applications etc.

User-following Display User-following Display Goal: Automatically steer display to appropriate surface at the Goal: Automatically steer display to appropriate surface at the Goal: Automatically steer display to appropriate surface at the appropriate position, size and orientation based on user location. appropriate position, size and orientation based on user location. appropriate position, size and orientation based on user location. Enabling Technologies: Enabling Technologies: 1. 3D model of the environment 1. 3D model of the environment 2. 3D calibration of projector(s), camera(s) 2. 3D calibration of projector(s), camera(s) 3. Dynamic localization of the user(s). 3. Dynamic localization of the user(s). 4. Geometric reasoning engine for: 4. Geometric reasoning engine for: virtual projection virtual projection virtual imaging virtual imaging virtual steering virtual steering occlusion detection occlusion detection computation of warping function computation of warping function

1. 3D Environment Modeling 1. 3D Environment Modeling VRML model of VRML model of environment: environment: ! Based on real world ! Based on real world measurements measurements

Modeling toolkit ! Basic 3D modeling software ! Basic 3D modeling software supporting simple geometries supporting simple geometries (planes, cubes) and import of (planes, cubes) and import of more complex geometries more complex geometries ! Provides annotations for all ! Provides annotations for all objects -- semantics can be objects -- semantics can be attached to objects attached to objects ! Includes “projectors”, “cameras”, ! Includes “projectors”, “cameras”, and “PJ surfaces” and “PJ surfaces” ! Model stored in XML format ! Model stored in XML format (objects are tags and annotations (objects are tags and annotations are attributes) are attributes)