Mobile Augmented Reality Tssio Knop de Castro Visgraf Lab, IMPA, - PowerPoint PPT Presentation

Mobile Augmented Reality Tássio Knop de Castro Visgraf Lab, IMPA, Rio de Janeiro 2011

Intro – Mobile Augmented Reality AR Toolkit Nearest Tube App ● Enhance the perception of the real world

Motivation Trip Advisor ● How to provide useful information without confusing the user? ● Is it possible to really insert a virtual object onto a real scene in a realistic way?

Goal To explore the possibilities and limitations of Mobile Augmented Reality concerning information insertion and objects rendering

Mobile Development

The device: Nokia N900 ● 600 MHz ARM processor ● 256 MB RAM ● 5MP programmable camera

Maemo Development ● MADDE (Maemo Application Development and Debugging Environment) ● Cross-compiling ● Multi-platform support (Linux (32-bit/64-bit), Windows, Mac OS X) ● Comes with Nokia Qt SDK =) ● It's a technology preview =( ● Scratchbox

Maemo Development ● Main challenges: ● Small community, poor documentation ● How to compile and configure external libraries for the N900 using this Qt-based environment? (however, Qt is a good tool to create interfaces)

AR Toolkit Plus ● Marker-based augmented reality: ● Given a physical marker, this toolkit is able to compute the camera position and orientation in real time ● Hundreds of markers templates ● There are marker-less methods as well (more expensive) ● We just need a good transformation matrix.

AR Toolkit Plus ● Main challenges: ● It's an external library: really tricky on MADDE ● No longer updated since June 2006

Coding in OpengGL ES 2.0 ● OpenGL ES 2.0 ● Programmable Shaders (GLSL) ● No fixed function pipeline

OpenGL ES 2.0

Overview Objects Setup VBO Setup Texture Detect Markers, Capture estimate Render! (GStreamer) transf. matrix (ARToolkitPlus) Compile and link shaders

GStreamer Powerful pipeline-based multimedia framework ●

Experiments ● 2D Text ● Texture Overlay ● 3D Text ● Shadow Mapping 2D Text

Texture Overlay

3D text ● More immersive than the 2D approach ● Array of vertices ● Simple shader Video: 3D text

Shadow Mapping Lance Williams, Casting curved shadows on curved surfaces. ● SIGGRAPH Comput. Graph. 12, 3 (August 1978) – objects light camera

Shadow Mapping Render objects from Light POV into a buffer ● This buffer shows which regions are illuminated ● objects light camera

Shadow Mapping Render from the Camera POV using the computed depth buffer ● The (x,y,z) coordinates which aren't visible from light POV are in shadow. ● objects light camera

Shadow Mapping Video: Shadow Mapping

Main Challenges ● Tracking stability ● Limited memory and speed ● Interface-related: one hand is always holding the device ● Small community ● OpenGL ES 2.0 and GLSL sintax

Future Work ● Improve shadow mapping and 2D text ● Experiment with arbitrary complex meshes ● Acquire light position in real time ● Render soft shadows ● Marker-less AR

Special Thanks ● Luiz Velho ● Djalma Lucio ● Aldo Zang ● Leonardo Carvalho

References ● Tangible Interfaces using Handheld Augmented Reality. P. Rojtberg and A. Olwal ● OpenGL ES specs: http://www.khronos.org/opengles/spec/ ● Understanding Virtual Reality. William R. Sherman and Alan Craig

Thank you -------- tknop@impa.br www.impa.br/~tknop