VU Augmented Reality on Mobile Devices VU Augmented Reality on - PowerPoint PPT Presentation

VU Augmented Reality on Mobile Devices VU Augmented Reality on Mobile Devices  Introduction  Introduction – What is AR What is AR  Interaction Techniques  Navigation Collaboration Navigation, Collaboration  Visualization Techniques  Visual Coherence Visual Coherence  Tracking  … Includes material from Denis Kalkofen, TU Graz, , , Georg Klein Oxford, Jan Fischer 1

Augmented Graphics Augmented Graphics  Information  Virtual Objects  Visualization  Graphics  Understanding  Realism Understanding Realism 2

Topics Topics  Occlusion  Occlusion  Lighting Lighting  Camera effects Camera effects  Visual effects 3 3

Scene models Scene models  Computer graphics completely models real scene  Computer graphics completely models real scene  Geometry accessible Geometry accessible for calculations  Occlusions  Lighting Li hti  Shadows  Camera parameters 4 4

Augmented Reality Augmented Reality  Less information available  Final image through compositing iti  Digital in video see-through  Physical in optical see- through, projection  Approximate real world  Approximate real world to simulate interaction  geometrical, appearance  post processing 5 5

Occlusion Handling Occlusion Handling  Virtual in front of real  Draw augmentation on top of video background video background  Virtual behind real QuickTime™ and a  Need strategy to distinguish decompressor are needed to see this picture. visible from occluded i ibl f l d d augmentations 6

Phantom Rendering Phantom Rendering  Render registered virtual representations g p (Phantoms) of real objects  Occlusions handled by graphics hardware hi h d 1. Draw Video 2. Disable writing to color buffer (glColorMask or glBlendFunc(0,1)) 3 Render phantoms of 3. Render phantoms of real scene ➔ sets depth buffer 4. Enable writing to color buffer 5. Draw virtual objects 7 7

Phantom Rendering Phantom Rendering 8

Phantom Rendering Phantom Rendering  Polygonal data [Breen96 Fiala06]  Polygonal data [Breen96,Fiala06] 9

Phantom Rendering Phantom Rendering  Volume data  Volume data  Phantom = Isosurface using ’FirstHit Raycasting’ [Fischer04] 10

Problems of Phantom Rendering Problems of Phantom Rendering  Requires accurate  Requires accurate  Model  Tracking data  Registration R i t ti 11

Occlusion Refinement Occlusion Refinement  To overcome poor  To overcome poor  Model  Tracking  Registration R i t ti [DiVerdi06] 12

Occlusion Refinement Occlusion Refinement  Image based post processing  Image based post-processing [Kl i 05] [Klein05] [DiV [DiVerdi06] di06] 13

Occlusion Refinement Occlusion Refinement 14

Probabilistic Occlusion Handling Probabilistic Occlusion Handling  Soft transition between occluding and hidden objects  Soft transition between occluding and hidden objects  Compensate for tracking and registration error, by reducing the occluder’s transparency depending on g p y p g the probability of occlusion [Fuhrmann99] 15

Model Free Occlusion Handling Model Free Occlusion Handling  Instead of tracked & registered g phantom model  Construct depth map from video  from Computer Vision  Stereo, Shading, Structured Light etc.  Consider Performance Co s de e o a ce [PointGrey Bumblebee] [Fischer07] 16

Modell Free Occlusion Handling Modell Free Occlusion Handling  Initialize rendering of augmentations with depth map  Initialize rendering of augmentations with depth map from video [Fischer07] 17

Model Free Occlusion Handling Model Free Occlusion Handling  Using a priori g p knowledge about scene layout Foreground  Foreground and  Foreground and background objects are known  Online separation by  Online separation by foreground illumination [Grundhoefer07] 18

Model Free Occlusion Handling Model Free Occlusion Handling  Online separation by foreground illumination  Online separation by foreground illumination [Grundhoefer07] 19

Model Free Occlusion Handling Model Free Occlusion Handling  Using a priori knowledge g p g about scene layout  Hands are always in front and visible visible Hand Occlusion Hand Occlusion Hand Occlusion Hand Occlusion  Hand segmentation by using hand color [Sandor07] [Sandor07] 20

Semi-Automatic Model Free Occlusion Handling Handling  Manual a priori foreground segmentation  Manual a priori foreground segmentation  Boundary tracking Boundary Boundary [Lepetit00] 21

Lighting Lighting  Shadows  Shadows  Global illumination Global illumination  Interaction between real and virtual objects Interaction between real and virtual objects 22

Shadows Shadows  Important for  Important for  Depth perception  Scene interpretation  Relations  real ➔ real ea ea  real ➔ virtual  virtual ➔ real  virtual ➔ virtual virtual ➔ virtual [Haller03] [Haller03] 23

Scene model Scene model  Virtual objects Virtual objects  Impostors for pos o s o real objects  Light sources  real must be reflected in shading  real must be used to shadow casting 24

Drop shadow on plane Drop shadow on plane  Simple scene geometry  Simple scene geometry  Only virtual shadows 25

Shadow volume rendering Shadow volume rendering  Multi pass shadow volume  Multi-pass shadow volume 1. Draw real scene from video image 2. Draw virtual ➔ real shadows 3 D 3. Draw virtual objects i t l bj t 4. Draw real, virtual ➔ virtual objects 26

Illumination Illumination  Real scenes have complex illumination  Real scenes have complex illumination  More than 1 light source  Complex light sources - area, inter-reflection, color  Global illumination through indirect light Gl b l ill i ti th h i di t li ht  Measure illumination  Measure illumination  Light probe 27

Light probe for environment maps Light probe for environment maps  Light probe  Shiny crome sphere  Create HDR  Create HDR environment map  Captures most directions [Devebec98] 28

Image-based Lighting and shadows Image-based Lighting and shadows  Illuminate surfaces with environment map map [Devebec98] [ ]  Approximate with light field g and blend multiple shadows shadows [Knecht10] 29

Image-based Lighting and shadows Image-based Lighting and shadows 30

Light acquisition without a sphere Light acquisition without a sphere [Pilet06]  Calibrate multi-camera system  planar target with feature point  planar target with feature point recognition  On-line and off-line estimation O li d ff li ti ti of a light map  no light occlusions  depends only on normal direction 31

Now all together Now all together  Interaction between close real and virtual objects  light bleeding  light bleeding  shadowing  refraction  Render dynamic environment maps per p p virtual object (a) Environment map (b) Light sources (c) Virtual objects (d) Impostors [Pessoa10] 32

Now all together Now all together 33

Camera Effects Camera Effects  Small & cheap cameras & lenses  Small & cheap cameras & lenses  Many problems  Distortion  Vignetting  Chromatic aberration  Softness in corners  Bayer pattern artifacts  Color compression  Arbitrary image processing Arbitrary image processing  Motion blur f=2.1 mm, f/2.0, around $10 34

A terrible image 35

Graphics don’t match 36

Lens distortion Lens distortion  Wide angle lenses g  good for tracking  not well modeled with a pinhole camera  Texture-based Approach  Watson & Hodges ’95  Render undistorted initially  Use texture mapping to distort pp g rendered image  Geometry-based Approach Geometry based Approach  Vertex shader to tweak geometry 37

Render un-distorted visuals Render un-distorted visuals [Klein05]  Render into off screen  Prepare a grid for buffer texture mapping texture mapping 38

Compositing with video background Compositing with video background [Klein05]  Overlay distorted  Draw video background texture texture 39

Final Image 40

Geometry-based Approach Geometry-based Approach  Vertex shader to move vertices in projected image  Requires geometry subsampling q g y p g  Artifacts at the image edge 41

More effects More effects  Lens  Lens  Vignetting Sony 1/4” CCD  Chromatic aberration 640x480@30Hz 640x480@30Hz  Softness in corners S ft i Bayer mask  Image acquisition g q  Antialiasing  Sharping  Quantization Quantization  Bayer CFA artifacts  Color compression 42

Emulating camera effects Emulating camera effects  Some effects require changes to 3D rendering  Some effects require changes to 3D rendering  E.g: Depth defocus, proper motion blur  Expensive ! p  Focus on post-processing:  Take single RGBA image rendered in OpenGL  Modify this image to emulate camera effects  Limitations Limitations  No full motion blur [Klein08] 43