VU Augmented Reality on Mobile Devices VU Augmented Reality on - PowerPoint PPT Presentation

VU Augmented Reality on Mobile Devices VU Augmented Reality on Mobile Devices  Introduction  Introduction – What is AR What is AR  Interaction Techniques  Navigation Collaboration Navigation, Collaboration  Visualization Techniques  Visual Coherence Visual Coherence  Tracking  … Based on material from Denis Kalkofen, TU Graz , 1

Visual Augmentation Visual Augmentation  Combine real and virtual imagery  Combine real and virtual imagery  Tracking & Registration data is used to align virtual objects within real imagery j g y + = 2

Augmented Graphics Augmented Graphics  Information  Virtual Objects  Visualization  Graphics  Understanding  Realism Understanding Realism 3

Spatial Arrangements Spatial Arrangements  Left, Right  In Front / Behind In Front / Behind How do we perceive spatial arrangements? 4

Making the Invisible visible Making the Invisible visible  Hidden structures & & information  Superman’s Xray Vision  Spatial arrangement? t? 5

Information display management Information display management  Selection  Placement  Clutter  Context Context  Task  Environment 6

XRay Vision XRay Vision 7

The Depth Problem The Depth Problem  Where is the ball in 3D?  Where is the ball in 3D?  Same position in 2D  Depth cues indicate position in 3D Depth cues indicate position in 3D 8

[Depth]: Dynamic Cues [Depth]: Dynamic Cues  Motion Parallax  Obtained by  Movement of the scene  Movement of the camera  Movement of the camera (head)  Speed as a function of di t distance  Far distant objects appear to move slower pp than near objects 9

[Depth]: Disparity [Depth]: Disparity  The difference in distance of the projected position of  The difference in distance of the projected position of the same point on the retina  Used in stereo displays p y 10

[Depth]: Static Cues [Depth]: Static Cues  Occlusion  Relative height  Relative size  Perspective  Brightness  Shadows  Texture Details 11

Transparent Occluder Transparent Occluder  Blend foreground pixel where object is hidden  Blend foreground pixel where object is hidden  Via pixel blending & stencil masking 12

Transparent Phantoms Transparent Phantoms  Entire occluder is transparent -> Need scene knowledge  Via pi el blending & Via pixel blending & stencil test Algorithm: Algorithm: 1. 1. Render Phantom to Render Phantom to stencil buffer stencil buffer stencil buffer stencil buffer 2. 2. Draw video outside the Draw video outside the mask mask 3. 3. Clear depth buffer Clear depth buffer 4. 4. Render virtual objects Render virtual objects 5. 5. Setup glBlendFunc Setup glBlendFunc 6. 6. Blend video inside mask Blend video inside mask 13

Transparent Phantoms Transparent Phantoms Problem: Blends with black if no virtual object is hidden hidden Sometimes hard to ’understand’ the object’s spatial relationship, especially in case of multiple occluding objects 14

Transparent Phantoms Transparent Phantoms  Better blend with video’s history  Better blend with video s history  Static viewpoint  Static hidden background background [Buchmann05] 15

Problems with Simple Blending Problems with Simple Blending  If occluder’s transparency is very high scene may be  If occluder s transparency is very high, scene may be hard to ’understand’  Object’s spatial relationship, especially in case of multiple occluding objects l di bj t  Important depth cues and landmarks are lost ! Missing Occlusion Cues ! ! Missing Occlusion Cues ! 17

Cut Aways Cut Aways  Display Fully opaque hidden objects Display Fully opaque hidden objects  Additional occlusion cues provided via cut-out [Sielhorst06] [Sielhorst06] 18

Cut Aways Cut Aways [Bichlmeier06] [Sielhorst06]  Video vs. Black background  Left: Only hidden objects are cut  Right: Video is removed in cutting Right: Video is removed in cutting 19

Cut Away Shapes Cut Away Shapes  2-1/2D Window  Screen aligned  Cut-away follows occluder‘s geometry  -> Need 3D Representation > Need 3D Representation [Furmanski04] [Bichlmeier06] 20

Cut Away Types Cut Away Types  2 1/2D Window  2-1/2D Window  Additional Box Rendering  Additional perspective cue p p [F [Fuch92] h92] [Furmanski04] 21

Cut Away for Underground Infrastructure Cut Away for Underground Infrastructure Fixed box cut away Geometry-based cut away 22

Cut Away Types Cut Away Types  2-1/2D Window Cuts  Additional Box  Box Cuts  Use a cubic geometry to cut away occluder  Need 3D occluder [Li07] [Coffin02] 23

Cut Away Types Cut Away Types  2-1/2D Window  Additional Box  Box Cuts Box Cuts  Tube Cuts  Perpendicular to primary axis [Li07] 24

Cut Away Types Cut Away Types  2 1/2D Window Cuts  2-1/2D Window Cuts  Additional Box  Box Cuts Box Cuts  Tube Cuts  Wedge Cuts Wedge Cuts  Similar to Box Cuts  Use wedge instead of box [Li07] Combined Wedge g and Tube Cut 25

Cut Away Types Cut Away Types  2-1/2D Window Cuts  Additional Box  Box Cuts Box Cuts  Tube Cuts  Wedge Cuts Wedge Cuts  Inset Cuts  Multi-Object j Occlusions [Li07] 26

Interactive Cut Away Creation Interactive Cut Away Creation  Interactive ’window’ placement  Interactive window placement  Define window corner [Coffin02] [Bichlmeier06] 27

Interactive Cut Away Creation Interactive Cut Away Creation  Li 2007  Interactive object classification  Based on B d classification, the system chooses the cutting type 29

Cut-Away Problems Cut-Away Problems  Big cuts  Big cuts  ’Hole’ can become as big as occluder  -> No occluder and no occlusion cue will remain  Mentally ’uncutting’ becomes difficult for big cuts M t ll ’ tti ’ b diffi lt f bi t => Use Ghosting in such cases <= Ghosting = Sparse representation of occluding objects 30

Ghosting Ghosting  Illustrators technique  Illustrators technique  Occluder‘s most important features are kept visible to preserve its shape, texture or landmarks. p p , khulsey.com 31

Types of Preservings Types of Preservings  Screen door  Screen door  Easy to compute  Easy scalable between sparse and dense representations  Introduces pattern I t d tt [Viola05] 32

Types of Preservings Types of Preservings  Perceptual driven preserving  Perceptual driven preserving  Interrante: Experiments with different types of occlusion pattern p [Interrante95] 33

Procedural Masks Procedural Masks  Static and dynamic masks  Static and dynamic masks 34

Types of Preservings: Discrete Preserving Types of Preservings: Discrete Preserving  Interrante: Curvature = Import shape indicator  Interrante: Curvature = Import shape indicator  Preserve Ridge and Valley Lines [Interrante95] 35

Types of Preservings: Continous Preserving Types of Preservings: Continous Preserving  Non uniform transparency modulation as continuous  Non-uniform transparency modulation as continuous function of occluders attributes  transp = f(curvature)  … [Krueger06] 36

Types of Preservings Types of Preservings  transparency = f(distance to hiddenobject)  transparency = f(distance_to_hiddenobject) [Krueger06] 37

Types of Preservings Types of Preservings  transparency = f(ShadingIntensity)  transparency = f(ShadingIntensity) [B [Bruckner06] k 06] 38

Types of Preservings Types of Preservings  transparency = combination of parameters  transparency = combination of parameters Curvature Dist(maskCenter) Angle(normal,viewDir) [Bichlmeier07] similar to f(ShadingIntensity) 39

Ghosting in AR Ghosting in AR 40

Features from Real World Imagery Features from Real World Imagery  Use image based filter g operation, e.g. any type of edge detector 41

Features from Tracked Model Features from Tracked Model  Object-based feature j detectors are independent from scene conditions like lighting conditions like lighting, texture ->gives ususally better results  May suffer from poor tracking / registration tracking / registration 42

Sources of Features in AR Sources of Features in AR  Feature from video feed  Easily over- or underpreserving  Reduce overpreserving with h brid approach hybrid approach  detect features only inside tracked mask 43

Discrete vs Continuous Preservings Discrete vs. Continuous Preservings  Continuous  Continuous  Can be difficult to understand  Less number of fully opaque pixel  Discrete Di t  Difficult to interpolate between level of sparseness 44

Enhanced vs. Non-Enhanced (Video) Preserving in AR Preserving in AR  Discrete non enhanced preservings are difficult to  Discrete, non-enhanced preservings are difficult to identify in video imagery  To ’understand’ the occluder, we need to perceive its , p ghosting as one object  Enhance preserving to perceive ghosting or use continous preserving ti i 45