Calibrated Image Acquisition for Multi-view 3D Reconstruction - PowerPoint PPT Presentation

Calibrated Image Acquisition for Multi-view 3D Reconstruction Sriram Kashyap M S Guide: Prof. Sharat Chandran Indian Institute of Technology, Bombay April 2009 Sriram Kashyap 3D Reconstruction 1/ 42

Motivation Given pictures of an object, how can I place the object in a virtual environment? Sriram Kashyap 3D Reconstruction 2/ 42

Motivation Given pictures of an object, how can I place the object in a virtual environment? Sriram Kashyap 3D Reconstruction 2/ 42

Traditional Graphics Render a 3D world on a 2D screen The world is authored using modeling tools, by artists Sriram Kashyap 3D Reconstruction 3/ 42

Traditional Graphics Render a 3D world on a 2D screen The world is authored using modeling tools, by artists Sriram Kashyap 3D Reconstruction 3/ 42

Going backwards: 2D to 3D Sriram Kashyap 3D Reconstruction 4/ 42

Going backwards: 2D to 3D Monocular reconstruction Salzmann et al. Local deformation models for monocular 3D shape recovery. CVPR 2008 Sriram Kashyap 3D Reconstruction 4/ 42

Going backwards: 2D to 3D Stereo Reconstruction http://www.cs.washington.edu/homes/indria/project/CSE576finalproject/ Sriram Kashyap 3D Reconstruction 5/ 42

Going backwards: 2D to 3D Multi-view Reconstruction vision.middlebury.edu Sriram Kashyap 3D Reconstruction 6/ 42

Visual Hull Visual hull is an approximate 3D structure obtained from reconstruction algorithms based on object silhouettes. Given a camera, a silhouette defines a back-projected generalized cone that contains the actual object Each camera gives us one such generalized cone The intersection of these cones is the Visual Hull Sriram Kashyap 3D Reconstruction 7/ 42

Visual Hull Visual hull is an approximate 3D structure obtained from reconstruction algorithms based on object silhouettes. Given a camera, a silhouette defines a back-projected generalized cone that contains the actual object Each camera gives us one such generalized cone The intersections of these cones is the Visual Hull Sriram Kashyap 3D Reconstruction 8/ 42

Visual Hull Visual hull is an approximate 3D structure obtained from reconstruction algorithms based on object silhouettes. Given a camera, a silhouette defines a back-projected generalized cone that contains the actual object Each camera gives us one such generalized cone The intersections of these cones is the Visual Hull Sriram Kashyap 3D Reconstruction 9/ 42

Visual Hull Visual hull is an approximate 3D structure obtained from reconstruction algorithms based on object silhouettes. Given a camera, a silhouette defines a back-projected generalized cone that contains the actual object Each camera gives us one such generalized cone The intersections of these cones is the Visual Hull Sriram Kashyap 3D Reconstruction 10/ 42

Simulating multiple cameras Object is placed on a turntable Camera is fixed Camera captures frames as the table rotates Sriram Kashyap 3D Reconstruction 11/ 42

Issues in visual hull construction Image segmentation: Obtain silhouette information from images Camera calibration: Find the camera projection matrix for each camera Sriram Kashyap 3D Reconstruction 12/ 42

Segmentation Background Subtraction Capture object and background images If background and object image pixels are similar, mark pixel as background Similarity tests performed in RGB and YCbCr color spaces Sriram Kashyap 3D Reconstruction 13/ 42

Segmentation Background Subtraction Capture object and background images If background and object image pixels are similar, mark pixel as background Similarity tests performed in RGB and YCbCr color spaces Sriram Kashyap 3D Reconstruction 13/ 42

Segmentation Segmented image Sriram Kashyap 3D Reconstruction 14/ 42

What went wrong? We are using a webcam Camera exposure, white balance, noise, compression artifacts What is the correct threshold value? This may vary from image to image Sriram Kashyap 3D Reconstruction 15/ 42

Adaptive thresholding Can we provide more information to the system? Sriram Kashyap 3D Reconstruction 16/ 42

Adaptive thresholding Can we provide more information to the system? Ratio of object pixels to total number of pixels remains within certain bounds Provide the expected upper bound for this ratio Sriram Kashyap 3D Reconstruction 16/ 42

Adaptive thresholding Can we provide more information to the system? Ratio of object pixels to total number of pixels remains within certain bounds Provide the expected upper bound for this ratio Start with a low threshold value Increase the threshold till the ratio is below this upper bound Sriram Kashyap 3D Reconstruction 16/ 42

Adaptive thresholding Segmented image (foreground ratio=0.07) Sriram Kashyap 3D Reconstruction 17/ 42

Camera Calibration Calibration Find the projection matrix corresponding to each camera view Sriram Kashyap 3D Reconstruction 18/ 42

Camera Calibration Calibration Find the projection matrix corresponding to each camera view Find a set of 2D to 3D point correspondences Use existing tools to compute matrices from these correspondences Sriram Kashyap 3D Reconstruction 18/ 42

Point Correspondences Calibration Pattern Sriram Kashyap 3D Reconstruction 19/ 42

Point Correspondences Calibration Pattern Sriram Kashyap 3D Reconstruction 20/ 42

Point Correspondences Feature to find: Centers of black squares Centers are easier to find and more robust against errors Fix an absolute ordering of boxes Use colored boxes to find orientation Thresholding to locate the boxes Find centers of these boxes Sriram Kashyap 3D Reconstruction 21/ 42

Example Calibration Pattern Sriram Kashyap 3D Reconstruction 22/ 42

Example Black threshold, first pass Sriram Kashyap 3D Reconstruction 23/ 42

Example Black threshold, second pass Sriram Kashyap 3D Reconstruction 24/ 42

Example Green threshold, first pass Sriram Kashyap 3D Reconstruction 25/ 42

Example Green threshold, second pass Sriram Kashyap 3D Reconstruction 26/ 42

Example Green threshold, final pass Sriram Kashyap 3D Reconstruction 27/ 42

Example Labeled calibration pattern Sriram Kashyap 3D Reconstruction 28/ 42

Camera Calibration In some cases, the algorithm may explicitly fail(vision is uncertain) Discard such views automatically Sriram Kashyap 3D Reconstruction 29/ 42

Camera Calibration In some cases, the algorithm may explicitly fail(vision is uncertain) Discard such views automatically In some cases, the algorithm may fail silently (returns an incorrect, but mathematically valid matrix) Cannot automatically discard such views, although tools can be written to help find bad views Sriram Kashyap 3D Reconstruction 29/ 42

Camera Calibration Visualizing a camera matrix   216 . 201 634 . 052 − 44 . 0839 3922 . 02 36 . 6629 − 52 . 2481 394 . 727 3119 . 24   − 0 . 69624 0 . 685745 − 0 . 212142 12 . 13 Sriram Kashyap 3D Reconstruction 30/ 42

Camera Calibration Visualizing a camera matrix   591 . 256 − 313 . 573 − 52 . 8837 3893 . 9 − 5 . 06723 − 59 . 6318 − 395 . 35 3148 . 67   − 0 . 0399772 0 . 974564 − 0 . 220515 11 . 9781 Sriram Kashyap 3D Reconstruction 31/ 42

Results Actual object Sriram Kashyap 3D Reconstruction 32/ 42

Results Reconstruction Sriram Kashyap 3D Reconstruction 33/ 42

Results Reconstruction Sriram Kashyap 3D Reconstruction 34/ 42

Results Reconstruction Sriram Kashyap 3D Reconstruction 35/ 42

Results Reconstruction Sriram Kashyap 3D Reconstruction 36/ 42

Results Reconstruction Sriram Kashyap 3D Reconstruction 37/ 42

Results Reconstruction Sriram Kashyap 3D Reconstruction 38/ 42

Results Reconstruction Sriram Kashyap 3D Reconstruction 39/ 42

Results Reconstruction Sriram Kashyap 3D Reconstruction 40/ 42

Future Work 1 Better calibration using optimization techniques 2 Textured rendering of visual hull 3 Image based Relighting support Sriram Kashyap 3D Reconstruction 41/ 42

References 1 OpenCV camera control and calibration: opencv.willowgarage.com/ 2 Image based Animation: http://www.cse.iitb.ac.in/ biswarup/projects/Motion/ Sriram Kashyap 3D Reconstruction 42/ 42