COMP37111 Advanced Computer Graphics 5: Model Acquisition - 3 - PDF document

COMP37111 Toby Howard The University of Manchester COMP37111 Advanced Computer Graphics 5: Model Acquisition - 3 toby.howard@manchester.ac.uk 1 The University of Manchester Geometry from images § Our goal: to automatically extract geometry from images § This is a hard problem, and the subject of much current research … § We will look at the concepts , not the mathematical detail § We’ll cover: 1. Geometry from static images (photographs) 2. Geometry from image sequences (video) 2 School of Computer Science The University of Manchester 1

COMP37111 Toby Howard The University of Manchester What is a camera? § The basic “pinhole camera” model § Point P forms an image P’ on an Image Plane at a distance F (the focal length) away from the optical centre O (lens) Y World frame J Camera frame Image Plane j P F X I O O k optical centre P’ K Z 3 The University of Manchester Camera parameters § Extrinsic parameters define: § location of camera origin with respect to world frame § orientation of camera frame with respect to world frame § Intrinsic parameters define how coordinates in camera frame map to pixel coordinates on image plane: § Focal length § Image plane size § Position of image plane § Image plane “skew” (angle between rows and columns of CCD sensor) § If we can find the Extrinsic and Intrinsic camera parameters for an image, we say the image is “calibrated”, and we can then extract geometry from the image 4 School of Computer Science The University of Manchester 2

COMP37111 Toby Howard The University of Manchester Intrinsic parameters § The intrinsic parameters of a camera are expressed by its calibration matrix K : f S f S O ⋅ ⋅ ⎡ ⎤ x x θ ⎢ ⎥ K 0 f S O = ⋅ y y ⎢ ⎥ 0 0 1 ⎢ ⎥ ⎣ ⎦ § Where § S x and S y are the dimensions of the image plane (in pixels) § O x and O y are the coordinates of the centre of the image plane with respect to the focal point § S θ is the skew of the image plane (deviation from square) § f is the focal length of the camera 5 The University of Manchester Finding the calibration matrix § How can we find K for a camera? § If we have the actual camera: § Photograph a known calibration object, and calculate K directly from the resulting image Calibration object § If we only have an image (which is the general case): § We can estimate K by finding calibration features in the image: § Parallel lines Calibration features § Orthogonal lines 6 School of Computer Science The University of Manchester 3

COMP37111 Toby Howard The University of Manchester Calculating K using a calibration object 7 The University of Manchester Estimating K using calibration features in an image Calibration features 8 School of Computer Science The University of Manchester 4

COMP37111 Toby Howard The University of Manchester Lens distortion § But before we try to do anything with an image, we must correct for distortions caused by the camera lens § All lenses distort to some extent 9 The University of Manchester Barrel distortion in an image “Straight lines” aren’t straight 10 School of Computer Science The University of Manchester 5

COMP37111 Toby Howard The University of Manchester Barrel distortion in an image 11 The University of Manchester Removing distortion using image warping § Removing distortion is an example of image warping § The user selects a distorted E straight edge E § Then marks a line L on the image to indicate how the original edge should have looked L § We can then create a non- linear WARP transformation which maps L to E 12 School of Computer Science The University of Manchester 6

COMP37111 Toby Howard The University of Manchester Barrel distortion in an image 13 The University of Manchester Barrel distortion in an image – removed by warping 14 School of Computer Science The University of Manchester 7

COMP37111 Toby Howard The University of Manchester Calibrating a single image § We can calibrate from a single image IF we can identify at least 2 vanishing points, and we choose an origin Y axis Y axis X axis User marks parallel lines and X axis orthogonal origin lines 15 The University of Manchester Case study 3: Geometry reconstruction with Icarus § In the Icarus system, the user interactively matches features in a calibrated image with a set of pre-defined shapes (cubes, spheres, cylinders etc) § Because the image is calibrated, the synthetic objects can be accurately positioned, scaled and rotated into the scene § Paper to read: S. Gibson and T.L.J. Howard, “Interactive reconstruction of virtual environments from photographs, with application to scene-of- crime analysis.” 16 School of Computer Science The University of Manchester 8

COMP37111 Toby Howard The University of Manchester Original single image 17 The University of Manchester Reconstructed geometry User interactively approximates scene geometry by drawing primitive shapes in the calibrated image space. 18 School of Computer Science The University of Manchester 9

COMP37111 Toby Howard The University of Manchester Geometry, simply rendered The approximated geometry can then be rendered. 19 The University of Manchester Textures extracted and rendered Or we can grab the pixel colours from the original image and map these as textures onto the reconstructed primitives. 20 School of Computer Science The University of Manchester 10

COMP37111 Toby Howard The University of Manchester Novel view of 3D geometry Because this is a true 3D geometric model, we can view it from any angle. However, some views may require information that was not available from the original image sequence. This is ongoing research. 21 The University of Manchester Calibrating from a single image § Unfortunately for this method, many images simply do not feature clearly parallel lines … § So we tend to work with sets of multiple images 22 School of Computer Science The University of Manchester 11

COMP37111 Toby Howard The University of Manchester Calibrating from multiple images § Suppose we have at least 2 images of a scene, taken with the same camera § If we can identify at least 8 corresponding points in each image, we can estimate the intrinsic and extrinsic camera parameters § For details, see the following papers ( not examinable : for interest only, but we recommend you read them, available on the course webpage) § H.C.Longuet-Higgins, “A Computer Algorithm for Reconstructing a Scene from Two Projections,” Nature, vol 293, 1981, pp. 133-135. § R. Hartley, “In Defense of The Eight-Point Algorithm” 23 The University of Manchester Calibrating from multiple images origin § The user interactively marks corresponding points in each image 24 School of Computer Science The University of Manchester 12

COMP37111 Toby Howard The University of Manchester World frame and camera frames world frame Y View 1 camera frame Image 1 View 2 camera X frame Image 2 Z 25 The University of Manchester Calibrating from multiple images Y Y origin Z X X Z § How the world frame would be seen from each view 26 School of Computer Science The University of Manchester 13

COMP37111 Toby Howard The University of Manchester Extraction of geometry § We now have a calibrated system § In other words, we know: § The position and orientation of the world frame § For each view we know the position and orientation of the camera frame (with respect to the world frame) § Now, we can interactively draw in 3D with the correct camera parameters 27 The University of Manchester Interactive extraction of geometry § We draw vertices over the images (compare Mme Gouraud), and since we know the camera positions, we can compute 3D coordinates of each vertex 28 School of Computer Science The University of Manchester 14

COMP37111 Toby Howard The University of Manchester Extracting pixel colours from the image § We can sample pixel colours from the original images and map them onto the polygon mesh as textures Rendered with textures Raw polygon mesh 29 The University of Manchester Geometry from video sequences § What is a video sequence? § It’s a set of images – 24 images per second § Usually, there is coherence between frames too (if no edits) § Goal: to automatically calibrate the camera in each frame, to obtain the camera motion for the whole sequence § Approach: As before, we identify corresponding points between images … § … but this time each “image” is a frame from a video sequence § We try to do this automatically , using computer vision techniques 30 School of Computer Science The University of Manchester 15

COMP37111 Toby Howard The University of Manchester Automatic feature detection 31 The University of Manchester Automatic feature detection These features have been automatically identified 32 School of Computer Science The University of Manchester 16