29/05/2018 Ocular convergence A binocular depth cue related to the tension in the eye muscles when the eyes track inward to focus on objects close to the viewer The more tension in the eye muscle, the closer the object is Works best at close distances 41 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Depth cues – Range of effectiveness 0-2 m 2-20 m 30+ m Type of cue Accomodation x Oculomotor Converngence x Oculomotor Occlusion x x x Pictorial Relative size x x x Pictorial Relative height x x Pictorial Familiar size x x x Pictorial Texture gradient x x Pictorial Shadows x x x Pictorial Motion parallax x x Motion 42 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 21
29/05/2018 History of stereo and 3D vision 2D Photography 1839 - Louis Daguerre, William Fox Talbot, John Herschel 2D (still or motion) picture technologies are well developed and well accepted 43 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision History of stereo and 3D vision Stereoscope invented in 1838 by Sir Charles Wheatsone 179 years ago! 44 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 22
29/05/2018 History of stereo and 3D vision In 1849 Sir David Brewster invented a lens based stereoscope Sterograms were popular in the early 1900’s A special viewer was needed to display different images to the left and right eyes 45 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision History of stereo and 3D vision 46 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 23
29/05/2018 History of stereo and 3D vision The first 3D movies in the 1950’s 47 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision History of stereo and 3D vision Anaglyphs provide a stereoscopic 3D effect when viewed with 2-color glasses (each lens a chromatically opposite color, usually red and cyan) 48 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 24
29/05/2018 History of stereo and 3D vision Current technology for 3D movies and computer displays is to use polarized glasses The viewer wears eyeglasses which contain circular polarizers of opposite handedness 49 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision History of stereo and 3D vision Active shutter 3D glasses Alternate frame sequencing • Alternately displays different perspective for each eye • Uses liquid crystal or active shutter glasses 50 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 25
29/05/2018 History of stereo and 3D vision Head mounted display Helmet or glasses with two small LCD or LED displays with magnifying lenses, one for each eye Stereo films, images, games, maintenance of complex systems,… 51 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision History of stereo and 3D vision Autostereoscopic • Eliminates the eyeglasses and presents the depth as it is • Initially developed by Sharp Provide multiple views of the same scene, rather than just two Tow main methods providing autostereoscopic vision : • Parallax barrier • Lenticular lens 52 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 26
29/05/2018 History of stereo and 3D vision Parallax Barrier A mask is placed over the LCD display which directs light from alternate pixel columns to each eye Instant switching between 2D and 3D modes 53 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision History of stereo and 3D vision Lenticular lens An array of cylindrical lenses directs light from alternate pixel columns to a defined viewing zone Each eye to receives a different image at an optimum distance 54 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 27
29/05/2018 History of stereo and 3D vision Correct viewing position of an autosteroscopic display 55 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision 3D displays 56 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 28
29/05/2018 3D displays 57 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision 3D displays 58 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 29
29/05/2018 3D displays 59 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Benefits of stereo vision Relative depth judgment Spatial localization Breaking camouflage Surface material perception Judgment of surface curvature 60 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 30
29/05/2018 Main issues of stereo Geometry: • What information is available? • How do the camera views relate? Correspondences: • What feature in view 1 corresponds to what feature in view 2? Triangulation, reconstruction • Interference in presence of noise 61 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Outline Introduction to stereo vision Image formation and projective geometry Epipolar geometry and epipolar constraints Correspondence problem Camera calibration Multi-view reconstruction 62 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 31
29/05/2018 Perspective projection The axis of the real image plane O is the center of the projection The axis of the front image plane From similar triangles we can say: x i x x x i f f z z 63 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Perspective projection to a 2D image In a camera we have a flat image Points in the 3D world are projected onto the image plane. 64 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 32
29/05/2018 Perspective projection to a 2D image x x f i c z c y y f i c z c 65 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Estimating depth with stereo If we have 2 images of a scene, and hence can define 2 viewing rays, we can find the 3D location of that point by finding the intersection of the two viewing rays. scene point image plane optical center 66 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 33
29/05/2018 Stereo vision 2 cameras, 2 simultaneous Single moving camera and views static scene 67 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Stereo principle If we know: • Intrinsic parameters of each camera • The relative pose between the cameras If we measure: • An image point in the left camera • The corresponding point in the right camera Each image point corresponds to a ray emanating from the camera We can intersect these rays (triangulate) to find the absolute point position 68 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 34
29/05/2018 Camera parameters Camera frame 2 Extrinsic parameters: Camera frame 1 Camera frame 2 Intrinsic parameters: Image coordinates relative to Camera camera Pixel coordinates frame 1 Extrinsic parameters : rotation matrix and translation vector Intrinsic parameters : focal length, pixel sizes (mm), image center point, radial distortion We’ll assume for now that these parameters are given and fixed. 69 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Stereo geometry – simple case Assuming parallel optical axes, known camera parameters (i.e., calibrated cameras) x x d Disparity l r 70 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 35
29/05/2018 Stereo geometry – simple case d x x Disparity l r 71 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Stereo geometry – simple case z x f x l z x b f x r z y y y y f l r x x d Disparity l r 72 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 36
29/05/2018 Stereo geometry – simple case b b z f f x x d l r z z x x b x l r f f z z y y y l f r f Disparity refers to the difference in the image location of the same 3D point when projected under perspective to 2 different cameras . d x x Disparity l r 73 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Stereo geometry – simple case Important equation!!! baseline Depth disparity x x d Disparity l r 74 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 37
29/05/2018 Stereo geometry – simple case Triangulation = determining depth from disparity Depth disparity Depth and disparity are inversely proportional. d x x Disparity l r 75 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Stereo geometry – general case Cameras not aligned, but relative pose known Assuming f=1, we have In principle, you can find P by intersecting the rays O L p L and O R p R However, they may not intersect Instead, find the midpoint of the segment perpendicular to the two rays 76 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 38
29/05/2018 Stereo geometry – general case The projection of P onto the left image is The projection of P onto the right image is where 77 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Stereo geometry – general case Note that p L and M L P are parallel, so their cross product should be zero Similarly for p R and M R P Point P should satisfy both This is a system of 4 equations Can solve for the 3 unknowns (X L , Y L, , Z L ) using least squares Method also works for more than 2 cameras 78 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 39
29/05/2018 Stereo geometry – general case The transformation of coordinate system, from left to right is described by a rotation matrix R and a translation vector T. More precisely, a point P described as P L in the left frame will be described in the 1 right frame as P R ( P T ) r l 79 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Stereo disparity Tie in with the intro: for our purposes Disparity = Parallax Disparity/Parallax inversely proportional to depth Near objects appear to move more faster than far away ones when the camera translates sideways 80 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 40
29/05/2018 Stereo process Extract features from the left and right images Match the left and right image features, to get their disparity in position (the “ correspondence problem ”) Use stereo disparity to compute depth (the “ reconstruction problem ”) Need to know focal length f, baseline b • use prior knowledge or camera calibration http://vision.middlebury.edu/stereo/data/scenes2003/ The correspondence problem is the most difficult 81 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Correspondence problem For every point in the left image, there are many possible matches in the right image Locally, main points look similar → matches are ambiguous 82 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 41
29/05/2018 Correspondence problem We have 2 images taken from cameras with different intrinsic and extrinsic parameters How do we match a point in the first image to a point in the second? How can we constrain our search? 83 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Given a point in the left image, do you need to search the entire right image for the corresponding point? 84 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 42
29/05/2018 Establishing correspondences We can use the (known) geometry of the cameras to help limit the search for matches - epipolar geometry The most important constraint is the epipolar constraint • We can limit the search for a match to be along a certain line in the other image • Reduces the search space to a one-dimensional line • Makes search for correspondences quicker 85 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Outline Introduction to stereo vision Image formation and projective geometry Epipolar geometry and the epipolar constraint Correspondence problem Camera calibration Multi-view reconstruction 86 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 43
29/05/2018 Epipolar constraint : normal image pair With aligned cameras, search for corresponding point is 1D along the corresponding row of the other camera 87 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Epipolar constraint : normal image pair The epipolar plane cuts the through the image plane(s) forming 2 epipolar lines. The match for P 1 in the other image, must lie on the same epipolar line . 88 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 44
29/05/2018 Matching example using epipolar lines • For a patch in the left image • Compare with the patches along the same line in the right image • Select patch with highest match score 89 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Epipolar constraint : general image pair If cameras are not aligned, a 1D search can still be determined for the corresponding point. P, C1, C2 determine a plane that cuts image I2 in a line: P2 will be on that line. 90 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 45
29/05/2018 Epipolar constraint : general image pair 91 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Epipolar constraint : general image pair Epipolar plane Image plane Epipolar line The optical centers of the 2 cameras, a point P, and the image points p 0 and p 1 of P all lie in the same plane : epipolar plane These vectors are co-planar: 92 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 46
29/05/2018 Example: converging cameras 93 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Example: parallel cameras Where are the epipoles? 94 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 47
29/05/2018 Stereo image rectification What happens when the cameras are not aligned? Arbitrary arrangements of camera result in image planes that are not parallel Complicated epipolar representation and correspondence search 95 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Stereo image rectification Re-project image planes onto a common plane parallel to the line between camera centers Pixel motion is horizontal after this transformation 96 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 48
29/05/2018 Rectification example Original image pair overlaid with several epipolar lines Images rectified so that epipolar lines are horizontal and in vertical correspondence 97 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision From geometry to algebra So far, we have the explanation in terms of geometry. Now, how to express the epipolar constraints algebraically? 98 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 49
29/05/2018 Fundamental matrix The fundamental matrix F is the algebraic representation of epipolar geometry F is a 3×3 matrix which relates corresponding points in stereo images. 99 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Fundamental matrix Let p be a point in left image, p’ in right image P Epipolar relation l l’ • p maps to epipolar line l’ p p’ • p’ maps to epipolar line l Epipolar mapping described by a 3x3 matrix F It follows that 101 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 50
29/05/2018 Stereo geometry, with calibrated cameras Main idea 102 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Stereo geometry, with calibrated cameras If the stereo rig is calibrated, we know : how to rotate and translate camera reference frame 1 to get to camera reference frame 2. Rotation: 3 x 3 matrix R ; translation: 3 x 1 vector T . 103 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 51
29/05/2018 Stereo geometry, with calibrated cameras If the stereo rig is calibrated, we know : how to rotate and translate camera reference frame 1 to get to camera reference frame 2. Rotation: 3 x 3 matrix R ; translation: 3 x 1 vector T . X ' RX T c c 104 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Reminder: cross product Vector cross product takes two vectors and returns a third vector that’s perpendicular to both inputs. So here, c is perpendicular to both a and b, which means the dot product = 0. 105 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 52
29/05/2018 From geometry to algebra X' RX T X T X X T RX T X T RX T T 0 Normal to the plane T RX 106 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Reminder: Matrix form of cross product 0 a a b 3 2 1 a b a 0 a b c 3 1 2 a a 0 b 2 1 3 Can be expressed as a matrix multiplication. 0 a a 3 2 a x a 0 a 3 1 a a 0 2 1 107 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 53
29/05/2018 From geometry to algebra X' RX T X T X X T RX T X T RX T T 0 Normal to the plane T RX 108 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Essential matrix X T RX 0 X [T ] RX 0 x E [T x ] R Let EX X T 0 E is called the essential matrix , and it relates corresponding image points between both cameras, given the rotation and translation. If we observe a point in one image, its position in other image is constrained to lie on line defined by above. 109 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 54
29/05/2018 Fundamental matrix This matrix E is called the “Essential Matrix” • when image intrinsic parameters are known This matrix F is called the “Fundamental Matrix” • more generally (uncalibrated case) Can solve for F from point correspondences • Each (p, p’) pair gives one linear equation in entries of F • 8 points give enough to solve for F (8-point algorithm) 110 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Summary: Epipolar geometry Epipolar plane : plane containing baseline and world point Epipole : point of intersection of baseline with image plane Epipolar line : intersection of epipolar plane with the image plane Baseline : line joining the camera centers Epipolar Line • Epipolar Plane Baseline Epipole Epipole 112 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 55
29/05/2018 Summary: Epipolar constraint All epipolar lines intersect at the epipole An epipolar plane intersects the left and right image planes in epipolar lines Epipolar Line • Epipolar Plane Baseline Epipole Epipole 113 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Summary: Epipolar constraint X X X x x’ x’ x’ Potential matches for x have to lie on the corresponding line l’ . Potential matches for x’ have to lie on the corresponding line l . 114 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 56
29/05/2018 Summary Epipolar geometry • Fundamental matrix maps from a point in one image to a line (its epipolar line) in the other • Can solve for F given corresponding points (e.g., interest points) Stereo depth estimation • Main idea is to triangulate from corresponding image points. • Estimate disparity by finding corresponding points along scanlines • Depth is inverse to disparity 115 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Outline Introduction to stereo vision Image formation and projective geometry Epipolar geometry and the epipolar constraint Correspondence problem Camera calibration Multi-view reconstruction 116 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 57
29/05/2018 Correspondence problem Epipolar geometry constrains our search, but we still have a difficult correspondence problem. Worst case scenarios • A white board (no features) • A checkered wallpaper (ambiguous matches) The problem is under constrained To solve, we need to impose assumptions about the real world: • Disparity limits • Appearance • Uniqueness • Ordering • Smoothness 117 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Disparity limits Assume that valid disparities are within certain limits • Constrains search Why usually true? When is it violated? 118 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 58
29/05/2018 Appearance Assume features should have similar appearance in the left and right images Why usually true? When is it violated? 119 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Uniqueness Assume that a point in the left image can have at most one match in the right image Why usually true? 120 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 59
29/05/2018 Uniqueness Assume that a point in the left image can have at most one match in the right image Why usually true? When is it violated? 121 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Ordering Assume features should be in the same left to right order in each image Why usually true? 122 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 60
29/05/2018 Ordering Assume features should be in the same left to right order in each image Why usually true? When is it violated? 123 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Smoothness Assume objects have mostly smooth surfaces, meaning that disparities should vary smoothly Why usually true? When is it violated? 124 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 61
29/05/2018 Methods of correspondence Match points based on local similarity between images Two general approaches • Correlation-based approaches • Feature-based approaches 125 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Correlation approach Matches image patches using correlation Assumes only a translational difference between the two local patches (no rotation, or differences in appearance due to perspective) A good assumption if patch covers a single surface, and surface is far away compared to baseline between cameras Works well for scenes with lots of texture 126 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 62
29/05/2018 Correlation approach Similarity measures • CC (cross-correlation) • SSD (sum of squared differences) • SAD (sum of absolute differences) 127 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Cross correlation approach Select a range of disparities to search For each patch in the left image, compute cross correlation score for every point along the epipolar line Find maximum correlation score along that line 128 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 63
29/05/2018 Correspondence search with similarity constraint Left Right scanline Matching cost disparity Slide a window along the right epipolar line and compare contents of that window with the reference window in the left image Matching cost: SSD or normalized correlation 129 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Correspondence search with similarity constraint Left Right scanline SSD 130 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 64
29/05/2018 Correspondence search with similarity constraint Left Right scanline Normalized correlation 131 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Effect on window size Larger windows: + Robust to noise - Reduced precision, less detail Smaller windows: + Good precision, more detail - Sensitive to noise 132 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 65
29/05/2018 Feature matching Matches edges, lines, or corners Gives a sparse reconstruction May be better for scenes with little texture 133 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Correspondence error sources Low-contrast ; textureless image regions Occlusions Camera calibration errors Violations of brightness constancy (e.g., specular reflections) Large motions 134 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 66
29/05/2018 Main steps: stereo with calibrated cameras Given image pair, R, L Detect some features Compute essential matrix E Match features using the epipolar and other constraints Triangulate for 3D structure 135 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Outline Introduction to stereo vision Image formation and projective geometry Epipolar geometry and the epipolar constraint Correspondence problem Camera calibration Multi-view reconstruction 136 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 67
29/05/2018 Uncalibrated case What if we don’t know the camera parameters? We can still reconstruct the 3D structure, up to certain ambiguities, if we can find correspondences between points… 137 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Image, camera and world frames There are 3 coordinate systems involved: • Image • Camera • World 138 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 68
29/05/2018 Intrinsic parameters Intrinsic parameters are the parameters necessary to link the image’s coordinate system (pixel coordinates) to the idealized coordinate system (camera reference frame): • Focal length • Pixel size • Distortion coefficients • Image center 139 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Extrinsic camera parameters Extrinsic parameters • Position • Orientation (pose) of camera 140 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 69
29/05/2018 Camera calibration Calibration means estimating extrinsic (external) and intrinsic (internal) parameters using observed camera data Key idea : write the projection equation linking the known coordinates of a set of 3D points and their projection onto the image, and solve for the camera parameters 141 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Possible applications Use these parameters to: • correct for lens distortion, • measure the size of an object in world units, • determine the location of the camera in the scene Used in applications such as: • machine vision to detect and measure objects, • robotics, • navigation systems, • 3D scene reconstruction 142 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 70
29/05/2018 Examples 143 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Camera model : pinhole camera A pinhole camera is a simple camera without a lens and with a single small aperture. Light rays pass through the aperture and project an inverted image on the opposite side of the camera. Describes the mathematical relationship between the coordinates of a 3D point and its projection onto the image plane of an ideal pinhole camera. 144 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 71
29/05/2018 Pinhole camera parameters The world points are transformed to camera coordinates using the extrinsics parameters. The camera coordinates are mapped into the image plane using the intrinsics parameters. 145 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Pinhole camera parameters The calibration algorithm calculates the camera matrix using the extrinsic and intrinsic parameters. The extrinsic parameters represent a rigid transformation from 3D world coordinate system to the 3D camera's coordinate system. The intrinsic parameters represent a projective transformation from the 3D camera's coordinates into the 2D image coordinates. 146 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 72
29/05/2018 Two steps process Modeling: • Determine the equation that approximates the camera behavior • Define the set of unknowns in the equation (camera parameters) • The camera model is an approximation of the physics & optics of the camera Calibration: • Get the numeric value of every camera parameter 147 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Perspective projection 3D world mapped to 2D projection in image plane Camera center Image plane y X Y Y f y ( X , Y , Z ) ( f , f ) Y f Z Z Z Z x X f Scene point Image coordinates x X f Z Z 148 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 73
29/05/2018 Perspective projection 3D world mapped to 2D projection in image plane Camera center Image plane y X Y Y f y ( X , Y , Z ) ( f , f ) Y f Z Z Z Z x X f Image coordinates Scene point x X f Z Z 149 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Perspective projection But “pixels” are in some arbitrary spatial units X X x f x Z Z Y Y y f y Z Z 150 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 74
29/05/2018 Perspective projection If pixels are not square X X x x Z Z Y Y y y Z Z 151 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Perspective projection We don’t know the origin of our camera pixel coordinates X X x x x 0 Z Z Y Y y y y Z Z 0 152 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 75
29/05/2018 Perspective projection If we have skew between camera pixel axes x’ x y’ θ y x ' sin x y ' y cos( ) x ' y cot( ) x X X Y x x x cot( ) x 0 0 Z Z Z Y Y y y y y Z 0 sin( ) Z 0 153 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Camera calibration matrix (K) x 0 0 y K 0 0 0 0 1 α and β represent the focal lengths in units of physical pixels x 0 , y 0 represent the coordinates of the principle point 154 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 76
29/05/2018 Camera calibration matrix (K) 0 x 0 y K 0 0 0 0 1 α and β represent the focal lengths in units of physical pixels x 0 , y 0 represent the coordinates of the principle point Intrinsic parameters 155 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Camera calibration matrix (K) x s 0 y K 0 0 0 0 1 α and β represent the focal lengths in units of physical pixels x 0 , y 0 represent the coordinates of the principle point s represents the skew coefficient 5 Intrinsic parameters 156 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 77
29/05/2018 Intrinsic parameters : Homogeneous coordinates Using homogenous coordinates, we can write this as: X 0 x 0 x 0 Y y y 0 0 Z 0 1 0 0 1 0 1 C In pixels: x K x In camera-based coords 157 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Lens distortion We have assumed that lines are imaged as lines Not quite true for real lenses • Significant error for cheap optics and for short focal lengths 158 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 78
29/05/2018 Radial distortion In pixel coordinates the correction is written 159 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Extrinsic parameters The extrinsic parameters consist of a rotation, R , and a translation, t . The origin of the camera's coordinate system is at its optical center and its x- and y- axis define the image plane. 160 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 79
29/05/2018 Extrinsic parameters World coordinate system Camera coordinate system X X t c x Y R Y t c y Z Z t c z 3x3 rotation matrix 3x1 translation vector 3 angles 161 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Extrinsic parameters P R ( P T ) c w Camera reference World reference frame frame T P X , Y , Z c 162 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 80
29/05/2018 Camera matrix x X X c y K Y K R Y t c 1 Z Z c 163 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Camera matrix x X X c y K Y K R Y t c 1 Z Z c X Y K R t Z 1 3x4 matrix 164 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 81
29/05/2018 Camera matrix x X X c y K Y K R Y t c 1 Z Z c X Y K R t Z 1 P Camera matrix 165 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Single camera calibration methods Calibration using calibration patterns • taking multiple images of a pattern from different viewpoints. • estimating camera calibaration matrix using these images Auto-calibration • estimating camera calibration matrix directly from real image sequences 166 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 82
29/05/2018 Single camera calibration methods Method of Hall • Lineal method • Transformation matrix Method of Faugeras-Toscani • Lineal method • Obtaining camera parameters Method of Faugeras-Toscani with distortion • Iterative method • Radial distortion Method of Tsai • Iterative method • Radial distortion • Focal distance estimation Method of Weng • Iterative method • Radial and tangential distortion … and many more 167 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Single camera calibration : resectioning Estimating the camera matrix P from known x, X X x P P P P i i 11 12 13 14 Y y P P P P i 21 22 23 24 i Z i 1 P P P P 31 32 33 34 1 x P X P Y P Z P ... i 11 i 12 i 13 i 14 168 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 83
29/05/2018 Single camera calibration : resectioning So for each feature point i, we have: A p 0 2n x 12 12x1 vector n = the number of correspondences P K R t 169 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Camera calibration with a calibration object Main idea Place “calibration object” with known geometry in the scene Get correspondences Solve for mapping from scene to image 170 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 84
29/05/2018 Camera calibration with a calibration object 171 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Camera calibration with a calibration object The use of a calibration pattern is one of the more reliable ways to estimate a camera’s intrinsic parameters • A planar target is often used along with multiple images taken at different poses You can move the target in a controlled or just move it in an uncontrolled way • It is best if the calibration object spans as much of the image as possible The strategy is to first solve for all calibration parameters except lens distortion by assuming there is no distortion • Then perform a final nonlinear optimization which includes solving for lens distortion 172 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 85
29/05/2018 Calibration problem P 1 … P n with known positions in [O w ,i w ,j w ,k w ] 173 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Calibration problem P 1 … P n with known positions in [O w ,i w ,j w ,k w ] p 1 … p n with known positions in the image Goal: compute intrinsic and extrinsic parameters 174 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 86
29/05/2018 Calibration problem How many correspondences do we need? • Our camera matrix has 11 unknown • We need 11 equations • 6 correspondences would do it 175 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Calibration problem In practice, using more than 6 correspondences enables more robust results 176 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 87
29/05/2018 Calibration problem Changed notation M = projective matrix 177 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Calibration problem 178 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 88
29/05/2018 Calibration problem 179 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Calibration problem 180 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 89
29/05/2018 Calibration problem 181 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Homogeneous M x N Linear Systems Rectangular system (M>N) • 0 is always a solution • To find non-zero solution ─ Minimize |P m| 2 ─ under the constraint |m| 2 =1 M = number of equations = 2n N = number of unknowns = 11 182 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 90
29/05/2018 Calibration problem How do we solve this homogenous linear system? Via SVD (Singular Value Decomposition) decomposition! 183 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Calibration problem 184 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 91
29/05/2018 Extracting camera parameters 185 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Extracting camera parameters 186 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 92
29/05/2018 Extracting camera parameters 187 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Extracting camera parameters 188 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 93
29/05/2018 Stereo camera calibration There are several tasks in developing a practical system for binocular stereo: 1. Calibrate the intrinsic parameters for each camera. 2. Solve the relative orientation problem. 3. Stereo image rectification. 4. Compute conjugate pairs by feature matching or correlation. 5. Solve the stereo intersection problem for each conjugate pair. 6. Determine baseline distance. 7. Solve the absolute orientation problem to transform point measurements from the coordinate system of the stereo cameras to an absolute coordinate system for the scene. 189 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Stereo camera calibration 190 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 94
29/05/2018 Stereo camera calibration Solve the relative orientation problem and determine the baseline by other means, such as using the stereo cameras to measure points that are at a known distance apart . Calibrates the rigid body transformation between the two cameras. Since the baseline has been calibrated, the point measurements will be in real units and the stereo system can be used to measure the relationships between points on objects in the scene. It is not necessary to solve the absolute orientation problem, unless the point measurements must be transformed into another coordinate system. 191 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Stereo camera calibration Solve the relative orientation problem and obtain point measurements in the arbitrary system of measurement that results from assuming unit baseline distance . The point measurements will be correct, except for the unknown scale factor. Distance ratios and angles will be correct. If the baseline distance is obtained later, then the point coordinates can be multiplied by the baseline distance to get point measurements in known units. 192 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 95
29/05/2018 Stereo camera calibration Solve the exterior orientation problem for each stereo camera. This provides the transformation from the coordinate systems of the left and right camera into absolute coordinates. The point measurements obtained by intersecting rays will automatically be in absolute coordinates with known units, and no further transformations are necessary. 193 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Outline Introduction to stereo vision Image formation and projective geometry Epipolar geometry and the epipolar constraint Correspondence problem Camera calibration Multi-view reconstruction 194 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 96
29/05/2018 Context A new trend in 3D Gfx: modeling by capturing the real world page 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision 195 Acquisition of 3D data 3D laser scanner • Direct acquisition • Renders in real-time the acquired points as coordinates • Advantages : very accurate, fast results • Disadvantages : quite expensive, difficulties with shiny, shimmering or transparent objects Structured light 196 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 97
29/05/2018 Acquisition of 3D data Photogrammetry • Indirect acquisition • Extracts spatial coordinates using different rendering techniques • Mono, stereo or multi images • Disadvantages : slower, less accurate • Advantages : cheaper, easier to use 197 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision 3D reconstruction from images Goal • Automatic construction of photo-realistic 3D models of a scene from multiple images taken from a set of arbitrary viewpoints • Image-based modeling; 3D photography Applications • Interactive visualization of remote environments or objects by a virtual video camera for fly-bys, mission rehearsal and planning, site analysis • Virtual modification of a real scene for augmented reality tasks 198 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 98
29/05/2018 Dense reconstruction : motivation Accurate 3D models → cultural heritage Reconstruction of houses, buildings, famous touristic sites 199 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Dense reconstruction: challenges Scene elements do not always look the same in the images Camera-related problems • Image noise 200 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 99
29/05/2018 Dense reconstruction: challenges Scene elements do not always look the same in the images Camera-related problems • Image noise • Lens distortion 201 30/05/2018 Institut Mines-Télécom IMA 4509 - Stereo vision Dense reconstruction: challenges Scene elements do not always look the same in the images Camera-related problems • Image noise • Lens distortion • Color/chromatic aberration 202 Institut Mines-Télécom IMA 4509 - Stereo vision 30/05/2018 100
Recommend
More recommend
