Lecture 7 Measurement Using a Single Camera Lin ZHANG, PhD School - PowerPoint PPT Presentation

Foundations of Projective Geometry • Lines in the homogeneous coordinate On a projective plane, please determine the line passing two   ' x ( , , ) , x ( , , ) T T points x y z x y z 1 1 1 2 2 2 x y z y z z x x y 1 1  1 1  1 1   0  0 x y z x y z 1 1 1 y z z x x y 2 2 2 2 2 2 x y z 2 2 2 T   y z z x x y 1 1 1 1 1 1 , ,    y z z x x y  2 2 2 2 2 2 Theorem ' x x , On the projective plane, the line passing two points is   ' l x x Lin ZHANG, SSE, 2016

Foundations of Projective Geometry • Lines in the homogeneous coordinate   x ( , , ) T l ( , , ) T A point is on the line x y z a b c 0 0 0 T    0 x l  x l 0 (It is ) Lin ZHANG, SSE, 2016

Foundations of Projective Geometry • Lines in the homogeneous coordinate Theorem: On the projective plane, the intersection of two   ' ' x l l l l , lines is the point    0    0, a x b y c z Proof: Two lines a x b y c z 2 2 2 1 1 1   ' l ( , , ) , l ( , , ) T T a b c a b c 1 1 1 2 2 2   x y   , Inhomogeneous form   X Y   z z   c b a c 1 1 1 1       0 a X bY c c b a c   1 1 1 2 2 2 2 ,  X Y    0 a b a b  a X b Y c 1 1 1 1 2 2 2 a b a b 2 2 2 2 Lin ZHANG, SSE, 2016

Foundations of Projective Geometry • Lines in the homogeneous coordinate Theorem: On the projective plane, the intersection of two   ' ' x l l l l , lines is the point Homogenous form of the cross point is     c b a c a b 1 1 1 1   1 1  k let   c b a c       a b c b a c a b   2 2 2 2 2 2 x , ,1 k   1 1 1 1 1 1 x , ,   a b a b    c b a c a b  1 1 1 1   2 2 2 2 2 2   a b a b   2 2 2 2   b c c a a b     1 1 1 1 1 1 '  x l l x , ,   b c c a a b  2 2 2 2 2 2 Lin ZHANG, SSE, 2016

Foundations of Projective Geometry • Lines in the homogeneous coordinate Theorem: On the projective plane, the intersection of two   ' ' x l l l l , lines is the point   1, 1 Example: find the cross point of the lines x y y Homogeneous form        1 0 ( 1) 0 x x x x x   1 , 2   x y 1 2 3    x x 3 3 y  1     0 1 ( 1) 0  x x x 1 2 3 Homogeneous coordinates of the two x o   lines are (1,0, 1) ,(0,1, 1) T T x  1 Cross point is       (1,0, 1) (0,1, 1) 1,1,1 T T Lin ZHANG, SSE, 2016

Foundations of Projective Geometry • Lines in the homogeneous coordinate Theorem: On the projective plane, the intersection of two   ' ' x l l l l , lines is the point   1, 2 Example: find the cross point of the lines x x y Homogeneous form      1 0 ( 1) 0   x x x x  x x 2   1 , 2   1 2 3 x y    x x 3 3     1 0 ( 2) 0  x x x 1 2 3 Homogeneous coordinates of the two x o   lines are (1,0, 1) ,(1,0, 2) T T x  1 Cross point is i j k         (1,0, 1) (1,0, 2) 1 0 1 0,1,0 T T  1 0 2 Lin ZHANG, SSE, 2016

Foundations of Projective Geometry • Duality In projective geometry, lines and points can swap their positions T  x l 0 How to interpret? If x is a variable, it represents the points lying on the line l ; If l is a variable, it represents the lines passing a fixed point x '   x x , ' l x x The line passing two points is   ' ' x l l l l , The cross point of two lines is Duality Principle: To any theorem of projective geometry, there corresponds a dual theorem, which may be derived by interchanging the roles of points and lines in the original theorem Lin ZHANG, SSE, 2016

Contents • Foundations of Projective Geometry • Matrix Differentiation • Lagrange Multiplier • Least‐squares for Linear Systems • What is Camera Calibration • Single Camera Calibration • Bird’s‐eye‐view Generation Lin ZHANG, SSE, 2016

Matrix differentiation • Function is a vector and the variable is a scalar    T ( ) ( ), ( ),..., ( ) f t f t f t f t 1 2 n Definition T   ( ) ( ) ( ) df df t df t df t   , ,..., 1 2 n    dt dt dt dt Lin ZHANG, SSE, 2016

Matrix differentiation • Function is a matrix and the variable is a scalar   ( ) ( ),..., ( ) f t f t f t 11 12 1 m   ( ) ( ),..., ( ) f t f t f t      21 22 2   ( ) m ( ) f t f t    ij   n m   ( ) ( ),..., ( )   f t f t f t 1 2 n n nm Definition   ( ) ( ) ,..., ( ) df t df t df t 11 12 1 m   dt dt dt   ( ) ( ) ( )   df t df t df t   ( ) 21 22 ,..., 2 m df t df      ij  dt dt dt     dt dt   n m     ( ) ( ) ( ) df t df t df t 1 2 ,...,   n n nm   dt dt dt Lin ZHANG, SSE, 2016

Matrix differentiation • Function is a scalar and the variable is a vector  ( ), x x ( , ,..., ) T f x x x 1 2 n Definition T      df f f f   , ,...,     x   d x x x 1 2 n In a similar way,  ( ), x x ( , ,..., ) f x x x 1 2 n      df f f f   , ,...,     x   d x x x 1 2 n Lin ZHANG, SSE, 2016

Matrix differentiation • Function is a vector and the variable is a vector       T T x , ,..., , y ( ), x ( ),..., x ( ) x x x x y y y 1 2 1 2 n m Definition     ( ) x ( ) x ( ) x  y y y , ,..., 1 1 1      x x x   1 2 n      ( ) x ( ) x ( ) x y y y 2 , 2 ,..., 2 y   d      x x x  1 2 n x T d         ( ) x ( ) x ( ) x y y y   , ,..., m m m        x x x 1 2  n m n Lin ZHANG, SSE, 2016

Matrix differentiation • Function is a vector and the variable is a vector       T T x , ,..., , y ( ), x ( ),..., x ( ) x x x x y y y 1 2 1 2 n m In a similar way,      ( ) x ( ) x ( ) x y y y 1 , 2 ,..., m      x x x   1 1 1     ( ) x ( ) x ( ) x  y y y 1 , 2 ,..., m y T   d      x x x  2 2 2 x d         ( ) x ( ) x ( ) x y y y   1 , 2 ,..., m        x x x  n n n n m Lin ZHANG, SSE, 2016

Matrix differentiation • Function is a vector and the variable is a vector Example:   x 1   ( ) x   y  1      2 2 y , x , ( ) x , ( ) x 3   x y x x y x x   2 1 1 2 2 3 2 ( ) x   y   2   x 3     ( ) x ( ) x y y 1 2     x x     1 1 2 0 x 1     y ( ) x ( ) x T   d y y    1 2 1 3       x d x x     2 2 0 2  x    3   ( ) x ( ) x y y 1 2       x x 3 3 Lin ZHANG, SSE, 2016

Matrix differentiation • Function is a scalar and the variable is a matrix   ( X X ),  m n f Definition      f f f       x x x   11 12 1 n ( X ) df     X   d    f f f           x x x 1 2 m m mn Lin ZHANG, SSE, 2016

Matrix differentiation • Useful results (1) n   1 x a ,  Then, a x x a T T d d   a , a x x d d How to prove? Lin ZHANG, SSE, 2016

Matrix differentiation • Useful results x dA     1   , x  m n n (2) Then, A A x T d x T T d A      1  , x  T m n n (3) Then, A A x d x x T d A     1    , x  n n n (4) ( ) x Then, T A A A x d a Xb T d       1 1  X  , a  , b  m n m n ab (5) T Then, X d a X b T T d       1 1  X  , a  , b  n m m n ba (6) T Then, X d x x T d  2 x  n  1 x  (7) Then, x d Lin ZHANG, SSE, 2016

Contents • Foundations of Projective Geometry • Matrix Differentiation • Lagrange Multiplier • Least‐squares for Linear Systems • What is Camera Calibration • Single Camera Calibration • Bird‐view Generation Lin ZHANG, SSE, 2016

Lagrange multiplier • Single‐variable function  ( , ) ( ) f x is differentiable in ( a , b ). At , f ( x ) achieves an x a b 0 extremum df  | 0 x 0 dx • Two‐variables function ( , ) ( , ) f x y is differentiable in its domain. At , f ( x, y ) achieves x y 0 0 an extremum   f f   | 0, | 0 ( , ) ( , )   x y x y 0 0 0 0 x y Lin ZHANG, SSE, 2016

Lagrange multiplier • In general case x   1 ( ), x x  n If is a stationary point of f 0    f f f    | 0, | 0,..., | 0 x x x    0 0 0 x x x 1 2 n Lin ZHANG, SSE, 2016

Lagrange multiplier • Lagrange multiplier is a strategy for finding the local extremum of a function subject to equality constraints     1 ( ), x x n y f Problem: find stationary points for   ( ) x 0, 1,2,..., g k m under m constraints k Solution: m       ( ; x ,..., ) ( ) x ( ) x F f g 1 m k k  1    k ( x , , ..., ) If is a stationary point 0 10 20 0 m of F , then, x f x ( ) is a stationary point of with constraints 0 Joseph‐Louis Lagrange Jan. 25, 1736~Apr.10, 1813 Lin ZHANG, SSE, 2016

Lagrange multiplier • Lagrange multiplier is a strategy for finding the local extremum of a function subject to equality constraints     1 ( ), x x n y f Problem: find stationary points for   ( ) x 0, 1,2,..., g k m under m constraints k Solution: m       ( ; x ,..., ) ( ) x ( ) x F f g 1 m k k  1 k   ( x , ,..., ) is a stationary point of F 0 10 0 m       F F F F F F       0, 0,..., 0, 0, 0,..., 0          x x x 1 2 1 2 n m at that point n + m equations! Lin ZHANG, SSE, 2016

Lagrange multiplier • Example Problem: for a given point p 0 = (1, 0), among all the points lying on the line y = x , identify the one having the least distance to p 0 . The distance is     2 2 ( , ) ( 1) ( 0) f x y x y Now we want to find the stationary point of f ( x , y ) under the constraint ?    ( , ) 0 g x y y x According to Lagrange multiplier method, p 0 construct another function y = x           2 2 ( , , ) ( ) ( ) ( 1) ( ) F x y f x g x x y y x F x y  ( , , ) Find the stationary point for Lin ZHANG, SSE, 2016

Lagrange multiplier • Example Problem: for a given point p 0 = (1, 0), among all the points lying on the line y = x , identify the one having the least  distance to p 0 . F  0   x       2( 1) 0   0.5 x x    F   0      2 0   0.5  y y ? y      0   1  x y   F  0  p 0   y = x F x y  ( , , ) (0.5,0.5,1) is a stationary point of (0.5,0.5) is a stationary point of f ( x , y ) under constraints Lin ZHANG, SSE, 2016

Contents • Foundations of Projective Geometry • Matrix Differentiation • Lagrange Multiplier • Least‐squares for Linear Systems • Homograpy Estimation • What is Camera Calibration • Single Camera Calibration • Bird‐view Generation Lin ZHANG, SSE, 2016

LS for Inhomogeneous Linear System Consider the following linear equations system      3     1 1 3 x x x 1 2  1           2 4 2 1 4        x x x 1 2 2 x b A Matrix form: A  x b   1 x 1  It can be easily solved  2  x 2 Lin ZHANG, SSE, 2016

LS for Inhomogeneous Linear System How about the following one?    3     1 1 3 x x 1 2        x 1     2 4 2 1 4    x x     1 2   x        2 1 2 6 2 6      x x 1 2 It does not have a solution! A  x b What is the condition for a linear equation system can be solved? Can we solve it in an approximate way? A: we can use least squares technique! Carl Friedrich Gauss Lin ZHANG, SSE, 2016

LS for Inhomogeneous Linear System Let’s consider a system of p linear equations with q unknowns      ... b a x a x a x 11 1 12 2 1 1 q q      ... b  a x a x a x 21 1 22 2 2 2   q q x b  A ...  unknowns      ... b a x a x a x  1 1 2 2 p p pq q p We consider the case: p > q , and rank( A )= q In general case, there is no solution! Instead, we want to find a vector x that minimizes the error: p  2       2 ( ) x ( ... b ) x b E a x a x A 1 1 i iq q i  1 i Lin ZHANG, SSE, 2016

LS for Inhomogeneous Linear System 2    * x arg min ( ) x arg min x b E A 2 x x    1  * x b T T A A A Pseudoinverse of A How about the pseudoinverse of A when A is square and non-singular? Lin ZHANG, SSE, 2016

LS for Homogeneous Linear System Let’s consider a system of p linear equations with q unknowns      ... 0 a x a x a x 11 1 12 2 1 q q      ... 0  a x a x a x 21 1 22 2 2   q q x 0  A ...  unknowns      ... 0 a x a x a x  1 1 2 2 p p pq q We consider the case: p > q , and rank( A )= q Theoretically, there is only a trivial solution: x = 0  x 1 So, we add a constraint to avoid the trivial solution 2 Lin ZHANG, SSE, 2016

LS for Homogeneous Linear System 2   x 1 ( ) x x We want to minimize , subject to E A 2 2   * x arg min ( ), . ., x x 1 (1) E s t 2 x Use the Lagrange multiplier to solve it,     2 2     * x arg min x 1 x (2) A   2 2 x Solving the stationary point of the Lagrange function,      2 2     x 1 x A     2 2 (3) 0    x      2 2     x 1 x A    2 2  0     Lin ZHANG, SSE, 2016

LS for Homogeneous Linear System     2 2     x 1 x A    2 2 0 (3)  x Then, we have   x x T A A x is the eigen‐vector of A T A associated with the eigenvalue    2       x x x x x x T T T E A A A 2 The unit vector x is the eigenvector associated with the minimum T eigenvalue of A A Lin ZHANG, SSE, 2016

Contents • Foundations of Projective Geometry • Matrix Differentiation • Lagrange Multiplier • Least‐squares for Linear Systems • Homograpy Estimation • What is Camera Calibration • Single Camera Calibration • Bird’s‐eye‐view Generation Lin ZHANG, SSE, 2016

Homography Estimation Problem definition:     ' x x Given a set of points and a corresponding set of points i i in a projective plane, compute the projective transformation ' x x that takes to i i   ' x x , 1,2,..., We know there existing an H satisfying H i n i i     x ' x Coordinates of and are known, we need to find H i i where H is a homography matrix   a a a 11 12 13     It has 8 degrees of freedom H a a a  21 22 23     a a a 31 32 33 Lin ZHANG, SSE, 2016

Homography Estimation 4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom           a a a a x a y a cu cu x 11 12 13 11 12 13             cv a a a y a x a y a cv       21 22 23 21 22 23           1     c    a a a a x a y a c 31 32 33 31 32 33    a x a y a  11 12 13 u     a x a y a 31 32 33    a x a y a   21 22 23 v     a x a y a 31 32 33 Lin ZHANG, SSE, 2016

Homography Estimation 4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves   two degrees of freedom a 11   a   12   a 13     a 21       1 0 0 0 x y ux uy u   0    a  22    0 0 0 1    x y vx vy v  a 23   a Thus, four correspondence pairs   31 generate 8 equations   a 32       a 33 Lin ZHANG, SSE, 2016

Homography Estimation 4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom  x 0 (1) A   9 1 8 9 Rank A  ( ) 8 Normally, ; thus (1) has 1 (9‐8) solution vector in its solution space Lin ZHANG, SSE, 2016

Homography Estimation • How about the case when there are more than 4 correspondence pairs? • Use the LS method (for homogeneous case) to solve the model Lin ZHANG, SSE, 2016

Contents • Foundations of Projective Geometry • Matrix Differentiation • Lagrange Multiplier • Least‐squares for Linear Systems • Homograpy Estimation • What is Camera Calibration • Single Camera Calibration • Bird’s‐eye‐view Generation Lin ZHANG, SSE, 2016

What is camera calibration? • Camera calibration is a necessary step in 3D computer vision in order to extract metric information from 2D images • It estimates the parameters of a lens and image sensor of the camera; you can use these parameters to correct for lens distortion, measure the size of an object in world units, or determine the location of the camera in the scene • These tasks are used in applications such as machine vision to detect and measure objects. They are also used in robotics, for navigation systems, and 3‐D scene reconstruction Lin ZHANG, SSE, 2016

What is camera calibration? Lin ZHANG, SSE, 2016

What is camera calibration? • Camera parameters include • Intrinsics • Extrinsics • Distortion coefficients Lin ZHANG, SSE, 2016

Contents • Foundations of Projective Geometry • Matrix Differentiation • Lagrange Multiplier • Least‐squares for Linear Systems • Homograpy Estimation • What is Camera Calibration • Single Camera Calibration • Bird’s‐eye‐view Generation Lin ZHANG, SSE, 2016

Single Camera Calibration • For simplicity, usually we use a pinhole camera model Lin ZHANG, SSE, 2016

Single Camera Calibration • To model the image formation process, 4 coordinate systems are required • World coordinate system (3D space) • Camera coordinate system (3D space) • Retinal coordinate system (2D space) • Pixel coordinate system (2D space) Lin ZHANG, SSE, 2016

Single Camera Calibration • To model the image formation process, 4 coordinate systems are required Lin ZHANG, SSE, 2016

Single Camera Calibration • From the world CS to the camera CS   T , , X Y Z is a 3D point represented in the WCS w w w In the camera CS, it is represented as,     X X c w       R t Y Y     c w         Z Z c w  3 1 a translation vector  3 3 a rotation matrix (orthogonal) Lin ZHANG, SSE, 2016

Single Camera Calibration • From the world CS to the camera CS   T , , X Y Z is a 3D point represented in the WCS w w w In the camera CS, it is represented as,     X X     Homogeneous X X c w     c w     form   R t   Y Y R t     Y Y       c w  (1) c w         0 1 T   Z Z     Z Z     c w c w 1 1     Lin ZHANG, SSE, 2016

Single Camera Calibration • From the camera CS to the retinal CS We can use a pin‐hole model to represent the mapping from the camera CS to the retinal CS Lin ZHANG, SSE, 2016

Single Camera Calibration • From the camera CS to the retinal CS We can use a pin‐hole model to represent the mapping from the camera CS to the retinal CS y f z x O is the optical center, f is the focal length, P = [ X c , Y c , Z c ] T is a scene point while P’= ( x, y ) is its image on the retinal plane Lin ZHANG, SSE, 2016

Single Camera Calibration • From the camera CS to the retinal CS We can use a pin‐hole model to represent the mapping from the camera CS to the retinal CS   X   X   c f X        c 0 0 0 Homogeneous  x f c     x Z        form Y    c Y      0 0 0 c (2) Z y f        c    y Y c   Z  c  f        Z  1 0 0 10 c      c   Z 1  c Lin ZHANG, SSE, 2016

Single Camera Calibration • From the retinal CS to the pixel CS The unit for retinal CS ( x-y ) is physical unit (e.g., mm, cm) while the unit for pixel CS ( u-v ) is pixel One pixel represents dx physical units along the x‐axis and represents dy physical units along the y‐axis; the image of the optical center is ( u 0 , v 0 )   1 0 u   0 dx       u x 1        0 v v y       0 dy       1 1     0 0 1       Lin ZHANG, SSE, 2016

Single Camera Calibration • From the retinal CS to the pixel CS If the two axis of the pixel plane are not perpendicular to each other, another parameter s is introduced to represent the skewness of the two axis     1 1 0 s u u     0 0 dx dx             u x u x 1 1               0 0 v v y (3) v v y             0 0 dy dy             1 1 1 1         0 0 1 0 0 1             Lin ZHANG, SSE, 2016

Single Camera Calibration From Eqs.1~3, we can have     1 f 0 s u sf u         0 X 0 X dx dx     c c   0 0 0       u f 1       Y   Y   f     0 . 0 0 0 c 0 0 . c Z v v f v             0 0 c dy Z dy Z         1 0 0 10     c c     0 0 1 0 0 1 0   1   1                   X X X              0 c 0 w w u  u   u   0 0 0   R t        Y Y Y              0 0 . c 0 0 . w 0 . R t w   v v v             0 0 0 0 1 T   Z Z Z             0 0 1 0 c 0 0 1 0 w 0 0 1 w        1 1 1      Intrinsic Extrinsic Lin ZHANG, SSE, 2016

Single Camera Calibration     X X       w w     u u 0     Y Y          K [ R t ] w 0 [ R t ] w Z v v         0 c Z Z     1     w 0 0 1 w     1 1     Intrinsic Extrinsic   , u v , the coordinates of the principal point in the image plane 0 0  and  , the scale factors in image u and v axes  , describing the skewness of the two image axes R and t determines the rigid transformation from the world coordinate system to the camera coordinate system Altogether, there are 11 parameters to be determined Lin ZHANG, SSE, 2016

Single Camera Calibration • To accurately represent an ideal camera, the camera model can include the radial and tangential lens distortion • Radial distortion occurs when light rays bend more near the edges of a lens than they do at its optical center; the smaller the lens, the greater the distortion Lin ZHANG, SSE, 2016

Single Camera Calibration • To accurately represent an ideal camera, the camera model can include the radial and tangential lens distortion • Radial distortion occurs when light rays bend more near the edges of a lens than they do at its optical center; the smaller the lens, the greater the distortion       2 4 6 1 u u k r k r k r 1 2 3 distorted       2 4 6 1 v v k r k r k r 1 2 3 distorted   2 2 2 r u v where , , k k k are the radial distortion coefficients of the lens 1 2 3 Lin ZHANG, SSE, 2016

Single Camera Calibration • To accurately represent an ideal camera, the camera model can include the radial and tangential lens distortion • Tangential distortion occurs when the lens and the image plane are not parallel Lin ZHANG, SSE, 2016

Single Camera Calibration • To accurately represent an ideal camera, the camera model can include the radial and tangential lens distortion • Tangential distortion occurs when the lens and the image plane are not parallel          2 2 + 2 2 u u uv r u 1 2 distorted           2 2 2 2 v v uv r v 2 1 distorted   , are the tangential distortion coefficients of the lens 1 2 Lin ZHANG, SSE, 2016

Single Camera Calibration • The purpose of the camera calibration is to determine the values for the extrinsics, intrinsics, and distortion coefficients • How to do? • Zhengyou Zhang’s method [1] is a commonly used modern approach • A calibration board is needed; several images of the board need to be captured; based on the correspondence pairs (pixel coordinate and world coordinate of a feature point), equation systems can be obtained; by solving the equation systems, parameters can be determined [1] Z. Zhang, A flexible new technique for camera calibration, IEEE Trans. Pattern Analysis and Machine Intelligence, 2000 Lin ZHANG, SSE, 2016

Single Camera Calibration Calibration board Lin ZHANG, SSE, 2016

Single Camera Calibration A set of Calibration board images (50~60) Lin ZHANG, SSE, 2016

Single Camera Calibration • Matlab provides a “Camera Calibrator” • Straightforward to use • However, based on my experience, it is not as accurate as the routine provided in openCV3.0, especially for large FOV cameras (such as fisheye camera); thus, for some accuracy critical applications, I recommend to use the openCV function, though a little more complicated • It exports “cameraParams” as the calibration result Lin ZHANG, SSE, 2016

Single Camera Calibration Lin ZHANG, SSE, 2016

Single Camera Calibration • For our purpose (measuring geometric metrics of a planar object), we use the camera parameters to undistort the image • The essence of this step is to make sure the transformation from a physical plane to the image plane can be represented by a linear projective matrix; or in other words, a straight line should be mapped to a straight line Lin ZHANG, SSE, 2016

Single Camera Calibration Original image Undistorted image Lin ZHANG, SSE, 2016

Contents • Foundations of Projective Geometry • Matrix Differentiation • Lagrange Multiplier • Least‐squares for Linear Systems • Homograpy Estimation • What is Camera Calibration • Single Camera Calibration • Bird’s‐eye‐view Generation Lin ZHANG, SSE, 2016

Bird’s‐eye‐view Generation • Our task is to measure the geometric properties of objects on a plane (e.g., conveyor belt) • Such a problem can be solved if we have its bird‐view image; bird’s‐eye‐view is easy for object detection and measurement Lin ZHANG, SSE, 2016

Bird’s‐eye‐view Generation • Three coordinate systems are required • Bird’s‐eye‐view image coordinate system • World coordinate system • Undistorted image coordinate system Y X X X Similarity Projective Y Y Bird’s‐eye‐view image WCS Undistorted image Lin ZHANG, SSE, 2016

Bird’s‐eye‐view Generation • Basic idea for bird’s‐eye‐view generation Suppose that the transformation matrix from bird’s‐eye‐view to WCS is and the transformation matrix from WCS to the P  B W undistorted image is P  W I   T , ,1 Then, given a position on bird’s‐eye‐view, we can get x y B B its corresponding position in the undistorted image as   x B    x P P y     I W I B W B   1     T , ,1 Then, the intensity of the pixel can be determined x y B B using some interpolation technique based on the x I neighborhood around on the undistorted image Lin ZHANG, SSE, 2016

Bird’s‐eye‐view Generation • Basic idea for bird’s‐eye‐view generation Suppose that the transformation matrix from bird’s‐eye‐view to WCS is and the transformation matrix from WCS to the P  B W undistorted image is P  W I P  P  The key problem is how to obtain and ? B W W I Lin ZHANG, SSE, 2016

Bird’s‐eye‐view Generation • Determine P  B W N (pixels) Y X H X (mm) M (pixels) Y Lin ZHANG, SSE, 2016

Bird’s‐eye‐view Generation • Determine P  B W   T , ,1 For a point on bird’s‐eye‐view, the corresponding x y B B point on the world coordinate system is,   H HN  0   2 M M         x x x W B B         H H    0 y y P y          W B B W B 2 M        1 1 1      0 0 1       Please verify!! Lin ZHANG, SSE, 2016

Bird’s‐eye‐view Generation • Determine P  W I The physical plane (in WCS) and the undistorted image plane can be linked via a homography matrix P  W I  x x P  I W I W   N x , x If we know a set of correspondence pairs , i  Ii Wi 1 P  can be estimated using the least‐square method W I Lin ZHANG, SSE, 2016

Bird’s‐eye‐view Generation • Determine P  W I A set of point correspondence pairs; for each pair, we know its coordinate on the undistorted image plane and its coordinate in the WCS Lin ZHANG, SSE, 2016