Optical Flow: Horn-Schunck 16-385 Computer Vision (Kris Kitani) - PowerPoint PPT Presentation

Optical Flow: Horn-Schunck 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University

Horn-Schunck Lucas-Kanade Optical Flow (1981) Optical Flow (1981) brightness constancy method of differences small motion ‘smooth’ flow ‘constant’ flow (flow can vary from pixel to pixel) (flow is constant for all pixels) global method local method (dense) (sparse)

Optical Flow (Problem definition) I ( x, y, t 0 ) I ( x, y, t ) Estimate the motion (flow) between these two consecutive images How is this different from estimating a 2D transform?

Key Assumptions (unique to optical flow) Color Constancy (Brightness constancy for intensity images) Implication: allows for pixel to pixel comparison (not image features) Small Motion (pixels only move a little bit) Implication: linearization of the brightness constancy constraint

Approach I ( x, y, t 0 ) I ( x, y, t ) Look for nearby pixels with the same color (small motion) (color constancy)

Brightness constancy ( x + u δ t, y + v δ t ) ( x, y ) ( x, y ) I ( x, y, t ) I ( x, y, t + δ t ) ( u, v ) ( δ x, δ y ) = ( u δ t, v δ t ) Optical flow (velocities): Displacement: I ( x + u δ t, y + v δ t, t + δ t ) = I ( x, y, t ) For a really small time step…

Optical Flow Constraint equation I ( x + u δ t, y + v δ t, t + δ t ) = I ( x, y, t ) I ( x, y, t ) + ∂ I ∂ x δ x + ∂ I ∂ y δ y + ∂ I ∂ t δ t = I ( x, y, t ) ∂ I dt + ∂ I dt + ∂ I dx dy ∂ t = 0 ∂ x ∂ y I x u + I y v + I t = 0

Solution lies on a straight line u I x u + I y v + I t = 0 v The solution cannot be determined uniquely with a single constraint (a single pixel)

Where can we get an additional constraint?

Horn-Schunck Lucas-Kanade Optical Flow (1981) Optical Flow (1981) brightness constancy method of differences small motion ‘smooth’ flow ‘constant’ flow (flow is smooth from pixel to pixel) (flow is constant for all pixels) global method local method

Smoothness most objects in the world are rigid or deform elastically moving together coherently we expect optical flow fields to be smooth

Key idea (of Horn-Schunck optical flow) Enforce brightness constancy Enforce smooth flow field to compute optical flow

Key idea (of Horn-Schunck optical flow) Enforce brightness constancy Enforce smooth flow field to compute optical flow

Enforce 
 brightness constancy I x u + I y v + I t = 0 For every pixel, � 2  I x u ij + I y v ij + I t min u,v

Enforce 
 brightness constancy I x u + I y v + I t = 0 For every pixel, � 2  I x u ij + I y v ij + I t min u,v lazy notation for I x ( i, j )

displacement of object Find the optical flow such that it satisfies: I y I x I t + = changes these changes these u v

Key idea (of Horn-Schunck optical flow) Enforce brightness constancy Enforce smooth flow field to compute optical flow

Enforce smooth flow field u i,j +1 u ( u i,j − u i +1 ,j ) 2 u ij u i − 1 ,j u i +1 ,j min u i,j − 1 u-component of flow

u ( u i,j − u i +1 ,j ) 2 min Which flow field optimizes the objective? X X ( u ij − u i +1 ,j ) 2 ? ( u ij − u i +1 ,j ) 2 ij ij

u ( u i,j − u i +1 ,j ) 2 min Which flow field optimizes the objective? big small

Key idea (of Horn-Schunck optical flow) Enforce brightness constancy Enforce smooth flow field to compute optical flow bringing it all together…

Horn-Schunck optical flow ⇢ � smoothness brightness constancy X min E s ( i, j ) + λ E d ( i, j ) u , v i,j weight

HS optical flow objective function � 2  Brightness constancy E d ( i, j ) = I x u ij + I y v ij + I t Smoothness  � E s ( i, j ) = 1 ( u ij − u i +1 ,j ) 2 + ( u ij − u i,j +1 ) 2 + ( v ij − v i +1 ,j ) 2 + ( v ij − v i,j +1 ) 2 4 i, j + 1 i, j + 1 i, j + 1 i, j + 1 ( v ij − v i +1 ,j ) ( u ij − u i +1 ,j ) ( u ij − u i,j +1 ) ( v ij − v i,j +1 ) ij i + 1 , j ij i + 1 , j ij i + 1 , j i − 1 , j ij i + 1 , j i − 1 , j i − 1 , j i − 1 , j i, j − 1 i, j − 1 i, j − 1 i, j − 1 why not all four neighbors?

How do we solve this minimization problem? ⇢ � X min E s ( i, j ) + λ E d ( i, j ) u , v i,j

How do we solve this minimization problem? ⇢ � X min E s ( i, j ) + λ E d ( i, j ) u , v i,j Compute partial derivative, derive update equations (gradient decent!)

Compute the partial derivatives of this huge sum! ( � 2 ) 1  �  ( u ij − u i +1 ,j ) 2 + ( u ij − u i,j +1 ) 2 + ( v ij − v i +1 ,j ) 2 + ( v ij − v i,j +1 ) 2 X + λ I x u ij + I y v ij + I t 4 ij smoothness term brightness constancy

Compute the partial derivatives of this huge sum! ( � 2 ) 1  �  ( u ij − u i +1 ,j ) 2 + ( u ij − u i,j +1 ) 2 + ( v ij − v i +1 ,j ) 2 + ( v ij − v i,j +1 ) 2 X + λ I x u ij + I y v ij + I t 4 ij it’s not so bad… ∂ E = ∂ u kl how many u terms depend on k and l?

Compute the partial derivatives of this huge sum! ( � 2 ) 1  �  ( u ij − u i +1 ,j ) 2 + ( u ij − u i,j +1 ) 2 + ( v ij − v i +1 ,j ) 2 + ( v ij − v i,j +1 ) 2 X + λ I x u ij + I y v ij + I t 4 ij it’s not so bad… ∂ E = ∂ u kl how many u terms depend on k and l? FOUR from smoothness ONE from brightness constancy

Compute the partial derivatives of this huge sum! ( � 2 ) 1  �  ( u ij − u i +1 ,j ) 2 + ( u ij − u i,j +1 ) 2 + ( v ij − v i +1 ,j ) 2 + ( v ij − v i,j +1 ) 2 X + λ I x u ij + I y v ij + I t 4 ij it’s not so bad… ∂ E = 2( u kl − ¯ u kl ) + 2 λ ( I x u kl + I y v kl + I t ) I x ∂ u kl how many u terms depend on k and l? FOUR from smoothness ONE from brightness constancy

Compute the partial derivatives of this huge sum! ( � 2 ) 1  �  ( u ij − u i +1 ,j ) 2 + ( u ij − u i,j +1 ) 2 + ( v ij − v i +1 ,j ) 2 + ( v ij − v i,j +1 ) 2 X + λ I x u ij + I y v ij + I t 4 ij i, j + 1 ( u ij − u i +1 ,j ) ( u 2 ij − 2 u ij u i +1 ,j + u 2 i +1 ,j ) ( u 2 ij − 2 u ij u i,j +1 + u 2 i,j +1 ) i − 1 , j ij i + 1 , j (variable will appear four times in sum) i, j − 1

Compute the partial derivatives of this huge sum! ( � 2 ) 1  �  ( u ij − u i +1 ,j ) 2 + ( u ij − u i,j +1 ) 2 + ( v ij − v i +1 ,j ) 2 + ( v ij − v i,j +1 ) 2 X + λ I x u ij + I y v ij + I t 4 ij i, j + 1 ( u ij − u i +1 ,j ) ( u 2 ij − 2 u ij u i +1 ,j + u 2 i +1 ,j ) ( u 2 ij − 2 u ij u i,j +1 + u 2 i,j +1 ) i − 1 , j ij i + 1 , j (variable will appear four times in sum) i, j − 1 ∂ E = 2( u kl − ¯ u kl ) + 2 λ ( I x u kl + I y v kl + I t ) I x ∂ u kl ∂ E = 2( v kl − ¯ v kl ) + 2 λ ( I x u kl + I y v kl + I t ) I y ∂ v kl ⇢ � u ij = 1 short hand for ¯ u i +1 ,j + u i − 1 ,j + u i,j +1 + u i,j − 1 local average 4

∂ E = 2( u kl − ¯ u kl ) + 2 λ ( I x u kl + I y v kl + I t ) I x ∂ u kl ∂ E = 2( v kl − ¯ v kl ) + 2 λ ( I x u kl + I y v kl + I t ) I y ∂ v kl Where are the extrema of E?

∂ E = 2( u kl − ¯ u kl ) + 2 λ ( I x u kl + I y v kl + I t ) I x ∂ u kl ∂ E = 2( v kl − ¯ v kl ) + 2 λ ( I x u kl + I y v kl + I t ) I y ∂ v kl Where are the extrema of E? (set derivatives to zero and solve for unknowns u and v) (1 + λ I 2 x ) u kl + λ I x I y v kl = ¯ u kl − λ I x I t λ I x I y u kl + (1 + λ I 2 y ) v kl = ¯ v kl − λ I y I t

∂ E = 2( u kl − ¯ u kl ) + 2 λ ( I x u kl + I y v kl + I t ) I x ∂ u kl ∂ E = 2( v kl − ¯ v kl ) + 2 λ ( I x u kl + I y v kl + I t ) I y ∂ v kl Where are the extrema of E? (set derivatives to zero and solve for unknowns u and v) (1 + λ I 2 x ) u kl + λ I x I y v kl = ¯ u kl − λ I x I t λ I x I y u kl + (1 + λ I 2 y ) v kl = ¯ v kl − λ I y I t A x = b this is a linear system how do you solve this?

ok, take a step back, why are we doing all this math?

We are solving for the optical flow (u,v) given two constraints ( � 2 ) 1  �  ( u ij − u i +1 ,j ) 2 + ( u ij − u i,j +1 ) 2 + ( v ij − v i +1 ,j ) 2 + ( v ij − v i,j +1 ) 2 X + λ I x u ij + I y v ij + I t 4 ij smoothness brightness constancy We need the math to minimize this (back to the math)

Partial derivatives of Horn-Schunck objective function E: ∂ E = 2( u kl − ¯ u kl ) + 2 λ ( I x u kl + I y v kl + I t ) I x ∂ u kl ∂ E = 2( v kl − ¯ v kl ) + 2 λ ( I x u kl + I y v kl + I t ) I y ∂ v kl Where are the extrema of E? (set derivatives to zero and solve for unknowns u and v) (1 + λ I 2 x ) u kl + λ I x I y v kl = ¯ u kl − λ I x I t λ I x I y u kl + (1 + λ I 2 y ) v kl = ¯ v kl − λ I y I t A x = b how do you solve this?

(1 + λ I 2 x ) u kl + λ I x I y v kl = ¯ u kl − λ I x I t λ I x I y u kl + (1 + λ I 2 y ) v kl = ¯ v kl − λ I y I t x = A − 1 b = adj A det A b Recall

(1 + λ I 2 x ) u kl + λ I x I y v kl = ¯ u kl − λ I x I t λ I x I y u kl + (1 + λ I 2 y ) v kl = ¯ v kl − λ I y I t x = A − 1 b = adj A det A b Recall Same as the linear system: { 1 + λ ( I 2 x + I 2 y ) } u kl = (1 + λ I 2 x )¯ u kl − λ I x I y ¯ v kl − λ I x I t (det A) { 1 + λ ( I 2 x + I 2 y ) } v kl = (1 + λ I 2 y )¯ v kl − λ I x I y ¯ u kl − λ I y I t (det A)