Motion Estimation Lecture 6 Announcement - Project proposal due - PowerPoint PPT Presentation

Motion Estimation Lecture 6

Announcement - Project proposal due on October 16 (next Wednesday) - Come to our office hours to discuss - My OH this week: tomorrow noon - Kevin Zakka started course notes (see Piazza) - bonus points for contributing

What will you take home today? Optical Flow What is it and why do you care? Assumptions Formulating the optimization problem Solving it Scene Flow Learning-based Approaches to Estimating Motion

Optical Flow - What is it? J. J. Gibson, The Ecological Approach to Visual Perception

Optical Flow - What is it? Image Credit: Wikipedia. Optical Flow.

Optical flow - What is it? B. Horn, Robot Vision, MIT Press Motion field = 2D motion field representing the projection of the 3D motion of points in the scene onto the image plane.

Optical flow - What is it? B. Horn, Robot Vision, MIT Press Optical flow = 2D velocity field describing the apparent motion in the images.

What is the motion field? What is the apparent motion? Lambertian (matte) ball rotating in 3D What does the 2D motion field look like? What does the 2D optical flow field look like? Slide Credit: Michael Black Image source: http://www.evl.uic.edu/aej/488/lecture12.html

What is the motion field? What is the apparent motion? Stationary Lambertian (matte) ball Moving Light Source What does the 2D motion field look like? What does the 2D optical flow field look like? Slide Credit: Michael Black Image source: http://www.evl.uic.edu/aej/488/lecture12.html

Optical flow - What is it? Motion Displacement of all image pixels Key Image pixel value at time t and 𝑣 𝑦, 𝑧 horizontal component Location 𝐲 = 𝑦, 𝑧 : 𝑤 𝑦, 𝑧 vertical component 𝐽(𝑦, 𝑧, 𝑢) Slide Credit: Michael Black

Optical Flow - What is it good for? Painterly effect Slide Credit: Michael Black

Optical Flow - What is it good for? Face morphing in matrix reloaded Slide Credit: Michael Black

Optical Flow - What is it good for? Slide Credit: Michael Black

Optical Flow - What is it good for? Slide Credit: Michael Black

Optical Flow - What is it good for?

Optical Flow - What is it good for? ! p v 2 2 ! p v 3 ! 3 p v 1 1 Optical Flow ! p v 4 4 ! I ( t ), { p } { i v } Velocity vectors i Slide Credit: CS223b – Sebastian Thrun

Compute Optical Flow Goal Compute the apparent 2D image motion of pixels from one image frame to the next in a video sequence.

Compute Optical Flow Step 1 - Assumptions Step 2 - Objective Function Step 3 - Optimization Source: Wikipedia.

Assumption 1 - Brightness Constancy 𝐽 𝑦 + 𝑣, 𝑧 + 𝑤, 𝑢 + 1 = 𝐽(𝑦, 𝑧, 𝑢) Slide Credit: Michael Black

Assumption 2 - Spatial Smoothness Slide Credit: Michael Black

Assumption 3 – Temporal Coherence Slide Credit: Michael Black

Compute Optical Flow Step 1 - Assumptions Step 2 - Objective Function Source: Wikipedia.

Optimization Function (𝐽 𝑦 3 + 𝑣 3 , 𝑧 3 + 𝑤 3 , 𝑢 + 1 − 𝐽 𝑦, 𝑧, 𝑢 ) 5 𝐹 / 𝐯, 𝐰 = 2 3 New Assumption: Gaussian noise

Optimization Function (𝑣 3 − 𝑣 7 ) 5 + 5 𝐹 6 𝐯, 𝐰 = 2 2 𝑤 3 − 𝑤 7 7∈9(3) 7∈9(3) New Assumptions: Flow field smooth Gaussian Deviations First order smoothness good enough Flow derivative approximated by first differences

Optimization Function 𝐹 𝑣, 𝑤 = ∑ 3 (𝐽 𝑦 3 + 𝑣 3 , 𝑧 3 + 𝑤 3 , 𝑢 + 1 − 𝐽 𝑦, 𝑧, 𝑢 ) 5 + 𝜇 ∑ 7∈9(3) (𝑣 3 − 𝑣 7 ) 5 + ∑ 7∈9(3) 𝑤 3 − 𝑤 7 5

Compute Optical Flow Step 1 - Assumptions Step 2 - Objective Function Step 3 - Optimization Source: Wikipedia.

Linear Approximation 𝐹 𝑣, 𝑤 = ∑ 3 (𝐽 𝑦 3 + 𝑣 3 , 𝑧 3 + 𝑤 3 , 𝑢 + 1 − 𝐽 𝑦, 𝑧, 𝑢 ) 5 + 𝜇 ∑ 7∈9(3) (𝑣 3 − 𝑣 7 ) 5 + ∑ 7∈9(3) 𝑤 3 − 𝑤 7 5 𝑣 3 = 𝑒𝑦, 𝑤 3 = 𝑒𝑧, 𝑒𝑢 = 1 𝐽 𝑦, 𝑧, 𝑢 + 𝑒𝑦 𝜀 𝜀𝑦 𝐽 𝑦, 𝑧, 𝑢 + 𝑒𝑧 𝜀 𝜀𝑧 𝐽 𝑦, 𝑧, 𝑢 + 𝑒𝑢 𝜀 𝜀𝑢 𝐽 𝑦, 𝑧, 𝑢 − 𝐽 𝑦, 𝑧, 𝑢 = 0

Optical Flow Constraint Equation 𝑣 𝜀 𝜀𝑦 𝐽 𝑦, 𝑧, 𝑢 + 𝑤 𝜀 𝜀𝑧 𝐽 𝑦, 𝑧, 𝑢 + 𝜀 𝜀𝑢 𝐽 𝑦, 𝑧, 𝑢 = 0 𝐽 ? 𝑣 + 𝐽 @ 𝑤 + 𝐽 A = 0 New Assumptions: Flow is small Image is differentiable First order Taylor series is a good approximation

Optical Flow Constraint Equation

Aperture Problem Slide Credit: CS223b – Sebastian Thrun

Aperture Problem Slide Credit: CS223b – Sebastian Thrun

What are the constraint lines? 𝑤 A C B 𝑣

Multiple Constraints Slide Credit: Michael Black

How do we solve this optimization problem?

How do we solve this optimization problem?

How do we solve this optimization problem?

How do we solve this optimization problem?

Image Gradient Examples - Edge

Image Gradient Examples – Low texture

Image Gradient Examples – Low texture

Bag of tricks Small motion assumption

Bag of tricks Reduce Resolution * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

u=1.25 pixels u=2.5 pixels u=5 pixels u=10 pixels image I t-1 image I image I image I t-1 Gaussian pyramid of image I t-1 Gaussian pyramid of image I

Scene Flow

What are the main challenges with this traditional formulation? 1. Assumptions a. Brightness constancy b. Small motion c. Etc 2. Occlusions 3. Large motion

Learning-based approaches 1. Since 2015 2. Availability of data

FlowNet - Learning Optical Flow with Convolutional Networks Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, P. Häusser, C. Hazırbaş, V. Golkov, P. Smagt, D. Cremers, Thomas Brox. IEEE International Conference on Computer Vision (ICCV), 2015

FlowNet - Learning Optical Flow with Convolutional Networks

Motion-based Object Segmentation based on Dense RGB-D Scene Flow

Motion-based Object Segmentation based on Dense RGB-D Scene Flow

Motion-based Object Segmentation based on Dense RGB-D Scene Flow

Motion-based Object Segmentation based on Dense RGB-D Scene Flow

Motion-based Object Segmentation based on Dense RGB-D Scene Flow