image motion 2 Tues. Feb. 6, 2018 1 Overview of Today Abstract - PowerPoint PPT Presentation

COMP 546 Lecture 8 image motion 2 Tues. Feb. 6, 2018 1

Overview of Today • Abstract computational problem: how to estimate the local image velocity ? • Solution based on V1 motion detectors 2

Intensity changes in XY 𝜖 𝜖 𝜖𝑦 𝐽 𝑦, 𝑧 , 𝜖𝑧 𝐽 𝑦, 𝑧 ) ( is the 2D spatial gradient at 𝑦, 𝑧 . 𝑧 + - + + - + - - ∇ 𝐽 - + + - 𝑦 3

Intensity changes in XYT 𝜖 𝜖 𝜖 𝜖𝑦 𝐽 𝑦, 𝑧, 𝑢 , 𝜖𝑧 𝐽 𝑦, 𝑧, 𝑢 , 𝜖𝑢 𝐽 𝑦, 𝑧, 𝑢 ) ( is the 3D spatio-temporal gradient at 𝑦, 𝑧, 𝑢 . 𝑧 ∇ 𝐽 𝑢 𝑦 4

Suppose that image “objects” are “moving”. (What do these terms mean?) 𝑤 𝑦 , 𝑤 𝑧 (𝑦, 𝑧, 𝑢) 5

Intensity conservation Assume a moving point’s intensity doesn’t change over time. 𝐽 𝑦, 𝑧, 𝑢 = 𝐽 𝑦 + 𝑤 𝑦 ∆𝑢, 𝑧 + 𝑤 𝑧 ∆𝑢, 𝑢 + ∆𝑢 This is similar to the assumption made last lecture, namely that the left and right images are related by a (disparity) shift. 6

Taylor series 𝐽 𝑦 + 𝑤 𝑦 ∆𝑢, 𝑧 + 𝑤 𝑧 ∆𝑢, 𝑢 + ∆𝑢 𝜖 = 𝐽 𝑦, 𝑧, 𝑢 + 𝜖𝑦 𝐽 𝑦, 𝑧, 𝑢 𝑤 𝑦 ∆𝑢 𝜖 + 𝜖𝑧 𝐽 𝑦, 𝑧, 𝑢 𝑤 𝑧 ∆𝑢 𝜖 + 𝜖𝑢 𝐽 𝑦, 𝑧, 𝑢 ∆𝑢 + H.O.T. 7

𝐽 𝑦 + ∆𝑦 ≈ 𝐽 𝑦 + 𝜖𝐽 𝜖𝑦 ∆𝑦 𝑦 𝑦 + ∆𝑦 𝜖𝐽 𝐽 𝑦 + 1 − 𝐽(𝑦 − 1) We can estimate by taking . 𝜖𝑦 2 8

𝐝𝐛𝐨𝐝𝐟𝐦 𝐜𝐟𝐝𝐛𝐯𝐭𝐟 𝐩𝐠 𝐣𝐨𝐮𝐟𝐨𝐭𝐣𝐮𝐳 𝐝𝐩𝐨𝐭𝐟𝐬𝐰𝐛𝐮𝐣𝐩𝐨 𝐽 𝑦 + 𝑤 𝑦 ∆𝑢, 𝑧 + 𝑤 𝑧 ∆𝑢, 𝑢 + ∆𝑢 𝜖 = 𝐽 𝑦, 𝑧, 𝑢 + 𝜖𝑦 𝐽 𝑦, 𝑧, 𝑢 𝑤 𝑦 ∆𝑢 𝜖 + 𝜖𝑧 𝐽 𝑦, 𝑧, 𝑢 𝑤 𝑧 ∆𝑢 ≈ 0 𝜖 + 𝜖𝑢 𝐽 𝑦, 𝑧, 𝑢 ∆𝑢 + H.O.T. 9

“Motion Constraint Equation” Previous slides (and assumptions) imply: 𝜖 𝐽 𝜖 𝐽 𝜖 𝐽 𝑤 𝑦 + 𝜖𝑧 𝑤 𝑧 + 𝜖𝑢 = 0 𝜖𝑦 ( 𝜖 𝐽 𝜖𝑦 , 𝜖 𝐽 𝜖𝑧 , 𝜖 𝐽 𝜖𝑢 ) ∙ ( 𝑤 𝑦 , 𝑤 𝑧 , 1 ) = 0 10

Motion constraint equation is a line in velocity space 𝜖 𝐽 𝑤 𝑦 + 𝜖 𝐽 𝜖 𝐽 𝜖𝑧 𝑤 𝑧 + 𝜖𝑢 = 0 𝜖𝑦 𝑤 𝑧 𝑤 𝑦 11

𝜖 𝐽 𝜖 𝐽 𝜖 𝐽 𝑤 𝑦 + 𝜖𝑧 𝑤 𝑧 + 𝜖𝑢 = 0 𝜖𝑦 But many velocities satisfy this equation. 𝑤 𝑧 𝑤 𝑦 12

𝜖 𝐽 𝜖 𝐽 𝜖 𝐽 𝜖𝑦 , 𝜖𝑧 , 𝜖𝑢 ) ∙ ( 𝑤 𝑦 , 𝑤 𝑧 , 1 ) = 0 ( The black vector in the figure below is the 3D image gradient. It is perpendicular to the gray plane. The red vectors are perpendicular to the 3D image gradient (see equation) and thus the red vectors lie in the grey plane. 𝑧 𝑢 𝑦 13

Normal velocity 𝜖 𝐽 𝜖 𝐽 𝜖 𝐽 𝑤 𝑦 + 𝜖𝑧 𝑤 𝑧 + 𝜖𝑢 = 0 𝜖𝑦 𝑤 𝑧 “Normal velocity” is the velocity component in the direction of the XY 𝑤 𝑦 gradient. 14

“Aperture Problem” The same issue arises with any 1D pattern e.g. single bar, edge, constant gradient. 15

Solution of Aperture Problem We need more than one line or edge (or gradient orientation). normal velocities shown Could be in the same aperture or neighboring aperture. 16

“Intersection of Constraints” (IOC) Suppose two nearby points have two different spatial gradients and the same image velocity . 𝑤 𝑧 normal velocities shown 𝑤 𝑦 17

Intersection of Constraints (IOC) 𝜖 𝐽 𝜖 𝐽 𝜖 𝐽 𝜖𝑦 (𝑦 1 , 𝑧 1 , 𝑢) 𝑤 𝑦 + 𝜖𝑧 (𝑦 1 , 𝑧 1 , 𝑢) 𝑤 𝑧 + 𝜖𝑢 (𝑦 1 , 𝑧 1 , 𝑢) = 0 𝜖 𝐽 𝜖 𝐽 𝜖 𝐽 𝜖𝑦 (𝑦 2 , 𝑧 2 , 𝑢) 𝑤 𝑦 + 𝜖𝑧 (𝑦 2 , 𝑧 2 , 𝑢) 𝑤 𝑧 + 𝜖𝑢 (𝑦 2 , 𝑧 2 , 𝑢) = 0 𝑤 𝑧 IOC gives a unique solution. 𝑤 𝑦 18

Another Example (counterintuitive) 𝑤 𝑧 𝑤 𝑦 19

Overview of Today • Abstract computational problem: how to estimate the local image velocity ? • Solution based on V1 motion detectors 20

Recall: XYT Gabor 2𝜌 2𝜌 sin 𝑂 (𝑙 0 𝑦 + 𝑙 1 𝑧) + 𝑈 𝜕 𝑢 𝐻(𝑦, 𝑧, 𝑢, 𝜏 𝑦 , 𝜏 𝑧 , 𝜏 𝑢 ) 𝑧 𝑦 sin 2𝜌 𝑂 (𝑙 0 𝑦 + 𝑙 1 𝑧) + 2𝜌 𝑈 𝜕 𝑢 𝑢 𝑧 𝑦 𝑧 𝑢 𝑢 2𝜌 2𝜌 cos 𝑂 (𝑙 0 𝑦 + 𝑙 1 𝑧) + 𝑈 𝜕 𝑢 𝐻(𝑦, 𝑧, 𝑢, 𝜏 𝑦 , 𝜏 𝑧 , 𝜏 𝑢 ) 𝑦 21

We should get maximum response from a Gabor whose frequency (𝑙 0 , 𝑙 1 , 𝜕 ) is parallel to intensity gradient. (2𝜌 𝑂 𝑙 0 , 2𝜌 2𝜌 𝜖 𝜖 𝜖 𝜖𝑦 𝐽 , 𝜖𝑧 𝐽, 𝜖𝑢 𝐽 ) ( 𝑂 𝑙 1 , 𝑈 𝜕 ) 𝑧 𝑢 𝑦 22

Each motion sensitive V1 cell is sensitive only to the 1D motion component perpendicular to its orientation. 23

Suppose true motion vector 𝑤 𝑦 , 𝑤 𝑧 is toward the right and the image contains lots of oriented structure in each local neighborhood. Which V1 motion cells would have a large response ? 𝑤 𝑧 𝑤 𝑦 24

Suppose true motion vector 𝑤 𝑦 , 𝑤 𝑧 is toward the right and the image contains lots of oriented structure in each local neighborhood. Which V1 motion cells would have a large response ? 𝑤 𝑧 𝑤 𝑦 25

Suppose true motion vector 𝑤 𝑦 , 𝑤 𝑧 is toward the right and the image contains lots of oriented structure in each local neighborhood. Which V1 motion cells would have a large response ? 𝑤 𝑧 𝑤 𝑦 26

Suppose true motion vector 𝑤 𝑦 , 𝑤 𝑧 is toward the right and the image contains lots of oriented structure in each local neighborhood. Which V1 motion cells would have a large response ? 𝑤 𝑧 𝑤 𝑦 27

Motion pathway in the brain (contains V1) MST ⟵ MT ⟵ V1 (temporal lobe contains motion area MT and MST : “middle temporal” & “medial superior temporal”) (next lecture) (today) 28

V1 → MT 𝑤 𝑧 MT cells receive inputs from orientation/motion tuned V1 cells. Many MT cells are velocity 𝑤 𝑦 tuned . They require multiple orientations in their receptive field (to avoid the aperture problem). 29

V1 → MT 𝑤 𝑧 MT cells receive inputs from orientation/motion tuned V1 cells. Many MT cells are velocity 𝑤 𝑦 tuned . They require multiple orientations in their receptive field (to avoid the aperture problem). 30

V1 → MT 𝑤 𝑧 For any velocity vector, there is a family of orientation/motion tuned V1 cells that would respond well, provided 𝑤 𝑦 the moving image contains the spatial orientaiton that this cell is sensitive to. 31

Gabors can have different sizes and spatial frequencies. dot pattern and velocity For full version of the MT model, see [Simoncelli and Heeger Vision Research 1998] 32

Random dot patterns (such as above) contain oriented structure at all frequencies. Later in the course when we discuss linear systems and Fourier transforms, you will learn a more formal (mathematical) statement of what this means. 33