Cue combinations, Bayesian models Thurs. March 1, 2018 1 Visual - PowerPoint PPT Presentation

COMP 546 Lecture 15 Cue combinations, Bayesian models Thurs. March 1, 2018 1

Visual Cues: image properties that can tell us about scene properties Image Scene texture depth gradient - size, shape, density - slant, tilt shading surface curvature binocular disparities depth motion (from moving observer) defocus blur 2

Last lecture: Likelihood 𝑞 𝐽 = 𝑗 𝑇 = 𝑡 ) • Probability of measuring image 𝐽 = 𝑗, when the scene is 𝑇 = 𝑡. (called “likelihood” of scene 𝑇 = 𝑡 , given the image 𝐽 = 𝑗 ). • Maximum likelihood method: Choose 𝑇 = 𝑡 that maximizes 𝑞 𝐽 = 𝑗 𝑇 = 𝑡 ) 3

This lecture: How to combine cues ? 𝑞 𝐽 1 , 𝐽 2 𝑇 ) 4

Example: texture only (monocular) stereo only texture and stereo [Hillis 2004] 5

Assume likelihood function is “conditionally independent”: 𝑞 𝐽 1 , 𝐽 2 𝑇 ) = 𝑞 𝐽 1 𝑇 ) 𝑞 𝐽 2 𝑇 ) e.g. 𝐽 1 is texture. 𝐽 2 is binocular disparity. 6

𝑞 𝐽 2 𝑇 ) 𝑞 𝐽 1 𝑇 ) 𝑇 = s Assume 𝑞 𝐽 1 = 𝑗 1 𝑇 = 𝑡 ) and 𝑞 𝐽 2 = 𝑗 2 𝑇 = 𝑡 ) are Gaussian shaped. 7

𝑞 𝐽 2 𝑇 ) 𝑞 𝐽 1 𝑇 ) 𝑇 = s 𝑡 1 𝑡 2 Assume 𝑞 𝐽 1 = 𝑗 1 𝑇 = 𝑡 ) and 𝑞 𝐽 2 = 𝑗 2 𝑇 = 𝑡 ) are Gaussian shaped. Their maxima might occur at different values of 𝑡 . Why ? 8

We want to find the 𝑡 that maximizes: − 𝑡 − 𝑡 1 2 − 𝑡 − 𝑡 2 2 2 𝜏 12 2 𝜏 22 𝑞 𝐽 1 | 𝑇 = 𝑡 𝑞 𝐽 2 | 𝑇 = 𝑡 = 𝑓 𝑓

We want to find the 𝑡 that maximizes: − 𝑡 − 𝑡 1 2 − 𝑡 − 𝑡 2 2 2 𝜏 12 2 𝜏 22 𝑞 𝐽 1 | 𝑇 = 𝑡 𝑞 𝐽 2 | 𝑇 = 𝑡 = 𝑓 𝑓 So, we want to find the 𝑡 that minimizes:

The lecture notes show that the solution 𝑇 = 𝑡 is 𝑡 = 𝑥 1 𝑡 1 + 𝑥 2 𝑡 2 where 𝑥 1 + 𝑥 2 = 1 0 < 𝑥 𝑗 < 1 “Linear Cue Combination”

The lecture notes show that the solution 𝑇 = 𝑡 is 𝑡 = 𝑥 1 𝑡 1 + 𝑥 2 𝑡 2 where 𝑥 1 + 𝑥 2 = 1 0 < 𝑥 𝑗 < 1 𝜏 2 2 𝜏 1 2 𝑥 1 = 𝑥 2 = 𝜏 12 + 𝜏 22 𝜏 12 + 𝜏 22 Thus, less reliable cue (larger 𝜏 ) get less weight.

Example: [Hillis 2004] texture only (monocular) stereo only Measure slant discrimination thresholds for cues in isolation . Estimate likelihood function parameters ( 𝑡 1 , 𝜏 1 , 𝑡 2 , 𝜏 2 ). 13

… then • present cues together texture and stereo • measure thresholds for 𝑇 • convert thresholds to likelihood parameters ( 𝑡 , σ ) 14

… then • present cues together texture and stereo • measure thresholds for 𝑇 • convert thresholds to likelihood parameters ( 𝑡 , σ ) • examine if these values are consistent with the model* 𝑡 = 𝑥 1 𝑡 1 + 𝑥 2 𝑡 2 *Model also makes prediction about σ in combined case. 15

𝑞 𝐽 2 𝑇 ) 𝑞 𝐽 1 𝑇 ) texture and stereo 𝑇 = s 𝑡 1 𝑡 2 Experimenter can manipulate 𝑡 1 , 𝑡 2 , 𝜏 1 , 𝜏 2 and predict effect on perception of slant. 16

COMP 546 Lecture 15 Cue combinations, Bayesian models Thurs. March 1, 2018 17

𝑞 𝐽 = 𝑗 𝑇 = 𝑡) ≠ 𝑞 𝑇 = 𝑡 𝐽 = 𝑗) Likelihood of scene 𝑡 , Probability of scene 𝑡 , given image 𝑗 given image 𝑗 What is the crucial difference ? 18

wire frame with independently chosen depths regular solid cube flat drawing All scenes above have the same likelihood 𝑞( 𝐽 = 𝑗 | 𝑇 = 𝑡 ). Why do we prefer the regular solid cube? [Kersten & Yuille 2003]

Some scenes may have a larger probability 𝑞(𝑇 = 𝑡 ). The marginal probably 𝑞(𝑇 = 𝑡 ) is called the "prior".

𝑞(𝐽, 𝑇 ) 𝑞 𝐽 𝑇 ) ≡ 𝑞(𝑇) 𝑞 (𝐽, 𝑇 ) 𝑞 𝑇 𝐽 ) ≡ 𝑞(𝐽) Thus, 𝑞 𝐽 𝑇 ) 𝑞 𝑇 = 𝑞 𝑇 𝐽 ) 𝑞 𝐽

Bayes Theorem 𝑞(𝐽, 𝑇 ) 𝑞 𝐽 𝑇 ) ≡ 𝑞(𝑇) 𝑞 (𝐽, 𝑇 ) 𝑞 𝑇 𝐽 ) ≡ 𝑞(𝐽) Thus, likelihood scene prior 𝑞 𝐽 𝑇 ) 𝑞 𝑇 𝑞 𝑇 𝐽 ) = 𝑞 𝐽 image prior posterior

Maximum ‘ a Posteriori’ (MAP) Given an image, 𝐽 = 𝑗, find the scene 𝑇 = 𝑡 that maximizes 𝑞( 𝑇 = 𝑡 | 𝐽 = 𝑗 ). likelihood scene prior 𝑞 𝐽 𝑇 ) 𝑞 𝑇 𝑞 𝑇 𝐽 ) = 𝑞 𝐽 image prior posterior

Maximum ‘ a Posteriori’ (MAP) Given an image, 𝐽 = 𝑗, find the scene 𝑇 = 𝑡 that maximizes 𝑞( 𝑇 = 𝑡 | 𝐽 = 𝑗 ). We don't care about 𝑞( 𝐽 = 𝑗 ). Why not ? likelihood scene prior 𝑞 𝐽 𝑇 ) 𝑞 𝑇 𝑞 𝑇 𝐽 ) = 𝑞 𝐽 image prior posterior

If the prior p(S) is uniform then maximum likelihood gives the same solution as maximum posterior (MAP). likelihood scene prior constant 𝑞 𝐽 𝑇 ) 𝑞 𝑇 𝑞 𝑇 𝐽 ) = 𝑞 𝐽 image prior posterior Interesting cases arise when the prior is non-uniform.

likelihood prior

Ames Room http://www.youtube.com/watch?v=Ttd0YjXF0no https://www.youtube.com/watch?v=gJhyu6nlGt8

Priors (“Natural Scenes Statistics”) • intensity • orientation of image lines, edges • disparity • motion • surface slant, tilt

orientation 𝜄 of lines, edges 𝑞(𝑇 = 𝜄) [Girshick 2011] People are indeed better at discriminating vertical and horizontal orientations than oblique orientations. Why? Because they use a prior ?

surface slant 𝜏 and tilt 𝜐 ceiling floor Here we represent (slant, tilt) using a concave hemisphere. See next slide.

𝑞(𝑇 = (𝜏, 𝜐)) Each disk shows 𝑞(𝜏, 𝜐) for surfaces represent slants and tilts using a concave visible over a range of viewing direction elevations, relative to line of sight. [Adams & Elder 2016]

𝑞(𝑇 = (𝜏, 𝜐))

𝑞(𝑇 = (𝜏, 𝜐))

Maximum a Posteriori (MAP) Choose the S = (slant,tilt) that maximizes the posterior. ∗ 𝑞( 𝑇 ) = 𝑞(𝐽 = 𝑗 | 𝑇 ) 𝑞 𝑇 𝐽 = 𝑗 ) posterior likelihood prior

Likelihood functions can have more than one maximum. overall (slant, tilt) 𝑞(𝐽 = 𝑗 | 𝑇 ) i.e. convex or concave ?

Depth Reversal Ambiguity and Shading (see Exercise) Likelihood (slant, tilt) 𝑞(𝐽 = 𝑗 | 𝑇 ) A valley illuminated from the right produces the same shading as a hill illuminated from the left.

What “priors” does the visual system use to resolve such twofold ambiguities ? Let’s look at a few related examples.

You can perceive the center point as a hill or a valley. When you see it as a hill, you perceive the tilt as 180 deg (leftward). But when you see it as a valley, the slant is 0 (rightward).

We tend to see the center as a hill. Why ?

We tend to see the center as a valley. Why ?

The visual system uses three priors to resolve the depth reversal ambiguity: - surface orientation: p(floor) > p(ceiling) - light source direction: p( above) > p( below) - ‘global’ surface curvature: p(convex) > p(concave)

Example in which all three priors assumptions are met light from above viewpoint from above (floor) shape is convex

Example in which all three prior assumptions fail shape is concave viewpoint from below (ceiling) light from below

Convex shape, illuminated from above the line of sight floor ceiling

Concave shape, illuminated from below the line of sight ceiling floor

We showed how people combined the three different "priors": Percent correct in judging local "hill" or "valley": = 50 +/- 10 floor vs. ceiling +/- 10 light from above vs. below +/- 10 globally convex/concave [Langer and Buelthoff, 2001]

Best Worst (80%) (20%)

These look weird, but in different ways. How ?

Reminder • A2 is due tonight • Midterm (optional) is first class after Study Break