Designing Applications that See Lecture 3: Image Processing Dan - PowerPoint PPT Presentation

Color Spaces � Use color matching functions to define a coordinate system for color � Each color can be assigned a triple of coordinates with respect to some color space (e.g. RGB) � Devices (monitors, printers, projectors) and computers can communicate colors precisely 15 January 2008 Lecture 3: Image Processing 60

McAdam Ellipses (10 times actual size) (Actual size) (courtesy of D. Forsyth) 15 January 2008 Lecture 3: Image Processing 61

Human Color Constancy � Distinguish between � Color constancy, which refers to hue and saturation � Lightness constancy, which refers to gray-level. � Humans can perceive � Color a surface would have under white light (surface color) � Color of the reflected light (limited ability to separate surface color from measured color) � Color of illuminant (even more limited) (courtesy of J.M. Rehg) 15 January 2008 Lecture 3: Image Processing 62

Spatial Arrangement and Color Perception 15 January 2008 Lecture 3: Image Processing 63

Spatial Arrangement and Color Perception 15 January 2008 Lecture 3: Image Processing 64

Spatial Arrangement and Color Perception 15 January 2008 Lecture 3: Image Processing 65

Spatial Arrangement and Color Perception 15 January 2008 Lecture 3: Image Processing 66

Spatial Arrangement and Color Perception 15 January 2008 Lecture 3: Image Processing 67

(courtesy of D. Forsyth) 15 January 2008 Lecture 3: Image Processing 68

Land’s Mondrian Experiments � Squares of color with the same color radiance yield very different color perceptions Photometer: 1.0, 0.3, 0.3 Photometer: 1.0, 0.3, 0.3 Colored light White light Audience: “Red” Audience: “Blue” (courtesy of J.M. Rehg) 15 January 2008 Lecture 3: Image Processing 69

Lightness Constancy Algorithm � The goal is to determine what the surfaces in the image would look like under white light. � Compares the brightness of patches across their common boundaries and computes relative brightness � Establish an absolute reference for lightness (e.g. brightest point is “white”) 15 January 2008 Lecture 3: Image Processing 70

Lightness Constancy Example (courtesy of John McCann) 15 January 2008 Lecture 3: Image Processing 71

Finite Dimensional Linear Models m ∑ ( ) = ( ) E λ ε i ψ i λ i = 1 ⎛ ⎞ ⎛ ⎞ m n ( ) ( ) ( ) ∑ ∑ ∫ ∫ ⎜ ⎟ = σ λ ⎜ ε ψ λ ⎟ ϕ λ λ L r d ⎜ ⎜ ⎟ ⎟ k k i i j j ⎝ ⎝ ⎠ ⎠ ⎝ ⎝ ⎠ ⎠ = = i 1 j 1 m , n ( ) ( ) ( ) ∑ ∫ = ε σ λ ψ λ ϕ λ λ r d i j k i j = = i 1 , j 1 m , n ∑ = ε r g n ∑ ( ) = ( ) i j ijk ρ λ r j ϕ j λ = = i 1 , j 1 j = 1 15 January 2008 Lecture 3: Image Processing 72

From Images to Objects and Regions � Attributes of regions � Bounding edges � Texture � The need to compute and reason about spatial aggregations of pixels leads us to filtering � Key problem is segmentation 15 January 2008 Lecture 3: Image Processing 73

Features and Filters: Questions � What is a feature? � What is an image filter? What is it good for? � How can we find corners? f d � How can we find edges? 15 January 2008 Lecture 3: Image Processing 74

What is a Feature? � In computer vision, a feature is a local, meaningful, detectable part of an image 15 January 2008 Lecture 3: Image Processing 75

Why Use Features? � High information content � Invariant to changing viewpoint or changing illumination � R d � Reduces computational burden t ti l b d (courtesy of Sebastian Thrun) 15 January 2008 Lecture 3: Image Processing 76

One Type of Computer Vision Image 1 Feature 1 Computer Feature 2 Vision : Algorithm Feature N Image 2 Feature 1 Feature 2 : Feature N (courtesy of Sebastian Thrun) 15 January 2008 Lecture 3: Image Processing 77

Where Features Are Used � Calibration � Image Segmentation � Correspondence in multiple images (stereo, structure from motion) t t f ti ) � Object detection, classification (courtesy of Sebastian Thrun) 15 January 2008 Lecture 3: Image Processing 78

What Makes a Good Feature? � Invariance � View point (scale, orientation, translation) � Lighting condition � Object deformations � Partial occlusion � Other Characteristics � Uniqueness � Sufficiently many � Tuned to the task (courtesy of Sebastian Thrun) 15 January 2008 Lecture 3: Image Processing 79

What is Image Filtering? � Modify the pixels in an image based on some function of a local neighborhood of the pixels 10 5 3 4 5 1 7 1 1 7 15 January 2008 Lecture 3: Image Processing 80

Linear Filtering � Linear case is simplest and most useful � Replace each pixel with a linear combination of its neighbors. � The specification of the linear combination is called the convolution kernel. called the convolution kernel. 10 5 3 0 0 0 = * 4 5 1 7 0 0.5 0 1 1 7 0 1.0 0.5 kernel 15 January 2008 Lecture 3: Image Processing 81

I(.) I(.) I(.) I(.) I(.) I(.) I(.) I(.) I(.) g 11 g 12 g 13 g 22 g 23 g 21 g 31 g 32 g 33 + g 12 I(i-1,j) + g 13 I(i-1,j+1) + f (i,j) = g 11 I(i-1,j-1) + g 22 I(i,j) + g 23 I(i,j+1) + g 21 I(i,j-1) + g 32 I(i+1,j) + g 33 I(i+1,j+1) g 31 I(i+1,j-1) 15 January 2008 Lecture 3: Image Processing 82

1 1 1 2 2 2 3 Step 1 -1 2 1 2 1 3 3 -1 -1 1 2 2 1 2 1 3 2 2 1 1 1 -1 1 4 4 2 2 2 2 2 2 2 2 3 3 5 5 2 1 3 3 -1 -2 1 2 2 1 2 1 3 2 2 (courtesy of Christopher Rasmussen)

1 1 1 2 2 2 3 Step 2 -1 2 1 2 1 3 3 -1 -1 1 2 2 1 2 1 3 2 2 1 1 1 -2 2 2 2 4 4 2 2 2 2 2 2 3 3 5 5 4 4 2 1 3 3 -2 -1 3 2 2 1 2 1 3 2 2 (courtesy of Christopher Rasmussen)

1 1 1 2 2 2 3 Step 3 -1 2 1 2 1 3 3 -1 -1 1 2 2 1 2 1 3 2 2 1 1 1 2 2 -2 2 2 2 4 4 2 2 3 3 3 3 5 5 4 4 4 4 2 1 3 3 -1 -3 3 2 2 1 2 1 3 2 2 (courtesy of Christopher Rasmussen)

1 1 1 2 2 2 3 Step 4 -1 2 1 2 1 3 3 -1 -1 1 2 2 1 2 1 3 2 2 1 1 1 2 2 2 2 -2 2 2 2 3 6 6 3 1 1 5 5 4 4 4 4 -2 2 2 1 3 3 -3 -3 1 2 2 1 2 1 3 2 2 (courtesy of Christopher Rasmussen)

1 1 1 2 2 2 3 Step 5 -1 2 1 2 1 3 3 -1 -1 1 2 2 1 2 1 3 2 2 1 1 2 2 2 2 2 2 2 2 2 2 3 3 5 5 4 4 4 4 -2 2 -1 4 2 1 1 3 3 9 -1 -2 2 2 2 1 2 1 3 2 2 (courtesy of Christopher Rasmussen)

1 1 1 2 2 2 3 Step 6 -1 2 1 2 1 3 3 -1 -1 1 2 2 1 2 1 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 5 5 4 4 4 4 -2 2 -2 2 2 1 3 3 3 9 6 -2 2 -2 2 1 1 2 1 3 2 2

1 1 1 Final Result -1 2 1 -1 -1 1 2 2 2 3 5 4 4 -2 2 2 1 1 3 3 3 3 9 9 6 6 14 14 5 5 11 7 6 5 2 2 1 2 1 3 2 2 9 12 8 5 I I ’ Why is I ’ large in some places and small in others? (courtesy of Christopher Rasmussen)

Filtering Example 15 January 2008 Lecture 3: Image Processing 90

Filtering Example 15 January 2008 Lecture 3: Image Processing 91

Filtering Example 15 January 2008 Lecture 3: Image Processing 92

Problem: Image Noise (courtesy of Forsyth & Ponce) 15 January 2008 Lecture 3: Image Processing 93

Solution: Smoothing Filter � If object reflectance changes slowly and noise at each pixel is independent, then we want to replace each pixel with something like the average of neighbors Disadvantage: Sharp (high-frequency) features lost Original image 7 x 7 averaging neighborhood 15 January 2008 Lecture 3: Image Processing 94

Smoothing Filter: Details � Filter types � Mean filter (box) 1 1 1 � Median (nonlinear) � Gaussian � Gaussian 1 1 1 1 1 1 � Can specify linear 1 1 1 operation by shifting 3 x 3 box filter kernel kernel over image and taking product 15 January 2008 Lecture 3: Image Processing 95

Gaussian Kernel � Idea: Weight contributions of neighboring pixels by nearness 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003 5 x 5, σ = 1 � Smooth roll-off reduces “ringing” seen in box filter 15 January 2008 Lecture 3: Image Processing 96

Gaussian Smoothing Example Original image Box filter 7 x 7 kernel σ = 1 σ = 3 15 January 2008 Lecture 3: Image Processing 97

Gaussian Smoothing Averaging Gaussian (courtesy of Marc Pollefeys) 15 January 2008 Lecture 3: Image Processing 98

“Box” vs. “Cone” Filters (courtesy Andries van Dam) 15 January 2008 Lecture 3: Image Processing 99

Smoothing Reduces Noise (courtesy of Marc Pollefeys) 15 January 2008 Lecture 3: Image Processing 100