Linear filtering Applications De-noising Subhransu Maji - PowerPoint PPT Presentation

Overview Linear filtering ‣ Mathematical model ‣ Implementation details Linear filtering Applications ‣ De-noising Subhransu Maji ‣ Sharpening ‣ Edge detection CMPSCI 670: Computer Vision Canny edge detector and recent advances September 27, 2016 Slides credit: S. Lazebnik and others 2 CMPSCI 670 Subhransu Maji (UMass, Fall 16) Motivation Moving average Let’s replace each pixel with a weighted average of its neighborhood How can we reduce noise in a photograph? The weights are called the filter kernel What are the weights for the average of a 3x3 neighborhood? 1 1 1 1 1 1 1 1 1 “box filter” Source: D. Lowe 3 4 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16)

Filtering Key properties Let f be the image and g be the kernel. The output of filtering f with g Linearity: filter( f 1 + f 2 ) = filter( f 1 ) + filter( f 2 ) denoted f * g is given by: Shift invariance: same behavior regardless of pixel location: X filter(shift( f )) = shift(filter( f )) ( f ∗ g )[ m, n ] = f [ m + k, n + l ] g [ k, l ] Theoretical result: any linear shift-invariant operator can be k,l represented as a convolution f Filtering computes the correlation between the g and f at each location Convolution is filtering with a flipped g (by notation) Source: F. Durand 5 6 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Properties in more detail Annoying details Commutative: a * b = b * a What is the size of the output? Conceptually no difference between filter and signal MATLAB: filter2(g, f, shape ) ‣ shape = ‘full’: output size is sum of sizes of f and g Associative: a * ( b * c ) = ( a * b ) * c ‣ shape = ‘same’: output size is same as f Often apply several filters one after another: ((( a * b 1 ) * b 2 ) * b 3 ) ‣ shape = ‘valid’: output size is difference of sizes of f and g This is equivalent to applying one filter: a * ( b 1 * b 2 * b 3 ) Distributes over addition: a * ( b + c ) = ( a * b ) + ( a * c ) full same valid Scalars factor out: ka * b = a * kb = k ( a * b ) g g Identity: unit impulse e = […, 0, 0, 1, 0, 0, …], a * e = a g g g g f f f g g g g g g 7 8 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16)

Annoying details Annoying details What about near the edge? What about near the edge? ‣ the filter window falls off the edge of the image ‣ the filter window falls off the edge of the image ‣ need to extrapolate ‣ need to extrapolate ‣ methods: ‣ methods (MATLAB): ➡ clip filter (black) ➡ clip filter (black): imfilter(f, g, 0) ➡ wrap around ➡ wrap around: imfilter(f, g, ‘circular’) ➡ copy edge ➡ copy edge: imfilter(f, g, ‘replicate’) ➡ reflect across edge ➡ reflect across edge: imfilter(f, g, ‘symmetric’) Source: S. Marschner Source: S. Marschner 9 10 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Practice with linear filters Practice with linear filters 0 0 0 0 0 0 ? 0 1 0 0 1 0 0 0 0 0 0 0 Original Original Filtered (no change) Source: D. Lowe Source: D. Lowe 11 12 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16)

Practice with linear filters Practice with linear filters 0 0 0 0 0 0 ? 0 0 1 0 0 1 0 0 0 0 0 0 Original Original Shifted left By 1 pixel Source: D. Lowe Source: D. Lowe 13 14 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Practice with linear filters Practice with linear filters 1 1 1 1 1 1 ? 1 1 1 1 1 1 1 1 1 1 1 1 Original Original Blur (with a box filter) Source: D. Lowe Source: D. Lowe 15 16 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16)

Practice with linear filters Practice with linear filters 0 0 0 1 1 1 0 0 0 1 1 1 - - ? 0 2 0 1 1 1 0 2 0 1 1 1 0 0 0 1 1 1 0 0 0 1 1 1 (Note that filter sums to 1) Original Original Sharpening filter: accentuates differences with local average Source: D. Lowe Source: D. Lowe 17 18 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Smoothing with box filter revisited Smoothing with box filter revisited What’s wrong with this picture? What’s wrong with this picture? What’s the solution? What’s the solution? ‣ To eliminate edge effects, weight contribution of neighborhood pixels according to their closeness to the center “fuzzy blob” Source: D. Forsyth 19 20 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16)

Gaussian Kernel Gaussian kernel 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003 5 x 5, σ = 1 σ = 2 with 30 x 30 σ = 5 with 30 x 30 kernel kernel Constant factor at front makes volume sum to 1 (can be ignored when computing the filter values, as we should renormalize weights to sum to 1 in any case) Standard deviation σ : determines extent of smoothing Source: C. Rasmussen Source: K. Grauman 21 22 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Choosing kernel width Choosing kernel width Rule of thumb: set filter half-width to about 3 σ The Gaussian function has infinite support, but discrete filters use finite kernels Source: K. Grauman 23 24 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16)

Gaussian vs. box filtering Gaussian filters Remove high-frequency components from the image ( low-pass filter ) Convolution with self is another Gaussian ‣ So can smooth with small- σ kernel, repeat, and get same result as larger- σ kernel would have ‣ Convolving two times with Gaussian kernel with std. dev. σ 
 2 is same as convolving once with kernel with std. dev. σ Separable kernel ‣ Factors into product of two 1D Gaussians ‣ Discrete example: 1 2 1 1 & # & # $ ! $ ! [ ] 2 4 2 2 1 2 1 = $ ! $ ! 1 2 1 1 $ ! $ ! % " % " Source: K. Grauman 25 26 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Separability of the Gaussian filter Why is separability useful? Separability means that a 2D convolution can be reduced to two 1D convolutions (one among rows and one among columns) What is the complexity of filtering an n × n image with an m × m kernel? ‣ O(n 2 m 2 ) What if the kernel is separable? ‣ O(n 2 m) Source: D. Lowe 27 28 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16)

Types of noise Gaussian noise Salt and pepper noise : Mathematical model: sum of many independent factors contains random occurrences Good for small standard deviations of black and white pixels Assumption: independent, zero-mean noise Impulse noise: contains random occurrences of white pixels Gaussian noise : variations in intensity drawn from a Gaussian normal distribution Source: S. Seitz Source: M. Hebert 29 30 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Reducing Gaussian noise Reducing salt-and-pepper noise noise 3x3 5x5 7x7 Gaussian smoothing with increasing standard deviation What is wrong with these results? Smoothing with larger standard deviations suppresses noise, but also blurs the image 31 32 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16)


 
 
 
 
 
 Alternative idea: Median filtering Median filter A median filter operates over a window by selecting the median What advantage does median filtering have over Gaussian filtering? intensity in the window 
 Robustness to outliers Question: is median filtering linear? Source: K. Grauman Source: K. Grauman 33 34 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Median filter Salt-and-pepper noise Median filtered Linear filtering Subhransu Maji CMPSCI 670: Computer Vision September 29, 2016 MATLAB: medfilt2(image, [h w]) Source: M. Hebert Slides credit: S. Lazebnik and others 35 CMPSCI 670 Subhransu Maji (UMass, Fall 16)

Announcements Sharpening Today’s lecture ends at 1:55pm ‣ Encourage you to attend Yoshua Bengio’s talk @ 2 Administrivia ‣ Mini-project 1 due today ‣ Mini-project 2 will be posted later today (due Oct 14) Source: D. Lowe 37 38 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) Sharpening Unsharp mask filter What does blurring take away? f ( f f g ) ( 1 ) f f g f (( 1 ) e g ) + α − ∗ = + α − α ∗ = ∗ + α − image blurred 
 unit impulse 
 (identity) image – = smoothed (5x5) detail original Let’s add it back: + α = unit impulse Gaussian Laplacian of Gaussian original detail sharpened 39 40 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16)

Hybrid Images Gaussian Filter Laplacian Filter A. Oliva, A. Torralba, P.G. Schyns, “Hybrid Images,” SIGGRAPH 2006 41 42 CMPSCI 670 Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16) dolphin and car 43 44 CMPSCI 670 motorcycle and bicycle Subhransu Maji (UMass, Fall 16) CMPSCI 670 Subhransu Maji (UMass, Fall 16)