Edge Preserving Filtering Median Filter Bilateral Filter Shai - PowerPoint PPT Presentation

Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University

Slide Credits � (partial list) • Rick Szeliski • Steve Seitz • Alyosha Efros • Yacov Hel-Or • Marc Levoy • Bill Freeman • Fredo Durand • Sylvain Paris

A Gentle Introduction to Bilateral Filtering and its Applications “Fixing the Gaussian Blur”: the Bilateral Filter Sylvain Paris – MIT CSAIL

Box Average square neighborhood output input average

Equation of Box Average � � � BA [ I ] B ( p q ) I � p q � q S intensity at result at pixel q pixel p sum over all pixels q normalized box function 0

Square Box Generates Defects • Axis-aligned streaks • Blocky results output input

Box Profile pixel weight pixel position unrelated related unrelated pixels pixels pixels

Strategy to Solve these Problems • Use an isotropic ( i.e. circular) window. • Use a window with a smooth falloff. box window Gaussian window

Gaussian Blur per-pixel multiplication * output input average

input

box average

Gaussian blur

Equation of Gaussian Blur Same idea: weighted average of pixels . � � � � � GB [ I ] G || p q || I � p q � q S normalized Gaussian function 1 0

Gaussian Profile �� � 2 1 x � � � G ( x ) exp � � pixel � � 2 � � 2 2 � � weight pixel position unrelated uncertain related uncertain unrelated pixels pixels pixels pixels pixels

Spatial Parameter � � � � � GB [ I ] G || p q || I input � p q � q S size of the window small � large � limited smoothing strong smoothing

Blur Comes from Averaging across Edges * output input * * Same Gaussian kernel everywhere.

Bilateral Filter No Averaging across Edges [Aurich 95, Smith 97, Tomasi 98] * output input * * The kernel shape depends on the image content.

Bilateral Filter Definition: an Additional Edge Term Same idea: weighted average of pixels . new not new new � � 1 � � � � � � BF [ I ] G || p q || G | I I | I � � p p q q W s r � q S p normalization space weight range weight factor I

Illustration a 1D Image • 1D image = line of pixels • Better visualized as a plot pixel intensity pixel position

Gaussian Blur and Bilateral Filter Gaussian blur p � � � � � GB [ I ] G || p q || I q � p q � q S space space Bilateral filter [Aurich 95, Smith 97, Tomasi 98] p � � 1 � � range � � � � BF [ I ] G || p q || G | I I | I � � p p q q W q s r � q S p space range normalization space

Space and Range Parameters � � 1 � � � � � � BF [ I ] G || p q || G | I I | I � � p p q q W s r � q S p • space � s : spatial extent of the kernel, size of the considered neighborhood. • range � r : “minimum” amplitude of an edge

How to Set the Parameters Depends on the application. For instance: • space parameter: proportional to image size – e.g., 2% of image diagonal • range parameter: proportional to edge amplitude – e.g., mean or median of image gradients • independent of resolution and exposure

A Few More Advanced Remarks

Bilateral Filter Crosses Thin Lines • Bilateral filter averages across features thinner than ~2 � s • Desirable for smoothing: more pixels = more robust • Different from diffusion that stops at thin lines close-up kernel

Iterating the Bilateral Filter � I BF [ I ] � ( n 1 ) ( n ) • Generate more piecewise-flat images • Often not needed in computational photo.

input

1 iteration

2 iterations

4 iterations

Bilateral Filtering Color Images input For gray-level images intensity difference � � 1 � � � � � � BF [ I ] G || p q || G | I I | I � � p p q q W s r � q S p scalar output For color images color difference � � 1 � � � � � � BF [ I ] G || p q || G || C C || C � � p p q q W s r � q S p 3D vector (RGB, Lab) The bilateral filter is The bilateral filter is extremely easy to adapt to your need. extremely easy to adapt to your need.

Applications • Image denoising • HDR Compression – Key idea: – Break image into base and detail layers – Compress base – Recompose image

Fast Implementation: 3D Kernel [Paris and Durand 06] • Idea: represent image data such that the weights depend only on the distance between points close in space 1D image Plot I = f ( x ) pixel far in range intensity pixel position

1 st Step: Re-arranging Symbols � � 1 � � � � � � BF [ I ] G || p q || G | I I | I � � p p q q W s r � q S p � � � � � � � � W G || p q || G | I I | � � p p q s r � q S Multiply first equation by W p � � � � � � � � W BF [ I ] G || p q || G | I I | I � � p p p q q s r � q S � � 1 � � � � � � W G || p q || G | I I | � � p p q s r � q S

1 st Step: Summary � � � � � � � � W BF [ I ] G || p q || G | I I | I � � p p p q q s r � q S � � 1 � � � � � � W G || p q || G | I I | � � p p q s r � q S • Similar equations • No normalization factor anymore • Don’t forget to divide at the end

2 nd Step: Higher-dimensional Space • “Product of two Gaussians” = higher dim. Gaussian space range p p

2 nd Step: Higher-dimensional Space • 0 almost everywhere, I at “plot location” space range p p

2 nd Step: Higher-dimensional Space • 0 almost everywhere, I at “plot location” • Weighted average at each point = Gaussian blur p p

2 nd Step: Higher-dimensional Space • 0 almost everywhere, I at “plot location” • Weighted average at each point = Gaussian blur • Result is at “plot location” p p

New num. scheme: • simple operations • complex space ���������������������������� ������������� ������ ������ ����������� ����������� ����������� ����������� ��������� ��������� �������� �������

Strategy: downsampled convolution ���������������������������� ������������������� �������������������� ������� ������� ����������� ����������� ��������������� �������� Conceptual view, ������� not exactly the actual algorithm

The Algorithm

NEW IDEA : ‘Joint’ or ‘Cross’ Bilateral’ Petschnigg(2004) and Eisemann(2004) Bilateral � two kinds of weights NEW : get them from two kinds of images. • Smooth image A pixels locally, but • Limit to ‘similar regions’ of image B Why do this? To get ‘best of both images’

Ordinary Bilateral Filter Bilateral � two kinds of weights, one image A : � � 1 � � � � � � BF [ A ] G || p q || G | A A | A � � p p q q W s r � q S p Image A: c c ����� ����� s s f(x) f(x) x x ������ ������

‘Joint’ or ‘Cross’ Bilateral Filter NEW: two kinds of weights, two images � � 1 � � � � � � BF [ A ] G || p q || G | B B | A � � p p q q W s r � q S p B: Clean,strong A: Noisy, dim (Flash image) (ambient image) c c c c s s s s

Image A: Warm, shadows, but too Noisy (too dim for a good quick photo) (too dim for a good quick photo) No-flash

Image B: Cold, Shadow-free, Clean (flash: simple light, ALMOST no shadows) (flash: simple light, ALMOST no shadows)

MERGE BEST OF BOTH: apply ‘Cross Bilateral’ or ‘Joint Bilateral’

(it really is much better!)

Video Enhancement Using Per Pixel Exposures (Bennett, 06) From this video: ASTA: A daptive S patio- S T emporal T A ccumulation Filter

ASTA Replace pixel difference with a general dissimilarity measure D(x,x)=0, D(x,y)=D(y,x)

Shot noise

A new dissimilarity measure Instead of comparing pixel intensities, look at their local spatial neighborhood

The Process for One Frame • Raw Video Frame: (from FIFO center) • Histogram stretching; (estimate gain for each pixel) • ‘Mostly Temporal’ Bilateral Filter: – Average recent similar values, – Reject outliers (avoids ‘ghosting’), spatial avg as needed – Tone Mapping