Introduction to Computer Graphics – Image Processing (1) – June 10, 2020 Kenshi Takayama
Today’s topics • Edge-aware image processing • Gradient-domain image processing 2
Image smoothing using Gaussian Filter • Smoothness parameter 𝜏 Original 𝜏 = 2 𝜏 = 5 𝜏 = 10 3
Equation of Gaussian Filter • 𝐽 𝐪 represents pixel value of image 𝐽 at position 𝐪 = 𝑞 " , 𝑞 # ∈ Ω • For given resolution e.g. 640 × 480, Ω ≔ 1, ⋯ , 640 × 1, ⋯ , 480 • GF $ [𝐽] represents filtered image with Gaussian parameter 𝜏 : ∑ 𝐫∈' 𝐻 $ 𝐪 − 𝐫 𝐽 𝐫 GF $ 𝐽 𝐪 ≔ ∑ 𝐫∈' 𝐻 $ 𝐪 − 𝐫 𝑋 𝐻 ! 𝑦 𝐪 • 𝐻 ' 𝑦 ≔ exp − ( ! )' ! ç Gaussian Kernel of radius 𝜏 𝑦 −3𝜏 −2𝜏 −𝜏 0 𝜏 2𝜏 3𝜏 4
Equation of Gaussian Filter • 𝐽 𝐪 represents pixel value of image 𝐽 at position 𝐪 = 𝑞 " , 𝑞 # ∈ Ω • For given resolution e.g. 640 × 480, Ω ≔ 1, ⋯ , 640 × 1, ⋯ , 480 • GF $ [𝐽] represents filtered image with Gaussian parameter 𝜏 : GF $ 𝐽 𝐪 ≔ 1 4 𝐻 $ 𝐪 − 𝐫 𝐽 𝐫 𝑋 𝐪 𝐫∈' 𝐻 ! 𝑦 • 𝐻 ' 𝑦 ≔ exp − ( ! )' ! ç Gaussian Kernel of radius 𝜏 𝑦 −3𝜏 −2𝜏 −𝜏 0 𝜏 2𝜏 3𝜏 5
Implementing Gaussian Filter 𝐻 ! 𝑦 • 𝐻 $ 3𝜏 ≈ 0 è Distant pixels can be ignored 𝑦 −3𝜏 −2𝜏 −𝜏 0 𝜏 2𝜏 3𝜏 2𝑠 + 1 • For fixed size 𝑠 ≔ ceil 3𝜏 , precompute weights 2𝑠 + 1 on a (2𝑠 + 1)×(2𝑠 + 1) stencil Stencil 6 http://people.csail.mit.edu/sparis/bf_course/
When kernel radius 𝜏 is very large • Direct computation takes a lot of time • Alternative: downsample è smooth with small 𝜏 è upsample 𝜏 = 20 downsample upsample 𝜏 = 5 7
Detail Extraction & Enhancement smoothed detail halos! 3× detail enhanced 8
When using edge-aware smoothing, ... smoothed detail 3× detail enhanced 9
Edge-aware smoothing using Bilateral Filter • Two parameters • 𝜏 / : Range of smoothing w.r.t. pixel’s location • 𝜏 0 : Range of smoothing w.r.t. pixel’s color BF $ ! , $ " 𝐽 𝐪 ≔ 1 4 𝐻 $ ! 𝐪 − 𝐫 𝐻 $ " 𝐽 𝐪 − 𝐽 𝐫 𝐽 𝐫 𝑋 𝐪 𝐫∈' In all cases, 𝜏 # = 10 Original 𝜏 " = 32 𝜏 " = 128 𝜏 " = 512 10
Application of Bilateral Filter: Stylization 11 Real-time video abstraction [Winnemöller SIGGRAPH06]
Application of Bilateral Filter: Tone Mapping • Range of each channel (24bit color image): 1~255 • Range of light intensity in the real world: 1~10 5 • H igh D ynamic R ange image • Can be obtained by photographing with different exposure times Short exposure Long exposure https://en.wikipedia.org/wiki/Tone_mapping 12 Fast bilateral filtering for the display of high-dynamic-range images [Durand SIGGRAPH02]
Application of Bilateral Filter: Tone Mapping HDR image γ correct. ( 𝑌 → 𝑌 𝛿 ) Bilateral Filter details lost γ correct. https://en.wikipedia.org/wiki/Tone_mapping details preserved 13 Fast bilateral filtering for the display of high-dynamic-range images [Durand SIGGRAPH02]
Naïve implementation of Bilateral Filter 1 4 𝐻 $ ! 𝐪 − 𝐫 𝐻 $ " 𝐽 𝐪 − 𝐽 𝐫 𝐽 𝐫 𝑋 𝐪 𝐫∈' • Recompute stencil for every pixel location 𝐪 ∈ Ω è slow • (Basic Assignment) 14
Another view toward Bilateral Filter # 𝐪 𝐪 • Define feature vector 𝐠 𝐪 ≔ " % , " ' for pixel location 𝐪 and intensity 𝐽 𝐪 𝐻 ! " 𝐪 − 𝐫 𝐻 ! # 𝐽 𝐪 − 𝐽 𝐫 • Weight of Bilateral Filter is equivalent to $ = exp − 𝐪 − 𝐫 $ 𝐽 𝐪 − 𝐽 𝐫 exp − Gaussian kernel applied to Euclidean distance $ $ 2𝜏 % 2𝜏 & in the feature space $ 𝐠 𝐪 − 𝐠 𝐫 = exp − 2 = 𝐻 ' 𝐠 𝐪 − 𝐠 𝐫 • Bilateral Filter is equivalent to applying Gaussian Filter of radius 1 to sample points {𝐠 𝐪 } in the feature space è Simpler computation 15
Bilateral Grid [Paris06; Chen07] • Define 3D feature space as (X-coord, Y-coord, intensity), map sample points {𝐠 𝐪 } to 3D grid • The larger 𝜏 3 & 𝜏 4 , the coarser the grid è lower comput. cost A Fast Approximation of the Bilateral Filter using a Signal Processing Approach [Paris ECCV06] 16 Real-time edge-aware image processing with the bilateral grid [Chen SIGGRAPH07]
Weight map generation using feature space White scribble è constraint of weight=1 Weight map Application: color adjustment Black scribble è constraint of weight=0 • Various names: Edit Propagation, Matting, Segmentation • Solve Laplace equation on Bilateral Grid 17 Real-time edge-aware image processing with the bilateral grid [Chen SIGGRAPH07]
Weight map generation using feature space Interpolation using Hermite RBF [Ijiri13] (Purpose: segmentation of CT volume) Interpolation using RBF [Li10] (Purpose: edit propagation for images/videos) https://www.youtube.com/watch?v=mL6ig_OaQAA Instant Propagation of Sparse Edits on Images and Videos [Li PG10] 18 Bilateral Hermite Radial Basis Functions for Contour-based Volume Segmentation [Ijiri EG13]
Extension to BF: Joint (Cross) Bilateral Filter Photo A: without flash Photo F: with flash J Correct color L Incorrect color L Noisy, blurred J Less noisy, sharp After applying JBF JBF ' " , ' # 𝐵, 𝐺 𝐪 ≔ 1 8 𝐻 ' " 𝐪 − 𝐫 𝐻 ' # 𝐺 𝐪 − 𝐺 𝐵 𝐫 𝐫 𝑋 𝐪 𝐫∈8 Digital Photography with Flash and No-Flash Image Pairs [Petschnigg SIGGRAPH04] 19 Flash Photography Enhancement via Intrinsic Relighting [Eisemann SIGGRAPH04]
Extension to BF: Non-Local Means Filter • Define feature space by neighborhood vector 𝐨 𝐪 representing 7 × 7 sub-image centered at 𝐪 NLMF ' 𝐽 𝐪 ≔ 1 8 𝐻 ' 𝐨 𝐪 − 𝐨 𝐫 𝐽 𝐫 𝑋 𝐪 𝐫∈8 Noisy input Bilateral NL Means 20 A non local algorithm for image denoising [Buades CVPR05]
Extension to BF: Rolling Guidance Filter Original Original Input Gaussian Input Guidance Joint Bilateral Filter Output Bilateral Rolling Guidance 21 Rolling Guidance Filter [Zhang ECCV14]
Today’s topics • Edge-aware image processing • Gradient-domain image processing 22
Scenario: insert source img. into destination img. Source Dest. Simple copying Boundary blurred Gradient-domain processing L L J Poisson image editing [Perez SIGGRAPH03] 23 Gradient Domain Manipulation Techniques in Vision and Graphics [Agrawal ICCV07 Course]
Scenario: generating panorama from several shots Simple tiling Gradient-domain processing 24 Efficient gradient-domain compositing using quadtrees [Agarwala SIGGRAPH07]
Simple case of 1D grayscale image Dest. Source Offset 25
2D case: offset by Laplace Membrane • Solve Laplace equation under Dirichlet boundary condition • Fast approximation using Mean Value Coordinates Mean value coordinates [Floater CAGD03] https://www.youtube.com/watch?v=AXvPeuc-wRw 26 Coordinates for instant image cloning [Farbman SIGGRAPH09]
Gradient-domain processing in general form (not just simple cloning) 27
Gradient-domain processing in general form (not just simple cloning) Modify gradients arbitrarily! 0 × 2 || 28
Gradient-domain processing in general form (not just simple cloning) Find 𝑔 that minimize Find function 𝑔 such that its 9 gradient closely matches 9:; − 9 ) 8 𝑔 9 − 𝑔 user-specified target ( 9 subject to: 𝑔| <8 = 𝑔 ∗ | <8 𝑔 ∗ 6 𝑔 ∗ Ω 29
1D case 2D case Find 𝑔 that minimize Find 𝑔 𝑦, 𝑧 that minimizes 9 9:; − 9 ) 8 𝑔 9 − 𝑔 𝛂𝑔 𝑦, 𝑧 − 𝐡(𝑦, 𝑧) ) F (,> ∈8 9 subject to: 𝑔| <8 = 𝑔 ∗ | <8 subject to: 𝑔| <8 = 𝑔 ∗ | <8 ⇔ • Basics of Gradient-domain image processing: Solve Poisson equation: Find image 𝑔 whose gradient best matches user-specified target gradient field 𝐡 Δ𝑔 = 𝛂 ⋅ 𝐡 by solving Poisson equation subject to: 𝑔| <8 = 𝑔 ∗ | <8 30
How to give target gradients: mixing • Copy source’s gradient only when its magnitude is larger è smooth part of source won’t be copied source source source dest. dest. dest. 31 Poisson image editing [Perez SIGGRAPH03]
How to give target gradient: Edge Brush • Copy gradients along object silhouette, paste along brush stroke • Real-time Poisson solver implemented on GPU Before After Before After https://www.youtube.com/watch?v=9MGjrsPzFc4 Real-time gradient-domain painting [McCann SIGGRAPH08] 32 Code: http://graphics.cs.cmu.edu/projects/gradient-paint/gradient_paint.r2403.tar.gz
How to give target gradient: modify original Amplify/suppress within selected region Set to zero except where detected as edges è Local Tone Mapping è Stylization 33 Poisson image editing [Perez SIGGRAPH03]
Extra: Gradient-domain geometry processing 34
Recommend
More recommend
