Introduction to Computer Graphics Image Processing (1) July 4, - PowerPoint PPT Presentation

Introduction to Computer Graphics – Image Processing (1) – July 4, 2019 Kenshi Takayama

Today’s topics • Edge-aware image processing • Gradient-domain image processing 2

Image smoothing using Gaussian Filter • Smoothness parameter 𝜏 Original 𝜏 = 2 𝜏 = 5 𝜏 = 10 3

Equation of Gaussian Filter • 𝐽 𝐪 represents pixel value of image 𝐽 at position 𝐪 = 𝑞 x , 𝑞 y ∈ Ω • For given resolution e.g. 640 × 480, Ω ≔ 1, ⋯ , 640 × 1, ⋯ , 480 • GF 𝜏 [𝐽] represents filtered image with Gaussian parameter 𝜏 : σ 𝐫∈Ω 𝐻 𝜏 𝐪 − 𝐫 𝐽 𝐫 GF 𝜏 𝐽 𝐪 ≔ σ 𝐫∈Ω 𝐻 𝜏 𝐪 − 𝐫 𝑋 𝐪 𝐻 𝜏 𝑦 𝑦 2 2𝜏 2  Gaussian Kernel of radius 𝜏 • 𝐻 𝜏 𝑦 ≔ exp − 𝑦 0 −3𝜏 −2𝜏 −𝜏 𝜏 2𝜏 3𝜏 4

Equation of Gaussian Filter • 𝐽 𝐪 represents pixel value of image 𝐽 at position 𝐪 = 𝑞 x , 𝑞 y ∈ Ω • For given resolution e.g. 640 × 480, Ω ≔ 1, ⋯ , 640 × 1, ⋯ , 480 • GF 𝜏 [𝐽] represents filtered image with Gaussian parameter 𝜏 : GF 𝜏 𝐽 𝐪 ≔ 1 ෍ 𝐻 𝜏 𝐪 − 𝐫 𝐽 𝐫 𝑋 𝐪 𝐫∈Ω 𝐻 𝜏 𝑦 𝑦 2 2𝜏 2  Gaussian Kernel of radius 𝜏 • 𝐻 𝜏 𝑦 ≔ exp − 𝑦 0 −3𝜏 −2𝜏 −𝜏 𝜏 2𝜏 3𝜏 5

Implementing Gaussian Filter 𝐻 𝜏 𝑦 • 𝐻 𝜏 3𝜏 ≈ 0  Distant pixels can be ignored 𝑦 0 −3𝜏 −2𝜏 −𝜏 𝜏 2𝜏 3𝜏 2𝑠 + 1 • For fixed size 𝑠 ≔ ceil 3𝜏 , precompute weights 2𝑠 + 1 on a (2𝑠 + 1) × (2𝑠 + 1) stencil Stencil 6 http://people.csail.mit.edu/sparis/bf_course/

When kernel radius 𝜏 is very large • Direct computation takes a lot of time • Alternative: downsample  smooth with small 𝜏  upsample 𝜏 = 20 downsample upsample 𝜏 = 5 7

Detail Extraction & Enhancement smoothed detail halos! 3 × detail enhanced 8

When using edge-aware smoothing, ... smoothed detail 3 × detail enhanced 9

Edge-aware smoothing using Bilateral Filter • Two parameters • 𝜏 s : Range of smoothing w.r.t. pixel’s location • 𝜏 r : Range of smoothing w.r.t. pixel’s color BF 𝜏 s , 𝜏 r 𝐽 𝐪 ≔ 1 ෍ 𝐻 𝜏 s 𝐪 − 𝐫 𝐻 𝜏 r 𝐽 𝐪 − 𝐽 𝐫 𝐽 𝐫 𝑋 𝐪 𝐫∈Ω In all cases, 𝜏 s = 10 Original 𝜏 r = 32 𝜏 r = 128 𝜏 r = 512 10

Application of Bilateral Filter: Stylization 11 Real-time video abstraction [Winnemöller SIGGRAPH06]

Application of Bilateral Filter: Tone Mapping • Range of each channel (24bit color image): 1~255 • Range of light intensity in the real world: 1~10 5 • H igh D ynamic R ange image • Can be obtained by photographing with different exposure times Short exposure Long exposure https://en.wikipedia.org/wiki/Tone_mapping 12 Fast bilateral filtering for the display of high-dynamic-range images [Durand SIGGRAPH02]

Application of Bilateral Filter: Tone Mapping HDR image γ correct. ( 𝑌 → 𝑌 𝛿 ) Bilateral Filter details lost https://en.wikipedia.org/wiki/Tone_mapping details preserved 13 Fast bilateral filtering for the display of high-dynamic-range images [Durand SIGGRAPH02]

Naïve implementation of Bilateral Filter 1 ෍ 𝐻 𝜏 s 𝐪 − 𝐫 𝐻 𝜏 r 𝐽 𝐪 − 𝐽 𝐫 𝐽 𝐫 𝑋 𝐪 𝐫∈Ω • Recompute stencil for every pixel location 𝐪 ∈ Ω  slow • (Basic Assignment) 14

Another view toward Bilateral Filter 𝐽 𝐪 𝐪 • Define feature vector 𝐠 𝐪 ≔ 𝜏 r for pixel location 𝐪 and intensity 𝐽 𝐪 𝜏 s , 𝐻 𝜏 s 𝐪 − 𝐫 𝐻 𝜏 r 𝐽 𝐪 − 𝐽 𝐫 • Weight of Bilateral Filter is equivalent to 2 = exp − 𝐪 − 𝐫 2 𝐽 𝐪 − 𝐽 𝐫 Gaussian kernel applied to Euclidean distance exp − 2 2 2𝜏 s 2𝜏 r in the feature space 2 𝐠 𝐪 − 𝐠 𝐫 = exp − 2 = 𝐻 1 𝐠 𝐪 − 𝐠 𝐫 • Bilateral Filter is equivalent to applying Gaussian Filter of radius 1 to sample points {𝐠 𝐪 } in the feature space  Simpler computation 15

Bilateral Grid [Paris06; Chen07] • Define 3D feature space as (X-coord, Y-coord, intensity), map sample points {𝐠 𝐪 } to 3D grid • The larger 𝜏 s & 𝜏 r , the coarser the grid  lower comput. cost A Fast Approximation of the Bilateral Filter using a Signal Processing Approach [Paris ECCV06] 16 Real-time edge-aware image processing with the bilateral grid [Chen SIGGRAPH07]

Weight map generation using feature space White scribble  constraint of weight=1 Weight map Application: color adjustment Black scribble  constraint of weight=0 • Various names: Edit Propagation, Matting, Segmentation • Solve Laplace equation on Bilateral Grid 17 Real-time edge-aware image processing with the bilateral grid [Chen SIGGRAPH07]

Weight map generation using feature space Interpolation using Hermite RBF [Ijiri13] (Purpose: segmentation of CT volume) Interpolation using RBF [Li10] (Purpose: edit propagation for images/videos) https://www.youtube.com/watch?v=mL6ig_OaQAA Instant Propagation of Sparse Edits on Images and Videos [Li PG10] 18 Bilateral Hermite Radial Basis Functions for Contour-based Volume Segmentation [Ijiri EG13]

Extension of Bilateral Filter: Joint (Cross) Bilateral Filter Photo A: without flash Photo F: with flash  Correct color  Incorrect color  Noisy, blurred  Less noisy, sharp After applying JBF JBF 𝜏 s , 𝜏 r 𝐵, 𝐺 𝐪 ≔ 1 ෍ 𝐻 𝜏 s 𝐪 − 𝐫 𝐻 𝜏 r 𝐺 𝐪 − 𝐺 𝐵 𝐫 𝐫 𝑋 𝐪 𝐫∈Ω Digital Photography with Flash and No-Flash Image Pairs [Petschnigg SIGGRAPH04] 19 Flash Photography Enhancement via Intrinsic Relighting [Eisemann SIGGRAPH04]

Extension of Bilateral Filter: Non-Local Means Filter • Define feature space by neighborhood vector 𝐨 𝐪 representing 7 × 7 sub-image centered at 𝐪 NLMF 𝜏 𝐽 𝐪 ≔ 1 ෍ 𝐻 𝜏 𝐨 𝐪 − 𝐨 𝐫 𝐽 𝐫 𝑋 𝐪 𝐫∈Ω Bilateral Noisy input NL Means 20 A non local algorithm for image denoising [Buades CVPR05]

Today’s topics • Edge-aware image processing • Gradient-domain image processing 21

Scenario: insert source img. into destination img. Source Dest. Simple copying Boundary blurred Gradient-domain processing    Poisson image editing [Perez SIGGRAPH03] 22 Gradient Domain Manipulation Techniques in Vision and Graphics [Agrawal ICCV07 Course]

Scenario: generating panorama from several shots Simple tiling Gradient-domain processing 23 Efficient gradient-domain compositing using quadtrees [Agarwala SIGGRAPH07]

Simple case of 1D grayscale image Dest. Source Offset 24

2D case: offset by Laplace Membrane • Solve Laplace equation under Dirichlet boundary condition • Fast approximation using Mean Value Coordinates Mean value coordinates [Floater CAGD03] https://www.youtube.com/watch?v=AXvPeuc-wRw 25 Coordinates for instant image cloning [Farbman SIGGRAPH09]

Gradient-domain processing in general form (not just simple cloning) 26

Gradient-domain processing in general form (not just simple cloning) Modify gradients arbitrarily! 0 × 2 || 27

Gradient-domain processing in general form (not just simple cloning) Find 𝑔 that minimize Find function 𝑔 such that its 𝑗 gradient closely matches 𝑗−1 − 𝑕 𝑗 2 ෍ 𝑔 𝑗 − 𝑔 user-specified target 𝑕 𝑗 𝑗 subject to: 𝑔ȁ 𝜖Ω = 𝑔 ∗ ȁ 𝜖Ω 𝑔 ∗ 𝑕 𝑗 𝑔 ∗ Ω 28

1D case 2D case Find 𝑔 that minimize Find 𝑔 𝑦, 𝑧 that minimizes 𝑗 𝑗−1 − 𝑕 𝑗 2 ෍ 𝑔 𝑗 − 𝑔 𝛂𝑔 𝑦, 𝑧 − 𝐡(𝑦, 𝑧) 2 න 𝑦,𝑧 ∈Ω 𝑗 subject to: 𝑔ȁ 𝜖Ω = 𝑔 ∗ ȁ 𝜖Ω subject to: 𝑔ȁ 𝜖Ω = 𝑔 ∗ ȁ 𝜖Ω ⇔ • Basics of Gradient-domain image processing: Solve Poisson equation: Find image 𝑔 whose gradient best matches user-specified target gradient field 𝐡 Δ𝑔 = 𝛂 ⋅ 𝐡 by solving Poisson equation subject to: 𝑔 ȁ 𝜖Ω = 𝑔 ∗ ȁ 𝜖Ω 29

How to give target gradients: mixing • Copy source’s gradient only when its magnitude is larger  smooth part of source won’t be copied source source source dest. dest. dest. 30 Poisson image editing [Perez SIGGRAPH03]

How to give target gradient: Edge Brush • Copy gradients along object silhouette, paste along brush stroke • Real-time Poisson solver implemented on GPU Before After Before After https://www.youtube.com/watch?v=9MGjrsPzFc4 Real-time gradient-domain painting [McCann SIGGRAPH08] 31 Code: http://graphics.cs.cmu.edu/projects/gradient-paint/gradient_paint.r2403.tar.gz

How to give target gradient: modify original Amplify/suppress within selected region Set to zero except where detected as edges  Local Tone Mapping  Stylization 32 Poisson image editing [Perez SIGGRAPH03]

Extra: Gradient-domain geometry processing 33