Edge-Adaptive Image Interpolation with Contour Stencils Pascal - PowerPoint PPT Presentation

Edge-Adaptive Image Interpolation with Contour Stencils Pascal Getreuer Dec 27, 2010

TV along Curves Let u be an image. For C a smooth simple curve, define � T � dt , � � ∂ � �� � u � TV ( C ) = γ ( t ) γ : [0 , T ] → C . ∂ t u 0 Strategy: Find approximate con- γ ′ ( t ) tours of u by finding curves C such that � u � TV ( C ) is small. C

Contour Stencils A contour stencil is a function S : Z 2 × Z 2 → R describing edges between pixels, and TV is estimated as � ( S ⋆ [ u ])( k ) := S ( m , n ) | u k + m − u k + n | ≈ � u � TV ( C ) m , n ∈ Z 2 where S describes edges that approximate C . + n 1 + n 2 � ( S ⋆ [ u ])( i , j ) = | u i , j − 1 − u i +1 , j | (0 , 0) + | u i − 1 , j − 1 − u i , j | + | u i , j − u i +1 , j +1 | � + | u i − 1 , j − u i , j +1 | .

Contour Stencils Estimate the contours locally by finding a stencil with low TV, S ⋆ ( k ) = arg min S∈ Σ ( S ⋆ [ u ])( k ) where Σ is a set of candidate stencils. � Σ = �

Contour Stencils Input Contour Stencils Sobel For each pixel, S ⋆ ( k ) is determined to estimate the local contour orientation.

Why TV? Total variation is invariant under diffeomorphisms on space. Consider the change of variables � � u ( t ) ϕ ( s ) u t = ϕ ( s ) dt = ϕ ′ ( s ) ds and suppose that ϕ ′ ( s ) > 0, then � T � ϕ − 1 ( T ) | u ′ ( t ) | dt = � u ′ � � ϕ ′ ( s ) ds � �� ϕ ( s ) ϕ − 1 (0) 0 � ϕ − 1 ( T ) � ds . � � ∂ �� � = ∂ s u ϕ ( s ) ϕ − 1 (0)

Interpolation Problem Given discrete image v and point spread function h ( x , y ), find function u ( x , y ) such that v i , j = ( h ∗ u )( i , j ) for all i , j . Input v Interpolation u

Edge Directed Interpolation Theorem (Edge Directed Interpolation) Consider approximating u ( x ) by b a x u ( x ) = (1 − λ ) u ( a ) + λ u ( b ) ˆ C 2 C 1 and let C = C 1 ∪ C 2 be a curve passing through a, x, and b. Then the approximation error is bounded by � � | ˆ u ( x ) − u ( x ) | ≤ max | 1 − λ | , | λ | � u � TV ( C ) . Choosing the stencil with the smallest TV minimizes the estimated interpolation error: |S ⋆ | ( S ⋆ ⋆ [ u ])( k ) = min 1 1 | ˆ u ( x ) − u ( x ) | ≤ � u � TV ( C ) ≈ S ( S ⋆ [ u ])( k ) . S∈ Σ

Contour Stencil Windowed Zooming Local reconstructions: if S = � c n ϕ n θ u k ( x ) = v k + S ⋆ ( k ) ( x − n ) , n ∈N k where k + N : k th pixel of input image v k N : neighborhood if S = ϕ n S ⋆ ( k ) : function oriented with the best-fitting stencil S ⋆ ( k ) k : coefficients such that c n ( h ∗ u k )( m ) = v k + m for m ∈ N k + N

Contour Stencil Windowed Zooming Combine local reconstructions with overlapping windows � u ( x ) = w ( x − k ) u k ( x − k ) , k ∈ Z 2 where w satisfies � k w ( x − k ) ≡ 1 s.t. method reproduces constants w ( k ) = 0 for k �∈ N s.t. ↓ ( h ∗ u ) ≈ v w compact support for computational efficiency Example: Cubic B-spline w ( x ) = B ( x 1 ) B ( x 2 ) 6 | t | 3 − 1 � 3 � + � 1 − | t | + 1 � � � 1 − | t | B ( t ) = 3

Original Image (332 × 300) Input Image (83 × 75) Estimated Orientations Proposed Interpolation (PSNR 25.97, 0.125 s)

Cubic B-spline (PSNR 25.92, 0.011 s) Fourier (PSNR 25.34, 0.062 s) TV Minimization (PSNR 25.73, 0.784 s) Proposed Interpolation (PSNR 25.97, 0.125 s)

AQua-2 (PSNR 24.72, 0.016 s) Fractal Zooming (PSNR 24.65) Roussos (PSNR 25.87, 2.518 s) Proposed Interpolation (PSNR 25.97, 0.125 s)

Zooming Comparison Average PSNR on the Kodak Image Suite Zoom Factor 2 × 3 × 4 × AQua-2 25.06 22.48 21.35 Fractal Zooming 29.00 27.20 25.25 Contour Stencils 29.87 27.77 25.93 Roussos 30.56 27.97 26.19 Computation Time (s) vs. Output Image Size Image Size 128 × 128 256 × 256 512 × 512 AQua-2 0.0048 0.017 0.068 Contour Stencils 0.025 0.088 0.34 Roussos 0.23 2.22 8.64

Analysis of Contour Stencils ˜ C C ˜ C ≈ C = ⇒ � u � TV (˜ C ) ≈ � u � TV ( C ) Curve Perturbation Let C and ˜ C be smooth curves parameterized by γ : [0 , T ] → ˜ γ : [0 , T ] → C and ˜ C . Then if u is twice continuously differentiable, � � � � u � TV (˜ C ) − � u � TV ( C ) � γ ′ − γ ′ | � � � � ≤ � |∇ u | � | ˜ � � ∞ 1 � | ˜ γ ′ − γ ′ | C | + | C | + �∇ 2 u � ∞ + 1 � � �� � � | ˜ � | ˜ γ − γ | ∞ . � � 2 4 1

Analysis of Contour Stencils u ∈ C 2 ⇒ = discrete TV is first-order accurate TV Discretization Suppose u ∈ C 2 [0 , T ] and 0 = t 0 < t 1 < · · · < t N = T , and define h i = t i − t i − 1 . Then N 3 T h 2 h avg � u ′′ � ∞ ≤ � � u � TV − 1 | u ( t i ) − u ( t i − 1 ) | ≤ � u � TV . max i =1

Analysis of Contour Stencils Let S ⋆ 2 ( k ) denote the second best-fitting stencil and define the separation sep v ( k ) between the first and second best, S ⋆ 2 ( k ) := arg min ( S ⋆ [ v ])( k ) , S∈ Σ \S ⋆ ( k ) S ⋆ 2 ( k ) ⋆ [ v ] S ⋆ ( k ) ⋆ [ v ] � � � � sep v ( k ) := ( k ) − ( k ) . Stability of the Best-Fitting Stencil Suppose that for two images v and ˜ v sep v ( k ) > 2 M � v − ˜ v � 2 , � 2 � 1 / 2 �� �� � � � S ( m , n ) + S ( n , m ) M = max . � S∈ Σ m ∈ Z n ∈ Z Then they have the same best-fitting stencil at k .

Contour Stencil Design Let { f 1 , . . . , f J } , f j : Z 2 → R , be a set of image features. f j ( x ) = x 1 sin π 8 j − x 2 cos π Example: 8 j , j = 0 , . . . , 7 We want to design stencils S 1 , . . . , S J that distinguish between these features.

Contour Stencil Design We want stencil S j to be the best-fitting stencil on f j , |S j | ( S j ⋆ [ f j ])(0) < |S i | ( S i ⋆ [ f j ])(0) 1 1 for all i � = j . Want: 1 Ignoring the |S| normalizations, this condition becomes ( S j ⋆ [ f j ])(0) < ( S i ⋆ [ f j ])(0) ( S j − S i ) ⋆ [ f j ] � � = ⇒ (0) < 0 . We can try to satisfy this condition by minimizing J J J ( S j − S i ) ⋆ [ f j ] � � � �S j � 1 � � min (0) + γ S 1 ,..., S J i =1 j =1 j =1 s.t. 0 ≤ S j ( m , n ) ≤ 1

Contour Stencil Design J J J ( S j − S i ) ⋆ [ f j ] � � � � � �S j � 1 min (0) + γ S 1 ,..., S J i =1 j =1 j =1 s.t. 0 ≤ S j ( m , n ) ≤ 1 The minimization has closed-form solution � � J n | − γ if | f j m − f j n | < 1 i =1 | f i m − f i 1 J , S j ( m , n ) = J � J n | > 1 n | − γ if | f j m − f j i =1 | f i m − f i 0 J . J

Corner-Shaped Stencils Example: Corner-shaped stencils designed from the features f j ( x ) = max { x 1 cos π 4 j − x 2 sin π 4 j , x 1 sin π 4 j + x 2 cos π 4 j } .

3D Stencils In d dimensions, a stencil S : Z d × Z d → R is applied at voxel k ∈ Z d as � ( S ⋆ [ u ])( k ) := S ( m , n ) | u k + m − u k + n | . m , n ∈ Z d Small TV detects isosurfaces . Some 3D stencils:

3D Stencils i 2 + j 2 + k 2 � Example: Stencils applied to u i , j , k = The results are visualized by assigning a color to the region of space having a particular best-fitting stencil.

3D Stencils Example: Stencils applied to an MRI brain volume

Demosaicing Original Mosaiced Demosaiced Bayer Grid

Demosaicing Stencils Centered on a green pixel: Axially-oriented Diagonally-oriented π 8 -oriented Centered on a red or blue pixel: Axially-oriented Diagonally-oriented π 8 -oriented

Demosaicing Stencils Stencil orientation estimation on mosaiced data:

Demosaicing Stencils Stencil orientation estimation on mosaiced data:

Preliminary Demosaicing Method Let f be the given mosaiced image. We consider demosaicing by the minimization of � � � � u ( k ) m − u ( k ) � � arg min w m , n � n u n ∈ Ω k ∈{ Y , C b , C r } m ∈N ( n ) + λ � � ( f n − u ( k ) n ) 2 2 k ∈{ R , G , B } n ∈ Ω ( k ) where w m , n : weights choosen according the best-fitting stencils N ( n ) : neighbors of pixel n λ : fidelity parameter Ω ( k ) : subset of Ω where k th channel is given

Exact Exact Exact Demosaiced Demosaiced Demosaiced

Zhang-Wu Exact Malvar et al. Gunturk et al. Proposed Bilinear Hamilton-Adams Li

Zhang-Wu Exact Malvar et al. Gunturk et al. Proposed Bilinear Hamilton-Adams Li

Zhang-Wu Exact Malvar et al. Gunturk et al. Proposed Bilinear Hamilton-Adams Li

Thanks! Webpage http://www.math.ucla.edu/ ∼ getreuer/contours Contact getreuer@gmail.com