Truncation Error in Image Interpolation Loc Simon SampTA 2013 - - PowerPoint PPT Presentation

Truncation Error in Image Interpolation Loïc Simon SampTA 2013 - Bremen 1

Collaborator Jean-Michel Morel 2

Truncation error: What is that? X k ’s X t ’s 3

Truncation error: What is that? X X t := X k sinc( t − k ) k ∈ Z 2 4

Truncation error: What is that? ? 5

Context • Motivations • Assumptions • Goal • Related work 6

Motivations • Image registration • optical flow • stereopsis • super-resolution • sub-pixel accuracy 7

Motivations • Image registration • optical flow • stereopsis • super-resolution • sub-pixel accuracy • error ~ quantization 8

Assumptions 9

Assumptions 10

Assumptions • a 1d random process X t ( t ∈ R ) • observed on k ∈ { − K, . . . , K } • weakly stationary µ, d Ψ X ( ω ) • no aliasing X t X X t := X k sinc( t − k ) k ∈ Z t − K 0 K 11

Goal • Linear shift-invariant ˜ X X t := X k h ( t − k ) k ≤ K • Practical bounds on r h i RMSE [ ˜ ( ˜ X t ] := X t − X t ) 2 E 12

Goal • Linear shift-invariant ⇢ h ( t ) = sinc( t ) ˜ X X t := X k h ( t − k ) h ( t ) = sincd K ( t ) k ≤ K • Practical bounds on r h i RMSE [ ˜ ( ˜ X t ] := X t − X t ) 2 E ➡ Sinc interpolation ➡ DFT interpolation 13

Related Work Truncation Error Approximation Jagerman 1966 Strang & Fix 1971 Yao & Thomas 1966 Blu & Unser 1999 Campbell 1968 Condat & al. 2005 Brown 1969 Xu & Huang & Li 2009 Other Moisan 2011 Jerri 1977 14

Related Work Truncation Error Approximation Strang & Fix 1971 Jagerman 1966 Yao & Thomas 1966 Blu & Unser 1999 Campbell 1968 Condat & al. 2005 Brown 1969 Xu & Huang & Li 2009 ➡ sinc only ➡ oversampled case Other Moisan 2011 Jerri 1977 15

Related Work Truncation Error Approximation Jagerman 1966 Strang & Fix 1971 Yao & Thomas 1966 Blu & Unser 1999 Campbell 1968 Condat & al. 2005 Brown 1969 ➡ K = ∞ Xu & Huang & Li 2009 Other Moisan 2011 Jerri 1977 16

Related Work Truncation Error Approximation Jagerman 1966 Strang & Fix 1971 Yao & Thomas 1966 Blu & Unser 1999 Campbell 1968 Condat & al. 2005 Brown 1969 Xu & Huang & Li 2009 Other Moisan 2011 Jerri 1977 17

Rest of the talk • Theoretical bounds • Experimental results • Discussion & conclusion 18

A bit of intuition... δ ( t ) t δ ( t ) 19

Theoretical bounds 0 ⇣ ⌘ 1 1 µ 2 O δ ( t ) 2 B C + B C X ]( t ) = sin 2 ( π t ) B C ⇣ ⌘ σ 0 2 MSE [ ˜ 1 α O B C × δ ( t ) 2 π 2 B C B C + B C @ ⇣ ⌘ A 1 σ 2 α O δ ( t ) 20

Theoretical bounds 0 ⇣ ⌘ 1 1 µ 2 O δ ( t ) 2 B C + B C X ]( t ) = sin 2 ( π t ) B C ⇣ ⌘ σ 0 2 MSE [ ˜ 1 α O B C × δ ( t ) 2 π 2 B C B C + B C @ ⇣ ⌘ A 1 σ 2 α O δ ( t ) 21

Theoretical bounds 0 ⇣ ⌘ 1 1 0 µ 2 O δ ( t ) 2 B C + B C X ]( t ) = sin 2 ( π t ) B C ⇣ ⌘ 2 σ 0 2 MSE [ ˜ 1 α O B C × δ ( t ) 2 π 2 B C B C + B C @ ⇣ ⌘ A 1 2 σ 2 α O δ ( t ) ➡ DFT modifications 22

Spectral representation � � 2 � � X � � MSE [ ˜ X ]( t ) = µ 2 1 − h ( t − k ) � � � � � � | k | ≤ K | {z } MSE [ µ ]( t ) + Z � � 2 � � 1 X � � e i ω t − e i ω k h ( t − k ) d Ψ X ( ω ) � � 2 π � � � � | k | ≤ K | {z } MSE [ d Ψ X ]( t ) ➡ Average component ➡ Spectral component 23

Spectral representation � � 2 � � X � � MSE [ ˜ X ]( t ) = µ 2 1 − h ( t − k ) � � � � � � | k | ≤ K | {z } MSE [ µ ]( t ) + Z � � 2 � � 1 X � � e i ω t − e i ω k h ( t − k ) d Ψ X ( ω ) � � 2 π � � � � | k | ≤ K | {z } MSE [ d Ψ X ]( t ) ➡ Aliasing is not forbidden 24

Spectral representation � � 2 � � X X � � MSE [ ˜ X ]( t ) = µ 2 sinc( t − k ) − h ( t − k ) � � � � � � k ∈ Z | k | ≤ K | {z } MSE [ µ ]( t ) + � � 2 � � Z 1 X X � � e i ω k sinc( t − k ) − e i ω k h ( t − k ) d Ψ X ( ω ) � � 2 π � � | ω | ≤ π � � k ∈ Z | k | ≤ K | {z } MSE [ d Ψ X ]( t ) ➡ Under no aliasing condition 25

Spectral representation � � 2 � � X X � � MSE [ ˜ X ]( t ) = µ 2 sinc( t − k ) − sinc( t − k ) � � � � � � k ∈ Z | k | ≤ K | {z } MSE [ µ ]( t ) + � � 2 � � Z 1 X X � � e i ω k sinc( t − k ) − e i ω k sinc( t − k ) d Ψ X ( ω ) � � 2 π � � | ω | ≤ π � � k ∈ Z | k | ≤ K | {z } MSE [ d Ψ X ]( t ) ➡ Under no aliasing condition ➡ Sinc 26

Average component MSE [ µ ]( t ) = sin 2 ( π t ) ✓ 1 ◆ µ 2 O π 2 δ ( t ) 2 ➡ Gibbs phenomenon 27

Average component MSE [ µ ]( t ) = 0 ➡ DFT 28

Spectral component Spectrum (dB) Spectrum (dB) Spectrum (dB) Spectrum (dB) 40 40 42.8dB 42.8dB 36.5dB 36.5dB 40 40 30 30 30 30 20 20 20 20 15.2dB 15.2dB 10 10 10 10 1.7dB 1.7dB 0.50 0.50 0.50 0.50 60 60 Spectrum (dB) Spectrum (dB) Spectrum (dB) Spectrum (dB) 41.6dB 41.6dB 54.2dB 54.2dB 40 40 50 50 40 40 30 30 30 30 23.0dB 23.0dB 20 20 20 20 9.8dB 9.8dB 10 10 10 10 0.50 0.50 0.50 0.50 29

Spectral decomposition d Ψ X ( ω ) ψ α ( ω ) d ω d Ψ 0 α ( ω ) σ 2 | ω |  π d ω α σ 2 α ω απ π 0 ➡ spectrum ≤ oversampled + white-noise 30

Oversampled case supp( d Ψ 0 α ) ⊂ {| ω | ≤ απ } = ⇒ α ]( t ) =sin 2 ( π t ) ✓ ◆ 1 σ 0 2 MSE [ d Ψ 0 α O π 2 δ ( t ) 2 where, α = 1 1 Z σ 0 2 1 + cos( ω ) d Ψ 0 α ( ω ) π | ω |  απ 31

Oversampled case supp( d Ψ 0 α ) ⊂ {| ω | ≤ απ } = ⇒ =sin 2 ( π t ) ⇢ � ⇢ � ✓ ◆ σ 0 2 1 MSE [ d Ψ 0 α ]( t ) O α µ 2 MSE [ µ ]( t ) π 2 δ ( t ) 2 where, α = 1 1 Z σ 0 2 1 + cos( ω ) d Ψ 0 α ( ω ) π | ω |  απ 32

White-noise d Ψ X ( ω ) ψ α ( ω ) d ω d Ψ 0 α ( ω ) σ 2 | ω |  π d ω α σ 2 α d Ψ ( ω ) = σ 2 | ω | ≤ π d ω ω α απ π 0 = ⇒ ✓ 1 MSE [ d Ψ ]( t ) =sin 2 ( π t ) ◆ σ 2 α O π 2 δ ( t ) ➡ Slow decay 33

Recap 0 ⇣ ⌘ 1 1 0 µ 2 O δ ( t ) 2 B C + B C X ]( t ) = sin 2 ( π t ) B C ⇣ ⌘ 2 σ 0 2 MSE [ ˜ 1 α O B C × δ ( t ) 2 π 2 B C B C + B C @ ⇣ ⌘ A 1 2 σ 2 α O δ ( t ) ➡ DFT modifications 34

Experimental results • Bound validity • Bound tightness • Order of magnitude ➡ online demo: IPOL · Image Processing On Line 35

Experimental results • Bound validity • Bound tightness • Order of magnitude ➡ online demo: IPOL · Image Processing On Line 36

Experimental results • Bound validity • Bound tightness • Order of magnitude ➡ online demo: IPOL · Image Processing On Line 37

Experimental results • Bound validity • Bound tightness • Order of magnitude ➡ online demo: IPOL · Image Processing On Line 38

Bound tightness Sinc Sinc w/o DFT µ q ➡ Smooth image ➡ E [quant 2 ] = 0 . 3 39

Bound tightness Sinc Sinc w/o DFT µ q ➡ Simulated white-noise ➡ E [quant 2 ] = 0 . 3 40

Bound tightness Sinc Sinc w/o DFT µ q ➡ Textured image ➡ E [quant 2 ] = 0 . 3 41

Other kernels? • Bound validity • Bound tightness • Order of magnitude ➡ online demo: IPOL · Image Processing On Line 42

Other kernels? Sinc w/o µ 43

Other kernels? Sinc + accel 44

Other kernels? Bilinear 45

Other kernels? Bicubic 46

Other kernels? B-Spline 3 47

Other kernels? B-Spline 11 48

Conclusion • Textures are nasty • Aliasing is not the worst thing in life • Is there a hope for image interpolation? 49

Empirical estimate - ... 50

Textures • What’s special about images? 51