Upscaling Beyond SuperResolution Using a Novel DeepLearning System - PowerPoint PPT Presentation

Upscaling Beyond Super–Resolution Using a Novel Deep–Learning System Pablo Navarrete Michelini pnavarre@boe.com.cn Hanwen Liu lhw@boe.com.cn BOE Technology Group Co., Ltd.

BOE Technology Group Co., Ltd.

BOE Ultra–HD Panels

Chapter I : “The Layer and the System”

Standard Upscaling � 1 / 4 1 / 2 1 / 4 � For example, a simple linear interpolation can be done with F = . 1 / 2 1 1 / 2 1 / 4 1 / 2 1 / 4

Standard Upscaling Efficient implementation avoids multiplying zeros. Break F into many filters W i .

Classic Upscalers Classic Upscalers: Nearest Neighbor, Linear, Bicubic, Lanczos, . . . Advanced Upscalers: Directional filters (NEDI), wavelets, . . . (a) Original (b) Nearest Neighbor (c) Bicubic Figure: Classic Upscalers

GTC–2016 MuxOut

GTC–2016 MuxOut

GTC–2016 MuxOut

GTC–2016 MuxOut

GTC–2016 MuxOut

GTC–2016 MuxOut

Similar Approaches SRCNN Dong C., et.al., “Learning a Deep Convolutional Network for Image Super–Resolution.” Sept 2014. BOE MuxOut Navarrete P., et.al., “Upscaling with Deep Convolutional Networks and Muxout Layers.” May 2016. Google RAISR Romano Y., “RAISR: Rapid and Accurate Image Super Resolution.” Jun 2016. Twitter ESPCN Shi W., et.al, “Real–Time Single Image and Video Super–Resolution Using an Efficient Sub–Pixel Convolutional Neural Network.” Sept 2016. Twitter GAN Ledig C., et.al., “Photo-Realistic Single Image Super–Resolution Using a Generative Adversarial Network.” Sept 2016. Twitter GAN Sønderby C.K., et.al, “Amortised MAP Inference for Image Super-resolution.” Oct 2016.

Similar Approaches Twitter ESPCN: “Sub–pixel convolution layer” = “MuxOut r × r ”. Differences: MuxOut considers several groups of r 2 features. MuxOut is design to factorize r and use as several layers within the network. Figure from: Shi W., et.al, “Real–Time Single Image and Video Super–Resolution Using an Efficient Sub–Pixel Convolutional Neural Network.” Sept 2016.

Similar Approaches Google RAISR: Uses a ML–approach to learn adaptive filters. Not based on convolutional–networks. Similarity: We will show how to interpret the convolutional–network approach as an adaptive filter. Figure from: Romano Y., “RAISR: Rapid and Accurate Image Super Resolution.” Jun 2016.

MuxOut Layer – Old Version Problems of MuxOut: Reduces processing features Works very well only with easy content (e.g. text). Why? Filter parameters W ahve 2 tasks: Downsampling: Which combination of a – b – c – d works better? Filter: Which values work better for interpolation?

MuxOut Layer – New Version New Version: Consider all (or most) possible combinations of features Can keep the same number of processing features. Filter parameters can focus on interpolation. SGD algorithms converge fast and stable.

MuxOut Usage A convolutional–layer block typically means: �� � c in x c in ∗ W c in , c out + b c out conv ( x c in ) = σ We will consider convolution and activation independently: x c in ∗ W c in , c out conv ( x c in ) � = c in σ ( x c + b c ) activ ( x c ) = And we use MuxOut like:

First Approach Problem: Large upscaling factors color might be misaligned with lumminance.

Full–Color Configuration Idea: RGB Input + RGB Output Problem: MuxOut mixes color channels. Need to process separately:

Chroma Sub–sampling Configuration Note: HVS is less sensitive to the position and motion of color than luminance.

MSE vs SSIM Traditional approach: H , W 1 � ( X i , j − Y i , j ) 2 Loss ( X , Y ) = MSE( X , Y ) = H · W i , j =0 Problem: Not well correlated with HVS Why not PSNR? → PSNR is unbounded. (2 µ X µ Y + C 1 )(2 σ XY + C 2 ) Loss ( X , Y ) = SSIM( X , Y ) = ( µ 2 X + µ 2 Y + C 1 )( σ 2 X + σ 2 Y + C 2 ) Well correlated with HVS. Differentiable. Behaves well with SGD.

Results (a) Standard (PSNR 24.82 dB – SSIM 0.8463) (b) Ours (PSNR 27.31 dB – SSIM 0.8990)

Chapter II : “The Analysis”

Analysis Linear Systems: Interpolation filter given by impulse response. CN: Is not Linear because of ReLU. (c) Activity Recorder (d) Mask Layer Use an input image and record activity. Replace all activations (ReLU) by a “Mask layer”. The system becomes linear! Check impulse response.

Analysis

Analysis

Analysis

Analysis

Analysis

Analysis

Analysis

Analysis

Analysis

Analysis

Analysis

Analysis

Analysis

Chapter III : “Hyper–Resolution”

Hyper–Resolution We say that x and y are aliases if Downscale ( x ) = Downscale ( y ) Many realistic images are aliased. MSE, SSIM, etc aim for only one alias. MSE, SSIM traget removes the innovation process! (e.g. linear regression). Give up the original content. We just want it to “look real”.

Hyper–Resolution Generator (Upscaler) Discriminator

Generative Adversarian Networks (GAN) Increasing attention and significant progress in the last year. We will refer to the following important references: WGAN: Arjovsky M., et.al., “Wasserstein GAN.” Jan 2017. Improved WGAN: Gulrajani I., et.al., “Improved Training of Wasserstein GANs.” March 2017. Losses: � x ) � 2 − 1) 2 � L D = E [ D ( x fake )] − E [ D ( x real )] + λ gp E ( �∇ ˆ x D (ˆ = − E [ D ( x fake )] + λ LR ∆ (Downscale( G ( x LR )) , x LR ) L G

Generative Adversarian Networks (GAN) We do not want to reveal the high–resolution content during the Upscaler’s training. We do not want to generate artificial images with no reference to the input. We ask the upscaler to be able to recover the low–resolution input with a standard downscaler (e.g. area). ∆ ( Downscale( G ( x LR )) , x LR ) with: ∆ ( x , y ) = MSE ( x , y ) or ∆ ( x , y ) = 1 − SSIM ( x , y )

Results (e) Standard (PSNR 29.78 dB) (f) Original (PSNR ∞ ) (g) Ours (PSNR 25.68 dB)

Results (h) Standard (PSNR 29.78 dB) (i) Original (PSNR ∞ ) (j) Ours (PSNR 25.68 dB)

Results (k) Standard (PSNR 29.78 dB) (l) Original (PSNR ∞ ) (m) Ours (PSNR 25.68 dB)

Conclusions Overview: System: Proposed improved MuxOut. Analysis: Novel approach to visualize CN as adaptive filter. Super–Resolution: Proposed SSIM loss and process color input/output. Hyper–Resolution: Hallucinating details using GAN can produce results comparable to original content. Next Steps: Larger upscaling factors. Use analysis to improve design and test for other problems. Improve generalization of GAN approach.

Questions & Answers Thank you! LinkedIn: https://www.linkedin.com/in/pnavarre ResearchGate: https://www.researchgate.net/profile/Pablo_Navarrete_Michelini