Efficient Neural Networks for Image Restoration Yulun Zhang Supervisor: Prof. Yun Fu SMILE lab, Northeastern University, Boston, US
Research summary Deep convolutional neural networks for Image Restoration 1.Residual dense network. [ CVPR-2018 ] Comparable STOA performance with much less parameters 2.Residual channel attention network. [ ECCV-2018 ] Very deep network with channel attention 3.Residual non-local attention network for image restoration. [ ICLR-2019 ] Channel and spatial mixed attention
Research status Residual Dense Network for Image Super-Resolution (CVPR-2018) Feature extraction in HR space SRCNN VDSR MemNet Limitations: Increase computation complexity; Blur original LR inputs Feature extraction in LR space FSRCNN SRResNet EDSR Limitations: Neglect to use hierarchical features; Hard to train very deep and wide networks Challenges : Objects have various: Hard to recover lost details. Scales; Hierarchical features in LR feature space Angles of view; Local feature extraction Aspect ratios; Hard to train very deep and wide network
Research status Residual Dense Network for Image Super-Resolution (CVPR-2018) Method Global Feature Fusion Shallow Global feature extraction Feature 1x1 Conv Extraction concat Upscale Block D Block 1 Block 2 Block d Conv Conv Conv Conv HR LR Global Residual Learning Upscale Block d Contiguous Local Memory Feature Fusion 1x1Conv Block d+1 Block d-1 concat ReLU ReLU ReLU ReLU Conv Conv Conv Conv Local Residual Learning Residual Dense Block
Research status Residual Dense Network for Image Super-Resolution (CVPR-2018) Study of D, C, and G. The number of RDB (denote as D for short), the number of Conv layers per RDB (denote as C for short), and the growth rate (denote as G for short). Analyses: Our RDN allows deeper and wider network, from which more hierarchical features are extracted for higher performance.
Research status Residual Dense Network for Image Super-Resolution (CVPR-2018) Ablation Investigation. Ablation investigation on the effects of contiguous memory (CM), local residual learning (LRL), and global feature fusion (GFF). Analyses: These quantitative and visual analyses demonstrate the effectiveness and benefits of our proposed CM, LRL, and GFF.
Research status Residual Dense Network for Image Super-Resolution (CVPR-2018) Analyses: These quantitative and visual analyses demonstrate the effectiveness and benefits of our proposed CM, LRL, and GFF.
Research status Residual Dense Network for Image Super-Resolution (CVPR-2018) Visual Results with BI Degradation Model.
Research status Residual Dense Network for Image Super-Resolution (CVPR-2018) Visual Results with BD Degradation Model.
Research status Residual Dense Network for Image Super-Resolution (CVPR-2018) Visual Results with DN Degradation Model. More results about image restoration arXiv-2018- Residual dense network for image restoration https://arxiv.org/abs/1812.10477
Research status Motivations for our next work (ECCV-2018-RCAN) Less GPU memory . Wide network could consume too much GPU memory. (4 GPUs, or 1 GPU with batch split) Smaller model size . Too further decrease network parameter number. (CVPRW-17-EDSR: 43M, CVPR-18-RDN: 22M) Better performance . Very deep network should achieve better performance.
Research status Image super-resolution using very deep residual channel attention networks (ECCV-2018) Limitations of previous methods Whether deeper networks can further contribute to image SR and how to construct very deep trainable networks remains to be explored. Deepest networks for image SR: ICCV-2017-MemNet_M10R10_212C64, CVPRW-2017-EDSR Previous networks lack distinguish ability across feature channels, and finally hinder the representational power of deep networks.
Research status Image super-resolution using very deep residual channel attention networks (ECCV-2018) Network architecture Residual Group Residual channel Residual group attention block Fg , b−1 Fg , b Fg−1 Fg Element-wise RCAB- 1 Upscale RCAB- b RCAB- B Conv sum module Short skip connection HR Residual in Residual LR F DF Fg−1 Fg RG- 1 RG- g RG- G Long skip connection Contributions We propose the very deep residual channel attention networks (RCAN) for highly accurate image SR. We propose residual in residual (RIR) structure to construct very deep trainable networks. The long and short skip connections in RIR help to bypass abundant low-frequency information and make the main network learn more effective information. We propose channel attention (CA) mechanism to adaptively rescale features by considering interdependencies among feature channels.
Research status Image super-resolution using very deep residual channel attention networks (ECCV-2018) Convergence analyses with RIR
Research status Image super-resolution using very deep residual channel attention networks (ECCV-2018) Channel attention H × W × C H × W × C 1 × 1 × C 1 × 1 × C 1 × 1 × C 1 × 1 × C r W U f W D HGP 𝑦 𝑑 𝑗, 𝑘 is the value at position ( i , j ) of c -th feature 𝑦 𝑑 . Residual channel attention block Channel attention Sigmoid Conv function Fg,b−1 Xg,b Fg,b Element-wise ReLU product Global Element-wise pooling sum
Research status Image super-resolution using very deep residual channel attention networks (ECCV-2018) Channel attention visualization c=8,s=0.0016 Low-level CA c=28,s=0.0017 c=51,s=0.9732 c=23,s=0.9998 c=48,s=0.0009 c=12,s=0.9578 High-level CA c=29,s=0.2244 c=1,s=0.2397 c=54,s=0.2699 c=56,s=0.5334 c=33,s=0.5457 c=13,s=0.5603 Figure. Channel attention visualization. Low-/high-level CAs and feature maps. c and s denote channel index and weight. In each row, we show 3 feature maps (indicated by index c) with the smallest channel weights (s) and other 3 feature maps with the largest channel weights.
Research status Image super-resolution using very deep residual channel attention networks (ECCV-2018) Ablation study Investigations of RIR and CA. We observe the best PSNR (dB) values on Set5 (2 × ) in 5 × 10 4 iterations
Research status Image super-resolution using very deep residual channel attention networks (ECCV-2018) Quantitative results
Research status Image super-resolution using very deep residual channel attention networks (ECCV-2018) Quantitative results
Research status Image super-resolution using very deep residual channel attention networks (ECCV-2018) Visual results
Research status Image super-resolution using very deep residual channel attention networks (ECCV-2018) Visual results
Research status Image super-resolution using very deep residual channel attention networks (ECCV-2018) Visual results
Research status Image super-resolution using very deep residual channel attention networks (ECCV-2018) Objective recognition performance and model size
Research status Motivations for our next work (ICLR-2019-RNAN) Effective attention mechanism . Channel attention to spatial attention, mixed attention, …. Tell noise apart from noisy input better Model generalization . Generalize our model to different image restoration tasks. Larger receptive field size . To make use of input in a more global way.
Research status Residual Non-local Attention Networks for Image Restoration (ICLR-2019) Limitations of previous methods T he receptive field size of these networks is relatively small. Distinctive ability of these networks is also limited. All channel-wise features are treated equally in those networks. Network architecture Framework Residual (non-)local attention block. Non-local block Contributions We propose the very deep residual non-local attention networks for high-quality image restoration. We propose residual non-local attention learning to train very deep networks by preserving more low-level features, being more suitable for image restoration. We demonstrate with extensive experiments that our RNAN is powerful for various image restoration tasks.
Research status Residual Non-local Attention Networks for Image Restoration (ICLR-2019) Quantitative results: color and gray-scale image denoising
Research status Residual Non-local Attention Networks for Image Restoration (ICLR-2019) Visual results: color image denoising
Research status Residual Non-local Attention Networks for Image Restoration (ICLR-2019) Visual results: gray-scale image denoising
Research status Residual Non-local Attention Networks for Image Restoration (ICLR-2019) Visual results: image demosaicing
Research status Residual Non-local Attention Networks for Image Restoration (ICLR-2019) Visual results: image compression artifact reduction
Research status Residual Non-local Attention Networks for Image Restoration (ICLR-2019) Visual results: image super-resolution
Thank you More works are available at: http://yulunzhang.com
Recommend
More recommend
