image restoration with neural networks
play

IMAGE RESTORATION WITH NEURAL NETWORKS Orazio Gallo Work with Hang - PowerPoint PPT Presentation

Apr 16, 2023 •8 likes •468 views

IMAGE RESTORATION WITH NEURAL NETWORKS Orazio Gallo Work with Hang Zhao, Iuri Frosio, Jan Kautz MOTIVATION The long path of images AF/AE Bad Pixel Black Correction Level Lens Demosaic Denoise Metering Correction Image Tone

  1. IMAGE RESTORATION WITH NEURAL NETWORKS Orazio Gallo Work with Hang Zhao, Iuri Frosio, Jan Kautz

  4. MOTIVATION The long path of images… AF/AE Bad Pixel Black Correction Level Lens Demosaic Denoise Metering Correction Image Tone Enhancing Mapping Image Signal Processor (ISP)

  5. Image credit: Wikipedia colors by interpolation DEMOSAICING Image credit: Marc Levoy

  6. DENOISING Several types of noise involved in the image formation: • Photon shot noise Dark current (AKA thermal noise) • • Photo-response non-uniformity Vignetting • Readout noise: • • Reset noise (charge-to-voltage transfer) White noise (during voltage amplification amplification) • • Quantization noise (ADC)

  7. DENOISING

  8. MOTIVATION AF/AE Bad Pixel Black Correction Level Lens Demosaic Denoise Metering Correction Image Tone Enhancing Mapping Demosaicing before denoising changes the statistics of the noise. And the best de-noising algorithms require to know what the noise looks like.

  9. MOTIVATION AF/AE Bad Pixel Black Correction Level Lens Denoise Demosaic Metering Correction Image Tone Enhancing Mapping Denoising first can change the color reproduction accuracy as the three channels may be denoised differently.

  10. FLEXISP 1 A Flexible Camera Image Processing Framework PSF CFA Noise [1] Heide et al., ACM SIGGRAPH Asia 2012 (ToG)

  11. CAN WE DO IT WITH A NEURAL NETWORK? Can we do it with a neural network, which moves the heavy lifting to the training stage and inference is very quick?

  12. JOINT DEMOSAICING AND DENOISING

  13. JOINT DEMOSAICING AND DENOISING bilinear interpolation Network architecture convolution convolution convolution

  14. MEASURING IMAGE QUALITY Original Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/

  15. MEASURING IMAGE QUALITY Higher sensitivity to errors in texture-less regions! Wang, et al. "Image quality assessment: from error visibility to structural similarity." IEEE TIP (2004)

  16. MEASURING IMAGE QUALITY Original 0.662 0.988 Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/

  17. MEASURING IMAGE QUALITY Higher sensitivity to errors in texture-less regions!

  18. JOINT DEMOSAICING AND DENOISING bilinear interpolation Network architecture convolution convolution convolution

  19. JOINT DEMOSAICING AND DENOISING Network training Training data 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) Input - noisy image (realistic noise model) - bilinear interpolation Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

  20. Ground truth

  21. Noisy

  24. RESULTS Visual comparison (+ unsharp masking) Noisy BM3D (state of the art) Ground truth

  25. Noisy

  28. RESULTS Visual comparison (+ unsharp masking) Noisy BM3D (state of the art) Ground truth

  29. JOINT DEMOSAICING AND DENOISING: RESULTS Average image quality metrics on the testing dataset

  30. DOES IT GENERALIZE? JPEG ARTIFACT REMOVAL & SUPER-RESOLUTION

  31. JPEG ARTIFACT REMOVAL Network training Training data 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) Input JPEG compressed image , 25% quality Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

  32. JPEG ARTIFACT REMOVAL: RESULTS Visual comparison (+ unsharp masking) JPEG L2 L1 + MS-SSIM Ground truth

  33. JPEG ARTIFACT REMOVAL: RESULTS Numerical comparison Average image quality metrics on the testing dataset

  34. SUPER-RESOLUTION Network training Training data 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) Input 2x downsampled image + upsampled with bilinear interpolation Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

  35. SUPER-RESOLUTION: RESULTS Visual comparison (+ unsharp masking) Low rez L2 L1 + MS-SSIM

  36. SUPERRESOLUTION: RESULTS Numerical comparison and literature

  37. LEARNINGS?

  38. LEARNINGS A closer look at the different losses

  39. LEARNINGS and

  40. LEARNINGS and 0.3939 0.3896 • seems to have more convergence issues. converges faster and speeds up the convergence or other losses, too. •

  41. LEARNINGS A closer look at the different losses

  42. LEARNINGS SSIM and MS-SSIM “Higher sensitivity to errors in texture - less regions!” Multi-scale is helpful when dealing with transition regions. •

  43. LEARNINGS A closer look at the different losses

  44. RESULTS Why mixing MS-SSIM and ?

  45. CONCLUSIONS What have we learnt? • Even a shallow network can produce state-of-the- art results… …if you train it carefully. • Perceptually-motivated loss functions can help! • • But you have to be aware of their limitations!

  46. Thanks! Zhao, Gallo, Frosio, and Kautz, “Loss Functions for Image Restoration with Neural Networks”, IEEE Trans. on Comp. Imaging, 2017

Recommend

Efficient Neural Networks for Image Restoration Yulun Zhang Supervisor: Prof. Yun Fu SMILE lab,

Efficient Neural Networks for Image Restoration Yulun Zhang Supervisor: Prof. Yun Fu SMILE lab,

Efficient Neural Networks for Image Restoration Yulun Zhang Supervisor: Prof. Yun Fu SMILE lab, Northeastern University, Boston, US Research summary Deep convolutional neural networks for Image Restoration 1.Residual dense network. [ CVPR-2018

375 views • 33 slides
Image Restoration Image Enhancement and Image Restoration both deal with improving images. Image

Image Restoration Image Enhancement and Image Restoration both deal with improving images. Image

Image Restoration Image Enhancement and Image Restoration both deal with improving images. Image Analysis In Image Enhancement, the quality of the improved image was Image Restoration and Reconstruction judged subjectively. In contrast, Image

349 views • 8 slides
Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional Neural Networks

648 views • 26 slides
Advanced Section #8: Neural Networks for Image Analysis Camilo Fosco CS109A Introduction to Data

Advanced Section #8: Neural Networks for Image Analysis Camilo Fosco CS109A Introduction to Data

Advanced Section #8: Neural Networks for Image Analysis Camilo Fosco CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader 1 Outline Image analysis: why neural networks? Multi Layer Perceptron refresher

899 views • 68 slides
Uncertainty Estimation in Deep Neural Networks for Dermoscopic Image Classification Marc

Uncertainty Estimation in Deep Neural Networks for Dermoscopic Image Classification Marc

Uncertainty Estimation in Deep Neural Networks for Dermoscopic Image Classification Marc Combalia, Ferran Hueto, Susana Puig, Josep Malvehy, Veronica Vilaplana Introduction Neural Networks in HealthCare High performance of AI in HealthCare

958 views • 22 slides
Learning and transferring mid-level image representions using convolutional neural networks

Learning and transferring mid-level image representions using convolutional neural networks

Willow project-team Learning and transferring mid-level image representions using convolutional neural networks Maxime Oquab, Lon Bottou, Ivan Laptev, Josef Sivic 1 mardi 5 aot 14 Image classification (easy) Is there a car ? Source :

776 views • 61 slides
Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional

444 views • 27 slides
Neural Network Basics Part II Content Image-to-image Why fully convolutional?

Neural Network Basics Part II Content Image-to-image Why fully convolutional?

Neural Network Basics Part II Content Image-to-image Why fully convolutional? Fully Convolutional Networks (FCN) Up-sampling Network architecture Recurrent Neural Networks Sequence data and representation

625 views • 45 slides
Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training Neural Networks 2.Leveraging heterogeneous hardware Source: wikipedia Modern Neural Networks Applications: Image classification Modern Neural

513 views • 20 slides
Image Reconstruction of EEG data using Convolutional Neural Networks Pieter Uys Biomedical

Image Reconstruction of EEG data using Convolutional Neural Networks Pieter Uys Biomedical

Image Reconstruction of EEG data using Convolutional Neural Networks Pieter Uys Biomedical Engineering Research Group Stellenbosch University Supervisor: Dr. D. van den Heever August 17, 2018 Thesis presentation Aug 17 th , 2018 Image

350 views • 9 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl Dickstein Tom Brady Deep Neural Networks Recent successes in using Deep neural networks for image classification,

1.32k views • 17 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Convolutional Neural Networks for Image Classification Dan C. Cirean , Ueli Meier, Jonathan

Convolutional Neural Networks for Image Classification Dan C. Cirean , Ueli Meier, Jonathan

Flexible, High Performance Convolutional Neural Networks for Image Classification Dan C. Cirean , Ueli Meier, Jonathan Masci, Luca M. Gambardella, Jrgen Schmidhuber IDSIA, USI and SUPSI Manno-Lugano, Switzerland

475 views • 15 slides
RNNs for Image Caption Generation James Guevara Recurrent Neural Networks Contain at least

RNNs for Image Caption Generation James Guevara Recurrent Neural Networks Contain at least

RNNs for Image Caption Generation James Guevara Recurrent Neural Networks Contain at least one directed cycle. Applications include: pattern classification, stochastic sequence modeling, speech recognition. Train using

681 views • 22 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides
Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides
Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides
Image Space Embeddings and Generalized Convolutional Neural Networks Nate Strawn September

Image Space Embeddings and Generalized Convolutional Neural Networks Nate Strawn September

Image Space Embeddings and Generalized Convolutional Neural Networks Nate Strawn September 20th, 2019 Georgetown University Table of Contents 1. Introduction 2. Smooth Image Space Embeddings 3. Example: Dictionary Learning 4. Convolutional

530 views • 49 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides
KD-MRI: A knowledge distillation framework for image reconstruction and image restoration in MRI

KD-MRI: A knowledge distillation framework for image reconstruction and image restoration in MRI

KD-MRI: A knowledge distillation framework for image reconstruction and image restoration in MRI workflow Balamurali Murugesan , Sricharan Vijayarangan, Kaushik Sarveswaran, Keerthi Ram, Mohanasankar Sivaprakasam Healthcare Technology Innovation

531 views • 12 slides

More recommend