A Concurrent Deep Learning Model to Remove Reflections Boxin Shi and - PowerPoint PPT Presentation

A Concurrent Deep Learning Model to Remove Reflections Boxin Shi and Renjie Wan shiboxin@pku.edu.cn, wanpeoplejie@gmail.com Collaborators: Ling-Yu Duan, Ah-Hwee Tan, and Alex C. Kot

Outline 2  Problem background  Two-stage framework based methods  Low-level image prior based methods  ICIP16, TIP18  Learning based solutions  Limitations  Breaking the limitations of two-stage framework  SIR 2 benchmark dataset  ICCV17  CRRN: a deep learning model to remove reflections  CVPR18

Problem background 3 Camera Reflection Glass Background

Problem background 4 Images are from “Li et al. Exploiting Reflection Change for Automatic Reflection Removal . ICCV 2013”

Problem background 5  Difficulties of this problem  Estimate two unknown parameters from one equations  The similarity between background and reflection Mixture image Background Reflection 𝐂 𝐒 𝐉

Related work 6  A two-stage framework: Detection and Removal. Results Detection Removal 𝑄 𝑀 𝐶 , 𝑀 𝑆 = 𝑄 1 (𝑀 𝐶 ) ∙ 𝑄 2 (𝑀 𝑆 ) Background Reflection AY07: Levin et al. User assisted separation of reflections from a single image using a sparsity prior. TPAMI 2007

Related work 7 Detection Image sequence Results Removal 𝑄 𝑀 𝐶 , 𝑀 𝑆 = 𝑄 1 (𝑀 𝐶 ) ∙ 𝑄 2 (𝑀 𝑆 ) Background edges Reflection edges Li et al. Exploiting Reflection Change for Automatic Reflection Removal . ICCV 2013

Related work 8 Result Detection Removal Mixture image DoF confidence map 𝑄 𝑀 𝐶 , 𝑀 𝑆 = 𝑄 1 (𝑀 𝐶 ) ∙ 𝑄 2 (𝑀 𝑆 ) Background edges Reflection edges WS16: Wan et al. “Depth of field guided reflection removal” ICIP 2016

Related work 9  Regional properties of reflections  Only cover a very small region WS18: Wan et al. “Region aware reflection removal with unified content and gradient priors” TIP 2018

Related work 10  Learning based methods with two-stage framework  Noroozi et al. ConvNet-based Depth Estimation, Reflection Separation and Deblurring of Plenoptic Images. ACCV 2016  Fan, et al. A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing. CVPR 2017 Edge extraction Image reconstruction

Related work 11  Not dependent on the two-stage framework  LB14: Li Yu et al. Single Image Layer Separation using Relative Smoothness  NR17: N Arvanitopoulos et al. Single image reflection suppression  SK15: Shih et al. Reflection Removal using Ghosting Cues Background image Reflection image Mixture image 2 ) 𝐧𝐣𝐨 𝑀 1 ,𝑀 2 ෍ (𝜍 𝑀 1 𝑗 + 𝜐(𝑀 2 ) 𝑘 𝑗,𝑘

Related work 12  Not dependent on the two-stage framework  LB14: Li Yu et al. Single Image Layer Separation using Relative Smoothness  NR17: N Arvanitopoulos et al. Single image reflection suppression  SK15: Shih et al. Reflection Removal using Ghosting Cues Image smoothing

Related work 13  Not dependent on the two-stage framework  LB14: Li Yu et al. Single Image Layer Separation using Relative Smoothness  NR17: N Arvanitopoulos et al. Single image reflection suppression  SK15: Shih et al. Reflection Removal using Ghosting Cues Mixture image Result

Limitations 14  The limitations of the two-stage framework.  Highly depend on specific scenarios.  Limited description ability to the reflection properties. Mixture image Result obtained by NR17 Blurring effects or ghosting effects.  Mixture image Result by SK15 Failure case of ghosting effects Failure case of blurring effects

Breaking the two-stage limitations 15 Benchmark data w/ g.t. A concurrent network

SIngle-image Reflection Removal dataset SIR 2 : Motivations 16 LB14 SK15

SIngle-image Reflection Removal dataset SIR 2 : Motivations 17 LB14 Not available SK15

SIngle-image Reflection Removal dataset SIR 2 : Motivations 18 LB14 Not enough Not available SK15 Not enough

SIngle-image Reflection Removal dataset SIR 2 : A benchmark dataset 19 Reflection Glass Background Background Reflection

SIngle-image Reflection Removal dataset SIR 2 : A benchmark dataset 20 Reflection Glass Black paper Background Background

SIngle-image Reflection Removal dataset SIR 2 : A benchmark dataset 21 Reflection Background

SIR 2 : Types of reflections 22

SIR 2 : Images with different reflections 23  Different parameters to explore the influence of different settings.  Seven different aperture sizes and 3 different thickness settings in the postcard and solid object dataset.  Different indoor and outdoor scenes in the uncotrolled scene dataset.

SIR 2 : Various scenarios 24  Image triplets taken in different scenarios.  The postcard dataset (200 image triplets and 600 images in total).  The solid object dataset (200 image triplets and 600 images in total).  The wild scene dataset (100 scenes and 300 images in total). Mixture image Background Reflection

SIR 2 : Various scenarios 25  Image triplets taken in different scenarios.  The postcard dataset (200 image triplets and 600 images in total).  The solid object dataset (200 image triplets and 600 images in total).  The wild scene dataset (100 scenes and 300 images in total). Mixture image Background Reflection

SIR 2 : Various scenarios 26  Image triplets taken in different scenarios.  The postcard dataset (200 image triplets and 600 images in total).  The solid object dataset (200 image triplets and 600 images in total).  The wild scene dataset (100 scenes and 300 images in total). Background Mixture image Reflection Accepted by ICCV 2017. More details can be found here: https://sir2data.github.io

SIR 2 : Limitations of evaluated methods 27  The ignorance of the regional properties of reflections  The highly dependence to specific priors  Ghosting effects and blurring effects Mixture image Result obtained by NR17 Mixture image Result by SK15 Failure case of ghosting effects Failure case of blurring effects

CRRN: Deep learning based methods 28 Depth extraction Image reconstruction Noroozi et al. ConvNet-based Depth Estimation, Reflection Separation and Deblurring of Plenoptic Images. ACCV 2016 Edge extraction Image reconstruction FY17: Fan et al. A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing. ICCV 2017

CRRN: Training data preparation 29 FY17 𝐉 = 𝐂 + 𝐒 𝐉 = 𝐂 + 𝐒 ∗ 𝒊 LB14, WS16… 𝐉 = 𝐂 + 𝐒 ∗ (𝜷𝜺 𝟐 + 𝜸𝜺 𝟐 ) SK15

CRRN: Training data preparation 30 FY17 𝐉 = 𝐂 + 𝐒 𝐉 = 𝐂 + 𝐒 ∗ 𝒊 LB14, WS16… 𝐉 = 𝐂 + 𝐒 ∗ (𝜷𝜺 𝟐 + 𝜸𝜺 𝟐 ) SK15

CRRN: Training data preparation 31 FY17 𝐉 = 𝐂 + 𝐒 𝐉 = 𝐂 + 𝐒 ∗ 𝒊 LB14, WS16… 𝐉 = 𝐂 + 𝐒 ∗ (𝜷𝜺 𝟐 + 𝜸𝜺 𝟐 ) SK15

CRRN: Training data preparation 32  3250 reflection images taken from different places

CRRN: Network structure 33 IiN: Image inference network 𝟓 × 𝟓 × 𝟑𝟔𝟕 𝟓 × 𝟓 × 𝟐𝟑𝟗 𝟓 × 𝟓 × 𝟕𝟓 𝟓 × 𝟓 × 𝟒𝟑 𝟓 × 𝟓 × 𝟐𝟕 𝟒 × 𝟒 × 𝟑𝟔𝟕 𝟒 × 𝟒 × 𝟐𝟕 𝟒 × 𝟒 × 𝟒 Estimated 𝐒 ∗ Estimated 𝐂 ∗ Fine-tuned VGG model Encoder Decoder 𝟖 × 𝟖 × 𝟐𝟏𝟑𝟓 𝟒 × 𝟒 × 𝟐𝟑𝟗 𝟓 × 𝟓 × 𝟐𝟑𝟗 𝟒 × 𝟒 × 𝟑𝟔𝟕 𝟓 × 𝟓 × 𝟑𝟔𝟕 𝟒 × 𝟒 × 𝟔𝟐𝟑 𝟓 × 𝟓 × 𝟔𝟐𝟑 𝟒 × 𝟒 × 𝟔𝟐𝟑 𝟓 × 𝟓 × 𝟔𝟐𝟑 𝟐 × 𝟐 × 𝟔𝟐𝟑 𝟒 × 𝟒 × 𝟑𝟔𝟕 𝟓 × 𝟓 × 𝟑𝟔𝟕 𝟓 × 𝟓 × 𝟐𝟑𝟗 𝟒 × 𝟒 × 𝟐𝟑𝟗 𝟒 × 𝟒 × 𝟕𝟓 𝟓 × 𝟓 × 𝟕𝟓 𝟒 × 𝟒 × 𝟕𝟓 𝟓 × 𝟓 × 𝟕𝟓 𝟒 × 𝟒 × 𝟒𝟑 𝟓 × 𝟓 × 𝟒𝟑 𝟓 × 𝟓 × 𝟕𝟓 𝟔 × 𝟔 × 𝟐 Input image Estimated gradient Input gradient GiN: Gradient inference network Multi-scale guided inference Cov layers (stride = 1, 2) Max-pooling layers De-conv layers (stride =2) Feature extraction layers A\B Concat operation

CRRN: Network structure 34 IiN: Image inference network 𝟓 × 𝟓 × 𝟑𝟔𝟕 𝟓 × 𝟓 × 𝟐𝟑𝟗 𝟓 × 𝟓 × 𝟕𝟓 𝟓 × 𝟓 × 𝟒𝟑 𝟓 × 𝟓 × 𝟐𝟕 𝟒 × 𝟒 × 𝟑𝟔𝟕 𝟒 × 𝟒 × 𝟐𝟕 𝟒 × 𝟒 × 𝟒 Estimated 𝐒 ∗ Estimated 𝐂 ∗ Fine-tuned VGG model Encoder Decoder 𝟖 × 𝟖 × 𝟐𝟏𝟑𝟓 𝟒 × 𝟒 × 𝟐𝟑𝟗 𝟓 × 𝟓 × 𝟐𝟑𝟗 𝟒 × 𝟒 × 𝟑𝟔𝟕 𝟓 × 𝟓 × 𝟑𝟔𝟕 𝟒 × 𝟒 × 𝟔𝟐𝟑 𝟓 × 𝟓 × 𝟔𝟐𝟑 𝟒 × 𝟒 × 𝟔𝟐𝟑 𝟓 × 𝟓 × 𝟔𝟐𝟑 𝟐 × 𝟐 × 𝟔𝟐𝟑 𝟒 × 𝟒 × 𝟑𝟔𝟕 𝟓 × 𝟓 × 𝟑𝟔𝟕 𝟓 × 𝟓 × 𝟐𝟑𝟗 𝟒 × 𝟒 × 𝟐𝟑𝟗 𝟒 × 𝟒 × 𝟕𝟓 𝟓 × 𝟓 × 𝟕𝟓 𝟒 × 𝟒 × 𝟕𝟓 𝟓 × 𝟓 × 𝟕𝟓 𝟒 × 𝟒 × 𝟒𝟑 𝟓 × 𝟓 × 𝟒𝟑 𝟓 × 𝟓 × 𝟕𝟓 𝟔 × 𝟔 × 𝟐 Input image Estimated gradient Input gradient GiN: Gradient inference network Multi-scale guided inference Cov layers (stride = 1, 2) Max-pooling layers De-conv layers (stride =2) Feature extraction layers A\B Concat operation