Self-Supervised Feature Learning by Learning to Spot Artifacts - PowerPoint PPT Presentation

Self-Supervised Feature Learning by Learning to Spot Artifacts Wonbin Kim

Self-Supervised Learning • “To exploit different labelings that are freely available besides or within visual data, and to use them as intrinsic reward signals to learn general- purpose features.”[5] Object Detection, Semantic Segmentation, Classification 2

Shortcut • “Trivial” and undesirable Solution to Self-Supervised Learning • Low-level statistics • Boundary Pattern or Texture • Edge continuity • The pixel intensity/color distribution • Chromatic aberration Wikipedia : Industrial_National_Bank_Building 3

Chromatic aberration • It cause different wavelengths of light to have differing focal lengths. Wikipedia : Chromatic aberration 4

Motivation • What words will come to the blank? “There is nothing noble in being superior to your fellow man; “There is nothing noble in being superior to your fellow man; true nobility is being superior to your former self.” true nobility is being to your former self.” - Ernest Hemingway - Ernest Hemingway 5

1. Context prediction Carl Doersch et al., "Unsupervised visual representation learning by context prediction.“, ICCV, (2015) 6

1. Context prediction – avoiding shortcuts • Sampling with gap and jittering • Color dropping Carl Doersch et al., "Unsupervised visual representation learning by context prediction.“, ICCV, (2015) 7

1. Context prediction[6] Carl Doersch et al., "Unsupervised visual representation learning by context prediction.“, ICCV, (2015) 8

2. Inpainting Deepak Pathak et al. "Context encoders: Feature learning by inpainting." ICCV, (2016) 9

2. Inpainting • Loss function • Reconstruction loss 2 ෡ 1 − ෡ ℒ 𝑠𝑓𝑑 𝑦 = 𝜇 𝑠𝑓𝑑 𝑁 ⊙ 𝑦 − 𝐺 𝑁 ⊙ 𝑦 2 • Adversarial loss 1 − ෡ ℒ 𝑏𝑒𝑤 = max 𝔽 𝑦∈𝜓 log 𝐸 𝑦 + log 1 − 𝐸 𝐺 𝑁 ⊙ 𝐸 • Aggregated Loss ℒ = 𝜇 𝑠𝑓𝑑 ℒ 𝑠𝑓𝑑 + 𝜇 𝑏𝑒𝑤 ℒ 𝑏𝑒𝑤 1 0

2. Inpainting Deepak Pathak et al. "Context encoders: Feature learning by inpainting." ICCV, (2016) 1 1

2. Inpainting Deepak Pathak et al. "Context encoders: Feature learning by inpainting." ICCV, (2016) 1 2

2. Inpainting Deepak Pathak et al. "Context encoders: Feature learning by inpainting." ICCV, (2016) 1 3

2. Inpainting Deepak Pathak et al. "Context encoders: Feature learning by inpainting." ICCV, (2016) 1 4

3. Solving Jigsaw puzzle Mehdi Noroozi et el., "Unsupervised learning of visual representations by solving jigsaw puzzles." ECCV, 2016. 15

3. Solving Jigsaw puzzle Mehdi Noroozi et el., "Unsupervised learning of visual representations by solving jigsaw puzzles." ECCV, 2016. 16

3. Solving Jigsaw puzzle Mehdi Noroozi et el., "Unsupervised learning of visual representations by solving jigsaw puzzles." ECCV, 2016. 17

3. Solving Jigsaw puzzle Mehdi Noroozi et el., "Unsupervised learning of visual representations by solving jigsaw puzzles." ECCV, 2016. 18

3. Distortion Simon Jenni et el., "Self-Supervised Feature Learning by Learning to Spot Artifacts." CVPR (2018). 19

4. Distortion Simon Jenni et el., "Self-Supervised Feature Learning by Learning to Spot Artifacts." CVPR (2018). 20

4. Distortion • Decoder with Repair Network. • i=1 ෠ 𝐸 1 = 𝜚 𝑦 + 1 − Ω ⊙ 𝑆 1 𝜚 𝑦 , Where ෠ 𝜚 x = Ω ⊙ 𝜚 x + (1 − Ω) ⊙ 𝑣 ∗ 𝜚 𝑦 • i=2, 3, 4 𝐸 𝑗 = 𝐸 𝑗−1 + 1 − 𝑉 𝑗−1 Ω ⊙ 𝑆 𝑗 𝐸 𝑗−1 Simon Jenni et el., "Self-Supervised Feature Learning by Learning to Spot Artifacts." CVPR (2018). 21

4. Distortion • Loss function • Discriminator adversarial Loss : {real , corruption} log 𝐷 𝑑𝑚𝑏𝑡𝑡 𝐸 𝜚 𝐲 ෡ + log 1 − 𝐷 𝑑𝑚𝑏𝑡𝑡 ෡ ℒ 𝑑𝑚𝑏𝑡𝑡 = min 𝑆 max ෍ 𝐸 𝜚 𝐲 . 𝐷 𝐲~p 𝐲 • Discriminator binary classification Loss : mask prediction 𝑛𝑏𝑡𝑙 ො 𝑛𝑏𝑡𝑙 ෡ ෡ ℒ 𝑛𝑏𝑡𝑙 = min 𝐷 ෍ ෍ Ω 𝑗𝑘 log 𝜏 𝐷 𝑗𝑘 𝐲 + 1 − Ω 𝑗𝑘 log 1 − 𝜏 𝐷 𝑗𝑘 𝐸 𝜚 𝐲 𝐲 ො 𝑗𝑘 Simon Jenni et el., "Self-Supervised Feature Learning by Learning to Spot Artifacts." CVPR (2018). 22

4. Distortion Simon Jenni et el., "Self-Supervised Feature Learning by Learning to Spot Artifacts." CVPR (2018). 23

Reference • [1] Jenni, Simon, and Paolo Favaro. "Self-Supervised Feature Learning by Learning to Spot Artifacts." CVPR (2018). • [2] Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016. • [3] Noroozi, Mehdi, and Paolo Favaro. "Unsupervised learning of visual representations by solving jigsaw puzzles." European Conference on Computer Vision. Springer, Cham, 2016. • [4] Doersch, Carl, Abhinav Gupta, and Alexei A. Efros. "Unsupervised visual representation learning by context prediction." Proceedings of the IEEE International Conference on Computer Vision. 2015. • [5] Wikipedia : Chromatic aberration; https://en.wikipedia.org/wiki/Chromatic_aberration 24