conditio ditional al generati ative adversa sarial al
play

Conditio ditional al Generati ative Adversa sarial al - PowerPoint PPT Presentation

Conditio ditional al Generati ative Adversa sarial al Networks works (cGANs Ns) conti tinued! nued! Prof. Leal-Taix and Prof. Niessner 1 Paired vs Unpair ired Settin ing Prof. Leal-Taix and Prof. Niessner 2 pix2pix ix: :


  1. Conditio ditional al Generati ative Adversa sarial al Networks works (cGANs Ns) conti tinued! nued! Prof. Leal-Taixé and Prof. Niessner 1

  2. Paired vs Unpair ired Settin ing Prof. Leal-Taixé and Prof. Niessner 2

  3. pix2pix ix: : Image-to to-Ima mage Transla latio ion slides credit: Isola / Zhu

  4. z G(z) G D real or fake? Generator Discriminator min 𝐻 max 𝔽 𝑨,𝑦 log 𝐸(𝐻 𝑨 ) + log(1 − 𝐸 𝑦 ) 𝐸 [Goodfellow et al. 2014] slides credit: Isola / Zhu

  5. x G(x) G D real or fake? Generator Discriminator min 𝐻 max 𝔽 𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 ) 𝐸 slides credit: Isola / Zhu

  6. x G(x) Real! G D Generator Discriminator min 𝐻 max 𝔽 𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 ) 𝐸 slides credit: Isola / Zhu

  7. x G(x) Real too! G D Generator Discriminator min 𝐻 max 𝔽 𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 ) 𝐸 slides credit: Isola / Zhu

  8. x G(x) G D real or fake pair ? min 𝐻 max 𝔽 𝑦,𝑧 log 𝐸(𝑦, 𝐻 𝑦 ) + log(1 − 𝐸 𝑦, 𝑧 ) 𝐸 fake pair real pair match joint distribution p G x , y ∼ p(x, y) slides credit: Isola / Zhu

  9. pix2pix ix 9

  10. pix2pix: ix: Paired Settin ing • Great when we have ‘free’ training data • Often called self-supervised • Think about these settings  10

  11. Edges → Images Input Output Input Output Input Output Edges from [Xie & Tu, 2015] slides credit: Isola / Zhu

  12. Sketches → Images Input Output Input Output Input Output Trained on Edges → Images Data from [Eitz, Hays, Alexa, 2012] slides credit: Isola / Zhu

  13. #edges2cats [Christopher Hesse] @gods_tail @matthematician Vitaly Vidmirov @vvid Ivy Tasi @ivymyt https://affinelayer.com/pixsrv/ slides credit: Isola / Zhu

  14. Input Output Groundtruth Data from [maps.google.com] slides credit: Isola / Zhu

  15. BW → Color Input Output Input Output Input Output Data from [Russakovsky et al. 2015] slides credit: Isola / Zhu

  16. Ideas behind Pix2Pi Pix • 𝑀 = 𝑀 𝐻𝐵𝑂 + 𝜇𝑀 1 (makes it more constrained) • Unet / skip connections for preserving structure • Noise only through dropout – cGANs tend to learn to ignore the random vector z – Still want probabilistic model Prof. Leal-Taixé and Prof. Niessner 16

  17. Ideas behind Pix2Pi Pix • L1 or L2 loss for low frequency details • GAN discriminator for high frequency details -> PatchGAN – GAN discriminator applied only to local patches – It’s fully -convolutional; i.e., can run on arbitrary image sizes Prof. Leal-Taixé and Prof. Niessner 17

  18. Pix2Pi PixHD • Expand the pix2pix idea to multi-scale • Coarse-to-fine generator + discriminator • G’s and D’s are the same but since they operate on different resolutions, they have effectively a larger receptive field Prof. Leal-Taixé and Prof. Niessner 18 [Wang et al. 18]

  19. Pix2Pi PixHD Prof. Leal-Taixé and Prof. Niessner 19 [Wang et al. 18]

  20. Pix2Pi PixHD • Use of multi-scale discriminators • min 𝑙=1,2,3 𝑀 𝐻𝐵𝑂 (𝐻, 𝐸 𝑙 ) max 𝐻 𝐸 1 ,𝐸 2 ,𝐸 3 • Can make various combinations of stacking discriminator and generator – E.g., have a single G and downsample generated and real images – or have intermediate real images (cf. ProGAN) Prof. Leal-Taixé and Prof. Niessner 20 [Wang et al. 18]

  21. Pix2Pi PixHD Prof. Leal-Taixé and Prof. Niessner 21 [Wang et al. 18]

  22. Pix2Pi PixHD Prof. Leal-Taixé and Prof. Niessner 22 [Wang et al. 18]

  23. Pix2Pi PixH xHD D (interactive ive result lts) Prof. Leal-Taixé and Prof. Niessner 23 [Wang et al. 18]

  24. Paired Label ↔ photo: per-pixel labeling Horse ↔ zebra: how to get zebras? … - Expensive to collect pairs. - Impossible in many scenarios. slides credit: Isola / Zhu

  25. Paired Unpaired … … … slides credit: Isola / Zhu

  26. x G(x) G D Generator No input-output pairs! slides credit: Isola / Zhu

  27. x G(x) Real! G D Generator Discriminator slides credit: Isola / Zhu

  28. x G(x) Real too! G D Generator Discriminator GANs doesn’t force output to correspond to input slides credit: Isola / Zhu

  29. mode collapse! slides credit: Isola / Zhu

  30. Cycle-Consistent Adversarial Networks ⋯ ⋯ [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

  31. Cycle-Consistent Adversarial Networks ⋯ ⋯ [Mark Twain, 1903] [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

  32. Cycle Consistency Loss x G(x) F(G x ) D Y (G x ) Reconstruction error F G x − x 1 [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

  33. Cycle Consistency Loss Large cycle loss Small cycle loss x G(x) F(G x ) D Y (G x ) Reconstruction error F G x − x 1 [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

  34. Cycle Consistency Loss x G(x) F(G x ) 𝑧 F(y) G(F x ) D Y (G x ) D G (F x ) Reconstruction Reconstruction error error F G x − x 1 G F y − 𝑧 1 [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

  35. Cycle GAN - Overvie view Prof. Leal-Taixé and Prof. Niessner 35 https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.]

  36. Cycle GAN: Objective ive Domain X Domain Y Cycle consistency Full Loss: Prof. Leal-Taixé and Prof. Niessner 36 https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.]

  37. Monet’s paintin ings → photos https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.] slides credit: Isola / Zhu

  38. https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.] slides credit: Isola / Zhu

  39. https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.] slides credit: Isola / Zhu

  40. Adminis istrative ive Prof. Leal-Taixé and Prof. Niessner 40

  41. Adminis istrative ive • Deadline for final projects – Wed Feb 6 th th , 11:59pm – Submission via moodle – Submission must contain • Code (results must be replicable) • 2-3 pages of final report (at most 1 page of text, rest results; i.e., images and tables) • Use CVPR templates: http://cvpr2019.thecvf.com/submission/main_conference/au thor_guidelines Prof. Leal-Taixé and Prof. Niessner 41

  42. Adminis istrative ive • Poster presentation – Friday Feb 8 th th , 1pm-3pm – Location: • Magistrale (preliminary – will update if it changes) • In the area next to the back entrance (parking lot direction) – Poster stands will be provided – You need to print posters yourself (poster@in.tum.de) – Hang posters 15 mins before presentation session starts Prof. Leal-Taixé and Prof. Niessner 42

  43. Guest Speakers • Oriol Vinyals: – https://ai.google/research/people/OriolVinyals – Time: Ja January 31 st st , 6pm – 8pm – Location: HS-1 (CS building – the big one) Prof. Leal-Taixé and Prof. Niessner 43

  44. Next Lectures • Next Lecture -> Jan 21 st • Keep working on the projects! Prof. Leal-Taixé and Prof. Niessner 44

  45. Conditio ditional al Generati ative Adversa sarial al Networks works (cGANs Ns) conti tinued! nued! Prof. Leal-Taixé and Prof. Niessner 45

Recommend


More recommend