Condit itio ional Generativ ive Adversaria ial Networks - PowerPoint PPT Presentation

Condit itio ional Generativ ive Adversaria ial Networks (cGANs) Prof. Leal-Taixé and Prof. Niessner 1

Conditional GANs (cGANs) • Gain control of output • Modeling (e.g., sketch-based modeling, etc.) – Add semantic meaning to latent space manifold • Domain transfer – Labels on A - > transfer to B, train network on ‘B’, test on B – More later Prof. Leal-Taixé and Prof. Niessner 2

GAN Manifold Sampled Data -> G(z) Train Data Prof. Leal-Taixé and Prof. Niessner 3 [Radford et al. 15]

GAN Manifold [Bojanowski et al 17] Prof. Leal-Taixé and Prof. Niessner 4

GAN Manifold Prof. Leal-Taixé and Prof. Niessner 5

GAN Manifold Linear interpolation in z space: 𝐻(𝑨 0 + 𝑢 ⋅ 𝑨 1 − 𝑨 0 ) 𝐻(𝑨 0 ) 𝐻(𝑨 1 ) Prof. Leal-Taixé and Prof. Niessner 6 [Radford et al. 15]

Conditional GANs (cGANs) Prof. Leal-Taixé and Prof. Niessner 7 Slide credit Zhu

iG iGANs: : Overv rview original photo different degree of image manipulation Project Edit Transfer Editing UI projection on manifold transition between the original and edited projection Slide credit Zhu / [Zhu et al. 16]

iG iGANs: : Overv rview original photo different degree of image manipulation Project Edit Transfer Editing UI projection on manifold transition between the original and edited projection Slide credit Zhu / [Zhu et al. 16]

iG iGANs: Pro roje jectin ing an Im Image onto the Manif ifold Input: real image 𝑦 𝑆 Output: latent vector z Optimization 0.196 0.238 0.332 Reconstruction loss 𝑀 Generative model 𝐻(𝑨) Slide credit Zhu / [Zhu et al. 16]

iG iGANs: Pro roje jectin ing an Im Image onto the Manif ifold Input: real image 𝑦 𝑆 Output: latent vector z Optimization 0.196 0.238 0.332 Inverting Network z = 𝑄 𝑦 Auto-encoder 0.242 0.336 0.218 with a fixed decoder G Slide credit Zhu / [Zhu et al. 16]

iG iGANs: Pro roje jectin ing an Im Image onto the Manif ifold Input: real image 𝑦 𝑆 Output: latent vector z Optimization 0.196 0.238 0.332 Inverting Network z = 𝑄 𝑦 0.242 0.336 0.218 Hybrid Method Use the network as initialization for the optimization problem Slide credit Zhu / [Zhu et al. 16] 0.167 0.268 0.153

iG iGANs: : Overv rview original photo different degree of image manipulation Project Edit Transfer Editing UI projection on manifold transition between the original and edited projection Slide credit Zhu / [Zhu et al. 16]

iG iGANs: Manip ipulatin ing the Latent Vector constraint violation loss 𝑀 𝑕 user guidance image Objective: Guidance 𝑤 𝑕 𝐻(𝑨 ) 𝑨 0 Slide credit Zhu / [Zhu et al. 16]

iG iGANs: : Overv rview original photo different degree of image manipulation Project Edit Transfer Editing UI projection on manifold transition between the original and edited projection Slide credit Zhu / [Zhu et al. 16]

iG iGANs: : Edit it Tra ransfer Motion (u, v)+ Color ( 𝑩 𝟒×𝟓 ): estimate per-pixel geometric and color variation 𝐻(𝑨 0 ) Linear Interpolation in 𝑨 space 𝐻(𝑨 1 ) Input Slide credit Zhu / [Zhu et al. 16]

iG iGANs: : Edit it Tra ransfer Motion (u, v)+ Color ( 𝑩 𝟒×𝟓 ): estimate per-pixel geometric and color variation 𝐻(𝑨 0 ) Linear Interpolation in 𝑨 space 𝐻(𝑨 1 ) Input Slide credit Zhu / [Zhu et al. 16]

iG iGANs: : Edit it Tra ransfer Motion (u, v)+ Color ( 𝑩 𝟒×𝟓 ): estimate per-pixel geometric and color variation 𝐻(𝑨 0 ) Linear Interpolation in 𝑨 space 𝐻(𝑨 1 ) Input Result

cGANs: : In Interactiv ive GANs Interactive GANs: projection to GAN embedding Prof. Leal-Taixé and Prof. Niessner 19 https://github.com/junyanz/iGAN [Zhu et al. 16.]

cGANs: : In Interactiv ive GANs Prof. Leal-Taixé and Prof. Niessner 20 https://github.com/junyanz/iGAN [Zhu et al. 16.]

cGANs: : In Interactiv ive GANs Prof. Leal-Taixé and Prof. Niessner 21 https://github.com/junyanz/iGAN [Zhu et al. 16.]

Mapping in in Latent Space is is Diff iffic icult! • Semantics are missing • In most cases, no labels available • Ideally, need some unsupervised disentangled rep. Prof. Leal-Taixé and Prof. Niessner 22 InfoGAN [Chen et al. 16]

Pair ired vs Unpair ired Settin ing Prof. Leal-Taixé and Prof. Niessner 23

pix ix2pix: : Im Image-to to-Image Tra ranslatio ion slides credit: Isola / Zhu

z G(z) G D real or fake? Generator Discriminator min 𝐻 max 𝔽 𝑨,𝑦 log 𝐸(𝐻 𝑨 ) + log(1 − 𝐸 𝑦 ) 𝐸 [Goodfellow et al. 2014] slides credit: Isola / Zhu

x G(x) G D real or fake? Generator Discriminator min 𝐻 max 𝔽 𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 ) 𝐸 slides credit: Isola / Zhu

x G(x) Real! G D Generator Discriminator min 𝐻 max 𝔽 𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 ) 𝐸 slides credit: Isola / Zhu

x G(x) Real too! G D Generator Discriminator min 𝐻 max 𝔽 𝑦,𝑧 log 𝐸(𝐻 𝑦 ) + log(1 − 𝐸 𝑧 ) 𝐸 slides credit: Isola / Zhu

x G(x) G D real or fake pair ? min 𝐻 max 𝔽 𝑦,𝑧 log 𝐸(𝑦, 𝐻 𝑦 ) + log(1 − 𝐸 𝑦, 𝑧 ) 𝐸 fake pair real pair match joint distribution p G x , y ∼ p(x, y) slides credit: Isola / Zhu

Edges → Images Input Output Input Output Input Output Edges from [Xie & Tu, 2015] slides credit: Isola / Zhu

Pix ix2Pix ix: : Pair ired Settin ing • Great when we have ‘free’ training data • Often called self-supervised • Think about these settings  31

Sketches → Images Input Output Input Output Input Output Trained on Edges → Images Data from [Eitz, Hays, Alexa, 2012] slides credit: Isola / Zhu

#edges2cats [Christopher Hesse] @gods_tail @matthematician Vitaly Vidmirov @vvid Ivy Tasi @ivymyt https://affinelayer.com/pixsrv/ slides credit: Isola / Zhu

Input Output Groundtruth Data from [maps.google.com] slides credit: Isola / Zhu

BW → Color Input Output Input Output Input Output Data from [Russakovsky et al. 2015] slides credit: Isola / Zhu

Paired Label ↔ photo: per-pixel labeling Horse ↔ zebra: how to get zebras? … - Expensive to collect pairs. - Impossible in many scenarios. slides credit: Isola / Zhu

Paired Unpaired … … … slides credit: Isola / Zhu

x G(x) G D Generator No input-output pairs! slides credit: Isola / Zhu

x G(x) Real! G D Generator Discriminator slides credit: Isola / Zhu

x G(x) Real too! G D Generator Discriminator GANs doesn’t force output to correspond to input slides credit: Isola / Zhu

mode collapse! slides credit: Isola / Zhu

Cycle-Consistent Adversarial Networks ⋯ ⋯ [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

Cycle-Consistent Adversarial Networks ⋯ ⋯ [Mark Twain, 1903] [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

Cycle Consistency Loss x G(x) F(G x ) D Y (G x ) Reconstruction error F G x − x 1 [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

Cycle Consistency Loss Large cycle loss Small cycle loss x G(x) F(G x ) D Y (G x ) Reconstruction error F G x − x 1 [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

Cycle Consistency Loss x G(x) F(G x ) 𝑧 F(y) G(F x ) D Y (G x ) D G (F x ) Reconstruction Reconstruction error error F G x − x 1 G F y − 𝑧 1 [Zhu*, Park*, Isola, and Efros, ICCV 2017] slides credit: Isola / Zhu

Cycle le GAN - Overv rvie iew Prof. Leal-Taixé and Prof. Niessner 48 https://junyanz.github.io/CycleGAN/ [Zhu et al. 17.]

Monet’s paintin ings → photos slides credit: Isola / Zhu

slides credit: Isola / Zhu

slides credit: Isola / Zhu

Next Lectures • Next Monday 24 th , – Xmas s GANs – No Lecture re Jan 14 th -> No lecture, but office hours • Next Lecture -> Jan 14 th • We are still working on feedback for presentations – will send • around asap… Keep working on the projects! • Prof. Leal-Taixé and Prof. Niessner 52

See you next year r  Prof. Leal-Taixé and Prof. Niessner 53