Photo Editing With Generative Adversarial Networks (GANs) GTC, May - PowerPoint PPT Presentation

Photo Editing With Generative Adversarial Networks (GANs) GTC, May 2017. Greg Heinrich.

PANDARUS: Alas, I think he shall be come approached and the day When little srain would be attain'd into being never fed, And who is but a chain and subjects of his death, I should not sleep. G AN : WHAT IS A GENERATIVE MODEL? In Machine Learning A generative model learns to generate samples that have the same characteristics as the samples in the dataset. Learn from Shakespeare novels: Produce: http://karpathy.github.io/2015/05/21/rnn- PANDARUS: effectiveness/ Alas, I think he shall be come approached and the day When little srain would be attain'd into being never fed, And who is but a chain and subjects of his death, 2 2 I should not sleep.

BASIC REMINDER: BACKPROP 𝜖𝐹 Calculating 𝑚 iteratively 𝜖𝑥 𝑗𝑘 Chain rule 𝜖𝐹 Calculation of 𝑚 : Output of each neuron 𝑘 of layer 𝑚 : 𝜖𝑨 𝑘 𝑚 = 𝜒 𝑨 𝑚 = 𝜒 𝑥 𝑗𝑘 𝑚 ℎ 𝑗 𝑚−1 + 𝑐 𝑚 ℎ 𝑘 𝑘 𝑘 𝑚 𝑚+1 𝑚+1 𝜖ℎ 𝑘 𝜖𝐹 𝜖𝐹 𝜖𝑨 𝑙 𝜖𝐹 𝜖𝑨 𝑙 𝑚 = 𝑚 = 𝑗 𝑙 𝑙 𝑚+1 𝑚+1 𝑚 𝑚 𝜖𝑨 𝑘 𝜖𝑨 𝑙 𝜖𝑨 𝑘 𝜖𝑨 𝑙 𝜖ℎ 𝑘 𝜖𝑨 𝑘 Gradient of E with respect to each weight: = 𝜖𝐹 𝑚+1 𝜒′ 𝑨 𝑚 𝑚+1 𝑥 𝑘𝑙 𝑘 𝑚 𝜖𝑨 𝑙 𝜖𝑨 𝑘 𝜖𝐹 𝜖𝐹 𝜖𝐹 𝑚−1 𝑚 = 𝑚 = 𝑚 ℎ 𝑗 𝑙 𝑚 𝜖𝑥 𝑗𝑘 𝜖𝑨 𝑘 𝜖𝑥 𝑗𝑘 𝜖𝑨 𝑘 𝑚 𝜖𝐹 𝑚+1 = 𝜒′ 𝑨 𝑚+1 𝑥 Calculated 𝑘 𝑘𝑙 𝜖𝑨 𝑙 during 𝑙 forward prop Multivariate chain rule 𝑚 𝜖𝑨 𝑘 Chain 𝑚−1 𝑚 only depends on ℎ 𝑗 rule 3 3 𝜖𝑥 𝑗𝑘

G A N : PLAYING THE ADVERSARIAL GAME Learning on a corpus of images Let’s play a game opposing two agents: - The Generator, a little imp in the computer who paints images. - The Discriminator: you are collectively responsible for playing the Discriminator. The game master (me) randomly picks images from either the corpus or the Generator and shows them to the Discriminator. The goal of the Discriminator is to identify the source of the images: real (from the corpus) or fake (painted by the little imp). The goal of the Generator is to fool the Discriminator. 4 4

PLAYING THE ADVERSARIAL GAME Is this a veelhoek* from our corpus? * veelhoek is the articulation Note: you don’t have to know what a veelhoek is, of a ubiquitous item in the you will learn through language of a tiny country in examples! Europe that is well known for the inferior quality of its cheese. Yes, this red square is a veelhoek! 5 5

PLAYING THE ADVERSARIAL GAME Is this a veelhoek from our corpus? No, those squiggly lines aren’t right! 6 6

PLAYING THE ADVERSARIAL GAME Is this a veelhoek from our corpus? Yes, even though it’s blue and tiny! 7 7

PLAYING THE ADVERSARIAL GAME Is this a veelhoek from our corpus? No, those rounded corners are a giveaway! 8 8

PLAYING THE ADVERSARIAL GAME Is this a veelhoek from our corpus? No, but it’s a very good fake! 9 9

PLAYING THE ADVERSARIAL GAME Is this a veelhoek from our corpus? No, it’s the same fake as before! 10 10

PLAYING THE ADVERSARIAL GAME Is this a veelhoek from our corpus? No, but it’s a very creative fake! 11 11

THE LATENT REPRESENTATION From features to images A veelhoek is characterized by three features: - colour, - size, - number of faces This set of features is known as the “ LATENT REPRESENTATION” . We can generate many real-looking veelhoeks by randomly picking reasonable values of each feature: 12 12

THE LATENT REPRESENTATION Arithmetic in latent space We can perform operations in latent space, have them reflected in feature space: 𝑚𝑏𝑠𝑕𝑓 𝑡𝑛𝑏𝑚𝑚 𝑛𝑓𝑒𝑗𝑣𝑛 1 𝑕𝑠𝑓𝑓𝑜 𝑧𝑓𝑚𝑚𝑝𝑥 𝑠𝑓𝑒 + = 2 3 𝑔𝑏𝑑𝑓𝑡 5 𝑔𝑏𝑑𝑓𝑡 4 𝑔𝑏𝑑𝑓𝑡 Equivalently: 𝑕𝑠𝑓𝑓𝑜 𝑠𝑓𝑒 1 + 𝑡𝑛𝑏𝑚𝑚 𝑚𝑏𝑠𝑕𝑓 + = 2 5 𝑔𝑏𝑑𝑓𝑡 3 𝑔𝑏𝑑𝑓𝑡 13 13

THE GAN SET-UP Connecting the Discriminator to the Generator and the Dataset Random Latent vector 14 14

GA N: NETWORK TOPOLOGY Radford (2015). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv:1511.06434 Generator Discriminator 15 15

TRAINING A GAN ON CELEBRITY FACES* Generating new faces by picking random values of the latent vector * CelebFaces dataset 16 16

ANALOGIES man is to woman as king is to queen Reproduction of the famous “king + woman - man = queen” analogy on faces: Blond Blue Looking Pointy Man Smile Hair Eyes Left Nose - - - + + + Top Right + + + - Bottom Left + Subtract Top + - - - - Left + - - - + + Bottom Right 17 17 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

MAPPING IMAGES TO LATENT VECTORS Transfer learning : from Discriminator to Encoder 18 18

IMAGE RECONSTRUCTIONS Visualizing 𝐻 𝐹 𝑗𝑛𝑏𝑕𝑓 19 19

ATTRIBUTES Calculating attribute vectors The encoder 𝐹 may be used to calculate the latent vector for each attribute. For each 𝑏𝑢𝑢𝑠 in 𝑏𝑢𝑢𝑠𝑗𝑐𝑣𝑢𝑓𝑡 : + − 𝐽 𝑏𝑢𝑢𝑠 = 𝑗𝑛 𝑏𝑢𝑢𝑠 and 𝐽 𝑏𝑢𝑢𝑠 = 𝑗𝑛 𝑏𝑢𝑢𝑠 are sets of images w/wo the attribute 1 1 𝑨 𝑏𝑢𝑢𝑠 = 𝐹(𝑗𝑛) − 𝐹(𝑗𝑛) − + 𝑗𝑛 ∈ 𝐽 𝑏𝑢𝑢𝑠 + 𝑗𝑛 ∈ 𝐽 𝑏𝑢𝑢𝑠 − 𝐽 𝑏𝑢𝑢𝑠 𝐽 𝑏𝑢𝑢𝑠 It is then straightforward to add or remove attributes from an image: F rom left to right: original image (OI); OI + “young” attribute; OI - “blond hair” + “black hair”; OI - “smile”; OI + “male” + “bald”. 20 20

PLAYING WITH ATTRIBUTES 21 21

EXTRACTING ATTRIBUTES …from portraits of illustrious people 22 22

DEGENERATOR Getting the essence of your dataset After convergence, stop updating the discriminator: 23 23

DATASET VISUALIZATION Projecting latent vectors on a sphere 24 24

THANK YOU Questions? 25 25