Self-Supervised Model Training and Selection for Disentangling GANs - PowerPoint PPT Presentation

Paper ID: 4410 Self-Supervised Model Training and Selection for Disentangling GANs Previous title: InfoGAN-CR: Disentangling Generative Adversarial Networks with Contrastive Regularizers Zinan Lin Kiran Thekumparampil Giulia Fanti Sewoong Oh CMU UIUC CMU UW

Generative Adversarial Networks (GANs) 𝑙 factors 𝑒 input noise 𝑨 ! • Hair color 𝑨 " • Rotation Generator … • Background 𝑨 # • Bangs • How do 𝑨 ! s control the factors? Vanilla GANs Disentangled GANs 𝑨 ! Factor ! 𝑑 ! Factor ! 𝑨 " Factor " 𝑑 " Factor " … … … … 𝑨 # Factor $ 𝑑 $ Factor $ 𝑨 % s ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 2

Examples of Disentanglement 𝑑 ! 𝑑 " 𝑑 & 𝑑 ' 𝑑 ( Changing only: Latent (CelebA dataset) codes 𝑑 ! 𝑑 " hair color rotation lighting background bangs Generator … 𝑑 # 𝑨 % s The remaining (dSprites dataset) noise dimensions y-position shape scale rotation x-position * CelebA example is generated by InfoGAN-CR. Dsprites example is synthetic for illustration. ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 3

Challenges in Learning Disentangled GANs 1. How to train disentangled GANs? GANs are good at generating high fidelity images, • but are reported to be bad at disentanglement (e.g. compared with VAEs) 2. How to do unsupervised model selection? In practice, we don’t have ground-truth factor labels • for selecting the best disentangled models. ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 4

Our Solution: Self-Supervision Assessing the Pairs of images disentanglement with 1. How to train disentangled GANs? quality self-supervised labels InfoGAN-CR • Contrastive Generator Regularizer … (CR) 2. How to do unsupervised model selection? ModelCentrality • Self-supervised All trained A pair of distance metric models: trained models: Pick the most central model ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 5

Benchmark dSprites Dataset Model FactorVAE Metric ↑ VAE 0.63 𝛾 -TCVAE 0.62 HFVAE 0.63 VAE Supervised 𝛾 -VAE 0.63 hyper-parameter CHyVAE 0.77 selection FactorVAE 0.82 InfoGAN 0.59 InfoGAN (modified) 0.83 GAN IB-GAN 0.80 InfoGAN-CR 0.90 ±𝟏. 𝟏𝟐 Unsupervised InfoGAN-CR model 0.92 ±𝟏. 𝟏𝟏 model selection selected with ModelCentrality Code & Paper & More results: https://github.com/fjxmlzn/InfoGAN-CR ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 6

Details 1. InfoGAN-CR (model training) 2. ModelCentrality (model selection) ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 7

Intuition of Contrastive Regularizer (CR) $ ) to generate a pair of images $ ,…, 𝑑 # $ ,…, 𝑑 ! • Use two latent codes ( 𝑑 " ,…, 𝑑 ! ,…, 𝑑 # ), ( 𝑑 " Equal Same i-th latent code Same shape 𝑗 = 1 Generator Same x-position 𝑗 = 2 ( 𝐻 ) 𝑗 = 3 Same y-position ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 8

Intuition of Contrastive Regularizer (CR) $ ) to generate a pair of images $ ,…, 𝑑 # $ ,…, 𝑑 ! • Use two latent codes ( 𝑑 " ,…, 𝑑 ! ,…, 𝑑 # ), ( 𝑑 " Equal Same i-th latent code 1 𝑗 = 1 Contrastive Generator 2 𝑗 = 2 Regularizer ( 𝐻 ) (CR) 𝑗 = 3 3 Classification task! ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 9

̂ InfoGAN-CR GAN Discriminator ℝ % ∈ ℝ # # Input Noise " InfoGAN Encoder ! ∈ ℝ ! ! $ ∈ ℝ " %′ ∈ ℝ # Latent Factors $ ∈ ℝ " %′′ ∈ ℝ # CR 𝐼 ℝ " InfoGAN-CR loss: %,',( max min 𝑀 *+, 𝐻, 𝐸 − 𝜇𝐽 𝐻, 𝑅 − 𝛽𝑀 - (𝐻, 𝐼) ) Mutual info GAN’s classification accuracy of CR loss adversarial loss InfoGAN [1] [1] InfoGAN. Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (NeurIPS 2016) ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 10

CR Achieves a Further Gain that Cannot be Gotten with InfoGAN Alone Disentanglement Split training #training iterations (dSprites dataset) ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 11

Details 1. InfoGAN-CR (model training) 2. ModelCentrality (model selection) ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 12

Intuition of ModelCentrality • Well-disentangled models are close to the (unknown) ground-truth model Model space Ground-truth model (unknown) Bad Good model model • Idea: pick the model at the “center” ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 13

ModelCentrality • How to find the model at the “center” ? Two models are close ó Their latent codes control similar factors (1) Treat as the Model 𝑗 ground-truth factors Any supervised Asymmetric similarity 𝑑 !" disentanglement metric Similarity 𝑒 !" ? Similarity 𝑒 !" = 𝑒 "! = (𝑑 !" + 𝑑 "! )/2 (e.g. FactorVAE metric [1]) Model 𝑘 (The higher, the more similar) ! (2) ./! ∑ 01% 𝑒 %0 ModelCentrality score 𝑁 % = High similarity Low similarity (3) Selection the model with the highest 𝑁 % [1] Disentangling by Factorising. Kim, H., & Mnih, A. (ICML 2018) ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 14

Results of ModelCentrality • Works well for GANs. Take InfoGAN-CR for example: Method FactorVAE Metric ↑ Supervised hyper-parameter selection 0.90 Unsupervised model selection with UDR Lasso [1] 0.86 Unsupervised model selection with UDR Spearman [1] 0.84 Unsupervised model selection with ModelCentrality 0.92 • Also works well for VAEs! Take FactorVAE for example: Method FactorVAE Metric ↑ Supervised hyper-parameter selection 0.83 Unsupervised model selection with UDR Lasso [1] 0.81 Unsupervised model selection with UDR Spearman [1] 0.79 Unsupervised model selection with ModelCentrality 0.84 [1] Unsupervised Model Selection for Variational Disentangled Representation Learning. Duan, S., Matthey, L., Saraiva, A., Watters, N., Burgess, C. P., Lerchner, A., & Higgins, I. (ICLR 2020). ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 15

Conclusion • InfoGAN-CR • New disentangled GANs, achieving state-of-the-art results • ModelCentrality • Unsupervised model selection approach for disentangled GANs and VAEs • InfoGAN-CR + ModelCentrality • Unsupervised disentangled generative model training and selection package, achieving state-of-the-art results Code & Paper & More results: https://github.com/fjxmlzn/InfoGAN-CR Including: More theoretical & empirical analysis of InfoGAN & InfoGAN-CR • Analysis of total correlation for GANs and CR for VAEs • New challenging disentanglement datasets • ID 4410: Self-supervised model training and selection for disentangling GAN Code & Paper: https://github.com/fjxmlzn/InfoGAN-CR 16