for Data-Efficient GAN Training Shengyu Zhao 1,2 Ji Lin 1 Jun-Yan Zhu - PowerPoint PPT Presentation

Differentiable Augmentation for Data-Efficient GAN Training Shengyu Zhao 1,2 Ji Lin 1 Jun-Yan Zhu 3,4 Song Han 1 Zhijian Liu 1 1 MIT 2 IIIS, Tsinghua University 3 Adobe Research 4 CMU NeurIPS 2020

Data Is Expensive Computation Algorithm Computation Algorithm Big Data FFHQ dataset: 70,000 selective post-processed human faces ImageNet dataset: millions of images from diverse categories Months or even years to collect the data, along with prohibitive annotation costs.

GANs Heavily Deteriorate Given Limited Data 100 images Obama Cat (Simard et al.) 160 images Dog (Simard et al.) 389 images Generated samples of StyleGAN2 (Karras et al.) using only hundreds of images

GANs Heavily Deteriorate Given Limited Data StyleGAN2 (baseline) + DiffAugment (ours) 40 36.0 35 30 23.1 25 FID ↓ 20 15 11.1 10 5 0 100% training data 20% training data 10% training data CIFAR-10

Discriminator Overfitting

#1 Approach: Augment reals only Generated images Artifacts from Color jittering Artifacts from Translation Artifacts from Cutout (DeVries et al.) Augment reals only: the same artifacts appear on the generated images.

#2 Approach: Augment reals & fakes for 𝑬 only Augment 𝑬 only: the unbalanced optimization cripples training.

#3 Approach: Differentiable Augmentation (Ours) fakes reals Color Color Translation Translation Cutout Cutout Our approach (DiffAugment): Augment reals + fakes for both 𝐸 and 𝐻

Our Results StyleGAN2 (baseline) StyleGAN2 (baseline) + DiffAugment (ours) + DiffAugment (ours) 40 40 36.0 36.0 35 35 30 30 23.1 23.1 25 25 FID ↓ FID ↓ 20 20 14.5 14.5 15 15 12.2 12.2 11.1 11.1 9.9 9.9 10 10 5 5 0 0 100% training data 20% training data 10% training data 100% training data 20% training data 10% training data CIFAR-10

Low-Shot Generation 100 images Obama Cat (Simard et al.) 160 images Dog (Simard et al.) 389 images

Fine-Tuning vs. Ours Scale/Shift (Noguchi et al.) MineGAN (Wang et al.) TransferGAN (Wang et al.) FreezeD (Mo et al.) Ours 60 100000 50 10000 # Training Images 40 No pre-training 1000 FID↓ 30 100 20 10 10 1 0 Data Performance 100-shot Obama

100-Shot Interpolation Our code, datasets, and models are publicly available at https://github.com/mit-han-lab/data-efficient-gans.