Autoencoders Lecture slides for Chapter 14 of Deep Learning - PowerPoint PPT Presentation

Autoencoders Lecture slides for Chapter 14 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-09-30

Structure of an Autoencoder Hidden layer (code) h f g x r Input Reconstruction Figure 14.1 (Goodfellow 2016)

Stochastic Autoencoders h p encoder ( h | x ) p decoder ( x | h ) x r Figure 14.2 (Goodfellow 2016)

Avoiding Trivial Identity • Undercomplete autoencoders • h has lower dimension than x • f or g has low capacity (e.g., linear g ) • Must discard some information in h • Overcomplete autoencoders • h has higher dimension than x • Must be regularized (Goodfellow 2016)

Regularized Autoencoders • Sparse autoencoders • Denoising autoencoders • Autoencoders with dropout on the hidden layer • Contractive autoencoders (Goodfellow 2016)

Sparse Autoencoders • Limit capacity of autoencoder by adding a term to the cost function penalizing the code for being larger • Special case of variational autoencoder • Probabilistic model • Laplace prior corresponds to L1 sparsity penalty • Dirac variational posterior (Goodfellow 2016)

Denoising Autoencoder h g f ˜ ˜ L x x C : corruption process (introduce noise) C (˜ x | x ) ss L = − log p decoder ( x | h = f ( ˜ x )) , xample x , obtained through a given corr x Figure 14.3 (Goodfellow 2016)

Denoising Autoencoders Learn a Manifold ˜ x g � f x ˜ x C (˜ x | x ) x Figure 14.4 (Goodfellow 2016)

Score Matching • Score: r x log p ( x ) . (14.15) • Fit a density model by matching score of model to score of data • Some denoising autoencoders are equivalent to score matching applied to some density models (Goodfellow 2016)

Vector Field Learned by a Denoising Autoencoder Figure 14.5 (Goodfellow 2016)

Tangent Hyperplane of a Manifold Figure 14.6 (Goodfellow 2016)

Learning a Collection of 0-D Manifolds by Resisting Perturbation 1 . 0 Identity 0 . 8 Optimal reconstruction 0 . 6 r ( x ) 0 . 4 0 . 2 0 . 0 x 0 x 1 x 2 x Figure 14.7 (Goodfellow 2016)

Non-Parametric Manifold Learning with Nearest-Neighbor Graphs Figure 14.8 Figure 14.8: Non-parametric manifold learning procedures build a nearest neighbor grap (Goodfellow 2016)

Tiling a Manifold with Local Coordinate Systems Figure 14.9 (Goodfellow 2016)

Contractive Autoencoders 2 � � ∂ f ( x ) � � Ω ( h ) = λ (14.18) . � � ∂ x � � F Input Tangent vectors point Local PCA (no sharing across regions) Contractive autoencoder Figure 14.10 (Goodfellow 2016)