Meta-Learning Unsupervised Update Rules Paper by Luke Metz, Niru - PowerPoint PPT Presentation

Meta-Learning Unsupervised Update Rules Paper by Luke Metz, Niru Maheswaranathan, Brian Cheung, Jascha Sohl-Dickstein

Outline Motivation Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Outer Loop Experimental Results Critiques

Motivation Motivation Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Unsupervised learning enables representation Outer Loop Results learning on mountains on unlabeled data for Critiques downstream tasks

Motivation Motivation Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Unsupervised learning enables representation Outer Loop Results learning on mountains of unlabeled data for Critiques downstream tasks. Unsupervised Learning Rules VAE: Severe overfitting to training space. ● GANs: Great for images, weak on discrete data (ex. text). ● Both: Learning rule not unsupervised (ex. surrogate loss). ●

Motivation Motivation Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Unsupervised learning enables representation Outer Loop Results learning on mountains of unlabeled data for Critiques downstream tasks Unsupervised Learning Rules VAE: Severe overfitting to training space. ● GANs: Great for images, weak on discrete data (ex. text). ● Both: Learning rule not unsupervised (ex. surrogate loss). ● Question: Can we meta-learn an unsupervised learning rule?

Motivation Semi-Supervised Few-Shot Classification Problem Breakdown Method Overview Meta-Learning Setup Labeled train Unlabeled train Inner Loop Outer Loop Results y 1 y 2 y 3 y 4 x 1 x 2 x 3 x 4 x 5 x 1 x 2 x 3 x 4 Critiques Apply unsupervised rule to Apply encoder to get tune encoder compact vector Fit Model

Motivation Semi-Supervised Few-Shot Classification Problem Breakdown Method Overview Meta-Learning Setup Labeled train Unlabeled train Inner Loop Outer Loop Results y 1 y 2 y 3 y 4 x 1 x 2 x 3 x 4 x 5 x 1 x 2 x 3 x 4 Critiques Apply unsupervised rule to Apply encoder to get tune encoder compact vector Can we meta-learn this unsupervised learning rule? Fit Model

Learning the Learning Rule Motivation Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Outer Loop Results Backpropagation: Critiques Unsupervised Update:

Method Overview Motivation Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Outer loop Outer Loop Results Optimize meta-objective: ● Critiques Inner loop Learn encoder using unsupervised update rule. ●

Motivation Meta-Learning Setup Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Outer Loop Results Critiques

Motivation Meta-Learning Setup Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Inner loop applies an Outer Loop unsupervised learning Results Critiques alg. on unlabeled data Outer loop evaluates unsupervised learning alg. using labeled data

Motivation Meta-Learning Setup Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Inner loop applies an Outer Loop unsupervised learning Results Critiques alg. on unlabeled data Outer loop evaluates unsupervised learning alg. using labeled data

Motivation Inner Loop Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Question: Given a base model, g(x; ɸ ), which encodes Outer Loop Results inputs into compact vectors, how do we learn its Critiques parameters ɸ to give useful features?

Motivation Inner Loop Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Question: Given a base model, g(x; ɸ ), which encodes Outer Loop Results inputs into compact vectors, how do we learn its Critiques parameters ɸ to give useful features? Idea: What if we use another neural network to generate a neuron-specific error signal? Then we can learn its parameters θ (the meta-parameters) to produce useful error signals

Motivation Inner Loop: Forward Pass Problem Breakdown Method Overview Meta-Learning Setup Inner Loop 1) Take an input Outer Loop Results Critiques 2) Generate intermediate activations 3) Produce a feature representation

Motivation Inner Loop: Generate Error Signal Problem Breakdown Method Overview Meta-Learning Setup Inner Loop 1) Input each Outer Loop Results layer’s activation Critiques through an MLP 2) Output error vector

Motivation Inner Loop: Backward Pass Problem Breakdown Method Overview Meta-Learning Setup Inner Loop 1) Initialize Outer Loop Results top-level error Critiques with output of MLP 2) Backprop the error 3) Linearly combine output from MLP with backpropagated error

Motivation Inner Loop: Update 𝝔 Problem Breakdown Method Overview Meta-Learning Setup 𝝔 consists of all base Inner Loop Outer Loop model parameters Results Critiques W i , V i , and b i Updates like Δ W i , Δ V i are linear* functions of local error quantities h i-1 and h i *There are also nonlinear normalizations within this function

Motivation Inner Loop: Key Points Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Error generating network replicates the mechanics ● Outer Loop Results of backprop for unsupervised learning Critiques An iterative updates tune 𝝔 for some higher-level ● objective Outer loop sets objective by modifying the error ● generating function

Motivation Inner Loop: Key Points Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Error generating network replicates the mechanics ● Outer Loop Results of backprop for unsupervised learning Critiques An iterative updates tune 𝝔 for some higher-level ● objective Outer loop sets objective by modifying the error ● generating function

Motivation Inner Loop: Key Points Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Error generating network replicates the mechanics ● Outer Loop Results of backprop for unsupervised learning Critiques An iterative updates tune 𝝔 for some higher-level ● objective Outer loop sets objective by modifying the error ● generating function

Motivation Inner Loop: Key Points Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Error generating network replicates the mechanics ● Outer Loop Results of backprop for unsupervised learning Critiques An iterative updates tune 𝝔 for some higher-level ● objective Outer loop sets objective by modifying the error ● generating function

Motivation Outer Loop Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Outer Loop Results Critiques

Motivation Outer Loop: Compute MetaObjective Problem Breakdown Method Overview Meta-Learning Setup Unlabeled support Labeled support Labeled query Inner Loop Outer Loop Results x * x * x 1 x 2 x 3 x 4 x 5 x 1 x 2 x 3 x 4 1 2 Critiques Apply Unsupervised Rule θ Apply encoder to tune Encoder Fit Linear Evaluate MS Error Model Model y 1 y 2 y 3 y 4 y * y * 1 2

Motivation Outer Loop: Compute MetaObjective Problem Breakdown Method Overview Meta-Learning Setup Unlabeled support Labeled support Labeled query Inner Loop Outer Loop Results x * x * x 1 x 2 x 3 x 4 x 5 x 1 x 2 x 3 x 4 1 2 Critiques Apply Unsupervised Rule θ Apply encoder to tune Encoder Fit Linear Evaluate MS Error Model Model x 1 x 2 x 3 x 4 x * x * 1 2

Motivation Outer Loop: Compute MetaObjective Problem Breakdown Method Overview Meta-Learning Setup Unlabeled support Labeled support Labeled query Inner Loop Outer Loop Results x * x * x 1 x 2 x 3 x 4 x 5 x 1 x 2 x 3 x 4 1 2 Critiques Apply Unsupervised Rule θ Apply encoder to tune Encoder Backprop all the way back to θ Fit Linear Evaluate MS Error Model Model x 1 x 2 x 3 x 4 x * x * 1 2

Motivation Outer Loop: Compute MetaObjective Problem Breakdown Method Overview Meta-Learning Setup Unlabeled support Labeled support Labeled query Inner Loop Outer Loop Results x * x * x 1 x 2 x 3 x 4 x 5 x 1 x 2 x 3 x 4 1 2 Critiques Apply Unsupervised Rule θ Apply encoder to tune Encoder Backprop all the way back to θ Truncated backprop Fit Linear Evaluate MS Error Model Model x 1 x 2 x 3 x 4 x * x * 1 2

Motivation Results Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Training Data: CIFAR10 & Imagenet. Outer Loop Results Critiques Generalization over datasets. ● Generalization over domains ● Generalization over network architectures ●

Motivation Results: Generalization over datasets Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Outer Loop Results Critiques What’s going on? - Evaluation of unsupervised learning rule on different datasets - Comparison to other methods.

Motivation Results: Generalization over Domains Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Outer Loop Results Critiques What’s going on? Evaluation of unsupervised learning rule on 2-way text classification. 30h vs 200h of meta-training.

Motivation Results: Generalization over Networks Problem Breakdown Method Overview Meta-Learning Setup Inner Loop Outer Loop Results Critiques What’s going on? Evaluation of unsupervised learning rule on different network architectures.