Lecture 1: Feedforward Princeton University COS 495 Instructor: - PowerPoint PPT Presentation

Deep Learning Basics Lecture 1: Feedforward Princeton University COS 495 Instructor: Yingyu Liang

Motivation I: representation learning

Machine learning 1-2-3 • Collect data and extract features • Build model: choose hypothesis class 𝓘 and loss function 𝑚 • Optimization: minimize the empirical loss

Features 𝑦 Color Histogram Extract build 𝑧 = 𝑥 𝑈 𝜚 𝑦 features hypothesis Red Green Blue

Features: part of the model Nonlinear model build 𝑧 = 𝑥 𝑈 𝜚 𝑦 hypothesis Linear model

Example: Polynomial kernel SVM 𝑦 1 𝑧 = sign(𝑥 𝑈 𝜚(𝑦) + 𝑐) 𝑦 2 Fixed 𝜚 𝑦

Motivation: representation learning • Why don’t we also learn 𝜚 𝑦 ? Learn 𝜚 𝑦 Learn 𝑥 𝜚 𝑦 𝑧 = 𝑥 𝑈 𝜚 𝑦 𝑦

Feedforward networks • View each dimension of 𝜚 𝑦 as something to be learned … 𝑧 = 𝑥 𝑈 𝜚 𝑦 … 𝑦 𝜚 𝑦

Feedforward networks 𝑈 𝑦 don’t work: need some nonlinearity • Linear functions 𝜚 𝑗 𝑦 = 𝜄 𝑗 … 𝑧 = 𝑥 𝑈 𝜚 𝑦 … 𝑦 𝜚 𝑦

Feedforward networks 𝑈 𝑦) where 𝑠(⋅) is some nonlinear function • Typically, set 𝜚 𝑗 𝑦 = 𝑠(𝜄 𝑗 … 𝑧 = 𝑥 𝑈 𝜚 𝑦 … 𝑦 𝜚 𝑦

Feedforward deep networks • What if we go deeper? … … … … 𝑧 … … ℎ 𝑀 ℎ 1 𝑦 ℎ 2

Figure from Deep learning , by Goodfellow, Bengio, Courville. Dark boxes are things to be learned.

Motivation II: neurons

Motivation: neurons Figure from Wikipedia

Motivation: abstract neuron model • Neuron activated when the correlation between the input and a pattern 𝜄 𝑦 1 exceeds some threshold 𝑐 𝑦 2 • 𝑧 = threshold(𝜄 𝑈 𝑦 − 𝑐) or 𝑧 = 𝑠(𝜄 𝑈 𝑦 − 𝑐) 𝑧 • 𝑠(⋅) called activation function 𝑦 𝑒

Motivation: artificial neural networks

Motivation: artificial neural networks • Put into layers: feedforward deep networks … … … … 𝑧 … … ℎ 𝑀 ℎ 1 𝑦 ℎ 2

Components in Feedforward networks

Components • Representations: • Input • Hidden variables • Layers/weights: • Hidden layers • Output layer

Components First layer Output layer … … … … 𝑧 … … ℎ 𝑀 Hidden variables ℎ 1 ℎ 2 Input 𝑦

Input • Represented as a vector • Sometimes require some Expand preprocessing, e.g., • Subtract mean • Normalize to [-1,1]

Output layers Output layer • Regression: 𝑧 = 𝑥 𝑈 ℎ + 𝑐 • Linear units: no nonlinearity 𝑧 ℎ

Output layers Output layer • Multi-dimensional regression: 𝑧 = 𝑋 𝑈 ℎ + 𝑐 • Linear units: no nonlinearity 𝑧 ℎ

Output layers Output layer • Binary classification: 𝑧 = 𝜏(𝑥 𝑈 ℎ + 𝑐) • Corresponds to using logistic regression on ℎ 𝑧 ℎ

Output layers Output layer • Multi-class classification: • 𝑧 = softmax 𝑨 where 𝑨 = 𝑋 𝑈 ℎ + 𝑐 • Corresponds to using multi-class logistic regression on ℎ 𝑨 𝑧 ℎ

Hidden layers • Neuron take weighted linear combination of the previous … layer • So can think of outputting one value for the next layer … ℎ 𝑗 ℎ 𝑗+1

Hidden layers • 𝑧 = 𝑠(𝑥 𝑈 𝑦 + 𝑐) • Typical activation function 𝑠 𝑠(⋅) • Threshold t 𝑨 = 𝕁[𝑨 ≥ 0] 𝑦 𝑧 • Sigmoid 𝜏 𝑨 = 1/ 1 + exp(−𝑨) • Tanh tanh 𝑨 = 2𝜏 2𝑨 − 1

Hidden layers • Problem: saturation 𝑠(⋅) 𝑦 𝑧 Too small gradient Figure borrowed from Pattern Recognition and Machine Learning , Bishop

Hidden layers • Activation function ReLU (rectified linear unit) • ReLU 𝑨 = max{𝑨, 0} Figure from Deep learning , by Goodfellow, Bengio, Courville.

Hidden layers • Activation function ReLU (rectified linear unit) • ReLU 𝑨 = max{𝑨, 0} Gradient 1 Gradient 0

Hidden layers • Generalizations of ReLU gReLU 𝑨 = max 𝑨, 0 + 𝛽 min{𝑨, 0} • Leaky- ReLU 𝑨 = max{𝑨, 0} + 0.01 min{𝑨, 0} • Parametric- ReLU 𝑨 : 𝛽 learnable gReLU 𝑨 𝑨