Computational Photography Si Lu Spring 2018 - PowerPoint PPT Presentation

Computational Photography Si Lu Spring 2018 http://web.cecs.pdx.edu/~lusi/CS510/CS510_Computati onal_Photography.htm 05/29/2018

Last Time o 3D Video Stabilization 2

Introduction of Neural Networks 3

Content • Introduction • Artificial vs. Biological Neural Nets • What are Neural Networks? • Neural Network-Basic • What is in the black box • Gradient Descent Method • Speed up training • Activation Function • Morden Neural Networks 4

Content • Introduction • Artificial vs. Biological Neural Nets • What are Neural Networks? • Neural Network-Basic • What is in the black box • Gradient Descent Method • Different Optimizors • Activation Function • Morden Neural Networks 5

Artificial vs. Biological Neural Nets http://7xiur2.com1.z0.glb.clouddn.com/0137.png

https://cdn-images-1.medium.com/max/1600/1*xR4m0oOKz_jRgQU4Oge53g.jpeg 7

https://cdn-images-1.medium.com/max/1600/1*xR4m0oOKz_jRgQU4Oge53g.jpeg 8

https://cdn-images-1.medium.com/max/1600/1*xR4m0oOKz_jRgQU4Oge53g.jpeg https://www.robotics.org/ 9

https://cdn-images-1.medium.com/max/1600/1*xR4m0oOKz_jRgQU4Oge53g.jpeg 10 https://www.geekwire.com

Artificial = Biological ? Neuron 90 Billion 11

Artificial = Biological ?

Artificial = Biological ?

Artificial = Biological ?

Artificial = Biological ?

Artificial = Biological ?

Artificial = Biological ?

Artificial = Biological ?

Artificial = Biological ?

Artificial = Biological ?

Artificial = Biological ?

Artificial = Biological ?

Artificial = Biological ? =

Content • Introduction • Artificial vs. Biological Neural Nets • What are Neural Networks ? • Neural Network-Basic • What is in the black box • Gradient Descent Method • Speed up training • Activation Function • Morden Neural Networks 24

What are Neural Networks? https://i.imgur.com/Vbsk7t5.jpg

What are Neural Networks? Neural network, or artificial neural network, is a computing system inspired by the biological neural networks that constitute animal brains wikipedia

What are Neural Networks? wikipedia

What are Neural Networks? Hidden Hidden Input Output layer 1 layer 2 wikipedia

What are Neural Networks?

What are Neural Networks?

What are Neural Networks?

What are Neural Networks? Trainging dataset Ground truth labels

What are Neural Networks?

What are Neural Networks?

What are Neural Networks?

What are Neural Networks? Error

What are Neural Networks? Error

What are Neural Networks? Repeat millions of times

What are Neural Networks?

What are Neural Networks?

Content • Introduction • Artificial vs. Biological Neural Nets • What are Neural Networks ? • Neural Network-Basic • What is in the black box ? • Gradient Descent Method • Speed up training • Activation Function • Morden Neural Networks • Overfitting-Batch Normalization, Dropout • From LeNet to ResNet 41

Neural Network Basics https://futureoflife.org

Content • Introduction • Artificial vs. Biological Neural Nets • What are Neural Networks ? • Neural Network-Basic • What is in the black box ? • Gradient Descent Method • Speed up training • Activation Function • Morden Neural Networks 43

What is in the black box ?

What is in the black box ?

What is in the black box ?

What is in the black box ?

What is in the black box ?

What is in the black box ? Neural networks are trained to extract higher and higher levels of abstract features to better represent the dataset via back-propogation

What is in the black box ? x 1 w 1 w 2 Σ AF x 2 y w 3 x 3 input parameters output Basic Unit: Neuron

What is in the black box ? x 1 w 1 w 2 Σ AF x 2 y w 3 x 3 input parameters output y = w 1 x 1 +w 2 x 2 +w 3 x 3

What is in the black box ? x 1 w 1 w 2 Σ AF x 2 y w 3 x 3 input parameters output y = Wx T

Content • Introduction • Artificial vs. Biological Neural Nets • What are Neural Networks ? • Neural Network-Basic • What is in the black box ? • Gradient Descent Method • Speed up training • Activation Function • Morden Neural Networks 53

Gradient Descent Method x 1 w 1 w 2 Σ AF x 2 y w 3 x 3 input parameters output y = Wx T

Gradient Descent Method y = Wx T Q1. What is the optimal W? Q2. How to obtain the optimal W?

Gradient Descent Method W opt = argmin W |y-Wx T | 2 Q1. What is the optimal W? Q2. How to obtain the optimal W?

Gradient Descent Method W opt = argmin W |y-Wx T | 2 loss Q1. What is the optimal W? Q2. How to obtain the optimal W?

Gradient Descent Method Optimization: Gradient Descent Q1. What is the optimal W? Q2. How to obtain the optimal W?

Gradient Descent Method Simplification Original loss function: f=|y-Wx T | 2

Gradient Descent Method Simplification Original loss function: f=|y-Wx T | 2 Simplify 1: single w/x/y: f=(y-wx) 2

Gradient Descent Method Simplification Original loss function: f=|y-Wx T | 2 Simplify 1: single w/x/y: f=(y-wx) 2 Simplify 2: y=0, x=1: f=w 2

Gradient Descent Method f w

Gradient Descent Method f w

Gradient Descent Method f w

Gradient Descent Method

Gradient Descent Method

Gradient Descent Method f w

Content • Introduction • Artificial vs. Biological Neural Nets • What are Neural Networks ? • Neural Network-Basic • What is in the black box ? • Gradient Descent Method • Speed up training • Activation Function • Morden Neural Networks 68

Speed up Training: dataset Large numbe of x, y, w y = Wx T

Speed up Training: dataset

Speed up Training: dataset Batch

Speed up Training: dataset Batch

Speed up Training: dataset Batch Stochastic Gradient Descent (SGD)

Speed up Training: optimizer Original: W += - LR * dx

Speed up Training: optimizer Original: W += - LR * dx Momentum: m += b 1 *m- LR * dx W += m Adding “dowhill”- inertia

Speed up Training: optimizer Original: W += - LR * dx AdaGrad: v += dx^2 W += -LR * dx/sqrt(v) Adding “breaking shoes”- resistance

Speed up Training: optimizer Momentum + AdaGrad RMSProp Adam (Popular)

Content • Introduction • Artificial vs. Biological Neural Nets • What are Neural Networks ? • Neural Network-Basic • What is in the black box ? • Gradient Descent Method • Speed up training • Activation Function • Morden Neural Networks 78

Activation Function x 1 w 1 w 2 Σ AF x 2 y w 3 x 3 input parameters output y = Wx T

https://medium.com/@shrutijadon10104776

Activation Function

Activation Function Activate different neurons for different input

Activation Function Activate different neurons for different input

Activation Function Team Salary Championship Essentially: adding non-linearty

Activation Function Team Salary Team Salary Championship Championship Essentially: adding non-linearty

Next Time • Introduction • Artificial vs. Biological Neural Nets • What are Neural Networks ? • Neural Network-Basic • What is in the black box ? • Gradient Descent Method • Speed up training • Activation Function • Morden Neural Networks 86