[PPT] - Neural Networks Continue Instructor: Bhiksha Raj Slides by Najim PowerPoint Presentation

SLIDE 1

Machine Learning for Signal Processing

Neural Networks Continue

Instructor: Bhiksha Raj Slides by Najim Dehak 1 Dec 2016

1

SLIDE 2

So what are neural networks??

What are these boxes?

N.Net Voice signal Transcription N.Net Image Text caption N.Net Game State Next move

18797/11755 2

SLIDE 3

So what are neural networks??

It began with this..
Humans are very good at the tasks we just saw
Can we model the human brain/ human intelligence?

– An old question – dating back to Plato and Aristotle..

18797/11755 3

SLIDE 4

MLP - Recap

MLPs are Boolean machines

– They represent Boolean functions over linear boundaries – They can represent arbitrary boundaries

Perceptrons are correlation filters

– They detect patterns in the input

MLPs are Boolean formulae over patterns detected by perceptron

– Higher-level perceptrons may also be viewed as feature detectors

MLPs are universal approximators

– Can model any function to arbitrary precision

Extra: MLP in classification

– The network will fire if the combination of the detected basic features matches an “acceptable” pattern for a desired class of signal

E.g. Appropriate combinations of (Nose, Eyes, Eyebrows, Cheek, Chin)  Face

4

SLIDE 5

MLP - Recap

MLPs are Boolean machines

– They represent arbitrary Boolean functions over arbitrary linear boundaries

Perceptrons are pattern detectors

– MLPs are Boolean formulae over these patterns

MLPs are universal approximators

– Can model any function to arbitrary precision

MLPs are very hard to train

– Training data are generally many orders of magnitude too few – Even with optimal architectures, we could get rubbish – Depth helps greatly! – Can learn functions that regular classifiers cannot

5

SLIDE 6

What is a deep network?

SLIDE 7

Deep Structures

In any directed network of computational

elements with input source nodes and output sink nodes, “depth” is the length of the longest path from a source to a sink

Left: Depth = 2. Right: Depth = 3

SLIDE 8

Deep Structures

Layered deep structure
“Deep”  Depth > 2

SLIDE 9

MLP as a continuous-valued regression

MLPs can actually compose arbitrary functions to arbitrary precision

– Not just classification/Boolean functions

1D example

– Left: A net with a pair of units can create a pulse of any width at any location – Right: A network of N such pairs approximates the function with N scaled pulses

9

x

1 T1 T2 1 T1 T2 1

1

T1 T2 x

f(x) x

+

SLIDE 10

MLP features

The lowest layers of a network detect significant features in the

signal

The signal could be reconstructed using these features

– Will retain all the significant components of the signal

10

DIGIT OR NOT?

SLIDE 11

Making it explicit: an autoencoder

A neural network can be trained to predict the input itself
This is an autoencoder
An encoder learns to detect all the most significant patterns in the signals
A decoder recomposes the signal from the patterns

11

𝒀 𝒁 𝒀 𝑿 𝑿𝑼

SLIDE 12

Deep Autoencoder

ENCODER DECODER

SLIDE 13

What does the AE learn

In the absence of an intermediate non-linearity
This is just PCA

13

𝒀 𝒀 𝒁 𝑿 𝑿𝑼

𝐙 = 𝐗𝐘 𝐘 = 𝐗𝑈𝐙 𝐹 = 𝐘 − 𝐗𝑈𝐗𝐘 2 Find W to minimize Avg[E]

SLIDE 14

The AE

With non-linearity

– “Non linear” PCA – Deeper networks can capture more complicated manifolds

14

ENCODER DECODER

SLIDE 15

The Decoder:

The decoder represents a source-specific generative

dictionary

Exciting it will produce typical signals from the source!

15

DECODER

SLIDE 16

The AE

ENCODER DECODER

Cut the AE

16

SLIDE 17

DECODER

The Decoder:

The decoder represents a source-specific generative

dictionary

Exciting it will produce typical signals from the source!

17

Sax dictionary

SLIDE 18

The Decoder:

The decoder represents a source-specific generative

dictionary

Exciting it will produce typical signals from the source!

18

DECODER

Clarinet dictionary

SLIDE 19

NN for speech enhancement

19

SLIDE 20

Story so far

MLPs are universal classifiers

– They can model any decision boundary

Neural networks are universal approximators

– They can model any regression

The decoder of MLP autoencoders represent

a non-linear constructive dictionary!

20

SLIDE 21

The need for shift invariance

In many problems the location of a pattern is not important

– Only the presence of the pattern

Conventional MLPs are sensitive to the location of the

pattern

– Moving it by one component results in an entirely different input that the MLP wont recognize

Requirement: Network must be shift invariant

=

SLIDE 22

History

Yann LeCun Hubel and Wiesel: 1959 (biological model), Fukushima: 1980 (computational model), Altas: 1988, Lecunn: 1989 (Backprop in convnets) Kunihiko Fukushima

Convolutional Neural Networks

SLIDE 23

Convolutional Neural Networks

A special kind of multi-layer neural networks.
Implicitly extract relevant features.
A feed-forward network that can extract topological

properties from an image.

CNNs are also trained with a version of back-propagation

algorithm.

SLIDE 24

All different weights Convolution layer has much smaller number of parameters by local connection and weight sharing All different weights Shared weights

Connectivity & weight sharing

SLIDE 25

25

Example: 200x200 image 40K hidden units

~2B parameters!!!

Spatial correlation is local
Waste of resources + we have not enough

utput layer

consider a second, green filter

Convolution Layer

SLIDE 54

32 32 3 Convolution Layer activation maps 6 28 28

For example, if we had 6 5x5 filters, we’ll get 6 separate activation maps: We stack these up to get a “new image” of size 28x28x6!

Convolution Layer

SLIDE 55

Preview: ConvNet is a sequence of Convolution Layers, interspersed with activation functions 32 32 3 28 28 6 CONV, ReLU e.g. 6 5x5x3 filters

CNN

SLIDE 56

Preview: ConvNet is a sequence of Convolutional Layers, interspersed with activation functions 32 32 3 CONV, ReLU e.g. 6 5x5x3 filters 28 28 6 CONV, ReLU e.g. 10 5x5x6 filters CONV, ReLU

….

10 24 24

CNN

SLIDE 57

57

Let us assume filter is an “eye” detector. Q.: how can we make the detection robust to the exact location of the eye?

Ranzato

Pooling Layer

SLIDE 58

58

By “pooling” (e.g., taking max) filter responses at different locations we gain robustness to the exact spatial location

f features.

Ranzato

Pooling Layer

SLIDE 59

makes the representations smaller and more manageable
perates over each activation map independently:

Pooling Layer

SLIDE 60

1 1 2 4 5 6 7 8 3 2 1 1 2 3 4 Single depth slice x y

max pool with 2x2 filters and stride 2

6 8 3 4

Max Pooling

SLIDE 61

61

Convol. Pooling One stage (zoom)

courtesy of

K. Kavukcuoglu

Ranzato

ConvNets: Typical Stage

SLIDE 62

Digit classification

SLIDE 63

ImageNet

1.2 million high-resolution images from ImageNet LSVRC-2010 contest
1000 different classes (sofmax layer)
NN configuration
NN contains 60 million parameters and 650,000 neurons,
5 convolutional layers, some of which are followed by max-pooling layers
3 fully-connected layers

Krizhevsky, A., Sutskever, I. and Hinton, G. E. “ImageNet Classification with Deep Convolutional Neural Networks” NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada

SLIDE 64

ImageNet

Figure 3: 96 convolutional

kernels of size 11×11×3 learned by the first convolutional layer on the 224×224×3 input images. The top 48 kernels were learned

n GPU 1 while the bottom 48

kernels were learned on GPU

2. See Section 6.1 for details.

Krizhevsky, A., Sutskever, I. and Hinton, G. E. “ImageNet Classification with Deep Convolutional Neural Networks” NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada

SLIDE 65

ImageNet

Krizhevsky, A., Sutskever, I. and Hinton, G. E. “ImageNet Classification with Deep Convolutional Neural Networks” NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada Eight ILSVRC-2010 test images and the five labels considered most probable by our model. The correct label is written under each image, and the probability assigned to the correct label is also shown with a red bar (if it happens to be in the top 5). Five ILSVRC-2010 test images in the first

column. The remaining columns show the six

training images that produce feature vectors in the last hidden layer with the smallest Euclidean distance from the feature vector for the test image.

SLIDE 66

CNN for Automatic Speech Recognition

Convolution over frequencies
Convolution over time

SLIDE 67

Neural network with specialized connectivity

structure

Feed-forward:
Convolve input
Non-linearity (rectified linear)
Pooling (local max)
Supervised training
Train convolutional filters by back-propagating error
Convolution over time
Adding memory to classical MLP network
Recurrent neural network

Feature maps Pooling Non-linearity Convolution (Learned) Input image

CNN-Recap

SLIDE 68

Recurrent networks introduce (RNN) cycles and a notion of time.

They are designed to process sequences of data 𝑦1, … , 𝑦𝑜 and

can produce sequences of outputs 𝑧1, … , 𝑧𝑛.

Recurrent Neural Networks (RNNs)

𝑦𝑢 𝑧𝑢 ℎ𝑢 ℎ𝑢−1

One-step delay

Recurrent Neural Network

SLIDE 69

Elman Nets (1990) – Simple Recurrent Neural Networks

Elman nets are feed forward networks with partial

recurrence

Unlike feed forward nets, Elman nets have a memory or

sense of time

Can also be viewed as a “Markovian” NN

SLIDE 70

(Vanilla) Recurrent Neural Network

The state consists of a single “hidden” vector h:

𝑦𝑢 𝑧𝑢 ℎ𝑢 ℎ𝑢−1

One-step delay

Simple Recurrent Neural Network

SLIDE 71

RNNs can be unrolled across multiple time steps. This produces a DAG which supports backpropagation. But its size depends on the input sequence length.

Unrolling RNNs

𝑦𝑢 𝑧𝑢 ℎ𝑢 ℎ𝑢−1

One-step delay

𝑦0 𝑧0 ℎ0 𝑦1 𝑧1 ℎ1 𝑦2 𝑧2 ℎ2

Recurrent Neural Network

SLIDE 72

Recurrent networks have one more or more

feedback loops

There are many tasks that require learning a

temporal sequence of events

– Speech, video, Text, Market

These problems can be broken into 3 distinct

types of tasks

1. Sequence Recognition: Produce a particular output pattern when a

specific input sequence is seen. Applications: speech recognition

2. Sequence Reproduction: Generate the rest of a sequence when the

network sees only part of the sequence. Applications: Time series prediction (stock market, sun spots, etc)

3. Temporal Association: Produce a particular output sequence in response

to a specific input sequence. Applications: speech generation

Learning time sequences

SLIDE 73

Often layers are stacked vertically (deep RNNs):

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features

Same parameters at this level Same parameters at this level



Recurrent Neural Network

SLIDE 74

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Activations

Recurrent Neural Network

Backprop still works: (it called Backpropagation Through Time)

SLIDE 75

Backprop still works:

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Activations

Recurrent Neural Network

SLIDE 76

Backprop still works:

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Activations

Recurrent Neural Network

SLIDE 77

Backprop still works:

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Activations

Recurrent Neural Network

SLIDE 78

Backprop still works:

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Activations

Recurrent Neural Network

SLIDE 79

Backprop still works:

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Activations

Recurrent Neural Network

SLIDE 80

Backprop still works:

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Activations

Recurrent Neural Network

SLIDE 81

Backprop still works:

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Gradients

Recurrent Neural Network

SLIDE 82

Backprop still works:

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Gradients

Recurrent Neural Network

SLIDE 83

Backprop still works:

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Gradients

Recurrent Neural Network

SLIDE 84

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Gradients

Recurrent Neural Network

Backprop still works:

SLIDE 85

Backprop still works:

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Gradients

Recurrent Neural Network

SLIDE 86

Backprop still works:

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 𝑧12 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Gradients

Recurrent Neural Network

SLIDE 87

Backprop still works:

RNN structure

𝑦0 𝑧00 ℎ00 𝑦1 𝑧01 ℎ01 𝑦2 𝑧02 ℎ02 𝑦00 𝑦01 𝑦02 𝑧10 𝑧11 ℎ10 ℎ11 ℎ12 Time Abstraction

Higher

level features Gradients

Recurrent Neural Network

𝑧12

SLIDE 88

The memory problem with RNN

RNN models signal context
If very long context is used -> RNNs become unable to learn the context

information

SLIDE 89

Standard RNNs to LSTM

Standard LSTM

SLIDE 90

LSTM illustrated: input and forming new memory

Input gate

New memory LSTM cell takes the following input

the input 𝑦𝑢
past memory
utput ℎ𝑢−1
past memory

𝐷𝑢−1 (all vectors) Forget gate Cell state

SLIDE 91

Forming the output of the cell by using output

gate

LSTM illustrated: Output

Overall picture:

SLIDE 92

LSTM Equations

92

𝑗 = 𝜏 𝑦𝑢𝑉𝑗 + 𝑡𝑢−1𝑋𝑗
𝑔 = 𝜏 𝑦𝑢𝑉𝑔 + 𝑡𝑢−1𝑋𝑔
𝑝 = 𝜏 𝑦𝑢𝑉𝑝 + 𝑡𝑢−1𝑋𝑝
𝑕 = tanh 𝑦𝑢𝑉𝑕 + 𝑡𝑢−1𝑋𝑕
𝑑𝑢 = 𝑑𝑢−1 ∘ 𝑔 + 𝑕 ∘ 𝑗
𝑡𝑢 = tanh 𝑑𝑢 ∘ 𝑝
𝑧 = 𝑡𝑝𝑔𝑢𝑛𝑏𝑦 𝑊𝑡𝑢
𝒋: input gate, how much of the new

information will be let through the memory cell.

𝒈: forget gate, responsible for information

should be thrown away from memory cell.

𝒑: output gate, how much of the information

will be passed to expose to the next time step.

𝒉: self-recurrent which is equal to standard

RNN

𝒅𝒖: internal memory of the memory cell
𝒕𝒖: hidden state
𝐳: final output

LSTM Memory Cell

SLIDE 93

LSTM output synchronization

SLIDE 94

(NLP) Applications of RNNs

Section overview

– Language Model – Sentiment analysis / text classification – Machine translation and conversation modeling – Sentence skip-thought vectors

SLIDE 95

RNN for

SLIDE 96

Sentiment analysis / text classification

A quick example, to see the idea.
Given text collections and their labels. Predict labels for unseen texts.

SLIDE 97

Translating Videos to Natural Language Using Deep Recurrent Neural Networks

Translating Videos to Natural Language Using Deep Recurrent Neural Networks Subhashini Venugopalan, Huijun Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, Kate Saenko North American Chapter of the Association for Computational Linguistics, Denver, Colorado, June 2015.

SLIDE 98

SLIDE 99

Composing music with RNN

http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/

SLIDE 100

CNN-LSTM-DNN for speech recognition

Ensembles of RNN/LSTM, DNN, & Conv

Nets (CNN) give huge gains (state of the art):

T. Sainath, O. Vinyals, A. Senior, H. Sak.

“Convolutional, Long Short-Term Memory, Fully Connected Deep Neural Networks,” ICASSP 2015.

SLIDE 101

The Impact of deep learning in speech technologies

Cortana

SLIDE 102

Conclusions

MLPs are Boolean machines

– They represent Boolean functions over linear boundaries – They can represent arbitrary boundaries

Perceptrons are correlation filters

– They detect patterns in the input

MLPs are Boolean formulae over patterns detected by perceptron

– Higher-level perceptrons may also be viewed as feature detectors

MLPs are universal approximators

– Can model any function to arbitrary precision – Non linear PCA

Convolute NN can handle shift invariance

– CNN

Special NN can model sequential data

– RNN, LSTM