Artificial Neural Networks (ANNs) What is an artificial neural - PDF document

2/13/17 100% Accuracy in Automatic Face Recognition Jenkins & Burton (2008) Big challenge for face recognition: coping with variation in facial appearance due to changing illumination, pose, expression, age, hair, etc. Store average face for each person?? Artificial Neural Networks (ANNs) What is an artificial neural network? What can an artificial neural network learn to do ? early success: ALVINN, handwritten zip codes, NETtalk A (very!) simple neural network Training a neural network with backpropagation 1

2/13/17 What is an artificial neural network? Network of simple neuron-like computing elements… … that can learn to associate inputs with desired outputs Artificial Neural Networks (ANNs) What is an artificial neural network? What can an artificial neural network learn to do? early success: ALVINN, handwritten zip codes, NETtalk A (very!) simple neural network Training a neural network with backpropagation 2

2/13/17 ALVINN learned to control steering actions Pomerleau (1991) • ALVINN learned to steer by observing a human driver (960 inputs) • Multiple networks for different roads (e.g. dirt road, two-lane road, highway (up to 70mph!)) Learning to recognize handwritten zip codes LeCun et al. (1989) System could recognize image samples provided by the US postal service, with high accuracy 3

2/13/17 NETtalk learned phonemes from text Sejnowski & Rosenberg (1989) features of phonemes https://www.youtube.com/watch?v=gakJlr3GecE Artificial Neural Networks (ANNs) What is an artificial neural network? What can an artificial neural network learn to do? early success: ALVINN, handwritten zip codes, NETtalk A (very!) simple neural network Training a neural network with backpropagation 4

2/13/17 Computing in an artificial neural network How does each unit integrate its inputs to produce an output? w 1 , w 2 : weights t : threshold I 1 w 1 H 1 or 0 w 2 I 2 How can such a network w 1 • I 1 + w 2 • I 2 > t perform a useful function? if true: H = 1 • if false: H = 0 • • A (very!) simple neural network H 1 O 1 I 1 Jack 1 Acquaintances 1 1 or 0 >0.5 >1.5 1 I 2 Jean 1 -1 O 2 H 2 I 3 Pam 1 >0.5 >-1.5 1 or 0 -1 1 Siblings I 4 Paul thresholds weights network inputs: 1 or 0, valid input combinations have exactly two 1’s network outputs: 1 or 0 5

2/13/17 A (very!) simple neural network H 1 O 1 I 1 1 Jack 1 Acquaintances 1 ?? >0.5 >1.5 1 I 2 1 Jean 1 -1 O 2 H 2 I 3 Pam 0 1 >-1.5 >0.5 ?? -1 1 Siblings I 4 0 Paul thresholds weights network inputs: 1 or 0, valid input combinations have exactly two 1’s network outputs: 1 or 0 A (very!) simple neural network H 1 O 1 I 1 0 Jack 1 Acquaintances 1 ?? >0.5 >1.5 1 I 2 0 Jean 1 -1 O 2 H 2 I 3 1 Pam 1 >-1.5 >0.5 ?? -1 1 Siblings I 4 1 Paul thresholds weights network inputs: 1 or 0, valid input combinations have exactly two 1’s network outputs: 1 or 0 6

2/13/17 A (very!) simple neural network H 1 O 1 I 1 1 Jack 1 Acquaintances 1 >0.5 ?? >1.5 1 I 2 0 Jean 1 -1 O 2 H 2 I 3 0 Pam 1 >-1.5 >0.5 ?? -1 1 Siblings I 4 1 Paul thresholds weights network inputs: 1 or 0, valid input combinations have exactly two 1’s network outputs: 1 or 0 Add “bias” units to simplify thresholds H 1 O 1 I 1 1 1 > 0 1 or 0 >1.5 1 I 2 1 -0.5 Do the same for +1 the output units -0.5 -1 O 2 H 2 I 3 1 > 0 >-1.5 1 or 0 -1 1 I 4 weights 7

2/13/17 Computing in a ”typical” neural network +1 sigmoid w 0 I 1 w 1 1 or 0 H w 2 I 2 w 0 • 1 + w 1 • I 1 + w 2 • I 2 > 0 • • activation • How does each unit integrate its inputs to produce an output? sum of weighted inputs à sigmoid function à output between 0 and 1 Artificial Neural Networks (ANNs) What is an artificial neural network? What can an artificial neural network learn to do? early success: ALVINN, handwritten zip codes, NETtalk A (very!) simple neural network Training a neural network with backpropagation 8

2/13/17 Learning in an artificial neural network network weights can be learned from training examples feedforward processing What’s in a set of training examples? Backpropagation method: compute output for each input training • sample, using current network • compute errors between actual and desired outputs work backwards from output layer to • input to determine how each weight backpropagation method can be adjusted to reduce errors to learn network weights update network and repeat • 9