How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) - PowerPoint PPT Presentation

Need for Machine . . . Neural Networks . . . How to Speed Up . . . Neural Networks: A . . . How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . . . Resulting Algorithm Faster by Taking Into How to Pre-Train a . . . How to Retain . . . Account Known Constraints Home Page Chitta Baral 1 , Martine Ceberio 2 , and Vladik Kreinovich 2 Title Page ◭◭ ◮◮ 1 Department of Computer Science, Arizona State University Tempe, AZ 85287-5406, USA, chitta@asu.edu ◭ ◮ 2 Department of Computer Science, University of Texas at El Paso El Paso, TX 79968, USA, mceberio@utep.edu, vladik@utep.edu Page 1 of 17 Go Back Full Screen Close Quit

Need for Machine . . . Neural Networks . . . 1. Need for Machine Learning How to Speed Up . . . • In many practical situations: Neural Networks: A . . . Biological Neuron: A . . . – we know that the quantities y 1 , . . . , y L depend on Artificial Neural . . . the quantities x 1 , . . . , x n , but Resulting Algorithm – we do not know the exact formula for this depen- How to Pre-Train a . . . dence. How to Retain . . . • To get this formula, we: Home Page – measure the values of all these quantities in differ- Title Page ent situations m = 1 , . . . , M , and then ◭◭ ◮◮ – use the corresponding measurement results x ( m ) i ◭ ◮ and y ( m ) to find the corresponding dependence. ℓ Page 2 of 17 • Algorithms that “learn” the dependence from the mea- Go Back surement results are known as machine learning alg. Full Screen Close Quit

Need for Machine . . . Neural Networks . . . 2. Neural Networks (NN): Successes and Limita- How to Speed Up . . . tions Neural Networks: A . . . • One of the most widely used machine learning tech- Biological Neuron: A . . . niques is the technique of neural networks (NN). Artificial Neural . . . Resulting Algorithm • It is is based on a (simplified) simulation of how actual How to Pre-Train a . . . neurons works in the human brain. How to Retain . . . • Multi-layer (“deep”) neural networks are, at present, Home Page the most efficient machine learning techniques. Title Page • One of the main limitations of neural networks is that ◭◭ ◮◮ their learning very slow. ◭ ◮ • The current neural networks always start “from Page 3 of 17 scratch”, from zero knowledge. Go Back • This inability to take prior knowledge into account drastically slows down the learning process. Full Screen Close Quit

Need for Machine . . . Neural Networks . . . 3. How to Speed Up Artificial Neural Networks: How to Speed Up . . . A Natural Idea. Neural Networks: A . . . • A natural idea is to enable neural networks to take Biological Neuron: A . . . prior knowledge into account. In other words: Artificial Neural . . . Resulting Algorithm – instead of learning all the data “from scratch”, How to Pre-Train a . . . – we should first learn the constraints. How to Retain . . . • Then: Home Page – when it is time to use the data, Title Page – we should be able to use these constraints to ◭◭ ◮◮ “guide” the neural network in the right direction. ◭ ◮ • In this paper, we show how to implement this idea. Page 4 of 17 Go Back Full Screen Close Quit

Need for Machine . . . Neural Networks . . . 4. Neural Networks: A Brief Reminder How to Speed Up . . . • In a biological neural network, a signal is represented Neural Networks: A . . . by a sequence of spikes. Biological Neuron: A . . . Artificial Neural . . . • All these spikes are largely the same, what is different Resulting Algorithm is how frequently the spikes come. How to Pre-Train a . . . • Several sensor cells generate such sequences: e.g., How to Retain . . . Home Page – there are cells that translate the optical signal into spikes, Title Page – there are cells that translate the acoustic signal into ◭◭ ◮◮ spikes. ◭ ◮ • For all such cells, the more intense the original physical Page 5 of 17 signal, the more spikes per unit time it generates. Go Back • Thus, the frequency of the spikes can serve as a mea- Full Screen sure of the strength of the original signal. Close Quit

Need for Machine . . . Neural Networks . . . 5. Biological Neuron: A Brief Description How to Speed Up . . . • A biological neuron has several inputs and one output. Neural Networks: A . . . Biological Neuron: A . . . • Usually, spikes from different inputs simply get to- Artificial Neural . . . gether – probably after some filtering. Resulting Algorithm • Filtering means that we suppress a certain proportion How to Pre-Train a . . . of spikes. How to Retain . . . • If we start with an input signal x i , then, after such a Home Page filtering, we get a decreased signal w i · x i . Title Page • Once all the inputs signals are combined, we have the ◭◭ ◮◮ n resulting signal � w i · x i . ◭ ◮ i =1 Page 6 of 17 • A biological neuron usually has some excitation level w 0 . Go Back • If the overall input signal is below w 0 , there is practi- Full Screen cally no output. Close Quit

Need for Machine . . . Neural Networks . . . 6. Biological Neuron (cont-d) How to Speed Up . . . • The intensity of the output signal thus depends on the Neural Networks: A . . . n def Biological Neuron: A . . . � difference d = w i · x i − w 0 . i =1 Artificial Neural . . . • Some neurons are linear, their output is proportional Resulting Algorithm to this difference. How to Pre-Train a . . . How to Retain . . . • Other neurons are non-linear, they output is equal to Home Page s 0 ( d ) for some non-linear function s 0 ( z ). Title Page • Empirically, it was found that the corresponding non- ◭◭ ◮◮ linear transformation is s 0 ( z ) = 1 / (1 + exp( − z )). ◭ ◮ • It should be mentioned that this is a simplified descrip- tion of a biological neuron: Page 7 of 17 – the actual neuron is a complex dynamical system, Go Back – its output depends not only on the current inputs, Full Screen but also on the previous input values. Close Quit

Need for Machine . . . Neural Networks . . . 7. Artificial Neural Networks and How They How to Speed Up . . . Learn Neural Networks: A . . . • For each output y ℓ , we train a separate neural network. Biological Neuron: A . . . Artificial Neural . . . • In the simplest (and most widely used) arrangement: Resulting Algorithm – the neurons from the first layer produce the signals How to Pre-Train a . . . � n � How to Retain . . . � y ℓ,k = s 0 w ℓ,ki · x i − w ℓ,k 0 , 1 ≤ k ≤ K ℓ , Home Page i =1 Title Page – these signals go into a linear neuron in the second ◭◭ ◮◮ layer, which combines them into an output ◭ ◮ K � y ℓ = W ℓ,k · y k − W ℓ, 0 . Page 8 of 17 k =1 Go Back • This is called forward propagation . Full Screen Close Quit

Need for Machine . . . Neural Networks . . . 8. How a NN Learns: Derivation of the Formulas How to Speed Up . . . • Once we have an observation ( x ( m ) , . . . , x ( m ) n , y ( m ) ), we Neural Networks: A . . . 1 ℓ first input the values x ( m ) , . . . , x ( m ) into the NN. Biological Neuron: A . . . n 1 Artificial Neural . . . • In general, the NN’s output output y ℓ,NN is different Resulting Algorithm from the observed output y ( m ) . ℓ How to Pre-Train a . . . • We want to modify the weights W ℓ,k and w ℓ,ki so as to How to Retain . . . minimize the squared difference Home Page def def = y ℓ,NN − y ( m ) = (∆ y ℓ ) 2 , where ∆ y ℓ J . Title Page ℓ ◭◭ ◮◮ • This minimization is done by using gradient descent: ◭ ◮ ∂J ∂J W ℓ,k → W ℓ,k − λ · , w ℓ,ki → w ℓ,ki − λ · . ∂W ℓ,k ∂w ℓ,ki Page 9 of 17 • The resulting algorithm for updating the weights is Go Back known as backpropagation . Full Screen • This algorithm is based on the following idea. Close Quit

Need for Machine . . . Neural Networks . . . 9. Derivation of the Formulas (cont-d) How to Speed Up . . . ∂J Neural Networks: A . . . • First, one can easily check that = − 2∆ y , so ∂W ℓ, 0 Biological Neuron: A . . . ∂J def Artificial Neural . . . ∆ W ℓ, 0 = − λ · = α · ∆ y ℓ , where α = 2 λ . ∂W ℓ, 0 Resulting Algorithm ∂J = 2∆ y ℓ · y ℓ,k , so ∆ W ℓ,k = − λ · ∂J How to Pre-Train a . . . • Similarly, = ∂W ℓ,k ∂W ℓ,k How to Retain . . . 2 λ · ∆ y ℓ · y ℓ,k , i.e., ∆ W ℓ,k = − ∆ W ℓ, 0 · y ℓ,k . Home Page • The only dependence of y ℓ on w ℓ,ki is via the depen- Title Page dence of y ℓ,k on w ℓ,ki , so, the chain rule leads to ◭◭ ◮◮ ∂J = ∂J · ∂y ℓ,k ◭ ◮ and ∂w ℓ,k 0 ∂y ℓ,k ∂w ℓ,k 0 Page 10 of 17 � n � ∂J Go Back � = 2∆ y ℓ · W ℓ,k · s ′ w ℓ,ki · x i − w ℓ,k 0 · ( − 1) . 0 ∂w ℓ,k 0 Full Screen i =1 Close Quit