The Power of Linear Recurrent Neural Networks Neural Networks Was - PowerPoint PPT Presentation

The Power of Linear Recurrent The Power of Linear Recurrent Neural Networks Neural Networks Was können lineare rekurrente neuronale Netze? Frieder Stolzenburg Overview Frieder Stolzenburg Introduction Recurrent Neural Networks Learning Functions Hochschule Harz, Fachbereich Automatisierung und Informatik, Friedrichstr. 57-59, 38855 Wernigerode, Deutschland Summary, Applications, Future Work E-Mail: fstolzenburg@hs-harz.de joint work with Oliver Obst, Olivia Michael, Sandra Litz, and Falk Schmidsberger in the Decorating project (DEep COnceptors for tempoRal dATa mINinG) funded by DAAD (Germany) and UA (Australia) 1 / 26

Overview The Power of Linear Recurrent Neural Networks 1 Introduction Frieder Stolzenburg Overview 2 Recurrent Neural Networks Introduction Recurrent Neural Networks 3 Learning Functions Learning Functions Summary, 4 Summary, Applications, Future Work Applications, Future Work 2 / 26

The Power of Linear Recurrent Neural Networks Introduction 1 Frieder Stolzenburg Recurrent Neural Networks 2 Introduction Time Series and Prediction Number Puzzles Learning Functions 3 Recurrent Neural Networks 4 Summary, Applications, Future Work Learning Functions Summary, Applications, Future Work 3 / 26

Time Series and Prediction The Power of Linear Definition Recurrent Neural A time series is a series of data points in d dimensions Networks S ( 0 ) , . . . , S ( n ) ∈ R d where d ≥ 1 and n ≥ 0. Frieder Stolzenburg Examples: Introduction Time Series and trajectories (of pedestrians, dance, sports, etc.) Prediction Number Puzzles stock quotations (of one or more companies) Recurrent weather forecast Neural Networks natural language processing Learning (speech recognition, text comprehension, question answering) Functions Time Series Analysis allows Summary, Applications, prediction of further values Future Work data compression, i.e. compact representation (e.g. by a function f ( t )) 4 / 26

Number Puzzles The Power of Linear Number puzzles can be understood as one-dimensional time series. Recurrent Neural Such exercises often are part of Networks intelligence tests, Frieder Stolzenburg entrance examinations, or job interviews. Introduction Time Series and Prediction Number Puzzles Recurrent Neural Networks Learning Functions Summary, Applications, Future Work 5 / 26

Number Puzzles The Power of Linear Number puzzles can be understood as one-dimensional time series. Recurrent Neural Such exercises often are part of Networks intelligence tests, Frieder Stolzenburg entrance examinations, or job interviews. Introduction Time Series and Examples: Which numbers continue the following series? Prediction Number Puzzles 1,3,5,7,9 1 Recurrent 1,2,4,8,16 2 Neural Networks Learning Functions Summary, Applications, Future Work 5 / 26

Number Puzzles The Power of Linear Number puzzles can be understood as one-dimensional time series. Recurrent Neural Such exercises often are part of Networks intelligence tests, Frieder Stolzenburg entrance examinations, or job interviews. Introduction Time Series and Examples: Which numbers continue the following series? Prediction Number Puzzles 1,3,5,7,9,11,13,15 (arithmetic series) 1 Recurrent 1,2,4,8,16 2 Neural Networks Learning Functions Summary, Applications, Future Work 5 / 26

Number Puzzles The Power of Linear Number puzzles can be understood as one-dimensional time series. Recurrent Neural Such exercises often are part of Networks intelligence tests, Frieder Stolzenburg entrance examinations, or job interviews. Introduction Time Series and Examples: Which numbers continue the following series? Prediction Number Puzzles 1,3,5,7,9,11,13,15 (arithmetic series) 1 Recurrent 1,2,4,8,16,32,64,128 (geometric series) 2 Neural Networks Learning Functions Summary, Applications, Future Work 5 / 26

Number Puzzles The Power of Linear Number puzzles can be understood as one-dimensional time series. Recurrent Neural Such exercises often are part of Networks intelligence tests, Frieder Stolzenburg entrance examinations, or job interviews. Introduction Time Series and Examples: Which numbers continue the following series? Prediction Number Puzzles 1,3,5,7,9,11,13,15 (arithmetic series) 1 Recurrent 1,2,4,8,16,32,64,128 (geometric series) 2 Neural Networks Question: Learning Functions Can number puzzles be solved automatically by computer programs? Summary, Applications, Future Work 5 / 26

Number Puzzles The Power of Linear Number puzzles can be understood as one-dimensional time series. Recurrent Neural Such exercises often are part of Networks intelligence tests, Frieder Stolzenburg entrance examinations, or job interviews. Introduction Time Series and Examples: Which numbers continue the following series? Prediction Number Puzzles 1,3,5,7,9,11,13,15 (arithmetic series) 1 Recurrent 1,2,4,8,16,32,64,128 (geometric series) 2 Neural Networks Question: Learning Functions Can number puzzles be solved automatically by computer programs? Summary, We will do this by means of artificial recurrent neural networks (RNN), namely Applications, Future Work predictive neural networks with a reservoir of randomly connected neurons, which are related to echo state networks (ESNs) [1]. 5 / 26

The Power of Linear Recurrent Neural Networks Introduction 1 Frieder Stolzenburg Recurrent Neural Networks 2 Introduction Recurrent Neural Learning Functions Networks 3 Artificial Neurons Feedforward Neural Nets RNN Architecture 4 Summary, Applications, Future Work Predictive Neural Networks Example Network Dynamics Long-Term Behaviour Learning Functions Summary, Applications, Future Work 6 / 26

Artificial Neurons The Power of A recurrent neural network (RNN) is a directed graph (usually fully connected), Linear Recurrent i.e. an interconnected group of N nodes, called neurons. Neural Networks The activation of a neuron y at (discrete) time t + τ for some time step τ is Frieder computed from the activation of the neurons x 1 , . . . , x n , that are connected to y Stolzenburg with the weights w 1 , . . . , w n , at time t : Introduction � � y ( t + τ ) = g w 1 · x 1 ( t ) + · · · + w n · x n ( t ) Recurrent Neural Networks g is called activation function. Artificial Neurons Feedforward Neural Nets Neural Unit RNN Architecture Predictive Neural Networks Example w 1 Network Dynamics x 1 . . . Long-Term Behaviour y Learning w n x n Functions Summary, Applications, Future Work 7 / 26

Feedforward Neural Nets The Power of Multi-Layer Feedforward Network Sigmoidal Activation Function Linear Recurrent Neural Output units a i 1 Networks Frieder w j,i Stolzenburg 0.5 Hidden units a j Introduction Recurrent w k,j Neural 0 Networks Input units a k -6 -4 -2 0 2 4 6 Artificial Neurons Feedforward Neural Nets RNN Architecture Predictive Neural Networks Network corresponds to directed acyclic graph with possibly multiple layers. Example Network Dynamics Activation function g often is sigmoidal (i.e. non-linear threshold function). Long-Term Behaviour Learning Complex functions can be learned by backpropagation (not here). Functions There are no internal states in the network (no memory or time). Summary, Applications, Future Work 8 / 26

RNN Architecture The Power of Linear In the reservoir, neurons may be Recurrent Recurrent Neural Network (RNN) Neural connected recurrently. Networks reservoir Frieder In the transition matrix W , an entry w ij input Stolzenburg in row i and column j states the weight output Introduction of the edge from neuron j to neuron i . in W Recurrent If there is no connection, then w ij = 0. Neural W out Networks Echo state networks [1]: Artificial Neurons Feedforward Neural Input and reservoir weights W in and Nets RNN Architecture W res form random matrices (but with Predictive Neural Networks stationary dynamics [3]). Example res W Network Dynamics Only the weights leading to the Long-Term Behaviour output neurons W out are learned. Learning Functions Summary, Applications, Future Work 9 / 26

Predictive Neural Networks The Power of Linear A predictive neural network (PrNN) is a RNN with the following properties: Recurrent Neural Networks 1 For all neurons we have linear activation, i.e., everywhere g is the identity. Frieder This simplifies learning a lot. Stolzenburg Still non-linear functions over time can be represented. Introduction 2 The weights in W in and W res are initially taken randomly, independently, and Recurrent Neural identically distributed from the standard normal distribution, whereas the Networks output weights W out are learned. Artificial Neurons Feedforward Neural Nets 3 There is no clear distinction of input and output but only one joint group of d RNN Architecture Predictive Neural input/output neurons. They may be arbitrarily connected like the reservoir Networks Example neurons. Network Dynamics Long-Term Behaviour x out ( t ) = x in ( t + 1 ) Learning Functions Summary, Applications, Future Work 10 / 26

Example The Power of t 0 1 2 3 4 Linear table : Recurrent f ( t ) 1 3 5 7 9 Neural Networks Frieder Stolzenburg Introduction Recurrent Neural Networks Artificial Neurons Feedforward Neural Nets RNN Architecture Predictive Neural Networks Example Network Dynamics Long-Term Behaviour Learning Functions Summary, Applications, Future Work 11 / 26