Learning visual motion Statistical models of spike trains Learning visual motion with recurrent neural networks Marius Pachitariu Gatsby Unit, UCL adviser: Maneesh Sahani Marius Pachitariu Learning visual motion with RNNs 1 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Outline Learning visual motion Spatiotemporal filtering Recurrent neural networks can compute visual motion Learning in generative RNN Statistical models of spike trains Recurrent GLM Instantaneous noise Results Marius Pachitariu Learning visual motion with RNNs 2 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Marr’s three levels of analysis Levels of analysis ◮ Computational ◮ Algorithmic / Representational ◮ Physical Marius Pachitariu Learning visual motion with RNNs 3 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Sequential data types ◮ Movies ◮ Spike trains ◮ Language Marius Pachitariu Learning visual motion with RNNs 4 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Spatio-temporal filters ◮ Dominant in both visual neuroscience and computer vision. ◮ Caveats: ◮ not real-time/requires copies of the past → bad for real-world systems, like the brain. Marius Pachitariu Learning visual motion with RNNs 5 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Spatio-temporal filters ◮ Dominant in both visual neuroscience and computer vision. ◮ Caveats: ◮ not real-time/requires copies of the past → bad for real-world systems, like the brain. ◮ too many parameters → bad for learning and generalization. ◮ high computational complexity → bad with high-bandwidth data. Marius Pachitariu Learning visual motion with RNNs 5 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Neural candidates for ST filters ◮ lagged LGN cells (Mastronarde, 1987) Marius Pachitariu Learning visual motion with RNNs 6 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Neural candidates for ST filters ◮ lagged LGN cells (Mastronarde, 1987) ◮ but LGN is an information bottleneck Marius Pachitariu Learning visual motion with RNNs 7 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Neural candidates for ST filters ◮ lagged LGN cells (Mastronarde, 1987) ◮ but LGN is an information bottleneck ◮ but LGN responds precisely to natural movies (Butts et al, 2011) Marius Pachitariu Learning visual motion with RNNs 7 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Compact parametrization of ST filters with an RNN ∞ x t = � W τ y t − τ τ =0 x t W τ W 2 W 1 W 0 y t − τ y t − 2 y t − 1 y t • • • • • • Spatiotemporal filtering Marius Pachitariu Learning visual motion with RNNs 8 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Compact parametrization of ST filters with an RNN ∞ x t = � W τ y t − τ τ =0 x t W τ W 2 W 1 W 0 y t − τ y t − 2 y t − 1 y t • • • • • • Spatiotemporal filtering W τ = ( R ) τ W 0 ∞ x t = W 0 y t + R � ( R ) τ W 0 y t − 1 − τ R R R R R x t − τ x t − 2 x t − 1 x t • • • • • • τ =0 x t = W 0 y t + R x t − 1 W 0 W 0 W 0 W 0 y t − τ y t − 2 y t − 1 y t • • • • • • Recurrent neural network Marius Pachitariu Learning visual motion with RNNs 8 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Compact parametrization of ST filters with an RNN ∞ x t = � W τ y t − τ τ =0 x t W τ W 2 W 1 W 0 y t − τ y t − 2 y t − 1 y t • • • • • • Spatiotemporal filtering W τ = ( R ) τ W 0 ∞ x t = W 0 y t + R � ( R ) τ W 0 y t − 1 − τ R R R R R x t − τ x t − 2 x t − 1 x t • • • • • • τ =0 x t = W 0 y t + R x t − 1 W 0 W 0 W 0 W 0 y t − τ y t − 2 y t − 1 y t • • • • • • Recurrent neural network As a simple example, we fit R to a diverse bank of spatiotemporal filters. Marius Pachitariu Learning visual motion with RNNs 8 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Reconstructions of the ST filters are good Marius Pachitariu Learning visual motion with RNNs 9 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN What do the connections look like? ◮ Spectrum of R Marius Pachitariu Learning visual motion with RNNs 10 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN What do the connections look like? ◮ Spectrum of R ◮ Strongest connections to a given neuron (animation). Marius Pachitariu Learning visual motion with RNNs 10 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Computational complexity and memory requirements ◮ l x by l y by n t filters (12 by 12 by 30) ◮ N (1600) ST filters ◮ Feedforward flops = 2 N l 2 x l 2 y n t Marius Pachitariu Learning visual motion with RNNs 11 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Computational complexity and memory requirements ◮ l x by l y by n t filters (12 by 12 by 30) ◮ N (1600) ST filters ◮ Feedforward flops = 2 N l 2 x l 2 y n t ◮ Recurrent flops = 2 N 2 + 2 N l 2 x l 2 y Marius Pachitariu Learning visual motion with RNNs 11 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Computational complexity and memory requirements ◮ l x by l y by n t filters (12 by 12 by 30) ◮ N (1600) ST filters ◮ Feedforward flops = 2 N l 2 x l 2 y n t ◮ Recurrent flops = 2 N 2 + 2 N l 2 x l 2 y ◮ 5 % non-zero connections in R . ◮ Recurrent flops = 2 · 0 . 05 · N 2 + 2 N l 2 x l 2 y Marius Pachitariu Learning visual motion with RNNs 11 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Computational complexity and memory requirements ◮ l x by l y by n t filters (12 by 12 by 30) ◮ N (1600) ST filters ◮ Feedforward flops = 2 N l 2 x l 2 y n t ◮ Recurrent flops = 2 N 2 + 2 N l 2 x l 2 y ◮ 5 % non-zero connections in R . ◮ Recurrent flops = 2 · 0 . 05 · N 2 + 2 N l 2 x l 2 y ◮ Recurrent flops < Feedforward flops Marius Pachitariu Learning visual motion with RNNs 11 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Advantages of recurrent neural networks ◮ the brain already has the hardware Marius Pachitariu Learning visual motion with RNNs 12 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Advantages of recurrent neural networks ◮ the brain already has the hardware ◮ do not require copies of the past → less memory usage → the brain has short timescales + bottleneck in LGN → no evidence for true delay lines in cortex Marius Pachitariu Learning visual motion with RNNs 12 / 48
Spatiotemporal filtering Learning visual motion Recurrent neural networks can compute visual motion Statistical models of spike trains Learning in generative RNN Advantages of recurrent neural networks ◮ the brain already has the hardware ◮ do not require copies of the past → less memory usage → the brain has short timescales + bottleneck in LGN → no evidence for true delay lines in cortex ◮ fewer parameters → important for learning and generalization Marius Pachitariu Learning visual motion with RNNs 12 / 48
Recommend
More recommend
