universality and individuality in recurrent neural
play

Universality and Individuality in Recurrent Neural Networks Niru - PowerPoint PPT Presentation

Apr 21, 2023 •43 likes •183 views

Universality and Individuality in Recurrent Neural Networks Niru Maheswaranathan, Alex Williams, Matthew D Golub, Surya Ganguli, David Sussillo Google Brain & Stanford University NeurIPS 2019 Poster #179 arxiv:1907.08549 Artificial neural

  1. Universality and Individuality in Recurrent Neural Networks Niru Maheswaranathan, Alex Williams, Matthew D Golub, Surya Ganguli, David Sussillo Google Brain & Stanford University NeurIPS 2019 Poster #179 arxiv:1907.08549

  2. Artificial neural networks in neuroscience Advantages: Can train ANNs to accomplish tasks analogous to those studied in animals. • Can inspect/probe/dissect artificial networks very easily. • Can easily initiate a huge number of in silico studies • ... Poster #179 arxiv:1907.08549

  3. Artificial & biological neural networks Motion context a b c Dots off Line Rotate Sort by attractor Attractor irrelevant Dots colour Dots on off Motion Colour Colour 1 Choice 1 Choice 2 Choice 1 Choice 2 Choice 1 Choice 2 6 Choice Choice Choice Motion Colour Strong Weak Strong Strong Weak Strong To choice 1 To choice 2 To choice 1 To choice 2 Colour context Motion context a b c Maheswaranathan et al, 2018 Sort by irrelevant colour Choice Rotate axis Dots Dots on off Dots a c Units off Motion Colour Colour 1 Go Unit 145 Go 1 Yamins & DiCarlo, 2014 RT Go Activitity (a.u.) 1.0 1.5 Dly Go Choice 1 Choice 2 Choice 1 Choice 2 Choice 1 Choice 2 Anti Choice Choice Choice 0.5 RT Anti Dly Anti Normalized task variance Motion Colour 0.0 DM 1 Strong Weak Strong Strong Weak Strong DM 2 0.0 Time (s) 1.5 Ctx DM 1 To choice 1 To choice 2 To choice 1 To choice 2 Ctx DM 2 MultSen DM b Dly DM 1 a Unit 145 Dly DM 2 grid-like Mante & Sussillo et al. Nature 2013 Ctx Dly DM 1 Task variance 0.05 Ctx Dly DM 2 MultSen Dly DM DMS DNMS 0.00 DMC Go DNMC DNMC 0 Task 1 2 3 4 5 6 7 8 9 10 11 12 Clusters b band-like d e 0.5 Go tSNE RT Go Dly Go Anti RT Anti Performance change after lesioning � Dly Anti DM 1 DM 2 Ctx DM 1 Ctx DM 2 c border MultSen DM Cueva & Wei, ICLR , 2018 Dly DM 1 Dly DM 2 Ctx Dly DM 1 Ctx Dly DM 2 Carnevale et al, NEURON , 2015 MultSen Dly DM DMS DNMS DMC DNMC –0.5 1 2 3 4 5 6 7 8 9 10 11 12 Clusters Yang, et al., Nature Neuroscience 2019 d e Rajan, Harvey, Tank, Neuron , 2016 Normalized activity target move onset Sussillo et al., Nature Neuroscience , 2015 Zipser & Andersen, Science , 1988 Orhan & Ma, Nature Neuroscience , 2019

  4. Artificial & biological neural networks Networks have surprisingly similar representations… Motion context a b c Dots off Line Rotate Sort by attractor Attractor irrelevant Dots colour Dots on off Motion Colour Colour 1 Choice 1 Choice 2 Choice 1 Choice 2 Choice 1 Choice 2 6 Choice Choice Choice Motion Colour Strong Weak Strong Strong Weak Strong To choice 1 To choice 2 To choice 1 To choice 2 Colour context Motion context a b c Maheswaranathan et al, 2018 Sort by irrelevant colour Choice Rotate axis Dots Dots on off Dots a c Units off Motion Colour Colour 1 Go Unit 145 Go 1 Yamins & DiCarlo, 2014 RT Go Activitity (a.u.) 1.0 1.5 Dly Go Choice 1 Choice 2 Choice 1 Choice 2 Choice 1 Choice 2 Anti Choice Choice Choice 0.5 RT Anti Dly Anti Normalized task variance Motion Colour 0.0 DM 1 Strong Weak Strong Strong Weak Strong DM 2 0.0 Time (s) 1.5 Ctx DM 1 To choice 1 To choice 2 To choice 1 To choice 2 Ctx DM 2 MultSen DM b Dly DM 1 a Unit 145 Dly DM 2 grid-like Mante & Sussillo et al. Nature 2013 Ctx Dly DM 1 Task variance 0.05 Ctx Dly DM 2 MultSen Dly DM DMS DNMS 0.00 DMC Go DNMC DNMC 0 Task 1 2 3 4 5 6 7 8 9 10 11 12 Clusters b band-like d e 0.5 Go tSNE RT Go Dly Go Anti RT Anti Performance change after lesioning � Dly Anti DM 1 DM 2 Ctx DM 1 Ctx DM 2 c border MultSen DM Cueva & Wei, ICLR , 2018 Dly DM 1 Dly DM 2 Ctx Dly DM 1 Ctx Dly DM 2 Carnevale et al, NEURON , 2015 MultSen Dly DM DMS DNMS DMC DNMC –0.5 1 2 3 4 5 6 7 8 9 10 11 12 Clusters Yang, et al., Nature Neuroscience 2019 d e Rajan, Harvey, Tank, Neuron , 2016 Normalized activity target move onset Sussillo et al., Nature Neuroscience , 2015 Zipser & Andersen, Science , 1988 Orhan & Ma, Nature Neuroscience , 2019

  5. Artificial & biological neural networks …but are composed of drastically different elements! Artificial neuron Biological neuron Poster #179 arxiv:1907.08549

  6. Central question When trained to perform the same task , why should we expect artificial and biological networks to be similar , given the drastic di ff erences in underlying mechanism ? Poster #179 arxiv:1907.08549

  7. This work: an empirical approach Network mechanisms RNN architectures (e.g. LSTMs, GRUs, …) Nonlinearities (e.g. ReLU, tanh) … Similarity measures Canonical correlation analysis (CCA) Centered kernel alignment (CKA) Tasks Decision making Pattern generation Working memory Poster #179 arxiv:1907.08549

  8. Evidence of both universality and individuality Poster #179 arxiv:1907.08549

  9. Evidence of both universality and individuality Pattern generation task Poster #179 arxiv:1907.08549

  10. Evidence of both universality and individuality Analyzing trained networks Pattern generation task PC #3 PC #1 PC #2 Poster #179 arxiv:1907.08549

  11. Evidence of both universality and individuality Analyzing trained networks Pattern generation task PC #3 PC #1 PC #2 Poster #179 arxiv:1907.08549

  12. Evidence of both universality and individuality Analyzing trained networks Pattern generation task PC #3 PC #1 PC #2 Network representations show individuality Poster #179 arxiv:1907.08549

  13. Evidence of both universality and individuality Analyzing trained networks Pattern generation task PC #3 PC #1 PC #2 Network representations show individuality but aspects of the computation are universal prediction Linear Target Poster #179 arxiv:1907.08549

  14. Learn more at Poster #179 Poster #179 arxiv:1907.08549

Recommend

CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I :

CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I :

Ugur HALICI - METU EEE - ANKARA 11/18/2004 CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I : Recurrent Neural Networks CHAPTER I Recurrent Neural Networks Introduction In this chapter first the

404 views • 27 slides
Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural circuits David Sussillo with Valerio Mante and Bill Newsome Table of contents Introduc(on Training recurrent neural networks(RNNs) Understanding how

1.57k views • 105 slides
Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020 Introduction Recurrent Neural Network LSTM Agenda Get introduced to different recurrent neural architecture e.g. , RNNs, LSTMs, GRUs etc. Get

743 views • 48 slides
Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The Problem of Long-Term Dependencies RNN short-term dependencies Language model trying to predict the next word based on the previous ones the clouds

729 views • 27 slides
IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks Stephan Oepen University of Oslo March 24, 2019 Our Roadmap Last Week Language structure: sequences, trees, graphs Recurrent Neural

217 views • 19 slides
CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer

CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer

CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer University of Washington [most slides from Yejin Choi] RECURRENT NEURAL L NE NETWOR WORKS Recurrent Neural Networks (RNNs) Each input

787 views • 29 slides
CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class Recurrent Neural Network Cell Recurrent Neural Networks (RNNs) Bi-Directional Recurrent Neural Networks (Bi-RNNs) Multiple-layer /

583 views • 47 slides
CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class Recurrent Neural Network Cell Recurrent Neural Networks (RNNs) Bi-Directional Recurrent Neural Networks (Bi-RNNs) Multiple-layer /

793 views • 21 slides
IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen University of Oslo March 10, 2019 Our Roadmap Today Language structure: sequences, trees, graphs Recurrent Neural Networks Different

780 views • 76 slides
Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep

Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep Learning textbook Ch. 10 Recurrent Neural Networks Long

667 views • 18 slides
Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang Introduction Recurrent neural networks Dates back to (Rumelhart et al. , 1986) A family of neural networks for handling

348 views • 34 slides
Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent

Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent

Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent Neural Networks Compact schematic drawing of standard multi-layer perceptron (MLP) output hidden input Modeling sequential data So far we

796 views • 50 slides
Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova

Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova

Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova March 9, 2020 Alex Graves, Supervised Sequence Labelling with Recurrent Neural Networks http://colah.github.io/posts/2015-08-Understanding-LSTMs/

538 views • 18 slides
Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random Variables Text Preprocessing Language Modeling Recurrent Neural Networks (RNN) LSTM Bidirectional RNN Deep RNN d2l.ai

798 views • 55 slides
RNN Input Layer RNN Hidden Layer RNN h t-1 h t x t (Picture adapted from Andrej

RNN Input Layer RNN Hidden Layer RNN h t-1 h t x t (Picture adapted from Andrej

Outline Recurrent Neural Networks (RNNs) Neural Machine Transla/on NMT basics (Sutskever et al., 2014) ABenCon mechanism (Bahdanau et al., 2015) Marcello Federico 2016 Recurrent Neural Networks (RNNs) Based on slides kindly provided

437 views • 24 slides
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio Presenter: Yu-Wei Lin Background: Recurrent Neural Network Traditional RNNs encounter

229 views • 18 slides
Understanding Hidden Memories of Recurrent Neural Networks Yao Ming , Shaozu Cao, Ruixiang Zhang,

Understanding Hidden Memories of Recurrent Neural Networks Yao Ming , Shaozu Cao, Ruixiang Zhang,

Understanding Hidden Memories of Recurrent Neural Networks Yao Ming , Shaozu Cao, Ruixiang Zhang, Zhen Li, Yuanzhe Chen, Yangqiu Song, Huamin Qu. THE HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY What is a Recurrent Neural Network? H K U S T

397 views • 37 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Recurrent Neural Network Agenda Recurrent Neural Network

Recurrent Neural Network Agenda Recurrent Neural Network

Recurrent Neural Network Agenda Recurrent Neural Network NER sequence tagging RNN

670 views • 46 slides
The Power of Linear Recurrent Neural Networks Neural Networks Was knnen lineare rekurrente

The Power of Linear Recurrent Neural Networks Neural Networks Was knnen lineare rekurrente

The Power of Linear Recurrent The Power of Linear Recurrent Neural Networks Neural Networks Was knnen lineare rekurrente neuronale Netze? Frieder Stolzenburg Overview Frieder Stolzenburg Introduction Recurrent Neural Networks

1.17k views • 66 slides
INF5820: Language Technological Applications Applications of Recurrent Neural Networks Stephan

INF5820: Language Technological Applications Applications of Recurrent Neural Networks Stephan

INF5820: Language Technological Applications Applications of Recurrent Neural Networks Stephan Oepen University of Oslo November 6, 2018 Most Recently: Variants of Recurrent Neural Networks To a first approximation, the de facto consensus in

339 views • 23 slides
Machine Learning for Computational Linguistics Recurrent neural networks (RNNs) ar

Machine Learning for Computational Linguistics Recurrent neural networks (RNNs) ar

Machine Learning for Computational Linguistics Recurrent neural networks (RNNs) ar ltekin University of Tbingen Seminar fr Sprachwissenschaft July 5, 2016 Neural networks: a quick summary Recurrent neural networks July 5, 2016

570 views • 35 slides
Sequential Data with Neural Networks Recurrent Neural Networks Sequential input / output Greg

Sequential Data with Neural Networks Recurrent Neural Networks Sequential input / output Greg

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Sequential Data with Neural Networks Recurrent Neural

303 views • 4 slides
NLP Programming Tutorial 8 - Recurrent Neural Nets Graham Neubig Nara Institute of Science and

NLP Programming Tutorial 8 - Recurrent Neural Nets Graham Neubig Nara Institute of Science and

NLP Programming Tutorial 8 Recurrent Neural Nets NLP Programming Tutorial 8 - Recurrent Neural Nets Graham Neubig Nara Institute of Science and Technology (NAIST) 1 NLP Programming Tutorial 8 Recurrent Neural Nets Feed Forward Neural

604 views • 34 slides

More recommend