on the algorithmic power of spiking neural networks
play

On the Algorithmic Power of Spiking Neural Networks Chi-Ning Chou - PowerPoint PPT Presentation

Jan 13, 2024 •527 likes •1.37k views

On the Algorithmic Power of Spiking Neural Networks Chi-Ning Chou Kai-Min Chung Chi-Jen Lu Harvard University Academia Sinica Academia Sinica ITCS 2019 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks

  1. On the Algorithmic Power of Spiking Neural Networks Chi-Ning Chou Kai-Min Chung Chi-Jen Lu Harvard University Academia Sinica Academia Sinica ITCS 2019 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 1/10

  2. What is Spiking Neural Networks (SNNs)? Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10

  3. What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10

  4. What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. * By Pennstatenews https://www.flickr.com/photos/pennstatelive/37247502805 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10

  5. What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons : Nerve cells * By Pennstatenews https://www.flickr.com/photos/pennstatelive/37247502805 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10

  6. What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons : Nerve cells Synapses : Connections between neurons * By Pennstatenews https://www.flickr.com/photos/pennstatelive/37247502805 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10

  7. What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons : Nerve cells Synapses : Connections between neurons Spikes : Instantaneous signals * By Pennstatenews https://www.flickr.com/photos/pennstatelive/37247502805 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10

  8. What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons Synapses Spikes Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10

  9. What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons Synapses • Various models since the 1900s. Spikes Integrate-and-fire [Lap07] , Hodgkin-Huxley [HH52] , • their variants [Fit61, Ste65, ML81, HR84, Ger95, KGH97, BL03, FTHVVB03, I+03, TMS14] . Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10

  10. What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons Synapses • Various models since the 1900s. Spikes Integrate-and-fire [Lap07] , Hodgkin-Huxley [HH52] , • their variants [Fit61, Ste65, ML81, HR84, Ger95, KGH97, BL03, FTHVVB03, I+03, TMS14] . • Study the behaviors/statistics of SNNs, e.g., firing rate. Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10

  11. What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons Synapses • Various models since the 1900s. Spikes Integrate-and-fire [Lap07] , Hodgkin-Huxley [HH52] , • their variants [Fit61, Ste65, ML81, HR84, Ger95, KGH97, BL03, FTHVVB03, I+03, TMS14] . • Study the behaviors/statistics of SNNs, e.g., firing rate. [Barret-Denève-Machens 2013] empirically showed a connection between • the firing rate of integrate-and-fire SNNs and an optimization problem. Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10

  12. What is Spiking Neural Networks (SNNs)? • Mathematical models for “biological neural networks”. Neurons Synapses • Various models since the 1900s. Spikes Integrate-and-fire [Lap07] , Hodgkin-Huxley [HH52] , • their variants [Fit61, Ste65, ML81, HR84, Ger95, KGH97, BL03, FTHVVB03, I+03, TMS14] . • Study the behaviors/statistics of SNNs, e.g., firing rate. [Barret-Denève-Machens 2013] empirically showed a connection between • the firing rate of integrate-and-fire SNNs and an optimization problem. SNNs seem to have non-trivial computational power. Can we understand them better through the lens of algorithms? Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 2/10

  13. Integrate-and-Fire (IAF) Model [Lapicque 1907] Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10

  14. Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 1 3 6 4 5 2 7 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10

  15. Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 𝜃 0 0 6 𝜃 0 4 𝜃 𝜃 0 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10

  16. Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 6 𝜃 0 4 𝜃 𝜃 0 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10

  17. Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 6 External charging 𝜃 0 4 𝜃 𝜃 0 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10

  18. Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 6 Spiking effects 𝜃 0 4 𝜃 𝜃 0 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10

  19. Integrate-and-Fire (IAF) Model [Lapicque 1907] 𝑱 4 𝑱 8 • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 𝑱 9 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝑱 𝟓 𝜃 0 0 6 • External charging: 𝑱 ∈ ℝ - 𝜃 𝑱 7 𝑱 5 0 4 𝑱 𝟖 𝜃 𝜃 0 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10

  20. Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 6 • External charging: 𝑱 ∈ ℝ - 𝜃 0 4 • Spikes: 𝒕 𝑢 ∈ 0,1 - Firing Rule 𝜃 𝜃 0 𝒕 < 𝑢 = 1 ⇔ 𝒗 < 𝑢 ≥ 𝜃 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10

  21. Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 −𝐷 8@ −𝐷 4@ 6 • External charging: 𝑱 ∈ ℝ - −𝐷 @@ 𝜃 −𝐷 9@ 0 4 • Spikes: 𝒕 𝑢 ∈ 0,1 - Firing Rule 𝜃 𝜃 0 𝒕 < 𝑢 = 1 ⇔ 𝒗 < 𝑢 ≥ 𝜃 • Connectivity: 𝐷 ∈ ℝ -×- −𝐷 A@ −𝐷 5@ 𝜃 −𝐷 7@ 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10

  22. Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 −𝐷 8@ −𝐷 4@ 6 • External charging: 𝑱 ∈ ℝ - −𝐷 @@ 𝜃 −𝐷 9@ 0 4 • Spikes: 𝒕 𝑢 ∈ 0,1 - Firing Rule 𝜃 𝜃 0 𝒕 < 𝑢 = 1 ⇔ 𝒗 < 𝑢 ≥ 𝜃 • Connectivity: 𝐷 ∈ ℝ -×- −𝐷 A@ −𝐷 5@ 𝜃 −𝐷 7@ 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10

  23. Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 6 • External charging: 𝑱 ∈ ℝ - 𝜃 0 4 • Spikes: 𝒕 𝑢 ∈ 0,1 - Firing Rule 𝜃 𝜃 0 𝒕 < 𝑢 = 1 ⇔ 𝒗 < 𝑢 ≥ 𝜃 • Connectivity: 𝐷 ∈ ℝ -×- 𝜃 5 2 7 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10

  24. Integrate-and-Fire (IAF) Model [Lapicque 1907] • Neurons: 𝑜 = {1,2, … , 𝑜} 𝜃 𝜃 • Potential: 𝒗 𝑢 ∈ ℝ - 1 3 • Dynamics: 𝒗 𝑢 + 1 = 𝒗 𝑢 − 𝐷𝒕 𝑢 + 𝑱 𝜃 0 0 6 • External charging: 𝑱 ∈ ℝ - 𝜃 0 4 • Spikes: 𝒕 𝑢 ∈ 0,1 - Firing Rule 𝜃 𝜃 0 𝒕 < 𝑢 = 1 ⇔ 𝒗 < 𝑢 ≥ 𝜃 • Connectivity: 𝐷 ∈ ℝ -×- 𝜃 5 2 7 • Firing rate: 𝒚 𝑢 = (#spikes before time 𝑢)/𝑢 0 0 0 Chi-Ning Chou (Harvard University) On the Algorithmic Power of Spiking Neural Networks 3/10

Recommend

Spiking Neural Networks Advanced Seminar Computer Engineering Eugen Rusakov Spiking Neural

Spiking Neural Networks Advanced Seminar Computer Engineering Eugen Rusakov Spiking Neural

Spiking Neural Networks Advanced Seminar Computer Engineering Eugen Rusakov Spiking Neural Networks Content Introduction &amp; Motivation Human Brain Project Basics and Background Simulators Conclusion

1.59k views • 35 slides
Fast classification using sparsely active spiking networks Hesham Mostafa Institute of neural

Fast classification using sparsely active spiking networks Hesham Mostafa Institute of neural

Fast classification using sparsely active spiking networks Hesham Mostafa Institute of neural computation, UCSD Artificial networks vs. spiking networks backpropagation output layer ...... ...... Multi-layer networks are extremely hidden

431 views • 18 slides
Why Spiking Neural Networks Are Limited in Time (cont-d) Efficient: A Theorem Shift- and Scale-

Why Spiking Neural Networks Are Limited in Time (cont-d) Efficient: A Theorem Shift- and Scale-

Why Spiking Neural . . . Looking for Basic . . . Dependence Is . . . Simplest Possible . . . Why Spiking Neural Networks Are Limited in Time (cont-d) Efficient: A Theorem Shift- and Scale- . . . Michael Beer 1 , Julio Urenda 2 Definitions

817 views • 40 slides
Effect of feedback strength in coupled spiking neural networks Javier Iglesias 1 , 2 in

Effect of feedback strength in coupled spiking neural networks Javier Iglesias 1 , 2 in

Effect of feedback strength in coupled spiking neural networks Javier Iglesias 1 , 2 in collaboration with a-Ojalvo 1 and Alessandro E.P . Villa 2 Jordi Garc 1 Departament de F sica i Enginyeria Nuclear, Universitat Polit` ecnica de

641 views • 35 slides
Scalable Multi-Precision Simulation of Spiking Neural Networks on GPU with OpenCL Dmitri

Scalable Multi-Precision Simulation of Spiking Neural Networks on GPU with OpenCL Dmitri

Scalable Multi-Precision Simulation of Spiking Neural Networks on GPU with OpenCL Dmitri Yudanov (Advanced Micro Devices, USA) Leon Reznik (Rochester Institute of Technology, USA) WCCI 2012, IJCNN, June 12 Agenda Motivation OpenCL.

336 views • 16 slides
Learning Spatio-Temporally Encoded Pattern Transformations in Structured Spiking Neural Networks

Learning Spatio-Temporally Encoded Pattern Transformations in Structured Spiking Neural Networks

Introduction Background Our Approach Results Summary Learning Spatio-Temporally Encoded Pattern Transformations in Structured Spiking Neural Networks 12 Andr e Gr uning, Brian Gardner and Ioana Sporea Department of Computer Science

540 views • 32 slides
GPU-Based Simulation of Spiking Neural Networks with Real-Time Performance &amp; High Accuracy

GPU-Based Simulation of Spiking Neural Networks with Real-Time Performance &amp; High Accuracy

GPU-Based Simulation of Spiking Neural Networks with Real-Time Performance &amp; High Accuracy Dmitri Yudanov, Muhammad Shaaban, Roy Melton, Leon Reznik Department of Computer Engineering Rochester Institute of Technology United States WCCI

522 views • 26 slides
Dynamics of pruning in simulated large-scale spiking neural networks Javier Iglesias 1 , 2 , 3

Dynamics of pruning in simulated large-scale spiking neural networks Javier Iglesias 1 , 2 , 3

Dynamics of pruning in simulated large-scale spiking neural networks Javier Iglesias 1 , 2 , 3 joint work with J. Eriksson 2 , F . Grize 1 , M. Tomassini 1 , A.E.P . Villa 2 , 3 Nonlinear Dynamics and Noise in Biological Systems Workshop: Torino,

418 views • 12 slides
Supervised Learning in Structured Spiking Neural Networks 12 (including a detour to SpiNNaker)

Supervised Learning in Structured Spiking Neural Networks 12 (including a detour to SpiNNaker)

Introduction Background Our Approach Results Results SpiNNaker References Supervised Learning in Structured Spiking Neural Networks 12 (including a detour to SpiNNaker) Andr e Gr uning, Eric Nichols, Brian Gardner and Ioana Sporea

787 views • 45 slides
Neural Networks Find a way to teach networks to do a certain computation (e.g. ICA) Network

Neural Networks Find a way to teach networks to do a certain computation (e.g. ICA) Network

Goals: Understand how (recurrent) networks behave Neural Networks Find a way to teach networks to do a certain computation (e.g. ICA) Network choices Mark van Rossum Neuron models: spiking, binary, rate (and its in-out relation). School of

207 views • 7 slides
Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic

Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic

Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic computation h ( x ) e ( x ) Backpropagation: algorithmic computation of weights M. Magdon-Ismail CSCI 4100/6100 recap: The Neural Network

294 views • 14 slides
Transient sequences in adaptive spiking networks: hypernetworks and spatiotemporal processing

Transient sequences in adaptive spiking networks: hypernetworks and spatiotemporal processing

Transient sequences in adaptive spiking networks: hypernetworks and spatiotemporal processing V.I. Nekorkin Institute of Applied Physics RAS Nizhny Novgorod Plan 1) Transient sequential dynamics in neural networks; 2) Cognitive dynamics are

363 views • 25 slides
The Power of Linear Recurrent Neural Networks Neural Networks Was knnen lineare rekurrente

The Power of Linear Recurrent Neural Networks Neural Networks Was knnen lineare rekurrente

The Power of Linear Recurrent The Power of Linear Recurrent Neural Networks Neural Networks Was knnen lineare rekurrente neuronale Netze? Frieder Stolzenburg Overview Frieder Stolzenburg Introduction Recurrent Neural Networks

1.17k views • 66 slides
On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl Dickstein Tom Brady Deep Neural Networks Recent successes in using Deep neural networks for image classification,

1.32k views • 17 slides
Musical Instrument Classification Using Spiking Neural Networks Jainesh Doshi, Vishrant Tripathi,

Musical Instrument Classification Using Spiking Neural Networks Jainesh Doshi, Vishrant Tripathi,

Musical Instrument Classification Using Spiking Neural Networks Jainesh Doshi, Vishrant Tripathi, Onkar Desai, Shreyas Mangalgi IIT Bombay November 6, 2015 Jainesh Doshi, Vishrant Tripathi, Onkar Desai, Shreyas Mangalgi Short title (UCLA)

398 views • 26 slides
A Temporal Coding Hardware Implementation A Temporal Coding Hardware Implementation for Spiking

A Temporal Coding Hardware Implementation A Temporal Coding Hardware Implementation for Spiking

A Temporal Coding Hardware Implementation A Temporal Coding Hardware Implementation for Spiking for Spiking Neural Networks Neural Networks Marco Aurelio Nuo-Maganda 1 1 , Cesar Torres-Huitzil , Cesar Torres-Huitzil 2 2 Marco Aurelio

339 views • 21 slides
Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating Deep (large) Neural Network (DNN) classifiers Neurons and DNN architectures Numerical training of DNNs (supervised deep learning) Spiking

630 views • 33 slides
Computationally Complete Spiking Neural P Systems Without Delay: Two Types of Neurons Are Enough

Computationally Complete Spiking Neural P Systems Without Delay: Two Types of Neurons Are Enough

Computationally Complete Spiking Neural P Systems Without Delay: Two Types of Neurons Are Enough Rudolf Freund 1 , Marian Kogler 1 , 2 1 Faculty of Informatics, Vienna University of Technology, Austria 2 Institute of Computer Science, Martin

848 views • 71 slides
Energy-Efficient Recurrent Spiking Neural Processor with Unsupervised and Supervised

Energy-Efficient Recurrent Spiking Neural Processor with Unsupervised and Supervised

Energy-Efficient Recurrent Spiking Neural Processor with Unsupervised and Supervised Spike-Timing-Dependent-Plasticity Yu Liu, Peng Li Dept. of Electrical &amp; Computer Engineering Texas A&amp;M University {yliu129, pli}@tamu.edu Neuromorphic

346 views • 9 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
CS 6316 Machine Learning Neural Networks Yangfeng Ji Department of Computer Science University

CS 6316 Machine Learning Neural Networks Yangfeng Ji Department of Computer Science University

CS 6316 Machine Learning Neural Networks Yangfeng Ji Department of Computer Science University of Virginia Overview 1. From Logistic Regression to Neural Networks 2. Expressive Power of Neural Networks 3. Learning Neural Networks 4.

871 views • 49 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides

More recommend