neural networks as function primitives
play

Neural Networks as Function Primitives Software/Hardware Support - PowerPoint PPT Presentation

Jan 04, 2024 •239 likes •336 views

Neural Networks as Function Primitives Software/Hardware Support with X-FILES/DANA Schuyler Eldridge 1 Tommy Unger 2 Marcia Sahaya Louis 1 Amos Waterland 3 Margo Seltzer 3 Jonathan Appavoo 2 Ajay Joshi 1 1 Boston University Department of Electrical

  1. Neural Networks as Function Primitives Software/Hardware Support with X-FILES/DANA Schuyler Eldridge 1 Tommy Unger 2 Marcia Sahaya Louis 1 Amos Waterland 3 Margo Seltzer 3 Jonathan Appavoo 2 Ajay Joshi 1 1 Boston University Department of Electrical and Computer Engineering 2 Boston University Department of Computer Science 3 Harvard University School of Engineering and Applied Sciences Boston Area Architecture Workshop ’16 BARC ’16 1/8

  2. Neural Networks as Function Primitives Motivation Neural networks and machine learning are everywhere ( again ) Broad use in high tech and big data, e.g., Google’s Tensorflow [1] Enable automatic parallelization, e.g., ASC [3] Provide a means for approximate computing, e.g., NPU [2] Our vision Neural networks are a new functional primitive useful at various scales of computation [4] With that in mind, we’ve developed software and hardware for the use of accelerator-backed neural network computation [1] Google TensorFlow, https://github.com/tensorflow/tensorflow [2] H. Esmaeilzadeh, A. Sampson et al. , “Neural acceleration for general-purpose approximate programs,” in Proc. MICRO , 2012. [3] A. Waterland, E. Angelino, R. P. Adams, J. Appavoo, and M. Seltzer, “Asc: Automatically scalable computation,” in Proc. ASPLOS , 2014. [4] S. Eldridge, A. Waterland et al. , “Towards general-purpose neural network computing,” in Proc. PACT , 2015. BARC ’16 2/8

  3. Our Contributions Towards this Vision X-FILES: Software/Hardware Extensions E x tensions f or the I ntegration of Machine L earning in E veryday S ystems A defined user and supervisor interface for neural networks This includes supervisor architectural state (hardware) DANA: An Example Multi-Transaction Accelerator D ynamically A llocated N eural Network A ccelerator An accelerator aligning with our multi transaction vision Neural Network Transactions A transaction encapsulates a request by a process to compute the output of a specific neural network for a provided input BARC ’16 3/8

  4. Or: A Drop in Accelerator for a RISC-V Microprocessor What does that mean? 1 Grab a Rocket Chip RISC-V Microprocessor [1] 2 Build a RISC-V toolchain 3 Grab a copy of our X-FILES/DANA accelerator [2] Implemented in Chisel [3] 4 Build an FPGA configuration for Rocket + X-FILES/DANA 5 User processes can safely throw transactions at X-FILES hardware With support for feedforward and learning computation [1] Rocket Chip git repository, UC Berkeley, Online: github.com/ucb-bar/rocket-chip [2] X-FILES/DANA git repository, Boston University, Online ( soon !): github.com/bu-icsg/xfiles-dana [3] J. Bachrach, H. Vo, B. Richards, Y. Lee, A. Waterman, R. Aviˇ zienis et al. , “Chisel: Constructing hardware in a scala embedded language,” in Proc. DAC , 2012, pp. 1216–1225. BARC ’16 4/8

  5. X-FILES Software Components Supervisor API Establishes sets of processes that can access neural network hardware Defines the neural networks that processes are allowed to access More details on the poster! User API Works at the level of transactions A complete request for access to neural network resources, communication of inputs, processing, and communication of outputs Initiating a new transaction Writing data Reading data BARC ’16 5/8

  6. X-FILES/DANA Hardware Rocket PE Table Core Local State Storage L1$ PE Configuration Cache ASID NNID State ASID Transaction Table ASID TID NNID State ASID-NNID Control Table Num ASIDs Walker RR Cache Arbiter Memory ASID-NNID Table Pointer X-Files Hardware Arbiter DANA Accelerator Figure: X-FILES/DANA hardware architecture Components X-FILES Hardware Arbiter maintaining transaction state DANA to move transactions towards completion With support for feedforward or learning computation BARC ’16 6/8

  7. Open Source Plans Remaining Items Linux kernel integration Support for asynchronous data transfer Open Source Availability Should be ready by the end of February On GitHub: github.com/bu-icsg/xfiles-dana BARC ’16 7/8

  8. Acknowledgments This work was supported by the following: A NASA Space Technology Research Fellowship An NSF Graduate Research Fellowship NSF CAREER awards A Google Faculty Research Award BARC ’16 8/8

Recommend

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
7. Artificial neural networks Introduction to neural networks Despite struggling to understand

7. Artificial neural networks Introduction to neural networks Despite struggling to understand

7. Artificial neural networks Introduction to neural networks Despite struggling to understand intricacies of protein, cell, and network function within the brain, neuroscientists would agree on the following simplistic description of how the

157 views • 11 slides
Neural Networks Linear regression (again) Radial basis function networks

Neural Networks Linear regression (again) Radial basis function networks

Neural Networks Linear regression (again) Radial basis function networks Self-organizing maps Recurrent networks Partially based on slides by John A. Bullinaria and J. Kok Linear Regression 7 6 5 4 3 2 1 0 0 1 2 3 4 5

685 views • 45 slides
Bayesian neural networks: a function space view tour Yingzhen Li Microsoft Research Cambridge

Bayesian neural networks: a function space view tour Yingzhen Li Microsoft Research Cambridge

Bayesian neural networks: a function space view tour Yingzhen Li Microsoft Research Cambridge Neural networks 101 Lets say we want to classify different types of cats x : input images; y : output label "cat" build a neural

1.05k views • 45 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
Nonparametric regression using deep neural networks with ReLU activation function Johannes

Nonparametric regression using deep neural networks with ReLU activation function Johannes

Nonparametric regression using deep neural networks with ReLU activation function Johannes Schmidt-Hieber February 2018 Caltech 1 / 20 Many impressive results in applications . . . Lack of theoretical understanding . . . 2 / 20

215 views • 20 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides
Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides
Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides
An Introduction to Neural Networks - Feedforward NN Backpropagation Agathe Merceron Beuth

An Introduction to Neural Networks - Feedforward NN Backpropagation Agathe Merceron Beuth

An Introduction to Neural Networks - Feedforward NN Backpropagation Agathe Merceron Beuth University of Applied Sciences Berlin, Germany 1 Agenda Artificial neuron Activation function Feedforward neural networks Forward

758 views • 39 slides
Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech

Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech

Training Neural Networks Some considerations Gaurav Kumar Center for Language and Speech Processing gkumar@cs.jhu.edu Universal Approximators Neural networks can approximate [1] function. any Capacity Layers hidden layer size

1.19k views • 17 slides
Learning Neural Networks Learning Neural Networks Neural Networks can represent complex Neural

Learning Neural Networks Learning Neural Networks Neural Networks can represent complex Neural

Learning Neural Networks Learning Neural Networks Neural Networks can represent complex Neural Networks can represent complex decision boundaries decision boundaries Variable size. Any boolean function can be Variable size. Any boolean

358 views • 14 slides
On the Impact of the Activation Function on Deep Neural Networks Training Soufiane Hayou

On the Impact of the Activation Function on Deep Neural Networks Training Soufiane Hayou

On the Impact of the Activation Function on Deep Neural Networks Training Soufiane Hayou University of Oxford soufiane.hayou@stats.ox.ac.uk Soufiane Hayou (OxCSML) University of Oxford 1 / 16 Overview Neural Networks as Gaussian Processes

1.02k views • 19 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides
How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

Need for Machine . . . Neural Networks . . . How to Speed Up . . . Neural Networks: A . . . How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . . . Resulting Algorithm Faster by Taking Into How to

464 views • 17 slides
Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron. A look at some current works. Adding an evolutionary strategy. Paul Nuytten CPSC 607 Why Model Neural Networks? The Neuron The nervous

697 views • 4 slides
Attractor neural networks Vi Tij, Tji Vj X U i = T ij V j Dynamics: j V i = sign( U i )

Attractor neural networks Vi Tij, Tji Vj X U i = T ij V j Dynamics: j V i = sign( U i )

Attractor neural networks Vi Tij, Tji Vj X U i = T ij V j Dynamics: j V i = sign( U i ) Energy function: Basins of attraction input recall Outer-product (Hebb) rule P ( ) P ( ) X T ij = i j P (1) P (1) + P (2) P (2) + P (3)

211 views • 17 slides
Towards Evaluating the Robustness of Neural Networks Nicholas Carlini and David Wagner

Towards Evaluating the Robustness of Neural Networks Nicholas Carlini and David Wagner

Towards Evaluating the Robustness of Neural Networks Nicholas Carlini and David Wagner University of California, Berkeley Background A neural network is a function with trainable parameters that learns a given mapping Given an image,

590 views • 58 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What

Neural Networks Hugo Larochelle ( @hugo_larochelle ) Google Brain 2 NEURAL NETWORKS What well cover ... f ( x ) computer vision architectures - convolutional networks - data augmentation 1 - residual networks ... ...

1.29k views • 75 slides

More recommend