Brain-Inspired Computing H.-S. Philip Wong Stanford University - PowerPoint PPT Presentation

The N3XT Technology for Stanford University Brain-Inspired Computing H.-S. Philip Wong Stanford University Stanford SystemX Alliance 2007.11.08 Department of Electrical Engineering Stanford University 1 H.-S. Philip Wong 2015.04.15

Stanford University 2 H.-S. Philip Wong 2015.04.15 Source: Google

Stanford University 3 H.-S. Philip Wong 2015.04.15 Source: vrworld.com

Stanford University 4 H.-S. Philip Wong 2015.04.15 Source: BDC Magazine

1988 Winter Olympic Games in Calgary, Canada Stanford University 5 H.-S. Philip Wong 2015.04.15

Stanford University 6 H.-S. Philip Wong 2015.04.15

Stanford University 7 H.-S. Philip Wong 2015.04.15 Source: Gogamguro.com

100’s of kW Source: Google Stanford University 8 H.-S. Philip Wong 2015.04.15

Scale Up Requires Energy Efficiency Estimated Application Hardware power used consumption Emulating 4.5% of human brain :10 13 Large scale Blue Gene/P: 2.9 MW synapses, 10 9 neurons 36,864 nodes, (LINPACK) 147,456 cores Deep sparse autoencoder : 1,000 CPUs ~100 kW 10 9 synapses, 10M images (16,000 cores) (cores only) 2 GPUs 1,200 W Convolutional neural net with 60M Small to moderate scale synapses, 650K neurons Restricted Boltzmann Machine : GPU 550 W 28M synapses; 69,888 neurons CPU 65 W GPU 238 W Processing 1 s of speech using deep neural network CPU (4 cores) 80 W S. B. Eryilmaz et al., IEDM 2015 Stanford University 9 H.-S. Philip Wong 2015.04.15

These nanotechnology innovations will have to be developed in close coordination with new computer architectures, and will likely be informed by our growing understanding of the brain — a remarkable, fault- tolerant system that consumes less power than an incandescent light bulb. Stanford University 10 H.-S. Philip Wong 2015.04.15

Approaches of Neuromorphic Hardware Stanford University 11 H.-S. Philip Wong 2015.04.15

Approaches of Neuromorphic Hardware Biology-based algorithms models / ML algorithms Conventional Stanford University 12 H.-S. Philip Wong 2015.04.15

Approaches of Neuromorphic Hardware Neuromorphic Conventional hardware hardware (CPU, GPU, supercomputers, etc) Stanford University 13 H.-S. Philip Wong 2015.04.15

Approaches of Neuromorphic Hardware Neuromorphic with analog non- Conventional hardware volatile memory hardware (CPU, GPU, synapses supercomputers, etc) Stanford University 14 H.-S. Philip Wong 2015.04.15

Approaches of Neuromorphic Hardware Neuromorphic with analog non- Conventional hardware volatile memory hardware (CPU, GPU, synapses supercomputers, etc) Biology-based Brain emulation on algorithms models / BlueGene [7] HTM [3] Stanford University 15 H.-S. Philip Wong 2015.04.15

Approaches of Neuromorphic Hardware Neuromorphic with analog non- Conventional hardware volatile memory hardware (CPU, GPU, synapses supercomputers, etc) Biology-based Brain emulation on algorithms models / BlueGene [7] HTM [3] ML algorithms Conventional “Cats on YouTube” ANNs: ConvNets, DNNs, DBNs [10-13] Stanford University 16 H.-S. Philip Wong 2015.04.15

Approaches of Neuromorphic Hardware Neuromorphic with analog non- Conventional hardware volatile memory hardware (CPU, GPU, synapses supercomputers, etc) Biology-based TrueNorth [16] Brain emulation on algorithms models / BlueGene [7] SpiNNaker [19] HTM [3] Human Brain Project [20] ML algorithms Conventional “Cats on YouTube” ANNs: ConvNets, DNNs, DBNs [10-13] Stanford University 17 H.-S. Philip Wong 2015.04.15

Approaches of Neuromorphic Hardware Neuromorphic with analog non- Conventional hardware volatile memory hardware (CPU, GPU, synapses supercomputers, etc) Biology-based TrueNorth [16] Hebbian learning Brain emulation on algorithms models / Spike-based ANN BlueGene [7] SpiNNaker [19] PCM, RRAM, HTM [3] Human Brain CBRAM Project [20] ML algorithms Conventional “Cats on YouTube” ANNs: ConvNets, DNNs, DBNs [10-13] Stanford University 18 H.-S. Philip Wong 2015.04.15

Approaches of Neuromorphic Hardware Neuromorphic with analog non- Conventional hardware volatile memory hardware (CPU, GPU, synapses supercomputers, etc) Biology-based TrueNorth [16] Hebbian learning Brain emulation on algorithms models / Spike-based ANN BlueGene [7] SpiNNaker [19] PCM, RRAM, HTM [3] Human Brain CBRAM Project [20] ML algorithms Conventional “Cats on YouTube” ANN, RBM, ANNs: ConvNets, sparse learning DNNs, DBNs [10-13] PCM, RRAM Stanford University 19 H.-S. Philip Wong 2015.04.15

Many of these breakthroughs will require new kinds of nanoscale devices and materials integrated into three-dimensional systems and may take a decade or more to achieve. Stanford University 21 H.-S. Philip Wong 2015.04.15

N3XT Nanosystems Computation immersed in memory Ultra-dense Memory vertical connections Computing logic Stanford University 22 H.-S. Philip Wong 2015.04.15

N3XT Nanosystems Computation immersed in memory Ultra-dense Memory vertical connections Computing logic Impossible with today’s technologies Stanford University 23 H.-S. Philip Wong 2015.04.15

N3XT: Computation Immersed in Memory 3D Resistive RAM Massive storage Not TSV 1D CNFET, 2D FET Compute, RAM access thermal MRAM Ultra-dense, Quick access fine-grained vias 1D CNFET, 2D FET Compute, RAM access thermal Silicon 1D CNFET, 2D FET Compute, Power, compatible Clock thermal Stanford University 24 H.-S. Philip Wong 2015.04.15

Aly et al., IEEE Computer , 2015 Stanford University 25 H.-S. Philip Wong 2015.04.15

Non-Volatile Memory (NVM) Top Electrode Active Top Electrode Top Electrode phase change material solid metal electrolyte oxide metal oxygen ion atoms oxide filament isolation switching oxygen region filament vacancy Bottom Electrode Bottom Electrode BottomElectrode partially reset state TiN Cu ion 25nm poly c-GST Electrolyte TiN TiOx/ amorphous SiO 2 HfOx Bottom electrode 12 nm 10 nm SiO 2 TiN Conductive bridge Phase change Metal oxide resistive memory (CBRAM) memory (PCM) switching memory (RRAM) D. Kuzum et al., Nano Lett . 2013, Y. Wu et al., IEDM 2013; A. Calderoni et al., IMW 2014 Stanford University 26 H.-S. Philip Wong 2015.04.15

Non- Volatile Memory (NVM) → Synapse  Analog programmable  Scalable to a few nm  Stack in 3D partially reset state TiN Cu ion 25nm poly c-GST Electrolyte TiN TiOx/ amorphous SiO 2 HfOx Bottom electrode 12 nm 10 nm SiO 2 TiN Conductive bridge Phase change Metal oxide resistive memory (CBRAM) memory (PCM) switching memory (RRAM) D. Kuzum et al., Nano Lett . 2013, Y. Wu et al., IEDM 2013; A. Calderoni et al., IMW 2014 Stanford University 27 H.-S. Philip Wong 2015.04.15

Nanoscale Memory as Synaptic Weights Synaptic updates in the brain: basis for learning Requirement: analog resistance change 100-step grey scale (1% resolution) 5 10 5 10 Resistance (Ohm) Resistance (Ohm) 4 4 10 10 Phase change Partial Partial synapse RESET SET 3 10 3 0 500 1000 1500 2000 2500 10 2100 2200 2300 Pulse Number Pulse Number D. Kuzum et al. , Nano Lett., p. 2179 (2012) Stanford University 28 H.-S. Philip Wong 2015.04.15

Nanoscale Memory Can Emulate Biological Synaptic Behavior STDP (spike-timing-dependent plasticity) Synaptic weight change  w (%) 100 100 Synaptic Weight Change  w (%) 120 10 ms 25 ms LTP-1 140  w (%)  w (%) 50 50 15 ms 35 ms LTP-2 120 80 0 0 LTP-3 20 ms 45 ms   =29.5 100 LTD-1 -50 -50 40 LTD-2 80 -50 0 50 -50 0 50  t (ms)  t (ms) LTD-3   =-18.6   =11 60 100 100 0   =-11.3 40   =20.7  w (%)  w (%) 50 50   =-29 20 -40 0 0 0 -50 -50 -60 -40 -20 0 20 40 60 0 20 40 60 80 100 -50 0 50 -50 0 50 Spike timing  t (ms) Number of pre/post spike pairs  t (ms)  t (ms) Various STDP kernels Various time constants Weight update saturation D. Kuzum et al. , Nano Lett., p. 2179 (2012) Stanford University 29 H.-S. Philip Wong 2015.04.15

Nanoscale Memory Can Emulate Biological Synaptic Behavior STDP (spike-timing-dependent plasticity) Synaptic weight change  w (%) 100 100 Synaptic Weight Change  w (%) 120 10 ms 25 ms LTP-1 140  w (%)  w (%) 50 50 15 ms 35 ms LTP-2 120 80 0 0 LTP-3 20 ms 45 ms   =29.5 100 LTD-1 -50 -50 40 LTD-2 80 -50 0 50 -50 0 50  t (ms)  t (ms) LTD-3   =-18.6   =11 60 100 100 0   =-11.3 40   =20.7  w (%)  w (%) 50 50   =-29 20 -40 0 0 0 -50 -50 -60 -40 -20 0 20 40 60 0 20 40 60 80 100 -50 0 50 -50 0 50 Spike timing  t (ms) Number of pre/post spike pairs  t (ms)  t (ms) Various STDP kernels Various time constants Weight update saturation D. Kuzum et al. , Nano Lett., p. 2179 (2012) Stanford University 30 H.-S. Philip Wong 2015.04.15