CoreNeuron : Morphologically Detailed Neuron Simulations Building, - PowerPoint PPT Presentation

CoreNeuron : Morphologically Detailed Neuron Simulations Building, Simulating and Optimizing Large Neuron Networks on GPUs Pramod Kumbhar, Michael Hines 7 th April 2016, GTC & Blue Brain HPC Team

Understanding Brain.. Brain/MINDS 2014 Human Brain Project 2013 BRAIN Initiative IBM: Cat’s brain scale 2013 simulation on BG-P 2009 Izhikevic: brain scale simulation on cluster 2007 1.5 billion cells 1 million cells Better understanding of brain See source1

Brain Simulations See source3 See source4 Morphologically Point Neurons Molecular Level Detailed Neurons Blue Brain Project, EPFL

Reverse Engineering Brain (10000 feet view) See source6 See source5

Reconstruction Workflow 15+ years of Experiments • 14,000 Neurons recorded • and labeled 2,052 Classified neurons • 1,009 Reconstructed • neurons 2,000+ Ion channel • recordings 4,000+ Electrical recordings • of single neurons 5,000+ Synaptic recordings • of pairs of neurons Markram et al. 2015, Cell

Modeling Neuron See source7 HH, 1952 See source8

Ion Channels Prominent components of nervous system • More than 300 ion channels • Every channel is a compute kernel, no single hotspot! Biologist view: compartment model

NMODL: Source to Source Compiler Lexical Analyzer tokens Syntax Analyzer parse tree NMODL Semantic Analyzer parse tree ~ Intermediate Representation backends Cuda Cyme: SIMD OpenACC C DSL Portable Performance

OpenACC Kernels User defined DS: Major challenge for many application auto-generated kernel wrap OpenACC and vectorisation hints related pragmas AoS/SoA, Vectorisation, Memory Coalescing etc.. OpenACC API’s to copy the complex data structure

node_index parent_index GPU: Cell level Parallelism (some kernels) Step 3 Step 2 Step 1 0 -1 5 -1 bksub: for(i = x; i < nodes; i++) { step 0 rhs[ i ] -= b[i] * rhs[ parent[i] ] rhs[i] /= d[i] 0 1 6 5 node : 1 2 3 4 } thread 0 parent[i]: 0 1 2 3 2 1 7 6 node : 6 7 8 9 thread 1 2 7 3 8 parent[i]: 5 6 7 8 Depth of Tree Stride 4 9 3 8 node : 156 157 158 159 thread 31 parent[i]: 155 156 157 158 warp 0 Memory addresses

Cell Interleaving Nodes Parents parent_index node_index 0 -1 Roots parent all cells root 0 -1 1 -1 1 2 -1 0 2 0 3 1 Cell 0 parent 4 2 3 5 Cell 0 and 5 4 3 1 Cell 1 nodes interleaved 6 4 5 6 7 2 5 6 Cell 1 parent 8 6 7 7 9 8 7 3 Permutations: ions, synapses, warp areas, point processes.. 9 8 0 Memory addresses

A B C e-types me-types me-combinations cAC cSTUT dNAC cNAC bAC Spike Exchange L1DAC (continuous Accommodating) (continuous Stuttering) L1NGC-DA L1NGC-SA L1HAC L1LAC L1SAC L23PC L23MC 37% 31% 5% L23BTC L23DBC bAC bSTUT L23 NBC L23BP (burst Accommodating) (burst Stuttering) L23NGC L23LBC L23NBC L23SBC L23ChC L4PC L4SP L4SS L4MC cNAC dSTUT • Electrical diversity: 11 e-types; 207 me-types L4BTC (continuous Non-accommodating) (delayed Stuttering) L4DBC L4BP L4NGC m-type L4LBC L4NBC L4SBC L4ChC L5TTPC1 • Number of connections increases exponentially cIR bNAC L5TTPC2 L5UTPC (continuous Irregular) (burst Non-accommodating) L5STPC L5MC L5BTC L5DBC L5BP L5NGC • Different types of events L5LBC L5NBC dNAC bIR L5SBC L5ChC (burst Irregular) (delayed Non-accommodating) L6TPCL1 L6TPCL4 L6UTPC L6IPC L6BPC L6MC L6BTC L6DBC 25% 1.5% 1.5% L6BP cAD L6NGC L6LBC (continuous Adapting) cAC bNAC cIR Buffering Mechanism NetCon List

CoreNeuron Compute engine of NEURON simulator Being developed for large scale simulations (28 racks BG-Q) Ion Channels: ~ 85% time Linear Algebra: ~ 5-7% Spike Exchange: 7-10%

Timeline GPU current copy initialize solve state threshold CPU load MPI queue queue dt setup mindelay

Toolchain

Simple model… Soma hh pas pas Compartment pas pas pas

Performance Kernels with cell level parallelism, low occupancy! Real World Models • K20x vs 8-core Xeon • Cray (OpenACC) • Cuda 7 Varying # : Rings, Cells, Model A Model B Model E Model C Model D • No “hand” tuning yet Branches, Compartments • Optimized CPU code with vectorization larger cells 1024 65536 3072 Ion channels are 4-8x faster in all models!

Performance 8-core Xeon / 8 MPIs • BG-Q Node / 32-64 threads • Xeon Phi 61 core @ 1.23 GHz, • Varying # : Rings, Cells, 180-240 threads Branches, Compartments K20X GPU • 4096 393216 65536 3072 Optimized Xeon/MIC code with • vectorization (XLC issue)

Cell Interleaving and Exposing Parallelism Heterogeneous Homogenous Ideal How much parallelism? How much imbalance? ill - suited

Morphological diversity challenge “Morphology Aware Scheduling of Kernels using Isomorphic Subtrees”

October 8, 2015 ª 2015 Elsevier Inc. http://dx.doi.org/10.1016/j.cell.2015.09.029 Resource Reconstruction and Simulation of Neocortical Microcircuitry Henry Markram, 1,2,19, * Eilif Muller, 1,19 Srikanth Ramaswamy, 1,19 Michael W. Reimann, 1,19 Marwan Abdellah, 1 Carlos Aguado Sanchez, 1 Anastasia Ailamaki, 16 Lidia Alonso-Nanclares, 6,7 Nicolas Antille, 1 Selim Arsever, 1 Guy Antoine Atenekeng Kahou, 1 Thomas K. Berger, 2 Ahmet Bilgili, 1 Nenad Buncic, 1 Athanassia Chalimourda, 1 Giuseppe Chindemi, 1 Jean-Denis Courcol, 1 Fabien Delalondre, 1 Vincent Delattre, 2 Shaul Druckmann, 4,5 Raphael Dumusc, 1 James Dynes, 1 Stefan Eilemann, 1 Eyal Gal, 4 Michael Emiel Gevaert, 1 Jean-Pierre Ghobril, 2 Albert Gidon, 3 Joe W. Graham, 1 Anirudh Gupta, 2 Valentin Haenel, 1 Etay Hay, 3,4 Thomas Heinis, 1,16,17 Juan B. Hernando, 8 Michael Hines, 12 Lida Kanari, 1 Daniel Keller, 1 John Kenyon, 1 Georges Khazen, 1 Yihwa Kim, 1 James G. King, 1 Zoltan Kisvarday, 13 Pramod Kumbhar, 1 Se ´ bastien Lasserre, 1,15 Jean-Vincent Le Be ´ , 2 Bruno R.C. Magalha ˜ es, 1 ´ rez, 6,7 Julie Meystre, 2 Benjamin Roy Morrice, 1 Jeffrey Muller, 1 Alberto Mun Angel Mercha ´ n-Pe ˜ oz-Ce ´ spedes, 6,7 Shruti Muralidhar, 2 Keerthan Muthurasa, 1 Daniel Nachbaur, 1 Taylor H. Newton, 1 Max Nolte, 1 Aleksandr Ovcharenko, 1 Juan Palacios, 1 Luis Pastor, 9 Rodrigo Perin, 2 Rajnish Ranjan, 1,2 Imad Riachi, 1 Jose ´ -Rodrigo Rodrı ´guez, 6,7 Juan Luis Riquelme, 1 Christian Ro ¨ ssert, 1 Konstantinos Sfyrakis, 1 Ying Shi, 1,2 Julian C. Shillcock, 1 Gilad Silberberg, 18 Ricardo Silva, 1 Farhan Tauheed, 1,16 Martin Telefont, 1 Maria Toledo-Rodriguez, 14 Thomas Tra ¨ nkler, 1 Werner Van Geit, 1 ´az, 1 Richard Walker, 1 Yun Wang, 10,11 Stefano M. Zaninetta, 1 Javier DeFelipe, 6,7,20 Sean L. Hill, 1,20 Jafet Villafranca Dı Idan Segev, 3,4,20 and Felix Schu ¨ rmann 1,20 ´ cole polytechnique fe 1 Blue Brain Project, E ´ de ´ rale de Lausanne (EPFL) Biotech Campus, 1202 Geneva, Switzerland 2 Laboratory of Neural Microcircuitry, Brain Mind Institute, EPFL, 1015 Lausanne, Switzerland 3 Department of Neurobiology, Alexander Silberman Institute of Life Sciences, The Hebrew University of Jerusalem, Jerusalem 91904, Israel 4 The Edmond and Lily Safra Center for Brain Sciences, The Hebrew University of Jerusalem, Jerusalem 91904, Israel 5 Janelia Farm Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA 6 Laboratorio Cajal de Circuitos Corticales, Centro de Tecnologı ´a Biome ´ dica, Universidad Polite ´ cnica de Madrid, 28223 Madrid, Spain 7 Instituto Cajal (CSIC) and CIBERNED, 28002 Madrid, Spain 8 CeSViMa, Centro de Supercomputacio ´ n y Visualizacio ´ n de Madrid, Universidad Polite ´ cnica de Madrid, 28223 Madrid, Spain 9 Modeling and Virtual Reality Group, Universidad Rey Juan Carlos, 28933 Mo ´ stoles, Madrid, Spain 10 Key Laboratory of Visual Science and National Ministry of Health, School of Optometry and Opthalmology, Wenzhou Medical College, Wenzhou 325003, China 11 Caritas St. Elizabeth’s Medical Center, Genesys Research Institute, Tufts University, Boston, MA 02111, USA 12 Department of Neurobiology, Yale University, New Haven, CT 06510 USA 13 MTA-Debreceni Egyetem, Neuroscience Research Group, 4032 Debrecen, Hungary 14 School of Life Sciences, University of Nottingham, Nottingham NG7 2UH, United Kingdom 15 Laboratoire d’informatique et de visualisation, EPFL, 1015 Lausanne, Switzerland 16 Data-Intensive Applications and Systems Lab, EPFL, 1015 Lausanne, Switzerland 17 Imperial College London, London SW7 2AZ, UK 18 Department of Neuroscience, Karolinska Institutet, Stockholm 17177, Sweden 19 Co-first author 20 Co-senior author *Correspondence: henry.markram@epfl.ch http://dx.doi.org/10.1016/j.cell.2015.09.029

THANK YOU!

Explanatory Graphic Sources source 1: dgallery.s3.amazonaws.com • source 2: clipartbest.com/cliparts, scaryforkids.com, geniusawakening.com • source 3: developer.humanbrainproject.eu • source 4: nature.com • source 5: deviantart.net, squarespace.com • source 6: lcn.epfl.ch • source 7: nature.com • source 8: genesis-sim.org •

Backup low occupancy! Model A Model B Model C Model E Model D larger cells 1024 65536 2048