CoreNeuron : Morphologically Detailed Neuron Simulations

Building, Simulating and Optimizing Large Neuron Networks on GPUs

Pramod Kumbhar, Michael Hines

7th April 2016, GTC & Blue Brain HPC Team
Understanding Brain..
Brain/MINDS
2014
Human Brain
Project
BRAIN 2013
Initiative
IBM: Cat’s brain scale 2013
simulation on BG-P
2009
Izhikevic: brain scale
simulation on cluster
2007

1 million cells 1.5 billion cells

Better understanding of brain

See source1
Brain Simulations

See source3 See source4

Point Neurons Morphologically Molecular Level

Detailed Neurons
Blue Brain Project, EPFL
Reverse Engineering Brain (10000 feet view)

See source6

See source5
Reconstruction Workflow

• 15+ years of Experiments

• 14,000 Neurons recorded
and labeled
• 2,052 Classified neurons
• 1,009 Reconstructed
neurons
• 2,000+ Ion channel
recordings
• 4,000+ Electrical recordings
of single neurons
• 5,000+ Synaptic recordings
of pairs of neurons

Markram et al. 2015, Cell

Modeling Neuron

See source7
HH, 1952

See source8
Ion Channels
• Prominent components of nervous system

• More than 300 ion channels

Every channel is a compute kernel, no

single hotspot!
Biologist view:
compartment model
NMODL: Source to Source Compiler

Lexical Analyzer
tokens

Syntax Analyzer
parse tree
NMODL
Semantic Analyzer
parse tree

~ Intermediate Representation

backends

C OpenACC Cuda Cyme: SIMD

DSL

Portable Performance
OpenACC Kernels

User defined DS: Major challenge for many application

auto-generated kernel
wrap OpenACC and vectorisation hints related pragmas

AoS/SoA, Vectorisation, Memory Coalescing etc.. OpenACC API’s to copy the complex data structure
node_index
parent_index GPU: Cell level Parallelism (some kernels) Step 3

Step 2

Step 1
0 -1 5 -1 bksub: for(i = x; i < nodes; i++) {
step 0

rhs[i] -= b[i] * rhs[ parent[i] ]

1 0 6 5 rhs[i] /= d[i]

}
node : 1 2 3 4
thread 0 parent[i]: 0 1 2 3
2 1 7 6

node : 6 7 8 9
3 2 8 7 thread 1
parent[i]: 5 6 7 8
Stride Depth of Tree
4 3 9 8

node : 156 157 158 159

thread 31
parent[i]: 155 156 157 158
warp

0
Memory addresses
 Cell Interleaving
parent_index Nodes Parents
node_index

0 -1 Roots parent
all cells root
0 -1
1 -1
1 -1 2 0
2 0 3 1
Cell 0 parent
3 5 4 2
Cell 0 and
4 1 Cell 1 nodes 5 3
interleaved
5 6 6 4
6 2 7 5
Cell 1 parent
7 7 8 6
8 3 9 7
Permutations: ions, synapses, warp
areas, point processes.. 9 8

0
Memory addresses
 A e-types B me-types C
cAC cSTUT dNAC cNAC bAC

Spike Exchange
(continuous Accommodating) (continuous Stuttering)
G
L1N G
L1N

37% 31% 5%
bAC bSTUT L23 NBC
L
L
(burst Accommodating) (burst Stuttering)
L2
L
L
L
L

cNAC dSTUT
• Electrical diversity: 11 e-types; 207 me-types (continuous Non-accommodating) (delayed Stuttering)

m-type
L

• Number of connections increases exponentially bNAC cIR L5

(burst Non-accommodating) (continuous Irregular) L5
L5
L

Different types of events

L

• dNAC
(delayed Non-accommodating)
bIR
(burst Irregular)
L6
L6
L6

25% 1.5% 1.5%

cAD L
(continuous Adapting)
cAC bNAC cIR
20 mV
200 ms

Figure 4. Markram et al.

NetCon List Buffering Mechanism

CoreNeuron

Compute engine of NEURON simulator

Being developed for large scale simulations (28 racks BG-Q)

Ion Channels: ~ 85% time

Linear Algebra: ~ 5-7%

Spike Exchange: 7-10%

 Timeline

GPU
copy initialize current solve state threshold

CPU
load queue MPI queue

setup dt

mindelay
Toolchain
Simple model…

hh Soma
pas

pas

Compartment
pas

pas

pas
 Performance Kernels with cell level
parallelism, low occupancy!

Real World Models

• K20x vs 8-core Xeon

• Cray (OpenACC)
Varying # : Rings, Cells, • Cuda 7
Model A Model B Model C Model D Model E
Branches, Compartments • No “hand” tuning yet
• Optimized CPU code with
vectorization
larger cells

65536 3072 1024

Ion channels are 4-8x faster in all models!

 Performance

• 8-core Xeon / 8 MPIs

• BG-Q Node / 32-64 threads
• Xeon Phi 61 core @ 1.23 GHz,
Varying # : Rings, Cells,
Branches, Compartments
180-240 threads
• K20X GPU
393216 65536 4096 3072
• Optimized Xeon/MIC code with
vectorization (XLC issue)
 Cell Interleaving and Exposing Parallelism

Heterogeneous
Homogenous

Ideal
How much parallelism? How much imbalance?

ill - suited
Morphological diversity challenge

“Morphology Aware Scheduling of Kernels using Isomorphic Subtrees”

d The Blue Brain Project digitally reconstructs and simulates a
part of neocortex

d Interdependencies allow dense in silico reconstruction from

sparse experimental data

d Simulations reproduce in vitro and in vivo experiments

without parameter tuning

d The neocortex reconfigures to support diverse information

processing strategies

Markram et al., 2015, Cell 163, 456–492

October 8, 2015 ª2015 Elsevier Inc.
http://dx.doi.org/10.1016/j.cell.2015.09.029

Resource

Reconstruction and Simulation

of Neocortical Microcircuitry
Henry Markram,1,2,19,* Eilif Muller,1,19 Srikanth Ramaswamy,1,19 Michael W. Reimann,1,19 Marwan Abdellah,1
Carlos Aguado Sanchez,1 Anastasia Ailamaki,16 Lidia Alonso-Nanclares,6,7 Nicolas Antille,1 Selim Arsever,1
Guy Antoine Atenekeng Kahou,1 Thomas K. Berger,2 Ahmet Bilgili,1 Nenad Buncic,1 Athanassia Chalimourda,1
Giuseppe Chindemi,1 Jean-Denis Courcol,1 Fabien Delalondre,1 Vincent Delattre,2 Shaul Druckmann,4,5
Raphael Dumusc,1 James Dynes,1 Stefan Eilemann,1 Eyal Gal,4 Michael Emiel Gevaert,1 Jean-Pierre Ghobril,2
Albert Gidon,3 Joe W. Graham,1 Anirudh Gupta,2 Valentin Haenel,1 Etay Hay,3,4 Thomas Heinis,1,16,17 Juan B. Hernando,8
Michael Hines,12 Lida Kanari,1 Daniel Keller,1 John Kenyon,1 Georges Khazen,1 Yihwa Kim,1 James G. King,1
Zoltan Kisvarday,13 Pramod Kumbhar,1 Sébastien Lasserre,1,15 Jean-Vincent Le Bé,2 Bruno R.C. Magalhães,1
Angel Merchán-Pérez,6,7 Julie Meystre,2 Benjamin Roy Morrice,1 Jeffrey Muller,1 Alberto Muñoz-Céspedes,6,7
Shruti Muralidhar,2 Keerthan Muthurasa,1 Daniel Nachbaur,1 Taylor H. Newton,1 Max Nolte,1 Aleksandr Ovcharenko,1
Juan Palacios,1 Luis Pastor,9 Rodrigo Perin,2 Rajnish Ranjan,1,2 Imad Riachi,1 José-Rodrigo Rodrı́guez,6,7
Juan Luis Riquelme,1 Christian Rössert,1 Konstantinos Sfyrakis,1 Ying Shi,1,2 Julian C. Shillcock,1 Gilad Silberberg,18
Ricardo Silva,1 Farhan Tauheed,1,16 Martin Telefont,1 Maria Toledo-Rodriguez,14 Thomas Tränkler,1 Werner Van Geit,1
Jafet Villafranca Dı́az,1 Richard Walker,1 Yun Wang,10,11 Stefano M. Zaninetta,1 Javier DeFelipe,6,7,20 Sean L. Hill,1,20
Idan Segev,3,4,20 and Felix Schürmann1,20
1Blue Brain Project, École polytechnique fédérale de Lausanne (EPFL) Biotech Campus, 1202 Geneva, Switzerland
2Laboratory of Neural Microcircuitry, Brain Mind Institute, EPFL, 1015 Lausanne, Switzerland
3Department of Neurobiology, Alexander Silberman Institute of Life Sciences, The Hebrew University of Jerusalem, Jerusalem 91904, Israel
4The Edmond and Lily Safra Center for Brain Sciences, The Hebrew University of Jerusalem, Jerusalem 91904, Israel
5Janelia Farm Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA
6Laboratorio Cajal de Circuitos Corticales, Centro de Tecnologı́a Biomédica, Universidad Politécnica de Madrid, 28223 Madrid, Spain
7Instituto Cajal (CSIC) and CIBERNED, 28002 Madrid, Spain
8CeSViMa, Centro de Supercomputación y Visualización de Madrid, Universidad Politécnica de Madrid, 28223 Madrid, Spain
9Modeling and Virtual Reality Group, Universidad Rey Juan Carlos, 28933 Móstoles, Madrid, Spain
10Key Laboratory of Visual Science and National Ministry of Health, School of Optometry and Opthalmology, Wenzhou Medical College,

Wenzhou 325003, China

11Caritas St. Elizabeth’s Medical Center, Genesys Research Institute, Tufts University, Boston, MA 02111, USA
12Department of Neurobiology, Yale University, New Haven, CT 06510 USA
13MTA-Debreceni Egyetem, Neuroscience Research Group, 4032 Debrecen, Hungary
14School of Life Sciences, University of Nottingham, Nottingham NG7 2UH, United Kingdom
15Laboratoire d’informatique et de visualisation, EPFL, 1015 Lausanne, Switzerland
16Data-Intensive Applications and Systems Lab, EPFL, 1015 Lausanne, Switzerland
17Imperial College London, London SW7 2AZ, UK
18Department of Neuroscience, Karolinska Institutet, Stockholm 17177, Sweden
19Co-first author
20Co-senior author

*Correspondence: henry.markram@epfl.ch
http://dx.doi.org/10.1016/j.cell.2015.09.029

SUMMARY their overlapping arbors form !8 million connections

We present a first-draft digital reconstruction of the
microcircuitry of somatosensory cortex of juvenile
rat. The reconstruction uses cellular and synaptic
organizing principles to algorithmically reconstruct
detailed anatomy and physiology from sparse experi-
mental data. An objective anatomical method defines
a neocortical volume of 0.29 ± 0.01 mm3 containing
!31,000 neurons, and patch-clamp studies identify
55 layer-specific morphological and 207 morpho-
electrical neuron subtypes. When digitally recon-
structed neurons are positioned in the volume and
synapse formation is restricted to biological bouton
densities and numbers of synapses per connection,
with !37 million synapses. Simulations reproduce
an array of in vitro and in vivo experiments without
parameter tuning. Additionally, we find a spectrum
of network states with a sharp transition from synchro-
nous to asynchronous activity, modulated by physio-
logical mechanisms. The spectrum of network states,
dynamically reconfigured around this transition, sup-
ports diverse information processing strategies.
INTRODUCTION
Since Santiago Ramón y Cajal’s seminal work on the neocortex
(DeFelipe and Jones, 1988; Ramón y Cajal, 1909, 1911), a vast
number of studies have attempted to unravel its multiple levels

456 Cell 163, 456–492, October 8, 2015 ª2015 Elsevier Inc.

THANK YOU!
Explanatory Graphic Sources

• source 1: dgallery.s3.amazonaws.com
• source 2: clipartbest.com/cliparts, scaryforkids.com, geniusawakening.com
• source 3: developer.humanbrainproject.eu
• source 4: nature.com
• source 5: deviantart.net, squarespace.com
• source 6: lcn.epfl.ch
• source 7: nature.com
• source 8: genesis-sim.org
Backup
low occupancy!

Model A Model B Model C Model D Model E

larger cells

65536 2048 1024