Professional Documents
Culture Documents
ISCA Tutorial
1
2 Speakers and Contributors
2
Outline
3
Participant Takeaways
Understand the key design considerations for
DNNs
4
Resources
5
Background of
Deep Neural Networks
6
Artificial Intelligence
Artificial Intelligence
7
AI and Machine Learning
Artificial Intelligence
Machine Learning
8
Brain-Inspired Machine Learning
Artificial Intelligence
Machine Learning
Brain-Inspired
9
How Does the Brain Work?
Artificial Intelligence
Machine Learning
Brain-Inspired
Spiking
11
Spiking Architecture
Brain-inspired
Integrate and fire
Example: IBM TrueNorth
Artificial Intelligence
Machine Learning
Brain-Inspired
Spiking Neural
Networks
13
Neural Networks: Weighted Sum
Artificial Intelligence
Machine Learning
Brain-Inspired
Spiking Neural
Networks
Deep
Learning
16
What is Deep Learning?
Volvo
Image XC90
350M images
uploaded per
day
2.5 Petabytes
of customer
data hourly
300 hours of
video uploaded
every minute
18
ImageNet Challenge
19
ImageNet: Image Classification Task
15
10
0
2010 2011 2012 2013 2014 2015 Human
21
Established Applications
Image
o Classification: image to object class
o Recognition: same as classification (except for faces)
o Detection: assigning bounding boxes to objects
o Segmentation: assigning object class to every pixel
Speech & Language
o Speech Recognition: audio to text
o Translation
o Natural Language Processing: text to meaning
o Audio Generation: text to audio
Games
22
Deep Learning on Games
Google DeepMind AlphaGo
23
Emerging Applications
Medical (Cancer Detection, Pre-Natal)
http://www.nextplatform.com/2016/09/14/next-wave-deep-learning-applications/ 24
Deep Learning for Self-driving Cars
25
Opportunities
26
Overview of
Deep Neural Networks
27
DNN Timeline
1940s: Neural networks were proposed
1960s: Deep neural networks were proposed
1989: Neural network for recognizing digits (LeNet)
1990s: Hardware for shallow neural nets
Example: Intel ETANN (1992)
2011: Breakthrough DNN-based speech recognition
Microsoft real-time speech translation
2012: DNNs for vision supplanting traditional ML
AlexNet for image classification
2014+: Rise of DNN accelerator research
Examples: Neuflow, DianNao, etc.
28
Publications at Architecture Conferences
29
So Many Neural Networks!
http://www.asimovinstitute.org/neural-network-zoo/ 30
DNN Terminology 101
Neurons
3
Y j = activation Wij Xi
W11 Y1 i=1
X1
Y2
X2
Y3
X3
W34 Y4
3
Y j = activation Wij Xi
W11 Y1 i=1
X1
Y2
X2
Y3
X3
W34 Y4
L2 Output
Activations
Sparsely-Connected
Fully-Connected NN
feed forward, a.k.a. multilayer perceptron (MLP)
Convolutional NN (CNN)
feed forward, sparsely-connected w/ weight sharing
Recurrent NN (RNN)
feedback
39
Inference vs. Training
40
Deep Convolutional Neural Networks
Low-Level High-Level
CONV CONV FC
Layer
Features
Layer
Features Layer Classes
1 3 Layers
41
Deep Convolutional Neural Networks
Low-Level High-Level
CONV CONV FC
Layer
Features
Layer
Features Layer Classes
Convolution Activation
42
Deep Convolutional Neural Networks
Low-Level High-Level
CONV CONV FC
Layer
Features
Layer
Features Layer Classes
Fully Activation
Connected
43
Deep Convolutional Neural Networks
Normalization Pooling
44
Deep Convolutional Neural Networks
High-Level
CONV NORM POOL CONV FC
Layer Layer Layer Layer
Features Layer Classes
45
Convolution (CONV) Layer
a plane of input activations
a.k.a. input feature map (fmap)
filter (weights)
H
R
S W
46
Convolution (CONV) Layer
input fmap
filter (weights)
H
R
S W
Element-wise
Multiplication
47
Convolution (CONV) Layer
S W F
Element-wise Partial Sum (psum)
Multiplication Accumulation
48
Convolution (CONV) Layer
S W F
Sliding Window Processing
49
Convolution (CONV) Layer
input fmap
filter C
output fmap
C
H
R E
S W F
Many Input Channels (C)
50
Convolution (CONV) Layer
C
H M
R E
1
S W F
Many
Output Channels (M)
C
R
M
S
51
Convolution (CONV) Layer
Many
Input fmaps (N) Many
C
Output fmaps (N)
filters
M
C
H
R E
1 1
S W F
C C
R E
H
S
N
N F
W
52
CNN Decoder Ring
53
CONV Layer Tensor Computation
Output fmaps (O) Input fmaps (I)
Biases (B) Filter weights (W)
54
CONV Layer Implementation
Nave 7-layer for-loop implementation:
55
Traditional Activation Functions
0 0
-1 -1
-1 0 1 -1 0 1
y=1/(1+e-x) y=(ex-e-x)/(ex+e-x)
0 0 0
-1 -1 -1
-1 0 1 -1 0 1 -1 0 1
x,
x0
y=max(0,x) y=
y=max(x,x) (ex-1),
x<0
= small const. (e.g. 0.1)
CHW
M = M
58
FC Layer from CONV Layer POV
input fmaps
C
output fmaps
filters
M
C
H
H 1
1 1
W W 1
C C
H 1
H
W
N
N 1
W
59
Pooling (POOL) Layer
Reduce resolution of each channel independently
Overlapping or non-overlapping depending on stride
61
Normalization (NORM) Layer
Batch Normalization (BN)
Normalize activations towards mean=0 and std.
dev.=1 based on the statistics of the training dataset
Convolution Activation
CONV
Layer BN
63
Normalization (NORM) Layer
Local Response Normalization (LRN)
Tries to mimic the inhibition scheme in the brain
Now deprecated!
Image Source: Caffe Tutorial 64
Relevant Components for Tutorial
65