HPC and Deep Learning –

New Developments in
Neural Network
Computing

Dr. Ren Wu
renw@novumind.com

© Dr. Ren Wu
Mark’s New Year Resolution

© Dr. Ren Wu
Mark’s New Year Resolution

© Dr. Ren Wu
Dr. Ren Wu
•  Founder and CEO, NovuMind Ltd.
•  Distinguished Scientist, Baidu
•  HSA Chief Software Architect, AMD
•  PI, HP Labs CUDA Research Center
•  World Computer Xiangqi Champion

•  AI expert
•  HPC expert
•  Big Data Analytics expert
•  Heterogeneous Computing expert
•  Computational scientist
© Dr. Ren Wu
© Dr. Ren Wu
Insights and Inspirations

Big
Data

Big
Compute

(Gu Ban, 32-92 AD) (Sun Tzu, 544-496 BC)

Meaning the more you see the More calculations win, few
more you know calculation lose © Dr. Ren Wu
 Human:

game was drawn, 1570

Computer:

no, Red wins, Red captures in 95

moves, 1997

Human:

"In certain kinds of positions, the computer sees so deeply that it plays like God.”
-- Gary Kasparov
© Dr. Ren Wu
 Human:

game was drawn, 1570

1 billion positions, Pentium Pro 200, 128MB, 92 hours!

Computer:

no, Red wins, Red captures in 95

Big data + Big compute = moves, 1997 knowledge
Complete

Human:

"In certain kinds of positions, the computer sees so deeply that it plays like God.”
-- Gary Kasparov
© Dr. Ren Wu
Eighteen Years Ago -
05/11/1997

Deep Blue 3.5:2.5 Gary Kasparov

© Dr. Ren Wu
Deep Blue

A classic example of application-specific system design comprised

of an IBM supercomputer with 480 custom-made VLSI chess chips, running
massively parallel search algorithm with highly optimized implementation.
Heuristic Search + Big compute > World champion
© Dr. Ren Wu
Computer Chess and Moore’s Law

© Dr. Ren Wu
Google’s AlphaGo Program
The brute force-style approach "has led to superhuman
performance in chess, checkers and Othello, but it was
believed to be intractable in Go due to the complexity
of the game," AlphaGo’s creators wrote in a paper
published Wednesday in the journal Nature.

1920 CPUs
280 GPUs
Months training time!

© Dr. Ren Wu
Deep Learning and AI
Rise of the machines
“We deepened our investment in
advanced technologies like Deep
Learning, which is already yielding near
term enhancements in customer ROI
and is expected to drive
transformational change over the
longer term.”
– Robin Li, Baidu CEO
© Dr. Ren Wu
Deep Learning
What have changed since 1980s

Big data

Big compute
© Dr. Ren Wu
Heterogeneous Computing

2000 world #1
ASCI White (IBM RS/6000SP)
6MW power, 106 tons
1993 world #1
12.3 TFlops
Think Machine CM5/1024
131 GFlops
2013
Two MacPro workstation
2013 (dual AMD GPUs each)
Samsung Note 3 smartphone 14 TFlops
(Qualcomm SnapDragon 800)
129 Gflops © Dr. Ren Wu
Project Minwa ( )
•  Minerva + Athena +
•  Custom designed supercomputer
•  100x more powerful than previous ones
•  hardware/software co-design, highly optimized
•  Very large scale
•  1+ billion parameters
•  10+ of billion training images
•  Best in class!

© Dr. Ren Wu
History is repeating itself !

© Dr. Ren Wu
Deep Learning: Two Steps
Supercomputers used for
training

And then deploy the trained

models everywhere!

Datacenters Tablets, smartphones Wearable devices IoTs

© Dr. Ren Wu
Cloud computing? !
What is missing?
© Dr. Ren Wu
Cloud Computing: What’s Missing?

Bandwidth?
Latency?
and
Power
consumption?

*Artem Vasilyev: CNN optimizations for embedded systems and FFT

Moving data around is expensive, very expensive! © Dr. Ren Wu

Cloud Computing: What’s Missing?

How about
privacy?

© Dr. Ren Wu
Cloud+Edge computing !

© Dr. Ren Wu
Heterogeneous Computing

“Human mind and brain is not a single general-purpose processor

but a collection of highly specialized components, each solving a
different, specific problem and yet collectively making up who we
are as human beings and thinkers. “ - Prof. Nancy Kanwisher
© Dr. Ren Wu
What’s Next?

* Mark Horowitz
Dedicated Hardware + Heterogeneous Computing
© Dr. Ren Wu
OpenCL Ecosystem
Implementers Working Group Members
Desktop/Mobile/FPGA Apps/Tools/Tests/Courseware

Single Source C++ Programming

Core API and Language Specs

Portable Kernel Intermediate Language

© Dr. Ren Wu
Big data and AI: Next Waves
Big data + Deep learning + High performance computing =
Intelligence Omnipotent

Big data + Deep learning + Heterogeneous computing =

Success Omnipresent

2
I oT - Intelligent Internet of Things
© Dr. Ren Wu
Intelligent Internet of Things
Everything Everything
Connected Intelligent

2
IPowered
oTby
NovuMind

Big data era AI era

© Dr. Ren Wu
Welcome to NovuPhalanx!

jobs@novumind.com
© Dr. Ren Wu
We are hiring …
•  High performance and heterogeneous computing
•  Artificial intelligence and deep learning
•  Computer vision, Speech recognition, NLP
•  Image processing, computational photography, camera system
•  Embedded system, smart hardware, SoC, FPGA, DSP, ASIC
•  Performance programming, CUDA/OpenCL/NEON etc.

•  Office and/or Executive Assistant

jobs@novumind.com
•  Finance/Marketing/PR/BD
© Dr. Ren Wu
Thank you!
© Dr. Ren Wu