Week 4: Convolutional Networks

Unit 5: Applications of CNNs

Applications of CNNs
What we covered in the last unit

Accelerating Deep CNN Training

Computational considerations
Batch normalization
Transfer learning
Residual networks

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 2

Applications of CNNs
Overview

Content:
Object detection
Two-stage detectors
One-stage detectors
Segmentation

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 3

Applications of CNNs
Overview

Content:
Object detection
Two-stage detectors
One-stage detectors
Segmentation

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 4

Applications of CNNs
Object detection: example

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 5

Applications of CNNs
Object detection Nave approach

1. Specify sliding window size

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 6

Applications of CNNs
Object detection Nave approach

1. Specify sliding window size

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 7

Applications of CNNs
Object detection Nave approach

1. Specify sliding window size

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 8

Applications of CNNs
Object detection Nave approach

1. Specify sliding window size

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 9

Applications of CNNs
Object detection Nave approach

1. Specify sliding window size

2. Run pre-trained object-classification CNN on each window (softmax output)

CNN

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 10

Applications of CNNs
Object detection Nave approach

1. Specify sliding window size

2. Run pre-trained object-classification CNN on each window (softmax output)
3. Associate current sliding window with object class if probability is sufficiently high (e.g. > 0.5)

CNN

Classification
2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 11
Applications of CNNs
Object detection Nave approach

1. Specify sliding window size

2. Run pre-trained object-classification CNN on each window (softmax output)
3. Associate current sliding window with object class if probability is sufficiently high (e.g. > 0.5)

CNN

Classification
2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 12
Applications of CNNs
Object detection Nave approach

1. Specify sliding window size

2. Run pre-trained object-classification CNN on each window (softmax output)
3. Associate current sliding window with object class if probability is sufficiently high (e.g. > 0.5)

Problem:
Intractable even
for small images

CNN

Classification
2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 13
Applications of CNNs
Two-stage detectors: key idea

Use proposal algorithm (e.g. selective search)

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 14

Applications of CNNs
Two-stage detectors: key idea

Use proposal algorithm (e.g. selective search)

Boxes are proposed based on e.g.:
Texture
Color
Intensity

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 15

Applications of CNNs
Two-stage detectors: key idea

Use proposal algorithm (e.g. selective search)

Boxes are proposed based on e.g.:
Texture
Color
Intensity

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 16

Applications of CNNs
Two-stage detectors: key idea

Use proposal algorithm (e.g. selective search)

Boxes are proposed based on e.g.:
Texture
Color
Intensity

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 17

Applications of CNNs
Two-stage detectors: key idea

Use proposal algorithm (e.g. selective search)

Boxes are proposed based on e.g.:
Texture
Color
Intensity
Evaluate only on proposals

CNN

Classification
2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 18
Applications of CNNs
Two-stage detectors: R-CNN [Girshick, Donahue, Darrell, Malik, 2013]

Proposal algorithm Warping Features

Object
SVM classification
CNN Improved
Lin. Reg bounding boxes

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 19

Applications of CNNs
Two-stage detectors: R-CNN [Girshick, Donahue, Darrell, Malik, 2013]

Proposal algorithm Warping Features

Object
SVM classification
CNN Improved
Lin. Reg bounding boxes

Advantages:
Several orders of magnitude faster
Allows improvement of bounding boxes

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 20

Applications of CNNs
Two-stage detectors: R-CNN [Girshick, Donahue, Darrell, Malik, 2013]

Proposal algorithm Warping Features

Object
SVM classification
CNN Improved
Lin. Reg bounding boxes

Advantages: Disadvantages:
Several orders of magnitude faster CNN cannot adapt to changes in SVM/linear
regression
Allows improvement of bounding boxes
Implication: Degrades power of model
CNN often evaluates boxes that have large
overlaps
Implication: Suboptimal performance
2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 21
Applications of CNNs
Two-stage detectors: Fast R-CNN [Girshick 2015]

Key Idea:
Softmax output replaces SVM
Combine regression and classification into one neural net (end-to-end)

Fully connected

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 22

Applications of CNNs
Two-stage detectors: Fast R-CNN [Girshick 2015]

Key Idea:
Softmax output replaces SVM
Combine regression and classification into one neural net (end-to-end)
Evaluate CNN only one time/image (share results among suggestions)
Region of Interest Pooling:

(same padding) Feature map Max Pooling Fully connected

Region of Interest Pooling

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 23
Applications of CNNs
Two-stage detectors: Faster R-CNN [Ren, He, Girshick, Sun, 2015]

Key Problem: Region proposal becomes bottleneck

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 24

Applications of CNNs
Two-stage detectors: Faster R-CNN [Ren, He, Girshick, Sun, 2015]

Key Problem: Region proposal becomes bottleneck

Solution: Add so-called Region Proposal Network

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 25

Applications of CNNs
Two-stage detectors: Faster R-CNN [Ren, He, Girshick, Sun, 2015]

Key Problem: Region proposal becomes bottleneck

Solution: Add so-called Region Proposal Network

Feature map

CNN

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 26

Applications of CNNs
Two-stage detectors: Faster R-CNN [Ren, He, Girshick, Sun, 2015]

Key Problem: Region proposal becomes bottleneck

Solution: Add so-called Region Proposal Network

Proposals

Feature map

CNN

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 27

Applications of CNNs
Two-stage detectors: Faster R-CNN [Ren, He, Girshick, Sun, 2015]

Key Problem: Region proposal becomes bottleneck

Solution: Add so-called Region Proposal Network

RoI-Pooling /
Proposals
Warping

Feature map

CNN

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 28

Applications of CNNs
Two-stage detectors: Faster R-CNN [Ren, He, Girshick, Sun, 2015]

Key Problem: Region proposal becomes bottleneck

Solution: Add so-called Region Proposal Network
Classifier / Regressor

RoI-Pooling /
Proposals
Warping

Feature map

CNN

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 29

Applications of CNNs
Two-stage detectors: comparison [Li, Karpathy, Johnson]

VOC 2017

Faster R-CNN

Fast R-CNN

R-CNN

0 10 20 30 40 50 60
Test time/image [s]

Faster R-CNN

Fast R-CNN

R-CNN

65,4 65,6 65,8 66 66,2 66,4 66,6 66,8 67

mAP
2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 30
Applications of CNNs
Overview

Content:
Object detection
Two-stage detectors
One-stage detectors
Segmentation

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 31

Applications of CNNs
Object detection: one-stage detectors

The R-CNN class detectors are accurate but very slow!

How can we make Methodology:
this faster? First stage: Generate region proposals
Second stage: Classify & refine bbox proposals

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 32

Applications of CNNs
Object detection: one-stage detectors

The R-CNN class detectors are accurate but very slow!

How can we make Methodology:
this faster? First stage: Generate region proposals
Second stage: Classify & refine bbox proposals

One-stage detectors:
Idea: Take advantage of fully convolutional networks to
CHANGE
compute bounding box regression and class probabilities
METHODOLOGY
in one pass
Implication: Architecture is considerably simpler!

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 33

Applications of CNNs
One-stage detectors e.g.: YOLO(9000) You Only Look Once [Redmon, Divvala, Girshick, Farhadi, 2015]

CNN
(Encoder)

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 34

Applications of CNNs
One-stage detectors e.g.: YOLO(9000) You Only Look Once [Redmon, Divvala, Girshick, Farhadi, 2015]

CNN Upsample
(Encoder) (Decoder)

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 35

Applications of CNNs
One-stage detectors e.g.: YOLO(9000) You Only Look Once [Redmon, Divvala, Girshick, Farhadi, 2015]

CNN Upsample
(Encoder) (Decoder)

Mask

Each pixel predicts

a class

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 36

Applications of CNNs
One-stage detectors e.g.: YOLO(9000) You Only Look Once [Redmon, Divvala, Girshick, Farhadi, 2015]

Box
coordinates

CNN Upsample Each pixel predicts

4 coordinates
(Encoder) (Decoder)

Mask

Each pixel predicts

a class

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 37

Applications of CNNs
One-stage detectors e.g.: YOLO(9000) You Only Look Once [Redmon, Divvala, Girshick, Farhadi, 2015]

Box
coordinates

CNN Upsample Each pixel predicts

(Encoder) (Decoder) 4 coordinates

Mask

Each pixel predicts

a class

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 38

Applications of CNNs
One-stage detectors e.g.: YOLO(9000) You Only Look Once [Redmon, Divvala, Girshick, Farhadi, 2015]

Box
coordinates

CNN Upsample Each pixel predicts

(Encoder) (Decoder) 4 coordinates

Mask

Each pixel predicts

a class

Also see: Single Shot Multibox Detector (SSD)

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 39
Applications of CNNs
One-stage detectors: comparison

Problems: One-stage detectors are less accurate than two-stage detectors:

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 40

Applications of CNNs
One-stage detectors: comparison

Problems: One-stage detectors are less accurate than two-stage detectors:

Number of bounding box predictions is considerably larger (c.f. R-CNNs 2k proposals vs. 100k in most
one-stage detectors)

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 41

Applications of CNNs
One-stage detectors: comparison

Problems: One-stage detectors are less accurate than two-stage detectors:

Number of bounding box predictions is considerably larger (c.f. R-CNNs 2k proposals vs. 100k in most
one-stage detectors)
This leads to more class imbalance since most of the bounding boxes belong to background

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 42

Applications of CNNs
One-stage detectors: RetinaNet [Lin, Goyal, Girshick, He, Dollar, 2017]

Problems: One-stage detectors are less accurate than two-stage detectors:

Number of bounding box predictions is considerably larger (c.f. R-CNNs 2k proposals vs. 100k in most
one-stage detectors)
This leads to more class imbalance since most of the bounding boxes belong to background

Remedy: Introduce so-called Focal Loss:

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 43

Applications of CNNs
One-stage detectors: RetinaNet [Lin, Goyal, Girshick, He, Dollar, 2017]

Problems: One-stage detectors are less accurate than two-stage detectors:

Number of bounding box predictions is considerably larger (c.f. R-CNNs 2k proposals vs. 100k in most
one-stage detectors)
This leads to more class imbalance since most of the bounding boxes belong to background

Remedy: Introduce so-called Focal Loss:

Down-weight contribution of examples that are particularly easy to learn (i.e. background patches)

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 44

Applications of CNNs
One-stage detectors: RetinaNet [Lin, Goyal, Girshick, He, Dollar, 2017]

Problems: One-stage detectors are less accurate than two-stage detectors:

Number of bounding box predictions is considerably larger (c.f. R-CNNs 2k proposals vs. 100k in most
one-stage detectors)
This leads to more class imbalance since most of the bounding boxes belong to background

Remedy: Introduce so-called Focal Loss:

Down-weight contribution of examples that are particularly easy to learn (i.e. background patches)

log , = +1
- L , =
log 1 , = 1

(1 ) log , = +1
- F , =
log 1 , = 1

For more details: RetinaNet https://arxiv.org/abs/1708.02002

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 45
Applications of CNNs
Overview

Content:
Object detection
Two-stage detectors
One-stage detectors
Segmentation

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 46

Applications of CNNs
Image segmentation: example

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 47

Applications of CNNs
Image segmentation: example

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 48

Applications of CNNs
Image segmentation: generic architecture

CNN Upsample
(Encoder) (Decoder)

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 49

Applications of CNNs
Image segmentation: generic architecture

CNN Upsample
(Encoder) (Decoder)

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 50

Applications of CNNs
Image segmentation e.g. U-Net architecture

Encoder

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 51

Applications of CNNs
Image segmentation e.g. U-Net architecture

Encoder Decoder

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 52

Applications of CNNs
Image segmentation e.g. U-Net architecture

Encoder Decoder

Upsampling: Transposed Convolutions (see Unit 3)

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 53

Applications of CNNs
Image segmentation e.g. U-Net architecture
Pixelwise
classification

Encoder Decoder

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 54

Applications of CNNs
Image segmentation e.g. U-Net architecture

Merge with hires features

Encoder Decoder

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 55

Applications of CNNs
Image segmentation e.g. U-Net architecture

Merge with hires features

Encoder Decoder

Also see: e.g. SegNet

e.g. Pyramid Scene Parsing Network
2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 56
Applications of CNNs
Image segmentation: challenges

Efficiency / Accuracy
Encoder downsampling (low resolution) versus dense prediction

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 57

Applications of CNNs
Image segmentation: challenges

Efficiency / Accuracy
Encoder downsampling (low resolution) versus dense prediction
Make use of dilated convolutions
Feature fusion

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 58

Applications of CNNs
Image segmentation: challenges

Efficiency / Accuracy
Encoder downsampling (low resolution) versus dense prediction
Make use of dilated convolutions
Feature fusion

Data availability
Ground truth: Hand-segmented pictures (very costly to acquire!)
Data augmentation is essential!

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 59

Applications of CNNs
Image segmentation: data augmentation

Data augmentation is
indispensable when there is little
training data
Use randomized on-the-fly
modifications

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 60

Applications of CNNs
Coming up next

Industry Applications of Deep Learning

Machine learning in customer service
Machine learning in banking
Medical image segmentation

2017 SAP SE or an SAP affiliate company. All rights reserved. PUBLIC 61

Thank you.
Contact information:

open@sap.com
 2017 SAP SE or an SAP affiliate company. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company.

The information contained herein may be changed without prior notice. Some software products marketed by SAP SE and its distributors contain proprietary software components
of other software vendors. National product specifications may vary.

These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP or its affiliated
companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP or SAP affiliate company products and services are those that are
set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.

In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release
any functionality mentioned therein. This document, or any related presentation, and SAP SEs or its affiliated companies strategy and possible future developments, products,
and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The
information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various
risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements,
and they should not be relied upon in making purchasing decisions.

SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company)
in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies.
See http://global.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices.