Fusion-based Multiclass AdaBoost for classification of pose

A Fusion-based Multiclass AdaBoost for Classifying Object Poses

Using Visual and Thermal IR Images

Presented by

Mohamed Hashim Changrampadi Communication Engineering Chalmers University of Technology

Supervisor and Examiner

Prof. Irene Gu Signal and System Chalmers University of Technology

Masters Thesis Presentation Sep 29-2011

Page 1

Fusion-based Multiclass AdaBoost for classification of pose

Outline
Challenges Classification model Equipment Used Fusion-based Multiclass AdaBoost Experiment and Results Conclusion Demo
Page 2

Masters Thesis Presentation Sep 29-2011

Fusion-based Multiclass AdaBoost for classification of pose

Motivation and Challenges

Need for Classification at all instance A practical fusion method (combination of Pros)
Variation in Lighting condition, postures, occlusion, background clutter etc. Cosmetics incase of Thermal image

Masters Thesis Presentation Sep 29-2011

Page 3

Fusion-based Multiclass AdaBoost for classification of pose

Guess

[1]

Masters Thesis Presentation Sep 29-2011

Page 4

Fusion-based Multiclass AdaBoost for classification of pose

Ooops.. !!

Masters Thesis Presentation Sep 29-2011

Page 5

Fusion-based Multiclass AdaBoost for classification of pose

Classification model
Feature Extraction Weak learner Boosted Classifier

Masters Thesis Presentation Sep 29-2011

Page 6

Fusion-based Multiclass AdaBoost for classification of pose

Classification model
Feature Extraction
Using useful data from input image that is unique Input image is represented in features

Masters Thesis Presentation Sep 29-2011

Page 7

Fusion-based Multiclass AdaBoost for classification of pose

Classification model
Weak learner
Gives predictions little better than random guessing Classifies using specific feature

Masters Thesis Presentation Sep 29-2011

Page 8

Fusion-based Multiclass AdaBoost for classification of pose

Classification model
Boosted Classifier
Combines the prediction of weak learner Outputs a strong hypothesis

Masters Thesis Presentation Sep 29-2011

Page 9

Fusion-based Multiclass AdaBoost for classification of pose

Classification model
Boosted Classifier Toy Example[2]

Masters Thesis Presentation Sep 29-2011

Page 10

Fusion-based Multiclass AdaBoost for classification of pose

Classification model
Boosted Classifier Toy Example Round 1

Masters Thesis Presentation Sep 29-2011

Page 11

Fusion-based Multiclass AdaBoost for classification of pose

Classification model
Boosted Classifier Toy Example Round 2

Masters Thesis Presentation Sep 29-2011

Page 12

Fusion-based Multiclass AdaBoost for classification of pose

Classification model
Boosted Classifier Toy Example Round 3

Masters Thesis Presentation Sep 29-2011

Page 13

Fusion-based Multiclass AdaBoost for classification of pose

Classification model
Boosted Classifier Toy Example Final

Masters Thesis Presentation Sep 29-2011

Page 14

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Thermal Camera
Captures Visual image and Thermal image

Created Chalmers Visual-Thermal database Used this database of training and testing

Masters Thesis Presentation Sep 29-2011

Page 15

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Thermal Camera
Captures Visual image and Thermal image

Created Chalmers Visual-Thermal database Used this database of training and testing
Masters Thesis Presentation Sep 29-2011 Page 16

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Thermal Camera
Captures Visual image and Thermal image

Created Chalmers Visual-Thermal database Used this database of training and testing
Masters Thesis Presentation Sep 29-2011 Page 17

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Thermal Camera
Captures Visual image and Thermal image

Created Chalmers Visual-Thermal database Used this database of training and testing

Masters Thesis Presentation Sep 29-2011

Page 18

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Thermal Camera
Captures Visual image and Thermal image

Created Chalmers Visual-Thermal database Used this database of training and testing

Masters Thesis Presentation Sep 29-2011

Page 19

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Chalmers Visual-Thermal database
Pose Right Front 506 Left 500 Up 456 Down 460 No. of 500 samples Cropped Visible image Cropped Thermal image

Masters Thesis Presentation Sep 29-2011

Page 20

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Thermal Camera
Captures Visual image and Thermal image

Created Chalmers Visual-Thermal database Used this database for training and testing

Masters Thesis Presentation Sep 29-2011

Page 21

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Visual and Near-Infrared Webcam
Modification of webcam to capture NIR images

Masters Thesis Presentation Sep 29-2011

Page 22

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Visual and Near-Infrared Webcam
Experimental Setup

Masters Thesis Presentation Sep 29-2011

Page 23

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Visual and Near-Infrared Webcam
Sample Images captured in DAY

Visual image
Masters Thesis Presentation Sep 29-2011

NIR image
Page 24

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Visual and Near-Infrared Webcam
Sample Images captured in NIGHT

Visual image
Masters Thesis Presentation Sep 29-2011

NIR image
Page 25

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Microsoft Kinect
Captures Visual, NIR and depth image

Masters Thesis Presentation Sep 29-2011

Page 26

Fusion-based Multiclass AdaBoost for classification of pose

Equipment Used
Microsoft Kinect
Captures Visual, NIR and depth image

Masters Thesis Presentation Sep 29-2011

Page 27

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Database: Chalmers Visual-Thermal

Masters Thesis Presentation Sep 29-2011

Page 28

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Pseudo code
Let = *1 , 2 , , + be the training images = *1 , 2 , , + be the corresponding class labels of , where ,1,2, = , Normalized weights for the input samples
1

Masters Thesis Presentation Sep 29-2011

Page 29

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Haar-like features

Masters Thesis Presentation Sep 29-2011

Page 30

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Haar-like features
Five types of feature

Masters Thesis Presentation Sep 29-2011

Page 31

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Haar-like features
Five types of feature 1 , , , 2 , + , , = In 19x19 image, there exist 35686 features which are used as weak learners

Masters Thesis Presentation Sep 29-2011

Page 32

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Pseudo code
Let = *1 , 2 , , + be the training images = *1 , 2 , , + be the corresponding class labels of , where ,1,2, = , Normalized weights for the input samples = 1 , 2 , be the feature set where = *1 , 2 , +
1

Masters Thesis Presentation Sep 29-2011

Page 33

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Multi-threshold Weak Learner

Masters Thesis Presentation Sep 29-2011

Page 34

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Multi-threshold Weak Learner
Weak Classifier: :
70 60 50

Right Frontal Left

Frequency

40 30 20 10 0 -15

-10

-5

10

15

Feature response of one weak laerner with three class

Masters Thesis Presentation Sep 29-2011 Page 35

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Pseudo code
Let = *1 , 2 , , + be the training images = *1 , 2 , , + be the corresponding class labels of , where ,1,2, = , Normalized weights for the input samples = 1 , 2 , be the feature set where = *1 , 2 , + Weak Classifier: :
Masters Thesis Presentation Sep 29-2011 Page 36

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Multiclass AdaBoost

Masters Thesis Presentation Sep 29-2011

Page 37

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Multiclass AdaBoost
K Strong Classifier, each of them strong in classifying class m Choses WL based on low false-positive and falsenegative, for K=1,Right
Right samples misclassified as Front/Left Front/Left samples wrongly classified as Right

Update weights
For the above samples
Weight the weak hypothesis,
Masters Thesis Presentation Sep 29-2011 Page 38

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Pseudo code For = 1, , ; boosting iterations
For = 1, , strong classifiers
Train a weak classifier for each feature

Choose the best weak hypothesis : with error = min( )

Calculate classifier weights Alpha, Update the weights of the samples:

End Loop

End Loop
Masters Thesis Presentation Sep 29-2011 Page 39

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Pseudo code Output the strong hypothesis,

=
=1 =1

= ,

, = 1, . . , The final estimation is given as

=
=1

= 1, . . ,

E = max
Masters Thesis Presentation Sep 29-2011 Page 40

Fusion-based Multiclass AdaBoost for classification of pose

Fusion-based Multiclass AdaBoost

Visual and Thermal Classifier

Masters Thesis Presentation Sep 29-2011

Page 41

Fusion-based Multiclass AdaBoost for classification of pose

Fusion of Visible and Thermal Classifier

Masters Thesis Presentation Sep 29-2011

Page 42

Fusion-based Multiclass AdaBoost for classification of pose

Fusion of Visible and Thermal Classifier

Masters Thesis Presentation Sep 29-2011

Page 43

Fusion-based Multiclass AdaBoost for classification of pose

Fusion of Visible and Thermal Classifier

Combine the confidence weights of both the classifier,
= + = + =1 1, . . ,

Output:

= max

Masters Thesis Presentation Sep 29-2011

Page 44

Fusion-based Multiclass AdaBoost for classification of pose

Experiment and Results

Classification Using Visual Images
0.5 0.45 Training error Test error

Classification Error(Only Visual Images)

0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0

10

15 20 Boosting Iterations

25

30

Masters Thesis Presentation Sep 29-2011

Page 45

Fusion-based Multiclass AdaBoost for classification of pose

Experiment and Results

Classification Using Thermal Images
0.5 0.45
Classification Error(Only Thermal Images)

Training error Test error

0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0

10

15 20 Boosting Iterations

25

30

Masters Thesis Presentation Sep 29-2011

Page 46

Fusion-based Multiclass AdaBoost for classification of pose

Experiment and Results

Fusion of Visual and Thermal IR Classification
0.5

Classification Error(Fused:Visual and Thermal Images)

0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0

Training error Test error

10

15 20 Boosting Iterations

25

30

Masters Thesis Presentation Sep 29-2011

Page 47

Fusion-based Multiclass AdaBoost for classification of pose

Experiment and Results

Fusion of Visual and Thermal IR Classification
Classification error comparison- Training

Masters Thesis Presentation Sep 29-2011

Page 48

Fusion-based Multiclass AdaBoost for classification of pose

Experiment and Results

Fusion of Visual and Thermal IR Classification
Classification error comparison- Testing
0.12 Visual Thermal Fused 0.1

Classification Error for Test Data

0.08

0.06

0.04

0.02

Masters Thesis Presentation Sep 29-2011

Page 49

Fusion-based Multiclass AdaBoost for classification of pose

Experiment and Results

Fusion of Visual and Thermal IR Classification
Classification rate comparison
Visual (%) Training Testing 95.08 89.4 Thermal IR (%) Fused (%) 98.42 98.4 99.7 99.34

Masters Thesis Presentation Sep 29-2011

Page 50

Fusion-based Multiclass AdaBoost for classification of pose

Conclusion
Fused-based Multiclass AdaBoost was proposed Fusion improves the performance of classification
Classification rate was improved from 95% to 99%

Pavement for the practical solution

Kinect and NIR webcam was investigated

Future work
Images to Video Classification to tracking
Masters Thesis Presentation Sep 29-2011 Page 51

Fusion-based Multiclass AdaBoost for classification of pose

Demo
Any Volunteer ..?

Masters Thesis Presentation Sep 29-2011

Page 52

Fusion-based Multiclass AdaBoost for classification of pose

Questions?

Masters Thesis Presentation Sep 29-2011

Page 53

Fusion-based Multiclass AdaBoost for classification of pose

Reference
[1] Antonio Torralba, Understanding Visual Scenes,NIPS09 Computer Science and Artificial Intelligence Laboratory (CSAIL) Department of Electrical Engineering and Computer Science [2] Rob Schapire, Theory and Applications of Boosting, NIPS09

Masters Thesis Presentation Sep 29-2011

Page 54