2 views

Uploaded by Aniket Roy

Optical Recognition of Handwritten Digits

- Dynamic PCA
- Data Mining Application
- PCA analysis
- 06608492
- Based on the SVM University Educatio
- nn
- slides01shkene
- Webster, Mathematical treatment of soil information
- Intrusion Detection
- Blast Disease Forecasting Using an a Neural Network
- [000049]
- SVM KNN Classifier
- Binder1 48.pdf
- Very Large SVM Training Using Core Vector Machines
- Contents
- Naphtha Leak Detection
- Facto Extra
- Gender Inequality
- 10.1.1.300.6258
- An Improved PCA Scheme for Sensor FDI- Application to an Air Quality Monitoring Network

You are on page 1of 6

Pradeep Rajendran

December 20, 2013

1 Introduction

Recognition of handwritten digits has many everyday uses. It is particularly applied in the auto-

mated sorting of postal addresses based on zipcode. A general implementation of such a system is

breifly described.

As the postal item moves on a conveyor belt, robotic mechanisms reposition the item such that

the address label is visible to a camera. Then, the camera registers an image of the label. The

image is then fed into a image processing system which dissects the image into constituent character

blocks. The digit blocks composing the zipcode field is then thresholded and pre-processed to ensure

uniform scale and orientation. After preprocessing, the binary image of a digit block is ready for

use in machine learning tools.

The pre-processing step is rather involved and out of the scope of this project as it requires many

image processing steps. It cannot be skipped as it is crucial to the success of most machine learning

tools.

In this project, I focus on how the performance of common machine learning tools compare with

each other.

2 Dataset

The MNIST(Mixed National Institute of Standards and Technology) dataset [1] is used in this

project. All the digit images have been pre-processed such that the digit is centered on a 28 28

block of 8-bit grayvalues. Fig. 1 shows an example of a digit block containing the character 8.

Fig. 2 shows a ensemble of digit blocks containing the character 5.

1

5

10

15

20

25

5 10 15 20 25

3 Tools applied

3.1 Multi-layer perceptron

A two layer perceptron implementation found in the Neural Network Toolbox is utlized for this

section. The first layer is the input layer and it has 784 inputs. The second layer has h hidden

neurons. And, the final layer is the output layer consisting of 10 neurons corresponding to the 10

class labels (i.e. 10 digits).

2

Fig. 3: An example of 2-layer neural network with 10 hidden neurons

The table below shows the various values of h that were tried and the corresponding performances.

10 8.46

20 8.92

100 2.09

120 11.22

200 21.27

From Table 1, it is clear that h = 100 gives better performance than h = 200. This might be due

to over-fitting associated with h = 200. The ROC plots for h = 100 are given in Fig. 4.

From the ROC, it appears that the number 4 is often misclassified as it is furthest away from the

point (0, 1).

The LIBSVM suite of tools developed by Chih-Chung Chang and Chih-Jen Lin is used in this section

[2]. The main tools are: svm-scale (used for scaling data), svm-train (used to obtain support vectors

3

from given data), svm-predict (used to make predictions based on a trained model).

Method

svm-scale is first used to to scale the input feature vectors in the training file to have a value of either

+1 or -1. A scale file is also produced along with the scaled training file. The scale file contains the

appropriate scale values that have to be applied to the testing data as a pre-processing step. The

scaled training file is then input to svm-train which produces a model file. This model file contains

the support vectors identified in the scaled training file. Once the model file is obtained, testing is

performed using svm-predict. svm-predict uses the model file to determine the classification of test

vectors and produces an output file containing these predictions.

Choice of parameters

There are many choices of parameters that can be chosen during the training phase of the SVM.

Table 2 shows the error rate corresponding different kernels and parameters.

Linear 6.85

Polynomial (Order 4) 2.18

Polynomial (Order 5) 2.01

Polynomial (Order 6) 1.92

Polynomial (Order 7) 1.92

Polynomial (Order 8) 1.99

Gaussian 2.66

Sigmoidal 10.71

According to Table 2, it seems that highest performance is achieved when a polynomial kernel of

order 6 or 7 is used.

K-NN is the simplest classification method. But, it is also a slow and memory intensive method.

This is because, during testing phase, a similarity metric between each test feature vector and all

the training feature vectors has to be calculated. Therefore, for this dataset, 60 000 similarity metric

calculations have to be performed for each test vector. Since there are 10 000 test vectors, the total

similarity metric calculation calls sums up to 600 000 000.

Eigen-digit method

The Eigen-digit method involves using PCA (Principal Component Analysis) to reduce the dimen-

sionality of the feature space from M to m. In this way, each training feature vector becomes

a m-dimensinal vector instead of the original 784-dimensional vector as shown in Fig. 5. While

performing PCA, the m eigenvectors (vj ) and 10 eigendigits (Ej = [v1 v2 . . . vm ]) are also obtained.

4

v1 Weight 1

v2 Weight 2

Projection

784-dim vector v3 Weight 3 5-dim vector

v4 Weight 4

v5 Weight 5

During the testing phase, each of 784-dimensional test feature vector is projected to the m-

dimensional feature space using the eigen-digits obtained in the training phase. The resulting

m-dimensional feature vector is then compared with the huge collection of labeled m-dimensional

training feature vectors (leftmost vectors in Fig. 6). During the comparision, K labled closest

matches from the collection are identified. Amongst the closest K matches, the most frequently

occuring label is taken as the classification output of the Eigen-digit method.

Distance bewteen

50-dim vector 50-dim training vector

n

io

e ct ..

oj ..

Pr .

.

Projection E5

784-dim test vector 50-dim vector 50-dim training vector

Pr

50-dim training vector

o je ..

ct . 50-dim training vector

io

n

E

9

50-dim vector

5

Combinations K=1 K=3 K=5 K=7 K = 11 Computation time (s)

m = 10 61.156 60.546 61.656 62.286 63.496 71

m = 50 8.981 8.931 9.841 10.991 13.851 410

m = 100 5.261 5.451 6.141 7.011 8.501 837

m = 200 4.01 4.23 4.78 5.381 6.621 1474

m = 250 3.77 4.03 4.59 5.161 6.391 1786

m = 500 3.49 3.76 4.16 4.65 5.731 3348

m = 784 3.47 3.7 4.12 4.59 5.661 5907

Table 3: Error rate and computation time for various combinations of m and K

Increasing K does not seem to improve error rate. And, increasing m has a diminishing improvement

on error rate. A trade-off between computation time and error rate can be acheived by picking a

value for m for which the classification rate specifications are still met.

4 Conclusion

The SVM seems to be particularly well suited for digit recognition as it has the the best performance

when compared to the MLP or KNN methods. The best error rates of SVM, MLP and KNN are

1.92, 2.09 and 3.47 respectively. SVM is not only accurate but also faster than the other methods

tested. KNN with m = 784 and K = 1 nearly took 1 hour and 40 minutes of computation time for

calculating labels of 10 000 test vectors. This translates to about 0.60 seconds for each test vector

which is too slow.

References

[1] Y. LeCun and C. Cortes, MNIST handwritten digit database.

http://yann.lecun.com/exdb/mnist/, 2010.

[2] C.-C. Chang and C.-J. Lin, LIBSVM: A library for support vector machines, ACM Transac-

tions on Intelligent Systems and Technology, vol. 2, pp. 27:127:27, 2011.

- Dynamic PCAUploaded bydisgusting4all
- Data Mining ApplicationUploaded byAbhinaw Kumar Singh
- PCA analysisUploaded byKarthik Arumugham
- 06608492Uploaded byGaurav Jaiswal
- Based on the SVM University EducatioUploaded bycastorypolux20762
- nnUploaded byandhracolleges
- slides01shkeneUploaded byh3492708
- Webster, Mathematical treatment of soil informationUploaded byScary Creatures
- Intrusion DetectionUploaded byArun Kumar
- Blast Disease Forecasting Using an a Neural NetworkUploaded bysyakira
- [000049]Uploaded byEka Puspa Rini
- SVM KNN ClassifierUploaded byMunaf Rashid
- Binder1 48.pdfUploaded byAbdul Rahman
- Very Large SVM Training Using Core Vector MachinesUploaded byravigobi
- ContentsUploaded byLahari Reddy
- Naphtha Leak DetectionUploaded byraj@siri7567
- Facto ExtraUploaded byAfulito
- Gender InequalityUploaded byGautam Bindlish
- 10.1.1.300.6258Uploaded byBruno Santos
- An Improved PCA Scheme for Sensor FDI- Application to an Air Quality Monitoring NetworkUploaded byYaqoob Ali
- Ambivalence in Moral SelfUploaded byDanilo Moggia
- Lecture1 JpsUploaded byChetan Desai
- Kn 2518431847Uploaded byAnonymous 7VPPkWS8O
- Secure Image Transmission using Visual SteganographyUploaded byEditor IJRITCC
- IJETT-V3I1P211Uploaded bysurendiran123
- MachineLearning-Lecture15.pdfUploaded bySethu S
- A Review of Optimization Methodologies in Support Vector MachinesUploaded byFrank Puk
- Leonardis Robusteigen Cviu00Uploaded bySandeep Devasrii
- Relationship Between LearningUploaded byShyamendra Kushwaha
- 1. IJAMSS - A PRINCIPAL COMPONENT REGRESSION MODEL FOR FORECASTING.pdfUploaded byiaset123

- benefits of video games apaUploaded byapi-272200454
- colour wheel lesson planUploaded byapi-298785195
- CERIMONIAL FORMATURA UNEDUploaded byanon-855162
- Modulo-fase Virtual, Seminario 2010Uploaded byJakori Chaupin
- Exposing the Global Road to Ruin through EducationUploaded byRivan Phoenix
- A Fuzzy Back Propagation AlgorithmUploaded byĐặng Sơn Tùng
- Senior Student ProfileUploaded byJohn Jones
- FINAL EXAM 201820192- Draf (270319)Uploaded byYuthishDheeranManiam
- persuasive writing lesson 1Uploaded byapi-337444779
- Kepentingan Design and TechnologyUploaded byanisku
- Beyond the constructivismUploaded bySuperlucido
- families3Uploaded byapi-319403975
- lesson3Uploaded byapi-301661997
- 520 drama objectivesUploaded byapi-298535721
- A Close Look at the Relationship Between Multiple Choice VocabularyUploaded bypalala89
- Thesis Revised 4 5Uploaded byCrystal Joy Malizon
- Ed 208683Uploaded byAnonymous j8Ge4cKI
- 22.APP Training Need AssessmentUploaded byImpact Journals
- A New Look at an Old Construct Cross-cultural Adaptation 1994Uploaded byValeria Lanata
- The Use of MTB.docxUploaded byVincentAlejandro
- Materials Design and Lesson Planning - PoetryUploaded byJonathan Acuña
- Grade 5Uploaded bypowell.gordon
- l o 1Uploaded byapi-217524579
- mastery matrix expectations 16-17Uploaded byapi-252206994
- Using Music to Accelerate Language LearningUploaded byRendel Rosalia
- IJTLHE55Uploaded byEl Avion Noel
- Lesson PlanUploaded byTee Pei Leng
- 208419490 Boxing Coaching Manual by Kevin Hickey Uk AbaUploaded byJose Angel San Roman
- 03 Chapter 4Uploaded byaashneet
- Pilot Cohort Corner Issue #6Uploaded byJoaquin Hernandez