21st International Conference on Pattern Recognition (ICPR 2012)
November 11-15, 2012. Tsukuba, Japan
Facial Expression Classification on Web Images
Matthias Richter? Tobias Gehrig?
?
Karlsruhe Institute of Technology, Institute for Anthropomatics, Karlsruhe, Germany
Istanbul Technical University, Faculty of Computer and Informatics, Istanbul, Turkey
matthias.richter@student.kit.edu, tobias.gehrig@kit.edu, ekenel@kit.edu
Abstract
In this paper, we present a novel database which, is
obtained from the web. It contains 4761 manually la-
beled images of seven basic expressions performed by
a large number of subjects of different gender, age and
ethnicity. Furthermore, we develop feature descriptors
based on the discrete cosine transform (DCT), local bi-
nary patterns (LBP), and Gabor filters, which share
a uniform formulation in terms of regions around key
points. We explore several strategies to find an opti-
mal selection of these key points. The system achieves
86.2%, 85.9% and 84.4% accuracy on the web image
database using the Gabor, LBP, and DCT descriptors,
respectively.
1 Introduction
Facial expression is one of the most important chan-
nels in human-to-human communication. At a low
level, they convey internal affective states, which are
often too complex to verbalize. Higher, more conscious
levels enable facial expression to emphasize and even
change the meaning of the spoken word. Computer-
ized access to this channel of communication opens up a
vast number of applications in health care [2], education
[10], human-computer-interaction [3], market research
[11] and entertainment [9]. Unsurprisingly, human fa-
cial expression has attracted a great deal of attention
from the scientific community. Darwin [4] proposed
that natural selection leads to the evolution of emotion
and deduced the existence of universal emotions. Ek-
man [7] confirmed this hypothesis by identifying six ba-
sic emotions anger, disgust, fear, joy, sadness and sur-
prise that show the same facial expression across dif-
ferent cultures. In this paper, we present a face database
that contains facial expressions that correspond to these
universal emotions. The images are collected from the
This study is funded by the Concept for the Future of Karl-
sruhe Institute of Technology within the framework of the German
Excellence Initiative.
978-4-9906441-1-6 2012 IAPR
Hazm Kemal Ekenel?,
web and include a large number of subjects of different
gender, age and ethnicity.
Two frequently used datasets utilized for facial
expression analysis are the Cohn-Kanade [8] and
GEMEP-FERA [1] databases. Both contain image se-
quences showing one or more facial expressions per-
formed by several subjects with varying intensities.
However, they lack in certain aspects. [8] only includes
subjects aged from 1830 years, while in [1] there are
just 10 subjects in total. Furthermore, both databases
exhibit laboratory conditions, which may result in mod-
els that represent real world use cases poorly. Recently,
Dhall et al. [5] addressed this issue by sampling static
images from 37 different movies using subtitle informa-
tion and a manual verification process. Unfortunately,
with only 700 images the database is relatively small.
Due to the extraction from a video stream some of the
facial images are very blurred or low resolution. Others
do not seem to show the target expression at all. Filter-
ing these images reduces the number of samples further.
The importance of automatic classification of facial
expressions has led to several studies. Littlewort et al.
[9] and Shan et al. [12] showed that AdaBoost se-
lected features for expression recognition can signifi-
cantly boost recognition performance. In [9], facial de-
scriptors are built by convolving face images with a Ga-
bor filter family. Using AdaBoost, the most discrimina-
tive of all available filter responses are selected and then
classified using support vector machines (SVM). This
AdaSVM approach outperforms classifying the com-
plete filter response both in discriminative quality and
speed. Because of the relatively high computational
costs of Gabor filters, [12] proposes feature descriptors
based on local binary patterns (LBP). Instead of select-
ing sub-features, AdaBoost is used to select significant
regions for LBP histogram extraction. While requir-
ing less computation time and being more compact, the
resulting descriptors outperform traditional grid-based
histogram extraction in different classification schemes
such as template matching and SVM classification.
Motivated by these results, we develop feature de-
scriptors based on the discrete cosine transform (DCT),
3517
LBP, and Gabor filters. We provide a unifying formula-
tion in terms of regions around key points. Several key
point selection strategies are compared using our web
image database. We show that LBP and DCT descrip-
tors benefit from key point selection, while the Gabor
descriptor does not. Furthermore, selecting too many
regions has a deteriorating effect on recognition perfor-
mance regardless of the feature extraction method.
2 Web Image Database
We have taken several points into consideration
while building the dataset. The corpus should contain
a large number of images of all basic expressions and
the neutral face. It should, furthermore, contain a large
number of male and female subjects from different age
groups and ethnicities. Since personal web sites, stock
photography agencies and services like Facebook and
flickr offer an enormous amount of partially tagged pic-
tures, the dataset can be built using web images. Google
Images offers a searchable index of these resources, and
was used to prepare an initial selection of images.
We used two word combinations as search terms.
One word was selected from a list of up to eight syn-
onyms of the adjectives describing the target expres-
sion, e.g. angry and aggravated. The other was
the term face or person or identified the subjects
age and gender. This produced 440 possible combi-
nations such as frightened man, which were used
to register more than 80000 images and corresponding
meta data, e.g. URL, containing document and matched
text. This initial selection contained a large number of
near duplicate images and suffered from other defects:
While the first few results of a given query were accu-
rate, the number of false positives generally increased,
when more items were taken into account. In addi-
tion, some images contained watermarks that covered
the face in a way that prevented recognition of the fa-
cial expression. Less frequent defects showed images
of only partial faces, heavily distorted images and pic-
tures that did not contain human faces at all.
To discard these images, a four step filtering pro-
cess was employed. First images were flagged whether
they show the desired expression or not. False positives
were removed after finishing labeling all images. In the
second step, faces were marked using rotated bounding
boxes. Previously undetected false positives were dis-
carded. Using the bounding boxes and the DCT feature
described in Sec. 3, near duplicate images were detected
by computing the L1-distance between the feature vec-
tor for the current face and the feature vectors of all the
other faces in the same class. Images to discard had to
be selected manually. This allowed to keep the picture
3518
Table 1. Number of samples per expres-
sion in the database.
Expression Samples Expression Samples
Anger 648 Disgust 368
Fear 288 Joy 2185
Sadness 327 Surprise 557
Neutral 388
with the best image quality and prevented discarding
false duplicates. In the last step, an eye detector was
used to semi automatically mark the eye center posi-
tions. User intervention was only prompted when the
deviation of marked bounding box rotation and rotation
computed from the eye centers exceeded a threshold.
The filtering steps reduced more than 80000 pic-
tures to a database of 4761 labeled faces with associ-
ated meta data.The images show all basic as well as the
neutral expression performed by a large number of sub-
jects of different gender, age and ethnicity under vary-
ing lighting conditions. While the head pose is non-
standardized, most of the images show full or near full
profile faces. Since a large portion of the images orig-
inated from stock photo services, most of the expres-
sions are artificial and non spontaneous. In some cases
watermarks partially occlude the facial region, but not
in a way that prevents humans to recognize the expres-
sion. As shown in Table 1, the database is not balanced,
which has to taken into account when training a classi-
fier using the dataset. The database information, i.e. the
image URLs, marked eye positions and other meta data
can be obtained from:
http://face.cs.kit.edu/datasets/web-expressions.
3 Facial Expression Classification
The web image database was used to evaluate differ-
ent feature description and classification methods. We
randomly selected a subset of images, so that there was
an equal amount of samples for each expression and di-
vided it into three equally sized parts. In six fold experi-
ments, each part was either used for key point selection,
classifier training or evaluation.
3.1 Facial Descriptors
The facial descriptors are formulated in terms of
square pixel regions with side length 2r around key
points pi = (xi , yi ), i.e.
Ri = {(x, y) : r x xi < r, r y yi < r}.
Input images were converted to gray scale and regis-
tered, so that the distance between eye centers became
48 pixels (px). A (96 + 2r) (96 + 2r)px facial region
was extracted, so that distance between left eye center
and the top and left image border was (24 + r)px. The
image histogram was equalized to account for changes
in lighting condition.
Formulating the facial feature descriptor in [6] in
terms of Ri yielded the DCT descriptor. The frequency
components in each region Ri were computed using the
DCT and ordered by applying a zig-zag scan. The lower
and higher frequency components were discarded by
dropping the first k1 and collecting the next k2 coeffi-
cients into the block features fi = (fik1 +1 , . . . , fik1 +k2 ).
To balance the individual regions impact on the overall
classification result, the block vectors were normalized
to kfi k2 = 1. The feature vector FDCT was formed
by concatenating the block-features fi . Throughout the
experiments we used k1 = 1 and k2 = 10.
Following [12], the second feature descriptor em-
ploys uniform LBP. Input images were processed by the
LBPu2 operator with R = 1 and P = 8. The feature
vector FLBP is the concatenation of histograms of la-
bels in the regions Ri . Note that [12] uses a LBPu2 op-
erator with P = 8 and R = 2, which focuses on larger
scale structures. However, since in [12] facial images
measure 110 150px, this constitutes only a minor de-
viation. The use of variable sized non-square regions in
[12] establishes a more substantial difference.
To obtain the Gabor descriptor, input images were
processed by a filter bank of 8 orientations and 5 spa-
tial frequencies. For each of the complex response im-
ages Gmn , the
i
P regions energy content was computed
as Emn = (x,y)Ri kGmn (x, y)k and then collected
into block feature vectors fi . Similar to the DCT de-
scriptor, the feature vector FGabor was obtained by con-
catenating the block features after normalisation. This
descriptor is fundamentally different from the one used
in [9]: Instead of considering the energy content of pixel
regions, their descriptor is built by selecting individual
filter responses using AdaBoost. Given a pi , the result-
ing feature vector may contain a single filter response
at that point, whereas our descriptor always includes all
responses at that location.
3.2 Key Point Selection
We employed several strategies to select regions Ri .
A trivial solution was to place the pi on a regular grid to
cover the whole facial area in a way that the regions do
not overlap. Doing so resulted in the well known block
based approaches used as baseline in [12] and [9]. A
second approach utilized AdaBoost to select a list of
discriminative regions. Initially, every pixel of the input
image was considered to be the region center for fea-
ture extraction. The descriptor F consisted of W H
sub-descriptors fi . With hj denoting linear SVM clas-
sifiers, we associated classifiers hji (f ) = hj (fi ) to the
3519
Table 2. Comparison of best performing
configurations by feature descriptor.
Feature Strategy n r Accuracy
DCT per-class 96 6 0.844
LBP per-class 96 6 0.859
Gabor grid 144 4 0.862
fi . AdaBoost was used to select the n most discrimi-
nate hjt , which were mapped back to pixel locations to
obtain a list of key points.
We trained AdaBoost in two separate ways: In per-
class selection, positive samples were chosen from the
target expression, while negative samples were ran-
domly selected from every other class. This emphasised
on features that discriminate one from all the other ex-
pressions. Expressive selection drew negative samples
from the Neutral class and selected positives from
all remaining classes. Doing so emphasised on fea-
tures that are useful to discriminate expressive from
non-expressive faces. We expected per-class selection
to perform better than expressive selection.
3.3 Classification
We used a 7-way forced choice to classify samples.
For each expression, a third degree polynomial ker-
nel SVMs discriminated the one expression from every
other category. The class of a sample was determined
by choosing the classifier that produced largest confi-
dence. SVM parameters were found by performing a
grid search and choosing the C and with highest ac-
curacy in a 5-fold cross validation on the training set.
Each class and feature selection method was allowed to
produce different parameters.
4 Experimental Results
We evaluated different key point selection methods
using Ri with r = 4 up to r = 12. The number of
regions varied between n = 5 and n = 144 for ex-
pressive and per-class selection and was defined by r
when using the grid based approach. Other feature de-
scriptor parameters were not varied. Table 2 lists the
best performing configuration for each feature extrac-
tion method. It can be seen that Gabor features show
best recognition performance, while the DCT descrip-
tor performs worst with a 1.8% difference in mean ac-
curacy. However, the DCT descriptor has a much lower
dimensionality and is faster to compute than both the
Gabor and LBP descriptors.
As expected, per-class key point selection is superior
to expressive selection. Consistent with [12], boosted
LBP features outperform the grid based approach, al-
 Table 3. Confusion matrix of the Gabor
feature descriptor.

Surprise
Sadness
Disgust

Neutral
Anger

Fear

Joy
Ang. 0.44 0.16 0.06 0.03 0.05 0.08 0.06
Dis. 0.19 0.41 0.08 0.06 0.06 0.16 0.05 Figure 1. Commonly misclassified im-
Fear 0.05 0.08 0.43 0.03 0.07 0.10 0.19 ages.
Joy 0.02 0.06 0.03 0.72 0.03 0.04 0.02
Neu. 0.12 0.07 0.06 0.05 0.64 0.13 0.07 cial expression recognition. Feature descriptors based
Sad. 0.11 0.18 0.13 0.08 0.09 0.42 0.05
Sur. 0.06 0.04 0.21 0.03 0.07 0.06 0.56 on the DCT, LBP, and Gabor filters were collectively
formulated in terms of regions around key points. Sev-
though not by such a large margin as observed in their eral strategies to find an optimal selection of key points
study. This might be attributed to the differences in the have been explored. The DCT and LBP based descrip-
feature extraction methods. The Gabor descriptor out- tors have been shown to benefit from key point selec-
performs both the DCT and the LBP descriptor, though tion, while the Gabor feature performed best when re-
surprisingly the best result is not achieved using boosted gions were uniformly distributed.
features, but by the grid based approach. Because we
observed that the Gabor descriptor yielded higher accu- References
racy when using smaller regions, we extended the ex-
periments to include regions with r = 1 and r = 0 (i.e. [1] T. Banziger and K. R. Scherer. Introducing the
only the filter responses). Still, grid based key point Geneva Multimodal Emotion Portrayal (GEMEP) Cor-
pus. Blueprint for affective computing: A sourcebook,
selection yielded highest mean accuracy. This directly pages 271294, 2010.
contradicts with the findings in [9], where AdaSVMs [2] J. Cockburn, M. Bartlett, J. Tanaka, J. Movellan,
performed considerably better than a traditional Gabor M. Pierce, and R. Schultz. SmileMaze: A Tutoring
System in Real-Time Facial Expression Perception and
descriptor. This result may be attributed by the differ- Production in Children with Autism Spectrum Disorder.
ent approach in feature selection: In [9] individual fea- In FGR08, pages 678986, 2008.
ture responses were selected, while we considered all [3] R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Vot-
sis, S. Kollias, W. Fellenz, and J. G. Taylor. Emotion
responses given a region Ri . recognition in human-computer interaction. Signal Pro-
Table 3 shows the confusion matrix for the Gabor cessing Magazine, 18(1):3280, January 2001.
feature descriptor. Worst recognition performance is [4] C. Darwin. The Expression of the Emotions in Man
observed with disgust, sadness, fear and anger. Dis- and Animals. Harper Perennial, anniversary edition,
1872/2009.
criminating anger from disgust, disgust from sadness [5] A. Dhall, R. Goecke, S. Lucey, and T. Gedeon. Static fa-
and fear from surprise produces the most mistakes. Joy cial expression analysis in tough conditions: Data, eval-
and neutral on the other hand are recognized with high uation protocol and benchmark. In ICCV Workshops,
pages 2106 2112, Nov. 2011.
confidence. Figure 1 shows commonly misclassified [6] H. K. Ekenel. A Robust Face Recognition Algorithm
images. From left to right images are tagged anger, fear, for Real-World Applications. PhD thesis, Universitat
anger and neutral and were classified as neutral, sad- Karlsruhe (TH), Fakultat fur Informatik, February 2009.
[7] P. Ekman. Basic emotions. In Handbook of cognition
ness, surprise and sadness, respectively. Some of these and emotion, volume 98, chapter 3, pages 4560. John
images, e.g. the second from left, are difficult to classify Wiley & Sons, 1999.
[8] T. Kanade, J. Cohn, and Y. Tian. Comprehensive
even for humans. Others show features shared between Database for Facial Expression Analysis. In FGR00,
expressions, such as wide opened eyes and raised inner pages 4653, 2000.
eye brows. [9] G. Littlewort, M. S. Bartlett, I. R. Fasel, J. Chenu,
T. Kanda, H. Ishiguro, and J. R. Movellan. Towards
social robots: Automatic evaluation of human-robot in-
5 Conclusion teraction by face detection and expression classification.
In NIPS, 2003.
[10] G. C. Littlewort, M. S. Bartlett, L. P. Salamanca, and
We presented a novel database compiled from web J. Reilly. Automated measurement of childrens facial
images containing 4761 labeled faces of male and fe- expressions during problem solving tasks. In FGR11,
pages 3035, 2011.
male subjects of different ethnicities and age groups. [11] R. W. Picard. Measuring affect in the wild. In ACII11,
Variations of expression intensity, head pose, and light- pages 33, Berlin, Heidelberg, 2011. Springer-Verlag.
ing conditions pose a new challenge for facial expres- [12] C. Shan, S. Gong, and P. W. McOwan. Facial expres-
sion recognition based on Local Binary Patterns: A
sion recognition systems. comprehensive study. Image and Vision Computing,
We furthermore developed a modular system for fa- 27(6):803816, 2009.

3520