HMM-based Human Action Recognition Using Multiview Image Sequences
Mohiuddin Ahmad and Seong-Whan Lee
Department of Computer Science and Engineering, Korea University,
Anam-dong, Seongbuk-gu, Seoul 136-713, Korea
{mohi, swlee}@image.korea.ac.kr
Abstract
In this paper, we present a novel method for human ac-
tion recognition from any arbitrary view image sequence
that uses the Cartesian component of optical flow veloc-
ity and human body silhouette feature vector information.
We use principal component analysis (PCA) to reduce the
higher dimensional silhouette feature space into lower di-
mensional feature space. The action region in an image
frame represents Q-dimensional optical flow feature vector
and R-dimensional silhouette feature vector. We represent
each action using a set of hidden Markov models and we
model each action for any viewing direction by using the
combined (Q + R)-dimensional features at any instant of
time. We perform experiments of the proposed method by
using KU gesture database and manually captured data.
Experimental results of different actions from any viewing
direction are correctly classified by our method, which in-
dicate the robustness of our view-independent method.
1 Introduction
This contribution addresses the human action recogni-
tion from any arbitrary view of an image sequence with
the assumption that each image sequence includes only one
person and performs a single activity. Usually, recognition
of human actions from image sequences is very popular
in computer vision community, which has applications in
video surveillance and monitoring, human-computer inter-
actions, etc. Since, there is no rigid syntax and well defined
structure for human action recognition is available, there-
fore, human action recognition is a more challenging and
sophisticated task. Several human action recognition meth-
ods have been proposed in the past decades. A detailed
excellent survey can be found in [1]. For recognizing ac-
tion, researchers use either explicit human body shape or
motion. For example, in [7] [8], authors use 2D and 3D
shape features for recognizing actions. Motion-based ac-
tion recognition was also performed by several researchers,
Proceedings of the 18th International Conference on Pattern Recognition (ICPR'06)
0-7695-2521-0/06 $20.00 © 2006
such as, in [9],[10], and [13], authors use motion feature for
action recognition. Most of the above techniques depend
on the viewing direction. The work of testing an action us-
ing multi-view motion learning is still unsolved. In [11] and
[12], authors presented view invariant recognition of action.
In this paper, we recognize human action from an ar-
bitrary view of an image sequence using the optical flow
and human body shape features. In this regard, human ac-
tion models are created in each viewing direction for some
specific actions using hidden Markov model (HMM). We
extract the optical flow feature and new body silhouette fea-
ture and then the feature vectors are converted into sym-
bols. Then, we learn the features and build HMMs in dif-
ferent viewing directions. Classification is finally achieved
by feeding a given (test) sequence in any viewing direction
to all the learned HMMs and employing a likelihood mea-
sure to declare the action performed in the image sequence
which makes our system view invariant. For training and
testing actions, we use the KU gesture database [6] and
manually captured data.
This paper is organized as follows: Section 2 describes
the features extraction from optical flow and human body
silhouette. Section 3 briefly describes the HMMs for action
modeling and recognition. Experimental results and anal-
ysis of the selected approaches are presented in Section 4.
Finally, conclusions are drawn in Section 5.
2 Feature Extraction
In this section, first, we extract the foreground and detect
the action region from the image sequence, second, we ex-
tract the human body shape feature and optical flow feature
in different viewing directions.
2.1 Foreground extraction
For extracting the foreground, we use a simplification of
the background subtraction algorithm used in [2], and the
color value of each background pixel is modeled by using a

(a) (b) (c)
Figure 1. Image preprocessing steps of an
image in action video. (a) sample frame (320×
240 pixels) (b) filtered foreground (320 × 240
pixels) (c) action region (80 × 150 pixels) and
(d) normalized image (60 × 80 pixels).
single Gaussian distribution. The probability density func-
tion that a background pixel will have a color value xt at
time t is estimated by using mean value, (μt ) and covari-
ance matrix. We assume the independence of different color
channels with variance σi2 , therefore, the estimated proba-
bility of xt is given by the following expression(i means
different color):
3 − 1 (xt i −μt,i )2
 1 2
σ2
P (xt ) = 2 2 1 e i
i=1 (2πσi )
Using this probability estimation, we compute the prob-
ability P (xt ) that a pixel value belongs to the background
or not. Then, we can use a threshold value that estimate
any pixel value will be the foreground pixel or not. We up-
date the background parameters, μt and σt2 continuously to
adopt the scene as [2]. Figure 1 (a) shows the sample frame
from the image sequence of a walking person, and Fig.1
(b) shows the extracted foreground image after background
subtraction and median filtering.
2.2 Action region extraction
We define the action region as the rectangular area where
action is occurred in the image frame. The action region
depends on body shape, types of action, and the distance
between the camera and the person who performs action.
Since distance is almost constant, therefore, we select the
action region inside the image frame which includes the av-
erage human body shape of the full image sequences. Fig-
ure 1 (c) shows the action region.
2.3 Shape features extraction
The action region silhouette image is normalized to 60 ×
80 pixels by using bi-cubic interpolation method, shown in
Fig. 1(d). For PCA analysis, let the training set of the nor-
malized images are Γ1 , Γ2 , Γ3 ,· · ·, ΓM . The average of
1 M
the set is defined by Ψ = M n=1 Γn . Each body sil-
houette image differs from the average Ψ by the vector
Proceedings of the 18th International Conference on Pattern Recognition (ICPR'06)
0-7695-2521-0/06 $20.00 © 2006
(a) Normalized silhouette images of run
(d)
(b) Feature images of the specified actions using new silhouette
features
Figure 2. Shape images and features.
Φi = Γi − Ψ. An example training set of the sequence
is shown in Figure 2(a). This very large set of vector is then
subject to PCA, which seeks a set of M ortho-normal vec-
tor, uM , which best describe the distribution of the data.
The k-th vector, uk , is chosen [4], such that,
1  T
M
λk = (u Φn )2 (2)
M n=1 k
(1) The vector uk and scalars λk are the eigenvectors and eigen-
values, respectively, of the covariance matrix [4]
1 
M
C= Φn ΦTn (3)
M n=1
The eigenvectors, uk , for i = 1, · · · , M are ranked accord-
ing to their associated eigenvalues, λk . Given a shape vector
λ, the projected new feature space is given by:
sk = uTk (Γ − Ψ) (4)
for k = 1, · · · · · · , N . Therefore, we use new feature
T
vectors, s = [s1 , s2 , · · · , sR ] for every frame of the se-
quence of any action in any viewing direction. Figure 2
(b)shows the new shape feature images of the specified ac-
tions, where x-axis represents the time and y-axis represents
the multiple features.
2.4 Motion features extraction
In this work, we use optical flow to estimate motion, be-
cause we can precisely determine the non-rigid motion of
the human action at any pixel in the image sequence. We
calculate the optical flow, v(vx , vy ) at any pixel x(x, y) in
time t from the action regions by using Lukas-Kanade’s op-
tical flow technique [3]. Figure 3(a) shows the optical flow
of five specified frame for walking, raising of the hand,
bowing, sitting on the floor and running, respectively, in
which optical flows are overlapped on the body parts. For
consistency of the analysis, we normalize the optical flow,
 (a) Optical flow in action regions (b) Flow features

Figure 3. The optical flow vectors and corre-

sponding flow feature vectors.

vn (vnx , vny ). Finally, for feature extraction, we partition

the action region into K blocks, B(k) of equal sizes. We
extract the flow features, vkx and vky by using the average Figure 4. Multiple views for training se-
value of optical flow motion at each block with n number quences (with sample images).
of pixels by using the following expression:
1 
[vkx,t , vky,t ] = [vnx (x, t), vny (x, t)] (5) rameters to achieve the local maximum. We use maximum
n
x∈B(k) likelihood approach for classifying each action:
T
Then, we arrange the vector, [v1x,t , ..., v Q x,t ] and λ = arg max P (O|λa ) (7)
2 λa ∈allActions
T
[v1y,t , ..., v Q y,t ] for any frame t of the sequence, where
2 where, P (O|λa ), the conditional probability for any ac-
Q = 72. The line plot of Fig. 3(b) shows the flow fea-
tion a and it is computed by P (O|λa ) = max P (O|λad ),
tures of the specified actions in Cartesian coordinate sys-
where O is the multi-observation feature vector sequence of
tem (x-flow and y-flow) by using equation above. For each
an unknown action.
image frame of any action, silhouette features and optical
flow features are combined and known as combined fea-
tures, It = [s1 , · · · , sR , v1x,t , · · · , v Q x,t , v1y,t , · · · , v Q y,t ]. 4 Experimental Results and Analysis
2 2

Experiments are performed on image sequences that

3 Action Modeling using HMMs
Hidden Markov Models have been successfully used for
speech recognition. We employ HMM for action recogni-
tion, since it can be applied to analyze the time series with
spatio-temporal variations. In this paper, we use HMM
for modeling and testing actions. We build HMM model
λad = {A, B, π} for any action a in any viewing direction
d to characterize the variation from the change of viewing
direction by using the symbols of the individual and com-
bined features. We can consider a set of hidden Markov
model for the multi-view directions which is expressed as:
λa = {λa1 , λa2 , · · · λad }
We use same topology of HMM , i.e. 6-state ergodic
model. The number of states is heuristically selected. Each
observation is modeled by using mixtures of Gaussian den-
sities. Two mixtures per feature is used in our experiment.
The model parameters are adjusted in such a way that they
can maximize the likelihood P (O|λa ) for classifying ac-
tion by using the given set of training data. We use Baum-
Welech algorithm [5] for iteratively re-estimate model pa-
have 320 × 240 pixel resolution and 30 frames per second.
We use 2D video data of KU gesture database [6] and man-
ually captured data of all directions for modeling and clas-
sifying actions. The training data set includes eight views,
which are shown in figure 4. We use five actions for training
and testing purposes, such as walking, raising of the hand,
bowing, running, and sitting on the floor. The duration of
each video is approximately 2 ∼ 8 seconds depends on per-
forming the action.
Table 1 shows the confusion matrix of action recogni-
tion using HMM where we use human body shape feature,
flow feature, and combined features. Each column in the ta-
ble represents the best match for each test sequence in any
arbitrary view direction. The 1st, 2nd, and 3rd value repre-
(6)
sents the recognition accuracy for silhouette, optical flow,
and combined features and the recognition rate becomes
81.25%, 77.5%, and 87.5%, respectively. The recognition
rate of combined features is better than the recognition rate
of individual feature. Some sequences are misclassified,
such as walking and running. They may occur due to the
high degree of similarity in the front view and rear view.
In table 2, we compare our test results with some previ-
ous results. The important thing is to note that we recognize

Proceedings of the 18th International Conference on Pattern Recognition (ICPR'06)

0-7695-2521-0/06 $20.00 © 2006
 References
Table 1. Confusion matrix. Action recognition
using (1) body silhouette feature, (2) optical [1] D. M. Gavrilla, “The visual analysis of human move-
flow feature, and (3) combined features. ment: A survey,” CVIU, Vol. 73, No. 1, January, 1999,
pp. 82-98.
Walk Raise Bow Run Sit
Walk 13,12,14 1,1,0 0,0,0 2,3,2 0,0,0 [2] C. Stauffer and W. Grimson, “Adaptive background
Raise 2,2,2 12,11,13 0,0,0 2,3,1 0,0,0 mixture model for real time tracking,” Proc. of IEEE
Bow 0,0,0 0,0,0 15,15,16 0,0,0 1,1,0 Conf. on CVPR, Vol. 2, 1999, pp. 246-252.
Run 3,3,3 1,2,0 0,0,0 12,11,13 0,0,0
Sit 1,2,1 0,0,0 1,0,1 1,1,0 13,13,14 [3] B. Lucas and T. Kanade, “An iterative image regis-
tration technique with an application to stereo vision,”
Proc. of IJCAI, 1981, pp. 674-679.
human action from any arbitrary view rather than any spe- [4] M. Turk and A. Pentland, “Eigenfaces for recogni-
cific view. We mainly face the problem of recognition in tion,” Journal of Cognitive Neuroscience, Vol. 3, No.
front view among all views. 1, 1991, pp. 71-80.
[5] R. Lawrence, and A. Rabiner, “Tutorial on hidden
Table 2. Comparison results of action recog- Markov models and selected applications in speech
nition with some previous researches. recognition,” Proc. of the IEEE, Vol. 77 No. 2, 1989,
pp. 257-286.
Researches Features View Recog(%)
Ali [13] Angle profile 78.8 [6] B. W. Hwang, S. Kim, and S. W. Lee, “A full-body
Masaud [9] motion parallel 92.8 gesture gatabase for automatic gesture recognition,”
Yacoob [10] parametric motion diagonal 82.0 Proc. of IEEE Conf. on FGR, April 2006, pp. 243-248.
Our approach shape & flow Independent 87.5 KU Gesture Database, http://gesturedb.korea.ac.kr/.
[7] I. Cohen and H. Li, “Inference of human postures
by classification of 3D human body shape,” Proc. of
5 Conclusions and Future Research IEEE WS on AMFG, 2003, pp. 74-81.
[8] S. Carlsson and J. Sullivan, “Action recognition by
In this paper, we proposed a novel method for HMM- shape matching to key frames,” Proc. of IEEE WS on
based view-independent human action recognition using Models versus Exemplars in CV, Florida, USA, 2002,
body silhouette feature, optical flow feature, and combined pp. 263-270.
feature. Based on these features, a set of HMMs were built
for each action to represent each action from different views [9] O. Masoud and N. Papanikolopoulos, “A method for
to enable recognizing from arbitrary views. The recogni- human action recognition,” IVC, Vol. 21, 2003, pp.
tion rate was found 87.5% for combined feature which is 729-743.
higher than the rate obtained by individual features. This
[10] Y. Yacoob and M. J. Black, “Parameterized modeling
result showed that our algorithm was robust to variations in
and recognition of activities,” CVIU, Vol. 73, No. 2,
view and duration. Although this rate was found lower than February 1999, pp. 232-247.
some previous researches. But it would be mentioned that
we recognize action from arbitrary views rather than any [11] C. Rao and M. Shah, “View-Invariance in action
specific view. Our future work includes the interaction of recognition,” Proc. of IEEE Conf. on CVPR, Hawaii,
multi-view learning using a single frame work with using December 2001, pp. 316-321.
the complex actions and more precise recognition of human
actions. [12] V. Parameswaran and R. Chellappa, “View invariants
for human action recognition,” Proc. of IEEE Conf. on
Acknowledgment CVPR, Vol. 2, 2003, pp. 613-619.
[13] A. Ali, J. K. Aggarwal, “Segmentation and recog-
This research was supported by the Intelligent Robotics nition of continuous human activity,” Proc. of IEEE
Development Program, one of the 21st Century Frontier Workshop on Detection and Recognition of Events in
R&D Programs funded by the Ministry of Commerce, In- Video, Canada, July 2001, pp. 28-35.
dustry and Energy of Korea.

Proceedings of the 18th International Conference on Pattern Recognition (ICPR'06)

0-7695-2521-0/06 $20.00 © 2006