ARTICLE IN PRESS

Signal Processing: Image Communication 23 (2008) 473– 489

Contents lists available at ScienceDirect

Signal Processing: Image Communication

journal homepage: www.elsevier.com/locate/image

A compressed-domain approach for shot boundary detection on

H.264/AVC bit streams
Sarah De Bruyne , Davy Van Deursen, Jan De Cock, Wesley De Neve,
Peter Lambert, Rik Van de Walle
Department of Electronics and Information Systems, Multimedia Lab, Ghent University, IBBT, Gaston Crommenlaan 8 bus 201, B-9050 Ledeberg-Ghent, Belgium

a r t i c l e in fo abstract

Article history: The amount of digital video content has grown extensively during recent years, resulting
Received 21 April 2008 in a rising need for the development of systems for automatic indexing, summarization,
Accepted 29 April 2008 and semantic analysis. A prerequisite for video content analysis is the ability to discover
the temporal structure of a video sequence. In this paper, a novel shot boundary
Keywords: detection technique is introduced that operates completely in the compressed domain
H.264/AVC using the H.264/AVC video standard. As this specification contains a number of new
Shot boundary detection coding tools, the characteristics of a compressed bit stream are different from
Temporal segmentation prior video specifications. Furthermore, the H.264/AVC specification introduces new
Video analysis
coding structures such as hierarchical coding patterns, which can have a major
influence on video analysis algorithms. First, a shot boundary detection algorithm is
proposed which can be used to segment H.264/AVC bit streams based on temporal
dependencies and spatial dissimilarities. This algorithm is further enhanced to exploit
hierarchical coding patterns. As these sequences are characterized by a pyramidal
structure, only a subset of frames needs to be considered during analysis, allowing the
reduction of the computational complexity. Besides the increased efficiency, experi-
mental results also show that the proposed shot boundary detection algorithm achieves
a high accuracy.
& 2008 Elsevier B.V. All rights reserved.

1. Introduction summarization are generally accepted as the key to

Advances in multimedia coding technology, combined
with the growth of the Internet and the advent of digital
television, have resulted in the widespread use and
availability of digital video. This rise will only be
strengthened by the increasing popularity of user-gener-
ated video content. Unfortunately, these video collections
are often not catalogued and only accessible by sequential
scanning. As a consequence, technologies and tools for
efficient browsing and retrieval of video content are
gaining importance. Semantic key frame extraction and
 Corresponding author. Tel.: +32 9 33 14957; fax: +32 9 33 14896.
E-mail address: Sarah.DeBruyne@UGent.be (S. De Bruyne).
URL: http://multimedialab.elis.ugent.be
quickly browse through a video sequence and to enable
easy navigation between relevant segments [33]. To
extract the semantics from the video content, a system
or agent is needed to analyze the content. Manual
annotation is very time consuming, expensive, and often
unfeasible for most applications, to the point that it is
almost impossible. Therefore, automatic analysis of video
content is of significant importance.
Since the identification of the temporal structure of
video is an essential task for video indexing and retrieval,
the first step commonly taken for video analysis is shot
boundary detection. A shot is usually conceived in the
literature as a series of interrelated consecutive pictures
taken contiguously by a single camera and representing a
continuous action in time and space [2]. According
to whether the transition between consecutive shots is

0923-5965/$ - see front matter & 2008 Elsevier B.V. All rights reserved.
doi:10.1016/j.image.2008.04.012
 ARTICLE IN PRESS
474 S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489
abrupt or not, boundaries are classified as cuts or gradual
transitions.
After identifying the shot boundaries, most of the
existing work focuses on the summarization of the
content. One common approach for visualizing video
content is a storyboard in which key frames are tiled
together to obtain a good overview of the video. Each key
frame generally represents the content of one shot and is
selected according to an analysis method that optimizes
the semantic coverage of the video content [4,13]. A
second approach is summarizing the video content by
selecting short, highlighted segments, resulting in a fast
preview of the content. A number of approaches have
been used to generate these so-called video skims,
including rate-distortion and motion analysis [10,11,24].
These techniques often rely on MPEG standards such as
MPEG-7 [28] and MPEG-21 [3] for the description and
analysis of the multimedia content and the multimedia
customization [5,10,11,27].
In order to preserve storage space and to reduce
bandwidth constraints, most video data in multimedia
databases are stored in compressed form. To avoid
unnecessary and time-consuming decompression during
video analysis, features available in the compressed
domain can be used. In the past, most video content was
stored using MPEG-1 Video, MPEG-2 Video, or MPEG-4
Visual, which resulted in the development of video
analysis algorithms only working on these specific coding
formats. As the H.264/AVC video coding standard [32]
performs significantly better than any prior standard in
terms of coding efficiency, it can be expected that a
significant amount of future video content will be
encoded in this format. The compression efficiency of
H.264/AVC can be attributed to a number of new coding
tools, which results in compressed video data having
specific characteristics. As a consequence, these charac-
teristics influence prior algorithms used in compressed
video analysis to a great extent.
In this paper, we propose a novel shot boundary
detection algorithm working directly on H.264/AVC bit
streams by relying on macroblock and sub-macroblock
types, and their related motion vectors and reference
picture indices. By only using compressed-domain in-
formation, this approach is characterized by a low
computational complexity. Furthermore, this algorithm
is enhanced to exploit hierarchical coding patterns, a new
coding pattern introduced in the H.264/AVC specification.
By exploiting the pyramidal coding structure, only a
subset of frames needs to be considered during analysis.
Consequently, the shot boundary detection step becomes
computationally less demanding, which is in line with the
idea of executing shot boundary detection in the com-
pressed domain.
The outline of the paper is as follows. Section 2
addresses related work in the area of shot boundary
detection. A discussion of the influence of H.264/AVC on
shot boundary detection algorithms for prior video coding
standards is also provided. Section 3 introduces our newly
developed shot boundary detection algorithm for H.264/
AVC bit streams, whereas Section 4 presents an enhance-
ment to this algorithm allowing to take into account
hierarchical coding patterns during shot detection. Section
5 provides performance results regarding the accuracy of
the proposed algorithm in terms of recall and precision.
This section also analyzes the decrease in complexity
originating from the enhanced approach. Finally, conclu-
sions are drawn in Section 6.
2. Related work
2.1. Existing algorithms
Algorithms for shot boundary detection can broadly be
classified into two major groups, depending on whether
the operations are performed in the pixel domain, or
whether they rely directly on compressed-domain fea-
tures. The most common algorithms that work in the pixel
domain are based on colour histogram differences,
changes in edge characteristics, and pixel differences
between successive frames [12,25,36]. Out of these
different methods, colour histogram-based algorithms
are considered to be the most reliable method for the
detection of abrupt transitions.
Full decompression of the video and the accompanying
computational overhead can be avoided by only using
compressed-domain features such as transform coeffi-
cients, macroblock types, or motion vectors. Hence, a
significant number of shot boundary detection algorithms
operate in the compressed domain. In the literature, most
of these algorithms work on MPEG-1 Video and MPEG-2
Video.
The concept of DC images is defined by Yeo and Liu
[34], where every 88 block is represented by the average
intensity of this block in the original image. For intra-
coded macroblocks, the DC coefficient of the block is
extracted as this represents the average energy. For inter-
coded macroblocks, these coefficients are predicted based
on the motion vector and the average intensity of the
referred blocks in the previous DC image. Based on these
DC images, shot detection algorithms based on colour
histogram differences can be modified to operate in the
compressed domain. Furthermore, the distribution of the
different macroblock types and motion information [29]
can also be used to detect shot boundaries. For example,
when an abrupt cut occurs at a P picture, it is expected
that most or all macroblocks are intra-coded since they
cannot be predicted well from prior reference frames.
Similarly, for an abrupt cut located at a B frame, it is
assumed that macroblocks are mainly intra-coded or
backward-predicted from future reference pictures.
Compared to abrupt changes, gradual transitions are
more difficult to detect; they take place over a variable
number of frames and can be generated using a great
variety of special effects. Consequently, the difference
between successive frames in a transition is substantially
reduced. Several techniques have been proposed in
literature for the detection of these gradual changes in
the pixel domain. In [36], Zhang et al. presented a twin-
comparison method. This approach takes into account the
cumulative differences between frames and requires two
thresholds: a higher threshold for detecting cuts and
 ARTICLE IN PRESS
S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489
a lower threshold for detecting gradual changes. As frames
are compared on a frame-to-frame basis by most shot
boundary detection algorithms, limited differences be-
tween some frames may result in undetected transitions.
An alternative is to compare every frame with the kth
following frames. When the transition is spread over more
than k frames, a plateau can be observed in the
dissimilarity measures [14,34]. However, selecting an
appropriate value for the parameter k is not trivial, as
each gradual change can have a different duration.
Although most of the proposed techniques for the
detection of gradual changes work on uncompressed
video, compressed-domain algorithms are gaining impor-
tance. Zhang et al. presented a compressed domain
approach in [37] based on DC coefficients and motion
vectors. These motion vectors are further examined to
distinct different types of camera motion and gradual
changes. In [1], Bescos disclosed a comparative study of
most of the metrics used for the detection of gradual
changes, both in the compressed and uncompressed
domain. He also proposed an algorithm for the detection
of abrupt as well as all types of gradual changes based on
DC images. A drawback of most approaches is that they
just focus on the detection of fades and dissolves, whereas
special editing effects such as wipes are hardly ever
considered.
2.2. Shot boundary detection on H.264/AVC bit streams
As mentioned in the introduction, H.264/AVC contains
a number of new coding tools compared to prior standards
for digital video coding [32]. Three of these features have a
major impact on existing shot boundary detection algo-
rithms: intra-prediction in the spatial domain, multi-
picture motion-compensated prediction, and decoupling
of display and referencing order.
Intra-prediction in H.264/AVC is conducted in the
spatial domain, by referring to neighbouring samples of
previously decoded blocks. Two primary types of intra-
coding are supported: Intra_44 and Intra_1616. In
the Fidelity Range Extensions of H.264/AVC, Intra_88 is
also introduced. Due to intra-prediction in the spatial
domain, DC coefficients in intra-coded pictures no longer
represent average energy, but only represent an energy
difference. Algorithms working on DC coefficients can
therefore no longer be applied to H.264/AVC bit streams.
The second feature, multiple reference picture motion
compensation, allows an encoder to use a larger number
of pictures as reference compared to previous standards.
Instead of only using the previous and following reference
picture for prediction, the encoder can choose from
multiple pictures that have been decoded and marked as
reference. In particular, the encoder is allowed to refer
further than the previous and following reference picture,
resulting in vagueness about random access in the bit
stream. Therefore, instantaneous decoding refresh (IDR)
pictures were introduced which indicate that no subse-
quent pictures in the bit stream will require references to
pictures prior to the IDR picture in decoding order. As the
prediction chain is broken, it is insufficient to only rely on
475
reference directions to detect shot boundaries. As a result,
algorithms developed for previous coding standards
cannot cope with the created gaps in the prediction chain
of H.264/AVC. Furthermore, to extract reference direc-
tions, a more complex approach is requisite as the display
numbers of the reference pictures first need to be
calculated in order to determine the reference direction.
Referencing order and display order are decoupled as
well, allowing the encoder to choose the ordering of
pictures with a higher degree of flexibility [32]. As a result,
new coding structures such as hierarchical coding pat-
terns are introduced [30]. In addition, there are no
restrictions on the coding type of reference frames and a
single coded picture may consist of a mixture of different
slice types. As such, the traditional concept of I, P, and B
frames is replaced by a highly flexible and general concept
that can be exploited by an encoder for different
purposes.1
Up to now, few publications report on shot boundary
detection algorithms working on H.264/AVC compressed
bit streams. In [35] and [26], an intra-mode histogram
distance based on the different intra-prediction mode
directions is presented to describe content changes in I
frames whereas reference directions are used for P and B
frames. In [22], the dissimilarity between I frames is
measured by relying on macroblock subdivisions (i.e.,
Intra_44 and Intra_1616 prediction); P and B frames
are not taken into consideration. A combination of both
approaches is proposed by the authors in [7], where
subdivisions of intra-coded macroblocks and reference
directions of inter-coded macroblocks are used during
analysis.
Whereas previously described algorithms for H.264/
AVC principally focus on the detection of abrupt changes,
[6] presents an algorithm for fade detection based on the
use of explicit weighting factors. This new coding tool in
H.264/AVC improves the coding efficiency of gradual
changes such as fades and dissolves [32]. However, a
drawback of algorithms relying on the use of luminance
weighted prediction factors is that not all encoders make
use of this feature; it is only advantageous for a limited
number of frames, while it increases coding complexity.
Furthermore, special effects during gradual changes, such
as wipes, do not benefit from this coding technique.
In theory, the set of motion vectors should also show
very typical behaviour during a gradual transition [37].
However, an increase in the amount of intra-coded
macroblocks as a result of content changes makes the
characteristics of the remaining motion vectors less
reliable. This rise is even strengthened in H.264/AVC by
the introduction of spatial prediction of intra-coded
macroblocks.
The approach presented in this paper is partially based
on previous work by the authors [7], where the analysis
relies on macroblock and sub-macroblock types, and their
related motion vectors and reference indices. In this paper,
1
In the context of H.264/AVC, we define an I frame as a frame that
consists entirely out of I slices. P and B frames are defined in a similar
way.
 ARTICLE IN PRESS
476 S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489
particular attention is paid to the analysis of I and IDR
frames, as these frames break the prediction chain.
Next, this algorithm is enhanced in order to exploit the
characteristics of hierarchical coding patterns. As elabo-
rated upon in Section 4, the coarse-to-fine structure leads
to the fact that a subset of lower layers can be seen as a
reduced version of the original video. Therefore, in the
enhanced algorithm, temporal dependencies between
pictures belonging to the base layer are first investigated
to decide whether they belong to the same shot or not.
Only in case this outcome is insufficient to accurately
detect shot boundaries, pictures belonging to higher levels
are further processed. As a consequence, only a subset of
pictures is taken into consideration during analysis,
resulting in a lower computational cost. Note that the
proposed algorithm can also be applied on conventional
coding structures such as IBP and IPP.
3. Shot boundary detection for traditional coding
patterns
In this section, algorithms for the detection of abrupt
and gradual transitions in traditional coding patterns will
be examined together with the accompanying selection of
thresholds. A flow chart of the algorithm is given at the
end in order to provide a complete overview.
3.1. Detection of abrupt transitions
Within a video sequence, a strong correlation exists
between successive frames. As an encoder typically tries
to exploit temporal and spatial correlation to a maximum
extent, this results in specific characteristics of motion
prediction information and macroblock subdivisions. By
analyzing these patterns, shot boundary detection can be
performed in the compressed domain.
For the detection of abrupt transitions, we propose to
examine the temporal dependencies between successive
frames. Thereafter, when the prediction chain is broken as
a result of certain I frames or IDR frames, spatial
dissimilarities are calculated as well.
3.1.1. Relying on temporal dependencies
P and B frames use temporal prediction to exploit
similarities with previously coded frames. However, when
the current frame is the starting frame of a new shot, this
frame will have hardly any resemblance with previously
displayed frames. As a consequence, the encoder will
prefer to make use of intra-coded macroblocks or macro-
blocks referring to following frames in display order
which are already decoded (called backward prediction).
An example is given in Fig. 1, where frame B2 is the first
frame of a shot in display order but not in decoding order.
The macroblocks in this frame will therefore mainly refer
backward to frame P3 . Frame P7 , which is the first encoded
and displayed frame of a new shot, will be mainly intra-
coded. On the other hand, the last frame of a shot in
display order will have hardly any correlation with
following frames. Therefore, this frame will mainly
contain macroblocks referring to previously displayed
0 1 2 3 4 5 6 7 8
Shot 1 Shot 2 Shot 3
Display order
0 3 1 2 5 4 7 6 8
Decoding order
Fig. 1. Example of a video sequence consisting of three shots. The arrows
represent the available reference frames; full arrows indicate frequently
used reference frames while the dashed arrows indicate the opposite.
frames (called forward prediction) or intra-coded macro-
blocks. This is the case for frames B1 and B6 .
In prior standards such as MPEG-1 Video and MPEG-2
Video, macroblocks could only be predicted from a single
previous (and a single following) reference frame. As a
result, the reference direction could directly be derived
from the macroblock types. As H.264/AVC allows multiple
reference picture motion compensation and decoupling of
referencing order and display order, macroblock types
only indicate which partitioning is applied and which
reference lists are utilized for each macroblock partition.
Based on the index in these reference lists, information
about the reference frame can be extracted. In particular,
information about the picture order count (POC) can be
utilized to determine the display number of a reference
frame. The reference direction of the macroblock partition
can then be determined by comparing the display
numbers of the current frame and the reference frame.
When the display numbers of all reference frames of a
partition are prior (resp. subsequent) to the current frame,
forward (resp. backward) prediction is used. When
reference frames are located before as well as after the
current frame, a partition is coded using bi-directional
prediction. As the smallest partition for which a reference
index can change is 88 pixels in size, this block size will
be used as the basic unit to calculate the reference
directions present in a frame.
These observations can be used to formulate two
conditions for the detection of shot boundaries. Let iðf i Þ,
jðf i Þ, bðf i Þ, and dðf i Þ, respectively, be the number of blocks
coded using intra-coding, forward prediction, backward
prediction, and bi-directional prediction of the current
frame f i ; let f i1 be the previous frame in display order
and let B denote the number of 8  8 blocks in a frame. By
relying on temporal dependencies, a shot boundary can be
declared at frame f i if the following condition is met:
1
ðiðf i1 Þ þ jðf i1 ÞÞ4T inter
B
1
^ ðiðf i Þ þ bðf i ÞÞ4T inter . (1)
B
Note that the percentage of bi-directional-predicted
blocks dðf i Þ is not used in the inequalities as a high value
for dðf i Þ typically corresponds to a low probability of a shot
boundary. When the percentages calculated in the
inequalities exceed a predefined threshold T inter , one can
conclude that an abrupt shot boundary is detected.
 ARTICLE IN PRESS
S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489
In Section 3.1.3, we will further elaborate on the
determination of optimal thresholds.
3.1.2. Relying on spatial similarities in case of I frames
As I frames are coded independently of other frames,
the second inequality in Condition 1 is always fulfilled.
When the previous frame in display order is only allowed
to use forward references, gaps in the temporal prediction
chain are introduced. In this case, both inequalities in
Condition 1 are met, resulting in a majority of falsely
detected shot boundaries. Indeed, every I frame that is not
used by previously displayed frames as reference, is then
seen as the start of a new shot. For classic I frames, this
phenomenon will typically occur when using IPP patterns,
or when variable GOP sizes are used. Furthermore, as a
result of the properties of an IDR frame, gaps in the
prediction chain will always occur for this particular type
of I frames. To overcome this problem, additional condi-
tions are required in case the prediction chain is broken as
a result of IDR frames and certain I frames.
As stated in Section 2.2, H.264/AVC supports several
intra-macroblock partitions, of which Intra_44 and
Intra_1616 are the most commonly used. The first
mode is generally preferred by an encoder in case of
significant detail, whereas the latter mode is more
suitable for coding smooth areas. As a result, the
subdivision of an I frame in different macroblock parti-
tions leads to a good representation of the detail of the
content, as can be seen in Fig. 2. By comparing the
distribution of the intra-macroblock types in the current
frame with previously displayed frames, dissimilarities
between frames can be calculated and therefore, changes
in content can be detected.
Comparing the current I frame with the previous I
frame is not recommended as the content can change
significantly between these two frames. For example, a
shot boundary can be located in between at a P or B frame,
or new objects can appear (Fig. 2). Therefore, an intra-
prediction map M i1 is constructed containing the last
intra-coded macroblock partitioning of each macroblock.
Every time an intra-coded macroblock has come across,
M i1 is updated with the new spatial information. This
map can then be used to represent the spatial distribution
of the content of the previous frame, in spite of the fact
that this frame contains inter-coded macroblocks.
By comparing the current I frame with the map M i1 ,
the dissimilarity between the current and previous frame
f10
I frame
Fig. 2. Distribution of Intra_44 and Intra_1616 macroblocks. Although the second frame is a P frame, it is mainly intra-coded as it is the first frame of a
new shot.
477
can be calculated. Instead of comparing macroblock types
at corresponding positions, a window of 33 macroblocks
is selected for each macroblock. This window is located
centrally around the current macroblock, as can be seen in
Fig. 2. This way, movement of objects or the camera will
lead to less false alarms.
Let f i denote the current I frame, M i1 the correspond-
ing intra-prediction map of the previous frame, and MB
the amount of macroblocks m in a frame. Define T as the
set of possible macroblock partitions ðT ¼ fIntra_44;
Intra_88; Intra_1616gÞ. Furthermore, let F be a place-
holder for f i and Mi1 , and n a macroblock in the
associated frame or intra-prediction map. The dissim-
ilarity metric O is defined as follows:
wFm;t ¼ fnjn 2 F ^ n 2 window associated with macroblock m
^ n is coded using partitioning mode tg
(2a)
P  
 fi i1 
t2T jwm;t j  jwM
m;t j
Wðf i ; mÞ ¼ , (2b)
f
2  jwmi j
1 X
Oðf i Þ ¼ Wðf i ; mÞ. (2c)
MB m2f
i
For I frames, when both inequalities in Condition 1 are
met and the dissimilarity Oðf i Þ also exceeds a predefined
threshold T intra , one can conclude that the current I frame
is the starting frame of a new shot. The next section
further elaborates on the determination of the threshold
T intra .
Note that for IDR frames, the gap in the temporal
prediction chain is not necessarily located at the IDR
frame, but can occur prior in display order. When the
decoded picture buffer is cleared as a result of the IDR
frame, frames located further in the bit stream but prior in
display order can still refer to the IDR frame, but not to
preceding frames in decoding order. As the amount of
frames influenced by an IDR frame is typically much larger
for hierarchical coding patterns than for traditional
structures, this will further be discussed in Section 4.2.
3.1.3. Threshold selection for abrupt transitions
A less extensively studied problem is the selection of
optimal thresholds for evaluating the computed frame-to-
frame differences. Most authors work with global thresh-
olds, which remain the same over the entire sequence.
window of 3x3
macroblocks
f 27 f 51
P frame I frame
 ARTICLE IN PRESS
478 S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489
Optimal values for global thresholds are often determined
by compromising between recall and precision ratios
[14,36]. An alternative is to work with adaptive thresh-
olds, which vary during the analysis and match the local
activity. The threshold for each frame is then based on the
frame-to-frame differences of surrounding frames located
in a corresponding search window. In general, maximum
values or mean and standard deviation of frame-to-frame
differences in a search window are used to determine the
local thresholds [15,34,36].
Selecting appropriate threshold values is a key issue in
successfully applying our proposed shot boundary detec-
tion algorithm. The detection of abrupt changes relies on
two thresholds T inter and T intra . The first threshold, which
is applied to both inequalities in Condition 1, is respon-
sible for detecting gaps in the temporal prediction chain.
For this purpose, the percentage of intra-coded, and
forward and backward referring blocks of the previous
and current frame are compared to T inter . As these
percentages represent the probability of a shot boundary
(from 0% to 100%), global thresholds can be applied [29].
The optimal value for T inter is heuristically chosen by
compromising between recall and precision ratios for
different threshold values. Experimental results shown in
Fig. 3 indicate that a wide range of threshold values (in
particular, the interval ½0:68; 0:88) is qualified for mini-
mizing the amount of missed detections and false alarms.
In order to select the optimal value from this interval, the
sum of the recall and precision curve is calculated, after
which the maximum of this curve is localized. As this
maximum corresponds to a threshold of 80%, this value is
used for T inter in the remainder of this paper. Note that for
determining the optimal values for other variables and
thresholds in the paper, the same methodology based on
recall and precision curves is applied.
To evaluate the spatial dissimilarity, T intra is used to
compare the distribution of the different macroblock
partitioning types in I frames. As the obtained results do
not represent probabilities, but rather indicate the
difference with previous frames, a more sophisticated
technique is requisite.
A technique often used in the literature is based on the
statistical distribution of the frame-to-frame differences
[15,36]. The obtained distribution is modelled by a
Gaussian function with parameters m and s which
Fig. 3. Influence of T inter on recall and precision ratios.
represent the mean and standard deviation of the frame-
to-frame differences. The corresponding threshold value
can be computed as
T ¼ m þ as, (3)
where a is the parameter related to the tolerated
probability for missed detections and false alarms. When
difference values fall out of the range 0 to m þ as, they can
be considered indicators of shot boundaries. Although this
threshold is based on the properties of the sequence, this
value remains the same for the entire sequence. Therefore,
it is not capable of anticipating on the properties of the
local content.
A second method using adaptive thresholds is based on
sliding windows [15,34]. For each frame k, a window
containing N elements is created where k is located in the
middle of the window. When the frame-to-frame differ-
ence between frame k  1 and k is the window maximum
and b times larger than the second largest difference
value, a shot boundary is detected. A major weakness of
this approach is the high sensitivity of b, originating from
motion.
In this paper, we have utilized a combination of both
approaches discussed above and applied several changes
to determine T intra . In particular, these techniques need to
be adjusted as they are mainly used in uncompressed-
domain algorithms, whereas we want to apply them on
compressed-domain information. First, as the spatial
dissimilarity can only be calculated for I frames, T intra is
not based on frame-to-frame differences; instead, only
difference values for I frames are considered. To adapt
T intra to the local properties of the content, a window
consisting of M elements is constructed. In contrast to the
above technique, all M elements are located before the
examined frame. By making this adjustment, I frames
located in the future and which are part of a high-motion
shot cannot cause missed detections. To control the
resulting false alarms, mean and standard deviation are
used. Let mO denote the mean of the dissimilarity values of
the M previous I frames and sO the corresponding
standard deviation. T intra can then be defined as
T intra ¼ mO þ asO . (4)
The values for M and a are computed heuristically in the
same way as for T inter, resulting in typical values lying
 ARTICLE IN PRESS
S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489
three intra-coded opening of intra-coded
objects macroblocks
Fig. 4. Extraction of foreground and background using the mathematical morphology operation opening.
around 5 and 3, respectively. Furthermore, an upper and
lower boundary is set to this threshold to avoid extreme
values. For example, when the start of a shot is static, the
corresponding dissimilarity values are close to zero.
A small amount of motion later in the shot would
therefore result in a FA. For high motion scenes, the
standard deviation could become too large, making the
T intra larger than its maximum. When observing typical
values for OðiÞ corresponding to abrupt transitions,
appropriate boundaries are ½0:2; 0:6.
3.2. Detection of gradual transitions
In this paper, the percentage of intra-coded macro-
blocks combined with motion vector information is used
as criterion for identifying gradual changes. First, a change
in the percentage of intra-coded macroblocks is used as an
indication for changes in content. When an increase in
this percentage is noticed, this can typically be attributed
to different events: abrupt changes, gradual changes, fast
local motion resulting from objects, or fast global motion
resulting from camera movement. To identify the gradual
changes from the abrupt changes and the motion, a
distinction between these different types of events needs
to be made. Abrupt changes can be identified by locating
gaps in the temporal prediction chain. Otherwise, motion
analysis is performed to distinguish gradual changes from
motion. The different steps of the algorithm are explained
below. A flow chart of this algorithm is given in Fig. 6 in
Section 3.3, containing a summary of the proposed
algorithm.
For content containing fast local motion, the moving
objects will typically be coded using a large amount of
intra-coded macroblocks, whereas the background re-
gions will mainly be coded using temporal prediction. For
gradual changes and fast global motion, the intra-coded
macroblocks are typically scattered over the entire frames.
By examining the motion intensity corresponding to the
intra-coded regions and inter-coded regions, a distinction
between the local motion, global motion, and gradual
changes can be made.
 Estimation of foreground and background: First, the
regions corresponding to the foreground and back-
ground are estimated. By calculating the mathematical
morphology operation opening [16] of the intra-coded
(resp. inter-coded) macroblocks, a rough estimation of
the foreground (resp. background) can be made, as
479
opening of inter-coded structure
macroblocks element
depicted in Figs. 4 and 5. By applying this operator, the
influence of isolated blocks can be ignored.
 Determination of motion intensity: As the estimated
foreground is intra-coded, the corresponding motion of
this region cannot be calculated from the same frame.
Therefore, all frames are divided into two groups: one
group which is used for the estimation of both regions,
and one to calculate the corresponding motion. As
frames marked as reference typically consist of a
higher amount of intra-coded macroblocks compared
to non-reference frames, the reference frames can be
used for the estimation of the foreground and back-
ground, whereas the latter type of frames can then be
used to measure the motion. This is for example the
case in Fig. 5 where the foreground and background are
determined from P 7 and where the motion intensity is
computed using B5 and B6 .
The corresponding motion intensity, a concept origi-
nating from the MPEG-7 specification [21], is then
calculated for both the foreground (MIF ðf i Þ) and back-
ground (MIB ðf i Þ). This is done by relying on the
standard deviation of the magnitude of the motion
vectors.2 The two obtained values for the foreground
and the background will therefore represent the
amount of motion present in both regions. Note that
as I and IDR frames cannot be used to make an
estimation of the foreground and background, the
previous reference frame is used as estimation instead.
 Distinction between motion and gradual changes: When
the calculated motion intensity of both foreground and
background is high, the change in content can most
likely be attributed to fast, global movement. High
motion intensity in only the foreground is typically
connected to local motion. Otherwise, when the
motion intensity is low although the amount of intra-
coded macroblocks indicates a change in content, the
origin is typically a gradual change. Note that relying
only on the motion intensity of an entire picture is
insufficient as the origin of intra-coded macroblocks is
unclear. More precisely, small moving objects can
result in an increase in intra-coded macroblocks, while
the percentage of corresponding motion vectors is too
low to influence the motion intensity of the complete
picture. Therefore, without separating foreground and
2
Note that each motion vector is first normalized in relation to the
difference between the display numbers since the distance to the
different reference frames can vary.
 ARTICLE IN PRESS
480 S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489
B5 B6
P4
intra-coded estimated estimated
macroblocks in P7 foreground of P7 background of P7
Fig. 5. Example of a sequence with local motion. The estimated foreground and background of P 7 is computed based on the opening of the intra-coded
and inter-coded macroblocks in P 7 . Using these estimations, the corresponding motion intensity of both regions in B6 is calculated.
background, it is difficult to distinguish gradual
transitions from local motion.
For the detection of gradual changes, two thresholds are
used. First, the amount of intra-coded macroblocks is
examined as an increase in the number of intra-coded
macroblocks can typically be attributed to a content
change. In order to anticipate the local properties of the
content, using an adaptive threshold is appropriate, as
explained in Section 3.1.3. Similar to the selection of T intra ,
only a certain amount of frames located before the current
frame is considered. When a content change starts (as a
result of a gradual transition or motion), the amount of
intra-coded macroblocks first increases, after which it
remains high for a certain time. When using future frames
as well, in combination with the original sliding window
technique, no real peak could be detected. However, by
comparing the percentage of intra-coded macroblocks in
the current frame with previous frames, the sudden
increase can be identified. As a result, the same technique
as for T intra is applied to calculate T grad . In particular, the
percentage of intra-coded macroblocks in the previous M
frames belonging to the base layer is used to compute the
mean mgrad and standard deviation sgrad . T grad can then be
defined as
T grad ¼ mgrad þ asgrad . (5)
Optimal values for M and a are computed heuristically in
the same way as for T inter, by compromising between
recall and precision curves obtained for different values
for M and a, resulting in typical values lying around 5 and
3, respectively. Again, in order to avoid extreme values,
this threshold is bounded to ½0:15; 0:60 for the same
reason as explained in Section 3.1.3.
Next, a conclusion must be drawn whether the change
in content is caused by motion or gradual transitions.
Therefore, the motion intensity of foreground and back-
ground is compared to the following threshold:
xl
T motion ¼ . (6)
F patterns. Although a hierarchical coding structure usually
P7
motion intensity of motion intensity of
foreground in B6 background in B6
The diagonal length l of a frame in pixels and the frame
rate F are applied to normalize the threshold. This
equation is defined in [18] to distinguish different degrees
in motion intensity. The value of x is selected equal to the
threshold dividing medium and high activity (i.e., 0.4267)
in [18]. Lower degrees of motion can typically be
compensated more easily by the encoder using temporal
prediction, which makes that these GOPs are not con-
sidered during the detection of gradual changes.
3.3. Summary of the proposed shot boundary detection
algorithm
To provide the reader with a complete overview of our
proposed shot boundary detection algorithm, a flow chart
is depicted in Fig. 6. First, temporal correlation between
successive frames is measured to detect gaps in the
temporal prediction chain (see Section 3.1.1). When this
gap is a result of certain I or IDR frames, the spatial
dissimilarity is calculated as well (as previously explained
in Section 3.1.2). This way, abrupt transitions can be
identified. Next, to locate the gradual changes, the
percentage of intra-coded macroblocks is compared to
prior frames. An increase can typically be attributed to
gradual changes as well as to local or global changes. By
analyzing the motion intensity of the estimated fore-
ground and background, these different types of content
changes can be distinguished.
4. Enhanced shot detection for hierarchical coding
patterns
In H.264/AVC, features like multiple reference picture
motion compensation and decoupling of display and
referencing order, among others, allow the creation of
arbitrary coding structures, which are not supported by
prior video coding standards. This flexibility in terms of
possible coding structures makes it possible to organize
the pictures in a bit stream in multiple ways. Often, this
flexibility is used for the creation of hierarchical coding
 ARTICLE IN PRESS
S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489 481

1
(ι ( fi −1 ) +  ( fi −1 )) > Tinter f i ∈ base layer
new frame fi if
B No if and No No transition
and
1
(ι ( fi ) + β ( fi )) > Tinter ι ( fi ) > Tgrad
B
Yes
Yes
Analysis of motion activity in
Gap in temporal foreground and background of
prediction chain intermediate non-reference frames fj

if gap is result of
MI F ( fj ) > Tmotion MI F ( f j ) < Tmotion
I or IDR frame
if MI B ( fj ) > Tmotion if and if and
Yes No MI B ( fj ) < Tmotion MI B ( f j ) < Tmotion
Yes Yes Yes
Detect abrupt
if Ω ( fi ) > Tintra
transition
fast global motion fast local motion
No Yes

Detect abrupt Detect gradual

No transition No transition No transition
transition transition

Fig. 6. Flow chart for the detection of the different types of content changes.

Layer 3 B1 B3 B5 B7

Layer 2 B2 B6

Layer 1 B4

Layer 0 I0 P8

Display order

I0 P8 B4 B2 B6 B1 B3 B5 B7

Decoding order

Fig. 7. Hierarchical coding pattern with four temporal layers.

introduces a higher end-to-end delay, it can improve
coding efficiency and offers multi-layered temporal scal-
ability in a straight-forward way [30,23]. Hierarchical
coding patterns consist of multiple layers, which result in
a coarse-to-fine structure, as shown in Fig. 7. The base
layer typically consists of I and P frames at a very low
frame rate, whereas the higher layers typically contain B
frames that are inserted between the frames of lower
layers in display order. Frames belonging to higher layers
can thus use frames of lower layers as references for the
decoding process (and possibly some preceding frames of
the current layer also).
To identify the hierarchical coding structure of a bit
stream, it is possible to rely on three supplemental
enhancement information (SEI) messages [8,19], which
are defined within the scope of sub-sequences and sub-
sequence layers [31]. When applied to hierarchical coding
patterns, a sub-sequence layer corresponds to a temporal
layer whereas a sub-sequence is a set of coded pictures
within a sub-sequence layer. The sub-sequence informa-
tion SEI message maps a picture to a subsequence and
sub-sequence layer. The sub-sequence layer characteris-
tics SEI message and the sub-sequence characteristics SEI
message provide statistical information (e.g., average bit
rate) on the indicated sub-sequence layer and sub-
sequence. When these messages are not inserted into
the bit stream, relying on the decoding order and display
order of the pictures is also feasible to detect the
hierarchical coding structure used. However, this solution
is more complex.
4.1. Exploitation of the pyramidal structure
Conventional algorithms for shot boundary detection
can be optimized by taking into account layered coding
 ARTICLE IN PRESS
482 S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489
structures. As a subset of lower layers can be seen as a
reduced version of the original video, pictures belonging
to higher layers only need to be considered when the
amount of information from the lower layers is insuffi-
cient to draw conclusions. First, temporal dependencies
between pictures belonging to the first layer are investi-
gated to decide whether they belong to the same shot or
not. Based on the outcome, pictures belonging to higher
layers are further processed in a recursive way or are
immediately discarded.
4.1.1. Abrupt transitions
Our algorithm for the detection of abrupt transitions is
described in more detail using the pseudo-code provided
below; it is executed for every sub-GOP in a bit stream.
This sub-GOP corresponds to two successive reference
frames in the base layer and the intermediate frames
located in higher levels.
Algorithm 1 RecursiveShotBoundaryDetection (startFrame, endFrame)
if ð1B ðiðstartFrameÞ þ jðstartFrameÞÞ4T inter ^
1 examination necessary. It is clear that we need to rise to
B ðiðendFrameÞ þ bðendFrameÞÞ4T inter Þ
// Temporal prediction chain is broken
if ðstartFrame þ 1 ¼ endFrameÞ
// Highest level reached
// Shot boundary detected
NewShotBoundary(endFrame)
else
// Go one level higher in the pyramidal structure
// to find the shot boundary
middleFrame ¼ intermediate frame in next level
RecursiveShotBoundaryDetection (startFrame, middleFrame)
RecursiveShotBoundaryDetection (middleFrame, endFrame)
end if
else
// No shot boundary detected
end if
The use of our recursive technique is shown in more detail
in Fig. 8, where a hierarchically coded sub-GOP is
displayed containing frames from two different shots. As
B1 B3
B2
B4
I0
shot boundary
: low temporal correlation
: not considered
Fig. 8. Recursive algorithm for detecting shot boundaries in a hierarchically coded sub-GOP.
explained below, only a subset of frames needs to be
examined in order to detect the abrupt transition.
As I0 and P 8 belong to different shots, P 8 will typically
not use I0 as a reference. Furthermore, the other frames
located after the shot boundary (i.e., B3 to B7 ) cannot be
used as a reference either since P 8 is coded before the
other frames, as illustrated in Fig. 7. As a result, the
percentage of intra-coded macroblocks in P 8 is high,
fulfilling the first if-statement in Algorithm 1. As a
consequence, the reference directions of the intermediate
frame B4 in the next layer need to be considered. For
this purpose, the same algorithm is executed for I0  B4
and B4  P 8 . As B4 and P8 belong to the same shot and
P 8 is present in the reference picture buffer of B4 , B4 is
mainly backward-predicted using P8 . Therefore, the
first part of the if-statement is not fulfilled, making
further investigation of frames located in between B4
and P 8 superfluous. The same technique is applied to
I0  B4 . As illustrated in Fig. 8, the shot boundary is
located in between these two frames, making further
the highest layer of the pyramidal structure using the
proposed technique in order to detect the exact location of
the shot boundary. Even then, a significant amount of
frames can be discarded, resulting in increased computa-
tional efficiency. This is for example the case for B1, B5 , B6 ,
and B7 .
When the content of a sub-GOP does not change
drastically, only the first level needs to be examined. On
the other hand, significant motion or gradual changes will
typically lead to the investigation of pictures residing at a
higher level. When the difference between pictures
residing at the lowest layer is relatively high, the
prediction can fail. However, when rising in the pyramidal
structure, the correlation between the frames becomes
more clear. Therefore, the recursive algorithm for the
detection of abrupt transitions is stopped once the
similarity is noticed.
B5 B7
B6
P8
Display order
: high temporal correlation
 ARTICLE IN PRESS
S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489
4.1.2. Gradual transitions
For the detection of gradual transitions in H.264/AVC
bit streams with hierarchical coding patterns, this bot-
tom-up approach can also be applied to enhance the
algorithm proposed in Section 3.2. First, the percentage of
intra-coded macroblocks in the base layer is compared to
the threshold T grad , which is calculated from previous
frames residing in the base layer. Only when the threshold
is exceeded, the next layer is examined as a result of the
content change. Based on the estimation of the foreground
and background derived from the base layer (i.e., f 24 in
Fig. 9), the motion intensity is calculated for the
intermediate frame in the second layer (i.e., f 22 ). As
previously explained in Section 3.2, this procedure allows
to make a distinction between fast global motion, fast
local motion, and gradual changes. However, as the
distance between the compared frames is larger than in
traditional coding patterns, the amount of intra-coded
macroblocks in higher layers can still be relatively large.
When a significant amount of co-located macroblocks is
still intra-coded, this procedure is repeated for higher
layers to make a reliable estimation of the motion
intensity. The estimation of foreground and background
is then executed for the current layer, whereas the motion
intensity is derived from the next layer. This is in line with
the idea used for the detection of abrupt shots, where only
these intervals are further investigated where the infor-
mation to draw conclusions about the temporal decom-
position is insufficient.
To exploit the hierarchical coding structures for the
detection of gradual changes, only one threshold needs to
be added. In particular, a new threshold T nextLayer is
introduced which decides whether higher layers need to
be considered to make a good estimation of the motion
intensity, or whether information from current frame is
enough to draw conclusion. Experimental results show
that when more than 50% of the macroblocks of the
considered regions are inter-coded, a good estimation can
be made.
f 21
f 22
f 20
390
> Tgrad
396
Fig. 9. Example of a gradual transition in a hierarchical coding structure. Intra-coded macroblocks are represented by their original colour, whereas inter-
coded macroblocks are blanched.
483
4.2. Spatial dissimilarity for IDR frames
As stated in Section 3.1.2, IDR frames break the
temporal prediction chain of a video sequence as they
indicate that no subsequent frame in the bit stream in
decoding order is allowed to use frames prior to the IDR
picture as a reference. As a result of the pyramidal
structure, this gap is located multiple frames prior to the
IDR frame in display order. An example is shown in Fig. 10,
where frames B33 to B39 are located before the IDR frame
in display order, but after the IDR frame in decoding order.
When only taking into account temporal dependencies
during shot boundary detection, an abrupt transition
between P32 and B33 will typically be falsely detected as
B33 is not allowed to use P32 as a reference.
In order to calculate the similarity between P 32 and B33 ,
spatial distributions need to be examined instead of
temporal dependencies. For the previous frame, the
intra-prediction map M32 introduced in Section 3.1.2 is
utilized. However, as B33 can be predicted from following
frames, the corresponding spatial distribution needs
to be derived as well. Therefore, a second intra-prediction
map M IDR;i is constructed containing a spatial representa-
tion of B33 . This map is composed of the intra-prediction
modes of following frames in display order. Initially,
this map contains the intra-prediction modes of IDR40 ,
which is then updated with intra-prediction modes
of macroblocks located closer to B33 . As this process
takes place during the analysis of the different layers in a
bottom-up approach, IDR40 , B36 , B34 , and B33 are used
during the update step. When a shot boundary takes
place within this sub-GOP, one of these frames will be
mainly intra-coded, which results in overwriting the
prediction modes of IDR40 . Consequently, M IDR;33 will
represent the spatial distribution of the content of B33 . To
calculate the correlation between P 32 and B33 , a new
dissimilarity metric OIDR ðiÞ is introduced. This metric is
based on Eqs. (2a)–(2c) for the intra-prediction maps M i1
and MIDR;i .
f 23
f 24
 ARTICLE IN PRESS
484 S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489
B33
B34
intra prediction map M32
B20
P16 P24 P32
P32 IDR40 B36
: temporal prediction prohibited
Fig. 10. The use of IDR frames results in a temporal prediction chain that is broken, as no subsequent frame in decoding order is allowed to use frames
prior to the IDR frame as reference.
During the construction of both maps, only the subset
of frames is used that is needed for the analysis of the
temporal dependencies. The other frames, which are not
considered, will typically contain very little spatial
information as they are located between highly similar
frames. Therefore, discarding these frames does not
influence the accuracy of the dissimilarity measurement.
Note that M i is kept up to date during the analysis of the
complete bit stream, whereas MIDR;i is only constructed for
sub-GOPs containing IDR frames.
As already declared in Section 3.1.2, this problem also
arises for traditional coding patterns. However, explana-
tions are provided in this section as the impact of IDR
frames is larger for hierarchical coding patterns since
more frames are involved.
4.3. Summary of the enhanced shot boundary detection
algorithm working on hierarchical coding patterns
To give a coherent overview of the algorithm working
on hierarchical coding patterns, a summarizing flow
chart is depicted in Fig. 11. This algorithm is executed
for every sub-GOP in a video sequence, which corresponds
to two successive frames belonging to the base
layer (further referred to as start frame and end frame),
and the intermediate frames located at higher layers.
Note that the end frame in the current sub-GOP
will therefore correspond to the start frame in the next
sub-GOP.
First, the algorithm responsible for detecting abrupt
transitions is applied. As elaborated upon in Section 4.1,
the temporal correlation between the start and end frame
is investigated to decide whether they belong to the same
shot or not. Based on the outcome, this algorithm is
recursively executed for higher layers or immediately
stopped. When arriving at the highest layer and the
temporal correlation is still very low, the probability of a
shot boundary is high. Only in case this gap in the
B35 B37 B39
B38
B36
IDR40
Display order
B34 B38 B33 B35 B37 B39
Decoding order
: temporal prediction allowed
temporal prediction chain is caused by an IDR frame or
certain I frames, the spatial dissimilarity is calculated as
well, as explained in Section 4.2. As a consequence, the
amount of FAs can be reduced.
Next, the algorithm responsible for detecting gradual
transitions is performed. In case the percentage of intra-
coded macroblocks in the end frame increases signifi-
cantly, T grad is exceeded, which indicates a change in
content. To discover the origin of this change (i.e., gradual
transition, global motion, or local motion), the foreground
and background of this frame are estimated, as discussed
in Section 4.1. For both regions, the corresponding motion
intensity in the intermediate frame belonging to the next
layer is calculated. Using T motion , the source of the content
change can be determined. In particular, when there is an
increase in intra-coded macroblocks although the motion
intensity is low, a gradual transition is detected. In case
the content change is very drastic, it is possible that
motion information from the current frame is insufficient
since the high amount of intra-coded macroblocks
gives a distorted view of the motion intensity. Therefore,
when this percentage exceeds T nextLayer , this technique is
performed in a recursive way on the frames located in the
next layer.
5. Performance results
To evaluate the performance of our shot boundary
detection algorithm, experiments have been carried out
on several video sequences with various characteristics in
terms of resolution, length, quality, and content. An
overview of the characteristics of the test sequences
used is given in Table 1. The first two sequences
are obtained from the publicly available MPEG-7 Content
Set [17] and represent a part of a news sequence
and a basketball sequence (i.e., V3 and V17 from [17]).
Since the quality and the resolution of these sequences
is low, three recent, proprietary sequences were added
 S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489 485

new sub-GOP
(startFrame,
endFrame) Abrupt transitions
1
(ι (startFrame ) + ϕ (startFrame )) > Tinter
Update Mi with B No transition in
if and No
startFrame sub-GOP
1
(ι (endFrame ) + β (endFrame )) > Tinter
B
Yes

parse intermediate frame in if (startFrame + 1 if gap is result of Construct No

No Yes Yes
next layer and update Mi = endFrame) I or IDR frame MIDR,endFrame

No

recursively repeat for

Detect abrupt
[startFrame, middleFrame] Yes if Ω IDR (endFrame ) > Tintr a
transition
and [middleFrame, endFrame]

Analysis of motion activity in if #(intra-coded MB in

parse intermediate
if ι (endFrame ) > Tgrad Yes foreground and background foreground or
frame in next layer
of middle frame fi background) > TnextLayer
No
Yes No
recursively repeat for
No transition in
[startFrame, middleFrame]
sub-GOP
and [middleFrame, endFrame] MIF ( fi ) > Tmotion MIF ( fi ) < Tmotion
if MIB ( fi ) > Tmotion if and if and
MI B ( fi ) < Tmotion MIB ( fi ) < Tmotion
Yes Yes Yes

Gradual transitions Detect gradual

No transition in No transition in
transition in
sub-GOP sub-GOP
sub-GOP

Fig. 11. Flow chart of the enhanced algorithm for the detection of the different types of content changes on hierarchical coding patterns.

Table 1
Characteristics of the test sequences

Name Resolution # Frames Frame rate (Hz) Length (s) # Transitions

Abrupt Gradual

News 1 352  288 26 000 25 1040.0 154 18

Basket 352  288 18 053 25 722.1 62 13
News 2 384  208 23 802 25 952.0 138 19
Soap 720  576 15 040 25 601.6 160 7
Trailer 848  352 3553 25 142.1 81 24

to the test set as well. The first sequence originates from a
news broadcast from Belgian public television, the second
sequence is part of an international television soap, and
the last sequence is the movie trailer of ‘‘Little miss
sunshine’’.3
These test sequences were coded a number of times
with different hierarchical coding patterns (i.e., two, three,
and four temporal layers). These configurations correspond
to a pyramidal structure containing two (hier_2), four
(hier_4), and eight frames (hier_8), respectively. Further-
more, these different coding patterns were generated two
times: once with I frames and once with IDR frames, which
were inserted every 32 frames. A distinction between these
two types is made as IDR frames lead to gaps in the
temporal prediction chain, which is not always the case for
classic I frames, as discussed in Section 3.1.2.
First, the accuracy of the proposed algorithm is
evaluated for different coding parameters, based on the
manually created ground truth. Next, these results are
compared to a publicly available uncompressed-domain
algorithm. Thereafter, a complexity analysis of the
enhanced algorithm for hierarchical patterns is given.
5.1. Accuracy of the proposed algorithm for shot boundary
detection

The accuracy of the proposed algorithm is evaluated by

3
This sequence can be found on the website of Apple. comparing the obtained results against the ground truth.
 ARTICLE IN PRESS
486 S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489

This comparison is based on the number of correct Table 2

detections (D), missed detections (MD), and false alarms Accuracy in precision (P) and recall (R) (%) of the proposed algorithm on
sequences coded with different hierarchical structures and I or IDR
(FA), expressed as recall and precision.
frames inserted every 32 frames. Additionally, the accuracy of the ‘‘TZI
D D Shotdetection TrecVID 2004’’ algorithm is depicted as comparison
Precision ¼ ; Recall ¼ . (7)
D þ FA D þ MD Name Coding pattern Abrupt Gradual
Table 2 presents the accuracy of our proposed algorithm.
P R P R
As can be deduced from the results, the accuracy for the
detection of abrupt transitions is very high. Only in case of News 1 I frames
IDR frames, a decrease in precision is observed which can hier_8 96 100 48 61
be attributed to the gaps in the temporal prediction chain hier_4 93 100 47 50
hier_2 91 100 75 67
and the intra-prediction map for calculating the spatial
dissimilarity. These FAs occur when the content slightly
IDR frames
changes as a result of slow motion. Since this motion can hier_8 91 99 53 77
generally be compensated using inter-prediction, the hier_4 88 99 40 44
intra-prediction map used for calculating the spatial hier_2 87 100 75 67
dissimilarity will not be updated accordingly. Conse-
quently, the intra-prediction map does no longer give a TZI 93 97 35 50
good representation the spatial distribution. This phe-
nomenon occurs less often for larger hierarchical struc- Basket I frames
hier_8 95 100 60 23
tures as the base layer typically contains more intra-coded
hier_4 91 100 50 15
macroblocks, which results in a more accurate intra- hier_2 94 98 60 23
prediction map. Note that the intra-prediction map is not
used in the test sequences coded with classic I frames, IDR frames
since the intermediate B frames prevent gaps in the hier_8 90 100 75 23
temporal prediction chain. Therefore, the accuracy of hier_4 90 98 50 15
hier_2 83 84 63 38
using intra-prediction maps can deduced from Table 2 by
comparing the precision and recall values for sequences
TZI 97 93 31 76
containing classic I frames with sequences containing IDR
frames.
News 2 I frames
For the gradual transitions, all false alarms are caused hier_8 99 100 76 84
by sudden changes in light intensity as the amount of hier_4 100 100 90 95
intra-coded macroblocks increases drastically, but no hier_2 98 100 100 95
significant motion is present. The missed gradual transi-
tions can for the major part be attributed to transitions IDR frames
taking place of a long period of time or to high motion hier_8 100 100 76 84
hier_4 93 98 82 95
present during the transition. Our proposed algorithm hier_2 80 99 95 95
falsely classifies these content changes as global motion,
instead of gradual changes. TZI 91 99 76 84
To compare the results of our algorithm with publicly
available algorithms, we employed the ‘‘TZI Shotdetection Soap I frames
TrecVID 2004’’ algorithm and added the accuracy results hier_8 99 100 36 57
to Table 2. This algorithm is developed within the scope of hier_4 100 100 58 100
the DELOS Network of Excellence [9], relies on uncom- hier_2 99 100 71 71

pressed-domain features for the detection of shot bound-

IDR frames
aries [20], and is also used in [14] for comparison
hier_8 99 100 43 86
purposes. When comparing the results of TZI with the hier_4 93 100 64 100
results of our algorithm in Table 2, it can be seen that our hier_2 83 99 71 71
proposed algorithm achieves a higher accuracy for the
detection of abrupt transitions. Although uncompressed- TZI 99 91 50 57
domain algorithms can rely on more features, our
algorithm exploits the decisions made by the encoder Trailer I frames
during its search for similarity with preceding frames, hier_8 100 99 92 96
hier_4 99 100 96 96
which is a computational intensive process. For gradual
hier_2 100 100 88 96
transitions, both algorithms struggle with the same
problems. Our proposed algorithm has the advantage that IDR frames
it uses features available in the compressed domain hier_8 99 98 96 96
instead of pixel-domain information, making full decom- hier_4 100 98 96 96
pression unnecessary. hier_2 100 100 92 96
Note that a pyramidal structure containing two frames
corresponds to a traditional IðBPÞ pattern. Compared to TZI 95 99 96 96
 ARTICLE IN PRESS
S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489
the algorithm for traditional coding patterns proposed in
Section 3, the enhanced algorithm proposed in Section 4
only investigates the second layer when the amount of
information from the base layer is insufficient to draw
conclusions. As examining the second layer corresponds
to investigating all layers in a traditional coding pattern,
the accuracy of both algorithms is the same for traditional
coding patterns. Therefore, the results obtained for the
proposed algorithm for traditional coding patterns are
reflected in the results for pyramidal coding structures
containing two frames.
5.2. Complexity analysis of the proposed shot boundary
detection algorithm
The main advantage of the enhanced algorithm for shot
boundary detection is the decreasing computational
complexity. As already stated in Section 4, only a subset
of frames present in the video is needed to detect shot
boundaries in the compressed domain. In this section,
the influence of the different coding parameters on the
complexity of the enhanced algorithm is discussed. The
type of content is taken into account as this also plays an
important role. Finally, the difference in complexity of
compressed- and uncompressed-domain shot boundary
detection algorithms is discussed.
As can be seen from Table 3, the percentage of frames
that needs to be analyzed during shot boundary detection
decreases when more temporal layers are included.
This results from the fact that the number of frames
present in the lowest layer decreases. Despite this strong
decrease in amount of processed frames, the execution
times decrease less rapidly. In some cases, one can even
notice an increase in execution speed for sequences
consisting of multiple layers. This can mainly be attrib-
uted to the fact that more frames from higher layers need
to be parsed when using more temporal layers. In
particular, frames located in the base layer need less time
to be parsed and processed as they typically use only one
reference, instead of two references for frames located in
higher levels.
Sequences containing IDR frames will generally need
more processing power than sequences with I frames as
spatial dissimilarity needs to be calculated as well. This
process becomes more expensive when the number of
temporal layers increases. In particular, more intermedi-
ate frames are necessary to create the second intra-
prediction map M IDR;i .
To illustrate the gain in efficiency, the execution speed
of the enhanced algorithm (where only the necessary
frames are parsed and analyzed) is compared to an
adapted version of this enhanced algorithm where all
frames are parsed. This is illustrated in Fig. 12 for the test
sequences containing I frames, where the relative execu-
tion speed of the enhanced algorithm is depicted. Indeed,
this graph represents the time needed to execute the
enhanced algorithm compared to the time needed to
perform the algorithm where all frames are parsed and
processed. From these measurements, it can be concluded
that a significant gain in execution time and therefore
487
efficiency can be achieved by only considering a well-
defined subset of frames present in the bit stream.
Furthermore, as can be seen from the percentage of
analyzed frames in Table 3 and the relative execution
speed in Fig. 12, the type of content has a major impact on
the efficiency. For video sequences with long shots and
little motion such as news sequences, the gain in
efficiency is very high. In this case, frames in the lowest
layer can often use each other as reference since the
content remains very similar over a long period of time. As
a result, higher layers only need to be examined rarely. On
the other hand, high-motion video streams such as sport
sequences demand more processing power. Since high
motion drastically changes the content of the video, the
similarity between pictures in the lowest layer is small
and the amount of intra-coded macroblocks increases
significantly. As this latter characteristic is also present
when gradual changes occur, higher layers need to be
examined in order to compute the motion intensity and to
distinguish motion from gradual changes. Although the
‘‘soap’’ and the ‘‘trailer’’ sequence do not contain a
significant amount of motion, the results show that their
complexity is also higher than the complexity of news
sequences. This can be attributed to the fact that the
average length of the shots is much smaller. As shot
boundaries occur more frequently, higher layers need to
be examined more often. To determine the exact location,
it is necessary to rise to the highest layer in the pyramidal
structure. As a result, the amount of frames that needs to
be processed highly increases. The same trends can be
observed for sequences containing IDR frames. Note that
the resolution of the video sequences also plays an
important role since the execution speed is inversely
proportional to the size of the content.
It would be interesting to compare the complexity
results of the proposed algorithm with uncompressed-
domain algorithms, and in particular the TZI algorithm.
However, making a decent comparison in terms of
complexity is non-trivial. To obtain a fair comparison
between the execution speeds of both algorithms, the
same code base should be used since the underlying
implementation determines the results to a great extent.
At best, this means that the parser used in our proposed
algorithm is part of the complete decoder used by
the uncompressed-domain algorithm before starting the
analysis phase. Even then, the implementation of the
remaining part of the decoder influences the complexity.
As a consequence, comparing the complexity in terms of
execution speed is very questionable, and hence, not
incorporated here.
However, one can clearly recognize that algorithms
working on compressed data are intrinsically less com-
plex. In particular, several time-consuming steps from the
decoding process such as entropy decoding, motion
compensation, inverse quantization, and inverse transfor-
mation do not need to be executed in our algorithm. Even
the parsing of a well-defined subset of frames is
eliminated.
Of course, the complexity of the algorithms themselves
also plays an important role. Uncompressed-domain
algorithms often compute information about edges and
 ARTICLE IN PRESS
488 S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489

Table 3
Complexity results for the enhanced shot detection algorithm

Name Coding pattern I frames IDR frames

Size (MB) Execution time (s) Analyzed frames (%) Size (MB) Execution time (s) Analyzed frames (%)

News 1 hier_8 141.0 83.2 26.3 144.9 114.2 23.8

hier_4 136.1 89.9 35.9 139.3 100.6 35.9
hier_2 129.5 125.8 58.7 131.3 125.3 58.6

Basket hier_8 118.4 57.8 32.7 120.7 72.3 29.7

hier_4 114.6 61.2 40.4 116.4 67.6 40.5
hier_2 109.7 84.4 60.9 110.7 84.9 60.8

News 2 hier_8 43.3 48.7 20.0 45.5 73.6 20.0

hier_4 43.7 55.9 31.4 45.0 65.4 31.5
hier_2 46.8 82.3 55.0 47.4 82.8 55.0

Soap hier_8 95.6 302.4 30.3 99.0 377.1 30.1

hier_4 91.1 280.9 38.7 93.5 295.9 38.9
hier_2 71.9 341.9 59.5 86.3 340.3 60.5

Trailer hier_8 14.1 56.0 29.4 14.9 67.0 30.9

hier_4 13.7 45.5 42.4 14.2 47.9 40.0
hier_2 13.5 58.0 60.5 13.7 58.0 60.7

50
Relative execution time (%)

45
40
35
30
25
20
15
10
5
0
Hier_8

Hier_4

Hier_2

Hier_8

Hier_4

Hier_2

Hier_8

Hier_4

Hier_2

Hier_8

Hier_4

Hier_2

Hier_8

Hier_4

Hier_2

News 1 Basket News 2 Soap Trailer

Fig. 12. Relative execution speed of the enhanced algorithm. These values represent the time needed to execute the enhanced algorithm compared to the
adapted version.

motion intensity to increase their accuracy. Compressed- a novel algorithm for shot boundary detection on H.264/
domain algorithms, on the other hand, can directly extract AVC bit streams. To identify abrupt transitions, the
this information from the bit stream as these character- algorithm examines the temporal dependencies between
istics are already exploited during the motion estimation successive frames. Since IDR frames and certain I frames
step and the search for optimal partitioning of macro- lead to gaps in this prediction chain, spatial dissimilarities
blocks in the encoding phase. As a result, the actual are considered as well. Gradual changes are detected by
analysis step of our compressed-domain algorithms is less relying on the amount of intra-coded macroblocks and
complex as well. motion intensity. Furthermore, an enhanced algorithm is
introduced for shot boundary detection on H.264/AVC bit
streams with hierarchical coding patterns. As these coding
6. Conclusions structures consist of multiple layers, a subset of lower
layers can be seen as a reduced version of the original
Temporal segmentation is a prerequisite for semantic video. Therefore, pictures residing in higher layers are
video analysis. Therefore, in this paper, we have proposed only considered in case information from lower layers is
 ARTICLE IN PRESS
S. De Bruyne et al. / Signal Processing: Image Communication 23 (2008) 473–489
insufficient to accurately detect shot boundaries. Experi-
mental results show that this algorithm achieves a high
accuracy in terms of recall and precision. Furthermore, by
exploiting the layered structure, the computational com-
plexity is drastically reduced, resulting in an increased
efficiency.
Acknowledgements
The authors would like to thank Davy De Schrijver for
his constructive and extensive feedback.
The research activities that have been described in this
paper were funded by Ghent University, the Interdisci-
plinary Institute for Broadband Technology (IBBT), the
Institute for the Promotion of Innovation by Science
and Technology in Flanders (IWT-Flanders), the Fund
for Scientific Research-Flanders (FWO-Flanders), and the
European Union.
References
[1] J. Bescós, Real-time shot change detection over online MPEG-2
video, IEEE Trans. Circuits Systems Video Technol. 14 (4) (2004)
475–484.
[2] G. Boccignone, A. Chianese, V. Moscato, A. Picariello, Foveated shot
detection for video segmentation, IEEE Trans. Circuits Systems
Video Technol. 15 (3) (2005) 365–377.
[3] I.S. Burnett, F. Pereira, R. Van de Walle, R. Koenen, The MPEG-21
Book, Wiley, New York, 2006.
[4] J. Ćalić, D.P. Gibson, N.W. Campbell, Efficient layout of comic-like
video summaries, IEEE Trans. Circuits Systems Video Technol. 17 (7)
(2007) 931–936.
[5] S.-F. Chang, A. Vetro, Video adaptation: concepts, technologies and
open issues, Proc. IEEE 93 (1) (2005) 148–158.
[6] B. Damghanian, M.R. Hashemi, M.K. Akbari, A novel fade detection
algorithm on H.264/AVC compressed domain, in: Lecture Notes in
Computer Science, vol. 4319, Springer, Berlin, 2006, pp. 1159–1167.
[7] S. De Bruyne, W. De Neve, K. De Wolf, D. De Schrijver, P. Verhoeve,
R. Van de Walle, Temporal video segmentation on H.264/AVC
compressed bitstreams, in: Lecture Notes in Computer Science, vol.
4351, Springer, Berlin, 2007, pp. 1–12.
[8] W. De Neve, D. Van Deursen, D. De Schrijver, K. De Wolf, R. Van de
Walle, Using bitstream structure descriptions for the exploitation of
multi-layered temporal scalability in H.264/AVC’s base specifica-
tion, in: Lecture Notes in Computer Science, vol. 3767, Springer,
Berlin, 2005, pp. 641–652.
[9] DELOS Network of Excellence on Digital Libraries hhttp://www.
delos.info/i.
[10] A. Divakaran, K.A. Peker, R. Radhakrishnan, Z. Xiong, R. Cabasson,
Video summarization using MPEG-7 motion activity and audio
descriptors, Technical Report TR-2003-34, Mitsubishi Electric
Research Laboratories, 2003.
[11] P.M. Fonseca, F. Pereira, Automatic video summarization based on
MPEG-7 descriptions, Signal Processing: Image Communications 19
(8) (2004) 685–699.
[12] U. Gargi, R. Kasturi, S.H. Strayer, Performance characterization of
video-shot-change detection methods, IEEE Trans. Circuits Systems
Video Technol. 10 (1) (2000) 1–13.
[13] A. Girgensohn, J. Boreczky, Time-constrained keyframe selection
technique, Multimedia Tools Appl. 11 (3) (2000) 347–358.
489
[14] C. Grana, R. Cucchiara, Linear transition detection as a unified shot
detection approach, IEEE Trans. Circuits Systems Video Technol. 17
(4) (2007) 483–489.
[15] A. Hanjalic, Shot-boundary detection: unraveled and resolved?,
IEEE Trans. Circuits Systems Video Technol. 12 (2) (2002) 90–105.
[16] H.J.A.M. Heijmans, Composing morphological filters, IEEE Trans.
Image Process. 6 (5) (1997) 713–723.
[17] ISO/IEC JTC1/SC29/WGll/N2467, Description of MPEG-7 content set,
1998.
[18] ISO/IEC 15938-3, Information technology—multimedia content
description interface—Part 3: visual, 2002.
[19] ITU-T and ISO/IEC JTC 1, Advanced video coding for generic
audiovisual services, ITU-T Rec. H.264 and ISO/IEC 14496-10 AVC,
2003.
[20] A. Jacobs, A. Miene, G.T. Ioannidis, O. Herzog, Automatic shot
boundary detection combining color, edge, and motion features of
adjacent frames, in: TRECVID 2004 Workshop Notebook Papers,
2004, pp. 197–206.
[21] S. Jeannin, A. Divakaran, MPEG-7 visual motion descriptors, IEEE
Trans. Circuits Systems Video Technol. 11 (6) (2001) 720–724.
[22] S.M. Kim, J. Byun, C.S. Won, A scene change detection in H.264/AVC
compression domain, in: Lecture Notes in Computer Science, vol.
3768, Springer, Berlin, 2005, pp. 1072–1082.
[23] A. Leontaris, P.C. Cosman, Compression efficiency and delay trade-
offs for hierarchical B-pictures and pulsed-quality frames, IEEE
Trans. Image Process. 16 (7) (2007) 1726–1740.
[24] Z. Li, G.M. Schuster, A.K. Katsaggelos, MINMAX optimal video
summarization, IEEE Trans. Circuits Systems Video Technol. 15 (10)
(2005) 1245–1256.
[25] R. Lienhart, Comparison of automatic shot boundary detection
algorithms, in: Proceedings of the SPIE Storage and Retrieval for
Image and Video Databases VII, vol. 3656, 1998, pp. 290–301.
[26] Y. Liu, W. Wang, W. Gao, W. Zeng, A novel compressed domain shot
segmentation algorithm on H.264/AVC, in: Proceedings of the
IEEE International Conference on Image Processing, vol. 4, 2004,
pp. 2235–2238.
[27] J. Magalhães, F. Pereira, Using MPEG standards for multimedia
customization, Signal Processing: Image Communications 19 (5)
(2004) 437–456.
[28] B.S. Manjunath, P. Salembier, T. Sikora, Introduction to MPEG-7:
Multimedia Content Description Interface, Wiley, New York,
2002.
[29] S.-C. Pei, Y.-Z. Chou, Efficient MPEG compressed video analysis
using macroblock type information, IEEE Trans. Multimedia 1 (4)
(1999) 321–333.
[30] H. Schwarz, D. Marpe, T. Wiegand, Analysis of hierarchical B
pictures and MCTF, in: Proceedings of the IEEE International
Conference on Multimedia and Expo, 2006, pp. 1929–1932.
[31] D. Tian, M.M. Hannuksela, M. Gabbouj, Sub-sequence video
coding for improved temporal scalability, in: Proceedings of
the IEEE International Symposium on Circuits and Systems, 2005,
pp. 6074–6077.
[32] T. Wiegand, G.J. Sullivan, G. Bjøntegaard, A. Luthra, Overview of the
H.264/AVC video coding standard, IEEE Trans. Circuits Systems
Video Technol. 13 (7) (2003) 560–576.
[33] Z. Xiong, R. Radhakrishnan, A. Divakaran, Y. Rui, T. Huang, A Unified
Framework for Video Summarization, Browsing & Retrieval: With
Applications to Consumer and Surveillance Video, Academic Press,
New York, 2005.
[34] B.-L. Yeo, B. Liu, Rapid scene analysis on compressed video, IEEE
Trans. Circuits Systems Video Technol. 5 (6) (1995) 533–544.
[35] W. Zeng, W. Gao, Shot change detection on H.264/AVC compressed
video, in: Proceedings of the IEEE International Symposium on
Circuits and Systems, vol. 4, 2005, pp. 3459–3462.
[36] H.J. Zhang, A. Kankanhalli, S.W. Smoliar, Automatic partitioning of
full-motion video, Multimedia Systems 1 (1) (1993) 10–28.
[37] H.J. Zhang, C.Y. Low, S.W. Smoliar, Video parsing and browsing using
compressed data, Multimedia Tools Appl. 1 (1) (1995) 89–111.