International Journal of Computer Science Engineering

and Information Technology Research (IJCSEITR)

ISSN(P): 2249-6831; ISSN(E): 2249-7943
Vol. 6, Issue 4, Aug 2016, 1-8
TJPRC Pvt. Ltd.

USING STORYBOARD TO IMPROVE VIDEO RETARGET

QUALITY BY PATTERN MATCHING METHOD
WEN-PINN FANG & JIU-SHENG KUO
Department of Information Communication, YuanZe University, Chung-Li, Taiwan, R.O.C.
ABSTRACT
Recently, there are many commercial products are invented to show videos. The products include television,
smart phone, tablet, and computer. People use these products somewhere because different situation. However the size
and aspect ratio of these devices are different. If the video is resized in width and height individually or resize according
to the minimum side, the presentation result will very bad. It is necessary to retarget the video for many display devices.
The author has proposed a retarget method by searching the location of face in the frames. It is not enough because
there are many frames in a video without face. This paper proposed a method to retarget the frames by the storyboard.
Based on color analysis, salient region detection and SURF method, the most important region can be distinguished, and
then retarget the corresponding frames. The result of this paper is better than the result that just resize the video by

KEYWORDS: Video Retarget, Storyboard, Feature Matching

Received: Apr 29, 2016; Accepted: May 11, 2016; Published: Jul 11, 2016; Paper Id.: IJCSEITRAUG20161

1. INTRODUCTION

Original Article

individual frames or just crop the frames without drama.

Now, it is very important to design a user friendly multimedia. A user friendly multimedia will make a lot
of profit with good hit rate. In this paper, the media is focus on video. There are many device used to display a
video. It causes some problems. One of the problems is the size of original video is different from the displayer.
Users feel uncomfortable if they watch the aspect ratio[1] of video is wrong. For example, there are many screen
size for smart phone[2]. The way to fit the video into the device is very important. There are many people
proposed methods to retarget the video[3-32]. The methods have a problem that, the program or algorithm cannot
distinguish the important region correctly with just color distance computing. The most precise method is to
retarget the video by asking the director. However, it is very expansive. This paper proposed a method to make the
cost of region marking lower.
The rest of this paper is organized as follows: the background knowledge is shown in Section 2; the
method is proposed in Section 3; Experimental results are shown in Section 4. Finally, the discussion is
represented in Section 5.

2. BACKGROUND KNOWLEDGE
Before describe the proposed method there are some background knowledge need to explain.
The background knowledge includes image retarget, video retarget, storyboard, and pattern matching. The detail is
shown as below:

www.tjprc.org

editor@tjprc.org

Wen-Pinn Fang & Jiu-Sheng Kuo

1.1 Image Resizing

In the decade, there are many researches discuss how to resize an image. The methods include image cropping[3]
[4] [5] [6], seam craving[7] [8] [9] [10] [11], wrapping[12] [13] [14] and hybrid approaches[15] [16] [17]. The cropping
method reserves the most important region and remove others. Avidn and Shamir [7] proposed a method to find out the
seams, remove the region negligible. Mansfield, etc[8]Rubinstein and Shamir[9] and Grundmann[10] improve the
quality of the retarget mage. In the other direction, some papers[12-14] discuss the image distortion after it has been
retargeted. .Rubinstein [15] combined cropping and scaling method to retarget an image. Sun, etc[16] proposed cyclic seam
caving algorithm. Don, etc. [17] discuss the relationship between reserved region and whole effect. Some paper [18] [19]
[20]discussed using coarse region instead of pixel analysis. Pritch [21] proposed shift-map to get better image quality.
Ding [22] designed filter to get better result.
1.2 Video Resizing
As shown as Figure 1there are an American patent that using cropping and wrapping to do video resizing.
There are some key points to retarget a video. If resize the images in a video individually and ignoring the relationship
between the frames, it causes many troubles. As shown in Figure 2 and Figure 3, The troubles include frame inconsistent,
ghost contour, color inconsistent. There are some paper discuss the cropping and wrapping methods to solve this problem.
For example, [23] [24] [25] [26]discuss cropping method. [27] [28] [29][30] discuss the motion vector and retargeting
method. [31] [32] discuss the relationship between moving object and retarget video.
1.3 Salient Region Detection
Different from traditional segmentation, salient region detection only detect the salient foreground object from the
background, rather than partition an image into regions of coherent properties as in general segmentation algorithms. As
shown in Figure 4, there is an original high resolution image, the salient region detection will find out the most important
object in the image. The common method of salient region detection is computing the different between every point first,
and then segment the image by the color distance.

Figure 1: Definition of Video Retarget

Impact Factor (JCC): 7.1293

Index Copernicus Value (ICV): 6.1

Using Storyboard To Improve Video Retarget Quality by Pattern Matching Method

Figure 2: An example of Video Retargeting by Wrapping [27]

Figure 3: An example of video retargeting by seam craving [31]

Figure 4: Given an Input Image (top), a Global Contrast Analysis is used to Compute a High Resolution
Saliency Map (Middle), which can be used to Produce an Unsupervised Segmentation Mask
(Bottom) for an Object of Interest. This Figure is the Example of [33], Not this Paper
1.4 Story Board
As

the

definition

of

Wikimedia

[36],

a storyboard is

graphic

organizer

in

the

form

of illustrations or images displayed in sequence for the purpose of pre-visualizing a motion picture, animation, motion
graphic or interactive media sequence.

www.tjprc.org

editor@tjprc.org

Wen-Pinn Fang & Jiu-Sheng Kuo

3. PROPOSED METHOD
This paper proposed a method to assist people to retarget a video. The method has four parts: (a) retrieve the
salient region.(b) split the images by texture analysis to important frames.(c) find out the important region from storyboard
by pattern matching (d) Using wrapping, cropping and seam craving method to retarget the video. The detail is shown as
below:

Retrieve the Salient Region

In this phase, we adopt the method which proposed by Cheng, etc,, el. [33]
First at all, compute the distance by equation (1)
(1)
S is the salient value in image I. D(Ik, Ii) is the color distance metric between pixels Ik and Ii in the L*a*b* space.
To decrease the computation time, compute the salient value by histogram as shown in equation(2).
(2)
fj is the probability of pixel color cj in image I.

Split the Images by Texture Analysis to Important Frames

In this phase, the method proposed by Jiang, etc, al[34] is adopted.
The idea is compute the probabilistic boundary as shown in equation (3)
(3)
where yt is a neighboring pixel of xt in frame t, and nv denotes the number of valid mapping. A mapping is defined

to be valid, if the projection points xt and yt in frame tare neither occluded nor out-of-view. If

is large, it is very

likely that there is a boundary across pixels xt and yt.

Find out the important region from storyboard by pattern matching

In this phase, the storyboard translated to black and white images first. In the second step, the frames of the video

translated to edges. And then compute the different between the storyboard images and frames. Mark the non-overlapped
region. The non-overlapped regions are the candidate region which can be eliminated.

Using Wrapping, Cropping and Seam Craving Method to Retarget the Video
In this phase, the method proposed by Bhamidipatiand and Raveendran[35] has been adopted. Using the

equation(4)-(6) to determine the energy and delete the pixels recursively.

GlobalLeftPixelEnergy=median(LeftNeighborhood)*W+(1W)*LeftPixelEnergy

(4)

GlobalRightPixelEnergy=median(RightNeighborhood)*W+(1W)*RightPixelEnergy

(5)

New Pixel=minimum(Globa lLeft Pixel Energy, Global Right Pixel Energy,

Impact Factor (JCC): 7.1293

Index Copernicus Value (ICV): 6.1

Using Storyboard To Improve Video Retarget Quality by Pattern Matching Method

Energy of pixel directly below current edge)

(6)

where W is a weighting factor.

4. EXPERIMENTAL RESULTS
An experiment example is shown as Figure 5. The storyboard is shown as Figure 5(a). The original frame is
shown in Figure 5 (b) and Figure 5 (c) is the retargeted frame. (video source: http://www.storyboardworld.com/
rights-of-the-game-points-optician-william-simpson)

(a)

(b)

(c)
Figure 5: An Example of Retargeting. (a) is the Storyboard,
(b) is the Original Frame, and (c) is the Retargeted Frame

5. DISCUSSIONS
In this paper, a video retarget assist method has been proposed. The method is not only detecting the salient region
for retarget a video, but also providing a user friendly check method for director or video producer. There are some cases
that the computer program cannot understand the drama of a video. For example, the key object maybe not so obvious in
the video. The key object is possible to be removed by the existing methods. This paper solves the problem and save the
time to check the quality of retargeted video.
REFERENCES
1.

https://en.wikipedia.org/wiki/Aspect_ratio

2.

http://developer.android.com/guide/practices/screens_support.html

3.

L. Chen, X. Xie, X. Fan, W. Ma, H. Zhang, H. Zhou, A visual attention model for adapting images on small displays, ACM
Multimedia System Journal, Vol. 9 (4), pp. 353364, 2004.

www.tjprc.org

editor@tjprc.org

Wen-Pinn Fang & Jiu-Sheng Kuo

4.

H. Liu, X. Xie, W.-Y. Ma, H. Zhang, Automatic browsing of large pictures on mobile devices, in: Proceedings of ACM
International Conference on Multimedia, pp. 148155 , 2003.

5.

B. Suh, H. Ling, B. Bederson, D. Jacobs, Automatic browsing of large pictures on mobile devices, in: Proceedings of UIST, pp.
95104, 2003.

6.

Santella, M. Agrawala, D. Decarlo, D. Salesin, M. Cohen, Gaze-based interaction for semiautomatic photo cropping, in:
Proceedings of CHI, pp. 771780, 2006.

7.

S. Avidan, A. Shamir, Seam carving for content-aware image resizing, ACM Transaction on Graphics Vol. 26 (3) Article 10,
2007.

8.

Mansfield, P. Gehler, L. V. Gool, C. Rother, Scene carving: Scene consistent image retargeting, Proceedings of European
Conference on Computer Vision, 2010.

9.

M. Rubinstein, A. Shamir, S. Avidan, Improved seam carving for video retargeting, ACM Transaction on Graphics Vol. 27 (3)
Article 16, 2008.

10. M. Grundmann, V. Kwatra, M. Han, I. Essa, Discontinuous seam carving for video retargeting, in: Proceedings of
International Conference on Computer Vision, 2010.
11. Y. Liang, Z. Su, X. Luo, Patchwise scaling method for content-aware image resizing, Signal Processing, Vol.92 (5), pp. 1243
1257, 2012.
12. R. Gal, O. Sorkine, D. Cohen-or, Feature-aware texturing, Computer, Vol. 11 (5) pp. 17 , 2006.
13. Y. Wang, C. Tai, O. Sorkine, T. LEE, Optimized scale-and-stretch for image resizing, ACM Transaction on Graphics Vol. 27 (5),
Article 118, 2008.
14. B. Li, Y. Chen, J. Wang, L. Duan, W. Gao, Fast retargeting with adaptive grid optimization, in: Proceedings of International
Conference on Multimedia & Expo, pp. 14, 2011.
15. M. Rubinstein, A. Shamir, S. Avidan, Multi-operator media retargeting, ACM Transaction on Graphics Vol. 28 (3), Article 23,
2009.
16. J. Sun, H. Ling, Scale and object aware image retargeting for thumbnail browsing, Proceedings of International Conference on
Computer Vision, 2011.
17. W. Dong, N. Zhou, J. Paul, X. Zhang, Optimized image resizing using seam carving and scaling, in: Proceedings of ACM
SIGGRAPH Asia 2009, Vol. 28, 2009.
18. T. Cho, M. Butman, W. F. S. Avidan and, The patch transform and its applications to image editing, Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition 2008, 2008.
19. D. Simakov, Y. Caspi, E. Shechtman, M. Irani, Summarizing visual data using bidirectional similarity, in: Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition 2008, 2008.
20. C. Barnes, E. Shechtman, A. Finkelstein, D. Gold-man, Patchmatch: A randomized correspondence algorithm for structural
image editing, ACM Transaction on Graphicsl. 28 (3) ,Article 24, 2009.
21. Y. Pritch, E. Kav-Venaki, S. Peleg, Shift-map image editing, in: Proceedings of International Conference on Computer Vision,
pp. 151158, 2009.
22. Y. Ding, J. Xiao, J. Yu, Importance filtering for image retargeting, Proceedings of IEEE Conference on Computer Vision and
Pattern Recognition 2008.
Impact Factor (JCC): 7.1293

Index Copernicus Value (ICV): 6.1

Using Storyboard To Improve Video Retarget Quality by Pattern Matching Method

23. F. Liu, M. Gleicher, Video retargeting: automating pan and scan, Multimedia pp.241250, 2006.
24. C. Tao, J. Jia, H. Sun, Active window oriented dynamic video retargeting, Proceedings of Workshop on Dynamical Vision,
ICCV, 2007.
25. T. Deselaers, P. Dreuw, H. Ney, Pan, zoom, scan time-coherent, trained automatic video cropping, Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition 2008.
26. M. L. Gleicher, F. Liu, Re-cinematography: Improving the camerawork of casual video, ACM Trans. Multimedia Comput.
Commun, Vol.5 (1), pp. 128, 2008.
27. L. Wolf, M. Guttmann, D. Cohen-Or, Non-homogeneous content-driven video-retargeting, in: Proceedings of International
Conference on Computer Vision, 2007.
28. Y. Zhang, S. Hu, R. R. Martin, Shrinkability maps for content-aware video resizing, Computer Graphics Forum Vol. 27 (7),
pp.17971804, 2008.
29. P. Krhenbhl, M. Lang, A. Hornung, M. Gross, A system for retargeting of streaming video, ACM Transaction on Graphics Vol.
28 (5), 2009.
30. Y.Wang, H. Fu, O. Sorkine, T. Lee, H.-P. Seidel, Motion-aware temporal coherence for video resizing, ACM Transaction on
Graphics Vol. 28 (5), 2009.
31. Y. Wang, H. Lin, O. Sorkine, T.-Y. Lee, Motion-based video retargeting with optimized crop-and-warp, ACM Transaction on
Graphics Vol. 29 (4), 2010
32. Y. Wang, J. Hsiao, O. Sorkine, T. Lee, Scalable and coherent video resizing with per-frame optimization, ACM Transaction on
Graphics Vol. 30 (4), 2011.
33. M. M. Cheng, N. J. Mitra, X. Huang, P. H. S. Torr, and S. M. Hu, Global Contrast based Salient Region Detection, IEEE
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, online.
34. H. Jiang , G. Zhang, H. Wang , and H. Bao,E, Spatio-Temporal Video Segmentation of Static Scenes and Its Applications, IEEE
TRANSACTIONS ON MULTIMEDIA, VOL. 17, NO. 1, JANUARY 2015.
35. P. K. Bhamidipati and V. R. Raveendran, Video retargeting using seam carving, https://www.google.com/patents/US9031358
36. https://en.wikipedia.org/wiki/Storyboard

www.tjprc.org

editor@tjprc.org