XVII SIMPOSIO DE TRATAMIENTO DE SE

NALES, IM

AGENES Y VISI

ON ARTIFICIAL - STSIVA 2012 1

Quality assessment for video streaming P2P
application over Wireless Mesh Network
Juan Pablo Urrea Duque

, Natalia Gaviria G omez

Universidad de Antioquia
{jpurrea, nagaviri}@udea.edu.co
AbstractCategory(4). Wireless mesh networks (WMN) en-
ables the expansion of communication networks and broadband
services. This expansion allows the use of applications like
streaming multimedia in P2P (Peer-to-Peer). P2P-TV and WMNs
share similar features like self-organization and decentralization
in dynamic network environments and multi-hop transmission.
In this context quality measurement has become essential due
to actual QoS (Quality of Service) parameters lack of user
experience metrics. In order to evaluate QoE in P2P streaming
over WMN, we use a Pseudo Subjective Quality Assessment
(PSQA) taking the advantages of both subjective an objective
approaches. The experimental testbed is implemented using
PeerStreamer (P2P video streaming application) running over
virtual machines, and the IEEE 802.11s NS-3 model as network
layer connected through the NS-3 emulation mode.
Index TermsMOS (Mean Opinion Score), PSNR (Peak Signal
to Noise Ratio), PSQA, P2PTV, QoE, Random Neuronal Net-
works (RNN), WMN (Wireless Mesh Networks).
I. INTRODUCTION
Wireless mesh networks (WMN) enables the expansion
of communications networks and broadband services. This
expansion has increased the use of applications like P2P
(Peer-to-Peer) streaming multimedia. P2PTV and WMNs share
similar features like self-organization and decentralization in
dynamic network environments and multi-hop transmission.
WMNs provides an easy deployment of a communication
network, with low investment, delivering broadband services
to sensitive communities (far from an infrastructure backbone).
Several communities have implemented WMNs [1] for Internet
access, educational purposes [2], and e-commerce [3].
In P2PTV the P2P networking technique is used for broad-
casting live television over video streaming. In this system
a peer downloads data from the server and other peers, and
uploads the downloaded data to those who require it. This
signicantly reduce the bandwidth requirements on the servers
and also provide peers with acceptable service [4]. These
applications requires high bandwidth with low delay and loss
constraints. However, streaming video presents degraded per-
formance in WMNs due to high compression and packet loss
ratios [5]. To overcome these drawbacks efcient resources
utilization of the wireless channel is required.
The identication of degrading factors in streaming video
is accomplished through performance metrics. Performance
metrics can be seen from the network operator side or from
the end-user side. In order to compensate the drawbacks, of
such generally not related measures, it is possible integrate
quality metrics (QoS and QoE) and obtain a better estimation
of the video quality assessment, in P2P video streaming over
WMN scenario.
II. STREAMING VIDEO QUALITY ASSESSMENT
The user satisfaction has become one of the main concerns
in real-time services and applications. So the network oper-
ators continuously monitor and control their resources while
maintaining user satisfaction. Therefore, they have to consider
both quality of service (QoS) and quality of experience (QoE).
QoE is the overall acceptability of an application or service,
as perceived by end-users [6].
Performance video quality metrics are evaluated through
objective and subjective measures. Streaming video degrada-
tion occurs due to channel and protocol features, while it is
transmitted through a communication network.
Objective metrics relies on different statistics obtained com-
paring the original and distorted video. One of such metrics
is the PSNR (Peak Signal to Noise Ratio), which is the ratio
between the maximum power of the original video and the
power of the interfering noise introduced in the transmission
process. It is dened via the Mean Squared Error (MSE)
between an original frame o and the distorted frame d as
following:
MSE =
1
M N
M

m=1
N

n=1
|o(m, n) d(m, n)|
2
(1)
where each frame has MN pixels, and o(m, n) and d(m, n)
are the luminance pixels in position (m, n) in the frame [7].
If the maximum luminance value in the frame is L (when the
pixels are represented using 8 bits per sample, L = 255) then:
PSNR = 10 log
255
2
MSE
(2)
PSNR can be computed only once the image is received, not
really appropriate in real-time mechanisms.
Other measures reecting the streaming performance are
frame loss rate, frame jitter, and mean loss burst size (MLBS).
Structural Similarity (SSIM: similarity between two images)
is designed to improve traditional methods like PSNR.
Video Quality Metric (VQM) method objectively measuring
video quality by comparing the original and distorted video
sequences using only a set of features extracted from each
video (it is demonstrated to perform much better on low
bit-rate encodings than for high bit-rate ones).
978-1-4673-2761-9/12/$31.00 c 2012 IEEE
PSNR [dB] MOS
>37 5 (Excellent)
31-37 4 (Good)
25-31 3 (Fair)
20-25 2 (Poor)
< 20 1 (Bad)
Table I
POSSIBLE PSNR TO MOS CONVERSION [8]
Subjective video evaluation is developed by individuals,
scoring in a determinate scale the video quality based on a
reference. The MOS (Mean Opinion Score) measure video
quality at the application level, on a scale that ranges from
1(worst) to 5(best) [8]. According to [8] there is a possible
relation between PSNR and MOS, as shown in table I.
Subjective video quality estimations are standardized by
ITU-R BT.500-11 specifying several variants (Single Stimu-
lus (SS), Double Stimulus Impairment Scale (DSIS), Double
Stimulus Continuous Quality Scale (DSCQS), etc.). Among all
these variants, the SS (Single Stimulus) is a better approach
in quality assessment with simplicity [9].
Streaming applications requires QoE evaluation in real-
time, and the methods mentioned use off-line evaluation based
on full reference or reduced reference videos (subjective or
objective). Besides objective methods are not well correlated
with human visual perception and they are time consuming
[10]. In most of P2PTV implementations it is no possible to
access to reference videos, and the quality evaluation must
be done without any reference. The MOS requires controlled
environments, it is not easy to reproduce the same results,
and cannot be automated [10]. So an hybrid approach called
Pseudo-subjective Quality Assessment (PSQA) is a viable
solution for overcoming the above conditions.
III. PSEUDO-SUBJECTIVE QUALITY ASSESSMENT
PSQA combines the best of both subjective and objective ap-
proaches and is based on statistic learning using random neural
network (RNN). This methodology start with the selection
of factors having impact on quality such as bandwidth, loss,
delay, and jitter. Then a distorted video database is created
varying the representative range in a set of selected factors.
In [7] [11] consider loss rate (LR) of video packet and MLBS
that is the average length of a sequence of consecutive lost
packets in a period of time.
With the dened database a panel of human observers
evaluates the distorted videos. Then MOS is computed using
the average score obtained by observers, generating training
and validation databases. Then the RNN is trained producing
the mapping of congurations and scores. The RNN is vali-
dated by comparing value given by the function, at the point
corresponding to each conguration in the validation database.
After verication and validation process, PSQA can be run in
real-time without any human interaction (Fig. 1).
A. Random neural network (RNN)
A Random neural network (RNN) exchange signals in form
of spikes. The neurons state is a non-negative integer called
Figure 1. PSQA Methodology using RNN [12].
potential, which increases by 1 when a positive signal (an
excitation) arrives to it, and decreases by 1 when a negative
signal (an inhibition) arrives [7]. Some signals arrive from
outside the network, or from other neurons. Each neuron
behaves as a ./M/1 queue with respect to positive signals. This
means that positive signals are interpreted as customers, these
customers arrive to the neurons, and they are served in a FIFO
order. The service rate at neuron i is denoted by
i
. A neuron i
receives positive customers from the environment according to
a Poisson process with rate
+
i
(no negative customers arrive
from the environment) [12]. When a positive signal leaves
neuron i it goes to neuron j with probability of p
+
ij
, negative
signals leave with probability of p

ij
, and signal departs the
network with probability d
i
[13].
d
i
+
N

j=1
(p
+
ij
+p

ij
) = 1 (3)
p
ij
= p
+
ij
+p

ij
(4)
When a signal leaves a neuron its not allowed to return to
same neuron (p
ii
= 0). The rates neuron i res positive and
negative signals when it is excited are
w
+
ij
=
i
p
+
ij
0 (5)
w

ij
=
i
p

ij
0 (6)
The network state (k
i
) is specied by the positive customers
waiting in queue. The steady-state probability guarantees that
the excitation level of each neuron remains nite with proba-
bility 1:
Pr(k
1
, , k
N
) =
N

i=1
(1
i
)
ki
i
(7)
where
i
is the load in the queue (i.e. the asymptotic proba-
bility that queue i is not empty). The loads are obtained solving
the non-linear system of equations, in terms of individual
services rates, environment arrivals and routing probabilities:
Figure 2. QoE estimation system proposed
Parameter Description
Application P2PTV (Peerstreamer)
Protocol UDP
Source bitrate 1 Mbps
GOP 25 (Transcoded)
Codec H.264
Testbed Real time emulation (NS-3 Virtualbox)
MAC IEEE 802.11s (Mesh NS-3)
Routing HWMP IEEE 802.11s
Topology Regular square
Distance 100m
Size 9 nodes
Channel model Constant Log Distance Propagation Loss Model
Bandwidth OFDM at 6 Mbps
Table II
TESTBED PARAMETERS DEFINITION

i
=

+
i
+

N
j=1

j
w
+
ij

i
+

N
j=1

j
w

ij
, i = 1, , N (8)
IV. QOE ESTIMATION
In this work the QoE estimation is carried out with the
implementation of RNN in PeerStreamer [14]. The topology
of the RNN has three layer, with 3 neurons at the input
layer, 10 neurons at hidden layer and one neuron at output
layer. In the rst layer the inputs are LR, MLBS and Bit-rate.
The weights were obtained using Multiple Back-Propagation
v.2.2.2 software [15]. The normalized output is scaled to obtain
a real-time PSNR value. Every 0.5 seconds the MOS value is
updated in real-time.
V. QOE ESTIMATION SYSTEM IN WMN
In Fig 2 we describe the system in order to predict the user
experience. The system consists of three main components
(streaming system, video parameters evaluation (bit-rate, LR,
MLBS), QoE estimation (RNN model)).
The streaming system is developed through emulation mode
(Fig 3) in a NS-3 mesh (IEEE 802.11s) implementation [16],
running a P2PTV application (PeerStreamer) inside several
Ubuntu virtual machines (Virtualbox [17]). In the emulation
testbed several videos were transmitted according to the pa-
rameters in table II. Each emulation session is controlled
through ssh and different traces were obtained during the
process, using packet capture les (.pcap) generated by NS-3.
Parameter Description
Name akiyo.mp4
Codec MPEG-4
Resolution 352x288
Frame rate 25 fps
GOP 12
Table III
TEST VIDEO PARAMETERS.
VI. RESULTS
The video parameters used are shown in table III. Several
trials were conducted transmitting one video, with one node
acting as server and several nodes receiving the streaming in
the P2PTV application. The original PeerStreamer operation
mode uses an Internet connection to receive live P2PTV
broadcasting and the streaming process is based on UDP.
The interprocess communication and initial synchronization
is based on TCP. The P2PTV application was recompiled
in order to stream only video (without audio) and disabling
loop mode. The application has the options of running without
graphical interface, and generating measures traces like MOS
(in PSNR scale), and I-B-P sequences.
In gure 4 increasing the number of peers reduce the bit-
rate (BR) due to the wireless shared media. There is at the
beginning of streaming a synchronization process, then the
bit-rate stabilizes. Faster synchronization occurs when adding
more peers. According to the graphic, the maximum bit-
rate when transmitting to one peer, maintains with two peers
and decreases with three (about 20%) and four peers (about
60%). Table IV summarizes different results showing the mean
() and standard deviation (), using 30 trials in each peer
situation. The average BR keep decreasing as the number of
peers increase. Also, the PLR, and MLBS keep in low values
(8% and 15%) but with four peers increases to 35%. The
average MOS measure is stable, even with 3 peers, and reduces
(about 50%) when the numbers of peers is four. In Fig. 5 is
shown how the real-time MOS holds over 4 score while three
peers are connected, and then the score falls to 1.5 when the
fourth peer is connected.
It is also possible to obtain QoS metrics (like Jitter,
Throughput and Delay) using the pcap les generated by NS-
3 at the end of each emulation. As an example in Fig 6,7
show the delay and jitter histogram of the receiving video
for two peers and one server. Also the trafc features can be
extracted like the transmission packet size which is about 550
bytes, and the most common size is 1440 bytes. Jitter was
measured according to [18] Metro Ethernet Forum (MEF) 10
specication 2004. All the pcap les were ltered with Tshark
[19] (enabling mesh dissector) and parsed using Python scripts.
Table V resume the statistics obtained for the setup mentioned
above.
VII. CONCLUSIONS
The PSQA methodology is a good choice in nding video
performance measures. Allows to present such measures in
real-time based on consistent trained database. The RNN is
Figure 3. Emulation Testbed integrating NS-3 emulation mode and Virtualbox.
1 Peer 2 Peers 3 Peers 4 Peers
QoE Metric
BR(Mbps) 1.49 0.17 1.53 0.13 1.21 0.15 0.74 0.27
PLR (%) 14.9 6.24 11.7 6.68 8.79 3.62 35.3 26.3
MLBS (%) 2.17 1.21 1.40 0.91 0.88 0.45 2.87 2.4
MOS 3.43 0.29 3.62 0.3 3.57 0.19 1.65 0.91
Table IV
QOE STATISTICS
Qos Metric
Delay (ms) 1.5964 0.6810
Jitter (ms) 0.5362 0.1429
Throughput (Mbps) 0.8968 0.5092
Table V
QOS STATISTICS
0 2 4 6 8 10 12
Time (s)
0
200
400
600
800
1000
1200
1400
B
i
t
r
a
t
e
(
k
b
p
s
)
One Peer
Two Peers
Three Peers
Four Peers
Figure 4. Bit-rate increasing the number of peers.
0 2 4 6 8 10 12
Time (s)
0
1
2
3
4
5
M
O
S
One Peer
Two Peers
Three Peers
Four Peers
Figure 5. MOS increasing the number of peers.
able to classify QoE even if the measures are scaled. A deeper
evaluation of trafc patterns can be done using the pcap
traces generated by the NS-3 emulation mode. This approach
offer a better performance estimation considering the operative
system stack. The emulation testbed developed will help in
understanding, as well as in the validation and verication
process determining a performance analytical model of a
WMN, when using a video streaming application.
ACKNOWLEDGMENT
This work is developed under the iTV macro-project and it
has been supported by ARTICA, The Colombian ICT Ministry
and Colciencias.
REFERENCES
[1] H. Song, B. C. Kim, J. Y. Lee, and H. S. Lee, IEEE 802.11-based
Wireless Mesh Network Testbed, 2007 16th IST Mobile and Wireless
Communications Summit, no. Mcl, pp. 15, Jul. 2007.
[2] R. C. Carrano, M. Bletsas, and L. C. S. Magalh aes, Mesh networks for
digital inclusion-testing OLPCs XO mesh implementation, published
in International Software Meet Porto Alegre, pp. 29, 2007.
0.0 0.5 1.0 1.5 2.0
Delay(ms)
0
5
10
15
20
P
e
r
c
e
n
t
(
%
)
Figure 6. Packet delay histogram emulating two peers.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Jitter (ms)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
P
e
r
c
e
n
t
a
g
e
(
%
)
Figure 7. Packet jitter histogram for emulation with two peers.
[3] J. Ishmael, S. Bury, D. Pezaros, and N. Race, Deploying Rural
Community Wireless Mesh Networks, IEEE Internet Computing,
vol. 12, no. 4, pp. 2229, Jul. 2008.
[4] L. Chang, Video-on-Demand streaming on Peer-to-Peer Networks,
grp.pan.uvic.ca, p. 6, 2006.
[5] X. Qiu, H. Liu, D. Ghosal, B. Mukherjee, J. Benko, W. Li, and
R. Bajaj, Enhancing the Performance of Video Streaming in Wireless
Mesh Networks, Wireless Personal Communications, vol. 56, no. 3,
pp. 535557, Apr. 2010.
[6] G. Rubino, M. Varela, and J.-M. Bonnin, Controlling Multimedia QoS
in the Future Home Network Using the PSQA Metric, The Computer
Journal, vol. 49, no. 2, pp. 137155, Dec. 2006.
[7] K. Piamrat, C. Viho, J.-M. Bonnin, and A. Ksentini, Quality
of Experience Measurements for Video Streaming over Wireless
Networks, 2009 Sixth International Conference on Information
Technology: New Generations, pp. 11841189, 2009.
[8] C. Ke and C. Shieh, An evaluation framework for more realistic
simulations of MPEG video transmission, Journal of information, vol.
440, pp. 425440, 2008.
[9] N. De Le on, Evaluaci on de calidad de video en
una aplicaci on P2P: Goalbit, Master Thesis, Universidad
de la Rep ublica de Uruguay, 2010. [Online]. Available:
http://www.ng.edu.uy/inco/pedeciba/bibliote/cpap/tesis-ndeleon.pdf
[10] S. Mohamed, S. Member, and G. Rubino, A study of real-time packet
video quality using random neural networks, IEEE Trans. on Circuits
and Systems for Video technology, vol. 12, no. 12, pp. 10711083,
2002.
[11] G. Rubino, Quantifying the quality of audio and video transmissions
over the Internet: the PSQA approach, Design and operations of
communication networks: a, 2005.
[12] P. Rodriguez-Bocca, Quality-centric design of Peer-to-Peer systems
for live-video broadcasting, Ph.D. dissertation, Universit e de Rennes,
2008. [Online]. Available: goalbit.sourceforge.net/publications/these.pdf
[13] K. Radhakrishnan and H. Larijani, Evaluating perceived voice
quality on packet networks using different random neural network
architectures, Performance Evaluation, vol. 68, no. 4, pp. 347360,
Apr. 2011.
[14] PeerStreamer, Open source P2P Media Streaming framework.
[Online]. Available: http://peerstreamer.org/
[15] Multiple Backpropagation v. 2.2.2. [Online]. Available:
http://dit.ipg.pt/MBP
[16] NS-3, Network Simulator. [Online]. Available: http://www.nsnam.org/
[17] Virtualbox, Virtualization software package. [Online]. Available:
https://www.virtualbox.org/
[18] Spirent, Jitter, pp. 110, 2007. [Online]. Available:
gospirent.com/whitepaper/Jitter Spirent Communications.pdf
[19] Wireshark, Network protocol analyzer. [Online]. Available:
www.wireshark.org/