Professional Documents
Culture Documents
Short
its
Description
sever
Current
Summary
State
Outage
No
Severity
Priority
Support
Sub-Type
Diagnose
Category
Software
Internal
Yes
Assignment Events
Product
Product
9926 DBS-WCDMA
Version
LR13.1W
Instance
Site Company
City
Abu Dhabi
Country
Contact
Name
Company
Phone
20 127 109 0616
Karim.Ibrahim@alcatel-lucent.com
Request Method
Email-CR
Dates
Occurred
(GMT+4)
06-Jan-2015 10:27
Time now
29-Aug-2015 13:21
Reported
06-Jan-2015 10:27
AR Created
06-Jan-2015 10:33
Service Start
06-Jan-2015 10:27
Next Customer Contact
07-Jan-2020 03:00
Responded
06-Jan-2015 23:00
SA - Calculated
Respond Target
07-Jan-2015 10:27
Restored
SA - None
23-Jan-2015 18:20
Restore Target
-- --- ----
Resolved
23-Jan-2015 18:27
SA - Calculated
Resolve Target
24-Mar-2015 03:15
23-Jan-2015 18:29
Last Modified
24-May-2015 16:05
Modified By
archive
Entitlement
Agreement
People
Owner
TSCr-MoA-WCDMA-SK : rories
Assignee
CloQ-WLS-WCDMA-vGlobal : unassigned
Referred 1
NorP-WLS-WCDMA-vGlobal : rories
Resolve Group
NorP-WLS-WCDMA-vGlobal
Submitter
ansikors
Description
Attachments
Hello,
Short Description
Current Summary
issue, and if its affecting the KPIs or not, the network deployed with PTP
sever for Sync.
Contact Surname
Contact Phone
Contact Company
Company
Contract Number
Karim
: Ibrahim
: Alcatel-Lucent
: OXIA 627085
Region
: EMEA
Country
City
: Abu Dhabi
Site
Product Instance
: 9926 DBS-WCDMA
Logs:
Severity
Priority
: HFB Attached.
: 3
: 3
Product Version
Outage
: LR13.3W
: No
Internal/External:
:Internal
-Best Regards,
Karim Ibrahim
LTE/WCDMA Integration Professional
M(Egy): +
<https://mail.eu.alcatel-lucent.com/owa/redir.aspx?
C=30449991f54743fd85063b63258fd6f2&URL=https%3a%2f%2fmail.eu.alcatellucent.com%2fowa%2fUrlBlockedError.aspx>
2010-668-44-810
ONNET:
2205 5534
Show
2. 06-Jan-2015 10:45
ansikors
ALCATEL-LUCENT PROPRIETARY
Dear Customer,
Kind Regards,
Anna Sikorska
ALCATEL-LUCENT
3. 07-Jan-2015 01:05
rkosorin
4. 07-Jan-2015 01:38
rkosorin
Hello Karim,
Are those NodeBs Do those NodeBs belong to the same RNC? Do they belong to
the
same network cluster/topology?
The first alarm appeared 27.12.2014 or is the sync related issue older
issue?
Since when is it present? Can you mp any network maintenance to this date?
Please choose NodeB where the issue is present and provide output of
following
commands:
/pltf_519/pltf/syn/ info
/pltf_519/pltf/syn/ synh
/pltf/ptp> ptph -a
Login to the NodeB as root and for the vlan used for synchronization please
setup
tcpdump:
xCCM-nodeb-nodeb> su root
Password:nodeb3G
Check your interface config:
xCCM-root-nodeb> route
and vlan config:
xCCM-root-nodeb> ifconfig
Please provide: IMT ZAP, hfb , network snapshot, sync_trace.pcap and the
output of
the commands.
Ticket will be assigned when the initial info & logs are available to me.
BR
Rasto
Update to Current Summary: additional logs & info needed
5. 07-Jan-2015 14:14
rkosorin
6. 08-Jan-2015 02:01
rories
Hi Rasto,
?
For the HFB I sent I take the 27.12.2014 as reference , @ logs I sent
you all
stability files from 1 Nov.
?
I performed Audit on the HFB I send to you and the most impacted site
is 2513,
so I performed the AP on it.
?
?
Please we need to find the RC of this issue, a recovery soln & if it
affect
the KPIs or not.
Thanks,
-BR,
Karim
7. 08-Jan-2015 02:01
rories
Hello Karim,
Thank you for the info. Ticket is now assigned to Robert, who will support
you.
BR
Rasto
Rastislav KOSORIN
8. 08-Jan-2015 02:02
rories
Hi Rasto, Reis,
Yes most of sites are impacted after integration, but not with the same
trend as
per attached, (Audit counted from 27/12/2014).
Thanks,
-BR,
Karim
9. 08-Jan-2015 02:02
rories
Hi Karim
Overview:
Based on logs, you sent we can see history of strong synchro break downs of
the
NodeB T_WR_MARRAGE_HALL_MZD_U_2513:
1) IMT has recorded in all flr files progressive streams of alarms like
this:
RNC_0028_00020
RNC_0030_00020
RNC_3004_29755
ANO_CELL_NOT_HSDPA_CAPABLE_INFORMATION
RNC_3015_29772
ANO_CELL_NOT_E-DCH_CAPABLE_INFORMATION
General
RNC_3034_29699
ANO_NODEB_IP_USER_PLANE_FAILURE
4) I needed to check DSCP values you have set for synch messages. But in tcp
dump
you sent, there are no such synch messages captured, because you captured
only OAM
virtual interface, although for synchronization you have set Telekom
interface:
/pltf_594/emo/ip$ VirtualInterface
: 2
[0] :
nbTypeOfTraffic
: 1
TypeOfTraffic
: OAM
IpAddress
: 10.235.158.76
VlanId
: 970
VirtualInterface
[1] :
nbTypeOfTraffic
: 3
TypeOfTraffic
: PTP
IpAddress
: 10.241.188.200
VlanId
: 907
/pltf_594/emo/ip$
AP:
Please ask your transport team what changes were performed around the 7th
Dec
either in settings of transport devices, or in general transport topology
especially in the path between TP5000 an NodeB(s).
Perform please tcp dump of telekom interface (VlanId 907). As tcp dump does
not
capture UP traffic, it`s size should not be very big.
And please inspect if you have correctly set next recommended network
settings for
PTP type of synchronization:
Set PTP packets (event & general messages) with highest priority (i.e.
maximum
precedence, typically DSCP CS7) in TP 5000 and in E2E between TP5000 and
NodeBs.
Make sure firewall and IPSec encapsulation are disabled between the TP 5000
servers and NodeB(s).
Configure the dither parameter as enable on the TP 5000.
Best Regards
Robert Ries
Hi Karim
Thanks for pcap trace. There is set DSCP value CS7 in PTP synchro message,
which
is correct:
However when I performed pings from RNC to NodeB and vice versa, the
transport
network showed it`s very bad quality. I set pings to packet size 1500b to
better
see the quality fluctuations. Look at attached document "ping log.docx",
please. I
highlighted lost messages in red in it.
While your transport network is not able to deliver packets without losses,
it has
no sense to investigate any UTRAN issue connected to transport (which is
almost
whichever UTRAN issue).
Best Regards
Robert Ries
ALCATEL-LUCENT PROPRIETARY
Hi Ries,
This is the same result we got from your team regarding the same site and 2
other
which had TX instabillty.
I Agree with it's a clear transport issue and nothing can be done from our
side
but the problem that when we eslcated the issue before to ETC TX team they
said no
issue from thier side, from your point of view what should be the problem
between
our UTRAN and TX( configration mismtsch or BW limitation , or what?) Do you
have
something from the log can give us any indication about why we recieve all
this
packet and the TX is OK?,
-BR
Karim Ibrahim
ALCATEL-LUCENT PROPRIETARY
Hi Karim
Path taken:
Hop 1:
10.238.14.93
(time = 2ms)
Hop 2:
0.0.0.0
(time = 0ms)
Hop 3:
10.241.188.200
(time = 4ms)
ok
2015-01-09 13:06:35.42
ok
Hop 1:
10.238.14.93
(time = 2ms)
Hop 2:
10.241.188.194
(time = 5ms)
Hop 3:
10.241.188.200
(time = 5ms)
2015-01-09 13:06:46.42
10.241.188.194 (10.241.188.194)
10.238.14.94 (10.238.14.94)
10.238.14.93 (10.238.14.93)
0.918 ms
5.269 ms
4.014 ms
0.736 ms
5.256 ms
3.739 ms
0.631 ms
10.241.188.194 (10.241.188.194)
10.238.14.94 (10.238.14.94)
10.238.14.93 (10.238.14.93)
0.774 ms
5.317 ms
3.910 ms
0.671 ms
0.722 ms
5.894 ms
3.960 ms
Thank you
Best Regards
Robert Ries
ALCATEL-LUCENT PROPRIETARY
Hello Reis,
For PTP alarms we still receive PTP alarms but from my mentoring without the
same
rate as before,
Please provide another AP, so you can check from your side.
Thanks,
-BR,
Karim
Hi Karim
I performed the same ping check as last time. Finally it is clear of packet
loss.
I checked PTP synch NodeB settings and all seems be OK.
I checked NodeB traffic. There are active sessions in progress.
I downloaded IMT file and no problem I found in it, except Alarms trace:
Filename: /rmem/pltf.Alarms.trace
---------------------------------------------------------TIME REF:
18.0000791630
STAMP REF:
1.0467287408
----------------------------------------------------------
As you can see, the 14th Jan there were three synchro interruptions at
01:54,
07:20, 10:48 (distinguished by different text colors).
The 15th Jan there is just one at 08:40.
Please look at the size of the hfb files from last days. While hfbs up to
the 11th
Jan have many MB, the hfbs from the 14th an 15th Jan are considerably less
than
1MB in size.
The 14th Jan there is just one alarm for NodeB "T_AR_U_2513_MarrageHallMzd"
at
01:28 - "loss of supervision".
The 15th Jan there is no any alarm for this NodeB.
Best Regards
Robert Ries
ALCATEL-LUCENT PROPRIETARY
Hello Reis,
I want to mention that the TX action had been taken also on below
sites(Alarm
audit done last week) as we already have KPIs degradation and TX alarms on
them(Also as mentioned before we receiving Sync alarms on them but we took
2513 as
example as it's
stable but
BTS Name
RNC Name
M_AR_BaynounhFodJnctn_U_2764 RNC231 88
EP_AR_JebelBarakhEast_U_2525 RNC235 41
ER_AR_BayyaJunction_U_2501
RNC235 32
O_WR_GAYATHI_SOUTH_U2253
RNC235 19
ET_AR_BayaShabiya_U_2375
RNC235 14
EP_AR_AlDhafraSportsAndCultureClub_U_2551
RNC231 10
Should we proceed for CCM change for all them? Or we can find another
corrective
action?, Also please we already receiving the Sync lock alarm all over the
live
sites as you know(see attached the audit done when we opened this AR).
For now we will proceed for CCM change for 2513 as trial and will keep you
updated, Thanks to check other high impacted sites and let us know if you
have any
other corrective action,
Thanks,
-BR,
Karim
ALCATEL-LUCENT PROPRIETARY
Hello Karim
However if you observe similar symptoms even on another sites, these issues
should
be investigated in another ARs. I keep our general rule => one ticket = one
issue.
So for another NodeBs, open new tickets, please.
But look at the attached log. I performed at least basic pings also for site
2764,
even though I shouldn`t.
There are clear visible the same backhaul defects caused loosing of ping
commands
as in case of site 2513.
Best Regards
Robert Ries
Hi Reis,
Thanks for your feedback, we will try to change the CCM and see the if there
are
any enhancement of the Sync alarms or not.
For the Rule we respect it for sure and following for all tickets, but this
ticket
opened to include all live sites on the network which have the same Sync
alarms..
so when we solve the issue for some sites or one sites from my point of view
not
mean to close the ticket and open another one with the same description.
As you know we still investigation with the transport team and will keep you
updated.
Thanks,
-BR,
Karim
Hi Reis,
Kindly note that we changed the CCM for the site 2513 @ 17.1.2015 but the
site
still suffering from PTP SYNC alarms as attached, as you know the site is
stable
from TX's side.
Please can you check and propose a soln for this issue.
Thanks,
-BR,
Karim
Hi Karim
Look at attached xls file, please. I added the new one sheet "Alarm
duration"
where are just exported columns H (Event Time) and L (Clear Time) from the
sheet
"HFB_2513". I highlighted in red those time values where there was any
observable
time shift present between these two time values (in any words, where there
is
some difference between Event and Clear Time). But alarms, which occurred
and
disappeared in the same second are in black. So in this sheet you can see
duration
of alarms from site 2513 more clearly.
In summary there is 87 alarms which occurred and disappeared just in the
same
second and only 36 alarms which shows any time duration. And as you can see
this
duration is in average only few minutes.
As issue was answered and if there is nothing else I can help you regarding
this
issue, kindly could I close the ticket ?
Best Regards
Robert Ries
Hello Reis,
Thanks for this detailed investigation, I have one question then we can
close this
AR, as you mentioned below regarding column Q, I can see also that the
Specific
Problem is "BTS/0 NO GRANT MESSAGE FROM PTP SERVER 1"
My question is: Based on your investigation, NO issue with our PTP server?
And
it's totally TX issue? Or may we have any configuration mismatch or PTP
failure
affecting this issue?
Probable Cause
Specific Problem
Thanks,
-BR,
Karim
Hello Karim
Look to the alarm sequences in the xls file HFB_2513 again, please. These
alarm
chunks exhibits a cyclic repetition in similar sequences like this (with
comments
copied from Alarm Reference Guide):
=================================================
BTS_0358_00002 - LOST HIGHER PRIORITY SYN SOURCE => This alarm indicates
that the
NodeB is not able to be synchronized.
BTS_0007_00002 - SYN NOT LOCKED TO THE NETWORK => The alarm indicates that
the BTS
is unable to derive reference timing from the Pulse Code Modulation (PCM) or
from
the synchronization sources.
BTS_0388_00033 - NO GRANT MESSAGE FROM PTP SERVER 1 => This alarm indicates
that
preferred PTP time server does not provide GRANT message to the NodeB
REQUEST
UNICAST TRANSPORTATION messages.
BTS_0333_00002 - SYN LOCK SOURCE CHANGE => This alarm indicates that SYN
LOCK
Source of a BTS is changed.
================================================
Important fact in this case is, that each chunk starts with alarm
BTS_0358_00002
with indicates synchronization lost (probable cause "lossOfSignal"). And
alarm
BTS_0388_00033 is just consequence of foregoing alarms indicated lost of
synchronization to the network.
In case you have any configuration mismatch in your PTP server, the NodeBs
synchronization problems would be permanent.
As issue was answered and if there is nothing else I can help you regarding
this
issue, kindly could I close the ticket ?
Best Regards
Robert Ries
Hi Reis,
Thanks,
-BR,
Karim
Update to Current Summary: Issue confirmed as backhaul instability.