August 21, 2006

UMTS Operations

Lucent Worldwide Services

Knowledge Transfer Workshop
Node B Alarm Management

Table of Contents
TASK DESCRIPTION ................................................................................................................................... 3
NODE B......................................................................................................................................................... 3
NE Event could not be Processed............................................................................................................. 3
NE Alarm Synchronization in Progress ..................................................................................................... 3
NE Communication Problem ..................................................................................................................... 3
UNMANAGED State Set for NE ................................................................................................................ 3
HeartBeatFailure ....................................................................................................................................... 3
E1T1 PORT................................................................................................................................................... 4
AISDetected .............................................................................................................................................. 4
BitErrorRateGreaterThan1PerMillion ........................................................................................................ 4
SignalLost .................................................................................................................................................. 4
FrameLost.................................................................................................................................................. 4
FrameMissed ............................................................................................................................................. 5
IMALinkAlarm ............................................................................................................................................ 5
CTU ............................................................................................................................................................... 5
ClockFailure30_72MHz ............................................................................................................................. 5
InitializationFailure..................................................................................................................................... 5
SelfTestFailure .......................................................................................................................................... 5
FRAME.......................................................................................................................................................... 6
CabinetDoorOpen...................................................................................................................................... 6
SHELFFAN.................................................................................................................................................... 6
EquipmentFault ......................................................................................................................................... 6
IUBINTERFACE ............................................................................................................................................ 6
ALCAPLinkFailure ..................................................................................................................................... 6
NBAP LINK FAILURE................................................................................................................................ 6
RADIO ........................................................................................................................................................... 7
CommunicationFailure............................................................................................................................... 7
MajorTxSpectrumFailure ........................................................................................................................... 7
MajorGainFailure ....................................................................................................................................... 8
MinorGainFailure ....................................................................................................................................... 8
LossOfPulse .............................................................................................................................................. 8
LossOfForwardBusLink ............................................................................................................................. 8
IMAGROUP................................................................................................................................................... 9
CriticalGroupAlarm .................................................................................................................................... 9
MajorGroupAlarm ...................................................................................................................................... 9
NonCriticalGroupAlarm.............................................................................................................................. 9
FILTER ........................................................................................................................................................ 10
DCConverterFailure................................................................................................................................. 10
Lucent Technologies PROPRIETARY
See notice on cover page

Node B Alarm Management

August 21, 2006,

RxAmpFailure .......................................................................................................................................... 10
EXTRXAMP................................................................................................................................................. 10
ExtRxAmpFailure..................................................................................................................................... 10
CELL ........................................................................................................................................................... 11
ReceiveAntennaTestThresholdExceeded ............................................................................................... 11
PowerTestThresholdExceeded ............................................................................................................... 11
AMPLIFIER ................................................................................................................................................. 12
OverTemperature .................................................................................................................................... 12
RFOverDrive............................................................................................................................................ 12
CommunicationFailure............................................................................................................................. 12
InitializationFailure................................................................................................................................... 12
RF PATH DIAGRAM ................................................................................................................................... 13
RXAIT EXAMPLE CONFIGURATION ........................................................................................................ 14

Lucent Technologies Proprietary

See notice on cover page
2

August 21, 2006,

Node B Alarm Management

TASK DESCRIPTION
This section describes Node B alarm management and the meaning of various Node B alarms.

NODE B
NE Event could
not be Processed

The data sent by the Node B has been corrupted.

Fault Clearance:

Verify Node B is Managed

Perform a Full Tree Sync

NE Alarm Sync

NE Alarm
Synchronization
in Progress

Typically reported when communication has been restored to the Node B. It is an information
alarm and can be ignored.
Fault Clearance:

Usually found in the cleared alarm list

NE
Communication
Problem

Alarm is generated when communication to the Node B over the logical Itf-B link has failed.

UNMANAGED
State Set for NE

The Node B link state has changed to Unmanaged. Call processing should still be possible;
however alarm events reporting trouble will not be visible.

Fault Clearance:

Verify the Node B is not in the Pre-Provisioned State, indicating that it has not been
integrated, however it has been mistakenly managed. If Pre-Provision state exists in
the Node B details, Unmanage the Node B to clear the alarm.

Ping the Universal Master Controller (UMC) IP Address. If ping fails troubleshoot
connectivity issues.

If ping is successful Unmanage and Manage the Node B.

If alarm still exists Perform an NE Alarm Sync.

Fault Clearance:

If the Node B is in the Pre-Provisioned state ignore the alarm and acknowledge the
alarm to let others know the alarm has been verified.

To clear the alarm Manage the previously integrated Node B.

HeartBeatFailure

The heartbeat synchronization between the Node B and the OMC-UPS has been lost.
Fault Clearance:

Verify the Node B is not in Pre-Provision mode. If it is, verify the Node B is
Unmanaged and ignore the alarm by acknowledging so that others are aware the
alarm has been troubleshot.

Ping the UMC IP address. If ping fails, troubleshoot connectivity.

UnManage / Manage the Node B

Perform an NE Alarm Sync.

Lucent Technologies Proprietary

See notice on cover page
3

Node B Alarm Management

August 21, 2006,

E1T1 PORT
AISDetected

The Node B has detected an AIS signal (Blue Alarm), which is most likely a failed T1.
Fault Clearance:

Verify the alarm in the PSAX.

Request Cingular DACs support to check for any loops, and to loop the Network
Interface Unit (NIU) and Channel Service Unit (CSU).

If loops are found and removed verify alarm has cleared and service has been
restored.

If the NIU (Demarc / smartjack) is not able to loop up, Cingular will open a ticket with
the T1 provider to report the outage.

If the NIU loops but the CSU does not, there is problem with the cabling between the
NIU and CSU (Node B) dispatch a tech to resolve.

BitErrorRateGrea
terThan1PerMillio
n

The CSU in the Node B (T1 port) has detected several bit errors, or Cyclic Redundancy
Checks (CRCs). The T1 may remain InService; however the link quality will suffer. The bit
errors are also called Dirty T1.
Fault Clearance:

Request Cingular DACs support to monitor the T1 for at least 15 minutes, or longer if
possible. If no trouble is found, request Cingular to break the circuit and stress the
circuit (run multiple patterns) to the CSU (but not the NIU).

SignalLost

The CSU (Node B T1 port) has detected a loss of signal.

Fault Clearance:

Verify the alarm in the PSAX.

Request Cingular DACs support to check for any loops, and to loop the NIU and
CSU.

If loops are found and removed verify alarm has cleared and service has been
restored.

If the NIU (Demarc / smartjack) is not able to loop up, Cingular will open a ticket with
the T1 provider reporting the outage.

If the NIU loops but the CSU does not, there is problem with the cabling between the
NIU and CSU (Node B) dispatch a tech to resolve.

FrameLost

The Node Bs CSU cannot detect the framing pattern. This is also known as a RED Alarm:
Fault Clearance:

Verify the alarm in the PSAX.

Request Cingular DACs support to check for any loops, and to loop the NIU and
CSU.

If loops are found and removed verify alarm has cleared and service has been
restored.

If the NIU (Demarc / smartjack) is not able to loop up, Cingular will open a ticket with
the T1 provider reporting the outage.

If the NIU loops but the CSU does not, there is problem with the cabling between the
NIU and CSU (Node B) dispatch a tech to resolve.

Lucent Technologies Proprietary

See notice on cover page
4

August 21, 2006,

Node B Alarm Management

Continued on next page

E1T1 PORT, CONTINUED

FrameMissed

Node B is detecting a framing error on the T1 line.

Fault Clearance:

Request Cingular DACs support to monitor the T1 for at least 15 minutes, or longer if
possible. If no trouble is found, request Cingular to break the circuit and stress the
circuit (run multiple patterns) to the CSU (but not the NIU).

IMALinkAlarm

A T1 that is a member of the Inverse Multiplexing on ATM (IMA) Group is either OutofService
or is InService but did not join the IMA Group.
Fault Clearance:

Verify the alarm in the PSAX.

Request Cingular DACs support to check for any loops, and to loop the NIU and
CSU.

If loops are found and removed verify alarm has cleared and service has been
restored.

If the NIU (Demarc / smartjack) is not able to loop up, Cingular will open a ticket with
the T1 provider reporting the outage.

If the NIU loops but the CSU does not, there is problem with the cabling between the
NIU and CSU (Node B) dispatch a tech to resolve.

CTU
ClockFailure30_7
2MHz

The Common Timing Unit (CTU) has failed to detect a 30.72MHz clock signal. Call
processing is affected because the Node B will not fully initialize. This alarm will typically be
found in the cleared alarm list because the CTU and Oscillator Module (OM) will automatically
reset to resolve. If the problem that caused the clock failure remains, the CTU will most likely
fail Initialization and report this alarm.
Fault Clearance:

Verify initial cause for clock failure replace suspect OM.

Replace failed CTU.

InitializationFailu
re

The CTU has failed to initialize properly either during Node B startup or because of an
autonomous reset to self-clear an alarm.
Fault Clearance:

Check cleared alarms to find out why the CTU re-initialized.

Reset the CTU if the reset does not resolve the alarm, replace CTU or OM.

SelfTestFailure

The CTU raises the the SelfTestFailure alarm anytime it resets and fails the Board Level Self
Tests (BLST).
Fault Clearance:

Reset CTU. If it fails again, replace it.

Lucent Technologies Proprietary

See notice on cover page
5

Node B Alarm Management

August 21, 2006,

FRAME
CabinetDoorOpe
n

The Outdoor Node B (modular/micro) has triggered an alarm indicating the door is open.
Fault Clearance:

Dispatch a Tech to the site to verify the door is closed and the alarm contact is
connected and unobstructed.

Onsite, verify that the rear heat exchanger hatch is secure and screws are tight.

SHELFFAN
EquipmentFault

The fan has failed because the power to the fan has been disconnected or the fan is faulty.
Fault Clearance:

Dispatch Tech to site to verify that the fan breaker is on.

Replace the faulty FanShelf.

IUBINTERFACE
ALCAPLinkFailur
e

Access Link Control Application Protocol (ALCAP) virtual circuit connection has failed.
Fault Clearance:

In the PSAX (Bpsax, and Rpsax), verify the Permanent Virtual Circuits (PVCs) exist
and pass 2-way traffic. If the PVCs do not exist, create them. If the PVCs do not
pass 2-way traffic, delete and add them back (PSAX Bug).

If the Node B has just rebooted, wait for up to 5 minutes for the alarm to clear.

Verify the IuB is not in alarm under the Lrnc.

NBAP LINK
FAILURE

Node B Application Part (NBAP) virtual circuit connection has failed.

Fault Clearance:

In the PSAX (Bpsax, and Rpsax), verify the PVCs exist and pass 2-way traffic. If the
PVCs do not exist create them. If the PVCs do not pass 2-way traffic, delete and
add them back (PSAX Bug).

If the Node B has just rebooted, wait for up to 5 minutes for the alarm to clear.

Verify the IuB is not in alarm under the Lrnc.

Lucent Technologies Proprietary

See notice on cover page
6

August 21, 2006,

Node B Alarm Management

RADIO
CommunicationF
ailure

Heartbeat synchronization of the radio has been lost. Call processing is not possible. The
following could be reasons that the radio can not communicate:

Faulty Radio

Faulty CTU or OM
Fault Clearance:

Reset the radio if the initialization fails, replace the radio and calibrate.

If alarms reappear troubleshoot the clock (replace CTU or OM).

MajorTxSpectrumFailure

MajorTxSpectrumFailure is a general alarm used for several purposes. A

majorTxSpectrumFailure can be triggered by:

Antennas not connected

Failure of edpdul

Radio hardware failure (known cold soldering issue)

Incorrectly connected Rf cables

Fault Clearance:

Execute RxAntennaTest if results are less than 14dB, dispatch an antenna

crew (anything close to 0.5db indicates an open or a short in the antenna
system).

Telnet to the radio and look for the following messages repeating every couple
minutes:
MCRedpdulFsm_generate: port=1 edpdulResult=19 blueResult=0 (Error)
MCRedpdulFsm_trainIt: port=1 (Generate error)
MCRedpdulFsm_trainIt: port=1 srxCopyCount=3 (Recapturing)
MCRedpdulFsm_estimate: port=1 srxCopyCount=3 (Estimating)
MCRedpdulFsm_generate: port=1 (Generating)
Or
MCRedpdulFsm_attemptToStartTraining: port=1 (Training) {Nothing in NV}
MCRedpdulFsm_startTransmission: port=1 session=2128
MCRedpdulFsm_startTransmission: port=1 pwr=31622 (Transmitting, backed
off)
MCRedpdulFsm_startTransmission: port=1 pwr=39810 (Transmitting, cell
power)
MCRedpdulFsm_ddcEventHandler: (HOLD complete)
MCRedpdulFsm_trainState: port=1 (Unlocked, discarding captures)
MCRedpdulFsm_trainState: port=1 clgcGainError=920
MCRedpdulFsm_attemptToStartTraining: port=2 (Not training) {Disabled}

If one of the above messages is repeating, dispatch a Tech to the site to

calibrate the radio. If the alarm returns replace the radio.
Verify antenna sharing configuration is correct at the site (div 0 = tx/rx; div
1=rx). Ensure that diversity 1 is not connected to the rx input of a duplexer.
Swap antennas with a known good sector and determine if the problem follows
the Node B or the antenna system

Lucent Technologies Proprietary

See notice on cover page
7

Node B Alarm Management

MajorGainFailure

August 21, 2006,

MajorGainFailure is generated when closed loop gain control has failed, and the radio has
switched over to OnBoard Gain Control.
Fault Clearance:

Execute Power Test if results are greater than 1dB, dispatch a Tech to the site to
verify power in closed loop and open loop mode.

Replace radio and calibrate (especially when MajorGainFailure is coupled with a

TxSpectrumFailure indication of the known HW cold soldering issue)

MinorGainFailure

MinorGainFailure can be caused by the following:

Missing Antenna Path descriptor data

Poor previous calibration

Node B internal RF cabling Issue

Fault Clearance:

Verify Antenna Path Descriptor data is present in the backplane using remote
maintenance terminal (RMT).

Dispatch an Installer to perform power calibration.

LossOfPulse

LossOfPulse can occur under the following conditions:

High temperature

Rx Signal Too strong (numerous Rx Blocks)

Faulty CTU

Faulty OM
Fault Clearance:

Verify the alarm is present on all 3 radios if yes replace OM or CTU

Telnet to the Multi-Carrier Radio (MCR) and type: tempSenseStats. If the radio
reports anything higher than 55 degrees C, report over-temperature condition to
Cingular Operations Team.

Dispatch Tech to the site to swap radios. If the problem followed the radio replace
the radio; else return the radio and troubleshoot antenna path and internal Node b
cables.

If the alarm appears with the antennas connected, and does not appear when
antennas are disconnected, dispatch antenna crew; else verify the internal Node B
cabling (recommend power calibration or swapping filters).

LossOfForwardB
usLink

One of the Field Programmable Gate Arrays has failed on the radio. The radio will
autonomously reset and perform BLST. If BLST passes, the radio will return to service; else
an alarm will be raised indicating Initialization Failure and will require the radio to be replaced.
FaultClearance:

This alarm is typically found in the alarm history and cleared alarm list.

Verify the radio has recovered; if not, replace radio and calibrate.

Lucent Technologies Proprietary

See notice on cover page
8

August 21, 2006,

Node B Alarm Management

IMAGROUP
CriticalGroupAlar
m

The IMA Group is no longer Operational. Most likely, all T1(s) have failed.
Fault Clearance:

Verify the alarm in the PSAX.

Request Cingular DACs support to check for any loops, and to loop the NIU and
CSU.

If loops are found and removed verify alarm has cleared and service has been
restored.

If the Network Interface Unit (NIU) (Demarc / smartjack) is not able to loop up,
Cingular will open a ticket with the T1 provider to report the outage.

If the NIU loops but the CSU does not, there is problem with the cabling between the
NIU and CSU (Node B) dispatch a Tech to resolve

MajorGroupAlar
m

The IMA Group remains operational; however it has lost one or more of the T1s associated
with the IMA Group.`
Fault Clearance:

Verify the alarm in the PSAX.

Request Cingular DACs support to check for any loops, and to loop the NIU and
CSU.

If loops are found and removed verify alarm has cleared and service has been
restored.

If the NIU (Demarc / smartjack) is not able to loop up, Cingular will open a ticket with
the T1 provider to report the outage.

If the NIU loops but the CSU does not, there is problem with the cabling between the
NIU and CSU (Node B) dispatch tech to resolve

NonCriticalGroup
Alarm

A NonCritical Configuration problem could indicate the following failure(s):

T1 outage

IMA GroupID does not match what is provisioned in the NDP on the OMC vs the
PSAX

PVCs blocked
Fault Clearance:

Verify the T1/IMA status and IMA configuration in the PSAX and compare that to
what is configured in the OMC.

Lucent Technologies Proprietary

See notice on cover page
9

Node B Alarm Management

August 21, 2006,

FILTER
DCConverterFail
ure

The DC converter inside the filter has detected a failure. Typically this alarm is caused by
someone onsite who has replaced a failed filter and has turned off or disconnected power to
the filter.
Fault Clearance:

Verify filter shelf breakers.

Verify DC cable is secure.

RxAmpFailure

RxAmp Failure can be caused by the following:

Lightning

Faulty filter

DC power/Comm cable issue

FaultClearance:

Dispatch a tech to the Site to reset the filter shelf. If filter remains in alarm replace
filter and calibrate sector.

EXTRXAMP
ExtRxAmpFailure

When the Node B provides the 12VDC power to the Tower Top Low Noise Amplifier (TTLNA)
(extRxAmp), the extRxAmp managed object should appear in the OMC GUI and should report
alarms based on the TTLNA Descriptor data for the Node B input during integration. If the
Filter Panel exceeds the limits specified, the DC bias powering the TTLNA will be turned off,
and the RF coverage area will shrink.
Fault Clearance:

Verify the TTLNA descriptor data is populated in the Filter Panel (note that any time a
filter is replaced, the TTLNA data must be re-entered and Node B reset in order for
changes to take effect).

Using the RMT, sample the current reading of the TTLNA. If it is out of the alarm
range, dispatch a Tech to troubleshoot further onsite.

Onsite request a Tech to swap antenna 0,1. If the problem followed the antenna
dispatch the antenna crew to resolve. If the problem followed the filter- replace the
filter and calibrate the associated sector.

Lucent Technologies Proprietary

See notice on cover page
10

August 21, 2006,

Node B Alarm Management

CELL
ReceiveAntennaT
estThresholdExc
eeded

RxAntennaTest is a useful tool to measure the Return Loss of the antenna system. However,
it has some limitations to be aware of:
1) Antenna test cannot sweep past TTLNA if the filter is not powering the TTLNA. The
path from the Node B External Antenna Connection (EAC) to the TTLNA Input will be
tested, but not beyond.
2) RxAntenna test cannot sweep past an RxAIT antenna-sharing configuration. RxAIT
splits the diversity into separate tx/rx paths. Anything beyond the RxAIT equipment
will be untested.
FaultClearance:

RxAntennaTest alarms indicate that the antenna test has failed and an
antenna crew should be dispatched to resolve issue.
Note: The RxAntennaTest could pass and indicate Test Unit Not Available. If this message
occurs or the RxAntennaTest Times Out Unmanage/Manage the Node B and try again.

PowerTestThresh
oldExceeded

Power Test is a useful tool to measure instantaneous Peak RF Ouput power at the Node B
Equipment Antenna Connector (EAC) without having to send someone to the site to verify.
Typically for a 40W configured system, the power output range should be between 36 and 38
dBm. The power test will fail if the expected power and the detected power differ by more
than 1dB (aka Err DET or Results).
Fault Clearance:

Execute 3 or more consecutive power tests. If all the results are

consistently within the 36-38dBm range, but the results are above the 1dB
threshold, dispatch someone to the site to perform calibration on the
associated sector.

If the test results pass/fail with peak measurements less than 15dB, verify
the logical cells are enabled, and that Common Pilot Channel (CPICH) and
Common Channel Configuration is configured on the Node B.

If the test results pass/fail with results greater than 40dBm, dispatch a Tech
to the site to verify Power in Open Loop and Closed Loop mode to
determine if the amplifier and radio are operating properly (typically the
failed radios is outputting messages, indicating that edpdul is training every
couple of minutes. Also, a MajorTxSpectrum alarm may be present. Check
/ troubleshoot radio alarms).

Lucent Technologies Proprietary

See notice on cover page
11

Node B Alarm Management

August 21, 2006,

AMPLIFIER
OverTemperature

The amplifiers temperature sensor has detected a temperature above the specified limit.
Fault Clearance:

If the alarm exists on all amplifiers dispatch a Tech to the site to verify the fans are
operating and the ambient temperature is within normal operating ranges.

Reset the amplifier if the alarm continuously reappears/clears replace amplifier.

RFOverDrive

The amplifier has detected an overdrive condition that could be caused by one of the
following:

Excessive Transmitter output

Faulty calibration

Failed Radio

Failed Amplifier
Fault Clearance:

Reset the amplifier if the alarm re-appears reset the radio.

Dispatch a Tech to the site to calibrate this sector.

Suspect Radio replace radio and calibrate.

Suspect Amplifier replace amplifier and verify RF power.

CommunicationF
ailure

Communication failure on the amplifier can be caused by the following:

Faulty I2C cables

Faulty PDPII

No Power to amplifier

Faulty amplifier
Fault Clearance:

Inspect all circuit breakers, cables.

Power cycle the amplifier circuit breaker.

Verify RF Power Closed Loop.

Replace amplifier (ensure power is off when RJ45 cable is plugged into amplifier
/Power Distribution Panel (PDP), or else the amp could go into a bench test mode
power cycle will resolve).

Perform a continuity check on the cables to ensure they are correctly wired and
completed terminated.

Replace PDP.

InitializationFailu
re

Amplifier has established communication to the Controller. However, the amplifier has failed
to initialize completely.
Fault Clearance:

Reset Amplifier. If no change replace amplifier and perform closed loop power
verification.

Lucent Technologies Proprietary

See notice on cover page
12

August 21, 2006,

Node B Alarm Management

RF PATH DIAGRAM

J41 J22-C

LNA
-1
TTLNA
-1
RF-ON
-1

J2-C

TTLNA
-0

4
1

J24

Dx1

J26

J31

J8

J28

J7

J11

J6

J5

J25

J4

2
4

Lucent

J8

J
Dx0

J4
1
J
1

RF-ON
-0

LNA
-0

Filter

AM
P
RF IN

Lucen
t

AC
T

B3

RF
OUT

B2
(RX2)

Antenna Test

B1

B2
(RX1)

ALM

MCR
MCR

Tx1

Lucen
t

Tx1

Tx On

Tx
Off

Tx2

R
1
C
RX1

Ton
e

Ton
e

STA
T
Tx1-ON
T
2-On
X

R
1
C

R
1
x

R
2
C

RX2

R
2
x

Lucent Technologies Proprietary

See notice on cover page
13

Node B Alarm Management

August 21, 2006,

RXAIT EXAMPLE CONFIGURATION

Lucent Technologies Proprietary

See notice on cover page
14