You are on page 1of 148

Technical

Information

SDU TROUBLESHOOTING

SOFTWARE RELEASE 2.16.3.X


SDU
CDMA2000 1X

ENGLISH
APR 2004
68P09258A80B

SPECIFICATIONS SUBJECT TO CHANGE WITHOUT NOTICE


Notice
While reasonable efforts have been made to assure the accuracy of this document, Motorola, Inc. assumes no liability resulting from any
inaccuracies or omissions in this document, or from use of the information obtained herein. The information in this document has been
carefully checked and is believed to be entirely reliable. However, no responsibility is assumed for inaccuracies or omissions. Motorola,
Inc. reserves the right to make changes to any products described herein and reserves the right to revise this document and to make
changes from time to time in content hereof with no obligation to notify any person of revisions or changes. Motorola, Inc. does not
assume any liability arising out of the application or use of any product, software, or circuit described herein; neither does it convey
license under its patent rights or the rights of others.
It is possible that this publication may contain references to, or information about Motorola products (machines and programs),
programming, or services that are not announced in your country. Such references or information must not be construed to mean
that Motorola intends to announce such Motorola products, programming, or services in your country.

Copyrights
This instruction manual, and the Motorola products described in this instruction manual may be, include or describe copyrighted
Motorola material, such as computer programs stored in semiconductor memories or other media. Laws in the United States and
other countries preserve for Motorola and its licensors certain exclusive rights for copyrighted material, including the exclusive
right to copy, reproduce in any form, distribute and make derivative works of the copyrighted material. Accordingly, any
copyrighted material of Motorola and its licensors contained herein or in the Motorola products described in this instruction manual
may not be copied, reproduced, distributed, merged or modified in any manner without the express written permission of Motorola.
Furthermore, the purchase of Motorola products shall not be deemed to grant either directly or by implication, estoppel, or
otherwise, any license under the copyrights, patents or patent applications of Motorola, as arises by operation of law in the sale of a
product.

Usage and Disclosure Restrictions


License Agreement
The software described in this document is the property of Motorola, Inc and its licensors. It is furnished by express license
agreement only and may be used only in accordance with the terms of such an agreement.

Copyrighted Materials
Software and documentation are copyrighted materials. Making unauthorized copies is prohibited by law. No part of the software or
documentation may be reproduced, transmitted, transcribed, stored in a retrieval system, or translated into any language or
computer language, in any form or by any means, without prior written permission of Motorola, Inc.

High Risk Activities


Components, units, or thirdparty products used in the product described herein are NOT faulttolerant and are NOT designed,
manufactured, or intended for use as online control equipment in the following hazardous environments requiring failsafe
controls: the operation of Nuclear Facilities, Aircraft Navigation or Aircraft Communication Systems, Air Traffic Control, Life
Support, or Weapons Systems (High Risk Activities). Motorola and its supplier(s) specifically disclaim any expressed or implied
warranty of fitness for such High Risk Activities.

Trademarks

MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are
the property of their respective owners.
Copyright 2003 Motorola, Inc.
Javat Technology and/or J2MEt: Java and all other Javabased marks are trademarks or registered trademarks of Sun
Microsystems, Inc. in the U.S. and other countries.
UNIXR: UNIX is a registered trademark of The Open Group in the United States and other countries.
REV091302

Table of Contents
SDU Troubleshooting
Software Release 2.16.3.x

List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

iii

List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

iv

Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vii

General Safety . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ix

Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xi

Chapter 1: Introduction
About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-1

SDU Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-2

Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-5

Chapter 2: SDU Card Reset Procedure


SDU Card Reset Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2-1

Chapter 3: Login/Passwords
Log in to SDU Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3-1

Chapter 4: Data Collection


Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4-1

Chapter 5: SDU Cage Reset Procedure


SDU Cage Reset Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-1

Chapter 6: SDU General Status


SDU General Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6-1

Chapter 7: SDU Initialization Failure Scenarios


SDU Initialization Failure Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

APR 2004

SDU Troubleshooting

7-1

Table of Contents continued

Chapter 8: ISB/SPROC Troubleshooting


ISB/SPROC Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8-1

Verifying ISB/SPROC Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8-2

Troubleshooting ISB/SPROC Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8-3

Troubleshooting ISB/SPROC Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8-9

Chapter 9: SDU Synchronization Failures


SDU Synchronization Failure Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9-1

Chapter 10: BPP Troubleshooting


BPP Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10-1

Verifying BPP Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10-2

Troubleshooting BPP Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10-3

Chapter 11: CCA Troubleshooting


CCA Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-1

Verifying CCA Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-2

Troubleshooting CCA Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-3

Chapter 12: OMCR Failure Scenarios


OMCR Failure Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12-1

Chapter 13: System Initialization Problems


System Initialization Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13-1

Chapter 14: Call Processing Failures


Call Processing Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14-1

Chapter 15: CFC Failures


SDU CFC Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ii

SDU Troubleshooting

15-1

APR 2004

List of Figures
SDU Troubleshooting
Software Release 2.16.3.x

APR 2004

Figure 1-1: SDU Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-2

Figure 1-2: SDU Cage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-3

Figure 1-3: SDU Cage Card Location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-3

Figure 1-4: SDU Top I/O Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-4

Figure 5-1: SDU Main Breaker Panel (MBP) . . . . . . . . . . . . . . . . . . . . . . . . . .

5-1

Figure 5-2: SDU Power Supply Modules and Circuit Breaker Modules . . . . . .

5-2

SDU Troubleshooting

iii

List of Tables
SDU Troubleshooting
Software Release 2.16.3.x

iv

Table 1-1: Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-5

Table 7-1: SPROC Initialization Does Not Complete . . . . . . . . . . . . . . . . . . . .

7-2

Table 8-1: SPROC Unable to get IP address from ISB . . . . . . . . . . . . . . . . . . .

8-4

Table 8-2: ISB Fails to establish Gigabit Link . . . . . . . . . . . . . . . . . . . . . . . . . .

8-6

Table 8-3: Common OMCR Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8-10

Table 8-4: SDU Communications Alarm 2822100:


Communication Fault, Physical Link Connection Error
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8-11

Table 8-5: SDU Quality of Service Alarm 2823220:


Quality of Serv Fault, Packet Routing Capacity Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8-11

Table 8-6: SDU Processing Error Alarm 2823390:


Processing Fault, Initialization Failure Troubleshooting Procedure . . . . . . . . .

8-12

Table 8-7: SDU Processing Error Alarm 2823420:


Processing Fault, Initialization Failure Troubleshooting Procedure . . . . . . . . .

8-13

Table 8-8: SDU Device Out of Service Alarm 2825000:


Device Out of Service Contact Lost Troubleshooting Procedure . . . . . . . . . .

8-13

Table 8-9: SDU Device Out of Service Alarm 2825040:


Device Out of Service Device Locked Troubleshooting Procedure . . . . . . . .

8-14

Table 8-10: SDU Alarm 2826555:


Application Heartbeat Failure Troubleshooting Procedure . . . . . . . . . . . . . . . .

8-15

Table 8-11: SDU Alarm 2826560:


Application Messaging Failure Troubleshooting Procedure . . . . . . . . . . . . . . .

8-15

Table 8-12: SDU BPP Alarm 2826570:


Communication Failure Application Messaging Failure
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8-16

Table 8-13: SDU Alarm 2826580:


Application Messaging Failure Troubleshooting Procedure . . . . . . . . . . . . . . .

8-16

Table 8-14: SDU Alarm 2826720:


Application Initialization Failure Troubleshooting Procedure . . . . . . . . . . . . . .

8-17

Table 8-15: SDU Alarm 2826730:


Application Initialization Failure Troubleshooting Procedure . . . . . . . . . . . . . .

8-17

Table 8-16: SDU Alarm 2826740:


Internal SCTP Stack Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . . .

8-18

Table 8-17: SDU Alarm 2826770:


Capacity Greatly Reduced Troubleshooting Procedure . . . . . . . . . . . . . . . . .

8-18

SDU Troubleshooting

APR 2004

List of Tables continued

APR 2004

Table 8-18: SDU Alarm 2826850:


Application Platform Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . .

8-19

Table 8-19: SDU Alarm 2826870:


Incomplete Dataload Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . .

8-19

Table 8-20: SDU Alarm 2826500:


Communication Failure Lost Connection with the MM
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8-20

Table 8-21: SDU Alarm 2826520:


Communication Failure Link Disabled Troubleshooting Procedure . . . . . .

8-21

Table 8-22: SDU Alarm 2826530:


Communication Failure Lost Communication with the PDSN
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8-21

Table 9-1: Troubleshoot SDU Heartbeat Failure . . . . . . . . . . . . . . . . . . . . . . . .

9-2

Table 10-1: Regenerate Corrupted Software . . . . . . . . . . . . . . . . . . . . . . . . . . .

10-3

Table 10-2: SDU BPP Alarm 2826000:


Application Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . .

10-4

Table 10-3: SDU BPP Alarm 2826001:


Application Error Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . .

10-4

Table 10-4: SDU BPP Alarm 2826010:


Call processing Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . .

10-5

Table 10-5: SDU BPP Alarm 2826011:


Call Processing Error Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . .

10-5

Table 10-6: SDU BPP Alarm 2826020:


Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10-6

Table 10-7: SDU BPP Alarm 2826030:


CPU Failure Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10-6

Table 10-8: SDU BPP Alarm 2826040:


Hardware Component Error Troubleshooting Procedure . . . . . . . . . . . . . . . . . .

10-7

Table 10-9: SDU BPP Alarm 2826050:


Hardware Component Failure Troubleshooting Procedure . . . . . . . . . . . . . . . .

10-7

Table 10-10: SDU BPP Alarm 2826060:


Partial Call Processing Capacity Loss Troubleshooting Procedure . . . . . . . . . .

10-8

Table 10-11: SDU BPP Alarm 2826070:


Full Call Processing Capacity Loss Troubleshooting Procedure . . . . . . . . . . . .

10-8

Table 10-12: SDU BPP Alarm 2826080:


Switch Processing Error Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . .

10-9

Table 10-13: SDU BPP Alarm 2826090:


ISB Link Loss Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . .

10-9

Table 10-14: SDU BPP Alarm 2826100:


File Access Error Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . .

10-10

Table 10-15: SDU BPP Alarm 2826110:


High Temperature Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . .

10-10

Table 10-16: SDU BPP Alarm 2826111:


Very High Temperature Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . .

10-11

Table 10-17: SDU BPP Alarm 2826140:


CPU Software Initialization Error Troubleshooting Procedure . . . . . . . . . . . . .

10-11

SDU Troubleshooting

List of Tables

vi

continued

Table 10-18: SDU BPP Alarm 2826150:


CPU Software Initialization Failure Troubleshooting Procedure . . . . . . . . . . . .

10-12

Table 10-19: SDU Communications Alarms 2821801 and


2821802: Communication Fault, Packet Bus Connection no. <12>
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10-12

Table 11-1: SDU Processing Error Alarm 2823380:


Processing Fault, Illegal Configuration Change Troubleshooting Procedure . .

11-3

Table 11-2: SDU Processing Error Alarm 2823410:


Processing Fault, Initialization Failure Troubleshooting Procedure . . . . . . . . .

11-4

Table 11-3: SDU Equipment Alarm 2823670:


Equipment Fault, Device Removal Detected Troubleshooting Procedure . . . . .

11-4

Table 11-4: SDU Equipment Alarm 2823690:


Equipment Fault, Hardware Fault Troubleshooting Procedure . . . . . . . . . . . . .

11-5

Table 11-5: SDU Equipment Alarm 2823790:


Equipment Fault, Subcomponent Fault Troubleshooting Procedure . . . . . . . . .

11-6

Table 11-6: SDU Environmental Alarm 2823900:


Environmental Fault, Power Supply Input Circuit Fault
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-6

Table 11-7: SDU Environmental Alarm 2824400:


Environmental Fault, AC Mains Under Volt Threshold Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-7

Table 11-8: SDU Environmental Alarm 2824440:


Environmental Fault, Bus Under Voltage Threshold Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-7

Table 11-9: SDU Environmental Alarm 2824460:


Environmental Fault, Cabinet Sensor Threshold Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-8

Table 11-10: SDU Environmental Alarm 2824480:


Environmental Fault, External Sensor Threshold Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-8

Table 11-11: SDU Environmental Alarm 2824700:


Environmental Fault, Temperature High Threshold Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-9

Table 11-12: SDU Environmental Alarm 2824740:


Environmental Fault, Temperature Very High Threshold Exceeded
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-10

Table 11-13: SDU Communications Alarms 2821351 through 2821362:


Communication Fault, Electronic ID 1wire Bus Connection no. <112>
Troubleshooting Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-11

Table 11-14: SDU Communications Alarms 2821801 and 2821802:


Communication Fault, Packet Bus Connection no. <12> Troubleshooting
Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-11

SDU Troubleshooting

APR 2004

Foreword

Scope of manual
This manual is intended for use by cellular telephone system
craftspersons in the day-to-day operation of Motorola cellular system
equipment and ancillary devices.
This manual is not intended to replace the system and equipment
training offered by Motorola, although it can be used to supplement or
enhance the knowledge gained through such training.
Obtaining Manuals
To view, download, or order manuals (original or revised), visit the
Motorola Lifecycles Customer web page at http://services.motorola.com,
or contact your Motorola account representative.
If Motorola changes the content of a manual after the original printing
date, Motorola publishes a new version with the same part number but a
different revision character.
Text conventions
The following special paragraphs are used in this manual to point out
information that must be read. This information may be set-off from the
surrounding text, but is always preceded by a bold title in capital letters.
The four categories of these special paragraphs are:
NOTE
Presents additional, helpful, non-critical information that
you can use.

IMPORTANT

Presents information to help you avoid an undesirable


situation or provides additional information to help you
understand a topic or concept.

CAUTION
Presents information to identify a situation in which
damage to software, stored data, or equipment could occur,
thus avoiding the damage.

WARNING
Presents information to warn you of a potentially
hazardous situation in which there is a possibility of
personal injury.
APR 2004

SDU Troubleshooting

vii

Foreword continued

The following typographical conventions are used for the presentation of


software information:
S In text, sans serif BOLDFACE CAPITAL characters (a type style
without angular strokes: for example, SERIF versus SANS SERIF)
are used to name a command.
S In text, typewriter style characters represent prompts and the
system output as displayed on an operator terminal or printer.
S In command definitions, sans serif boldface characters represent
those parts of the command string that must be entered exactly as
shown and typewriter style characters represent command output
responses as displayed on an operator terminal or printer.
S In the command format of the command definition, typewriter
style characters represent the command parameters.
Reporting manual errors
To report a documentation error, call the CNRC (Customer Network
Resolution Center) and provide the following information to enable
CNRC to open an MR (Modification Request):
the document type
the manual title, part number, and revision character
the page number(s) with the error
a detailed description of the error and if possible the proposed solution
Motorola appreciates feedback from the users of our manuals.
Contact us
Send questions and comments regarding user documentation to the email
address below:
cdma.documentation@motorola.com
Motorola appreciates feedback from the users of our information.
Manual banner definitions
A banner (oversized text on the bottom of the page, for example,
PRELIMINARY) indicates that some information contained in the
manual is not yet approved for general customer use.
24-hour support service
If you have problems regarding the operation of your equipment, please
contact the Customer Network Resolution Center (CNRC) for immediate
assistance. The 24 hour telephone numbers are:
North America
Europe, Middle East, Africa
Asia Pacific
Japan & Korea . . . . . . . . . . .
Latin American Countries

+18004335202
+44 (0) 1793565444
+861088417733
+81354633550
+5112124020

For further CNRC contact information, contact your Motorola account


representative.
viii

SDU Troubleshooting

APR 2004

General Safety

Remember! . . . Safety
depends on you!!
The following general safety precautions must be observed during all
phases of operation, service, and repair of the equipment described in
this manual. Failure to comply with these precautions or with specific
warnings elsewhere in this manual violates safety standards of design,
manufacture, and intended use of the equipment. Motorola, Inc. assumes
no liability for the customers failure to comply with these requirements.
The safety precautions listed below represent warnings of certain dangers
of which we are aware. You, as the user of this product, should follow
these warnings and all other safety precautions necessary for the safe
operation of the equipment in your operating environment.
Ground the instrument
To minimize shock hazard, the equipment chassis and enclosure must be
connected to an electrical ground. If the equipment is supplied with a
three-conductor ac power cable, the power cable must be either plugged
into an approved three-contact electrical outlet or used with a
three-contact to two-contact adapter. The three-contact to two-contact
adapter must have the grounding wire (green) firmly connected to an
electrical ground (safety ground) at the power outlet. The power jack and
mating plug of the power cable must meet International Electrotechnical
Commission (IEC) safety standards.
NOTE
Refer to Grounding Guideline for Cellular Radio
Installations 68P81150E62.

Do not operate in an explosive


atmosphere
Do not operate the equipment in the presence of flammable gases or
fumes. Operation of any electrical equipment in such an environment
constitutes a definite safety hazard.
Keep away from live circuits
Operating personnel must:

S not remove equipment covers. Only Factory Authorized Service


Personnel or other qualified maintenance personnel may remove
equipment covers for internal subassembly, or component
replacement, or any internal adjustment.

S not replace components with power cable connected. Under certain


conditions, dangerous voltages may exist even with the power cable
removed.

S always disconnect power and discharge circuits before touching them.


APR 2004

SDU Troubleshooting

ix

General Safety continued

Do not service or adjust alone


Do not attempt internal service or adjustment, unless another person,
capable of rendering first aid and resuscitation, is present.
Use caution when exposing or
handling the CRT
Breakage of the CathodeRay Tube (CRT) causes a high-velocity
scattering of glass fragments (implosion). To prevent CRT implosion,
avoid rough handling or jarring of the equipment. The CRT should be
handled only by qualified maintenance personnel, using approved safety
mask and gloves.
Do not substitute parts or
modify equipment
Because of the danger of introducing additional hazards, do not install
substitute parts or perform any unauthorized modification of equipment.
Contact Motorola Warranty and Repair for service and repair to ensure
that safety features are maintained.
Dangerous procedure
warnings
Warnings, such as the example below, precede potentially dangerous
procedures throughout this manual. Instructions contained in the
warnings must be followed. You should also employ all other safety
precautions that you deem necessary for the operation of the equipment
in your operating environment.
WARNING
Dangerous voltages, capable of causing death, are present in this
equipment. Use extreme caution when handling, testing, and
adjusting .

SDU Troubleshooting

APR 2004

Revision History

Manual Number
68P09258A80B
Manual Title
SDU Troubleshooting
Software Release 2.16.3.x
Version Information
The following table lists the manual version, date of version, and
remarks on the version.
Version
Level

Date of Issue

Remarks

APR 2004

Added notes stating that logging into SDU boards should be done during
a maintenance window.

APR 2004

SDU Troubleshooting

xi

Chapter 1: Introduction
Table of Contents

APR 2004

About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Associated Manuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-1
1-1
1-1

SDU Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SDU Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-2
1-2
1-2

Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1-5
1-5

SDU Troubleshooting

Table of Contents

continued

Notes

SDU Troubleshooting

APR 2004

About this Manual

About this Manual


This manual includes procedures for troubleshooting the hardware
platform and associated software for the CDMA SDU (Selection
Distribution unit) system version 2.16.3.
IMPORTANT

The procedures in this manual end when the symptom(s)


that caused the alarm(s) has been corrected. They DO NOT
supercede Scheduled Preventive Maintenance procedures.

IMPORTANT

Some of the procedures in this manual must be performed


only during a scheduled maintenance window. Logging
into the SPROC, ISB, and BPP devices to perform
troubleshooting activities should be performed only during
a maintenance window to avoid system disruption.

NOTE
Additional performance tests may be required to insure
that the SDU still meets ATP specifications after any
device is replaced. Refer to the applicable Field
Replacement Unit (FRU) and ATP manual for this
information.

Associated Manuals
The following manuals and online help systems are referenced in this
manual.

S
S
S
S
S
S
S
S
S

APR 2004

AN & SDU FRU (Field Replacement Procedures)


CDMA2000 1X RAN Hardware Installation
CDMA2000 1X RAN Functional Description
CDL/CFC Reference Manual
SMAP Operators Guide
Cellular System Administration Guide (Online Help)
System Command Reference (Online Help)
System Output Messages (Online Help)
SDU ATP (Acceptance Test Procedure)

SDU Troubleshooting

1-1

SDU Overview

Overview

The SDU performs the Selection/Distribution Function and Packet


Control Function for the 2.16.1.x network. For details, refer to the
CDMA2000 1X RAN Functional Description.
SDU Frame

The following figures illustrate the SDU frame, card cage, and top I/O
Panel.
Figure 1-1: SDU Frame

75

75

75

MAIN BREAKER PANEL

75

20

20

20

20

20

20

20

20

20

20

POWER SUPPLY
MODULES (4)
CIRCUIT BREAKER
MODULES (2)

UPPER SDU CAGE

FAN MODULES

LOWER SDU CAGE

FANS MODULES

NOTE:
1. Frame shown with doors removed
2. System configuration determines if upper cage is used.

1-2

SDU Troubleshooting

APR 2004

SDU Overview

continued

Figure 1-2: SDU Cage

Figure 1-3: SDU Cage Card Location


ISB card has a SPROC card mounted on it
CCA

APR 2004

BPP

ISB

ISB

SDU Troubleshooting

CCA

1-3

SDU Overview

continued

Figure 1-4: SDU Top I/O Panel

1-4

SDU Troubleshooting

APR 2004

Acronyms

Acronyms
Table 1-1 defines acronyms used in this manual.
Table 1-1: Acronyms
Item

Description

AN

Access Node

BPP

Bearer Payload Processor

CBM

Circuit Breaker Module

CCA

Customer Cabinet Alarms

DVM

Digital Volt Meter

FRU

Field Replaceable Unit

GND

Ground

ISB

Interface Switch Board

MBP

Main Breaker Panel

MMI

ManMachine Interface

PDU

Power Distribution Unit

PSM

Power Supply Module

RTN

Return

SDU

Selection Distribution Unit

SPROC

System Processor

STP

Shielded Twisted Pair

T1C

Tier 1 Controller

NOTE
T1C is a different name for the ISB/SPROC.
UTP

APR 2004

SDU Troubleshooting

Unshielded Twisted Pair

1-5

Acronyms

continued

Notes

1-6

SDU Troubleshooting

APR 2004

Chapter 2: SDU Card Reset Procedure


Table of Contents

SDU Card Reset Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Hardware vs. Software Reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Location of Reset Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Resetting the Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Lock/Unlock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Verify Successful Reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

APR 2004

SDU Troubleshooting

2-1
2-1
2-1
2-1
2-1
2-2
2-2

Table of Contents

continued

Notes

SDU Troubleshooting

APR 2004

SDU Card Reset Procedure

Overview

Many SDU BPP and SDU CCA alarm troubleshooting procedures


require a reset to resolve the alarm.

Generally, the alarm troubleshooting procedures conform to the


following variations:
S Reset not required

S Reset only required if the systems automatic reset fails


S Reset is up to the operator to perform when to reset depends upon
the urgency of the alarm.
WARNING
When manually resetting SDU cards, the call processing
resources will temporarily be unavailable. If possible,
attempt the reset during an appropriate maintenance
window (determine the severity of the alarm).

Hardware vs. Software Reset

When resetting cards to resolve an alarm, use the hardware reset method
(frontpanel reset switch) as opposed to the software reset method (e.g.
an OMCR reset command).

Location of Reset Switches

The BPP cards and CCA cards have hardware reset switches. The reset
switch is located in an unlabeled pinhole in the front panel of the card.

Resetting the BPPs/CCAs

To reset the card, insert an appropriately sized, nonconductive tool


(such as a toothpick), then press straight in with very light pressure.

Resetting the ISB/SPROCs

Since the ISB/SPROC does not have a frontpanel reset switch, it is


only possible to reset the ISB/SPROC by using the lock/unlock
commands from the OMCR, and by reseating the board (to do so refer
to the RAN Hardware Installation).
WARNING
Never reseat or reset the SPROC for any reason, unless
Motorola CNRC has given the instruction to do so. If this
is the case, first lock the SPROC from the OMCR.
APR 2004

SDU Troubleshooting

2-1

SDU Card Reset Procedure

continued

Lock/Unlock

When manually resetting cards (as opposed to an automatic reset by the


system), use the OMCR lock and unlock commands. Lock removes the
card from service and unlock restores the card to service. For details,
refer to the System Command Reference.

Verify Successful Reset

NOTE
This case applies to both an automatic system reset and a
manual reset.

To verify that the reset of the card completes successfully:

S The cards frontpanel LED transitions from red (alarm state) to


blinking green to indicate that it is resetting. Once the reset completes,
the LED appears solid green.

S The OMCR should issue a state change event indicating that the card
has come back into service. You can use an OMCR Display
command to verify this. For example, if you reset a BPP card with
SDF software, the OMCR will indicate that the SDF is in service.

2-2

SDU Troubleshooting

APR 2004

Chapter 3: Login/Passwords
Table of Contents

3
Log in to SDU Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Accessing Devices Using the OMCR . . . . . . . . . . . . . . . . . . . . . . . . .
Retrieving Information from Circuit BTS Configurations . . . . . . . . . .
Accessing the MMI Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Recovery Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

APR 2004

SDU Troubleshooting

3-1
3-1
3-1
3-3
3-4
3-4
3-5

Table of Contents

continued

Notes

SDU Troubleshooting

APR 2004

Log in to SDU Cards

Overview

Occasionally, an SDU troubleshooting procedure requires logging in to a


card. The following information provides directions on how to login to a
device remotely without accessing the front panel of the device. This
section explains how to log in to the various SDU cards.
IMPORTANT

Logging into the SPROC, ISB, and BPP devices should be


performed only during a maintenance window to avoid
system disruption.

Accessing Devices Using the


OMCR

Most of the SDU devices can be accessed remotely (not on site using the
front panel MMI interface) using the OMCR. This action makes it easy
to gain access to the device and collect data. The following procedures
provide the information needed to access the SDU devices using the
OMCR.
Locating SDU IP Addresses
At the OMCR enter the following command:

S DISP SDU<sduid> IPADDRESS


The output of the display command provides the following information:
Table 3-1: Interpreting and Retrieving IP Addresses from the
DISPLAY Command Output
Item

APR 2004

Description

SDU1 Management Link

IP address of the
active SPROC

SDU1 Router 2

IP address of the
active ISB

SDU2 Router 2

IP address of the
standby ISB

BPP<frameid><cageid><slotid> BCP

IP of LOGIN
ADDRESS of SDF
or PCF

BPP<frameid><cageid><slotid> CPE

IP address of the first


DSP of the SDF or
PCF

SDU Troubleshooting

3-1

Log in to SDU Cards

continued

WARNING
Any of the following actions could prevent future access to
the SDU boards:
S Never use the telnet command to log into any of the
SDU boards.

S Never use the Exit command to logout of any of the


SDU boards.

S Never used CTRL+X to logout of the SDU boards.

SPROC
At the OMCR use the following command to access to the SPROC.

S rlogin SDU1 MANAGEMENT_LINK<IP address>


To start a CLI session, type CLI at the prompt.
To exit the SPROC type LOGOUT at the prompt.
ISB
At the OMCR use the following command to access to the ISB.

S rlogin SDU1 ROUTERX<IP address>


To start a CLI session, type CLI at the prompt.
To exit the SPROC type LOGOUT at the prompt.
BPP
At the OMCR use the following command to access to the BPP.

S rlogin SDUBCP<IP address>


To start a CLI session, type CLI at the prompt.
To exit the SPROC type LOGOUT at the prompt.

3-2

SDU Troubleshooting

APR 2004

Log in to SDU Cards

continued

Retrieving Information from


Circuit BTS Configurations

The following two items contain directions on how to display items from
a circuit BTS configuration site.
Locating the PSI Address (for Circuit BTS Configurations)
At the OMCR enter the following command:

S DISP XC<xcid> STATUS ISO ALL PKTIF

Determine which PKTIF by entering the following command:

S DISP PKTIF<cageid>>slotid> PKTIFCONF


Display the PSI IP address of the PKTIF by entering the following
command:

S DISP PSI<cageid><slotid> BOARDCONF


PKTIF (on the Transcoder)
At the OMCR enter the following command:

S DISP XCX STATUS ISO ALL PKTIF


S DISP PKTIFXX PKTIFCONF
S DISP PSI## ipaddress
user: psi
password: rulesdata

S To start a CLI session enter: psiStart


To exit enter CTRL+}
send escape
quit

APR 2004

SDU Troubleshooting

3-3

Log in to SDU Cards

continued

Accessing the MMI Console

The MMI console is used to enter commands. Each SDU card has an
MMI console connection.
IMPORTANT

*
3

Logging into the SPROC, ISB, and BPP devices should be


performed only during a maintenance window to avoid
system disruption.

Configuration

To log in to SDU cards, you will need to obtain a PC with a Serial port,
and a terminal emulation program such as Hyperterminal. A
standalone dumb terminal can also be used.
The settings consist of:

S Baud Rate: 9600 Bps


Data: 8 Bit

S
S
S
S

Parity: None
Stop bit: 1
Flow control: none
VT100 Emulation

It is necessary to use a Null Modem cable to connect the PC to the


ISB/SPROC.
NOTE

S The MMI Console Interface is started when the network


element is booted; it is not necessary to manually start
the MMI Console Interface.

S After the terminal/PC is connected to the card, you may


need to press the <Enter> key several times.

S SPROC
Login: motorola
Password: developer
Exit MMI Password: partymoto
NOTE
Use startMMI to enter MMI mode again.

3-4

SDU Troubleshooting

APR 2004

Log in to SDU Cards

continued

S BPP
Login: motorola
Password: developer
Secure MMI: altivec

S CCA
Login: motorola
Password: developer

S ISB
Login: motorola
Password: developer
Enter logout to quit.
Recovery Procedure

If the rlogin command for the BPP command does not work, the
following procedure can be performed at the site.
Connect a serial port to the front panel of the BPP board.
At the prompt enter the following:

S lock
S shell Restart
If the lock and restart commands do not work, perform an lock/unlock
command to see if the session can be recovered.

APR 2004

SDU Troubleshooting

3-5

Log in to SDU Cards

continued

Notes

3-6

SDU Troubleshooting

APR 2004

Chapter 4: Data Collection


Table of Contents

Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Overview of Black Box Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Running collectbox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Running bboxlisten . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Accessing the Collected Black Boxes . . . . . . . . . . . . . . . . . . . . . . . . . .

APR 2004

SDU Troubleshooting

4-1
4-1
4-1
4-1
4-2
4-3
4-4

Table of Contents

continued

Notes

SDU Troubleshooting

APR 2004

Data Collection

Data Collection

When contacting Motorola CNRC for assistance with SDU problems,


consider collecting the following data:

S Log current status information from the OMCR


S Optionally, also collect black box data from the OMCR.
This section describes the process for collecting the black box data.
Overview of Black Box
Programs

collectbox
The collectbox program is intended to be used on an OMCR to collect
debug information from the SDU. The debug information being gathered
is called a blackbox (airplane lingo) and contains information recorded
on the controller (SPROC).
bboxlisten
The bboxlisten program collects the blackbox logs from the SDU in the
case of an SPROC reset. Bboxlisten also provides some automatic bbox
file maintenance on the OMCR.
Prerequisites

The OMCR Software Installation should include the black box


programs for SDU. Installing software on the OMCR is beyond the
scope of this manual.
The following is a brief overview of the install process for reference
purposes: untar collect.tar into a temporary directory. Run installbbox
from the temporary directory to copy the executables and scripts to the
correct directory on the OMCR. Run cronbbox INSTALL to start the
server.

APR 2004

SDU Troubleshooting

4-1

Data Collection

continued

Running collectbox

When to Run collectbox


In a nonreset situation, run collectbox to collect data from the SDU.
Running collectbox
From the OMCR, use the following command to run collectbox:
collectbox [ne ne# | ip ipaddress]
[p | f filter]
[active | standby | both ]
[d directory]

4
Setting the ne# (network element id or sdu number) or ipaddress is
mandatory. Optional parameters can be entered in any order.
NOTE
If not specified, the ne # is resolved from ip address. If the
ip address is not specified, the OMCR will pull the
address from the MIB based on the ne # specified.

Optional Parameters
Optional parameters:

S [f filter] Grabs the full blackbox using the specified filter, the
blackbox is not reset as part of the collection. The default behavior is
to collect the filtered blackbox from the active side. If neither the f or
p parameters are specified, the full unfiltered blackbox is collected.
Default behavior for the full blackbox is collection from the active
side.

S [p] Grabs the preserved blackboxes from the last reset. The default
behavior is to grab the preserved blackboxes from both the active and
the standby controllers. If neither the f or p parameters are
specified, the full unfiltered blackbox is collected. Default behavior
for the full blackbox is collection from the active side.

S [d directory] Specify the directory for output to be stored on the


OMCR. The default directory when none is specified is
/sc/spool/bbox.

S [active] Collect the blackbox from only the active side.


S [standby] Collect the blackbox from only the standby side.
S [both] Collect the blackbox from both sides.

4-2

SDU Troubleshooting

APR 2004

Data Collection

continued

Examples
The following examples illustrate the use of collectbox.

S collectbox ne 5
Collects a full blackbox from the active side of SDU5.

S collectbox ne 8 p
Collects the preserved blackboxes from both sides of SDU8.

S collectbox ne 1 f SNMP
Collects a filtered blackbox based on the keyword SNMP from the
active side of SDU1.

S collectbox ip 10.4.5.1 p standby


Collects the preserved blackboxes from the standby side of the SDU
with a management address of 10.4.5.1.

Description of Operation
This command does not require the CLI to be up when the command is
executed. The command will attempt to find the SDU in the database
and determine the correct IP address to connect to. The command will
normally ping the platform before attempting to collect the specified
blackbox. The command will display the name and location of the file
collected. Any temporary files created on the platform will be cleaned
up. This program telnets to the platform which grabs console output
temporarily from the front panel serial port.
Running bboxlisten

When to Run bboxlisten


When the blackbox server at the OMCR detects that the SPROC has
reset, bboxlisten automatically runs collectbox to collect data from the
SDU.
Running bboxlisten
No user intervention is required for the automatic upload of black boxes.
Description of Operation
The server will wake up when an SDU sends it a message after an
SPROC reset. The server will check every 30 seconds for collection
requests and invoke the collection script. The script will attempt to
upload the preserved black boxes from either SPROC (active and
standby). The script will turn off nagging on the SPROC so no more
requests to upload are generated from this reset. The script will check the
number of stored black boxes on the OMCR and clean up if needed.

APR 2004

SDU Troubleshooting

4-3

Data Collection

continued

Accessing the Collected Black


Boxes

The black box output is cleaned up on the OMCR and stored in the
/sc/spool/bb directory. The collected black boxes are named based on the
sdu#, time of collection, active/standby, and original file name on
platform.
The following is an example:
/sc/spool/bb/s210041939a0.out
/sc/spool/bb/s210041939s0.out
s2 SDU2
1004 October 4
1939 7:39 PM
a0 preserved file /data/blackbox/0 from
active side
s1 preserved file /data/blackbox/1 from
standby side
sh preserved file
/data/blackbox/hardResetBBox from standby side
af0 preserved file /active/init/hungbbox0
from standby side

4-4

SDU Troubleshooting

APR 2004

Chapter 5: SDU Cage Reset Procedure


Table of Contents

SDU Cage Reset Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SDU Cage Power Supply Overview . . . . . . . . . . . . . . . . . . . . . . . . . . .
Cage Reset Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Cage Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5-1
5-1
5-1
5-2
5-2

APR 2004

SDU Troubleshooting

Table of Contents

continued

Notes

SDU Troubleshooting

APR 2004

SDU Cage Reset Procedure

Introduction

If the SDU should become completely unresponsive for some reason, the
method of last resort for recovery would be a power cycle of the SDU
cage (which in turn would power cycle every FRU in the SDU cage).
SDU Cage Power Supply
Overview

This section provides a high level overview of the SDUs power supply
and distribution in order to familiarize the user with the location and
purpose of the various power supplies and circuit breakers on the SDU
frame.
There are two power supplies per SDU cage, for a total of 4 per frame.
Each power supply pair (2 vertically stacked on the left and the right)
feeds a group of 7 circuit breakers which in turn feed 7 power zones
(or groups of slots) in a given cage.
There are 7 pushpull switches that correspond to the 7 circuit breakers
(running vertically alongside the power supplies). However, there is a
metal plate over these to prevent them from being manually opened.
These should not be used to perform a cage reset.
In addition to the 7 circuit breakers per cage, there are also 2 circuit
breakers per cage which correspond to the main power supplies to the
cage. These are located near the top of the front side of the SDU cage on
the panel labeled MAIN BREAKER PANEL. There is one circuit
breaker pushpull switch for each power supply and two power supplies
for each cage.
Figure 5-1: SDU Main Breaker Panel (MBP)

75

APR 2004

75

75

SDU Troubleshooting

75

5-1

SDU Cage Reset Procedure

continued

Figure 5-2: SDU Power Supply Modules and Circuit Breaker Modules

20

20

20

20

20

20

20

20

20

20

Cage Reset Procedure

To cycle power to a given cage, the two circuit breaker switches for the
two power supplies to that cage must be pulled. The left two switches
control power to the lower cage and the right two switches control power
to the upper cage. Again, both switches must be pulled since the power
supply feeds are redundant.

Once both breaker switches are pulled, wait 1015 seconds and then
push the same two switches back in towards the frame. At this point, the
system should initialize completely and return to operation.
Cage Recovery

Once both circuit breaker switches are pushed back in for the Power
Supply, the SDU Frame will begin to initialize the SPROC and other
Payloads. During this process, the OMCR (NE Manager) gets an Alarm
which indicates the Loss of Communication with SDU Frame. Once
the NE (SPROC) completes the initialization and the ISB learns the
Routes from Access Node, then the SPROC will start sending the
SYNC Required to the OMCR. The OMCR also clears the Loss of
Communication Alarm. Then, the OMCR SYNCs with NE. Once the
NE SYNCs up with OMCR, the SDU Frame will be in the Operational
State.

5-2

SDU Troubleshooting

APR 2004

Chapter 6: SDU General Status


Table of Contents

SDU General Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


General SDU Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6-1
6-1

APR 2004

SDU Troubleshooting

Table of Contents

continued

Notes

SDU Troubleshooting

APR 2004

SDU General Status

General SDU Status

The following LEDs states indicate basic SDU status:

S Once the SDU boots up after a reset, the LED lights on the front panel
of an SDU frame should appear solid green.

S If the LED light on a board appears red, the board is having an alarm.
S If the LED light on a board appears blinking green, the board is
resetting.

S If the LED light is in a constant blinking green, the board may not be
provisioned at the moment.
Boot progress
To monitor the boot progress of a board in an SDU frame, a console
needs to be connected to the board. Without a console connection, only
the LEDs can provide a limited status. Once the LED light turns into
solid green, the board is finished rebooting.
State Determination
Once the SDU finishes rebooting, the OMCR can be used to check the
state of the SDU. For OMCR Status/Display Commands, please refer
to the System Command Reference.

APR 2004

SDU Troubleshooting

6-1

SDU General Status

continued

Notes

6-2

SDU Troubleshooting

APR 2004

Chapter 7: SDU Initialization Failure Scenarios


Table of Contents

SDU Initialization Failure Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The SPROC (either active or standby) fails to join in
the cluster and is unable to boot up . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SPROC (either active or standby) initialization does not complete . . .
The SPROC (either active or standby) displays
going unmanaged, and resets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Hung threads occur on the SPROC (active or standby)
and the SPROC resets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The software load on one SPROC (either active or standby)
becomes corrupt, and the SPROC reboots . . . . . . . . . . . . . . . . . . . . . . .
A failure of the active LAN causes the SPROC to reset . . . . . . . . . . . .

7-1
7-1
7-1
7-2
7-3
7-4
7-5
7-6

APR 2004

SDU Troubleshooting

Table of Contents

continued

Notes

SDU Troubleshooting

APR 2004

SDU Initialization Failure Scenarios

Introduction

This section contains troubleshooting procedures for SDU initialization


failure scenarios.
NOTE
Refer to chapter 6 for descriptions of LED states during
various boot progress stages.

WARNING
Never reseat or reset the SPROC for any reason, unless
Motorola CNRC has given the instruction to do so. If this
is the case, first lock the SPROC from the OMCR.

The SPROC (either active or


standby) fails to join in the
cluster and is unable to boot
up

Symptom
If the SPROC (either active or standby) fails to join in the cluster and is
unable to boot up, use the following information to troubleshoot the
problem.
Possible Causes
Possible causes for this symptom include one of the following:

S LAN failure
S While the OMCR is configuring the ISBs, the SPROCs unmanaged
timeout timer expires.
Recovery Action
Reset the SPROC, and verify that the ISBs are in a stable state.

APR 2004

SDU Troubleshooting

7-1

SDU Initialization Failure Scenarios

continued

SPROC (either active or


standby) initialization does not
complete

Symptom
The SPROC initialization does not complete if the SPROC console
displays node not in database and then resets.
Possible Causes
Possible causes for this symptom include one of the following:

S The configuration data file (baseNECDF .xml) for the SPROC either
contains errors or is corrupted.
For example, the configuration data may include an incorrect cage or
slot number.

S The SPROC is not properly seated.


Recovery Action
Perform the following recovery actions.
Table 7-1: SPROC Initialization Does Not Complete
Step
1

Action
If the SPROC is not correctly seated, correctly seat the
SPROC.

NOTE
Refer to the CDMA2000 1X RAN Hardware Installation for
details.
2

If the SPROC is correctly seated, regenerate the SDU Data


Files from the OMCR.
Enter the following commands from the OMCR CLI.

NOTE
Refer to the CDMA2000 1X Cellular System Administration
Guide for additional details.

7-2

2a

generate sdu# datafiles <Enter>

2b

activate sdu# <Enter>

SDU Troubleshooting

APR 2004

SDU Initialization Failure Scenarios

continued

The SPROC (either active or


standby) displays going
unmanaged, and resets

Symptom
If the SPROC console displays going unmanaged and then resets,
this indicates that the SPROC lost management control and is
performing a reset to clear the underlying problem.
Possible Causes
This symptom may be due to one of the following causes:

S With a 2N redundant setup, if both the ISBs reset and initialize again,
one of the SPROCs will display going unmanaged and reset as
a result of a split brain resolution process. There is a possibility that
the node which was a manager (the active one) may also reset.

S Both ISBs are resetting.


Recovery Action
There is no recovery action. The SPROC automatically resets itself to
resolve the problem.

APR 2004

SDU Troubleshooting

7-3

SDU Initialization Failure Scenarios

continued

Hung threads occur on the


SPROC (active or standby) and
the SPROC resets

Symptom
NOTE
A hung thread is like a hanging software task, only it is a
thread instead of a process.

If the hung thread problem occurs on the SDU, the SPROC console
displays something like the following output:
[D2D37D8] PANIC: Starting Emergency Exit
Handlers
[D2D37D8] PANIC: Called Default Emergency Exit
Handler
[D2D37D8] Writing Emergency Exit Trace
Information for 1 threads
[D2D37D8] goahead/hap Error: HUNG THREADS
DETECTED: srk tid = 14000011 (OS id = c43fa70)
[D2D37D8] Default EEH: PANIC for srk tid =
14000011 (OS id = c43fa70).
[D2D37D8] Finished Default Emergency Exit
Handler
[D2D37D8] PANIC: Finished Emergency Exit
Handlers
[D2D37D8] PANIC: Closing SRK
[D2E3D50] Closing Extensions
[D2E3D50] RIM Extension Shutting Down.
NEI : NEIX: Possible GA hung thread. Stack trace
in black box.Rebooting in 30 secs...

The SPROC will reboot.


If the standby SPROC has a higher slot number than the active SPROC,
and the hung thread problem occurs on the active SPROC, this
problem may cause a site reset (if the standby SPROC loses arbitration
for the cluster manager role).
Possible Causes
A GA hung thread is the cause. The GoAhead software shuts itself
down, which causes the SPROC to reboot.

7-4

SDU Troubleshooting

APR 2004

SDU Initialization Failure Scenarios

continued

Recovery Action
A full SDU reset from the OMCR may help to clear out the hung thread
problem, but use caution.
WARNING
A full SDU reset will cause a site outage. If possible,
perform the reset during an appropriate maintenance
window.

The software load on one


SPROC (either active or
standby) becomes corrupt, and
the SPROC reboots

Symptom
If the SPROC software becomes corrupted, the SPROC will reboot.
Possible Causes
See the Symptom description.
Recovery Action
No recovery action is necessary.
The SDU automatically resolves the problem using one of the following
methods:

S For a single SPROC node setup, the SPROC will reboot ten times,
then it will switch the partition and attempt to initialize using the other
partition.

S For a 2N redundant SPROC setup, if the other node becomes the


manager, the node running the corrupted software load will: become
the nonpotential manager; FTP the load from the manager; and
reset itself after switching the partition.

APR 2004

SDU Troubleshooting

7-5

SDU Initialization Failure Scenarios

continued

A failure of the active LAN


causes the SPROC to reset

Symptom
If the active LAN fails, the active SPROC will transition to the OOS
state, and the redundant SPROC will take over.
The SPROC console displays output similar to the following:
NEIX: active interface x with an ip address
x.x.x.x and subnet x failed
Possible Causes
The ISB connected to the active LAN is not functioning properly.
Recovery Action
None. This is a normal behavior.

7-6

SDU Troubleshooting

APR 2004

Chapter 8: ISB/SPROC Troubleshooting


Table of Contents

APR 2004

ISB/SPROC Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Overview of ISB/SPROC Troubleshooting . . . . . . . . . . . . . . . . . . . . . .

8-1
8-1

Verifying ISB/SPROC Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Verifying ISB/SPROC Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Logging in to the ISB/SPROC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8-2
8-2
8-2

Troubleshooting ISB/SPROC Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
ISB Code Load Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The ISB is unable to provide DHCP service to the SPROC . . . . . . . . .
The ISB reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
ISB reset causes payloads to reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The ISB fails to establish the gigabit link . . . . . . . . . . . . . . . . . . . . . . .
During SPROC failover, the ISB automatically reset . . . . . . . . . . . . . .
Pulling out the standby ISB and the standby SPROC
causes the payloads to reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The Active ISB reset causes the SPROC (active and standby) to reset
The Standby ISB reset causes SPROC to reset . . . . . . . . . . . . . . . . . . .

8-3
8-3
8-3
8-4
8-5
8-5
8-6
8-7

Troubleshooting ISB/SPROC Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


ISB/SPROC Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SDU ISB Communications Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SDU ISB Quality of Service Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . .
SDU ISB Processing Error Alarm . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SDU SPROC Device Out of Service Alarms . . . . . . . . . . . . . . . . . . . .
SPROC Application Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SPROC Communication Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8-9
8-9
8-11
8-11
8-12
8-13
8-15
8-20

SDU Troubleshooting

8-7
8-7
8-8

Table of Contents

continued

Notes

SDU Troubleshooting

APR 2004

ISB/SPROC Troubleshooting

Overview of ISB/SPROC
Troubleshooting

This chapter deals specifically with the procedures for troubleshooting


the ISB/SPROC cards in the SDU.

APR 2004

SDU Troubleshooting

8-1

Verifying ISB/SPROC Status

Verifying ISB/SPROC Status

When troubleshooting the ISB/SPROC, it is recommended to verify the


current ISB/SPROC status by checking LEDs and verifying board states
using the OMCR Display/Status commands.
For details, refer to chapter 6, General SDU Status.
Logging in to the ISB/SPROC

Some procedures may require logging in to the ISB/SPROC directly.


Refer to chapter 3 for details on how to log in to SDU boards.
IMPORTANT

Logging into the SPROC, ISB, and BPP devices should be


performed only during a maintenance window to avoid
system disruption.

8-2

SDU Troubleshooting

APR 2004

Troubleshooting ISB/SPROC Failures

Overview

This section contains troubleshooting procedures for various ISB failure


scenarios.
WARNING
Never reseat or reset the SPROC for any reason, unless
Motorola CNRC has given the instruction to do so. If this
is the case, first lock the SPROC from the OMCR.

ISB Code Load Failure

Symptom
The ISB is unable to download the software load from the SPROC.
Possible Causes
The possible cause may be one of the following. The ISB:

S Is new from factory;


S Does not have a programmed route to the SPROC;
S Is not receiving the GETCODELOAD from the proxy component.
Recovery Action
Contact CNRC for assistance.

APR 2004

SDU Troubleshooting

8-3

Troubleshooting ISB/SPROC Failures

continued

The ISB is unable to provide


DHCP service to the SPROC

Symptom
The SPROC is unable to get IP service from the ISB.
Possible Causes
Ensure that the debug ethernet port of the SPROC isnt connected to an
outside network. If, when the SPROC boots up, the console display
indicates it is loading up on the debug interface, the debug port is
connected to the network.
Recovery Action
Perform the recovery actions in Table 8-1.
Table 8-1: SPROC Unable to get IP address from ISB
Step

Action

Ensure that the front panel Ethernet Jack does not have a
cable connected to it. The front panel Ethernet Jack is the
debug port.

Reboot the SPROC and ISB. Refer to chapter 2 for details on


resets.

8-4

SDU Troubleshooting

APR 2004

Troubleshooting ISB/SPROC Failures

continued

The ISB reset

Symptom
The symptoms include at least one of the following:

S The ISB resets


S The SPROC console displays keep alive timed out for the ISB.
Possible Causes
The SPROC reset the ISB, because the SPROC did not receive any
acknowledgement from the ISB.
Recovery Action
None. The automatic reboot should correct the problem.
ISB reset causes payloads to
reset

Symptom
The ISB reset causes payloads (BPPs, CCAs) to reset.
Possible Causes
If the ISB failover didnt occur quickly enough, the payloads will also
reset.
Recovery Action
None. The automatic ISB reset should correct the problem.

APR 2004

SDU Troubleshooting

8-5

Troubleshooting ISB/SPROC Failures

continued

The ISB fails to establish the


gigabit link

Symptom
The ISB fails to establish the gigabit link.
Possible Causes
Possible causes include:

S Cable failure
S Span I/O failure
S The SPROC database was not updated.
Recovery Action
Check/replace cables and/or span I/O (swap out); check/update the
SPROC database for gigabit links.
Table 8-2: ISB Fails to establish Gigabit Link
Step
1

Action
At the OMCR, verify the settings for the
MLS_SDU_CON links, the Gigabit connections from the
AN to the SDU.
If necessary, correct or change those settings.

NOTE
If you change these settings, additional steps from the
OMCR will be necessary. For example, if you modify a
configuration from the OMCR, you need to regenerate the
data files (the Generate Data File command) and reactive the
devices.
For more details, and the complete procedural context, refer
to the System Command Reference and the Cellular System
Administration Guide.

8-6

If it is not a configuration problem, continue at the next step.

Check and if necessary replace the gigabit cable(s).

If it is not a cable problem, swap out the span I/O. Refer to


the SDU FRU manual for the applicable procedure.

SDU Troubleshooting

APR 2004

Troubleshooting ISB/SPROC Failures

continued

During SPROC failover, the ISB


automatically reset

Symptom
During the SPROC failover, the ISB automatically reset.
Possible Cause
The route updates were not fast enough.
Recovery Action
None. The automatic reset should correct the problem.
Pulling out the standby ISB
and the standby SPROC
causes the payloads to reset

Symptom
The Payloads reset after a Dual standby pull.
Possible Causes
The route updates were not fast enough.
Recovery Action
None. The automatic reset should correct the problem.
The Active ISB reset causes
the SPROC (active and
standby) to reset

Symptom
The active ISB reset causes both SPROCs to reset.
Possible Causes

The route updates were not fast enough.


Recovery Action
None. The automatic reset should correct the problem.

APR 2004

SDU Troubleshooting

8-7

Troubleshooting ISB/SPROC Failures

continued

The Standby ISB reset causes


SPROC to reset

Symptom
When the standby ISB reset, the SPROC also reset.
Possible Causes
The route updates were not fast enough.
Recovery Action
None. The automatic reset should correct the problem.

8-8

SDU Troubleshooting

APR 2004

Troubleshooting ISB/SPROC Alarms

ISB/SPROC Alarms

This section contains troubleshooting procedures for


ISB/SPROCrelated alarms.
Though the ISB or SPROC reports these alarms, in some cases another
card may be causing the underlying problem.
WARNING
Never reseat or reset the SPROC for any reason, unless
Motorola CNRC has given the instruction to do so. If this
is the case, first lock the SPROC from the OMCR.

NOTE
In addition to maintaining troubleshooting logs, you may
also want to notify CNRC of problems if they frequently
reappear.

NOTE
For a more detailed description of each alarm, refer to
System Output Messages. Also, if you are not sure when
you need to resolve the problem, refer to the alarm
documentation (in the System Output Messages) for the
severity level.

APR 2004

SDU Troubleshooting

8-9

Troubleshooting ISB/SPROC Alarms

continued

Table 8-3: Common OMCR Commands


Use

OMCR command

Verify that the device is


present and operational

Display/Status commands for SDU cards:


status sdusdu#
status sdubppsdubpp#
status sduccasdu#sducage#sducca#
status sdusdu#sducage#sduisb#
status sdusdu#sducage#sdusproc#
status sdfrgsdu#
status pcfrgsdu#

Verify that the links are


functional or correctly
configured

display mmsdfralinksdu#cbsc#mmsdfralink#
mmsdfralinkconf
display mmpcfralinksdu#cbsc#mmpcfralink# status
display mmpcfralinksdu#cbsc#mmpcfralink#
mmpcfralinkconf
status sdulinksdulink#
display
sdupdsnlinksdu#sdupdsncluster#sdupdsnlink#
pdsnlinkconf

NOTE
For information on editing, deleting and reprovisioning devices, refer to the Cellular
System Administration Guide.

8-10

SDU Troubleshooting

APR 2004

Troubleshooting ISB/SPROC Alarms

continued

SDU ISB Communications


Alarms

2822100 Communication Fault, Physical Link Connection


Error
This alarm is reported when a Communications Protocol Error causes a
fault on the Physical Link Connection.
Perform the following procedure to resolve this alarm.
Table 8-4: SDU Communications Alarm 2822100:
Communication Fault, Physical Link Connection Error Troubleshooting Procedure
Step

Action

Check the physical link connection at the SDU and the farend connection of the physical link at the
AN IP Switch frame (Catalyst MLS chassis). Determine whether it is a physical link connectivity
error or a bad cable.
Refer to the CDMA2000 1X RAN Hardware Installation manual for details on physical cabling and
connectivity between the SDU and AN.

If it is not a physical link problem, replace the ISB. Refer to the CDMA2000 1X AN & SDU FRU for
details.

SDU ISB Quality of Service


Alarms

2823220 Quality of Service Fault, Packet Routing Capacity


Exceeded
This alarm is reported when heavy traffic and/or congestion causes the
Packet Routing Processor on the ISB card to reach its operating limit.
Perform the following procedure to resolve this alarm.
Table 8-5: SDU Quality of Service Alarm 2823220:
Quality of Serv Fault, Packet Routing Capacity Exceeded Troubleshooting Procedure
Step
1

Action

Add additional processing capacity to the site or reduce the traffic loading.

APR 2004

SDU Troubleshooting

8-11

Troubleshooting ISB/SPROC Alarms

continued

2823221 Quality of Service Fault, Packet Routing Capacity


Severely Exceeded
This alarm is reported when the systems Packet Routing Queue cannot
process the volume of traffic directed to it. The alarm indicates that the
Packet Routing Queue Threshold has been exceeded because the Packet
Routing Capacity was severely exceeded.
Perform the procedure in Table 8-5 to resolve this alarm.
SDU ISB Processing Error
Alarm

2823390 Processing Fault, Initialization Failure


This alarm is reported when a configuration or customization error
causes an Initialization failure of a hardware device.
Perform the following procedure to resolve this alarm.
Table 8-6: SDU Processing Error Alarm 2823390:
Processing Fault, Initialization Failure Troubleshooting Procedure
Step
1

Action
The system automatically attempts to reset the card. The card will initialize if the automatic reset is
successful.
If the card initializes, no further action is necessary.
If the card fails to initialize, continue at the next step.

NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state.
2

If the card fails to initialize, attempt a hardware reset.

If the reset fails, lock and replace the card using the procedures contained in the CDMA2000 1X AN &
SDU FRU manual.

8-12

SDU Troubleshooting

APR 2004

Troubleshooting ISB/SPROC Alarms

continued

2823420 Processing Fault, OSPF Inactivity Timer Expired


This alarm indicates an OSPF Inactivity Timer Expired fault due to a
Software Program Error. This alarm represents a series and the last two
digits of the Alarm ID indicates the connection number. Typical range:
2342023422.
Perform the following procedure to resolve this alarm.
Table 8-7: SDU Processing Error Alarm 2823420:
Processing Fault, Initialization Failure Troubleshooting Procedure
Step

Action

If there are (in addition to this alarm) alarms on the farend ISB, or if other BPPs and CCAs are also
reporting similar alarms, the problem is most likely with the ISB. In that case, the ISB problems
should be addressed first. Otherwise the problem is probably local to the BPP, and the BPP should be
reset at a convenient time.

If the problem continues, replace the BPP. Refer to the CDMA2000 1X AN & SDU FRU.

SDU SPROC Device Out of


Service Alarms

IMPORTANT

If a card in the SDU is out of service, such as a BPP or


CCA, the SPROC reports the alarm.

2825000 Device Out of Service Contact Lost


This alarm is raised when the network element is unable to communicate
with a device. This may result from a failed device or the absence of a
device. A failure to communicate with a device constitutes a failure of a
device.
The CCA provides SDU monitoring; the BPP provides either selection
or packet data functions; the ISB provides transport, while the SPROC
provides the main SDU applications.

Perform the following procedure to resolve this alarm.


Table 8-8: SDU Device Out of Service Alarm 2825000:
Device Out of Service Contact Lost Troubleshooting Procedure
Step

Action

Verify that the device specified in the alarm output is present and operational, using the applicable
OMCR CLI commands.

If the device is not present or operational, replace it with an operational device. Refer to the
CDMA2000 1X AN & SDU FRU manual.

APR 2004

SDU Troubleshooting

8-13

Troubleshooting ISB/SPROC Alarms

continued

2825030 Device Out of Service Software Failure


This alarm is reported when a software application fails to respond to the
system heartbeat that polls the applications for operational status.
No immediate action is required. Monitor the device and ensure that it is
successfully restored to service.
If devices are reset often due to application failures, requesting a new
software load may be beneficial.
2825040 Device Out of Service Device Locked
This alarm is reported when the operator locks a device.
Perform the following procedure to resolve this alarm.
Table 8-9: SDU Device Out of Service Alarm 2825040:
Device Out of Service Device Locked Troubleshooting Procedure
Step
1

Action
Once the device is unlocked using the unlock command, the alarm will clear.
Refer to the Cellular System Administration Guide for details on the unlock command.

2825050 Device Out of Service Diagnostics


This alarm is reported when the operator runs an OutofService
Diagnostic Test on a device by using the test command.
No operator action is needed. When the Diagnostic Test is complete, the
alarm will clear automatically.

8-14

SDU Troubleshooting

APR 2004

Troubleshooting ISB/SPROC Alarms

continued

SPROC Application Alarms

2826555 Application Heartbeat Failure


A lowlevel software element on the SPROC which controls the link has
failed.
Perform the following procedure to resolve this alarm.
Table 8-10: SDU Alarm 2826555:
Application Heartbeat Failure Troubleshooting Procedure
Step
1

Action
The system disables the affected MMSDFRALINK or MMPCFRALINK. To clear the alarm, it is
necessary to delete and reprovision the link from the OMCR.

NOTE
For details, refer to the Cellular System Administration Guide for topics that deal with the SDULINK
software object and its subelements (the MMSDFRALINK and MMPCFRALINK).

2826560 Application Messaging Failure


An error has occurred in the software on the SPROC. Application
messaging among tasks on the SPROC has been corrupted to some
degree. The error may be in an application attempting to establish
onboard communications. Other causes may be bind or send failures in
the onboard message system.
Perform the following procedure to resolve this alarm.
Table 8-11: SDU Alarm 2826560:
Application Messaging Failure Troubleshooting Procedure
Step

Action

Wait for the alarm to clear.


The card resets in an attempt to recover. The standby SPROC becomes the active SPROC during the
reset. The alarm clears when the SPROC is reset.

The failed SPROC becomes the standby SPROC. If the problem is not resolved in the reset, the
standby SPROC continues to fail. In this case, replace the card.
Refer to the CDMA2000 1X AN & SDU FRU manual for details.

APR 2004

SDU Troubleshooting

8-15

Troubleshooting ISB/SPROC Alarms

continued

2826570 Application Messaging Failure


An error has occurred in the software on the SPROC. Application
messaging among tasks on the SPROC has been corrupted to some
degree. The error may be in an application attempting to establish
onboard communications. Other causes may be bind or send failures in
the onboard message system.
Perform the following procedure to resolve this alarm.
Table 8-12: SDU BPP Alarm 2826570:
Communication Failure Application Messaging Failure Troubleshooting Procedure
Step
1

Action
The system disables the affected SDUSPROC. To clear the alarm, it is necessary to delete the
SDUSPROC from the OMCR and reprovision in an attempt to bring it back into service.

NOTE
For details, refer to the Cellular System Administration Guide and System Command Reference.

2826580 Application Messaging Failure


An error has occurred in the software on the SPROC. Application
messaging among tasks on the SPROC has been corrupted to some
degree. The error may be in an application attempting to establish
onboard communications. Other causes may be bind or send failures in
the onboard message system.
Perform the following procedure to resolve this alarm.
Table 8-13: SDU Alarm 2826580:
Application Messaging Failure Troubleshooting Procedure
Step

Action

The system disables the affected device shown in the alarm. To clear the alarm, it is necessary to delete
the affected device from the OMCR and reprovision in an attempt to bring it back into service.
For details, refer to the Cellular System Administration Guide and System Command Reference.

NOTE

For details, refer to the Cellular System Administration Guide and System Command Reference.

8-16

SDU Troubleshooting

APR 2004

Troubleshooting ISB/SPROC Alarms

continued

2826720 Application Initialization Failure


An internal software error has occurred on the SPROC. A software
application controlling the link implicated in this alarm failed to
complete its initialization.
Perform the following procedure to resolve this alarm.
Table 8-14: SDU Alarm 2826720:
Application Initialization Failure Troubleshooting Procedure
Step

Action

The system disables the affected MMSDFRALINK, or MMPCFRALINK, or SDUPDSNLINK. To


clear the alarm, it is necessary to delete the affected link type from the OMCR and reprovision in an
attempt to bring it back into service.

NOTE
For details, refer to the Cellular System Administration Guide and System Command Reference.

2826730 Application Initialization Failure


An internal software failure has occurred on the SPROC in the onboard
communications protocol software. This board will be unable to
complete initialization.
Perform the following procedure to resolve this alarm.
Table 8-15: SDU Alarm 2826730:
Application Initialization Failure Troubleshooting Procedure
Step
1

Action
Wait for the alarm to clear. The card resets in an attempt to recover. The alarm clears when the failed
SPROC recovers.

NOTE
If the standby SPROC fails to initialize, the active SPROC is not affected.
If the active SPROC has this alarm, a failover occurs.
2

The failed SPROC becomes the standby SPROC. If the problem is not resolved in the reset, the
standby SPROC continues to fail. In this case, replace the card.
Refer to the CDMA2000 1X AN & SDU FRU manual for details.

APR 2004

SDU Troubleshooting

8-17

Troubleshooting ISB/SPROC Alarms

continued

2826740 Internal SCTP Stack Failure


An internal software failure has occurred on the SPROC in the
application controlling the implicated link. The software was unable to
bind to the onboard communications protocol software.
Perform the following procedure to resolve this alarm.
Table 8-16: SDU Alarm 2826740:
Internal SCTP Stack Failure Troubleshooting Procedure
Step
1

Action
The system disables the affected MMSDFRALINK, or MMPCFRALINK. To clear the alarm, it is
necessary to delete the affected link type from the OMCR and reprovision in an attempt to bring it
back into service.
For details, refer to the Cellular System Administration Guide and System Command Reference.

NOTE
For details, refer to the Cellular System Administration Guide and System Command Reference.

2826750 IP Failure
An internal software failure occurred in the protocol stack on the
SPROC in the onboard communications protocol software.
Perform the procedure in Table 8-15 to resolve this alarm.
2826760 Multicast Failure
An application on the SPROC was unable to join a multicast group.
Perform the procedure in Table 8-15 to resolve this alarm.
2826770 Capacity Greatly Reduced
Call capacity has been greatly reduced
Perform the following procedure to resolve this alarm.

Table 8-17: SDU Alarm 2826770:


Capacity Greatly Reduced Troubleshooting Procedure
Step
1

8-18

Action
Check the status and alarms for all SDF boards, and correct problems as indicated for those alarms.
Resolving the SDF problems should restore the capacity.

SDU Troubleshooting

APR 2004

Troubleshooting ISB/SPROC Alarms

continued

2826780 Capacity Reduced


Call capacity was reduced.
Perform the procedure in Table 8-17 to resolve this alarm.
2826830 Application Platform Failure
The software associated with the device encountered an error in
initializing.
Perform the procedure in Table 8-14 to resolve this alarm.
2826840 Application Platform Failure
The software associated with the SPROC encountered a failure in
initializing.
Perform the procedure in Table 8-15 to resolve this alarm.
2826850 Application Platform Failure
A memory allocation failure occurred.
Perform the following procedure to resolve this alarm.
Table 8-18: SDU Alarm 2826850:
Application Platform Failure Troubleshooting Procedure
Step
1

Action
To clear the alarm, it is necessary to lock, delete, and reprovision the BPP board to attempt bringing
it back into service. Refer to the Cellular System Administration Guide and System Command
Reference for details.

2826870 Incomplete Dataload


Some or all of the objects that the SDU requires for initialization are not
present.
Perform the following procedure to resolve this alarm.
Table 8-19: SDU Alarm 2826870:
Incomplete Dataload Troubleshooting Procedure
Step
1

Action
The SDU is disabled.
To clear the alarm, regenerate the objects for download. This can be accomplished with the Generate
SDU Data File CLI command at the OMCR. Refer to Table 10-1, step 2.

APR 2004

SDU Troubleshooting

8-19

Troubleshooting ISB/SPROC Alarms

continued

2826880 All Active Calls Released


A transient error caused all active calls to be released.
No action is necessary. Calls that originate after the alarm are
unaffected.
SPROC Communication
Alarms

2826500 Communication Failure, Lost Connection with MM


The connection at the protocollevel between the MM and SDU could
not be established or was dropped, although Access Node connectivity
is intact.
Perform the following procedure to resolve this alarm.
Table 8-20: SDU Alarm 2826500:
Communication Failure Lost Connection with the MM Troubleshooting Procedure
Step
1

Action
No action necessary. The system will take the link out of service. The MM will try to reestablish the
link. When the link is reestablished, the alarm will clear.

NOTE
The following link types exist between the SDU and the MM: MMSDFRALINK (for SDU SDF
resource allocation) and MMPCFRALINK (for SDU PCF resource allocation). For details on
configuring and displaying the status of these links, refer to the Cellular System Administration Guide
and/or the System Command Reference.
2

If other links are failing locally, a problem may exist with the network connection or the ISB. In this
case, check the ISB for alarms and check the farend MM for failures.

2826510 Communication Failure, Lost Communication with


the MM
Communication at the applicationlevel between the MM and SDU
could not be established or was dropped, although Access Node
connectivity is intact.

Perform the Table 8-20 procedure to resolve this alarm.

8-20

SDU Troubleshooting

APR 2004

Troubleshooting ISB/SPROC Alarms

continued

2826520 Communication Failure, Link Disabled


This alarm indicates that the MM has reported that the link is disabled.
Perform the following procedure to resolve this alarm.
Table 8-21: SDU Alarm 2826520:
Communication Failure Link Disabled Troubleshooting Procedure
Step
1

Action
No action is necessary. The system will take the link out of service. The MM will try to reestablish
the link. When the link is reestablished, the alarm will clear.

NOTE
The following link types exist between the SDU and the MM: MMSDFRALINK (for SDU SDF
resource allocation) and MMPCFRALINK (for SDU PCF resource allocation). For details on
configuring and displaying the status of these links, refer to the Cellular System Administration Guide
and/or the System Command Reference.
2

Check the farend MM for lock or failure status. Refer to the Cellular System Administration Guide
and System Command Reference for details on displaying the status of the MM.

2826530 Communication Failure, Lost Communication with


the PDSN
Communication between the SDU and PDSN could not be established or
has been lost. This alarm implicates the link to the PDSN.
Perform the following procedure to resolve this alarm.
Table 8-22: SDU Alarm 2826530:
Communication Failure Lost Communication with the PDSN Troubleshooting Procedure
Step
1

Action
Wait for the alarm to clear as the system automatically attempts to reestablish the SDUPDSNLINK.
The alarm will clear when the SDU reestablishes the SDUPDSNLINK, the call processing link
between the SDU and PDSN.
From the OMCR, you can verify the status of the SDUPDSNLINK.

NOTE
The SDUPDSNLINK is the call processing link between the SDU and the PDSN. Until this link is
reestablished, the SDU cannot assign new calls to the PDSN. For more details on SDUPDSNLINK,
refer to the Cellular System Administration Guide and System Command Reference.
2

Check for other external communication failures.

Verify that the PDSN is functioning normally from the OMCR.


For more details on PDSN status (e.g. connectivity to the AN, the status of SDUPDSNLINK, etc.)
refer to the Cellular System Administration Guide and System Command Reference. For other aspects
of PDSN functionality, refer to the PDSN vendor documentation.

APR 2004

SDU Troubleshooting

8-21

Troubleshooting ISB/SPROC Alarms

continued

Notes

8-22

SDU Troubleshooting

APR 2004

Chapter 9: SDU Synchronization Failures


Table of Contents

SDU Synchronization Failure Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SDUOMCR Audit Synchronization failure . . . . . . . . . . . . . . . . . . . .
SDU fails to send sanity (heartbeat) message to OMCR . . . . . . . . . . .
SDU fails OMCR reset command . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9-1
9-1
9-1
9-1
9-2

APR 2004

SDU Troubleshooting

Table of Contents

continued

Notes

SDU Troubleshooting

APR 2004

SDU Synchronization Failure Scenarios

Introduction

This chapter contains SDU synchronization failure scenarios and the


applicable recovery actions.
SDUOMCR Audit
Synchronization failure

Symptom
Any command from the OMCR to the SDU fails.
Possible Cause
The SDU is not in a state where it can synchronize with OMCR (or the
SDU is not connected to the OMCR).
Recovery Action
To verify that the SDU and OMCR are not in sync, type
snmpConnStateShow at the SPROC command line.
IMPORTANT

Logging into the SPROC should be performed during a


scheduled maintenance window to avoid system disruption.

At the OMCR, initiate the audit command to the SPROC. Refer to the
System Command Reference for details.
SDU fails to send sanity
(heartbeat) message to OMCR

Symptom
The SDU fails to send the sanity (heartbeat) message to the OMCR
every two minutes.
NOTE
The OMCR monitors heartbeat messages to verify that it
is still in contact with the device.

Possible Cause

Connectivity does not exist between the SDU and the OMCR.
Recovery Action
If the OMCR is not receiving heartbeat messages from the SDU,
perform the following procedure.
APR 2004

SDU Troubleshooting

9-1

SDU Synchronization Failure Scenarios

continued

Table 9-1: Troubleshoot SDU Heartbeat Failure


Step
1

Action
Log in to the SPROC and enter:
snmpConnStateShow
The command output will indicate whether the SPROC is
communicating with the OMCR.

* IMPORTANT
Logging into the SPROC should be performed during a
scheduled maintenance window to avoid system disruption.
2

If step 1 indicates that the SPROC is not communicating with


the OMCR, enter the ping command and note the results:

S Ping the SPROC from the OMCR;


S Ping the OMCR from the SPROC.
3

If the results of steps 1 and 2 indicate that the SPROC and


OMCR are not communicating (and there is no existing
network link failure, for example, an AN link alarm) perform
a manual synchronization.
Log in to the OMCR CLI and enter the following command:
sync sdu# all

If the OMCR does not recover the SDU heartbeat, contact


CNRC.

SDU fails OMCR reset


command

Symptom
The OMCR cannot reset the SDU or the particular SDU card.
Possible Cause
The OMCR command does not reach the SDU.
This could be due to a transient error or a connectivity problem.
Recovery Action
Retry the OMCR command.
If it still doesnt work, determine whether there are any connectivity
problems (are there alarms for contact being lost with SDU).
Troubleshoot those issues based on the applicable alarm procedure.

9-2

SDU Troubleshooting

APR 2004

Chapter 10: BPP Troubleshooting


Table of Contents

BPP Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Overview of BPP Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10-1
10-1

Verifying BPP Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Verifying BPP Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10-2
10-2

Troubleshooting BPP Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Procedure for Regenerating Corrupted Software . . . . . . . . . . . . . . . . .
Troubleshooting BPP Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10-3
10-3
10-3
10-4

10
APR 2004

SDU Troubleshooting

Table of Contents

continued

Notes

10
SDU Troubleshooting

APR 2004

BPP Troubleshooting

Overview of BPP
Troubleshooting

This chapter deals specifically with the procedures for troubleshooting


the BPP cards in the SDU.

10
APR 2004

SDU Troubleshooting

10-1

Verifying BPP Status

Verifying BPP Status

Whenever troubleshooting the BPP cards, it is recommended to verify


the current BPP status by checking LEDs and verifying board states
using the OMCR Display/Status commands.
For details, refer to chapter 6, General SDU Status.

10
10-2

SDU Troubleshooting

APR 2004

Troubleshooting BPP Alarms

Overview

This section contains troubleshooting procedures for BPPrelated


alarms.
These alarms deal with the BPP hardware when implemented with
Packet Control Function (PCF) software and Selection Distribution
Function (SDF) software.
The following procedures address the alarm number, alarm name, and
procedure.
NOTE
In addition to maintaining troubleshooting logs, you may
also want to notify CNRC of problems if they frequently
reappear.

NOTE
For a more detailed description of each alarm, refer to
System Output Messages. Also, if you are not sure when
you need to resolve the problem, refer to the alarm
documentation (in the System Output Messages) for the
severity level.

Procedure for Regenerating


Corrupted Software

If an alarm troubleshooting procedure requires replacing invalid or


corrupt software, use the following procedure.
Table 10-1: Regenerate Corrupted Software
Step

Action

Reinstall the software load as required by the alarm (alarm


dealing with invalid or corrupt software load or files).

After reinstalling software, it is necessary to regenerate the


SDU datafiles:

S From the OMCR, type generate sdu#


datafiles <Enter>

S From the OMCR, type activate sdu# <Enter>


NOTE
Refer to the Cellular System Administration Guide for
additional assistance.
APR 2004

SDU Troubleshooting

10
10-3

Troubleshooting BPP Alarms

continued

Troubleshooting BPP Alarms

2826000 Application Failure


This alarm is reported when communication is lost with a critical, but
noncall processing task on a processor on the alarming BPP Card.
Perform the following procedure to resolve this alarm.
Table 10-2: SDU BPP Alarm 2826000:
Application Failure Troubleshooting Procedure
Step
1

Action
The system automatically attempts to reset the card. The card will initialize if the automatic reset is
successful.
If the card initializes, no further action is necessary.
If the card fails to initialize, continue at the next step.

NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.
2

Reset the hardware using the reset switch on the front panel of the specific BPP card.

If the hardware reset fails, lock and replace the card using the procedures contained in the CDMA2000
1X AN & SDU FRU manual.

2826001 Application Error


This alarm indicates a nonfatal runtime software error.
Perform the following procedure to resolve this alarm.
Table 10-3: SDU BPP Alarm 2826001:
Application Error Troubleshooting Procedure
Step

Action

Verify whether the software load is valid and complete. To do so, check that the current software load
is actually the one intended to be running, and that all of the pertinent software files are present. Refer
to the SDU Software Installation documentation (in the Release Binder) for details.

If the software load is corrupt, replace it with a valid load and codeload the card with the new
software. Refer to the SDU Software Installation documentation (in the Release Binder) for details.

If the software load is valid, reset the BPP card.

10
10-4

SDU Troubleshooting

APR 2004

Troubleshooting BPP Alarms

continued

2826010 Call Processing Failure


This alarm is reported when communication is lost with a Call
Processing software program on a processor on a BPP Card.
Perform the following procedure to resolve this alarm.
Table 10-4: SDU BPP Alarm 2826010:
Call processing Failure Troubleshooting Procedure
Step
1

Action
The system automatically attempts to reset the card. The card will initialize if the automatic reset is
successful.
If the card initializes, no further action is necessary.
If the card fails to initialize, continue at the next step.

NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.
2

Reset the hardware using the reset switch on the front panel of the specific BPP card.

If the hardware reset fails, lock and replace the card using the procedures contained in the CDMA2000
1X AN & SDU FRU manual.

2826011 Call Processing Error


This alarm indicates an error in a call processing task on a processor. It
is most likely due to a software error.
Perform the following procedure to resolve this alarm.
Table 10-5: SDU BPP Alarm 2826011:
Call Processing Error Troubleshooting Procedure
Step

Action

NOTE
As this alarm has minimal impact, the reset can be delayed almost indefinitely.
1

If the alarm does not clear, reset the card.

10
APR 2004

SDU Troubleshooting

10-5

Troubleshooting BPP Alarms

continued

2826020 General Software Failure


This alarm is reported when a software program on the CPU of a BPP
Card fails.
Perform the following procedure to resolve this alarm.
Table 10-6: SDU BPP Alarm 2826020:
Failure Troubleshooting Procedure
Step
1

Action
The system automatically attempts to reset the card. The card will initialize if the automatic reset is
successful.
If the card initializes, no further action is necessary.
If the card fails to initialize, continue at the next step.

NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.
2

If the reset fails, reset the card using the reset switch on the front panel of the specific BPP card.

If the hardware reset fails, lock and replace the card using the procedures contained in the CDMA2000
1X AN & SDU FRU manual.

2826030 CPU Failure


This alarm indicates a processor failure. It is mostly likely due to a
hardware error.
Perform the following procedure to resolve this alarm.
Table 10-7: SDU BPP Alarm 2826030:
CPU Failure Troubleshooting Procedure
Step

Action

The system automatically resets the card. No action is necessary, unless the card fails to initialize. If
the card fails to initialize, attempt a hardware reset using the reset switch on the frontpanel of the
BPP card.

If the hardware reset fails, lock and replace the card. Refer to the CDMA2000 1X AN & SDU FRU for
details.

10
10-6

SDU Troubleshooting

APR 2004

Troubleshooting BPP Alarms

continued

2826040 Hardware Component Error


This alarm is reported when an onboard (nonDSP) component
experiences a Runtime error.
Perform the following procedure to resolve this alarm.
Table 10-8: SDU BPP Alarm 2826040:
Hardware Component Error Troubleshooting Procedure
Step

Action

If the additional text in the alarm output indicates that the board should be replaced and returned,
replace the card and return it to Motorola. To replace the card, refer to the CDMA2000 1X AN & SDU
FRU manual.

If the additional text does not indicate that the board should be replaced and returned, reset the card.

If the card fails to reinitialize properly after a reset, or if the alarm continues to occur, the card should
be replaced.

2826050 Hardware Component Failure


This alarm is reported when an onboard (nonDSP) component
experiences a runtime failure.
Perform the following procedure to resolve this alarm.
Table 10-9: SDU BPP Alarm 2826050:
Hardware Component Failure Troubleshooting Procedure
Step
1

Action
The system automatically attempts to reset the card. The card will initialize if the automatic reset is
successful.
If the card initializes, no further action is necessary.
If the card fails to initialize, continue at the next step.

NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.
2

If the card fails to initialize, reset the hardware using the reset switch on the front panel of the specific
BPP card.

If the reset fails, or if the problem recurs several times, lock and replace the card using the procedures
contained in the CDMA2000 1X AN & SDU FRU manual.

10
APR 2004

SDU Troubleshooting

10-7

Troubleshooting BPP Alarms

continued

2826060 Partial Call Processing Capacity Loss


This alarm is reported when one or more DSPs on the alarming BPP
Card are unavailable.
This alarm indicates that some of the call processing capacity of the
card has been lost. The unavailability may be due to a failure of a
callprocessing component or of another component.
Perform the following procedure to resolve this alarm.
Table 10-10: SDU BPP Alarm 2826060:
Partial Call Processing Capacity Loss Troubleshooting Procedure
Step
1

Action
Reset the card at the earliest opportunity.

NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.
2

If the reset fails, lock and replace the card using the procedures contained in the CDMA2000 1X AN &
SDU FRU manual.

2826070 Full Call Processing Capacity Loss


This alarm is reported when all DSPs are unavailable.
This alarm indicates that all of the call processing capacity of the card
has been lost.
Perform the following procedure to resolve this alarm.
Table 10-11: SDU BPP Alarm 2826070:
Full Call Processing Capacity Loss Troubleshooting Procedure
Step
1

Action
The system automatically resets the card. The card should be replaced if the automatic reset does not
result in a complete recovery.
To replace the card, refer to the CDMA 2000 1X AN & SDU FRU manual.

NOTE
If the card initializes properly, there should be a state change event indicating that the board has
returned to the inservice state. Also, there should be an attribute value change indicating that the
SDU capacity (either SDF or PCF) has increased by one BPPs worth of capacity.

10
10-8

SDU Troubleshooting

APR 2004

Troubleshooting BPP Alarms

continued

2826080 Switch Processing Error


This alarm is reported when the onboard switch experiences a nonfatal
error.
Perform the following procedure to resolve this alarm.
No immediate action is necessary. If the problem persists, follow the
procedure below. Unless other symptoms (for instance, high percentage
of dropped calls on the card) indicate a serious problem, the action may
be performed whenever it is convenient.
Table 10-12: SDU BPP Alarm 2826080:
Switch Processing Error Troubleshooting Procedure
Step

Action

If the problem persists, reset the card.

If a reset does not resolve the problem, replace the card.

2826090 ISB Link Loss


This alarm indicates a loss of the link with an ISB. The ISB is named in
additional text in the alarm output.
Perform the following procedure to resolve this alarm.
Table 10-13: SDU BPP Alarm 2826090:
ISB Link Loss Troubleshooting Procedure
Step

Action

If there are any ISBtype alarms occurring concurrently with this particular alarm, address the ISB
alarms first. Resolving the ISB problems should also resolve the BPP problems triggering this alarm.

If there are no ISB alarms, but other cards are also reporting link alarms, the ISB is probably the
source of the problem. Reset the ISB at a convenient time. When the ISB recovers, the BPP alarm
should clear.

If the ISB and other cards appear to be fine, or if resolving the ISB problems does not clear the alarm,
reset the BPP at a convenient time. If the problem remains, replace the BPP.

10
APR 2004

SDU Troubleshooting

10-9

Troubleshooting BPP Alarms

continued

2826100 File Access Error


This alarm is reported when an attempt to access a file located on the
alarming BPP fails.
Perform the following procedure to resolve this alarm.
Table 10-14: SDU BPP Alarm 2826100:
File Access Error Troubleshooting Procedure
Step

Action

Reset the card.

If the problem persists, replace the card.

2826110 High Temperature


This alarm indicates that the temperature on the card has exceeded the
lower of two hightemperature thresholds.
Perform the following procedure to resolve this alarm.
Table 10-15: SDU BPP Alarm 2826110:
High Temperature Troubleshooting Procedure
Step
1

Action
Determine whether the problem is a general SDU system problem or whether it is specific to just one
card.
The trouble is local to the one card if all of the following cases are true:

S
S
S
S

the fans are working properly


the air flow through the SDU cage is satisfactory
the general environmental conditions around the frame are acceptable
no other cards in the SDU frame are reporting temperaturerelated alarms

If the problem is specific to the one card, reset the card. If the problem persists, replace the card.

If the problem is not specific to the one card, it is necessary to assess and correct the general
environmental conditions (e.g. fan, airflow) surrounding the SDU frame. When the environmental
problem has been corrected, the alarm will clear.
The environmental conditions must meet the specifications outlined for the SDU frame in the
CDMA2000 1X RAN Hardware Installation manual.

If a card reports high temperature for an extended time and later exhibits unusual behavior, the card
may have been damaged and should be replaced.

10
10-10

SDU Troubleshooting

APR 2004

Troubleshooting BPP Alarms

continued

2826111 Very High Temperature


This alarm indicates that the temperature on the card has exceeded the
higher of two hightemperature thresholds.
Perform the following procedure to resolve this alarm.
Table 10-16: SDU BPP Alarm 2826111:
Very High Temperature Troubleshooting Procedure
Step
1

Action
Follow the procedure in Table 10-15.

2826120 Change Request Error


This alarm is reported when the alarming BPP could not process a
Change Request. This alarm is extremely unlikely and is included here
for reference purposes only.
These alarms clear after the operator resets the card. Note that for all
scenarios, reset refers to a hardware reset (accomplished using the
frontpanel reset button on the alarming card).
2826130 Change Request Failure
This alarm is reported when the alarming BPP could not process a
Change Request. This alarm is extremely unlikely and is included here
for reference purposes only.
These alarms clear after the operator resets the card. Note that for all
scenarios, reset refers to a hardware reset (accomplished using the
frontpanel reset button on the alarming card).
2826140 CPU Software Initialization Error
This alarm is reported when a software program on the Bearer Payload
Processor (BPP) Card failed to properly initialize.
Perform the following procedure to resolve this alarm.
Table 10-17: SDU BPP Alarm 2826140:
CPU Software Initialization Error Troubleshooting Procedure
Step

Action

If the alarm indicates Waiting for Personality, assign either a PCF or an SDF personality to the
BPP card. Refer to the Cellular System Administration Guide for details on assigning PCF or SDF
functionality to a BPP card.
Otherwise, skip to the next step.

Refer to Table 10-3.

10
APR 2004

SDU Troubleshooting

10-11

Troubleshooting BPP Alarms

continued

2826150 CPU Software Initialization Failure


This alarm is reported when a software program on the BPP card failed
to properly initialize.
Perform the following procedure to resolve this alarm.
Table 10-18: SDU BPP Alarm 2826150:
CPU Software Initialization Failure Troubleshooting Procedure
Step

Action

The system automatically resets the card. The alarm will clear after the card successfully initializes.

Verify that the software load is valid and complete. To do so, check that the current software load is
actually the one intended to be running, and that all of the pertinent software files are present. If the
software load is corrupt, replace it with a valid load and codeload the card with the new software.

NOTE
Refer to the SDU Software Installation documentation (in the Release Binder) for details.

2827801 through 27802, Packet Bus Connection


This alarm is reported when a LAN Error or Loss of Frame condition
causes a Packet Bus Connection Fault on the identified Packet Bus
Connection (1 or 2). The last two digits of the alarm number identify the
connection number. The devices that could cause this alarm are: Bearer
Payload Processor (BPP) Card and Cabinet and Customer Alarm (CCA)
Card.
Perform the following procedure to resolve this alarm.
Table 10-19: SDU Communications Alarms 2821801 and 2821802:
Communication Fault, Packet Bus Connection no. <12> Troubleshooting Procedure
Step

Action

Physically check the SDU card (BPP or CCA) specified in the alarm output. Determine whether it is
properly connected into the SDU backplane.
Refer to the CDMA2000 1X RAN Hardware Installation manual for details on card placement and
connections.

If it is not a card connectivity problem, then it is necessary to replace the FRU identified in the alarm.
Refer to the CDMA2000 1X AN & SDU FRU manual for details.

10
10-12

SDU Troubleshooting

APR 2004

11

Chapter 11: CCA Troubleshooting


Table of Contents

APR 2004

CCA Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Overview of CCA Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-1
11-1

Verifying CCA Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Verifying CCA Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11-2
11-2

Troubleshooting CCA Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Troubleshooting CCA Processing Faults . . . . . . . . . . . . . . . . . . . . . . . .
Troubleshooting CCA Equipment Faults . . . . . . . . . . . . . . . . . . . . . . .
Troubleshooting CCA Environmental Faults . . . . . . . . . . . . . . . . . . . .
Troubleshooting CCA Communication Faults . . . . . . . . . . . . . . . . . . .

11-3
11-3
11-3
11-4
11-7
11-11

SDU Troubleshooting

11

Table of Contents

continued

Notes

SDU Troubleshooting

APR 2004

CCA Troubleshooting

11

Overview of CCA
Troubleshooting

This chapter deals specifically with the procedures for troubleshooting


the CCA cards in the SDU.

APR 2004

SDU Troubleshooting

11-1

11

Verifying CCA Status

Verifying CCA Status

Whenever troubleshooting the CCA cards, it is recommended to verify


the current CCA status by checking LEDs and verifying board states
using the OMCR Display/Status commands.
For details, refer to chapter 6, General SDU Status.

11-2

SDU Troubleshooting

APR 2004

Troubleshooting CCA Alarms

11

Overview

This section contains troubleshooting procedures for CCArelated


alarms.
The following procedures address the alarm number, alarm name, and
procedure.
NOTE
In addition to maintaining troubleshooting logs, you may
also want to notify CNRC of problems if they frequently
reappear.

NOTE
For a more detailed description of each alarm, refer to
System Output Messages. Also, if you are not sure when
you need to resolve the problem, refer to the alarm
documentation (in the System Output Messages) for the
severity level.

Troubleshooting CCA
Processing Faults

2823380 Illegal Configuration Change


This alarm is reported when corrupt data causes an illegal configuration
change. Device List: CCA, Fan Tray, PSM DC
Perform the following procedure to resolve this alarm.
Table 11-1: SDU Processing Error Alarm 2823380:
Processing Fault, Illegal Configuration Change Troubleshooting Procedure
Step
1

Action
If this alarm occurs, contact CNRC.

APR 2004

SDU Troubleshooting

11-3

11

Troubleshooting CCA Alarms

continued

2823410 Initialization Failure


This alarm indicates an Invalid Configuration Data Loaded fault due to a
Configuration or Customization error. Device List: cabinetSensor,
customerSensor, controlRelay
Perform the following procedure to resolve this alarm.
Table 11-2: SDU Processing Error Alarm 2823410:
Processing Fault, Initialization Failure Troubleshooting Procedure
Step
1

Action
If this alarm occurs, contact CNRC.

Troubleshooting CCA
Equipment Faults

2823670 Device Removal Detected


This alarm is reported when a monitored Fan or Power Supply Module is
removed from the system.
Perform the following procedure to resolve this alarm.
Table 11-3: SDU Equipment Alarm 2823670:
Equipment Fault, Device Removal Detected Troubleshooting Procedure
Step

Action

Verify that maintenance was being performed on the alarmed device. If not, the device requires
replacement. Refer to the CDMA2000 1X AN & SDU FRU manual.

If maintenance was being performed, either a fan tray or power supply module was removed from the
system. The specified device should be placed back into the system.

NOTE
When a device is removed from the system wait 10 seconds before replacement to allow for software
adjustments needed for EID reread (and other internal adjustments) upon reinsertion.

11-4

SDU Troubleshooting

APR 2004

Troubleshooting CCA Alarms

continued

11

2823690 Hardware Fault


This alarm is reported when a hardware fault occurs due to one or more
of the following reasons:

S Equipment malfunction
S Multiplexer problem
S I/O Device failure
Device List: CCA, ISB, SDUFANTRAY, SDUPSMDCDC.
Perform the following procedure to resolve this alarm.
Table 11-4: SDU Equipment Alarm 2823690:
Equipment Fault, Hardware Fault Troubleshooting Procedure
Step

Action

Replace the FRU specified in the alarm output. Refer to the CDMA2000 1X AN & SDU FRU manual
for details.

If this alarm is a fan tray alarm, the fan tray alarm indicates a problem with the fan tray hardware
has occurred. When this alarm condition is detected on any fan all other fan trays in the system will
boost until the alarm condition clears.
Causes/debugging:

S Something is lodged in the fan, the voltage has dropped, or the motor is bad, causing the rotation of
the fan blades to stop or slow down.

S Fan alarms are produced by faulty fan units which should be replaced.
S The automatic boost of the fans may place the rotation speed within the alarm threshold which in
turn will clear the alarm. The fans will then unboost and the alarm will occur again creating a cycle
or bounce condition on the alarm and fan boost.
3

If this alarm is a power supply controller board alarm, this indicates a hardware failure and could be
caused by an overstress condition. Replace the hardware since it is not functional.

APR 2004

SDU Troubleshooting

11-5

11

Troubleshooting CCA Alarms

continued

2823790 Subcomponent Fault


This alarm is reported when a subcomponent fails due to equipment
malfunction.
Perform the following procedure to resolve this alarm.
Table 11-5: SDU Equipment Alarm 2823790:
Equipment Fault, Subcomponent Fault Troubleshooting Procedure
Step

Action

Replace the FRU specified in the alarm output. Refer to the CDMA2000 1X AN & SDU FRU manual
for details.

If it is a power supply fan failure alarm, check the following:

S
S
S
S

Check the power supply fan rotor for blockage.


Check the power supply air intake.
Check the power supply drive signal to the fan(s), they could be low or lost.
If this alarm occurs the power supply will overheat thereby generating the power supply high
temperature and very high temperature alarms and shutdown.

2823900 Power Supply Input Circuit Fault


A fault in the power supply input circuit due to a low bus voltage or a
blown fuse.
A power supply fuse alarm indicates all other power supply
alarms have been detected at the same time and the power supply
module is still present in the system.
Table 11-6: SDU Environmental Alarm 2823900:
Environmental Fault, Power Supply Input Circuit Fault Troubleshooting Procedure
Step

Action

Check the input power to the frame, and the circuit breaker to the PSM. Trace the fuse to make sure
that it is connected.

Check the alignment and replace the connecting module if necessary.

11-6

SDU Troubleshooting

APR 2004

Troubleshooting CCA Alarms

continued

11

Troubleshooting CCA
Environmental Faults

2824400 AC Mains Under Volt Threshold Exceeded


This alarm is reported when the number of Undervoltage incidents
exceeds the AC Mains Under Volt Threshold value. A potential Power
Supply Module failure or a service disruption is possible.
A power supply voltage alarm indicates that the power into the
power supply module (AC mains, 48V, etc.) has been interrupted.
Perform the following procedure to resolve this alarm.
Table 11-7: SDU Environmental Alarm 2824400:
Environmental Fault, AC Mains Under Volt Threshold Exceeded Troubleshooting Procedure
Step

Action

Verify that the input voltage is within the specified range for the SDU. (Refer to the CDMA2000 1X
SDU ATP manual for details on the acceptable input voltage range for the SDU).

Carefully check possible causes for this alarm. This alarm is indeterminate and could be caused by
power supply hardware, the AC filter, etc.

If the power supply voltage alarm condition also exists, this may indicate that the power supply
modules have been operating on batteries and the batteries are near being fully discharged.

If the power supply voltage alarm condition does not exist, this alarm indicates an unknown hardware
failure, so replace the power supply module.

2824440 Bus Under Voltage Threshold Exceeded


This alarm is reported when the number of Bus Undervoltage incidents
exceeds the Bus Under Voltage Threshold value. A potential Power
Supply Module failure or a service disruption is possible.
Perform the following procedure to resolve this alarm.
Table 11-8: SDU Environmental Alarm 2824440:
Environmental Fault, Bus Under Voltage Threshold Exceeded Troubleshooting Procedure
Step

Action

Evaluate and correct the power situation by verifying and ensuring that the input voltage is within the
specified range for the SDU.
Refer to the CDMA2000 1X SDU ATP manual for details on the acceptable input voltage range for the
SDU.

APR 2004

SDU Troubleshooting

11-7

11

Troubleshooting CCA Alarms

continued

2824460 Cabinet Sensor Threshold Exceeded


This alarm is reported when the number of Cabinet Sensor incidents
exceeds the Cabinet Sensor Threshold value.
Perform the following procedure to resolve this alarm.
Table 11-9: SDU Environmental Alarm 2824460:
Environmental Fault, Cabinet Sensor Threshold Exceeded Troubleshooting Procedure
Step
1

Action
Evaluate and correct the environmental condition cited by the alarm.

NOTE
Cabinet sensor 1 alarm occurs due to the cabinet / cage door being open.
Cabinet sensors 2 5 have not yet been defined.

2824480 External Sensor Threshold Exceeded


This alarm is reported when the number of External Sensor incidents
exceeds the External Sensor Threshold value.
Perform the following procedure to resolve this alarm.
Table 11-10: SDU Environmental Alarm 2824480:
Environmental Fault, External Sensor Threshold Exceeded Troubleshooting Procedure
Step
1

Action
Evaluate and correct the situation being alarmed by the customer defined sensors.

NOTE
Since the sensors are customer defined, the troubleshooting action will depend upon the specific
implementation. To troubleshoot this problem it is necessary to know how the sensors were wired for
the particular installation.

11-8

SDU Troubleshooting

APR 2004

Troubleshooting CCA Alarms

continued

11

2824700 Temperature High Threshold Exceeded


This alarm is reported when the number of high temperature incidents
exceeds the Temperature High Threshold value. This alarm indicates that
the Power Supply Module (PSM) has reached a minor over temperature
threshold. This alarm represents an early warning of potential
overheating in the power supplies. This alarm also indicates that the
lower of the two high temperature thresholds has been crossed.
Perform the following procedure to resolve this alarm.
Table 11-11: SDU Environmental Alarm 2824700:
Environmental Fault, Temperature High Threshold Exceeded Troubleshooting Procedure
Step
1

Action
Correct this problem by cooling the SDU environment or by removing dust from the fan input.
Check the following:

S
S
S
S

Check ambient temperature conditions.


Check power supply and cage ventilation.
Check internal power supply fans.
Check fan tray modules.

NOTE
When working with hardware follow all safety precautions specified in the CDMA2000 1X RAN Data
Hardware Installation and CDMA2000 1X AN & SDU FRU manuals.

APR 2004

SDU Troubleshooting

11-9

11

Troubleshooting CCA Alarms

continued

2824740 Temperature Very High Threshold Exceeded


This alarm is reported when the number of high temperature incidents
exceeds the Very Temperature High Threshold value.This alarm indicates
that the Power Supply Module (PSM) has overheated and shut down for
thermal overload protection. This is a critical condition because the
other power supplies are now carrying the load and are at high risk for
shutting down as well. This is true unless something is wrong, such as
a power supply fan failure, which is also a minor alarm.
Perform the following procedure to resolve this alarm.
Table 11-12: SDU Environmental Alarm 2824740:
Environmental Fault, Temperature Very High Threshold Exceeded Troubleshooting Procedure
Step
1

Action
Evaluate and correct the environmental condition cited by the alarm:

S If this alarm is coupled with fan failure alarms, correcting the fan problem should resolve this issue.
S If this alarm is occurring without any fan failure indications, evaluate and correct the temperature
conditions which may be causing the power supplies to overheat.
2

To evaluate temperaturerelated conditions, check the following:

S
S
S
S

11-10

Check ambient temperature conditions.


Check power supply and cage ventilation.
Check internal power supply fans.
Check fan tray modules.

SDU Troubleshooting

APR 2004

Troubleshooting CCA Alarms

continued

11

Troubleshooting CCA
Communication Faults

2827351 through 27362, Electronic ID 1wire Bus Failure


This alarm is reported when a Communications Subsystem Failure
causes an Electronic ID 1Wire Bus Connection Fault on the identified
Bus Connection (1 through 12). The last two digits of the Alarm Number
identify the Connection Number.
The devices that could cause this alarm include:

S DC/DC Power Supply


Perform the following procedure to resolve this alarm.
Table 11-13: SDU Communications Alarms 2821351 through 2821362:
Communication Fault, Electronic ID 1wire Bus Connection no. <112> Troubleshooting Procedure
Step
1

Action
Reseat the PSM. This alarm will occur if the PSM is either missing or not seated correctly.

NOTE
Refer to the appropriate documentation for installing/seating the PSM.
2

If you reseated the PSM and the alarm still occurs, contact CNRC.

2827801 through 27802, Packet Bus Connection


This alarm is reported when a LAN Error or Loss of Frame condition
causes a Packet Bus Connection Fault on the identified Packet Bus
Connection (1 or 2). The last two digits of the alarm number identify the
connection number. The devices that could cause this alarm are: Bearer
Payload Processor (BPP) Card and Cabinet and Customer Alarm (CCA)
Card.
Perform the following procedure to resolve this alarm.
Table 11-14: SDU Communications Alarms 2821801 and 2821802:
Communication Fault, Packet Bus Connection no. <12> Troubleshooting Procedure
Step

Action

Physically check the SDU card (BPP or CCA) specified in the alarm output. Determine whether it is
properly connected into the SDU backplane.
Refer to the CDMA2000 1X RAN Hardware Installation manual for details on card placement and
connections.

If it is not a card connectivity problem, then it is necessary to replace the FRU identified in the alarm.
Refer to the CDMA2000 1X AN & SDU FRU manual for details.

APR 2004

SDU Troubleshooting

11-11

11

Troubleshooting CCA Alarms

continued

Notes

11-12

SDU Troubleshooting

APR 2004

Chapter 12: OMCR Failure Scenarios


Table of Contents

OMCR Failure Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
OMCR Alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

APR 2004

SDU Troubleshooting

12-1
12-1
12-1

12

Table of Contents

continued

Notes

12

SDU Troubleshooting

APR 2004

OMCR Failure Scenarios

Overview

This section provides a listing of OMCR alarms that apply for multiple
network elements but also apply to the SDU.
Refer to the System Output Messages manual for the alarm description
and operator action.
OMCR Alarms

The following list describes related OMCR alarms:

S ALARM: 2878 Platform Out of Service


S ALARM: 2879 Platform Communication Failure with OMCR
S ALARM: 28102 SNMP MIB Support Unavailable for Committed
Version

S
S
S
S

ALARM:28103 Management Synchronization Failed with NE


ALARM: 306168 OMCR /SDU Database Mismatch
ALARM: 306170 OMCR GCMGR Queue to NE Inactive
ALARM:286820 Synchronization Failed: Unexpected Response
from Network Element

S ALARM:286821 Synchronization Failed: No Response from


Network Element

S ALARM 28 6851 Config File Corrupted

APR 2004

SDU Troubleshooting

12-1

12

OMCR Failure Scenarios

continued

Notes

12

12-2

SDU Troubleshooting

APR 2004

Chapter 13: System Initialization Problems


Table of Contents

13
System Initialization Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SPROC Boot Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
OMCR Commissioning Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . .

APR 2004

SDU Troubleshooting

13-1
13-1
13-1
13-2

Table of Contents

continued

Notes

13

SDU Troubleshooting

APR 2004

System Initialization Problems

Introduction

This chapter contains troubleshooting procedures for system


initialization problems.
SPROC Boot Problems

SPROC Fails to Receive DHCP from the ISB

13

Refer to chapter 8, section ISB unable to provide DHCP service to


SPROC.
SPROC Reboots Due to Bad NE_LIF.xml
If the software loaded on the SPROC does not agree with the
NE_LIF.xml file, then the SPROC will fail to initialize and will reset.
The following line will be displayed during init indicating a problem.
NEI : NSIF files missing or files not consistant
with LIF in /active/software.
NEI : Node is Not Potential Master based on
internal criteria[dd99220]
In this situation, the NE_LIF.xml file (located in the /active/software
directory) needs to be regenerated and downloaded to the SPROC. Note
that the NSIFsprocsproc file should NEVER be deleted from the
NE_LIF file.
Contact CNRC.
SPROC Component Factory Startup Fails
The following output indicates this SPROC error:
OMIX: palPDRSopen ERROR S_errno_EAGAIN for
/active/config/dynamic/baseNECDF.xml
OMIX: palPDRSseek ERROR S_errno_EAGAIN
This problem occurs due to a problem with permissions on the
.../data/pdrs directory. This directory needs permissions open to create a
file called safe. The component factory failure message is:
NEI : sending Data to NEIP
Component Factory started.
usage value is 0
hapua: onGoActive
availability value is 255
usage value is 1
Component Factory failed.
availability value is 255
APR 2004

SDU Troubleshooting

13-1

System Initialization Problems

continued

Call CNRC to resolve this problem.


OMCR Commissioning
Problems

SDU Sync Command Fails


If the SDU sync command fails during commissioning, investigate the
following:

13

S Verify that you can ping the SPROC external IP address from the
OMCR.

S Verify the active version on the OMCR (disp sdu# version).


S Verify that no errors are seen on the SPROC. Use the following
command to display more debug information on the SPROC:
SetOAMPLogDebug(1)

IMPORTANT

13-2

Logging into the SPROC should be performed during a


scheduled maintenance window to avoid system disruption.

SDU Troubleshooting

APR 2004

Chapter 14: Call Processing Failures


Table of Contents

Call Processing Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
SPROC MM RA/RM Links Not Active . . . . . . . . . . . . . . . . . . . . . .
No PCF Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Problems with SDF Resource Assignment . . . . . . . . . . . . . . . . . . . . . .
Problems with cBTS Proxy Request . . . . . . . . . . . . . . . . . . . . . . . . . . .
Problems with Circuit IOS Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Interface Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

APR 2004

SDU Troubleshooting

14-1
14-1
14-1
14-1
14-1
14-1
14-1
14-2
14-2

14

Table of Contents

continued

Notes

14

SDU Troubleshooting

APR 2004

Call Processing Failures

Introduction

This chapter describes SDUrelated call processing failures.


Use

This section describes the basic subsections of call processing related to


a system with the SDU architecture. If either of these points fail, CFCs
and/or alarms will occur. In that case, follow the applicable
troubleshooting procedures for those CFCs/alarms. Some
troubleshooting actions specified here may also be used to supplement
those procedures.

14

SPROC MM RA/RM Links Not


Active

The following SPROC MM links must be active or else calls cannot be


established and CFCs and alarms will occur. If such CFCs and/or alarms
occur, follow the related troubleshooting procedures.

S MMSDFRALINK (used for SDF resource setup)


S MMPCFRALINK (used for PCF resource setup)
No PCF Resources

PCF resources must be available to establish new SDU 1X packet data


calls.
If a CFC indicates that there are no PCF resources available, check the
following (OMCR CLI commands can accomplish this):

S Verify that the PDSN exists.


S Verify that the PCF resources are INS.
S Verify that the PCF CPEs (Channel Processing Elements) are INS.
Each PCF card has a certain number of CPEs.

S Verify that the SDUPDSNLINK is INS.


Problems with SDF Resource
Assignment

SDF resources must be available to establish any SDU 1X calls. If there


is a problem with the SDF assignment portion of call setup, a CFC will
occur. Refer to the applicable CFC for the appropriate troubleshooting
procedure.
Problems with cBTS Proxy
Request

The XC PKTIF must receive the cBTS Proxy Request in order to set up
the call on that end. If this doesnt happen, a CFC will occur.
APR 2004

SDU Troubleshooting

14-1

Call Processing Failures

continued

Problems with Circuit IOS


Traffic

If CFCs occur due to problems with sending/receiving Circuit IOS traffic


between devices, you can use the following methods to further inspect
the problem.
Sniffer on the Gigabit Link
Use a sniffer tool on the gigabit link to see if bidirectional traffic frames
can be seen.
Reverse Link
To determine the traffic being seen on the sniffer, examine the 4th byte
of the data field.

14

4th Byte (Frame Type):


00 = Idle
7E = Invalid Speech
01 20 = Valid Speech
Forward Link
5th Byte (Frame Type):
04 = 8th Rate Frame
Other = Valid speech frames will vary in rate
Look at XC KSW Port Connections
From the OMP, the port audit command can be used to make sure that
the desired connections are being made. This can be compared to the
TDM ports used by the MMS and the PSICE to verify the proper
connection is in place.
A Interface Problems

The A interfaces related to SDU are A8/A9 (between SDU SDF and
SDU PCF) and A10/A11 (between the SDU PCF and the PDSN).
The following steps must occur or else CFCs will occur and packet data
calls will not be established. In such a case, refer to the troubleshooting
procedure for the applicable CFC.

S
S
S
S

A9 Setup A8 must be sent by SDF to PCF


A9 Setup A8 must be received and seen by PCF
PCF must send A11 Registration Request to PDSN
PDSN must send an A11 Registration Reply
NOTE
For details on A interface call setup, refer to the
CDMA2000 1X RAN Functional Description.

14-2

SDU Troubleshooting

APR 2004

Chapter 15: CFC Failures


Table of Contents

SDU CFC Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Common Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Troubleshooting CFC Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15-1
15-1
15-2
15-5

15

APR 2004

SDU Troubleshooting

Table of Contents

continued

Notes

15

SDU Troubleshooting

APR 2004

SDU CFC Troubleshooting

Overview
Introduction
This chapter contains CFC troubleshooting procedures for SDU.
NOTE
When displaying status data from the OMCR, it may be
helpful to log the data to a file, especially if you plan on
contacting CNRC.

OMCR Commands and Database Timers


This section refers to the use of OMCR commands as well as call
processing timers. For details on the commands and timers, refer to the
Cellular System Administration Guide and System Command Reference.
CFCs and Call Flow Context
When using tools to isolate call flow breakdowns, such as sniffer logs,
SMAP, Mobile DM, and callproc, an advanced knowledge of the call
flow context surrounding the particular CFC(s) may be helpful.
For example, if you are troubleshooting CFC 06, it may help to be
familiar with the call processing context in which CFC 06 occurs when
investigating the possible causes, such as what call processing messages
are sent and received in that particular segment of call setup.
The CDL/CFC Reference manual contains call flows with CFCs plotted
accordingly.
In addition, refer to the CDL/CFC Reference manual for more detailed
descriptions of the CFCs.
SMAP and Mobile DM
Many troubleshooting procedures deal with operation of SMAP and a
Mobile Diagnostic Monitor (DM) to collect call processing messages
and verify what messages were actually sent and received on each end of
the connection. For details on SMAP operations, and descriptions of the
messages types and fields that can be logged (what is received on the
reverse link, what is sent on the forward link), refer to the SMAP Control
GUI and SMAP Data Analyzer Operators Guides. For details on the
same capabilities for the Mobile DM, refer to your vendor
documentation.

APR 2004

SDU Troubleshooting

15-1

SDU CFC Troubleshooting continued


Common Procedures

The following procedures are common throughout this chapter. Refer to


this section when performing these procedures.
Viewing BPP Spel Logs
IMPORTANT

Logging into the BPP should be performed during a


scheduled maintenance window to avoid system disruption.

To view errors and events logged by the BPP, use the spel <option>
command at the BPP command line.
The command format is spel <option> where <option> may be:

S
S
S
S
S
S
S
S
S
S
S
S

celm: Change the Event Log filter settings


cl: Clear event log
cls: Change Logging State
cmm: Change realtime MMI filter settings
cfs: Change outgoing MMI filter settings
le: Log event
ler: Log error (Used internally by event logger)
se: Show events
sfe: Show filtered events
ser: Show errors
selm: Show current Event Log filter settings
slogs: Show current logging states of MMI and EL

For example, to view events reported by the BPP:


spel se
For example, to view errors reported by the BPP:
spel ser
Collecting Sniffer Logs
Sniffer tools are optional aids for troubleshooting. Refer to the
documentation for the sniffer application for details.

15-2

SDU Troubleshooting

APR 2004

SDU CFC Troubleshooting continued

Checking the Service Option Database


In certain troubleshooting procedures, it may be necessary to check the
service option database.
From the OMCR, use the following commands to check for service
option settings:

S display servoptcbsc#servopt# alt displays the


alternate service options associated with the specified Service Option.

S display servoptcbsc#servopt# soserv command


displays the Service Option parameters for the specified Service
Option. For example, it displays the values such as XcdrCkt and
TchCkt (used in troubleshooting CFCs).

S The following commands check for forward and reverse frame rates
(they also display other Mux settings such as Radio Configuration):
for 1X: display servoptcbsc#servopt# so1xcrc
for IS95A/B:
display servoptcbsc#servopt# so95abcrc
Corresponding edit commands are available to correct the settings if
necessary.
IMPORTANT

Refer to the System Command Reference before changing


these settings, and for the correct ranges. For some
parameters, it is recommended to consult Motorola first
before making any changes.

SdfCpePrintStore
Use the SdfCpePrintStore command to determine whether calls are being
allocated across all CPEs (Channel Processing Elements) on the SDF
card.
This command will also identify the IP addresses of the individual
CPEs.
At the SPROC, enter: SdfCpePrintStore

IMPORTANT

APR 2004

Logging into the SPROC should be performed during a


scheduled maintenance window to avoid system disruption.

SDU Troubleshooting

15-3

SDU CFC Troubleshooting continued

For example:
[SDUSPROC2]> SdfCpePrintStore
***** Printing Cpe Store for SDF ******
//
SDF Cpe Index = 0
status = INS
ipAddress
= a01013d
capacity
= 3200
availableBwu = 3200
usedBwu
= 0
//
//
SDF Cpe Index = 1
status = INS
ipAddress
= a01013e
capacity
= 3200
availableBwu = 3200
usedBwu
= 0
//

15-4

SDU Troubleshooting

APR 2004

SDU CFC Troubleshooting continued


Troubleshooting CFC Failures

CFC=3 RF Layer 2 Failure


This CFC is pegged when a RF Layer 2 error is reported by a device.
There are many reasons that a RF Layer 2 error might be reported.
Causes:
Detailed Cause
0x1114
0x1115
0x1119

Description
Layer 2 Failure Reported by SDU DSP
Layer 2 Failure Reported by SDU L2
RF Layer 2 Failure

Troubleshooting

S Dump the BPP SPEL Log


IMPORTANT

Logging into the BPP should be performed during a


scheduled maintenance window to avoid system disruption.

CFC=4 RF Loss
This CFC is generated when the SDF CPE detects an RF loss condition.
RF Loss is defined by L3CC on the SDF receiving a Traffic Channel
State Change indication for Invalid Speech (erasure) frames from a valid
speech state. This will start the RF Loss Timer, if valid speech frames
are not received before the timer expires, the call will be torn down for
RF Loss.
Causes:
Detailed Cause
0x111A

Description
RF Loss

Troubleshooting:
RF Optimization may be necessary if RF Losses occur often.

CFC=5 No TCH Preamble Detected


This CFC is generated when the BTS tears down the call due to not
seeing any valid Circuit IOS frames on the forward link. Forward Ckt
IOS frames are generated by the SDF DSP after the SDU has received
the CDMA Channel Assign and completed the cBTS Proxy messaging
(for circuit BTS case) or completed the RF resource messaging (Packet
BTS).
The other cause could be for MccCpT1 timer expiring on the BTS, this
timer monitors for the mobile appearing on the traffic channel. The
Mobile moves from the paging channel to the traffic channel when it
receives the CDMA Channel Assign message from the BTS. This
message specifies what traffic channel (TCH) to move to.
APR 2004

SDU Troubleshooting

15-5

SDU CFC Troubleshooting continued

Causes:
Detailed Cause
0x111B
0x2005

Description
No TCH Preamble Detected
MccCpT1 Expired

Troubleshooting:
CFC 05 indicates either an RF environment problem or a possible
internal BTS problem. Use the following options to narrow down the
problem.
Problem is due to the RF environment:

S If possible, collect a mobile DM (Diagnostic Monitor) log to see what


information the mobile is seeing and what actions it is taking.

S Compare with an SMAP log to get the full picture of what is sent and
received on both ends.

S Messages not received/acknowledged on either side indicate RF


environment problems are occurring. For example, the BTS may not
have received the TCH Preamble frames from the mobile, or the
mobile may have not received the message from the BTS prompting
the transmission of TCH Preamble frames e.g. did not receive the
Channel Assignment/Extended Channel Assignment message.
Problem is due to the network:

S Collect a sniffer log to determine what bearer frames are present.


S Reset the BTS, force a conf load.
CFC=6 No Strau Synch
The bearer path connection between the XC/SDU and an assigned traffic
channel could not be established during call setup.
Causes:
Detailed Cause
0x111C

Description
No STRAU Synch

Troubleshooting:

S Collect a sniffer log to determine what bearer frames are present.


S Collect a sniffer log at multiple points to determine the source of
disconnect; from the Catalystt to the SDU ISB, and the SDU ISB to
the SDU BPP.

15-6

SDU Troubleshooting

APR 2004

SDU CFC Troubleshooting continued

CFC=7 CP Timeout Awaiting MS Acquisition


The XC/SDU subsystem did not acquire the mobile (did not receive null
frames from the mobile) during call setup.
Causes:
Detailed Cause
0x1121

Description
CP Timeout Awaiting MS Acquisition

Troubleshooting:
S If CFC 05 occurs, then CFC 07 will also occur. If this is the case,
instead troubleshoot CFC 05.
S If CFC 05 did not occur, and CFC 07 did occur, then there is probably
a disconnect:
Collect a sniffer log to determine what bearer frames are present.
Collect a sniffer log at multiple points to determine the source of
disconnect; from the Catalyst to the SDU ISB, and the SDU ISB to
the SDU BPP.

CFC=12 CPP Call Setup Timeout


This CFC is pegged when one of a number of various call setup timers
expires during call setup.
Causes:
Detailed Cause
0x114C
0x114D
0x114E
0x154E

Description
SDF Call Setup Timeout
No Vocoder Connect Received at XC
No connect order received
SDF Assignment Timeout. No SDF Resource
Update is received at the SPROC before

Troubleshooting:
S Verify that the SDF Call Setup Timer is shorter than the Vocoder
Connect Timer at the XC before troubleshooting this failure, as this
can be a misleading failure cause.
S Dump some SPEL events on the BPP to see if any errors are being
reported by the SDF.
IMPORTANT

Logging into the BPP should be performed during a


scheduled maintenance window to avoid system disruption.

CFC=13 CP Timeout Awaiting Service Option Ack


For P_REV_IN_USE=2 mobiles (800MHz cellular band), the XC/SDU
fails to receive an L2 ACK for the Service Option Response Order.
For P_REV_IN_USE=1 (1.9GHz PCS band), P_REV_IN_USE=3
(800MHz cellular band) and above mobiles, the CPP fails to receive an
L2 ACK or Service Connect Completion message.
APR 2004

SDU Troubleshooting

15-7

SDU CFC Troubleshooting continued

Causes:
Detailed Cause
0x1123

Description
No service option ack received from MS

Troubleshooting:

S From a Mobile DM (Diagnostic Monitor) tool, determine if the MS


received the Service Option Response Order/Service Connect message
and is responding with a L2 Ack./Service Connect Completion
message.

S Compare with SMAP data to see what was actually sent and received
by the BTS.

S Check the value of the XcdrCkt. ServType in the Servopt database for
the specified service option.

S Check the value of the TchCkt. Serv Type in the Servopt database for
the specified service option.

S Check the value of the Sdf Service Connect Complete timer:


At the OMCR CLI: display sdusdf1 sdfsn

S Check the value of the Setup Action Time, SDF SN Setup A Timer.
S Determine if the SDU (SDF L3CC) is sending a Service Option
Response Order/Service Connect message. You can use SMAP to
monitor these messages.

S Dump some SPEL events on the BPP to determine if there are any
errors being reported by the BPP:
Check to see if Timer 23 (Sdf Service Connect Complete Timer) is
expiring. This cant be done on a system under call load.
Check that L2 received the Service Option Response Order/Service
Connect message. This cant be done on a system under load.
Check that the BTS received the Scap Service Configuration
Change Request message.
IMPORTANT

15-8

Logging into the BPP should be performed during a


scheduled maintenance window to avoid system disruption.

SDU Troubleshooting

APR 2004

SDU CFC Troubleshooting continued

CFC=14 Not enough Mobile Status Information


Received
This CFC is pegged when there is no response from the Mobile to the
Status Request message on expiration of the Sdf Status Response timer,
or if the Mobile rejects the Status Request message by sending an MS
Reject Order message.
Causes:
Detailed Cause
0x1125

Description
Not enough status information received from Mobile

Troubleshooting:

S From a Mobile DM tool, determine if the Mobile received the Status


Request message at least 3 times.

S From the Mobile DM tool, determine if the Mobile responded to the


Status Request message with a Status Response/MS Reject Order
message.

S Also determine whether the BTS received the Status Response using
SMAP.

S Check the value of the Sdf Status Response timer:


At the OMCR CLI: display sdusdf1 sdfsn

S Dump some SPEL events on the BPP to determine if there are any
errors being reported by BPP.
IMPORTANT

Logging into the BPP should be performed during a


scheduled maintenance window to avoid system disruption.

S Collect a BTS TCH Audit Log to determine if the BTS is getting any
erased frames in the Status Response message.

S Check the RF Calibration to determine if the RF environment can be


further improved.

APR 2004

SDU Troubleshooting

15-9

SDU CFC Troubleshooting continued

CFC=15 Negotiation Failure


This CFC is pegged when the Mobile and the SDU are unable to arrive
at a mutually acceptable service configuration. This CFC is pegged on
expiration of the Sdf Negotiation timer, or if the Mobile rejects the
Service Request/Response/Connect /Option Response Order message by
sending a MS Reject order message.
Causes:
Detailed Cause
0x1124

Description
No mutual service option/configuration

Troubleshooting

S Verify the value of the Sdf Negotiation Timer.


S From a Mobile DM tool and SMAP determine if the BTS is sending
and the Mobile is receiving a Service Request/Response message.

S From the Mobile DM tool and SMAP, determine the response of the
Mobile to the Service Request/Response message and whether the
response is received by the BTS.
The Mobile can respond with a Service Response/Request or a MS
Reject order or send no response at all.

S From the Mobile DM tool determine if the Mobile is receiving a


Service Connect/Option Response Order message.

S From the Mobile DM tool and SMAP, determine the response of the
Mobile to the Service Connect /Option Response Order message and
whether the BTS receives the response.

S Using SMAP, check to see if the SDU (SDF L3CC) sends out a
Service Connect Message.

S In the servopt database, check the Alternative Service Option list of


the service option for which negotiation is taking place.

S In the servopt database, check the IS95 A/B and 1x Forward and
Reverse frame rates for the service option for which negotiation is
taking place.

S Using a sniffer, it is also possible to check to see if there are any


entries in the log which indicate the proposable Service Option List is
empty. Check the service option list supported by the Mobile by
examining the logging of the Status Response message.

15-10

SDU Troubleshooting

APR 2004

SDU CFC Troubleshooting continued

CFC=35 Resource Allocation Timeout


This CFC is pegged when the MM believes that it has a valid connection
with the SDURA, but does not receive a resource response message
before the MM Resource Response timer expires.
Causes:
Detailed Cause
0x1148

Description
No response from SDURA

Troubleshooting:
Collect a sniffer log to verify that the MM is sending resource requests
over SCTP.

CFC=36 No SDU Resources Available


This CFC is pegged when the resource response from the SDU indicates
that all available SDF resources are congested.
Causes:
Detailed Cause
0x114A

Description
No SDF Resources available

Troubleshooting:

S If there is a congestion alarm occurring for the SDU, troubleshoot the


alarm.

S At the OMCR, use SDUrelated STATUS and DISPLAY commands


to verify BPP usage.

S Check the OMCR event logs to verify that the SDFRG (SDF
Resource Group) is not reporting capacity changes.

S Check the resource allocation at the SPROC, is it close to the limit for
the SDFs?

S At the SPROC, run SdfCpePrintStore . Are calls being


allocated across all CPEs?
IMPORTANT

APR 2004

Logging into the SPROC should be performed during a


scheduled maintenance window to avoid system disruption.

SDU Troubleshooting

15-11

SDU CFC Troubleshooting continued

CFC=52 Equipment Failure at BSC


This CFC is pegged as an indication of a general equipment failure
somewhere at the CBSC.
Causes:
Detailed Cause
0x0101
0x0102
0x0103
0x0104
0x154F
0x1550
0x1551
0x154B
0x154C
0x154D
0x1552
0x1553

Description
MM RM Link Failure
MM RM Session ID Change
MM RM Deregistration
MM RM ReRegistration Failure
SDU CP Internal Failure
SDU CPE Failure
SDU BPP Failure
PSISIG Failure
PSICE Failure
PSI SDU Failure
Backhaul Bearer Loss
Network Bearer Loss

Troubleshooting:

S Determine device failure based on detailed cause


S Collect log info (OMP logs, event logs, etc) dependent upon specific
device failing

S Address any alarms that are occurring. Follow the troubleshooting


procedures for the appropriate alarms.

CFC=60 Error Between MSC and BSC


Troubleshooting:

S Look at MM callproc logs.


S Use CDLs to try to isolate the problem to a device.
CFC=61 Protocol Error Between MM and XC/SDU
MmCpT14 Timeout waiting for response from XC or SDU.
Causes:
Detailed Cause
NA
NA

Description
No PSILINK in MM DB
No SDFRA in MM DB

Troubleshooting:

S Look at MM callproc logs.


S Use CDLs to try to isolate a device.

15-12

SDU Troubleshooting

APR 2004

SDU CFC Troubleshooting continued

CFC=62 XC Detected Error


Troubleshooting:

S Look at MM callproc logs.


S Use CDLs to try to isolate a device.
CFC=80 MM Internal Error
This CFC will be pegged if the MM has completed SDU registration and
the SDFRA Link is setup, but the MM is missing something else in its
database that is preventing the call from being setup properly.
Causes:
Detailed Cause
NA
NA

Description
No PSILINK in MM DB
No SDFRA in MM DB

Troubleshooting:

S Look at MM callproc logs.


CFC=112 Pkt Oriented Data Call Setup Failure
This CFC will be pegged if the call setup fails during the PCFPDSN
setup messaging phase.
Causes:
Detailed Cause
0x1140

Description
PCF Pkt Oriented Setup Timer Expired

Troubleshooting:

S Use a PDSN system monitoring tool to view where the A10/A11


(RP) control messaging experienced problems. This should help
isolate the problem to the PCF or the PDSN. Refer to the CDMA2000
1X RAN Functional Description for a detailed description of A10/A11
call setup processing.

S Investigate the status of the PDSN.


S Verify that the SDU PCF CPE IP addresses are in the PDSN VHA
pool. To determine the PCF CPE IP addresses, enter the following
command at the SPROC: SdfCpePrintStore
IMPORTANT

APR 2004

Logging into the SPROC should be performed during a


scheduled maintenance window to avoid system disruption.

SDU Troubleshooting

15-13

SDU CFC Troubleshooting continued

CFC=113 Pkt Oriented Data Call Protocol Violation


This CFC will be pegged usually because the SDF TA8 inactivity timer
expired. This will also be pegged if the PPP session fails to establish.
Causes:
Detailed Cause
0x1141
0x1143

Description
Protocol error between MS and BS
TA8 Inactivity Timer Expired

Troubleshooting:

S Troubleshoot the PDSN.


S Get a sniffer log to look at PPP messaging. If the CFC occurred due
protocol error, this means that the PPP session did not establish
between the mobile and the PDSN due to the fault of the mobile or
PDSN. For a description of what successful PPP setup looks like
between the mobile and the PDSN, refer to the CDMA2000 1X RAN
Functional Description.

CFC=138 No PSI_SDU Available


This CFC is pegged when the MM is not aware of any SDUs being
available.
Causes:
Detailed Cause
NA
NA

Description
MM SDFRA link is OOS
SDU is OOS (no MMSDU connection)

Troubleshooting:

S Troubleshoot any alarms for MMSDFRALINK (the call processing


link between the MM and SDU SDFRA). Refer to the applicable
alarm procedure.

S Troubleshoot the SDU OOS alarm. Refer to the applicable alarm


procedure.

S Verify you can ping the SPROC from the MM. Refer to your IP
planning documentation for the applicable IP addresses.

15-14

SDU Troubleshooting

APR 2004

SDU CFC Troubleshooting continued

CFC=139 No PSICE Available


This CFC will be pegged when all available PSICEs have been utilized
on the system.
Causes:
Detailed Cause
0x114B
0x114F
0x1150

Description
cBTS Proxy Resource Request Timeout
cBTS Proxy Resource Configure Timeout
cBTS Proxy Resource Mismatch

Troubleshooting:

S Consider increasing PSICE capacity.


S To verify actual capacity/usage, you can enter Disp_psice at the
XC OMP to determine PSICE availability. You can also get a sniffer
log to look at the actual cBTS Proxy messaging.

CFC=142 No PDSN Resources Available


The A10/A11 interface was not setup due to no PDSN resources being
available.
Causes:
Detailed Cause
0x1D79

Description
PDSN Resources Not Available

Troubleshooting:

S Consider increasing PDSN capacity/resources.


CFC=143 No PCF Resources Available
This CFC is pegged when PCFRA indicates that there are no available
PCF resources.
Causes:
Detailed Cause
0x1D32

Description
No PCF Resources available

Troubleshooting:

S Consider increasing SDU PCF capacity/resources.

APR 2004

SDU Troubleshooting

15-15

SDU CFC Troubleshooting continued

CFC=146 A11 Registration Denied


This CFC is pegged when the PDSN is denying registration in the A11
Reg Reply message.
Causes:
Detailed Cause
0x1D80
0x1D81
0x1D82
0x1D83
0x1D85
0x1D86
0x1D88

Description
PDSN Reg Denied Reason Unspecified
PDSN Reg Denied Administratively Prohibited
PDSN Reg Denied Insufficient Resources
PDSN Reg Denied MS Failed Authentication
PDSN Reg Denied Identification Mismatch
PDSN Reg Denied Poorly formed request
PDSN Reg Denied Unknown PDSN Address

Troubleshooting:

S Examine the PDSN for possible failures. Refer to vendor


documentation for details.

15-16

SDU Troubleshooting

APR 2004

*68P09258A80B*
68P09258A80B

Technical
Information

SDU TROUBLESHOOTING

SOFTWARE RELEASE 2.16.3.X


SDU
CDMA2000 1X

ENGLISH
APR 2004
68P09258A80B

SDU TROUBLESHOOTING
SOFTWARE RELEASE 2.16.3.X
SDU
CDMA2000 1X

APR 2004

ENGLISH

68P09258A80B

Technical Information Products and Services

STANDARD MANUAL PRINTING INSTRUCTIONS


Filename:

Part Number: 68P09258A80B

APC:

Title: SDU Troubleshooting


Volume

of

Print Vendor:

Date

STANDARD SPECIFICATIONS FOR REFERENCEDO NOT MODIFY

PAPER:

1st. LEVEL TABS: 2nd. LEVEL TABS:

Body: 70 lb.
Inside Cover: 65 lb. Cougar
Tabs: 110 lb. Index
Binder Cover: Standard TED
cover 10 pt. Carolina

Single Sided
5 Cuts
Clear Mylar
Pantone 2706C
Black Ink

Single Sided
7 Cuts
Clear Mylar
White
Black Ink

FINISHING:

3Ring Binder
SlantD
3Hole Punched
(5/16in. dia.)
Shrink Wrap Body

Black ink for body, inside cover, and binder cover.

TAB and SHEET SIZE/QUANTITY

7X9

8.5x11

11x17

1st Level Tabs

Sheets = (Total Pages) / 2

2nd Level Tabs

NONSTANDARD SPECIFICATIONS
Tape Bound

Corner Stitch

Other: Meet with manager to determine the deliverable.

SPECIAL INSTRUCTIONS

You might also like