SJ-20130416095742-019-ZXUN ICX (MSCS) (V4.13.10) MSC Server Troubleshooting Guide - 504807

ZXUN iCX(MSCS)
MSC Server
Troubleshooting Guide
Version: V4.13.10
ZTE CORPORATION
No. 55, Hi-tech Road South, ShenZhen, P.R.China
Postcode: 518057
Tel: +86-755-26771900
Fax: +86-755-26770801
URL: http://ensupport.zte.com.cn
E-mail: support@zte.com.cn
LEGAL INFORMATION
Copyright © 2013 ZTE CORPORATION.
The contents of this document are protected by copyright laws and international treaties. Any reproduction or
distribution of this document or any portion of this document, in any form by any means, without the prior written
consent of ZTE CORPORATION is prohibited. Additionally, the contents of this document are protected by
contractual confidentiality obligations.
All company, brand and product names are trade or service marks, or registered trade or service marks, of ZTE
CORPORATION or of their respective owners.
This document is provided “as is”, and all express, implied, or statutory warranties, representations or conditions
are disclaimed, including without limitation any implied warranty of merchantability, fitness for a particular purpose,
title or non-infringement. ZTE CORPORATION and its licensors shall not be liable for damages resulting from the
use of or reliance on the information contained herein.
ZTE CORPORATION or its licensors may have current or pending intellectual property rights or applications
covering the subject matter of this document. Except as expressly provided in any written license between ZTE
CORPORATION and its licensee, the user of this document shall not acquire any license to the subject matter
herein.
ZTE CORPORATION reserves the right to upgrade or make technical change to this product without further notice.
Users may visit ZTE technical support website http://ensupport.zte.com.cn to inquire related information.
The ultimate right to interpret this product resides in ZTE CORPORATION.
Revision History
Revision No. Revision Date Revision Reason
R1.0 2013-05-31 First edition
Serial Number: SJ-20130416095742-019
Publishing Date: 2013-05-31 (R1.0)
SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

Contents
About This Manual ......................................................................................... I
Chapter 1 Introduction to Troubleshooting ............................................. 1-1
1.1 Fault Classification ............................................................................................. 1-1
1.2 Requirements for Maintenance Personnel............................................................ 1-2
1.3 Common Fault Handling Procedure ..................................................................... 1-2
1.4 Emergency Fault Handling Procedure.................................................................. 1-5
1.5 Common Troubleshooting Tools .......................................................................... 1-6
1.6 Common Troubleshooting Methods ..................................................................... 1-7
Chapter 2 Hardware Faults ........................................................................ 2-1

2.1 Board Faults ...................................................................................................... 2-2
2.1.1 The H/S Indicator Is Solid On .................................................................... 2-2
2.1.2 The H/S Indicator Is Flashing .................................................................... 2-3
2.1.3 The OK Indicator Is Off ............................................................................. 2-4
2.1.4 The OK Indicator Is Solid On in Red .......................................................... 2-5
2.1.5 The HOST Indicator Is Flashing in Red ...................................................... 2-6
2.1.6 The HD1/HD2 Indicator Is Solid On in Red................................................. 2-6
2.1.7 Failure in Board Powering On/Off When Operated with the Extractors ......... 2-7
2.1.8 A Board Is Abnormally Powered Off with the H/S Indicator Solid On ............ 2-7
2.2 Power Supply Faults........................................................................................... 2-8
2.2.1 The H/S Indicator Is Solid On .................................................................... 2-8
2.2.2 The H/S Indicator Is Flashing .................................................................... 2-9
2.2.3 All the Devices or Half of the Devices in a Shelf Are Powered Off .............. 2-10
2.2.4 Half of the Devices in a Shelf Are Powered Off After a PEM Is Removed ......2-11
2.2.5 Some Boards Are Suddenly Powered Off..................................................2-11
2.3 Fan Faults ....................................................................................................... 2-12
2.3.1 The H/S Indicator Is Solid On .................................................................. 2-12
2.3.2 The H/S Indicator Is Flashing .................................................................. 2-13
2.3.3 The RUN Indicator Is Flashing in Green but the Fans always Run in the
Full Speed ............................................................................................ 2-14
2.3.4 The RUN Indicator Is Off but the Fans always Run in the Full Speed ......... 2-15
Chapter 3 NMS Faults ................................................................................ 3-1

3.1 Communication Faults ........................................................................................ 3-1
3.1.1 Timeout When the LMT Transfers Data to the OMP.................................... 3-1

3.1.2 IE Cannot Access the OMM Server ........................................................... 3-3
3.2 Authority Faults .................................................................................................. 3-4
3.3 Database Faults ................................................................................................. 3-5
3.3.1 Failure to Connect the Database ............................................................... 3-5
3.3.2 Export Function of an LMT Is Unavailable .................................................. 3-6
3.4 Performance Management Faults........................................................................ 3-7
3.4.1 Time Delay in Performance Data Reporting................................................ 3-8
3.4.2 Cannot Report Performance Data ............................................................. 3-8
3.5 Alarm Management Faults .................................................................................. 3-9
3.5.1 Alarm Filtering Rules Do Not Work .......................................................... 3-10
3.5.2 Alarms Cannot Reach a Well-Connected Alarm Box ................................. 3-10
Chapter 4 Service Faults............................................................................ 4-1

4.1 Basic Call Service Faults .................................................................................... 4-1
4.1.1 Analyzing Instance 1 (Call Failure for Inconsistent Interconnection
Parameters) ........................................................................................... 4-4
4.1.2 Analyzing Instance 2 (Roaming Subscribers Cannot Receive Calls) ............ 4-5
4.1.3 Analyzing Instance 3 (Defaulting Subscriber Fails to Call Special Service
Numbers) ............................................................................................... 4-6
4.2 SMS Faults ........................................................................................................ 4-6
4.2.1 Analyzing Instance 1 (MS/UE Cannot Receive the Short Message) ........... 4-10
4.2.2 Analyzing Instance 2 (Office P Cannot Receive A Short Message from
Extranet W) .......................................................................................... 4-10
4.3 Location Update Service Faults......................................................................... 4-12
4.3.1 Analyzing Instance 1 (Call Failure for Setting the Periodic Location
Update) ................................................................................................ 4-14
4.3.2 Analyzing Instance 2 (Location Update Fails to Complete) ........................ 4-15
Appendix A Cautions for Routine Maintenance ..................................... A-1

Appendix B Troubleshooting Records .................................................... B-1
Glossary .......................................................................................................... I
II

About This Manual
Purpose
This manual describes how to handle the faults that might occur on the ZXUN iCX(MSCS).
Intended Audience
This manual is intended for the maintenance engineers.
Prerequisite Skill and Knowledge

To use this manual effectively, you should have a general understanding of mobile
telecommunications technology. Familiarity with the following is helpful:
l The ZXUN iCX(MSCS) product and its various components
l The ZXUN iCX(MSCS) OMM operations
l Purpose and usage of each functions of OMM, especially configuration management,
signaling trace, alarm management, and failure observation
What Is in This Manual

This manual contains the following chapters:
Chapter Summary
1, Introduction to Describes the failure classification, requirements for maintenance

Troubleshooting personnel, fault handling procedures, troubleshooting methods
and tools, requirements and precautions for troubleshooting,
methods for collecting fault information and the ways to get ZTE
technical supports.
2, Hardware Faults Describes how to handle hardware faults, including board, power
supply and fan.
3, NMS Faults Describes how to handle NMS faults.
4, Service Faults Describes how to handle service faults.
Appendix A, Cautions for Lists the hazard operations and cautions.

Routine Maintenance
Appendix B, Troubleshooting Gives a sheet of troubleshooting records that is recommended to

Records fill in after troubleshooting a fault.
Related Documentation
The following documentation is related to this manual:
l ZXUN iCX(MSCS) MSC Server Alarm Management Operation Guide

l ZXUN iCX(MSCS) MSC Server Alarm Handling

l ZXUN iCX(MSCS) MSC Server Performance Management Operation Guide
l ZXUN iCX(MSCS) MSC Server Data Configuration Guide (Basic Data)
l ZXUN iCX(MSCS) MSC Server Data Configuration Guide (Service Data)
l ZXUN iCX(MSCS) MSC Server General Operation Guide
l ZXUN iCX(MSCS) MSC Server Parts Replacement Guide
l ZXUN iCX(MSCS) MSC Server Hardware Description
l ZXUN iCX(MSCS) MSC Server Software Installation Guide
l ZXUN iCX(MSCS) MSC Server Trace Management Operation Guide
FCC Compliance Statement

This device complies with part 15 of the FCC Rules. Operation is subject to the following
two conditions.
1. This device may not cause harmful interference.
2. This device must accept any interference received, including interference that may
cause undesired operation.
Changes or modifications not expressly approved by the party responsible for compliance
could void the user's authority to operate the equipment.
Conventions
This manual uses the following typographical conventions:
Typeface Meaning
Italics Variables in commands. It may also refer to other related manuals and
documents.
Bold Menus, menu options, function names, input fields, option button names, check
boxes, drop-down lists, dialog box names, window names, parameters, and
commands.
Constant width Text that you type, program codes, filenames, directory names, and function
names.
[] Optional parameters.
{} Mandatory parameters.
| Separates individual parameter in series of parameters.
Note: Provides additional information about a certain topic.
II

Chapter 1
Introduction to
Troubleshooting
Table of Contents
Fault Classification .....................................................................................................1-1
Requirements for Maintenance Personnel ..................................................................1-2
Common Fault Handling Procedure............................................................................1-2
Emergency Fault Handling Procedure ........................................................................1-5
Common Troubleshooting Tools .................................................................................1-6
Common Troubleshooting Methods ............................................................................1-7
1.1 Fault Classification

The faults that may occur on the ZXUN iCX(MSCS) can be divided into the following types:
l Hardware faults
The common hardware faults that may occur on the ZXUN iCX(MSCS) can be divided
into the following types:
à 2.1 Board Faults
à 2.2 Power Supply Faults
à 2.3 Fan Faults
l NMS faults
The common faults that may occur when you are using the Network Management
System (NMS) can be divided into the following types:
à 3.1 Communication Faults
à 3.2 Authority Faults
à 3.3 Database Faults
à 3.4 Performance Management Faults
à 3.5 Alarm Management Faults
l Service faults
The common service faults that may occur on the ZXUN iCX(MSCS) can be divided
into the following types:
à 4.1 Basic Call Service Faults
à 4.2 SMS Faults
1-1

ZXUN iCX(MSCS) Troubleshooting Guide
à 4.3 Location Update Service Faults

This manual gives the guidelines for locating and handling the faults mentioned above.
1.2 Requirements for Maintenance Personnel

Knowledge
l Knowing telecommunication theories, including mobile telecommunication theories
and soft switch theories.
l Familiar with signaling protocols including SS7, MAP, SIGTRAN and H.248.
l Familiar with computer and network science and applications, including LAN, TCP/IP,
client/server structure, WEB application and database.
l Familiar with Windows and CGSL.
l Knowing the ZXUN iCX(MSCS) principles, system structure, interfaces, protocols,
service flows and working modes.
l Knowing ATCA and its principles.
l Knowing what service interruptions may cause subscriber complaints.
Equipment Operation Skills

l Familiar with the IP planning and networking of the ZXUN iCX(MSCS).
l Knowing dangerous operations on the ZXUN iCX(MSCS), that is, what operations
may cause interruption of all or some services, or equipment damage. For more
details about the dangerous operations, refer to Appendix “A Cautions for Routine
Maintenance”.
l Professional in the common operations on the servers and databases of the ZXUN
iCX(MSCS).
l Professional in daily operations on the ZXUN iCX(MSCS).
l Professional in backing up and restoring the databases of the ZXUN iCX(MSCS).
Instrument-Using Skills
Skilled in using various instruments and meters. The most commonly used tools include
multi-meters and SS7 signaling analyzers.
1.3 Common Fault Handling Procedure

Fault Handling Procedure
To locate a fault, you should start with the work flows and the system composition, analyze
and judge according to the symptoms, and then exclude normal modules and find out the
problem module. With the problem module located, you can handle the fault with any
method that you know or that is described in this manual.
For the common procedure for handling a fault, see the following figure.
1-2

Chapter 1 Introduction to Troubleshooting
Figure 1-1 Common Fault Handling Procedure
Fault Handling Operations

During a fault handling procedure, the maintenance persons should do the following in
turn.
1. Know the situations: In case of a fault, do simple service tests to know the situation of
the fault.
2. Collect fault information: Collect or record detailed information about the fault,
including the symptom, alarms and working information displayed on the NMS,
operations you have made to handle this fault, and other information that you can
collect with the maintenance tools (such as signaling trace, failure observation, and
performance management).
1-3

3. Classify the fault: Based on the symptom and the information that you have collected,
analyze the fault initially and classify it.
4. Locate the fault: Locate the fault and find out the possible causes by analyzing the
work flows and NEs.
5. Remove the fault: Handle according to the fault location and possible causes.
6. Record the fault handling information: Record details about the fault handling,
including the symptom and the operations. Such information may be a helpful
reference for handling similar faults. You can design a table to record the details or
use the recommended table, refer to Appendix “B Troubleshooting Records”.
Precautions
When you are troubleshooting, note the following:
l Rules and regulations for fault handling and trace should be made to be manifest
to all maintenance persons. Only authorized and relevant persons are allowed to
participate the troubleshooting, thus to prevent misoperations from causing worse
faults.
l The maintenance persons should do by following the instructions in the manuals of
the ZXUN iCX(MSCS). Before touching any hardware device, you should wear an
antistatic wrist strap, so as to avoid hazards and accidents caused by human factors.
l Back up service data and system running settings. Make a detailed record about
the fault symptom, versions, and configuration changes and operations that you have
done. Collect also other data about the fault. All the data may be used for analyzing
and removing the fault.
l Trace and record in detail the handling of each fault. For a fault that may last for days,
detailed records about the changes of shift should be made, so that the responsibilities
can be clear.
l Handle every fault promptly. In case of any fault that you cannot remove, contact ZTE
promptly.
In any of the following situations, you should contact ZTE for technical support.
à Emergency faults, for example, all services or some services interrupted
à Faults that you cannot remove with the methods described in this manual
à Faults that you cannot remove with you own knowledge
à Faults that you cannot remove by referring to the cases of successfully removing
similar faults
l Past a list of contacts of ZTE in a conspicuous place, and remember to confirm and
update the contacts frequently.
ZTE Global Customer Support Center (GCSC) provides 7×24 technical supports.
Contacts of GCSC:
à Tel: +86-755-26771900
à Fax: +86-755-26770801
1-4

à URL: http://ensupport.zte.com.cn
à E-mail: support@zte.com.cn
l When you are contacting ZTE for technical support, you may be required to provide
the following information:
à Detailed symptom about the fault, including time, place and events
à The calling number and the called number, and the time when the call was made
à Relevant alarms, performance statistical data, signaling trace result, and failure
observation result
à Operations that you have done after the fault occurred
à The way to remotely log in to the system and the telephone numbers of persons
for contact
1.4 Emergency Fault Handling Procedure

Definition of Emergency Fault
An emergency fault is a fault that causes equipment failures (such as equipment
breakdown) in providing basic services, system being unable to work for more than
30 minutes or human safety hazards, or other problems that should be solved in
emergencies.
Emergency faults of the ZXUN iCX(MSCS) can be classified into five types:
l Failures in providing basic services to any causes such as equipment breakdown,
environmental or human factors. The faults are not removed after the preliminary
handling and need to be handled immediately.
l The rate of successful service handling and service provisioning operations declines
5% or more, or many subscribers or important customers complaint about the
interruption or poor quality of services.
l Unable to maintain devices with the NMS.
l Influences on basic services provisioning by other equipment
l Hazards to human safety caused by use of the product
Principles for Emergency Fault Handling

Once emergency faults on the equipment are reported or found, to restore the system as
soon as possible, you should do according to the procedure, see Figure 1-2. Meanwhile,
contact the local ZTE office for technical support.
System faults may comprise complete or partial power failures, network failures, database
faults and other faults. You can do troubleshooting in these key aspects. When you have
confirmed that power and communication is OK, you can use the alarm management
function of the NMS to locate the node where problem possibly lies in.
1-5

Procedure of Emergency Fault Handling

Once an emergency fault happens, contact and organize relevant persons and departs
to do emergency fault handling, and meanwhile report via a telephone to the authority
department and supervisors immediately. (After recovery, submit a written failure report
to the authority departments and supervisors.) Besides, organize relevant technicians,
units and equipment suppliers to figure out causes so that lessons may be drawn from it
and effective measures may be taken in the subsequent operations to avoid its happening
again. After recovery, fill in a Detailed Emergency fault Record carefully and archive it.
For the procedure of handling emergency faults, see the following figure.
Figure 1-2 Procedure of Emergency Fault Handling
1.5 Common Troubleshooting Tools

To do troubleshooting, you may use the following software and hardware tools:
1-6

l OMM
The OMM provides signaling trace, failure observation, performance management,
alarm management and log management tools. These tools may be useful for you to
locate and remove a fault.
à With the alarm management function, you can query and view the alarms that
have occurred due to a fault. Many alarms can directly tell you the causes of a
fault.
à Signaling trace and failure observation can help you to locate a current service
fault.
à With the log management function, you can see the operation logs of the OMM .
You can know what operations have been done before an alarm, then you can
analyze whether this fault has something to do with these operations.
à You may be unable to directly locate a fault with the performance statistics
function. But analysis on performance statistical data is an very good assistant.
l Ethereal network packet sniffing tool
Ethereal is a free program. You can use it to capture and save wanted network
interfacing data and convert the data into a readable format.
l Instruments and meters
Commonly used instruments and meters including (flat-head and cross-head)
screwdrivers, a voltage tester, a signalling analyzer, network cable pliers and a
multi-meter should be ready at hand.
1.6 Common Troubleshooting Methods

Common Methods
l Comparison
Comparison refers to comparing the faulty configuration data or devices with correct
configuration data or devices and performing analysis to find the differences and locate
the fault.
l Replacement
Replacement refers to replacing a faulty hardware component (such as a board or a
cable) with a normal one (it would be better if the model is the same). If the fault is
removed, then the replaced component or board is faulty.
l Minimum system
A minimum system refers to a system with some hardware devices removed and only
the minimum number of necessary hardware devices kept. If the system with the
minimum hardware configuration (that is, the minimum system) is faulty, then the fault
locates in a remaining necessary device. If the minimum system is not faulty, then
the fault locates in a removed device. Install the removed devices one by one while
1-7

checking for the device that when it is installed the system becomes faulty again. This
device may be faulty, and you can replace it.
Note:
When you do troubleshooting with the minimum system method, be cautious that this
method might influence the services.
You can use the minimum system method for troubleshooting during the
commissioning procedure, and this method is not recommended if the equipment is
in commercial use.
l Signaling analysis
With the signaling trace function of the OMM, you can obtain and analyze the signaling.
By referring to the standard signaling, you can use the signaling analysis method to
locate a fault in inter-office signaling coordination.
l Failure code analysis
Failure code analysis is helpful for you to locate a service fault on the local office. The
system provides a failure code and reasons for every service failure.
l Performance statistics
Performance statistics is helpful for you to find out the when a fault occurred, what
services and what devices are influenced.
Recommendations
l To troubleshoot a hardware fault, you can check the indicators also when you are
using a troubleshooting method such as comparison or replacement.
l To troubleshoot a software or service fault, you are recommended to use the
maintenance tools of the OMM, while analyzing logs and other data.
l It may be difficult to tell whether a fault is a hardware fault or a software fault. In this
case, you need to use multiple troubleshooting methods. Therefore, proficiency in all
these troubleshooting methods helps you to quickly remove a fault.
1-8

Chapter 2
Hardware Faults
Types of Hardware Faults
For the types of common hardware faults that may occur on the ZXUN iCX(MSCS), refer
to the following table.
Table 2-1 Types of Hardware Faults
Type of Hardware Faults Reference
Board faults 2.1 Board Faults
Power supply faults 2.2 Power Supply Faults
Fan faults 2.3 Fan Faults
Introduction to Board Indicators

To check whether a hardware device is faulty, you can just see the indicators. For the
meaning of each indicator on a device, refer to the following table.
Table 2-2 Board Indicators
Indicator Meaning Color Description
OOS Service state Red Off: in service

indicator Solid on: out of service
H/S Hot-swap state Blue Off: The device is working, and you should
indicator not unplug the board.
Solid on: The device is not working, and
you can swap the board.
Flashing: The device is being activated or
deactivated.
OK Healthy state Red/green Off:

indicator l For the NCMM: No power supply.
l For other boards: The board has not
started up.
Flashing in green: Running normally.
On in red: Abnormal. There may be an
alarm.
HOST Running state/alarm Red/green Flashing in green: Running normally.

indicator Flashing in red: Abnormal. There may be
alarms. The higher frequency, the higher
alarm level.
2-1

Indicator Meaning Color Description
HD1/HD2 Hard disk state Red/green Flashing in green: The disk is reading or
indicator writing.
On in red: Indicates that the hard disk fails
or is off position.
RUN Running state Green Flashing in green: Running normally.

indicator of the fan Off: out of service
ALM Alarm indicator of Red Flashing in red: Abnormal. There may be

the fan alarms.
Off: Running normally.
Table of Contents
Board Faults...............................................................................................................2-2
Power Supply Faults ..................................................................................................2-8
Fan Faults ................................................................................................................2-12
2.1 Board Faults

Common board faults may have the following symptoms:
Symptom Reference
The H/S indicator of a board is solid on 2.1.1 The H/S Indicator Is Solid On
The H/S indicator of a board is flashing 2.1.2 The H/S Indicator Is Flashing
The OK indicator of a board is off 2.1.3 The OK Indicator Is Off
The OK indicator of a board is on in red 2.1.4 The OK Indicator Is Solid On in Red
The HOST indicator of a board is flashing in red 2.1.5 The HOST Indicator Is Flashing in Red
The HD1/HD2 indicator of a board is solid on in 2.1.6 The HD1/HD2 Indicator Is Solid On in Red
red
Failure in board powering on/off when operated 2.1.7 Failure in Board Powering On/Off When
with the extractors Operated with the Extractors
A board is abnormally powered off with the H/S 2.1.8 A Board Is Abnormally Powered Off with
indicator solid on the H/S Indicator Solid On
2.1.1 The H/S Indicator Is Solid On

Symptom
The H/S indicator (blue) of a GPBB0, a GPBX1, a SLB, a SWBB0/SWBB1, or a NCMM is
solid on.
2-2

Chapter 2 Hardware Faults
Analysis
When a board is working properly, the hot-swap (H/S) indicator is off. If the H/S indicator
is solid on, the board is not in the working state.
Do the following to locate the fault:
l Check whether the extractors are secured.
l Check the contact between the front board and the rear board.
Handling
You are recommended to handle this fault in the following procedure.
1. Check whether the extractors of the board are secured, that is, whether the sliders of
the extractors are fastened.
l Yes→Go to Step 3.
l No→Go to Step 2.
2. Secure the extractors, and check whether the H/S indicator flashes for a little while
and turns off.
l Yes→End.
3. Open the extractors and then fastened them, and check whether the H/S indicator
flashes for a little while and turns off.
l Yes→End.
Note:
If the H/S indicator is always flashing, handle by referring to “2.1.2 The H/S Indicator
Is Flashing”.
4. Contact ZTE for help.
Confirmation
The H/S indicator of a board is off.
2.1.2 The H/S Indicator Is Flashing

Symptom
The hot-swap state (H/S) indicator (blue) of a GPBB0, a GPBX1, a SLB, a SWBB0/SWBB1,
or a NCMM is flashing for a long time.
2-3

Analysis
When a board is working, the H/S indicator is off. If the indicator is flashing, the board has
not started up.
Handling
1. Open the extractors of the board and then fastened them, and check whether the
indicator becomes solid off after flashing for little while.
l Yes→End.
2. Open the extractors of the NCMM to do active/standby switchover and then secure
the extractors after the switchover. See whether the indicator becomes solid off after
flashing for little while.
l Yes→End.
3. Open the extractors of the board and then fastened them, and check whether the
indicator becomes solid off after flashing for little while.
l Yes→End.
Confirmation
The H/S indicator is solid off.
2.1.3 The OK Indicator Is Off

Symptom
The health state (OK) indicator of a GPBB0, a GPBX1, a SLB, a SWBB0/SWBB1, or a
NCDM is off.
Analysis
If the OK indicator is off, the board has not started up successfully.
Handling
1. Check whether the IPMC version of this board is being updated.
2. Wait for the update to complete, and then check whether the OK indicator is flashing
in green.
l Yes→End.
2-4

3. Check whether this board is being powered off or on.
4. Wait for the board to start up, and then check whether the OK indicator is flashing in
green.
l Yes→End.
5. Reboot the board and check whether the OK indicator is flashing in green.
l Yes→End.
Confirmation
The OK indicator of the board is flashing in green.
2.1.4 The OK Indicator Is Solid On in Red

Symptom
The health state (OK) indicator of a GPBB0, a GPBX1, a SLB, a SWBB0/SWBB1, or a
NCDM is on in red.
Analysis
If the OK indicator is on in red, there are alarms for this board.
Handling
1. In the Fault Management tab page of the Local Maintenance Terminal, check
whether there are alarms for this board.
l No→End.
2. Handle and remove the alarms by referring to the recommended measures attached
to the alarm information, and see whether this fault is removed.
l Yes→End.
Confirmation
The OK indicator is flashing in green.
2-5

2.1.5 The HOST Indicator Is Flashing in Red

Symptom
The running/alarm (HOST) indicator of a GPBB0, a GPBX1, a SLB, a SWBB0/SWBB1, or
a NCMM is flashing in red.
Analysis
If the HOST indicator flashes in red, the board may be working abnormally or have alarms.
The higher frequency that the indictor is flashing at, the higher level the alarms are.
Handling
whether there are alarms for this board.
l No→End.
to the alarm information, and see whether this fault is removed.
l Yes→End.
Confirmation
The HOST indicator is flashing in green.
2.1.6 The HD1/HD2 Indicator Is Solid On in Red

Symptom
The hard disk (HD1/HD2) indicator of a GPBB0/GPBX1 board is solid on in red.
Analysis
If the HD1/HD2 indicator of a GPBB0/GPBX1 board is solid on in red, the corresponding
hard disk is faulty.
Handling
1. Replace the faulty hard disk, and then check whether the HD1/HD2 indicator becomes
normal.
l Yes→End.
l No→Go to Step 2
2-6

Confirmation
The HD1/HD2 indicator flashes in green when the board is reading data from or writing
data into the hard disk. At other times, the indicator is off.
2.1.7 Failure in Board Powering On/Off When Operated with the

Extractors
Symptom
After the extractors are closed and the board is powered on, the HOST indicators flashes.
Analysis
If a board cannot successfully get power on after the extractors are closed or cannot
successfully get power off after the extractors are opened, the sliders of the extractors
may be damaged.
Handling
1. Check whether the micro switch of the board is damaged.

2. Replace the board, and check whether the HOST indicator is flashing in green.
l Yes→End.
Confirmation
The board can successfully power on and the HOST indicator shows no alarm.
2.1.8 A Board Is Abnormally Powered Off with the H/S Indicator

Solid On
Symptom
When a board is working, the H/S indicator is lit, the power supply turns off. After the board
is removed and plugged in again, the board can be powered on and can work normally.
Analysis
The fault may be caused by the internal control mechanism. If the temperature is over
high, the self-protection process starts and gets the power off.
2-7

Handling
whether there are temperature related alarms.
2. Check the equipment room temperature and see whether the environmental
temperature is higher than the required temperature.
3. Lower down the environmental temperature, and check whether the board is powered
on and the H/S indicator is off.
l Yes→End.
Confirmation
The board is powered on and the H/S indicator is off.
2.2 Power Supply Faults

Common power supply faults may have the following symptoms:
Symptom Reference
The H/S indicator is solid on 2.2.1 The H/S Indicator Is Solid On
The H/S indicator is flashing 2.2.2 The H/S Indicator Is Flashing
All the devices or half of the devices in a shelf 2.2.3 All the Devices or Half of the Devices in a
are powered off Shelf Are Powered Off
Half of the devices in a shelf are powered off 2.2.4 Half of the Devices in a Shelf Are Powered
after a power supply module is removed Off After a PEM Is Removed
Some boards are suddenly powered off 2.2.5 Some Boards Are Suddenly Powered Off

Symptom
The hot-swap (H/S) indicator (blue) of a power supply module is solid on.
Analysis
When a power supply module is working properly, the H/S indicator is off. If the H/S
indicator is solid on, the power supply module is not in the working state.
2-8

Handling
1. Press the sunk H/S key ( ) on the panel with a sharp tool (such as a forceps), and
check whether the H/S indicator flashes and then turns off.
l Yes→End.
2. Remove and then install the power supply module, and check whether the H/S
indicator flashes and then turns off.
l Yes→End.
Note:
Is Flashing”.
Confirmation
The H/S indicator of the power supply module is off.

Symptom
The hot-swap state (H/S) indicator (blue) of a power supply module is flashing for a long
time.
Analysis
When a power supply module is working, the H/S indicator is off. If the indicator is flashing,
the power supply module has not started up.
Handling
l Yes→End.
2. Open the extractors of the NCMM to do active/standby switchover and then secure the
extractors after the switchover. See whether the indicator of the power supply module
flashes and then turns off.
2-9

l Yes→End.
l Yes→End.
Confirmation
2.2.3 All the Devices or Half of the Devices in a Shelf Are Powered
Off
Symptom
All the devices or half of the devices in a shelf are suddenly powered off.
Analysis
The external power supply may be failed.
Handling
1. Check whether the UPS is normal.
l No→Check and restore the power supply.
2. Check whether any power state indicator (-48/-60 VA or -48/-60 VB) on the panel of
the power supply module is on in red.
3. The switch of the power supply module is not turned on.
4. Turn on the two switches, and check whether the fault is resolved.
l Yes→End.
5. Check the -48 V/-60 V and RTN terminals of the PEM with a multimeter and see
whether the voltages are normal and whether the polarities are correct.
6. According to the engineering documentation, check whether the switches or lines that
provide power to the cabinet have a problem.
2-10

7. Remove the problem in the switches or lines, and then check whether the fault is
resolved.
l Yes→End.
Confirmation
The devices can be successfully powered on and operate properly.
2.2.4 Half of the Devices in a Shelf Are Powered Off After a PEM Is
Removed
Symptom
Half of the devices in a shelf are powered off after a power supply module is removed.
Analysis
The two power supply modules work in hot-backup redundancy mode. If half of the devices
in a shelf are powered off after a power supply module is removed, the wiring may be
incorrect.
Handling
1. Check the engineering documentation and see whether the wiring of power supply
lines is correct.
2. Correct the wiring according to the engineering documentation, and check whether the
fault is resolved.
l Yes→End.
Confirmation
The devices in the cabinet work properly after a power supply module is removed.
2.2.5 Some Boards Are Suddenly Powered Off

Symptom
Some boards are suddenly powered off when the system is running. For example, When
a board is inserted, some other boards in this shelf are powered off.
2-11

Analysis
This fault may be due to the following causes:
l The voltage and power of the power supply to this shelf are abnormal.
l The contacts of power supply lines are of poor quality.
Handling
1. Check whether voltage and power of the power supply to this shelf are high enough.
l No→Correct the power supply problem.
2. Check whether the contacts of the power supply lines are in good conditions. There
should be no heating spots and no loose, especially the crimp connections in the
distribution box.
3. Correct the wiring by referring to the engineering documentation, and check whether
the fault is resolved.
l Yes→End.
Confirmation
The boards in the shelf are powered on, and the devices functions properly.
2.3 Fan Faults

Common fan faults may have the following symptoms:
Symptom Reference
The H/S indicator is solid on 2.3.1 The H/S Indicator Is Solid On
The H/S indicator is flashing 2.3.2 The H/S Indicator Is Flashing
The RUN indicator is flashing in green but the 2.3.3 The RUN Indicator Is Flashing in Green but
fans always run in the full speed the Fans always Run in the Full Speed
The RUN indicator is off but the fans always run 2.3.4 The RUN Indicator Is Off but the Fans
in the full speed always Run in the Full Speed

Symptom
The hot-swap (H/S) indicator (blue) of a fan module is solid on.
2-12

Analysis
When a fan module is working properly, the H/S indicator is off. If the H/S indicator is solid
on, the fan module is not in the working state.
Handling
1. Press the sunk hot-swap key ( ) on the panel with a sharp tool (such as a forceps),
and check whether the H/S indicator flashes and then turns off.
l Yes→End.
2. Remove and then install the fan module, and check whether the H/S indicator flashes
and then turns off.
l Yes→End.
Note:
Is Flashing”.
Confirmation
The H/S indicator of the fan module is off.

Symptom
The hot-swap (H/S) indicator (blue) of a fan module is flashing for a long time.
Analysis
When a fan module is working, the H/S indicator is off. If the H/S indicator is flashing, the
fan module has not started up.
Handling
l Yes→End.
2-13

2. Open the extractors of the NCMM to do active/standby switchover and then secure
the extractors after the switchover. See whether the H/S indicator of the fan module
flashes and then turns off.
l Yes→End.
3. Press the sunk H/S key on the panel with a sharp tool (such as a forceps), and check
whether the H/S indicator flashes and then turns off.
l Yes→End.
Confirmation
2.3.3 The RUN Indicator Is Flashing in Green but the Fans always
Run in the Full Speed
Symptom
The RUN indicator on the panel of the fan module is normal (flashing in green), but the
speed of fans cannot be adjusted and the fans always run in the full speed.
Analysis
The system may have an over-temperature alarm or the parts of the fan are damaged.
Handling
whether there is any over-temperature alarm.
to the alarm information, and then check whether the fault is resolved.
3. Check whether the parts of the fan are damaged.
4. Replace the fan, and then check whether the fault is resolved.
l Yes→End.
2-14

Confirmation
The fans are working properly and the rotate speed can be adjusted.
2.3.4 The RUN Indicator Is Off but the Fans always Run in the Full
Speed
Symptom
The fans always run in the full speed, but the RUN indicator is not flashing in green.
Analysis
It is caused by a failure of NCMM to control the fan module.
Handling
1. Remove the fan module and plug it in again. Check whether the fault is resolved.
l Yes→End.
Confirmation
The fans are working properly and the rotate speed can be adjusted.
2-15

This page intentionally left blank.
2-16

Chapter 3
NMS Faults
For the types of common NMS faults that may occur on the ZXUN iCX(MSCS), refer to the
following table.
Table 3-1 Types of NMS Faults
Type of NMS Faults Reference
Communication faults 3.1 Communication Faults
Authority faults 3.2 Authority Faults
Database faults 3.3 Database Faults
Performance management faults 3.4 Performance Management Faults
Alarm management faults 3.5 Alarm Management Faults
Table of Contents
Communication Faults................................................................................................3-1
Authority Faults ..........................................................................................................3-4
Database Faults .........................................................................................................3-5
Performance Management Faults...............................................................................3-7
Alarm Management Faults .........................................................................................3-9
3.1 Communication Faults

Common communication faults may have the following symptoms:
Symptom Reference
Timeout when the LMT transfers data to the OMP 3.1.1 Timeout When the LMT Transfers Data to
the OMP
IE cannot access the OMM server 3.1.2 IE Cannot Access the OMM Server
3.1.1 Timeout When the LMT Transfers Data to the OMP

Symptom
The system returns a response notifying operation time out after the SYNA:STYPE="AL
L" command is run in the Terminal tab page of the Local Maintenance Terminal (LMT).
3-1

Analysis
The cable connection between the OMM server and the OMP is faulty or the link is not
created.
Handling
1. On the OMM server, ping the IP address of the OMP and see whether the
communication is through.
2. Check whether the OMM server and the OMP are connected through network cable
of a GPI1 card.
3. Check whether any network cable connector is loose between the OMM server and
the OMP.
4. Secure the cable connection, and see whether the fault is resolved.
5. In the Terminal tab page of the Local Maintenance Terminal, run the SYNA:STY
PE="ALL" command to transfer the data to the OMP, and see whether the system
returns a timeout error.
l No→End.
6. In the Terminal tab page of the Local Maintenance Terminal, run the CHECK OMC
LINK command (for showing the links between the OMM server and the OMP) and
see whether the returned result is null (no link).
7. In the Terminal tab page of the Local Maintenance Terminal, run the SET OMP:RE
LINK="YES" command to recreate the link, and see whether the fault is resolved.
l Yes→End.
Confirmation
Run the SYNA:STYPE="ALL" command to transfer the data to the OMP. The operation
should succeed.
3-2

Chapter 3 NMS Faults
3.1.2 IE Cannot Access the OMM Server

Symptom
The OMM interface does not appear when IE is used to access the OMM server.
Analysis
The possible causes of this fault may be the following:
l The network communication between the OMM server and the computer where the
IE is broken.
l The OMM server processes have not started up successfully.
l The HTTP service on the OMM server is not started.
Handling
1. On the computer of the IE, ping the IP address of the OMM server and see whether
the communication is through.
2. Check whether the network communication between the OMM server and the
computer is normal and whether the network cable is loose.
3. Secure the cable connection and see whether the fault is resolved.
4. Access the OMM server with the IE, and check whether the OMM interface appears.
l Yes→End.
5. In the Terminal window of the OMM server, run the ps –ef|grep service to check
whether all OMM server processes have started up.
6. Restart the OMM server and see whether the fault is resolved.
l Yes→End.
8. In the Terminal window of the OMM server, run the service httpd status command to
check whether the HTTP service has started up.
3-3

9. In the Terminal window of the OMM server, run the service httpd start command to
start the HTTP service and see whether the fault is resolved.
l Yes→End.
Confirmation
The OMM interface does not appear when Internet Explorer is used to access the OMM
server.
3.2 Authority Faults

Symptom
After a user is created, the following problems occur:
l The user cannot be used to log in to the LMT.
l The user cannot operate with assigned permissions.
Analysis
The permissions may be not correctly assigned to this user, for example:
l The user is not valid.

l The user is not related roles.
l No proper permission is assigned to the role of the user.
Handling
1. In the Terminal tab page of the Local Maintenance Terminal, run the SHOW USER
to check whether this user is valid.
2. In the Terminal tab page of the Local Maintenance Terminal, run the SET USE
R command with the Valid User parameter set to Yes. Check whether the fault is
resolved.
l Yes→End.
3. Assign permissions to this user (if you are allowed to do so). In the Terminal tab page
of the Local Maintenance Terminal, run the SHOW USER ROLE to check whether
related roles are assigned to this user.
3-4

4. Assign permissions to this user (if you are allowed to do so). In the Terminal tab page
of the Local Maintenance Terminal, run the SHOW ROLE to check whether this role
is valid.
5. Assign permissions to this user (if you are allowed to do so). In the Terminal tab
page of the Local Maintenance Terminal, run the ADD ROLE to add the role. Check
whether the fault is resolved.
l Yes→End.
6. In the Terminal tab page of the Local Maintenance Terminal, run the ADD USER
ROLE to assign the related roles to the user. Check whether the fault is resolved.
l Yes→End.
7. Assign permissions to this role (if you have the permission to do so). In the Terminal
tab page of the Local Maintenance Terminal, run the SHOW ROLE CMDSET to
check related operation permissions are assigned to the role.
8. In the Terminal tab page of the Local Maintenance Terminal, run the ADD ROLE CM
DSET to assign operation permissions to the role. Check whether the fault is resolved.
l Yes→End.
Confirmation
The user can operate the LMT with the assigned permissions.
3.3 Database Faults

Common Database faults may have the following symptoms:
Symptom Reference
Failure to connect the database 3.3.1 Failure to Connect the Database
Export function of an LMT is unavailable 3.3.2 Export Function of an LMT Is Unavailable
3.3.1 Failure to Connect the Database

Symptom
The system reports Db not connect after a command is executed in the OMM interface.
The command execution still fails after the OMM server is restarted.
3-5

Analysis
Possible causes:
l The Firebird service on the OMM server has not started up.
l The Firebird database is faulty and inaccessible.
Handling
1. In the Terminal window of the OMM server, run the service firebird status command
and check whether the database service has started up.
2. Run the service firebird start command to start the database service, and check
whether the fault is resolved.
l Yes→End.
3. Enter the /opt/firebird/bin directory and run the following command to check
whether the database is accessible.
./isql -u SYSDBA -p SYSDBA /home/ngomm_data/ngomm_db/OfficeID/FIREBIRD_C

M.FDB
In this command, the parameter after “-u” is the Firebird user name, the parameter
after “-p” is the user password, and “OfficeID” in “OfficeID” refers to the office ID. You
should set the command parameters according to the actual condition.
l No→The Firebird database is damaged. Go to Step 4.
4. Reinstall the database, and check whether the fault is resolved.
For how to install the Firebird database, refer to Section “Configuring System Services”
of ZXUN iCX(MSCS) MSC Server Software Installation Guide.
l Yes→End.
Confirmation
Commands can be executed on the OMM interface.
3.3.2 Export Function of an LMT Is Unavailable

Symptom
After an LMT logs in to the OMM server, export function is not available, such as the export
function in the Performance Management tab page.
3-6

Analysis
The export function of the Local Maintenance Terminal needs to enable ActiveX
controls option, but the default setting of the IE is disable.
Handling
1. Open the IE on the Local Maintenance Terminal.
2. Enter the address of the OMM server in the Address Bar of the IE. The prompt is
displayed, see Figure 3-1.
Figure 3-1 ActiveX Controls Dialog Box
Note:
Just you visit the OMM server address for the first time, the dialog box is displayed.
3. Click the dialog box and select Add-on Disabled > Run Add-on shortcut menu.
4. Click the Running button to install “Zte OMM Assistant ActiveX Control Module” in the
message box.
5. After it is installed, and log in to the OMM server again, and check whether export
function is available.
l Yes→End.
Confirmation
After a Local Maintenance Terminal logs in to the OMM server, export function is
available.
3.4 Performance Management Faults

Common performance management faults may have the following symptoms:
Symptom Reference
Time delay in performance data reporting 3.4.1 Time Delay in Performance Data Reporting
Cannot report performance data 3.4.2 Cannot Report Performance Data
3-7

3.4.1 Time Delay in Performance Data Reporting

Symptom
A performance measurement task is created and activated. After 10 data collection
granularities, the performance data is reported but the collected performance data is less
than 10 granularities.
Analysis
The time is different between the OMP module and the OMM server.
Handling
1. In the Terminal tab page of the Local Maintenance Terminal, run the SHOW TIME
command to check whether the OMP time is the same as the OMM server.
2. In the Terminal tab page of the Local Maintenance Terminal, run the SYNC NTPTIM
E to synchronize the OMP time with the OMM server or run the UPD TIME command
to change the OMP time. Check whether the fault is resolved.
l Yes→End.
Confirmation
Wait for some collection granularities later after a measurement task is activated, check the
collected performance data. The collected performance data should be same as collection
granularities.
3.4.2 Cannot Report Performance Data

Symptom
A performance measurement task is created and activated. After 10 data collection
granularities, no performance data is reported.
Analysis
Possible causes:
l The link between the OMP and the OMM server is abnormal.
l The time difference between the OMP and the OMM server is too big.
l The processes of the OMM server is abnormal.
3-8

Handling
1. In the Terminal tab page of the Local Maintenance Terminal, run the SHOW TIME
command to check whether the OMP time is the same as the OMM server.
2. In the Terminal tab page of the Local Maintenance Terminal, run the SYNC NTPTIM
E to synchronize the OMP time with the OMM server or run the UPD TIME command
to change the OMP time. Check whether the fault is resolved.
l Yes→End.
3. In the Terminal tab page of the Local Maintenance Terminal, run the CHECK OMC
LINK to check whether the link between the OMP and the OMM server is created.
The returned result should not be null.
4. In the Terminal tab page of the Local Maintenance Terminal, run the SET OMP:RE
LINK="YES" to create a link again. Check whether the fault is resolved.
l Yes→End.
5. In the Terminal window of the OMM server, run the ps –ef|grep service to check
whether all OMM server processes have started up.
6. Restart the OMM server and see whether the fault is resolved.
7. Reboot the board and check whether this problem is removed.
l Yes→End.
Confirmation
Create and activate a performance measurement task. After 10 data collection
granularities, performance data should be reported.
3.5 Alarm Management Faults

Common alarm management faults may have the following symptoms:
Symptom Reference
Alarm filtering rules do not work 3.5.1 Alarm Filtering Rules Do Not Work
3-9

Symptom Reference
Alarms cannot reach a well-connected alarm 3.5.2 Alarms Cannot Reach a Well-Connected Alarm
box Box
3.5.1 Alarm Filtering Rules Do Not Work

Symptom
An alarm filtering rule is created and activated in the Fault Management tab page of the
Local Maintenance Terminal, but this rule does not work and the current alarms are not
filtered.
Analysis
An alarm filtering rule can filter only the alarms that occur after the rule is created and
activated.
Handling
1. In the Fault Management tab page of the Local Maintenance Terminal, manually
filter the alarms and then check whether the same alarms are filtered. The same
alarms may also occur but should be filtered automatically.
l Yes→End.
Confirmation
After an alarm filtering rule is created and activated, newly generated alarms that meet the
conditions of the rule are automatically filtered.
3.5.2 Alarms Cannot Reach a Well-Connected Alarm Box

Symptom
An alarm box is configured in the Fault Management tab page of the Local Maintenance
Terminal and it is connected normal, but alarm information cannot be sent to the alarm
box.
Analysis
An alarm box is valid for only the alarms that occur after the alarm box is configured.
3-10

Handling
1. Check whether new alarms are reported to the alarm box.
l Yes→End.
Confirmation
New alarms generated after an alarm box is configured are sent to the alarm box.
3-11

3-12

Chapter 4
Service Faults
For the types of common service faults that may occur on the ZXUN iCX(MSCS), refer to
the following table.
Table 4-1 Types of Service Faults
Type of Service Faults Reference
Basic Call Service Faults 4.1 Basic Call Service Faults
SMS Faults 4.2 SMS Faults
Location Update Service Faults 4.3 Location Update Service Faults
Table of Contents
Basic Call Service Faults............................................................................................4-1
SMS Faults ................................................................................................................4-6
Location Update Service Faults ................................................................................4-12
4.1 Basic Call Service Faults

Background
Call service is a basic service implemented by the ZXUN iCX(MSCS). The basic call
services fall into the Mobile Originated (MO) and the Mobile Terminated (MT). For the
complete flow when one mobile subscriber calls another mobile subscriber, see Figure
4-1.
4-1

Figure 4-1 Basic Call Flow
Descriptions of the Flow:
1. The calling subscriber MS1 dials the phone number of the MS2, informing MSCS1
through the RNS/BSS system.
2. Analyzing the phone number of the called subscriber MS2, the MSCS1 finds the
MS2-homed HLR and sends the routing request to the HLR.
3. HLR queries the current location information of MS2, and finds that MS2 is served by
MSCS2/VLR2. HLR sends a request for the routing information to MSCS2/VLR2.
4. MSCS2/VLR2 distributes the routing information (that is, the roaming number MSRN)
and sends the MSRN to HLR.
5. HLR sends the MSRN to the calling party MSCS1.
6. MSCS1 establishes the call with MSCS2 according to the MSRN.
7. MSCS2/VLR2 sends the paging message to the called subscriber MS2.
8. MSCS2/VLR2 receives the message that MS2 can access.
9. MSCS2 sends a request for establishing the bearer terminal to the MGW2.
10. MSCS2 sends an APM signal to MSCS1, carrying the call bearer address information.
11. MSCS1 sends the bearer address information of the call and the request for
establishing the bearer terminal to MGW1.
12. The bearer between MGW1 and MGW2 is successfully established. Meanwhile, the
call circuit between MGW and MS is successfully set up.
13. MSCS1 sends the ring back tone for the calling subscriber. If the called subscriber
hooks off at this time, the called office will sends the answer signal to the calling office.
The MSCS of each party will respectively ask MGW to connect the speech channel.
The calling subscriber and the called subscriber can communicate normally.
A basic call procedure involves such NEs as RNS/BSS, MSCS, MGW and HLR.
In CS domain, the CN-related NEs are MSCS and MGW. The MSCS is responsible for
call connection and control. The MGW is responsible for establishing and maintaining the
bearing channel, and providing various resources.
4-2

Chapter 4 Service Faults
Fault Phenomena
l After successfully updating the location, the subscriber cannot originate a call
normally.
l After successfully updating the location, the subscriber cannot receive a call normally.
l There are a great number of call losses in the Failure Observation tab page.
Fault Handling
For the flow of troubleshooting the call service fault, see Figure 4-2.
Figure 4-2 Handling the Call Service Fault

1. Open the signal trace system, and specify the subscriber signaling to be traced and
then trace the signaling. Use the subscriber terminal with call fault to perform the call
service, and analyze the traced signaling result.
2. If no signaling can be successfully traced, check whether the terminal is faulty, and
use another terminal to make a call attempt.
4-3

3. If some signaling can be successfully traced, analyze them. If the fault is possibly
caused by the local office by analyzing the signaling, open the Failure Observation
tab page to check the failure reason in the system, and locate the internal fault.
4. If it is the interconnection fault, contact the opposite-end office to handle it corporately.
5. If the call failure happens to many subscribers, check whether the links between the
local office and important office directions are normal, and whether all SMPs in local
office are normal.
4.1.1 Analyzing Instance 1 (Call Failure for Inconsistent

Interconnection Parameters)
Fault Phenomena
During the inter-working test between the ZXUN iCX(MSCS) and another vendor’s UMTS,
when the opposite-end MS dials the local-end MS of ZTE, the opposite-end HLR requests
the roaming number from the ZXUN iCX(MSCS). After receiving the IAM message from
the opposite-end MSCS, the ZXUN iCX(MSCS) immediately sends a REL message to the
opposite-end MSCS. The call is released.
Fault Analysis and Location

Location flow:
1. Check the MM message in the ZXUN iCX(MSCS), and find that the message call has
been established with normal signaling flow.
2. Check the IAM field in the ZXUN iCX(MSCS) and find that the sent roaming number is
1585310034FF, which contains FF (two Fs). F is the flag of number effective ending.
3. The roaming number contained in the SRI ack message is 1585310034F in the ZXUN
iCX(MSCS), the F should be added by the opposite-end MSCS.
Analysis flow:
l The roaming number of the opposite end is 11 digits, but the ZXUN iCX(MSCS) is 10
digits. The opposite-end MSCS receives 1585310034F from the ZXUN iCX(MSCS).
After receiving 11 digits, the opposite-end MSCS adds F to the 12th digit. That’s why
there are two Fs in the IAM message.
l This number is sent to the ZXUN iCX(MSCS) for the number analysis. Because F
is the end flag, the ZXUN iCX(MSCS) analyzes 1585310034F during the number
analysis, but there is no matched number analysis data. Therefore, the system
considers it is a vacant number, and sends a REL to the opposite-end office, causing
the service failure.
Fault Handling
4-4

1. In the Terminal tab page , run the ADD MRNPFX command to configure the roaming
number of the local end to 11 digits.
2. Test again and the dial-up test is successful. The problem is solved.
4.1.2 Analyzing Instance 2 (Roaming Subscribers Cannot Receive

Calls)
Fault Phenomena
One subscriber homed to HLR of location A is roaming in an office of location B. The mobile
phone can normally implement the location update, make a call, and receive a call from a
subscriber homed to local network of the same carrier. He/she fails to receive a call from
a subscriber homed to other mobile or fixed network.

Location flow:
Check the GT data about this subscriber on the tandem office and local gateway office,
and find that the GT translation incorrectly points to local other HLR.
Analysis flow:
l When a subscriber homed to foreign mobile or fixed network calls to this subscriber,
the signaling should be transferred to local gateway office. The local gateway office
initiates a routing information request to HLR. Therefore, probably, there is a problem
with the local gateway office, which did not perform this subscriber’s GT analysis
pointing to HLR.
l The local MSCS end office is connected to the HLR in an associated mode, while the
local gateway office is connected to the HLR in a quasi-associated mode, which is
forwarded by the local tandem office. Therefore, the problem may be located in the
local tandem office, which did not perform the GT analysis pointing to HLR.
Fault Handling
1. In the Terminal tab page of the tandem office, run the SET GT command to modify
that the GT translation points to local HLR.
2. On the Terminal tab page of the local gateway office, run the SET GT command to
modify that the GT translation points to local HLR.
4-5

4.1.3 Analyzing Instance 3 (Defaulting Subscriber Fails to Call

Special Service Numbers)
Fault Phenomena
A subscriber fails to dial the special service number in case of overdue, and hears "Sorry,
your telephone is defaulting". But normally, the special service calls can be connected by
subscribers even if they are defaulting.

1. Initially judge that this defaulting subscriber triggers the IN service, which may restrict
the calls on the SCP.
2. Query the VLR information of this subscriber, and find that it has subscribed to the IN
service that implements the defaulting control.
3. Trace the GSMSSF signaling message to the SCP, and find that the SCP delivers a
Play Announcement message to notify the SSP to play tones after sending an RRBE
message to the SSP. And then, the calling subscriber hears avoice notification for
owning fee. As a result, the call fails.
Fault Handling
1. Set these number to "Not trigger caller IN service" at the SSP side.
2. In the Terminal tab page of the ZXUN iCX(MSCS), run the SET TPDNAL command
to enable NTC option for these numbers.
SET TPDNAL:ENTR=1,DIGIT="110",ENOPT="NTC";
4.2 SMS Faults

Background
SMS message faults refer to that MS screen prompts the sending failure when MS sends
a short message or that the destination MS does not received a short message.
SMS processing flow is divided into three parts.

l MO short message processing flow (MO)
l MT short message processing flow (MT)
l Alert message transmission.
MO flow and MT flow are most frequently used.

l For the MO SM processing flow, see Figure 4-3.
4-6

Figure 4-3 SM MO Processing Flow

1. MS sends a CM_SER_REQ message to the MSCS/VLR. MSCS/VLR returns a
CM_SER_RSP message or a CM_SER_RJT message to the MS. If MSCS/VLR
has sent service response message, it will authenticate MS.
2. MS sends a short message CPDATA to MSCS/VLR. After receiving message,
MSCS/VLR checks whether the short message service is restricted or barred by
the supplementary service. If SM processing is not barred, MSCS/VLR will send
it to IWMSC (inter-working MSC), the interface with SC.
3. After SC receives short message sent from MS, it will return the processing result
to the MS. After receiving Delivery Report message, MSCS/VLR will send an
acknowledgement message to the MS.
l For the MT SM processing flow, see Figure 4-4.
4-7

Figure 4-4 MT SM Processing Flow

1. If short message that SC receives is destined for a mobile subscriber, SC will
send it to the ingress GMSCS that short message belongs to. Ingress GMSCS
then sends a route request message to HLR. And HLR finds out MSCS/VLR
route and then sends routing information to the GMSCS. According to the routing
information, GMSCS finds out MSCS/VLR to which the MS belongs, and sends
short message to MSCS/VLR.
2. After receiving Forward_short_message, the MSCS/VLR will directly returns a
delivery failure message to the GMSCS if VLR does not have any data related to
MS, and will send a PAGE message to MS if the VLR contains data related to MS.
After MS returns a PAGE-RSP message, SC will perform authentication to verify
whether the subscriber is legal.
3. After subscriber is proved legal, MSCS/VLR will send a short message to MS.
After receiving short message, MS will return processing result to MSCS/VLR:
If processing is successful, returned result is an acknowledgement message,
otherwise, it is the cause of the failure. If MS has no enough space to store short
message, it will inform the SC of storage capacity shortage in the failure cause.
4. After receiving acknowledgement message from MS, MSCS/VLR will convert
the message into MAP signaling and sends it to GMSCS, and then GMSCS will
forward it to SC.
Fault Phenomena
l The MO SM always fails.
4-8

l The MT SM fails.
Fault Handling
For the flow of handling the SMS fault, see Figure 4-5.
Figure 4-5 Handling SMS Faults

1. The MS has its own setting problems, such as the set SMC number is wrong.
2. Check whether the data configuration is incorrect. Chiefly check the SCCP data and
the SM-related security variables.
3. Contact the HLR corresponding personnel to check the subscriber subscription data
on the HLR to see whether the subscriber's SMS settings is correct.
4. Contact the RNC/BSC corresponding personnel to check whether the RNC/BSC data
is normal.
4-9

4.2.1 Analyzing Instance 1 (MS/UE Cannot Receive the Short

Message)
Fault Phenomena
When the ZXUN iCX(MSCS) serves as the IW/GMSCS to interconnect with the SC, the
MS/UE cannot receive the short message.

1. After tracing the signaling on the IW/GMSCS, find that the MO flow of the MS/UE is
successful, but there is no MT flow. Therefore, the SC possibly does not deliver the
short message.
2. Through tracing the signaling at the SC side, find that the SC has sent an MT message,
but the IW/GMSCS does not receive this message.
3. Continue to analyze this message, and find that the office number of the message to
be sent is SC, but the IW/GMSCS adopts another office number. Therefore, based on
this office number and module number, the SC cannot correctly send the MT message
to the IW/GMSCS.
Fault Handling
1. After confirming there are errors in the SC configuration, add the office number of the
IW/GMSCS in the SC configuration.
2. Besides adding the office number of the IW/GMSCS, it is required to modify the SMP
module of the corresponding IW/GMSCS in the SC system configuration.
3. Test again and find that MS/UE can receive the short message. The problem is solved.
4.2.2 Analyzing Instance 2 (Office P Cannot Receive A Short

Message from Extranet W)
Fault Phenomena
The short message interworking between Office P and Extranet W is special.
l Extranet W sends an incoming message to the PGMSCS of Office P through its own
WGMSCS, and transfers to HLR through STP of Office P. The PVMSCS of Office P
returns a response message to the WVMSCS of Extranet W, and transfers to SMSCS
through STP of Extranet W.
l An outgoing short message is vice versa.
Office P cannot receive a short message from Extranet W after the route of incoming short
message of Office P is cut over from old PGMSCS to the new PGMSCS.
For the networking diagram, see Figure 4-6.
4-10

Figure 4-6 Networking Diagram
Fault Handling
1. On the HLR and PVMSCS side, trace the number of the incoming short message
simultaneously. The routing request message of this incoming short message can
be traced on the HLR, and the HLR has returned a routing response message.
However, the short message delivery message cannot be traced on the PVMSCS,
which indicates that the PGMSCS can normally forward the SCCP-layer message to
the HLR. The problem probably occurs when the PGMSCS forwards a message to
the PVMSCS or STP forwards a message to the PVMSCS.
2. Check the GT configuration of PGMSCS, and find that all the GT data are configured,
including that of PVMSCS, HLR, extranet WGMSCS, and SMC. Particularly, the
configured PGMSCS GT is directly connected to the PVMSCS, not transferred
through STP. Therefore, something is wrong when the PGMSCS forwards an
SCCP-layer message to the PVMSCS.
3. Check the PVMSCS-related configuration on the PGMSCS, and find that the SCCP
protocol between PGMSCS and PVMSCS is not configured in the SIO-locate-AS
configuration.
Both PGMSCS and PVMSCS adopt the full-IP networking mode, all the MAP and
SCCP messages are forwarded through STP. Before cutover, this problem did not
happen because the short message of the extranet did not use the SCCP forwarding.
After cutover, this problem happened because the short message of the extranet
4-11

needed to use the SCCP forwarding but the SCCP between PGMSCS and PVMSCS
is not configured.
Fault Handling
1. In the Terminal tab page of the PGMSCS office, run the ADD SIOLOCAS command
to add SIO-locate-AS configuration between PGMSCS and PVMSCS and service
indication selected SCCP.
2. Test again and find that MS can receive the short message. The problem is solved.
Summary
When the IP networking is adopted, the inter-office signaling configuration of the adjacent
office should be selected in the SIO-locate-AS configuration as required.
4.3 Location Update Service Faults

Background
Location update includes:
l General location update
l Periodic location update
l IMSI attachment/detachment
For the signaling flow of a general location update, see Figure 4-7.
Figure 4-7 General Location Update Flow
4-12

1. MS sends a location update request to MSCS/VLR. After receiving location update

request from MS, MSCS/VLR checks correctness of subscriber data and judges
location update category, to determine subsequent operations.
2. According to some conditions, MSCS/VLR determines whether to send location update
request to HLR. In general, this operation is originated in the following cases:
l Subscriber starts the system for the first time.
l Subscriber roams across the MSCS/VLR system.
l Subscriber data is incorrect or inconsistent with that in the HLR due to VLR reset
or individual reasons.
3. MSCS/VLR receives location canceling request from HLR. According to the subscriber
identity IMSI in the parameters, MSCS/VLR deletes this record from subscriber data
to release subscriber’s TMSI.
4. No matter whether subscriber has ever been registered in VLR, MSCS/VLR will return
location canceling acknowledgement to HLR and close dialogue.
5. MSCS/VLR receives an Activate_Trace_Mode request from HLR and directly returns
the Activate_Trace_Mode acknowledgement to HLR. MSCS sets a subscriber tracing
flag in the corresponding Data Area (DA), to implement real-time subscriber tracing.
6. MSCS/VLR sends a location update request to HLR, which causes HLR to originate
subscriber data insertion operation, to transmit subscriber data from HLR to VLR.
7. MSCS/VLR acknowledges that all subscriber data from HLR is correct and returns
acknowledgement primitive to HLR.
8. MSCS/VLR receives Forward_Check_SS_Indication request from HLR, unnecessary
for acknowledgement.
9. At the end of location update processing, HLR will return acknowledgement primitive
to MSCS/VLR.
10. After location update originated by MSCS/VLR to MSCS is completed, MSCS/VLR will
return acknowledgement message to MS.
The signaling flows of the periodic location update and the IMSI attachment/detachment
are similar to that of the general location update. Therefore, their faults can be handled
based on the processing method of the general location update fault. In the location update
flow, MSCS/VLR chiefly implements the access and authentication of subscribers, HLR
provides the subscription data of subscribers, and MS originates the location-update/IMSI
attachment request.
In the location update flow, MGW implements the SG function, switching the signaling of
the access side to the MSCS without any processing.
Fault Phenomena
l The subscriber cannot log in the network after updating the location.
l The subscriber cannot be found in the VLR after the MS is powered on.
l The location update failure results in the call fault.
Fault Handling
4-13

1. Analyzing the subscriber with fault, make sure whether it is the single subscriber fault
or multi-subscriber fault, and then open the Signaling Trace tool to trace the faulty
subscriber.
2. For the single subscriber fault, trace the subscriber signaling. If no signaling is suc-
cessfully traced, the fault possibly lies in the subscriber side. In this way, contact the
subscriber for handling it. If the signaling of the subscriber location update request is
successfully traced, but the request is refused by HLR, check whether this subscriber
has subscribed the roaming restriction data.
3. For the multi-subscriber fault, analyze the information of faulty subscribers. The
analysis method is shown as follows.
l Analyze whether the subscribers are located in the same location area or cell. If
they are, it indicates that the fault is related to the geographical position where
the subscribers are located. Ask service personnel at the wireless side to check
whether the radio-side equipment at that location area is normal. Check whether
the radio-related configuration data at the local office has been modified and is
correct.
l Check whether the subscriber IMSI is regular, whether the SMP corresponding to
the IMSI load sharing of the faulty subscriber is normal, and whether the service
fault is caused by the SMP fault.
l Check whether these subscribers are home to the same HLR. If they are, check
the mobile number analysis, GT and MTP configuration of the faulty subscribers.
Check the HLR status. Contact the HLR maintenance personnel to perform the
signaling trace, and corporately solve the problem.
l If the joint location update fails, check whether the Gs interface between the SGSN
and the MSCS/VLR is normal.
4.3.1 Analyzing Instance 1 (Call Failure for Setting the Periodic

Location Update)
Fault Phenomena
When making a general voice call, the calling party hears the announcement saying that
“The called party is powered off”, but actually the called MS is always on. After the called
MS is restarted, the call is normal. Such case has happened for many times.

Location flow:
Check the VLR system data configuration. At the CN side, the period of general location
update is 45 minutes by default (this duration is: VLR location update protection duration
+ periodic location update duration), however, at the RNC side, the duration is set as 60
minutes.
Analysis flow:
4-14

l At the beginning, the duration of general location update at the CN side is 45 minutes,
less than the 60 minutes set at the RNC side. When the location update timer at the
CN side expires, but the subscriber does not perform the periodic location update and
is in inactive status, VLR marks this subscriber with “IMSI detach”, which causes this
subscriber unable to be paged.
l After the subscriber is restarted after power-off, location update is performed again,
and VLR marks the subscriber with “ATACH”. Therefore, this subscriber can be paged,
and may make a call normally.
Fault Handling
1. At the CN side, set the periodic location update duration as 60 minutes, which is
specified in the VLR system parameters. So, the general location update duration
at the CN side is 75 minutes (the periodic location-update duration is 60 minutes, and
the VLR location-update protective duration is 15 minutes), larger than that at the RNC
side (60 minutes).
Summary
In the OMM system of CN, the default duration of periodic location update is 30 minutes,
and the duration of the VLR location update protection is 15 minutes. Adjust these
parameters according to actual conditions to make the duration of general location update
at the CN side larger than that at the RNC side.
The duration of periodic location update originated by UE is determined by the related
configuration of RNC. RNC data configuration must be consistent with that of CN.
4.3.2 Analyzing Instance 2 (Location Update Fails to Complete)

Fault Phenomena
After the MSCS is interconnected with the RNC. During performing the location update
test, the engineer finds that the SCCP message from the RNC is not processed by the
SCCP module on the MSCS. By decoding the SCCP message from the RNC, the engineer
finds that it contains subscriber’s Location Updating Request message.

1. The engineer already sees the SCCP message on the MSCS, indicating that the MGW
already transfers the message of the RNC. The problem is probably related with the
attribute configuration of the SMP modules on the MSCS. The engineer checks the
attributes of the SMP, and finds that the SMP, VMSC and MSCBASECMP are selected
on the MSCS. In addition, the CMP_MGW is already selected for the SMP attributes on
the MGW. Therefore, this problem may be not related with the module configuration.
4-15

2. Open the Signaling Trace tab page and perform the location update test. The
engineer finds that the MSCS prompts an RNC_id ERR message after received a
SCCP message, and then directly returns a Disconnect message. It is related with
the RNC-related data configured on the MSCS. The problem probably lies in the
inconsistency between the data configured on the MSCS and that on the RNC.
Fault Handling
1. In the Terminal tab page , run the SHOW RNCOFC command to check the RNC ID
configuration on the MSCS and find that the RNC ID is set to 2, while the RNC side is
set to 1.
2. In the Terminal tab page , run the SET RNCOFC command to modify the RNC ID to
1 on the MSCS.
3. Perform the location update test. The SCCP normally processes the subscriber
message from RNC. The problem is solved.
Summary
The RNC ID configuration on the MSCS is inconsistent from that on the RNC, which causes
MSCS not to process subscriber information reported by RNC. As a result, the location
update fails.
4-16

Appendix A
Cautions for Routine
Maintenance
Table A-1 Cautions for Routine Maintenance
Operation Caution Possible Results Caused by Misoperation
Board-related Do not unplug online l The data synchronization between active and standby boards
operations active board. requires a certain period when the standby board is operating
properly. When the active board is unplugged online, the latest
data on it cannot be completely and automatically backed up
to the standby board, although the system will switch over
the boards automatically. It will easily cause statistical errors
and data loss.
l When the standby board operate abnormally, this operation
will interrupt all the service processing of corresponding
module, resulting in partial or global service block in the
system.
Do not press the RST l A board will be implemented the hardware reset by force in
button on the board case of the RST button on the board panel being pressed.
panel at will. Only qualified maintenance personnel can press it when
serious fault occurs in the system.
l Pressing the RST button on the active board panel will reset
the active board. The consequence is the same as that of
"Unplugging online active board".
Do not plug and unplug The electrostatic on human bodies may cause great damage to
boards without wearing the components. If you plug or unplug boards without wearing
antistatic wrist straps. antistatic wrist straps, the boards may suffer electrostatic damage
easily. As a result, boards are damaged or run unstably.
Cable-related Do not plug or unplug This operation probably will interrupt services and the
operations network cables inside communication between the front-end and back-end systems.
the cabinet at will.
Power-supply- Do not operate the power You can operate various power switches by following operation
related operations switches on the cabinet procedure only when serious fault occurs in the system, or
at will. during network upgrade, network expansion, and component
replacement. This operation will stop system running and interrupt
services.
A-1

A-2

Appendix B
Troubleshooting Records
Device name Device serial number
Time of fault occurrence Time of fault removal

(year-month-day-hour) (year-month-day-hour)
Type of fault:
Origin of fault:
Symptom:
Handling:
Conclusion:
Signature of the person on duty: Signature of the maintenance person:
B-1

B-2

Glossary
ATCA
- Advanced Telecommunications Computing Architecture
BSS
- Base Station Subsystem
CGSL
- Carrier Grade Server Linux
CS
- Circuit Switched
GT
- GSM/TD Dual Mode
H.248
- ITU-T Rec. H.248 Gateway Control Protocol
HD
- Hard disk
HLR
- Home Location Register
HTTP
- Hypertext Transfer Protocol
IMSI
- International Mobile Subscriber Identity
IP
- Internet Protocol
IPMC
- Intelligent Platform Management Controller
LAN
- Local Area Network
LMT
- Local Maintenance Terminal
MAP
- Mobile Application Part
MO
- Mobile Originated
MS
- Mobile Station

MSRN
- Mobile Subscriber Roaming Number
MT
- Mobile Terminated
MTP
- Message Transfer Part
NCDM
- New Chassis Data Module
NCMM
- New Chassis Management Module
NMS
- Network element Management System
OMM
- Operation & Maintenance Module
PEM
- Power Entry Module
RNS
- Radio Network Subsystem
RTN
- Return
SCP
- Service Control Point
SG
- Signaling Gateway
SGSN
- Serving GPRS Support Node
SIGTRAN
- Signalling Transport
SMS
- Short Message Service
SS7
- Signaling System No. 7
SSP
- Service Switching Point
TCP
- Transmission Control Protocol
UMTS
- Universal Mobile Telecommunication System
II

Glossary
UPS
- Uninterruptible Power Supply
III

SJ-20130416095742-019-ZXUN ICX (MSCS) (V4.13.10) MSC Server Troubleshooting Guide - 504807

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

SJ-20130416095742-019-ZXUN ICX (MSCS) (V4.13.10) MSC Server Troubleshooting Guide - 504807

Uploaded by

Copyright:

Available Formats

ZXUN iCX(MSCS)

Revision No. Revision Date Revision Reason

R1.0 2013-05-31 First edition

Serial Number: SJ-20130416095742-019

Publishing Date: 2013-05-31 (R1.0)

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

Chapter 2 Hardware Faults ........................................................................ 2-1

Chapter 3 NMS Faults ................................................................................ 3-1

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

Chapter 4 Service Faults............................................................................ 4-1

Appendix A Cautions for Routine Maintenance ..................................... A-1

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

Prerequisite Skill and Knowledge

What Is in This Manual

1, Introduction to Describes the failure classification, requirements for maintenance

3, NMS Faults Describes how to handle NMS faults.

4, Service Faults Describes how to handle service faults.

Appendix A, Cautions for Lists the hazard operations and cautions.

Appendix B, Troubleshooting Gives a sheet of troubleshooting records that is recommended to

l ZXUN iCX(MSCS) MSC Server Alarm Management Operation Guide

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

FCC Compliance Statement

| Separates individual parameter in series of parameters.

Note: Provides additional information about a certain topic.

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

1.1 Fault Classification

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

à 4.3 Location Update Service Faults

1.2 Requirements for Maintenance Personnel

Equipment Operation Skills

1.3 Common Fault Handling Procedure

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

Figure 1-1 Common Fault Handling Procedure

Fault Handling Operations

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

1.4 Emergency Fault Handling Procedure

Principles for Emergency Fault Handling

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

Procedure of Emergency Fault Handling

Figure 1-2 Procedure of Emergency Fault Handling

1.5 Common Troubleshooting Tools

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

1.6 Common Troubleshooting Methods

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

Table 2-1 Types of Hardware Faults

Type of Hardware Faults Reference

Board faults 2.1 Board Faults

Power supply faults 2.2 Power Supply Faults

Fan faults 2.3 Fan Faults

Introduction to Board Indicators

Table 2-2 Board Indicators

Indicator Meaning Color Description

OOS Service state Red Off: in service

OK Healthy state Red/green Off:

HOST Running state/alarm Red/green Flashing in green: Running normally.

SJ-20130416095742-019|2013-05-31 (R1.0) ZTE Proprietary and Confidential

Indicator Meaning Color Description

RUN Running state Green Flashing in green: Running normally.

ALM Alarm indicator of Red Flashing in red: Abnormal. There may be

2.1 Board Faults

The OK indicator of a board is off 2.1.3 The OK Indicator Is Off