IMT - iBTS Maintenance Tool: Training IBK

IMT iBTS Maintenance Tool
Training IBK
Nicolas Renot - IceBreaker
October 2012
COPYRIGHT 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.

ALCATEL-LUCENT INTERNAL PROPRIETARY USE PURSUANT TO COMPANY INSTRUCTION
Main goal
Domain & Level definition
IMT contents
imt_zap.txt
lec_[1..6].txt & flr_[1..6].txt
slave_<slotID>.txt
Decoding tools
L1 error code
ICAR
PQ2 tools
2
IMT Main goal

The IMT main purpose is to collect information that will allow users to monitor, collect
traces, understand and diagnose the root cause of problems at a site without requiring
technicians and equipment to be brought on-site.
The IMT provides the following functionality:
IBTS System snapshot: provide a complete view of hardware & software configuration
of the iBTS
IBTS ZAP tool: collect all necessary information for problem investigation, in a human
readable form
IBTS Trace manager: select the traces to activate
3
Main goal
IMT contents
imt_zap.txt
lec_[1..6].txt & flr_[1..6].txt
slave_<slotID>.txt
Decoding tools
L1 error code
ICAR
PQ2 tools
4
IMT - Domain
Possible domains are:
[SYS]: System resources - CPU, OS, I&C parameters, Trap info, SW errors;
[RADIO]: OAM config (local cell, sector, VCC, ), TRM channelizers,
Rlink/Flink, SYN, RF cabling;
[CALLP]: CallP tables, Load balancing information;
[OAM]: HW snapshot, OAM physical objects;
[CNX_INT_L1/L2/L3, CNX_EXT_L1/L2/L3]: internal/external,
HSSPC, PCM, TC, Ethernet, ATMswitch, ATMconn, IP, IPOA, sMsg, SSCOP;
[MCTL] [BD_MCTL]: MCTL counters + Bottleneck detector
114020 iBTS
[CECTL] [BD_CECTL]: CECTL counters + Bottleneck detector
Bottleneck
[OC] [OCP]: OC/OCP statistics
Detector
[GLOBAL]: synthetic state of the iBTS
5
IMT - Level
Possible levels are:
[CONFIG]: static configuration information
[PERF]: all information captured by Performance Manager (Buffer
occupancy, CPU consumption, Longest task lock)
[FAULT]: returns Errors and Alarms detected by the system
[OPSTATE]: all dynamic information (depends on domain set)
[STATISTICS]: Counters/Statistics
[GLOBAL]: synthetic state of the iBTS
6
IMT - Section header

IMT output: Domains and levels are separated by a regular string
Format for iCCM or i/x/eCEM:
===== IMT DATA for Card: C, BoardId: S Domain: D , Level: L, By: E

Format for xCCM:
===== IMT DATA Domain: D, Level: L, SWel E

With,
C equal to CCM, iCCM, CEM, iCEM, xCEM, TRM, iTRM, hTRM, xTRM
S is the slot id of the card/board,
D is the Domain,
L is the Level,
E is the software element.
7
Main goal
IMT contents
imt_zap.txt
lec_[1..6].txt & flr_[1..6].txt
slave_<slotID>.txt
Decoding tools
L1 error code
ICAR
PQ2 tools
8
IMT contents
Current LEC
Current IMT zap
Current FLR
tar tvf tar tvf imt_2008-03-06T12h41.tar

-rw------- root/root
347 2008-03-06 13:41:37 imt_2008-03-06T12h41/conf.txt.gz
16919 2008-03-06 13:41:42 imt_2008-03-06T12h41/lec_current.txt.gz
16626 2008-03-06 13:41:49 imt_2008-03-06T12h41/imt_zap.txt.gz
291230 2008-03-06 13:41:59 imt_2008-03-06T12h41/flr_current.txt.gz
37830 2008-03-06 13:42:00 imt_2008-03-06T12h41/lec_1.txt.gz
152201 2008-03-06 13:42:05 imt_2008-03-06T12h41/flr_1.txt.gz
LEC history 1..6

36179 2008-03-06 13:42:06 imt_2008-03-06T12h41/lec_2.txt.gz
238030 2008-03-06 13:42:14 imt_2008-03-06T12h41/flr_2.txt.gz
36939 2008-03-06 13:42:15 imt_2008-03-06T12h41/lec_3.txt.gz
249752 2008-03-06 13:42:24 imt_2008-03-06T12h41/flr_3.txt.gz
38994 2008-03-06 13:42:25 imt_2008-03-06T12h41/lec_4.txt.gz
223365 2008-03-06 13:42:33 imt_2008-03-06T12h41/flr_4.txt.gz
40483 2008-03-06 13:42:33 imt_2008-03-06T12h41/lec_5.txt.gz
238780 2008-03-06 13:42:42 imt_2008-03-06T12h41/flr_5.txt.gz
40609 2008-03-06 13:42:43 imt_2008-03-06T12h41/lec_6.txt.gz
210916 2008-03-06 13:42:51 imt_2008-03-06T12h41/flr_6.txt.gz
FLR history 1..6
82907 2008-03-06 13:43:01 imt_2008-03-06T12h41/slave_8.txt.gz
116887 2008-03-06 13:43:23 imt_2008-03-06T12h41/slave_4.txt.gz
Note: If RE used are RRH or TRDU, corresponding logs will be

available in directories RRH
9
Slave slot-ID
IMT contents imt_zap.txt

1/2
imt_zap snapshot contains the complete view of hardware & software
configuration of the NodeB.
Basic checks: SOFTWARE ELEMENT EMOAM

Section ===== IMT DATA Domain: global, Level: global, SWel EMOAMAlso used to know Slave slot-ID
Global view of NodeB configuration. Command: /pltf_*/emo/Bts
slot-ID=LogicalSlot
Section ===== IMT DATA Domain: oam, Level: config, SWel EMOAM
(-= OAM configuration =-) contains all the MIM parameters (DLU or CFileDump)
Section ===== IMT DATA Domain: oam, Level: fault, SWel EMOAM
We can check when alarm raised @ OAM

if any slave is reset we can check when CCM-Slave link alarm is raised from CCM POV
- Section ===== IMT DATA Domain: radio, Level: config, SWel EMOAM
(-= OAM Radio Configuration =-) contains radio configuration, RF cabling, BBLink configuration
Basic checks: SOFTWARE ELEMENT IUBMGR

Section ===== IMT DATA Domain: radio, Level: config, SWel IUBMGR
to check which cell is setup and the cell configuration

Section ===== IMT DATA Domain: oam, Level: opstate, SWel IUBMGR
to check last RSI and Audit messages triggered from NodeB
10
IMT contents imt_zap.txt

2/2
Basic checks: SOFTWARE ELEMENT CCM CALLP
Section ===== IMT DATA Domain: global, Level: global, SWel CCM callP
(CAC NodeB R99 / H-BBU / E-BBU internal data) contains CAC current state to check nbr of
call per BBU & limitation
Basic checks: SOFTWARE ELEMENT REPROXY_SW

Section ===== IMT DATA Domain: oam, Level: opstate, SWel REPROXY_SW
Harware/Software configuration of Radio Equipment
LastResetReason of each Radio Equipment
Basic checks: SOFTWARE ELEMENT DWL_SW

Section ===== IMT DATA Domain: enum 13, Level: statistics, SWel DWL_SW
check state of CCM and slave boards active/passive partitions
Debug transaction management

Section BeginDebugTransMgrTimeStampDump and Slave time stamp table informations
to get CCM internal messages and messages sent to slave board
Note: Same section exist for iCCM

Search section Domain: global , Level: global, By: EMOAM
instead of ===== IMT DATA Domain: global, Level: global, SWel EMOAM
11
IMT contents lec_[1..6].txt & flr_[1..6].txt

1/3
Log rotation principle (lec & flr)
When a BTS is running, the current FLRs are present in flr_current.txt.
When the BTS traps and comes back up, this flr_current.txt is copied to flr_1.txt, flr_1.txt is
copied to flr_2.txt and so on.
Supposing, you captured the IMT after the BTS reset twice and you want the reason for the
reset before the current one, you will have to look at flr_2.txt.
LEC = Linux Error Collector

Tool used for CCM platform investigation
Section [lec_snap_print_file] name LEC_RESTART_CAUSE
contains information concerning the CCM last reset reason
CCM command: pltf_* > pltf > lec > cause 1 (1 is the history no.; the values being 1 6)
[lec_snap_print_file] name LEC_RESTART_CAUSE_V1.7432.1970-01-01_01h00m57.ppc.dump; label
16842752; pid 569; size:461 cpu:PPC
Restart cause:Uptime
: 0 days 01:00:58
Sequence num: 7432

Thread name : MsgSeq
Thread ID : 0x323294b0
Error number: 514
Reset Cause : 1339
Description :resetHardware() oamLecErr(trap=514 sev=1 itl=0) Restart due to timeout
connection
12

2/3
LEC = Linux Error Collector
[lec_snap_print_file] name
LEC_MONITOR_V1.7432.1970-0101_01h00m57.ppc.dump; label 16842766; pid 563;
size:1464 cpu:PPC
Tool used for CCM platform

investigation
LEC monitor data:

Sequence number:
Section [lec_snap_print_file] name

LEC_MONITOR
7432
CPU load:
to check the CCM CPU load when the trap has

occurred
Load
- User:
0.60% 0.60% 1.20%

97.59% 96.60% 97.59%
CPU last idle:

-
Max
1.60% 1.20% 2.00%
- System:
- Idle:
Min
Idle
Min
Max
96.99% 96.60% 97.59%
[lec_snap_print_file] name
LEC_BOS_MBX_V1.7432.1970-0101_01h00m57.ppc.dump; label 16842770; pid 569;
size:33600 cpu:PPC
IDENT
CURRCOUNT PEAKCOUNT LIMITCOUNT
COMMENT
Section [lec_snap_print_file] name

LEC_BOS_MBX
to check Mail Box usage when the trap has
occurred
13
0x626d007f
32 mSiteLAN
0x626d0080
36
256 roam_mailbox
0x626d0081
64 mbx_bdetect
0x626d0082
20 SYN_BAL
0x626d0083
32 HsspcMailbox
0x626d0084
30
0x626d0085
110 SEPEProxy_Mbx
0x626d0086
104 NBAP_CCMMBX

139 PMMailbox

3/3
FLR = FLight Recorder
Tool used for CCM CallP investigation
Section Filename: /tmp/flr/pltf.Init
to check the CCM initialization
Section Filename: "/tmp/flr/pltf.SwErrors
to check Errors and Warnings just before the trap has occurred
Section Filename: /tmp/flr/pltf.SwLogs
to check SwLogs just before the trap has occurred
Section Filename: /tmp/flr/pltf.Alarms
to check Alarms history (RAISED & CLEARED) from CCM point of view
Corresponding section for
flr_current.txt:
- Filename: /rmem/pltf.Init.trace
- Filename: /rmem/pltf.SwErrors
- Filename: /rmem/pltf.SwLogs.trace
- Filename: /rmem/pltf.Alarms.trace
14
IMT contents slave_[slotID].txt

1/4
Slave files are divided in 4 parts
1-2-3) Section ==== IMT DATA for Card: ECEM, BoardId: 2 Domain: SYS , Level: FAULT, By:
EEC_SW
contains 3 records for the 3 last CEM trap (partial or full CEM trap)
Partial reset info is
4) All other section concerns the current session (when the UA08.1.4/UA09.1:
IMT is collected)
only saved in RAM (4th part) due to
o ItfLogs & Dumps & Sysdump: Section =====731409
IMT DATA
fixfor Card: ECEM, BoardId: 2
Domain: CALLP , Level: FAULT, By: FLR_RELAY
o SwLogs & SwErrs: Section ===== IMT DATA for Card: ECEM, BoardId: 2 Domain: SYS
, Level: FAULT, By: FLIGHT RECORDE
o Init logs: Section ===== IMT DATA for Card: ECEM, BoardId: 2 Domain: SYS , Level:
GLOBAL, By: FLIGHT RECORDE
o Link ucuCcId <-> ucuCellIndex: Section ===== IMT DATA for Card: ECEM, BoardId:
2 Domain: CALLP , Level: CONFIG, By: cellDump
o HSU scheduler active + tunables: Section ===== IMT DATA for Card: ECEM,
BoardId: 2 Domain: CALLP , Level: CONFIG, By: MacEScheduler
o Alarms: Section ===== IMT DATA for Card: ECEM, BoardId: 2 Domain: OAM , Level:
FAULT, By: FLIGHT RECORDE
o KPI statistics: Section ===== IMT DATA for Card: ECEM, BoardId: 2 Domain: MCTL ,
Level: STATISTICS, By: KPI_COUNTERS, ===== IMT DATA for Card: ECEM, BoardId: 2
Domain: CECTL , Level: STATISTICS, By: CeCtl statisti, ===== IMT DATA for Card:
ECEM, BoardId: 2 Domain: OC , Level: STATISTICS, By: OneChip statis, ===== IMT
DATA for Card: ECEM, BoardId: 2 Domain: OCP , Level: STATISTICS, By: OneChipPlus st
15

2/4
3 records for the last 3 traps
Section ==== IMT DATA for Card: ECEM, BoardId: 2 Domain: SYS , Level: FAULT, By:
EEC_SW
contains 3 records for the 3 last CEM trap
(partial Number
or full CEM trap)
Sequence
Logs session N3
Date & Time

Sequence Number
Logs session N2
SwLogs
Date & Time
Sequence Number
Logs session N1
SwErrs
Alarms
Init
Date & Time
ItfLogs
Dumps
EcCectl
SL_ITF
16

3/4
Logs session N-1

Trap main info
Sequence Number: 362
Boot Number: 506
TRAP LOG
Type: Non-Fatal SW Error
Time Of Exception: 0 days 16:08:58.553559450 after boot
App: ECEM
Longest WD Pacify : 15.3252723297
0:00:15.3252723297 after boot
Last WD Pacify : 0.1016651755
Longest Task Lock : 0.015867386
15:44:03.3246909416 after boot
by (0x0E169748) tIMToolbox
Free Heap Size : 0x0806C970
Cpu Load : 7
errorCode: 999 (0x000003E7)
error: CE Failure raised: OCP0, FORCED RESET, FaultMgr
TASK
Task: tTrapLogCol
17

4/4
Logs session N-1
Flight Recorder RamBuffer SwLogs
Flight Recorder RamBuffer SwErrs
Contains Warnings L1 OC/OCP, PQ2 and PQ3
Online decoding tool: Script Modem Warnings (IBK tool)
Flight Recorder RamBuffer Alarms
Contains CEM internal alarms
Flight Recorder RamBuffer Init
Contains initialization messages until GENERIC_RSP_ACK BOARD_INIT
Flight Recorder RamBuffer ItfLogs
Contains all ITF messages
Online decoding tool: ITF1/2/4/8/9 decoding tool (PQ2 tool)
Flight Recorder RamBuffer Dumps
Contains CE Monitor records of all 5 CE devices (OneChip 0, 1, 2, 3 and OneChipPlus DSP)
Online decoding tool: Online CPIF/OC/OCP Control decoder (PQ2 tool)
Flight Recorder RamBuffer EcCectl
Filled in case of CECTL trap
Flight Recorder RamBuffer SL_ITF
Contains messages ITF-3 (CCM_OAM<->CEM_SLOAM) & ITF-4 (CEM_MCTL<->CEM_SLOAM)
18
Main goal
IMT contents
imt_zap.txt
lec_[1..6].txt & flr_[1..6].txt
slave_<slotID>.txt
Decoding tools
L1 error code
ICAR
PQ2 tools
19
Decoding tools L1 error code

Severity of Err/War from OC
ReportstoCECTLerrorsdetectedbyASICpartitionsaswellasfirmwareblocks(filteringisalso
implemented)
Severity:
0 -CE Fatal Error:CECTLtriggersCEDevicereset(andCEMresetwithoutBBUFaultManagement)
1 -Call Fatal Error:CECTLtriggersChannelFailuretogetthechanneldeleted(CallFatalerrors
reportedforCommonChannelsaretranslatedbyCECTLasCEFatal)
2 -Warning:appearinFLRSWerrs,loggedinCEMonitor
3 -Information:doesnotappearinFLRSWerrs(toavoidlogpollution),loggedinCEMonitor.
Type: SeeMessageCatalogue
https://wcdma-ll.app.alcatel-lucent.com/livelink/livelink.exe?func=ll&objId=45442447&objAction=browse&viewType=1
20

Severity of Err/War from OCP
0 - CE Fatal DSPcannotcontinueandneedstobereset.
1 - Call FatalThecall(ormorecorrectly,object,thatisHS-DSCHchannel,HS-DPCCHchannel,orCell)
identifiedbytheCECommunicationContextID,cannotcontinue,andneedstobeDeleted.
2 - WarningAnunexpectedconditionhasoccurred,howevertheCallcancontinue.
3 - E-DCH FatalFPGABorChashadafailuresothatallcallswithE-DCHcannotcontinue,andfuture
callswithE-DCHmaynotbeconfigured.
4 - CE_OR_EDCH_FATALTheDSPcantdetermineifthisisCEFatalorE-DCHfatal,theUCUCneeds
todosomethingtodetermineifitisCEorEDCHfatal.
5 - INFOTheCEissendingoutinformationthatitwouldliketogetloggedfordebuggingpurposes.
Type:
SeeMessageCatalogue
https://wcdma-ll.app.alcatel-lucent.com/livelink/livelink.exe?func=ll&objId=45442447&objAction=browse&viewType=1
21

CEM partial trap <=> OC/OCP handleCeFailure
FLR SWLogs/SwErrs should contain the detailed cause just before
handleCeFailure(), with same timestamp.
CEM OC1 partial trap:
2010-11-03 06:19:52:013 3B714051 WAR: 005 02:06:26, CECTL4 [2010-11-03 06:19:51.140]
UCU_CP
5318 !W WARNING UcuMgr(0x20111a4): rxCeErrorIndNormal(): ce=1: ce_id=1,
sev=0x0, type=0x2f05, sfn=0xfd0, event=0x0, occ=0x1..
2010-11-03 06:19:52:013 3B715741 WAR: 005 02:06:26, CECTL4 [2010-11-03 06:19:51.140]
UCU_CP
5319 !W WARNING CeMgrOc(0x20111a4): err_info 0x00:
008000800080008000FC00800080003E008F003E0002EEEE..
2010-11-03 06:19:52:013 3B7167C7 WAR: 005 02:06:26, CECTL4 [2010-11-03 06:19:51.140]
UCU_CP
5320 !W WARNING *** UBM WARNING ***
File:/vobs/ubtssw_ucu/ucu3/control/callprocess/cectl/src/CeMgrOc.cc line:745 ***..
2010-11-03 06:19:52:013 3B71789F WAR: 005 02:06:26, CECTL4 [2010-11-03 06:19:51.140]
UCU_CP
5321 !W WARNING CeMgrOc(0x20111a4): handleCeFailure(): ce_id=1; cause=STATUS
Access
to OC error code data base: http://lanIND..
bcf.fr.alcatel-lucent.com/build/doc/ocdsp1.0_firm_p8pre_integ/occ/doc/html
22

CEM partial traps <=> OC/OCP handleCeFailure
FLR SWLogs/SwErrs should contain the detailed cause just before
handleCeFailure(), with same timestamp.
CEM OCP/DSP partial trap:
1970-01-06 23:30:36:748 0EFE1FC9 WAR: 005 23:29:25, CECTL3 [1970-01-06 23:30:35.700]
UCU_CP
141986 !W WARNING 030000012, CellBbrOcp.cc-1050: CellBbrOcp(0x2192f58):
rxCeErrorIndNormal(): ceId=16, ucu_cc_id=0x360, ce_cc_id=0x7fbf, ucu_cell_idx=3,
1970-01-06 23:30:36:748 0EFE39F3 WAR: 005 23:29:25, CECTL3 [1970-01-06 23:30:35.700]
UCU_CP
141987 !W WARNING 030000012, CellBbrOcp.cc-1050: CellBbrOcp(0x2192f58):
rxCeErrorIndNormal(): ceId=4, sev=0, type=0x100, sfn=0xdc2, occ=1, thread=2,
1970-01-06 23:30:36:748 0EFE4E1D WAR: 005 23:29:25, CECTL3 [1970-01-06 23:30:35.700]
UCU_CP
141988 !W WARNING HwAl: CR 65889 NO ALARM trigger - no: 1 1033006 detect: 1
1970-01-06 23:30:36:748 0EFE6090 WAR: 005 23:29:25, CECTL3 [1970-01-06 23:30:35.700]
UCU_CP
File:/vobs/ubtssw_ucu/ucu3/ptfx/alarm_monitor/cectl/src/hw_alarm_mon.cc line:1681 ***
1970-01-06 23:30:36:748 0EFE73F1 WAR: 005 23:29:25, CECTL3 [1970-01-06 23:30:35.700]
UCU_CP
141990 !W WARNING HwAl: CR 65889 CPLD register - reg: 0x60cc 0xffcc 0x4000
0xe400
Access to DSP error code data base:
1970-01-06 23:30:36:748 0EFE866D WAR: 005 23:29:25, CECTL3 [1970-01-06 23:30:35.700]

http://www.iprc.lucent.com/~umtsl1/
UCU_CP
File:/vobs/ubtssw_ucu/ucu3/ptfx/alarm_monitor/cectl/src/hw_alarm_mon.cc
line:1687
***
Choose your release, then click on the link column Error
Codes
1970-01-06 23:30:36:748 0EFE99F6 WAR: 005 23:29:25, CECTL3 [1970-01-06 23:30:35.700]
UCU_CP
141992 !W WARNING 030000012, CeMgrOcPlus.cc-2556:
CeMgrOcPlus(0x218c3dc): handleCeFailure(): ce_id=4; cause=STATUS IND
23
Decoding tools ICAR

ICAR Icebreaker Centrallized Automatic Reports
Direct access to ICAR tools
- Script Modem Warnings
to list and decode L1 OC/OCP from SwErrs
MIM Parameters Check
to check if the NodeB configuration of the tester is inline
with the RAN Model recommendations
ICAR Main page

Main
24
Decoding tools PQ2 tools

Subscribe to RSS feed
RSS PQ2 tools
Direct access to PQ2 tools

CPIF decoding tool
to decode CPIF messages from ItfLogs
ITF 1/2/4/8/9 decoding tool
to decode ITF-1/2/4/8/9 from ItfLogs
ITF-1: Iub signalling link RNC <--> MCTL
ITF-2: NBAP messages CCM CallP <--> MCTL
ITF-4: fault reporting + configuration messages MCTL <--> SLOAM
ITF-8: configuration messages MCTL <--> CECTL
ITF-9: messages to setup/release AAL2/AAL5 connection MCTL <--> CECTL
- CPIF/OC decoding tool

to decode CE monitor (OC/OCP) messages from Dumps
25
26
Additional information
Doc reference
UMT/BTS/APP/006244 IMT Cookbook
Tools: Useful links
iBTS_Standard_Logs
ICAR main
ICAR Mim Parameters Check
ICAR Script Modem Warnings
RSS PQ2 tools
27
28

IMT - iBTS Maintenance Tool: Training IBK

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

IMT - iBTS Maintenance Tool: Training IBK

Uploaded by

Copyright:

Available Formats

IMT iBTS Maintenance Tool

COPYRIGHT 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.