Stand Alone OMS

Troubleshooting Notes
 For internal use

1. Version Control.............................................................................2

2. Introduction...................................................................................3

3. Installation and Commissioning....................................................4

3.1. Datafill....................................................................................................................4
3.2. Reconfiguring the RAID array................................................................................4
3.3. Reinstalling SOMS using a USB Stick...................................................................6
3.4. Configuring iLO......................................................................................................6
3.5. Reconfiguring the SOMS to use separate interfaces for North (NetAct) and
South (RNC and NodeB) connections.............................................................................7

4. Integration...................................................................................12
4.1. RNC Integration...................................................................................................12
4.2. NetAct integration................................................................................................14
4.3. WBTS Integration.................................................................................................15

5. Technical Notes..........................................................................16
5.1. Ports and Protocols..............................................................................................16
5.2. LDAP................................................................................................................... 16
5.3. OMS – OMU Communication...............................................................................18
5.4. Tracing (S)OMS-OMU communication in OMU....................................................18
5.5. SOMS Redundancy.............................................................................................20
5.6. Linux IP routing....................................................................................................21

6. Troubleshooting Cases...............................................................23
6.1. No IP connectivity between WBTS and Southbound SOMS address...................24
6.2. RNC Topology Upload from SOMS failing...........................................................26
6.3. RNC RNW Plan Upload failing from NetAct.........................................................27
6.4. WBTS Site Configuration File Upload Failing from NetAct...................................28
6.5. License Download to WBTS failing from NetAct..................................................30
6.6. SOMS Internal Firewall misconfiguration.............................................................34
6.7. Recreating the (S)OMS Topology Database........................................................36
6.8. Increment installation could not be performed in a single step.............................36

7. References.................................................................................37

8. Appendixes.................................................................................37
8.1. OMS Pronto Symptom List...................................................................................37
8.2. OMS MySQL Databases......................................................................................37

2
 For internal use

1. Version Control

Issu
Date e Author Summary of changes

31/10/12 V1.0 Luis Abreu First Release

3
 For internal use

2. Introduction

The purpose of this document is to share the knowledge gained during the installation,
commissioning and integration of a factory Stand Alone OMS in a customer’s live network.
Some project specific implementation aspects are included. Most steps common to the
integrated OMS are omitted. Background information on some technical areas and a few
troubleshooting cases are provided.

The Stand Alone OMS replaces the integrated OMS. It will perform the same functions as
the Integrated OMS but is installed as a separate HW unit, possibly in a different location
than the RNC. Up to 10 RNCs can be controlled by one Standalone OMS. There are also
limitations for the total number of WBTS and Cells that can be managed by one SOMS.

The HW used to implement the SOMS is an HP ProLiant DL360 G6 1U rackmount server with
either four 146 GB or four 300 GB 2,5” Serial Attached SCSI hard disk drives.

Figure 1 HP ProLiant DL360

Figure 2 Network Schematic

There are two Ethernet ports in the SOMS, configurable in two different ways. Normally the
two physical ports are grouped into one logical interface named bond0. This logical interface
is then configured with two different IPs. One of those IPs is used to connect to NetAct and
the other one to the RNC and WBTSs. The connection towards NetAct is referred to as the

4
 For internal use

Northbound connection, the one towards the RNC and WBTS as the Southbound
connection. There is a second configuration option where one physical Ethernet port is
configured with one IP and used for the Southbound connection, the other physical port
configured with a different IP and used for the Northbound connection.

In the text that follows, actual command line inputs are highlighted in bold, command
outputs and text logs are enclosed in a border box. Where parts of command outputs have
been omitted, this is indicated with square brackets […] .

3. Installation and Commissioning

3.1. Datafill

If you proceed with a brand new installation of the SOMS, you will be asked for integration
and commissioning data right at the start. The attached excel file contains an example blank
datafill with all the parameters required.

SOMS__Datafill_Exa
mple.xls

3.2. Reconfiguring the RAID array

The disk structure with which the HP server comes from factory may not be the correct one.
The first step is to verify and reconfigure it if necessary. The target configuration is
composed of two logical drives, each logical drive a RAID0 array. These RAID0 arrays are in
turn composed of two physical hard disks which are not mirrored copies of one another. The
mirroring between the two RAID0 arrays is performed by the OMS SW. The two left-most
physical disks in the HP server will form logical drive 1 and the right-most disks logical drive
2.

As the disk structure is different, the commands to display and configure the disks will be
different than the ones used with the Integrated OMS. The disk structure can be
interrogated with the command:

#fsswcli –m-s
/dev/cciss/c0d0 UNLOCKED
/dev/cciss/c0d1 UNLOCKED

where c0d0 and c0d1 are the two RAID0 arrays. To look into the individual RAIDs:

#cd /proc/driver/ccis
#cat cciss0
cciss0: HP Smart Array P410i Controller
5
 For internal use

[…]
Sequential access devices: 0
cciss/c0d0: 599.93GB RAID 0
cciss/c0d1: 599.93GB RAID 0

From the last output each RAID0 array has a capacity of around 600 GB, the sum of the two
300GB physical disks. To examine the actual physical disks the hp tool hpacucli is available:

#hpacucli
HP Array Configuration Utility CLI 8.35-7.0
Detecting Controllers...Done.
Type "help" for a list of supported commands.
Type "exit" to close the console.
=> ctrl all show config detail
[…]
Array: A
Interface Type: SAS
Unused Space: 0 MB
Status: OK

Logical Drive: 1
Size: 558.7 GB
Fault Tolerance: RAID 0
Heads: 255
[…]

Disk Name: /dev/cciss/c0d0

[…]
physicaldrive 1I:1:1
Port: 1I
Box: 1
Bay: 1
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 300 GB
[...]

3.3. Reinstalling SOMS using a USB Stick

6
 For internal use

The reinstallation procedure is the same as for an Integrated OMS, only the type of OMS has
to be chosen as fp4_hprm. The documentation states that during USB installation the OMS
hardware clock must be set to “UTC/GMT time (United Kingdom’s time)”; for some of the
year UK time is not UTC/GMT. UTC/GMT is always the time that should be set.

As in an Integrated OMS the installation script has to be run,

./CompleteGOMS_USBInstallation.sh

and commissioning data has to be entered as per the excel sheet included above.

3.4. Configuring iLO

iLO is HPs out-of-band management technology, implemented by separate HW inside the HP

server. The fact that this feature is implemented by separate HW allows that even when the
main HW components of the server are restarted remote supervision is maintained. The
connection to the iLO management utility is made via a separate Ethernet port in the HP
chassis. This port has its own IP, which can be reconfigured from the default (obtained by
DHCP) to the desired one. The default login and password for iLO, which are also the
administrator credentials are written in a label on the chassis of the server.

Figure 1 IP Configuration tab in iLO

A very useful feature of iLO is a remote console session that can be used to monitor a restart
as would a local session. This remote console allows direct login with the root user,
something not possible with telnet or ssh.

Figure 2 iLO Remote Login Prompt

 For internal use

With this feature, unlike with the Integrated OMS, it is possible to connect and reboot the
SOMS even when IP connectivity to the southbound and northbound SOMS IPs is lost.

3.5. Reconfiguring the SOMS to use separate interfaces for North (NetAct) and South (RNC and
NodeB) connections

By default, the SOMS’s two Ethernet interfaces are grouped into one logical interface,
bond0, whether using the same or different IP addresses for the Northbound and
Southbound IP connections. The following procedure was used to reconfigure the SOMS so
that the two physical Ethernet interfaces were each assigned to its own IP, one for
Northbound one for Southbound IP. As this procedure touches upon several interesting
areas of the SOMS it will be included here.

Start by viewing the current configuration of the bond0 interface:

#fsipnet get ifacename bond0

fsipnet bond add bond0 node CLA-0 addphase nwmgrstart slave eth0 slave eth1
dependency
fsipNetworkObjectName=FSIPEthernetIface@eth0,fsFragmentId=Network,fsipHostName=CL
A-0,fsFragmentId=Nodes,fsFragmentId=HA,fsClusterId=ClusterRoot dependency
fsipNetworkObjectName=FSIPEthernetIface@eth1,fsFragmentId=Network,fsipHostName=CL
A-0,fsFragmentId=Nodes,fsFragmentId=HA,fsClusterId=ClusterRoot owner /CLA-0 address
inherit dynamic yes mtu 1500 active eth0 priority 786432 delphase nwmgrstop from eth0
rstp yes

The fsipnet command is used to interrogate network configuration from the LDAP database.
The same parameters can be checked by directly inspecting the database with the
Parameter Tool from application launcher.

View all Ethernet interfaces:

#fsipnet ethiface get

fsipnet ethiface add eth0 node CLA-0 addphase nwmgrstart delphase never owner /CLA-0
address RTE dynamic yes mtu 1500 hw fshwConnectorId=eth-1,fshwPIUId=piu-
0,fshwEquipmentHolderId=chassis-1,fshwEquipmentHolderId=cabinet-
1,fsFragmentId=HW,fsClusterId=ClusterRoot priority 524288

fsipnet ethiface add eth1 node CLA-0 addphase nwmgrstart delphase never dependency
fsipNetworkObjectName=FSIPEthernetIface@eth0,fsFragmentId=Network,fsipHostName=CL
A-0,fsFragmentId=Nodes,fsFragmentId=HA,fsClusterId=ClusterRoot owner /CLA-0 address
inherit dynamic yes mtu 1500 hw fshwConnectorId=eth-2,fshwPIUId=piu-
0,fshwEquipmentHolderId=chassis-1,fshwEquipmentHolderId=cabinet-
1,fsFragmentId=HW,fsClusterId=ClusterRoot priority 524289 from eth0

8
 For internal use

View OMS dedicated IP addresses:

#fsipnet get type Dedicated | cut –f1-26-d’’

fsipnet address add 172.17.5.150/24 node CLA-0 addphase nwmgrstart delphase

nwmgrstop dependency
fsipNetworkObjectName=FSIPBondIface@bond0,fsFragmentId=Network,fsipHostName=CLA
-0,fsFragmentId=Nodes,fsFragmentId=HA,fsClusterId=ClusterRoot owner /CLA-0 dynamic
yes user /CLA-0/NWI3TUServer user /FMAdapter user /CLA-
0/FSDirectoryServer/LDAPServer user /CLA-0/FSClusterNTPServer user /CLA-
0/EmiPMPlatMeaServer/EmiPMPlatMea

View OMS static routes:

#fsipnet nexthop get

fsipnet nexthop add 0.0.0.0/0 table 254 tos 0 via 172.17.5.1 node CLA-0 iface bond0 name
bond0@172.17.5.1 weight 1

To proceed with the reconfiguration, the existing configuration is removed. Starting with the
default routes:

fsiproute delete 0.0.0.0/0 owner /CLA-0

Remove first IP of SOMS:

fsipaddress dedicated delete 172.17.5.150/24 iface bond0 owner /CLA-0

The same IP address was being used for Northbound and Southbound connections, so no
additional removal command was necessary.

Remove bond0 interface:

fsipbond delete bond0 owner /CLA-0

The next step will be associating the different recovery groups with the South or North
interfaces. This is done by configuring the environment variables NLIST and SLIST with the
required services. Services to be associated with the Northbound interface are configured
on NLIST, Southbound on SLIST. The correct mapping is obtained by looking at the
assignment performed by the script oms_rnc_script.sh. This script is called by
zmodifynetworksettings during commissioning. In addition to the recovery groups that have
to be assigned manually, the daemons /Vsftpd, /sshd, /httpd listen on both interfaces by
default.

To inspect the mapping performed by the script:

cd /opt/Nokia/SS_OMSINST/script
9
 For internal use

# grep ‘NLIST=’oms_rnc_ipscript.sh
NLIST="user /Directory user /ClusterDNS user /SNMPMediator user /HTTPDPlat user
/NWI3Adapter user /PAP user /RuimReplicator user /RNWEvent user /NWI3TU user
/FMAdapter user /PMPres user /NWI3CM user /MeaHandler user /FMUIGate user
/PMGeneric user /OMSEventHandler user /Nwi3SWAgent user /Nwi3SecuCorba user
/CMaus user /THRKPI user /Nwi3LMAgent user /OMSAccessQuery user /OMSAppControl
user /MMI user /PMOnline user /RNWCM user /NELogManager user /NWI3HW user
/Nwi3Certificate user /EmiPMPlatMea user /Nwi3AuditTrailHandler"

#grep ‘SLIST=’oms_rnc_ipscript.sh

SLIST="user /ClusterNTP user /OMSBTSOM"

To assign values to NLIST and SLIST:

#NLIST="user /Directory user /ClusterDNS user /SNMPMediator user /HTTPDPlat

user /NWI3Adapter user /PAP user /RuimReplicator user /RNWEvent user
/NWI3TU user /FMAdapter user /PMPres user /NWI3CM user /MeaHandler
user /FMUIGate user /PMGeneric user /OMSEventHandler user /Nwi3SWAgent
user /Nwi3SecuCorba user /CMaus user /THRKPI user /Nwi3LMAgent user
/OMSAccessQuery user /OMSAppControl user /MMI user /PMOnline user
/RNWCM user /NELogManager user /NWI3HW user /Nwi3Certificate user
/EmiPMPlatMea user /Nwi3AuditTrailHandler"

SLIST="user /ClusterNTP user /OMSBTSOM"

Add dedicated addresses to both Ethernet interfaces, associating the services of NLIST and
SLIST with each one:

fsipaddress dedicated add 172.17.5.150/24 iface eth0 owner /CLA-0 $NLIST

fsipaddress dedicated add 172.17.29.22/28 iface eth1 owner /CLA-0 $SLIST

Define the default route for main routing table (in a following section, there is a short
explanation of Linux routing tables):

fsiproute add 0.0.0.0/0 via 172.17.5.1 iface eth0 owner /CLA-0

Define the routes in the routing table for the southbound interface. The first of the following
commands defines the subnet that can be reached directly by the interface; the second one
defines a gateway on that interface:

fsiproute add 172.17.29.16/28 iface eth1 owner /CLA-0 table 20

10
 For internal use

fsiproute add 0.0.0.0/0 via 172.17.29.17 iface eth1 owner /CLA-0 table 20

Create the routing rules. These specify which of the routing tables should be used
depending on the source and destination addresses configured on the IP packet:

fsiprule add 20000 owner /CLA-0 src 172.17.29.16/28 table 20

fsiprule add 20001 owner /CLA-0 dst 172.17.29.16/28 table 20

The next steps verify the new configuration before it is applied by system restart. View the
dedicated IP addresses:

#fsipnet get type Dedicated | cut –f1-26-d’’

fsipnet address add 172.17.5.150/24 node CLA-0 addphase nwmgrstart delphase
nwmgrstop
dependency
fsipNetworkObjectName=FSIPEthernetIface@eth0,fsFragmentId=Network,fsipHostName=CL
A-0,fsFragmentId=Nodes,fsFragmentId=HA,fsClusterId=ClusterRoot owner /CLA-0 dynamic
yes user /CLA-0/FSClusterDNSServer user /FMUIGate user /CLA-0/FSFTPServer user /CLA-
0/FSDirectoryServer/LDAPServer user /CLA-0/MeaHandlerServer/MeaHandler fsipnet
address add 172.17.29.22/28 node CLA-0 addphase nwmgrstart delphase nwmgrstop
dependency
fsipNetworkObjectName=FSIPEthernetIface@eth1,fsFragmentId=Network,fsipHostName=CL
A-0,fsFragmentId=Nodes,fsFragmentId=HA,fsClusterId=ClusterRoot owner /CLA-0 dynamic
yes user /ClusterNTP user /CLA-0/FSClusterNTPServer user/CLA-0 user /OMSBTSOM user
/CLA-0/OMSBTSOMServer/OMSBTSOM

View the static routes:

#fsipnet nexthop get

fsipnet nexthop add 0.0.0.0/0 table 254 tos 0 via 172.17.5.1 node CLA-0 iface eth0 name
eth0@172.17.5.1 weight 1
fsipnet nexthop add 172.17.29.16/28 table 20 tos 0 node CLA-0 iface eth1 name eth1@
weight 1
fsipnet nexthop add 0.0.0.0/0 table 20 tos 0 via 172.17.29.17 node CLA-0 iface eth1 name
eth1@172.17.29.17 weight 1View both routing tables.

View routes contained in routing table 254:

#ip route list table 254

172.17.29.16/28 dev eth1 proto kernel scope link src 172.17.29.22
172.17.5.0/24 dev eth0 proto kernel scope link src 172.17.5.150
default via 172.17.5.1 dev eth0 proto static notify

11
 For internal use

View the routes contained in routing table 20:

#ip route list table 20

172.17.29.16/28 dev eth1 proto static notify
default via 172.17.29.17 dev eth1 proto static notify

View the IP rules contained in the Routing Policy Database:

#ip rule show

0: from all lookup 255
20000: from 172.17.29.16/28 lookup 20
20001: from all to 172.17.29.16/28 lookup 20
32766: from all lookup main
32767: from all lookup default

Restart the system to apply all new settings:

fshascli -r /

Note that with this configuration, the default route when the IP rules 2000 and 20001 don’t
apply will still be the northbound interfaces’ gateway. The SOMS services however will use
as their source address the address configured in the interface (North or Southbound) to
which they are assigned to. This assignment is made via the NLIST and SLIST environment
variables.

4. Integration

4.1. RNC Integration

In RU30, a new user has to be created in the OMU for the (S)OMS to login with. This is done
automatically by the RU20 to RU30 upgrade macros, but has to be done manually in the
case of a new RNC. The command to create the user is:

ZIAH:OMSUSR:PROFILE;

After entering the command, the MML program asks for the password, the value that
should be set unless there are project specific instructions is SYSTEM. (This information was
added to Issue 08 of WCDMA RAN and I-HSPA, Rel. RU30, Operating Documentation).

12
 For internal use

After that the RNC ID and SOMS IPs (Primary, Secondary and Backup) have to be configured
in the RNC using the RUOSTEQX service terminal extension:

ZLE:1,RUOSTEQX;
Z1CR:<RNC_ID>,<OMS_IP_Address>,<RNC_name>;

Interrogating we get the output:

>Z1C;
0000-RUO>C;
Connecting to RNW database...

RNC object data in RNW database:

-----------------------------------------------------
RNC ID: 707
RNC name: RNSNA
Primary OMS IP address: 172.17.29.22
Secondary OMS IP address: 172.17.29.23
Backup OMS IP address: 172.17.29.22
Current serving OMS: 1 (primary OMS)
Serving OMS selection: 0 (automatic)
Connection retry count: 10
-----------------------------------------------------

In case the connection is not automatically established, BOI and BOE can be restarted in
both the active and standby OMU as a workaround:

ZLP:Q,SQQ;
ZQRR:506;
ZQRR:A0F;

The OMS IPs configured in the RNC can also be seen from SpringyUI. Only the backup OMS
IP Address can be edited there:

Figure 3 SOMS IP in Parameter Editor

13
 For internal use

As soon as this IP is configured in the RNC, the OMU will initiate the connection towards the
(S)OMS. This can be verified by ZQRS in the OMU and netstat in the SOMS. If the connection
is made correctly the RNC will appear in Springy’s main view. If only the RNC appears in
SpringyUI without any objects under it, and if the RNC’s RNW is not blank, then most likely
even though the BTSOM connection between OMU and (S)OMS is correctly established the
RNW plan upload that occurs immediately after the establishment is failing. It is only after
this initial plan transfer the topology database in (S)OMS is initialized with the objects
configured in the RNC.

4.2. NetAct integration

NetAct integration is the same as for an Integrated OMS, Initial Registration Username and
Password as well as the IOR string were configured in parameter editor:

14

Figure 4 Netact registration parameters in SOMS

 For internal use

It is possible to verify the correct registration of the SOMS to NetAct by logging in to the
connectivity server and changing directory to:

cd /var/opt/nokia/oss/global/nw3n3c/log

In the directory there are several files with the name structure

n3cregmx_<xxx>_i1.log

and the correct registration should be found in the file with the newest date by doing

# grep OMS n3cregmx_<xxx>_i1.log | grep register

4.3. WBTS Integration

It must be verified that there is IP connectivity between the Southbound IP address of the
SOMS and the WBTS O&M address. All protocols used for communication between OMS
and WBTS must be allowed by any firewall that might exist. The firewall could be between
ESA and (S)OMS, between WBTS and NPGE or between WBTS and ESA. One particularity in
the case of Active – Standby SOMS configuration where the SOMS is the NTP server for the
WBTS: the Southbound addresses of both SOMS have to be configured in the NTP Server tab

15

Figure 5 WBTS NTP Addresses

 For internal use

in order for SOMS switchover to be seamless:

In case of an ATM Iub WBTS, IP routing has to be defined between the SOMS and the OMU.
The WBTS will then be reachable via the IPoATM connection that starts at the OMU. In case
of an IP IuB WBTS the O&M connection can go via the NPGE(P) card over the same IP path
as User and Control Plane, or it can be carried in a separated O&M network going through
the ESA card. In both cases OMU has to have routing defined to reach the WBTS. In the case
where the NPGE is used, the NPGE has to have routing defined to be able to reach the SOMS
southbound interface, usually via the OMU card.

5. Technical Notes

5.1. Ports and Protocols

Attached is an excerpt from the RU30 WCDMA RAN communication matrix detailing the IP
connections and ports used between OMS, OMU and NetAct:

WCDMA RAN IP
protocol and port list RU30_20110825_approved.xls

5.2. LDAP

There is an LDAP daemon server running in the OMS to which Parameter Editor connects
when it is used to edit the LDIF.

The user accounts are stored n LDIF in ClusterRoot->Security- >People , and they are
accessed by the OMS processes when user credentials need to be verified.

Wiith ps –ef | grep ldap it can be seen that the LDAP process was started in SOMS with
commands:

16
 For internal use

/usr/local/libexec/slapd -d 0 -h ldap://CLA-0:49501 ldap://localhost:49501 -f

/opt/Nokia_BP/etc/ldapfiles/fsPlatformSlave.conf

/usr/local/libexec/slapd -d 0 -h ldap://localhost:389 ldaps://172.17.5.150:636

ldap://directory:49172 ldap://directory:389 -f fsPlatform.conf

The –d 0 option means no debugging, so no LDAP logging gets written to syslog by default.
There are two listeners, LDAP with TSL on localhost:49501 and ldap over TSL on
localhost:636.

Many of SOMS’s processes have a connection to the LDAP daemon:

# netstat -apeen | grep 61898

tcp 0 0 127.0.0.4:49501 127.0.0.4:61898 ESTABLISHED 0 30850 11415/slapd
tcp 0 0 127.0.0.4:61898 127.0.0.4:49501 ESTABLISHED 0 30846 13528/httpd

# netstat -apeen | grep 56013

tcp 0 0 127.0.0.37:56013 127.0.0.37:389 ESTABLISHED 0 36766 14391/EmiCMOS
tcp 0 0 127.0.0.37:389 127.0.0.37:56013 ESTABLISHED 0 36767 12354/slapd

# netstat -apeen | grep 57286

tcp 0 0 127.0.0.37:57286 127.0.0.37:389 ESTABLISHED 0 54384 17201/fumigate
tcp 0 0 127.0.0.37:389 127.0.0.37:57286 ESTABLISHED 0 54385 12354/slapd

Taking a wireshark trace on the loopback address in OMS, the processes can be seen
interrogating LDIF database:

17
 For internal use

5.3. OMS – OMU Communication

Figure 6 LDAP Communication

In RU30 the EMT protocol between (S)OMS and OMU is replaced with BTSOM. The
interrupted lines in the figure below indicate that the communication is encapsulated in
BTSOM protocol packets between OMS and the BOIMED program block in OMU, and
forwarded to the other program blocks inside the OMU or to the WBTS via the O&M
connection. This is just for O&M messages, and does not include the HTTP, FTP, HTTPS, SFTP
connections used for file (Licences, Site Configuration File, RNC Plan) transfer between WBTS
and SOMS.

Figure 7 (S)OMS - OMU Communication

5.4. Tracing (S)OMS-OMU communication in OMU

18
 For internal use

Monitoring on the OMU side can be performed with the DMXMacro. A monitoring condition
that could capture the relevant program blocks would be BOR(DER) 0A0F, and BOI(MED)- 0506,
which are included by default in the DMXMon macro:

Figure 8 DMXMon Monitorings

Capturing a SCF upload from a WBTS in OMU (after filtering out the NA messages the sack could
not decode):

And the same message exchange on the SOMS Southbound interface:

19
 For internal use

The user data in the BOR message starting at user_data[3]. can be matched to the OM message
starting with the. checksum contained in the fileInfo information element.

5.5. SOMS Redundancy

The OMU has a Primary and Secondary SOMS address configured. The configuration
between Primary and Secondary OMS has to be the same regarding usernames and
passwords, but not in terms of IPs or SOMS ID. This enables the OMU to switchover SOMS
automatically if the connection fails. This is feature RAN1874 Automatic OMS resiliency.

It is also possible to initiate an SOMS switchover manually from SOMS:

Figure 9 SOMS Switchover from SpringyUI 20

 For internal use

After the RNC has established a connection with the target SOMS, the latter will initiate a
plan upload transferring the topology and measurement plans. It will also initiate an alarm
upload. A trace on the source OMS shows:

Figure 10 SOMS switchover on Source SOMS

Figure 11 SOMS switchover on Target SOMS

A trace on the target SOMS shows:

5.6. Linux IP routing

Linux uses multiple routing tables, identified by numbers from 0 to 255. By default, two
tables are configured, 254 called the default routing table and 255 called the local routing
table. The local table is automatically manipulated by Linux, for example when an interface
is configured with the ifconfig command. The default table is the one manipulated when
routes are added to a Linux machine by the user, for example with the ip route add
command.

There is a mapping file under /etc/iproute2/rt_tables that shows the correspondence

between number and name for the routing tables.

There is also another separate database that defines which routing table is used for each IP
packet. This is the Routing Policy Database, RPDB. This table contains a series of rules, each
with its own priority. A packet is compared sequentially to each rule, when a match is found,
the corresponding routing table is used. The command to examine the RPDB was given
above and is

21
 For internal use

#ip rule show

0: from all lookup 255
20000: from 172.17.29.16/28 lookup 20
20001: from all to 172.17.29.16/28 lookup 20
32766: from all lookup main
32767: from all lookup default

According to this output, the first routing table to be checked for any packet is the local
routing table, routing table 255. Examining in this routing table in a standard (S)OMS:

# ip route list table 255

broadcast 172.17.29.31 dev eth1 proto kernel scope link src 172.17.29.22
broadcast 172.17.29.16 dev eth1 proto kernel scope link src 172.17.29.22
local 172.17.29.22 dev eth1 proto kernel scope host src 172.17.29.22
broadcast 172.17.5.255 dev eth0 proto kernel scope link src 172.17.5.150
local 172.17.5.150 dev eth0 proto kernel scope host src 172.17.5.150
local 127.0.0.14 dev lo proto kernel scope host src 127.0.0.1
[..]
local 127.0.0.16 dev lo proto kernel scope host src 127.0.0.1
broadcast 172.17.5.0 dev eth0 proto kernel scope link src 172.17.5.150
local 127.0.0.17 dev lo proto kernel scope host src 127.0.0.1
[...]
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1

This routing table contains only local or broadcast routes, that is, for destination IPs that are
either configured locally on one of the host’s interfaces or are directly accessible (without
resorting to a gateway) via one of those interfaces. For each of the external interface
addresses, there is a broadcast route to reach its network and broadcast address.

After rule 0, the next rules to be examined where the two introduced during the SOMS
reconfiguration in section 2.5. If either the source (rule 20000) or destination (rule 20001)
IPs in the packet match 172.29.16/28 then routing table 20 will be used.

If neither the source nor the destination address in the packet is within that range, then the
main routing table will be used, table 254. Lastly, the local routing table will be tried.

22
 For internal use

6. Troubleshooting Cases

In this section some troubleshooting cases are presented. They are for a Network setup
where all the TLSMode parameters are set to Off, meaning some message exchanges are
sent in the clear that normally would be encrypted.

Figure 12 TLS Parameters in Parameter Editor

With TLSModeOMOff the BTSOM traffic between the Southbound OMS interface and the
OMU will not go over TLS and can be viewed in wireshark. The same applies for BTSOM
traffic between OMU and WBTS as TLSMode RNCBTS=Off. TLSModeFT=Off means that the
WBTS will transfer files from OMS using HTTP instead of HTTPS, file transfer between OMU
and OMS will be performed using FTP instead of SFTP and file transfer between SOMS and
Netact will be performed with HTTP instead of HTTPS.

NSN’s proprietary Wireshark includes dissectors for BTSOM and GIOP, the protocol used
between SOMS and Netact. Therefore all traces taken at the SOMS should be analyzed with
it instead with of the standard open source Wireshark.

NSN’s Wireshark can be obtained from isource

https://isource.access.nokiasiemensnetworks.com/

A Wireshark trace can be captured in the (S)OMS using the command:

# /usr/sbin/tshark -i bond0 -V -R "ip.addr=10.10.10.10" -w /tmp/test.cap

In this case the options determine that the capture will be only on interface bond0 (-i), for
packets with the specified filter (-R) and written to a file (-w). The produced file can be
transferred out of the SOMS and opened directly with Wireshark.

23
 For internal use

IP connectivity between two interfaces below means that it is possible to ping one
interface’s IP address from the other interface, using the latter’s IP address as the source
address in the ping command

ping –I <source ip address> <destination ip address>

Simlarly with traceroute:

traceroute –I <source ip address> <destination ip address>

6.1. No IP connectivity between WBTS and Southbound SOMS address

After a SOMS recommissioning where a bond0, single IP implementation was changed to

separate IPs for eth0 and eth1, no IP connectivity between WBTS O&M address and the
Southbound IP in the SOMS existed. There was IP connectivity between the WBTSO&M and
the Northbound address.

The transmission network chain in this case (or what we could see from RAN) was:

Figure 13 SOMS - WBTS IP path

In the tracert from SOMS Southbound we saw:

24
 For internal use

# tracert -i eth1 10.56.13.113

traceroute to 10.56.13.113 (10.56.13.113), 30 hops max, 40 byte packets

1 172.17.29.17 (172.17.29.17) 1.375 ms 1.296 ms 1.223 ms
2 172.17.31.83 (172.17.31.83) 2.165 ms 1.966 ms 2.078 ms
3 10.200.71.38 (10.200.71.38) 17.264 ms 17.224 ms 17.245 ms
4 10.200.71.45 (10.200.71.45) 18.989 ms 13.814 ms 26.861 ms
5 ***
6 ***

In the tracert from SOMS Northbound we saw:

# tracert -i eth0 10.56.13.113

traceroute to 10.56.13.113 (10.56.13.113), 30 hops max, 40 byte packets

1 172.17.5.1 (172.17.5.1) 1.193 ms 1.147 ms 1.147 ms
2 172.17.31.83 (172.17.31.83) 1.796 ms 1.863 ms 1.974 ms
3 10.200.71.38 (10.200.71.38) 12.437 ms 12.555 ms 12.567 ms
4 10.200.71.45 (10.200.71.45) 26.109 ms 12.763 ms 18.611 ms
5 10.10.192.129 (10.10.192.129) 13.123 ms 15.865 ms 17.331 ms
6 10.56.25.1 (10.56.25.1) 18.349 ms 16.947 ms 18.274 ms
7 10.56.13.113 (10.56.13.113) 22.229 ms 31.245 ms 31.334 ms

Adding a route to the router between NPGE and WBTS, the problem was cleared.
Note that even in the correct case the NPGE only sends an ICMP Time Exceeded from the
IFFE interface to the ESA, not from the VLAN/IFGE interface towards the Iub transmission
network. This causes that the IFGE0/VLAN interface with IP 10.10.192.132 does not show up
in the traceroute output and was an argument used by the customer to say the packet was
being lost inside the NPGE.

6.2. RNC Topology Upload from SOMS failing

In this case the Topology Upload when requested via graphical user interface from SOMS
failed. In the Wireshark trace it was visible that the SOMS was trying to ftp to the OMU to
25
 For internal use

retrieve the plan using as source address 172.17.5.150 instead of the correct southbound
interface address of 172.17.29.22.

In Wireshark (10.200.81.36 is the OMU address):

Figure 14 SYN packets opening TCP connection from SOMS to OMU

In the OMS logs in CMausSrv_trace.txt the following was visible:

Tue Jul 24 11:34:44 2012 #0000000438 OMS::TRACE DEBUG <common.cpp:28> CLA-0 :

read_conf_value() configuration key=omsParameterId=szIPAddress, omsFragmentId=OMSS,
omsFragmentId=CUAccess, omsFragmentId=Network, omsFragmentId=System,
fsFragmentId=OMS, fsClusterId=ClusterRoot name=omsParameterValue (0,0) 721ms
pgrp=19900 pid=19900 tid=1643579712 uid=0 gid=0

Tue Jul 24 11:34:44 2012 #0000000441 OMS::TRACE DEBUG <common.cpp:342> CLA-0 :

getFileFromFtp() FTPServer: 10.200.81.36, FTPUser: OMSUSR, remote file:
/RUNNING/PLATMP/Topology.Z, local file:
/var/opt/OMSftproot/topology_upload_files//796Topology.Z, bind address: 172.17.5.150,
connections: < [FTP]=21,> (0,0) 722ms pgrp=19900 pid=19900 tid=1643579712 uid=0 gid=0

Which meant the CMausSrv was looking up the value of szIPAddress in the LDAPs database,
and that the value there was still 172.17.5.150.

Logging into Parameter Editor and checking under:

key=omsParameterId=szIPAddress, omsFragmentId=OMSS, omsFragmentId=CUAccess,

omsFragmentId=Network, omsFragmentId=System, fsFragmentId=OMS,
fsClusterId=ClusterRoot

26
 For internal use

This was confirmed. After changing the parameter to the correct value of 172.17.29.22 the
problem was cleared.

6.3. RNC RNW Plan Upload failing from NetAct

After the previous problem had been cleared, RNW plan upload was failing from Netact.
Problem could be seen in OMU computer logs, Wireshark logs and OMS logs for Nwi3CM.
The southbound source address that was showing up in the CMPlanManager configuration
dump included in in CMausSrv_trace.txt still had the old value:

------------------------------------------------------------------------------------
| CMPlanManager configuration dump |
| setting | type | value |
------------------------------------------------------------------------------------
| CM_FTP_ROOT | S | /var/opt/OMSftproot |
| CM_FTP_HOST | S| 172.17.5.150 |
| CM_FTPS_HOST | S| 172.17.5.150 |
| CM_OMSID | S| NE-OMS-11 |
| CM_MAX_THREAD_COUNT | I| 100 |

So the change in Parameter Editor had not yet been propagated to CMPlanManager. After
restarting Nwi3CM with

#fshascli –r /Nwi3CM

the value was correct:

| CM_FTPS_HOST | S| 172.17.29.22 |

and the problem was cleared.

6.4. WBTS Site Configuration File Upload Failing from NetAct

27
 For internal use

Attempting a WBTS SCF upload from NetAct from CM Operations Manager, there was the
following error message:

Figure 15 CM Operations Manager error message

Taking a wireshark on the Northbound interface, the upload request was correctly arriving
at the Northbound interface of the SOMS:

Figure 16 Licence upload request sent from NetAct

Taking a wireshark trace on the Southbound interface of the SOMS:

28
Figure 17 SYN packets opening TCP connection From SOMS to WBTS
 For internal use

the SOMS was correctly attempting to open a TCP session to the WBTS, but not reply was
ever received.

It is visible in the trace from the destination mac address that the packet is being correctly
sent to the gateway for the SouthBound interface. Confirmed in the SOMS:

#arp –a

nar1oss2.r01.netact.vfl.vodafone (172.17.5.46) at 1C:C1:DE:6D:A5:20 [ether] on eth0

? (172.17.5.1) at 00:10:DB:FF:40:05 [ether] on eth0
? (172.17.29.17) at 00:10:DB:FF:40:05 [ether] on
eth1nar1gu1b.winr01.r01.netact.vfl.vodafone (172.17.5.26) at 68:B5:99:B2:4A:E4 [ether] on
eth0
nar1ds1b.r01.netact.vfl.vodafone (172.17.5.6) at 68:B5:99:B9:D5:C8 [ether] on eth0
nar1la1b.r01.netact.vfl.vodafone (172.17.5.16) at 68:B5:99:C9:79:00 [ether] on eth0
nar1cs2b.r01.netact.vfl.vodafone (172.17.5.12) at 1C:C1:DE:6D:A5:20 [ether] on eth0

The same error message seen in Netact was recorded in Nwi3CMPlanSrv_trace.txt:

Thu Aug 9 16:37:19 2012 #0000042210 OMS::TRACE DEBUG <Error Log Entry:0> CLA-0 :
(../src/NemuFTP.cpp:968): CNemuFTP::GetFile: file transfer failed:
http://10.29.52.21:6000/bts//ram/CUPL_BTSC_252.xml.gz ->
/var/opt/OMSftproot/cmplan/ul/ne/11924/RNC_464_WBTS_252_SCF, responseCode=0,
curlError: Couldn't connect to server(7)
Error code: 0x80004005 (0,0) 825ms pgrp=16775 pid=16775 tid=1429371200 uid=0 gid=0

The problem was the firewall existing in the transmission network between SOMS and
WBTS.

6.5. License Download to WBTS failing from NetAct

29
 For internal use

In the normal process for license download, NetAct first informs (S)OMS that it has a license
to download, then transfers that license to a directory in SOMS. The following is from a
wireshark trace taken on a working OMS where the Northbound and Southbound OMS IPs
are 10.200.68.167, the OMU IP is 10.200.68.164 and the WBTS IP is 10.29.12.1. The process
can be seen starting with a license download request coming from NetAct to the
(Northbound) address of the OMS, protocol GIOP:

Figure 18 Licence download request sent from NetAct

The OMS then sends to the OMU a fileLoadPrepare message. This message will be
processed by the OMU internal program blocks and forwarded to the correct WBTS via the
O&M connection. Among the information that this message provides to the WBTS are the
protocols it can use for the license transfer, the (OMS) IP, login and passwords that it should
use, and the directory in the OMS where the license file is located.

Figure 19 fileLoadPrepare sent from OMS to OMU

30
 For internal use

On a correctly working transfer, the WBTS establishes an http session to the OMS with the
provided credentials and gets the license file:

Figure 20 HTTP GET command sent from the WBTS to OMS

In Nwi3LMagent.txt, the operation starts with the creation of the directory (145334 in the
excerpts below which are from a different transfer, it was 143648 in the screenshot above):

Wed Aug 15 11:14:01 2012 #0006082939 OMS::TRACE DEBUG <OMSFlexiHome.h:33> CLA-0 : OMSFlexiHome::lease_service
called, home interface=IDL:NWI3/LicenceManagement/LicenceOperation_MS_V1:1.0 (0,0) 813ms pgrp=12491 pid=12491
tid=129248144 uid=0 gid=0

Wed Aug 15 11:14:01 2012 #0006082976 OMS::TRACE DEBUG <ImplLicenceOperation_MS_V1.cpp:571> CLA-0 : REV 50975
checkLicenceName - License name: IE000284.XML (0,0) 888ms pgrp=12491 pid=12491 tid=42281872 uid=0 gid=0

Wed Aug 15 11:14:01 2012 #0006082980 OMS::TRACE DEBUG <ImplLicenceOperation_Base.cpp:53> CLA-0 : REV 50975
GetMoidBaseIdAndLocalMoid - BaseId: RNC-463, localmoid: WBTS-151 (0,0) 888ms pgrp=12491 pid=12491 tid=42281872
uid=0 gid=0

Wed Aug 15 11:14:01 2012 #0006082987 OMS::TRACE DEBUG <LicenseFileHelper.cpp:83> CLA-0 : REV 52127 createOneDir -
Directory: /var/opt/OMSftproot/Nwi3LMAgent/145334/ created (0,0) 889ms pgrp=12491 pid=12491 tid=42281872 uid=0
gid=0

Wed Aug 15 11:14:01 2012 #0006083027 OMS::TRACE DEBUG <BTSOMInterface.cpp:167> CLA-0 : REV 52127
downloadRequest - targetId: 0 - baseId: RNC-463, localMoid: WBTS-151 (0,0) 891ms pgrp=12491 pid=12491 tid=3012955024
uid=0 gid=0

Wed Aug 15 11:14:01 2012 #0006083028 OMS::TRACE DEBUG <BTSOMInterface.cpp:170> CLA-0 : REV 52127
downloadRequest - License: IE000284.XML (0,0) 891ms pgrp=12491 pid=12491 tid=3012955024 uid=0 gid=0

Wed Aug 15 11:14:01 2012 #0006083033 OMS::TRACE DEBUG <BTSOMInterface.cpp:127> CLA-0 : REV 52127 ReadFtpKeys -
sFtpUsername: omsFtpUser2, sFtpPassword: hg8rbsE8Z5D8c8X (0,0) 892ms pgrp=12491 pid=12491 tid=3012955024 uid=0
gid=0

and there is the fileLoadPrepare message being composed:

31
 For internal use

Wed Aug 15 11:14:01 2012 #0006083059 OMS::TRACE DEBUG <LicMgrBTSOMMsgHandler.cpp:244> CLA-0 : REV 52127
SendFileLoadPrepare - License file: /var/opt/OMSftproot/Nwi3LMAgent/145334/RNC-463_WBTS-151/IE000284.XML exist.
(0,0) 895ms pgrp=12491 pid=12491 tid=3000372112 uid=0 gid=0

In OMSBTSOMSrv_trace.txt it is possible to see the socket towards OMU being opened:

Wed Aug 15 11:14:52 2012 #0070073635 OMS::TRACE DEBUG <TcpEngine.cpp:878> CLA-0 : epoll_wait-
loop(2289892240)::modifyWanted, DataSocket: DataSocket: fd=7(this=0x7be03358): user data: [empty SocketUserData] peer:
10.200.68.164:62966 local: 10.200.68.167:8002 evs: events[read,write,err,hup] (0,0) 683ms pgrp=13247 pid=13247
tid=2289892240 uid=0 gid=0

and the message that will be sent to the wbts

Wed Aug 15 11:14:53 2012 #0070073651 OMS::TRACE DEBUG <CBTSOMProcSingleton.cpp:1205> CLA-0 :

CBTSOMProcSingleton::send_message, MSG: 30 80 a0 80 80 03 00 8e b1 81 02 01 cf 83 01 3f 00 00 a1 80 30 80 82 01 00 00 00
00 00 82 01 ff 83 01 06 00 00 (0,0) 069ms pgrp=13247 pid=13247 tid=2992438160 uid=0 gid=0

The hex dump following „MSG:“ can be matched to the hex dump visible in wireshark, it is
the actual BTSOM message sent to OMU. In the WBTS logs the download request would be
visible in the system module’s runtime log:

6a FCM-1011 <15.08 12:04:07.609144> 100F7 INF/LGC/GEN, HardHGW::handleData: file_load_prepare message received

6b FCM-1011 <15.08 12:04:07.609907> 100F0 INF/LGC/HGW, SoftHGW: FileLoadPrepare received. msn: 36887, protocol: HTTP,
fileType: licFile, fileId: /Nwi3LMAgent/145386/RNC-463_WBTS-151/IE000285.XML
7d FCM-1011 <15.08 12:04:07.746308> 10061 INF/LGC/SW_DL, FCM_Master: Starting to handle file IE000285.XML, it is
needed by: FSM1/FCM FSM2/FCM
7e FCM-1011 <15.08 12:04:07.746683> 100F0 INF/LGC/HGW, SoftHGW: evFileReq received. msn: 36887, filePath:
/Nwi3LMAgent/145386/RNC-463_WBTS-151/, fileName: IE000285.XML
7f FCM-1011 <15.08 12:04:07.747228> 10061 INF/LGC/SW_DL, FCM_Master: Starting to download file:
/Nwi3LMAgent/145386/RNC-463_WBTS-151/IE000285.XML
80 FCM-1011 <15.08 12:04:07.901383> 100FA INF/LGC/SW_DL,
TFtpTransfer::getFile(192.168.255.129,HTTP://omsFtpUser1:bmH9388Glo6bvT2@10.200.68.167:80/Nwi3LMAgent/145386/R
NC-463_WBTS-151/IE000285.XML,/ram/Tmp1.bin), which calls AaFileTftpGet() returned code = 0
81 FCM-1011 <15.08 12:04:07.902217> 10061 INF/LGC/SW_DL, FCM_Master: Download ok and distribution started
82 FCM-1011 <15.08 12:04:07.903554> 10061 INF/LGC/SW_DL, FCM_Master: Starting UDP file transfer.
a1 FCM-1011 <15.08 12:04:09.967287> 10061 INF/LGC/SW_DL, FCM_Master: File update reply received from remote slave:
FSM1/FCM (360053)
a2 FCM-1011 <15.08 12:04:09.967703> 10061 INF/LGC/SW_DL, FCM_Master: File distribution completed for file IE000285.XML
a3 FCM-1011 <15.08 12:04:09.967770> 10061 INF/LGC/SW_DL, FCM_Master: Distribution completed
a4 FCM-1011 <15.08 12:04:09.968168> 10061 INF/LGC/SW_DL, FCM_Master: File IE000285.XML saved to flash in unit
FSM2/FCM
a5 FCM-1011 <15.08 12:04:09.968518> 10061 INF/LGC/SW_DL, FCM_Master: No more files to download
a6 FCM-1011 <15.08 12:04:09.972021> 10061 INF/LGC/SW_DL, FCM_Slave: File: /rom/licences/IE000285.XML stored.

32
 For internal use

In the problem case faced, the messaging between OMS and NetAct could be seen in the
Northbound interface (IP 172.17.5.150):

Figure 21 Licence download request sent from NetAct

and also between OMU and OMS in the Southbound interface (172.17.29.22) the
fileLoadPrepare message is visible being sent to OMU with the correct WBTS ID, but the
connection attempt coming from the WBTSwas not seen in the trace:

Figure 22fileLoadPrepare sent from OMS to WBTS

This was again a firewall issue between WBTS and SOMS.

6.6. SOMS Internal Firewall misconfiguration

33
 For internal use

RNW Plan upload from the RNC to Netact was failing with a time out error. The configuration
was SOMS with single IP; 10.80.134.149 is the SOMS IP, 10.72.3.35 is the NetAct IP. Taking a
wireshark trace on the SOMS (in this configuration the IP and interface are the same for
North and Southbound):

Figure 23 TCP SYN packets opening HTTP connection from NetAct to SOMS

The SYN message that starts the http connection used to transfer the file from SOMS to
NetAct can be seen arriving to the SOMS, but there is no reply going out.

The httpd daemon was running correctly:

# ps -ef | grep http

nobody 1070 2982 0 Sep13 ? 00:00:01 /usr/sbin/httpd -DFOREGROUND -f /tmp/HTTPDPlat/conf/httpd.conf -k start
root 2964 11839 0 Sep13 ? 00:00:00 /bin/sh /usr/sbin/apachectl -DFOREGROUND -f /tmp/HTTPDPlat/conf/httpd.conf -k
start
root 2982 2964 0 Sep13 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND -f /tmp/HTTPDPlat/conf/httpd.conf -k start
nobody 2999 2982 0 Sep13 ? 00:00:01 /usr/sbin/httpd -DFOREGROUND -f /tmp/HTTPDPlat/conf/httpd.conf -k start
nobody 3000 2982 0 Sep13 ? 00:00:01 /usr/sbin/httpd -DFOREGROUND -f /tmp/HTTPDPlat/conf/httpd.conf -k start
nobody 3002 2982 0 Sep13 ? 00:00:01 /usr/sbin/httpd -DFOREGROUND -f /tmp/HTTPDPlat/conf/httpd.conf -k start
nobody 3003 2982 0 Sep13 ? 00:00:01 /usr/sbin/httpd -DFOREGROUND -f /tmp/HTTPDPlat/conf/httpd.conf -k start
nobody 3004 2982 0 Sep13 ? 00:00:01 /usr/sbin/httpd -DFOREGROUND -f /tmp/HTTPDPlat/conf/httpd.conf -k start
nobody 26446 2982 0 Sep13 ? 00:00:01 /usr/sbin/httpd -DFOREGROUND -f /tmp/HTTPDPlat/conf/httpd.conf -k start
root 31818 8868 0 10:45 pts/0 00:00:00 grep http

And the process was listening on port 80 (and 443 for https) :

# netstat -a | grep http

tcp 0 0 *:www-http *:* LISTEN
tcp 0 0 *:https *:* LISTEN

The opening of the connection could be simulated by manually attempting a telnet

connection choosing the port as 80:

C:\>telnet 172.17.5.151 80

Performing a telnet from the SOMS itself to the loopback address a connection on the port
can be opened:

34

Figure 24 SYN/ACK packets between loopback address in SOMS

Figure 25 Correct Telnet connection establishment to SOMS

 For internal use

Connecting to a working SOMS the message exchange is the following:

The problem was the IPSec configuration in the SOMS, which is stored in the file

/var/mnt/local/sysimg/sets/[…]/opt/Nokia/etc/IPSec/ipsec-policy.xml

and can be examined with the command

#ipmop -show-policy

By running the command zmodifyOMSHTTPandFTPPorts in the SOMS and enabling the HTTP
port the problem was cleared.

A tool that can also be useful in connectivity issues is the script

./opt/Nokia/SS_Inet/script/verify.sh

which tests outgoing transfers:

# ./verify.sh
====================
--- Start InitPOA!
ConfigPOA done
--- InitPOA done!
--- InitPOA succeeded!
--- ConfigORB done!
--- ConfigORB() succeeded!
TLSMode is "off"

Testing OMS file transfers

FTP :WORKING (should be enabled) :OK
SFTP :NOT WORKING (should be disabled) :OK
HTTP :WORKING (should be enabled) :OK

35
 For internal use

HTTPS :NOT WORKING (should be disabled)

:OK
====================

6.7. Recreating the (S)OMS Topology Database

There are situations where the Topology Upload fails in the SOMS without an upload
request even being sent to OMU; RNW operation such locking/unlocking cells take too long
or timeout before completing. In those cases there are two scripts are available to erase and
recreate the Topology Database:

/var/mnt/remote/sysimg/opt/Nokia/SS_Affe/script/oms_reinsta\ll_topology_db.sh

/opt/Nokia/SS_Affe/script/oms_reinstall_topology_db.sh

6.8. Increment installation could not be performed in a single step

When installing the corrs from the default directory described in the documentation
/root/swpack ,the installation failed. This is due to lack of space in the partition where this
directory is located /var/mnt/local/localimg. The problem was cleared by transferring
corrections to home/_nokfsoperator located in partition /var/mnt/local/sysimg/.

There is also a different known issue whereby increment installation may stall. The
workaround for this is described in the document “RNC OMS 1.0 Impact Document for 1.2
v2.0.pdf”, also part of part of the release notes for the RU30 base SW for standalone OMS.

7. References
a) Installing and Commissioning OMS.pdf
b) Integrating IPA-RNC.pdf
c) HP Integrated Lights-Out 2 User Guide.pdf

36
 For internal use

8. Appendixes

8.1. OMS Pronto Symptom List

Attached a list of symptoms required by R&D for (S)OMS cases:

Pronto checklist.doc

8.2. OMS MySQL Databases

The same MySQL databases present in the Integrated OMS are present in the Standalone
OMS. In the latter case, they will include data for multiple RNCs.

The running databases are

ps -ef | grep "mysqld"

root 2570 13639 0 12:20 pts/0 00:00:00 grep mysqld

520 12821 1 0 Jun20 ? 05:55:51 /opt/MySQL/sbin/mysqld --defaults-
file=/var/mnt/local/MySQL_DB_Alarm/my.cnf
507 13887 1 0 Jun20 ? 00:14:41 /opt/MySQL/sbin/mysqld --defaults-
file=/var/mnt/local/MySQL_DB_CosNaming/my.cnf
root 16236 1 0 Jun20 ? 00:09:36 /opt/MySQL/sbin/mysqld --defaults-
file=/var/mnt/local/MySQL_DB_CMPlan/my.cnf --user=root
root 16522 1 0 Jun20 ? 00:10:41 /opt/MySQL/sbin/mysqld --defaults-
file=/var/mnt/local/MySQL_DB_AidS/my.cnf --user=root
root 16728 1 0 Jun20 ? 00:15:07 /opt/MySQL/sbin/mysqld --defaults-
file=/var/mnt/local/MySQL_DB_PMData/my.cnf --user=root
root 17022 1 0 Jun20 ? 00:20:16 /opt/MySQL/sbin/mysqld --defaults-
file=/var/mnt/local/MySQL_DB_Topology/my.cnf --user=root

To log in to these databases, the user and password can be input in the following
manner:

/opt/MySQL/bin/mysql --database=db_aids --host=AidSDB --port=49898

--user=dbman --password=`fsgetcred db_aids dbman`

37
 For internal use

The port varies for each database

/opt/MySQL/bin/mysql --database=db_cmplan --host=CMPlanDB --port=49896

--user=dbman --password=`fsgetcred db_cmplan dbman`

/opt/MySQL/bin/mysql --database=db_pmdata --host=PMDataDB --port=49899

--user=dbman --password=`fsgetcred db_pmdata dbman`

/opt/MySQL/bin/mysql --host=AlarmDB --port=49711 --user=fsalarmagent

--password=`fsgetcred db_alarm fsalarmagent`

/opt/MySQL/bin/mysql --database=db_pmdata --host=PMDataDB --port=49899

--user=dbman --password=`fsgetcred db_pmdata dbman`

The tables that compose the database, in this case AlarmDB, can be examined with:

mysql> show tables;

+---------------------+
| Tables_in_db_cmplan |
+---------------------+
| cm_feedback_errors |
| cm_feedbacks |
| cm_operations |
| cm_plans |
| cm_plans_dns |
| cm_task_id_dns |
+---------------------+

6 rows in set (0.00 sec)

And then the description for each of the table:

#mysql> use cm_plans

#mysql> describe cm_plans;

+---------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------+---------+------+-----+---------+-------+
| plan_id | int(11) | NO | PRI | 0 | |
| task_id | int(11) | NO | PRI | 0 | |
+---------+---------+------+-----+---------+-------+

2 rows in set (0.00 sec)

38