You are on page 1of 44

Training

Back Office &


Troubleshooting
IWF (MSC Server)
April, 2017
IWF (MSC Server) System Back Office

Check Transaction Log


Check Incoming TPS of Each Site
PROCESS FLOW

Below is the process flow of IWF (MSC Server) Back Office:

DB

MME SGS LB-01 SGS-01

MME SGS LB-02 SGS-02

STP MAP MAP-01

STP MAP MAP-02

IBN HTTP OAM-1

IBN HTTP OAM-2


1. CHECK TRANSACTION LOG
1. CHECK TRANSACTION LOG

User can check Transaction Log that contains Detail Transaction from Call
Flow Request and Response.

Below are Call Flows on the IWF (MSC Server) System:

1.1 SGsAP-LOCATION-UPDATE-IMSI-ATTACH

1.2 SGsAP-NORMAL-LOCATION-UPDATE

1.3 SGsAP-EPS-DETACH

1.4 SGsAP-IMSI-DETACH

1.5 SEND NOTIFICATION IBN


CDR CONTENTS
Below is the CDR log example: CDR File Name Format

[root@MSCsJKTSGS1 cdr]# more cdr_2017032815.txt


02:30:15 io0 [/10.45.64.97:60000 sn:1 bs:114 ppid:0][assoc id:160 in:4 out:4] msg:9 upd:2
addr:510098850396241 imei:866178020798846
02:30:15 io1 [/10.45.64.97:60000 sn:1 bs:18 ppid:0][assoc id:160 in:4 out:4] msg:10 upd:2
addr:510098850396241
02:30:15 io0 [/10.45.64.81:60000 sn:1 bs:114 ppid:0][assoc id:162 in:4 out:4] msg:9 upd:2
addr:510098010291026 imei:868477021660742

CDR Format:
Field Name Type Description
time datetime (hh:ii:ss) Datetime of the event sent to IWF
transaction String Transaction type
Internet protocol (IP) : port Source IP
host_source
e.g., [/10.12.12.178:35611
sn String Stream number
bs String Byte size
ppid String Payload id
id String Assoc id
in String Max stream number in
out String Max stream number out
msg_type String Message type
location_update String Location update type
imsi String IMSI number
imei String IMEI number
CDR CONTENTS

Below is the description for each value on the CDR log:

Field Name Value Description


Io0 request
transaction Io1 response
Io2 disconnected
upd:1 IMSI Attach
location_update
upd:2 Normal Location Update
msg:1 SGsAP-PAGING-REQUEST
msg:2 SGsAP-PAGING-REJECT
msg:6 SGsAP-SERVICE-REQUEST
msg:7 SGsAP-DOWNLINK-UNITDATA
msg:8 SGsAP-UPLINK-UNITDATA
msg:9 SGsAP-LOCATION-UPDATE-REQUEST
msg_type
msg:10 SGsAP-LOCATION-UPDATE-ACCEPT
msg:11 SGsAP-LOCATION-UPDATE-REJECT
msg:12 SGsAP-TMSI-REALLOCATION-COMPLETE
msg:13 SGsAP-ALERT-REQUEST
msg:14 SGsAP-ALERT-ACK
msg:15 SGsAP-ALERT-REJECT
CDR CONTENTS

Below is the description for each value on the CDR log:

Field Name Value Description


msg:16 SGsAP-UE-ACTIVITY-INDICATION
msg:17 SGsAP-EPS-DETACH-INDICATION
msg:18 SGsAP-EPS-DETACH-ACK
msg:19 SGsAP-IMSI-DETACH-INDICATION
msg:20 SGsAP-IMSI-DETACH-ACK
msg:21 SGsAP-RESET-INDICATION
msg_type
msg:22 SGsAP-RESET-ACK
msg:23 SGsAP-SERVICE-ABORT-REQUEST
msg:26 SGsAP-MM-INFORMATION-REQUEST
msg:27 SGsAP-RELEASE-REQUEST
msg:29 SGsAP-STATUS
msg:31 SGsAP-UE-UNREACHABLE
1. CHECK TRANSACTION LOG

1.1 SGsAP-LOCATION-UPDATE-IMSI-ATTACH

a. Check Request
Check CDR Log from LOCATION-UPDATE-IMSI-ATTACH Request with command
below:
tail -f /project_iwf/cdr/cdr_YYYYMMDDHHtxt |grep "msg:9 upd:1

Result:
[root@MSCsJKTSGS1 ~]# tail -f /project_iwf/cdr/cdr_2017042013.txt|grep "msg:9 upd:1"13:26:19 io0
[/10.45.64.113:60000 sn:1 bs:114 ppid:0][assoc id:161 in:4 out:4] msg:9 upd:1 addr:510098000749713
imei:868455020614639
13:26:19 io0 [/10.45.64.113:60000 sn:1 bs:114 ppid:0][assoc id:161 in:4 out:4] msg:9 upd:1
addr:510095000024819 imei:862798030068270
13:26:19 io0 [/10.45.64.97:60000 sn:1 bs:114 ppid:0][assoc id:160 in:4 out:4] msg:9 upd:1
addr:510281981397801 imei:869123021147806
13:26:19 io0 [/10.45.64.65:60000 sn:1 bs:114 ppid:0][assoc id:159 in:4 out:4] msg:9 upd:1
addr:510098000798873 imei:860280031250142
13:26:19 io0 [/10.45.64.97:60000 sn:1 bs:114 ppid:0][assoc id:160 in:4 out:4] msg:9 upd:1
addr:510289951541265 imei:869123021157631
1. CHECK TRANSACTION LOG

1.1 SGsAP-LOCATION-UPDATE-IMSI-ATTACH

b. Check Accept
Check CDR Log from LOCATION-UPDATE-IMSI-ATTACH Accept with command
below:

tail -f /project_iwf/cdr/cdr_YYYYMMDDHHtxt |grep "msg:10 upd:1"

Result:
[root@MSCsJKTSGS1 ~]# tail -f /project_iwf/cdr/cdr_2017042013.txt|grep "msg:10 upd:1"13:27:16 io1
[/10.45.64.97:60000 sn:1 bs:18 ppid:0][assoc id:160 in:4 out:4] msg:10 upd:1 addr:510098841697553
13:27:16 io1 [/10.45.64.113:60000 sn:1 bs:18 ppid:0][assoc id:161 in:4 out:4] msg:10 upd:1
addr:510098841515692
13:27:16 io1 [/10.45.64.65:60000 sn:1 bs:18 ppid:0][assoc id:159 in:4 out:4] msg:10 upd:1
addr:510095001261999
13:27:16 io1 [/10.45.64.113:60000 sn:1 bs:18 ppid:0][assoc id:161 in:4 out:4] msg:10 upd:1
addr:510095000324722
13:27:16 io1 [/10.45.64.81:60000 sn:1 bs:18 ppid:0][assoc id:162 in:4 out:4] msg:10 upd:1
addr:510091200880970
1. CHECK TRANSACTION LOG

1.2 SGsAP-NORMAL-LOCATION-UPDATE
a. Check Request
Check CDR Log from NORMAL-LOCATION-UPDATE-Request with command below:
tail -f /project_iwf/cdr/cdr_YYYYMMDDHHtxt |grep "msg:9 upd:2"

Result:
[root@MSCsJKTSGS1 ~]# tail -f /project_iwf/cdr/cdr_2017042013.txt|grep "msg:9 upd:2"13:27:54 io0
[/10.45.64.65:60000 sn:1 bs:114 ppid:0][assoc id:159 in:4 out:4] msg:9 upd:2 addr:510091103728108
imei:868477023648166
13:27:54 io0 [/10.45.64.81:60000 sn:1 bs:114 ppid:0][assoc id:162 in:4 out:4] msg:9 upd:2
addr:510098811139740 imei:868477026100298
13:27:54 io0 [/10.45.64.65:60000 sn:1 bs:114 ppid:0][assoc id:159 in:4 out:4] msg:9 upd:2
addr:510091261465624 imei:866263026240538
13:27:54 io0 [/10.45.64.113:60000 sn:1 bs:114 ppid:0][assoc id:161 in:4 out:4] msg:9 upd:2
addr:510098001112245 imei:860280030232851
13:27:54 io0 [/10.45.64.65:60000 sn:1 bs:114 ppid:0][assoc id:159 in:4 out:4] msg:9 upd:2
addr:510289910598408 imei:869123020352605
13:27:54 io0 [/10.45.64.97:60000 sn:1 bs:114 ppid:0][assoc id:160 in:4 out:4] msg:9 upd:2
addr:510289961476697 imei:868236020083144
1. CHECK TRANSACTION LOG

1.2 SGsAP-NORMAL-LOCATION-UPDATE

b. Check Accept
Check CDR Log from NORMAL-LOCATION-UPDATE-Accept with command below:
tail -f /project_iwf/cdr/cdr_YYYYMMDDHHtxt |grep "msg:10 upd:2"

Result:
[root@MSCsJKTSGS1 ~]# tail -f /project_iwf/cdr/cdr_2017042013.txt|grep "msg:10 upd:2
13:29:01 io1 [/10.45.64.113:60000 sn:1 bs:18 ppid:0][assoc id:161 in:4 out:4] msg:10 upd:2
addr:510091200852578
13:29:01 io1 [/10.45.64.113:60000 sn:1 bs:18 ppid:0][assoc id:161 in:4 out:4] msg:10 upd:2
addr:510098811599318
13:29:01 io1 [/10.45.64.65:60000 sn:1 bs:18 ppid:0][assoc id:159 in:4 out:4] msg:10 upd:2
addr:510098841170580
13:29:01 io1 [/10.45.64.81:60000 sn:1 bs:18 ppid:0][assoc id:162 in:4 out:4] msg:10 upd:2
addr:510289970386992
13:29:01 io1 [/10.45.64.65:60000 sn:1 bs:18 ppid:0][assoc id:159 in:4 out:4] msg:10 upd:2
addr:510098850583704
1. CHECK TRANSACTION LOG

1.3 SGsAP-EPS-DETACH

a. Check Request
Check CDR Log from EPS-DETACH-INDICATION with command below:

tail -f /project_iwf/cdr/cdr_YYYYMMDDHHtxt |grep "msg:17"

Result:
[root@MSCsSBYSGS1 ~]# tail -f /project_iwf/cdr/cdr_2017031916.txt |grep "msg:9 upd:2"
16:34:49 io0 [/10.46.64.81:60000 sn:1 bs:104 ppid:0][assoc id:23 in:4 out:4] msg:9 upd:2 addr:510289961555109
16:34:49 io0 [/10.46.64.81:60000 sn:1 bs:104 ppid:0][assoc id:23 in:4 out:4] msg:9 upd:2 addr:510098001585873
16:34:49 io0 [/10.46.64.81:60000 sn:1 bs:104 ppid:0][assoc id:23 in:4 out:4] msg:9 upd:2 addr:510098001280388
16:34:49 io0 [/10.46.64.97:60000 sn:1 bs:104 ppid:0][assoc id:24 in:4 out:4] msg:9 upd:2 addr:510095000596175
16:34:50 io0 [/10.46.64.81:60000 sn:1 bs:104 ppid:0][assoc id:23 in:4 out:4] msg:9 upd:2 addr:510098811523002
1. CHECK TRANSACTION LOG

1.3 SGsAP-EPS-DETACH

b. Check Accept
Check CDR Log from EPS-DETACH- ACK with command below:

tail -f /project_iwf/cdr/cdr_YYYYMMDDHHtxt |grep "msg:18"

Result:
[root@MSCsSBYSGS1 ~]# tail -f /project_iwf/cdr/cdr_2017031916.txt |grep "msg:18"
16:37:06 io1 [/10.46.64.113:60000 sn:1 bs:11 ppid:0][assoc id:25 in:4 out:4] msg:18 addr:510095000151603
16:37:06 io1 [/10.46.64.97:60000 sn:1 bs:11 ppid:0][assoc id:24 in:4 out:4] msg:18 addr:510091200347113
16:37:07 io1 [/10.46.64.97:60000 sn:1 bs:11 ppid:0][assoc id:24 in:4 out:4] msg:18 addr:510098001683619
16:37:07 io1 [/10.46.64.97:60000 sn:1 bs:11 ppid:0][assoc id:24 in:4 out:4] msg:18 addr:510091200487399
16:37:08 io1 [/10.46.64.97:60000 sn:1 bs:11 ppid:0][assoc id:24 in:4 out:4] msg:18 addr:510091261606664
1. CHECK TRANSACTION LOG

1.4 SGsAP-IMSI-DETACH

a. Check Request
Check CDR Log from IMSI-DETACH-INDICATION with command below:

tail -f /project_iwf/cdr/cdr_YYYYMMDDHHtxt |grep "msg:19"

Result:
[root@MSCsSBYSGS1 ~]# tail -f /project_iwf/cdr/cdr_2017031916.txt |grep "msg:19"
16:37:42 io0 [/10.46.64.97:60000 sn:1 bs:71 ppid:0][assoc id:24 in:4 out:4] msg:19 addr:510091200060770
16:37:42 io0 [/10.46.64.97:60000 sn:1 bs:71 ppid:0][assoc id:24 in:4 out:4] msg:19 addr:510281981838374
16:37:42 io0 [/10.46.64.113:60000 sn:1 bs:71 ppid:0][assoc id:25 in:4 out:4] msg:19 addr:510098840876289
16:37:43 io0 [/10.46.64.81:60000 sn:1 bs:71 ppid:0][assoc id:23 in:4 out:4] msg:19 addr:510098010192498
16:37:43 io0 [/10.46.64.97:60000 sn:1 bs:71 ppid:0][assoc id:24 in:4 out:4] msg:19 addr:510289961408303
1. CHECK TRANSACTION LOG

1.4 SGsAP-IMSI-DETACH

b. Check Accept
Check CDR Log from IMSI-DETACH-ACK with command below:

tail -f /project_iwf/cdr/cdr_YYYYMMDDHHtxt |grep "msg:20"

Result:
[root@MSCsSBYSGS1 ~]# tail -f /project_iwf/cdr/cdr_2017031916.txt |grep "msg:20"
16:38:27 io1 [/10.46.64.65:60000 sn:1 bs:11 ppid:0][assoc id:19 in:4 out:4] msg:20 addr:510098000393823
16:38:28 io1 [/10.46.64.97:60000 sn:1 bs:11 ppid:0][assoc id:24 in:4 out:4] msg:20 addr:510095000684602
16:38:29 io1 [/10.46.64.113:60000 sn:1 bs:11 ppid:0][assoc id:25 in:4 out:4] msg:20 addr:510098811735134
16:38:30 io1 [/10.46.64.81:60000 sn:1 bs:11 ppid:0][assoc id:23 in:4 out:4] msg:20 addr:510095000514079
1. CHECK TRANSACTION LOG

1.5 SEND NOTIFICATION IBN

a. Check Log
To check Send Notification IBN log, use this following command :

tail f /var /log/send_notification_YYYY-MM-DD.log in VM MSCsJKTOAM1


1. CHECK TRANSACTION LOG

1.5 SEND NOTIFICATION IBN

Result : Request

2017-03-15 17:37:36 Request : <soapenv:Envelope


xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:urn="urn:provisioning"><soapenv:Header/><soapenv:Body><urn:provisioningAddIbnWhitelist><data
><username>iwf</username><password>iwfsmart2017</password><customer_name>iwf</customer_na
me><campaign_id>iwf_074_03201713092924_20170313</campaign_id><imsi>510281981872726</imsi><
advertising_content>&lt;img src=&quot;%PATH%132_15rb_Desktop.jpg&quot; height=%HEIGHT%
width=%WIDTH%&gt;&lt;/img&gt;</advertising_content></data></urn:provisioningAddIbnWhitelist></soa
Response
penv:Body></soapenv:Envelope>

2017-03-15 17:37:36 Response : <?xml version="1.0" encoding="ISO-8859-1"?><SOAP-ENV:Envelope


xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-
instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-
ENV:Body><ns1:provisioningAddIbnWhitelistResponse
xmlns:ns1="urn:provisioning"><return><result>SUCCESS</result><description>success add white
list</description><error_code></error_code></return></ns1:provisioningAddIbnWhitelistResponse></SOA
P-ENV:Body></SOAP-ENV:Envelope>
2. CHECK INCOMING TPS OF EACH SITE
2.1 INCOMING TPS ON JKT SITE
User can check the incoming TPS on Jakarta site

To check incoming TPS on JKT Site, use this following command on VM MSCsJKTLB1:
iwf_start.sh display

Result:
2.2 INCOMING TPS ON SBY SITE
User can check the incoming TPS of Surabaya site

To check incoming TPS on SBY Site, use this following command on VM MSCsSBYLB1:
iwf_start.sh display

Result:
IWF (MSC Server) System Troubleshooting

Check Application Status


Check Daily Maintenance
Trace and Correct Faults
1. CHECK APPLICATION STATUS
1. CHECK APPLICATION STATUS

1.1 LB Module Application on VM MSCsLB1

a. Check Status
To check status use this following command:
iwf_start.sh status

b. Stop Application
To stop application, use this following command:
iwf_start.sh Stop

c. Start Application
To start application, use this following command:
iwf_start.sh Start

d. Restart Application
To restart application, use this following command:
iwf_start.sh restart
1. CHECK APPLICATION STATUS

1.1 LB Module Application on VM MSCsLB1

e. Check Application Log


To check application log, use this following command:
iwf_start.sh display

Result:
1. CHECK APPLICATION STATUS

1.1 LB Module Application on VM MSCsLB1


e. Check Log Transaction
To check transaction log, use this following command:
tail -f /project_iwf/screen/scr_YYYYMMDDHH.txt

Result:

RX : Incoming
Request
TX : Response given
by IWF (MSC Server)
to MME
1. CHECK APPLICATION STATUS

1.2 SGS Module Application on VM MSCsJKTSGS1

a. Check Status
To check status use this following command:
iwf_start.sh status

b. Stop Application
To stop application, use this following command:
iwf_start.sh Stop

c. Start Application
To start application, use this following command:
iwf_start.sh Start

d. Restart Application
To restart application, use this following command:
iwf_start.sh restart
1. CHECK APPLICATION STATUS

1.2 SGS Module Application on VM MSCsJKTSGS1

e. Check Application Log


To check application log, use this following command:
iwf_start.sh display

Result:
1. CHECK APPLICATION STATUS

1.2 SGS Module Application on VM MSCsJKTSGS1


e. Check Incoming CDR Log
To check Incoming CDR log, use this following command:
tail -f /project_iwf/cdr/cdr_YYYYMMDDHH.txt

Result:
2. CHECK DAILY MAINTENANCE
2. CHECK DAILY MAINTENANCE

2.1 Monitoring on NMS Browser

User can monitor Network Elements of IWF (MSC Server) System on the NMS browser:

If theres excess load (overload) on


each element from IWF, it will send
alerts through the symbols of each
element in color.

Minor Alert

Major Alert

Critical Alert
2. CHECK DAILY MAINTENANCE

2.2 Check Log of LB Module Application

To check Log of LB Module Application, use this following command:


iwf_start.sh display

And below is the Log of LB Module Application :


2. CHECK DAILY MAINTENANCE

2.2 Check Log of LB Module Application

Normal value for queue thread is 0, if the value keeps increasing, it means theres a problem
on the application.
For example:

INFO 17:20:25,017 [InfoTh] - 0=0 1=0 2=0 3=0 4=0 5=0 6=0 7=0 tot=0

On the sample log above, all the value is 0. It means no problem found on the application.

Check the amount of connected assocs for TPS. The normal Condition amount for TPS is more
than 0.
For example:
assocs=[/10.45.64.97:60000][assoc id:64 maxIn:4 maxOut:4][uptime:16h 43m 58s][ins:0
outs:0][pps:690 tps:345]

On the sample log above, the TPS value is more than 0, it means that the TPS condition is
normal.
2. CHECK DAILY MAINTENANCE

2.3 Check Log of SGS Module Application

To check Log of SGS Module Application, use this following command:


iwf_start.sh display

And below is the Log of SGS Module Application :


2. CHECK DAILY MAINTENANCE

2.3 Check Log of SGS Module Application

Normal value for queue thread is 0, if the value keeps increasing, it means theres a problem
on the application.
For example:
INFO 19:39:05,015 [InfoTh] - 0=0 1=0 2=0 3=0 4=0 5=0 6=0 7=0 tot=0
On the sample log above, all the value is 0. It means no problem found on the application:

Check the amount of incoming TPS, normal condition for TPS amount is more than 0.
On this log below the TPS amount is more than 0, it means that the TPS condition is normal.

assocs=[/192.168.1.16:12345, /10.17.151.90:12345, /192.168.1.19:12345, /10.17.151.92:12345,


/10.17.151.132:12345, /10.17.151.134:12345][assoc id:23 maxIn:10 maxOut:10][uptime:18h 23m
26s][ins:0 outs:0][pps:2401 tps:1200]
3. TRACE AND CORRECT FAULTS
3. TRACE AND CORRECT FAULTS

To trace and correct faults on the IWF (MSC Server) System, user can
perform these following activities:

3.1 Check Anomaly on LB Module Application

3.2 Check Anomaly on SGS Module Application

3.3 Check Incoming IMSI


3. TRACE AND CORRECT FAULTS

3.1 Check Anomaly on LB Module Application

To check anomaly on module LB application (increasing thread), use this following command
on VM MSCsJKTLB01:
iwf_start.sh display

And below is the LB Module Application Log:


3. TRACE AND CORRECT FAULTS

3.1 Check Anomaly on LB Module Application


Check CPU Usage on IWF_MODULE Application on VM MSCsJKTLB01:

And if on the Thread log, the value of buffer keeps increasing and CPU Usage of IWF is high,
LB Module Application needs to be restart using this following command:
iwf_start.sh restart
3. TRACE AND CORRECT FAULTS

3.2 Check Anomaly on SGS Module Application

To check Anomaly on Module SGS Application (Increasing Thread), use this following
command on VM MSCsJKTSGS01:
iwf_start.sh display

And below is the SGS Module Application Log:

And then check the value of thread on the log below:


INFO 19:39:05,015 [InfoTh] - 0=0 1=0 2=0 3=0 4=0 5=0 6=0 7=0 tot=0
3. TRACE AND CORRECT FAULTS

3.2 Check Anomaly on SGS Module Application


Check CPU Usage on IWF_MODULE Application on VM MSCsJKTSGS01:

If on the Thread log, the value of buffer keeps increasing and CPU Usage of IWF is high,
SGS Module Application needs to be restart using this following command:
iwf_start.sh restart
3. TRACE AND CORRECT FAULTS

3.3 Check Incoming IMSI

Check incoming CDR Log on VM MSCsJKTSGS01 using this command below:


tail -f /project_iwf/cdr/cdr_YYYYMMDDHH.txt |grep IMSI
Log Result:

Check via Wireshark application using command below:


tshark -i eth2 -w /root/capture/name_file.cap
Log Result:
3. TRACE AND CORRECT FAULTS

3.3 Check Incoming IMSI

Open using Wireshark application:

You might also like