NMS Redudancy and Failover - IDX 3.3 Rev B

NMS Redundancy and Failover
iDX Release 3.3
Technical Notes
November 20, 2014

Copyright 2014, Inc. All rights reserved. Reproduction in whole or in part without permission is prohibited.
Information contained herein is subject to change without notice. The specifications and information regarding the
products in this document are subject to change without notice. All statements, information and recommendations
in this document are believed to be accurate, but are presented without warranty of any kind, express, or implied.
Users must take full responsibility for their application of any products. Trademarks, brand names and products
mentioned in this document are the property of their respective owners. All such references are used strictly in an
editorial fashion with no intent to convey any affiliation with the name or the product's rightful owner.
A Family of iDirect Companies
iDirect, a subsidiary of VT Systems, is a global leader in IP-based satellite communications providing technology that
enables our partners to optimize their networks, differentiate and expand their businesses. The iDirect Intelligent
Platform allows our partners to run their entire business operations more efficiently via a single, unified IP-based
satellite architecture, whether it's providing core IP applications to the enterprise or specialized services to any
number of diverse vertical markets. iDirect is the #1 name in global satellite communications in key industries
including maritime, military/government, and oil and gas.
iDirect
Company web site: www.idirect.net ~ Main Phone: 703.648.8000
TAC Contact Information: Phone: 703.648.8151 ~ Email: tac@idirect.net ~ Web site: tac.idirect.net
iDirect Government Technologies (iGT), created in 2007, is a wholly owned subsidiary of iDirect and was formed to
better serve the U.S. government and defense communities.
iDirect Government Technologies

Company web site: http: www.idirectgt.com ~ Main Phone: 703.648.8118
TAC Contact Information: Phone: 703.648.8111 ~ Email: tac@idirectgt.com ~ Web site: tac.idirectgt.com
iDirect Asia Pte Ltd was established in Singapore during 2008 to enhance iDirects value-add and responsiveness to
customers in the Asia Pacific region.
iDirect Asia Pte Ltd

Company web site: www.stengg.com ~ Main Phone: 65.6521.7888
TAC Contact Information: Phone: 703.648.8151 ~ Email: tac@idirect.net ~ Web site: tac.idirect.net
Document Name: TN_NMS_Redundancy_3.3_Rev B_112014.pdf

Document Part Number: T0000605
ii NMS Redundancy and Failover

iDX Release 3.3
Revision History
The following table shows all revisions for this document. To determine if this is the latest
revision, check the TAC Web site. Refer to Getting Help on page x for TAC access information.
Revision Date Updates

A 7/31/2014 Initial release of document.
B 11/20/2014 Updated the document template to use the Co-branded template.
Replaced all Warning messages with Caution messages to adhere to
the document convention.
Corrected an error in a command syntax for backing up the NMS
database.
Network Upgrade Procedure Guide for iDX Release 3.0.0.x and Later to iii
iDX Release 3.3.x.x
Revision History
iv Network Upgrade Procedure Guide for iDX Release 3.0.0.x and Later to
iDX Release 3.3.x.x
Contents
Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
About . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Purpose. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Document Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Getting Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x
iDirect Contact Information: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x
iGT Contact Information: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Document Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
1 Single NMS Redundancy and Failover . . . . . . . . . . . . . . . . . . . . . . . . . . .1

1.1 Configuring Single NMS Server Redundancy. . . . . . . . ... ... ... ... .... . . .1
1.1.1 Configuring the CRONTAB on the Servers . . . . . . . . ... ... ... ... .... . . .1
1.1.1 Configuring the CRONTAB on the Primary NMS . . ... ... ... ... .... . . .2
7. Configuring the crontab on the Backup NMS . . . . . . ... ... ... ... .... . . .3
1.1.2 Verify That the System Backup is Working . . . . . . . ... ... ... ... .... . . .3
1.2 Configuring Single NMS Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.1 Configuring the Network Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Distributed NMS Redundancy and Failover . . . . . . . . . . . . . . . . . . . . . . . 9

2.1 Distributed NMS Redundancy and Replication . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Configuring Distributed NMS Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.1 Configuring the CRONTAB on the Servers . . . . . . . . . . . . . . . . . . . . . . . . . . 10
NMS Redundancy and Failover v

iDX Release 3.3
Contents
2.2.1 Configuring the crontab on the Primary NMS . . . . . . . . . . . . . . . . . . . . . 10

7. Configuring the crontab on NMS 2 and NMS 3 . . . . . . . . . . . . . . . . . . . . . . . 11
5. Configuring the crontab on the Backup NMS . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.2 Verify That the System Backup is Working . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Configuring Distributed NMS Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.1 Preparing the Backup NMS Server to Take Over . . . . . . . . . . . . . . . . . . . . . . 13
2.3.2 Configuring Backup to Take Over for Primary . . . . . . . . . . . . . . . . . . . . . . . 14
2.3.3 Backup NMS Server Takes Over for an Auxiliary Server . . . . . . . . . . . . . . . . . 14
vi NMS Redundancy and Failover

iDX Release 3.3
Figures
Figure 1-1. ifcfg-eth0 Results Display. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Figure 1-2. Network File Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Figure 1-3. Etc/Hosts File Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
NMS Redundancy and Failover vii

iDX Release 3.3
Tables
Table 1-1. ifcfg-eth0 File Entries ........................................................................... 5

Table 1-2. Network File Commands ........................................................................ 7
NMS Redundancy and Failover viii

iDX Release 3.3
About
Purpose
This technical note presents the following procedures:
Configuring the crontab files for redundancy in a single NMS network.
Configuring the crontab files for redundancy in a distributed NMS network.
Bringing the Backup NMS server online in single NMS network.
Bringing the Backup NMS server online in a distributed NMS network.
Audience
This technical note is written for network operators using the iDirect iDX Release 3.3 software
suite (iBuilder/iMonitor/iSite), network architects, and any other personnel who operate or
monitor an iDirect network. A basic knowledge of TCP/IP concepts, satellite communications,
Linux and MS Windows operating systems and tools (including WinSCP and PuTTY) is assumed.
Prior experience with the operation and monitoring of an iDirect network is desirable but not
required.
Contents
This document contains the following major sections:
Single NMS Redundancy and Failover
Distributed NMS Redundancy and Failover
Document Conventions
This section illustrates and describes the conventions used throughout this document.
Convention Description Example

Command Used when the user is required to Enter the command:
enter a command at a command cd /etc/snmp/
line prompt or in a console.
NMS Redundancy and Failover ix

iDX Release 3.3
About
Convention Description Example

Terminal Used when showing resulting crc report all
Output output from a command that was 8350.3235 : DATA CRC [ 1]
entered at a command line or on a 8350.3502 : DATA CRC [5818]
console. 8350.4382 : DATA CRC [ 20]
Screen Used when referring to text that 1. To add a remote to an inroute group, right-click
Reference appears on the screen on a the Inroute Group and select Add Remote.
Graphical User Interface (GUI). The Remote dialog box has a number of user-
Used when specifying names of selectable tabs across the top. The Information
commands, menus, folders, tabs, tab is visible when the dialog box opens.
dialogs, list boxes, and options.
Hyperlink Used to show all hyperlinked text For instructions on adding a line card to the
within a document or external network tree, see Adding a Line Card on
links such as web page URL. page 108.
WARNING: A Warning highlights an essential operating or maintenance

procedure, practice, condition, or statement which, if not strictly observed,
could result in injury, death, or long term health hazards.
CAUTION: A Caution highlights an essential operating or maintenance procedure,

practice, condition, or statement which, if not strictly observed, could result in
damage to, or destruction of, equipment or a condition that adversely affects
system operation.
NOTE: A Note is a statement or other notification that adds, emphasizes, or

clarifies essential information of special importance or interest.
Getting Help
The iDirect Technical Assistance Center (TAC) and the iDirect Government Technologies
Technical Assistance Center (iGT TAC) are available to provide assistance 24 hours a day, 365
days a year. Software user guides, installation procedures, FAQs, and other documents that
support iDirect products are available on the respective TAC Web site.
iDirect Contact Information:

Web site: http://tac.idirect.net
Telephone: (703) 648-8151
E-mail: tac@idirect.net
For sales or product purchasing information contact iDirect Corporate Sales at:
Telephone: (703) 648-8000
E-mail: sales@idirect.net
x NMS Redundancy and Failover

iDX Release 3.3
About
iGT Contact Information:

Web site: http://tac.idirectgt.com
Telephone: (703) 648-8111
E-mail: tac@idirectgt.com
iDirect and iGT produce documentation that is technically accurate, easy to use, and helpful
to our customers. Please assist us in improving this document by providing feedback. Send
comments to:
iDirect: techpubs@idirect.net
iGT: techpubs@idirectgt.com
Document Set
The following iDirect documents are available on the TAC Web site and contain information
relevant to installing and using iDirect satellite network software and equipment.
iDX 3.3 Release Notes
iDX 2.3 and Later to iDX 3.3 Network Upgrade Procedure
iDX iBuilder User Guide
iDX iMonitor User Guide
iDX Technical Reference Guide
iDX Satellite Router Commissioning and Installation Guide
iDX Features and Chassis Licensing Guide
iDX Upgrade/New Installation Survey
iDX Link Budget Analysis Guide
iDX Hardware Matrix
iDX Software Release Matrix
NMS Redundancy and Failover xi

iDX Release 3.3
About
xii NMS Redundancy and Failover

iDX Release 3.3
1 Single NMS Redundancy
and Failover
This chapter describes how to configure the Primary NMS to back up the databases and how to
bring the Backup NMS server online in the event that the Primary NMS fails. It includes the
following major topics:
Configuring Single NMS Server Redundancy
Configuring Single NMS Failover
Beginning with iDX Release 3.1, iDirect supports near real-time replication of NMS databases
from the Primary NMS server to one or more Backup NMS servers in addition to the traditional
idsBackup and idsRestore. When the NMS Database Replication feature is enabled, changes to
the NMS databases on the Primary NMS are automatically replicated to the Backup NMS server
and the idsBackup script is automatically configured to run on the Backup server. For a
complete description of the NMS Database Replication feature, including configuration, see
the Technical Reference Guide for iDX Release 3.3.
If the NMS Database Replication feature is currently being used, the sections in this document
on configuring single and distributed NMS redundancy are not applicable. However, there are
slight differences to the procedures to bring the Backup NMS server on line in case of a
Primary NMS failure. These differences are noted in Configuring Single NMS Failover on page 4
and in Configuring Distributed NMS Failover on page 13.
1.1 Configuring Single NMS Server Redundancy

This section describes how to configure redundancy in a single NMS network.
NOTE: This procedure is not applicable if NMS Database Replication feature is

enabled on the NMS Servers.
1.1.1 Configuring the CRONTAB on the Servers

The crontab file must be configured in order to correctly run the idsBackup and idsRestore
scripts.
NMS Redundancy and Failover 1

iDX Release 3.3
Configuring the CRONTAB on the Primary NMS

Perform the following on the Primary NMS:
1. Log on to the Primary NMS and switch to root.
2. Edit the crontab file by entering the command:
crontab -e
The crontab file opens in the Vi editor.
3. On the Primary NMS, remove the comment symbol (#) from the beginning of all lines in
the crontab file, except for this line:
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --
keep 2
4. Add the following line to the crontab file, replacing the IP address shown with the Backup
NMS servers address, and, if desired, substituting the filename tokens shown with your
own:
30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --inst
idirect@192.168.77.33:/var/idirect/backup/\%ip-\%date --keep 2
NOTE: The idsBackup command creates a local backup, and when using the
--inst option, it creates a remote backup and installs it on the specified
NMS.
NOTE: A forward slash character (\) is required before the token %

character for the crontab to execute properly. For example, use \%ip and
not %ip
5. Optionally, change the time and frequency with which backups are performed by
modifying the string 30 0 * * * (highlighted above). This is a five-field string set by
default to run a backup every night at 12:30am. The time and frequency of the backups
can be modified by changing these fields according to the following definitions. The fields
below are numbered left to right.
Field 1 (00 by default): Minute of the specified Hour (0 - 59)
Field 2 (1 by default): Hour of the Day (0 - 23)
Field 3 (* by default): Day of the Month (1 - 31)
Field 4 (* by default): Month (1 - 12)
Field 5 (* by default): Day of the week (0 - 6, with Sunday = 0)
6. Optionally, the number of local backup copies kept can be changed by modifying --keep
2, for example, to --keep 1 to save disk space.
7. Save changes to the file and exit by pressing Esc and entering:
:wq!
An example of an edited crontab file for the Primary NMS is shown below.
0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >>
/opt/idirect/nms/utils/db_maint/cons.output 2>&1
0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >>
/opt/idirect/nms/utils/db_maint/check_db.output 2>&1
keep 2
2 NMS Redundancy and Failover

iDX Release 3.3

Configuring the crontab on the Backup NMS

Perform the following steps on the Backup NMS:
1. Log on to the Backup NMS and switch root.
crontab -e
3. Remove the comment symbol (#) from the following line:
This command performs a recommended, daily maintenance check on the database.
4. Ensure comment symbols (#) appear in front of the following lines:
#0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >>
keep 2
:wq!
An example of an edited crontab file for the Backup NMS is shown below:
keep 2
1.1.2 Verify That the System Backup is Working

This procedure provides all necessary steps to verify the system backup script is working
condition and that the appropriate backups are made on the Backup NMS server.
The CRONTAB time setting can be modified to run the commands 1-3 minutes after the
current time. Be sure to check the idsBackup.log to see if the backup and install is running
properly.
Perform the following:
1. On the Primary NMS, log on as idirect and switch to root.
2. Enter the following command to push SSH keys to the Backup NMS:
pushSSHKeys <Backup NMS IP address>
The following sample output displays:
Pushing SSH key for user root to idirect@172.20.4.253:

iDX Release 3.3
The authenticity of host 172.20.4.253 (172.20.4.253) cant be

established.
RSA key fingerprint is 3e:28:f2:a5:50:22:a5:2f:51:33:69:39:ca:a4:8d:63.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 172.20.4.253 (RSA) to the list of known hosts.
idirect@172.20.4.253s password: remote_users_password
SSH key for user root successfully pushed to idirect@172.20.4.253
3. On the Primary NMS, get the current time.
date
Tue Aug 31 15:21:45 UTC 2010
4. In the crontab file, set the idsBackup command to run two minutes from the current
time.
crontab -e
keep 2
23 15 * * * /usr/bin/idsBackup --inst
5. Check the idsBackup.log to monitor if the commands ran as expected.
tail f /var/idirect/log/idsBackup.log
6. Restore the original settings in the crontab file.
1.2 Configuring Single NMS Failover
NOTE: This procedure is not applicable if NMS Database Replication feature is

enabled on the NMS Servers.
This procedure describes how to bring a Backup NMS on line if the Primary NMS fails.
1. If the primary NMS is still accessible, reconfigure the primary NMS eth0 interface with
another unused IP address and physically unplug the eth0 interface. If the interface has
been reconfigured, the NMS server must be rebooted. Restart the network services by
entering the following command at the prompt:
service network restart
2. If the primary NMS remains connected to the network, the iDirect NMS services MUST be
shutdown by entering the following commands and the IP address must be different.
Additionally, these services MUST remain down while the other new (backup) NMS server
is online.
service idirect_nms stop

iDX Release 3.3
service idirect_nms stop nms_config

cd /etc/init.d
chkconfig --level 2345 idirect_nms off
3. Log on to the Backup NMS server as idirect and switch to root.
4. Verify that the routing configuration on the Backup NMS server is correct. Any static
routes that were configured on the Primary NMS server should also be configured on the
Backup NMS server. Make sure they are configured as persistent routes so they remain in
effect after a reboot.
5. If NMS Database Replication is enabled, enter the following command on the Backup NMS
server to stop the server from being a MySQL slave:
cr8DbSlave -d
6. Proceed to Configuring the Network Scripts.
1.2.1 Configuring the Network Scripts

Perform the following steps to configure the network interface configuration:
1. Go to the /etc/sysconfig/network-scripts directory.
cd /etc/sysconfig/network-scripts
2. Using the Vi editor, open the ifcfg-eth0 file for editing by entering the following
command:
vi ifcfg-eth0
3. Change the command lines in the ifcfg-eth0 file as shown in Table 1-1 (do not change any
other values, and retain all other lines within this file).
NOTE: Command lines are case-sensitive.
Table 1-1. ifcfg-eth0 File Entries
Command Description
DEVICE eth0
BOOTPROTO none
ONBOOT yes
IPADDR upstream IP address of the Primary NMS server
NETMASK subnet mask of the NMS server
GATEWAY IP address of the upstream router interface
MTU 1500
An example of a correctly configured network-script file for eth0 is shown in Figure 1-1.

iDX Release 3.3
Figure 1-1. ifcfg-eth0 Results Display
CAUTION: Do not delete any lines from the existing file. Doing so may have an
adversed effect on network performance.
4. Record the hardware address (HWADDR) as displayed in Figure 1-1.

5. Save the file and exit the Vi editor.
:wq!
6. Contact the TAC to see if a new license file is needed. Typically, the primary and backup
NMS server MAC addresses are listed in the license file, so a new license file is not
required.
7. Return to the /etc/sysconfig directory.
cd /etc/sysconfig
8. Using the vi editor, open the network file for editing by entering the following command:
vi network
The network file displays as shown in the example in Figure 1-2.
Figure 1-2. Network File Display

iDX Release 3.3
9. Change the command lines in the network-scripts file as shown in Table 1-2 (do not
change any other values, and retain all other lines within this file).
Table 1-2. Network File Commands
Command Description
NETWORKING yes
HOSTNAME Company_Name_NMS_Primary
GATEWAYDEV eth0
10. Save the file and exit the editor by entering the following command:
:wq!
11. Go to the /etc directory by entering the following command:
cd /etc
12. Using the vi editor, open the hosts file for editing by entering the following command:
vi hosts
13. Add the following lines to the hosts file, as applicable to the IP addressing scheme being
implemented. The two lines for 127.0.0.1 must be present. Enter the following command
lines:
127.0.0.1 COMPANY_NMS_PRIMARY localhost.localdomain localhost
192.168.1.3 COMPANY_NMS_PRIMARY
The hosts file displays as shown in the example in Figure 1-3.
Figure 1-3. Etc/Hosts File Display
14. Save the file and exit the editor by entering the following command:
:wq!
15. Configure the NMS services to start automatically at boot time by entering the following
command:
chkconfig idirect_nms on
16. Reboot the Backup NMS.

iDX Release 3.3
The backup NMS becomes primary upon boot up. It may take a few minutes for the
upstream router to update the ARP tables. After this update is complete, IP connectivity
to the server is restored.

iDX Release 3.3
2 Distributed NMS
Redundancy and Failover
Special consideration is required for Hubs that are using a Distributed NMS. If the current
system is a Distributed NMS, then additional configuration changes must be made on the
Primary NMS server, Auxiliary servers, and Backup server (event server (evtsvr), latency
server (latsvr), and the archive server (nrdsvr)). This chapter includes the following major
topics:
Distributed NMS Redundancy and Replication
Configuring Distributed NMS Redundancy
Configuring Distributed NMS Failover
2.1 Distributed NMS Redundancy and Replication

Beginning with iDX Release 3.1, iDirect supports near real-time replication of NMS databases
from the Primary NMS server to one or more Backup NMS servers in addition to the traditional
idsBackup and idsRestore. When enable the NMS Database Replication feature, changes to the
NMS databases on the Primary NMS are automatically replicated to the Backup NMS server and
the idsBackup script is automatically configured to run on the Backup server.
On a distributed NMS with NMS Database Replication enabled, each NMS server replicates its
database to a different backup machine. Assuming a standard DNMS setup with all databases
being replicated, the system consists of one Primary NMS server (NMS 1), two Auxiliary NMS
servers (NMS 2 and NMS 3), and three Backup NMS servers. During normal operation, each of
the three on line Servers (NMS 1, NMS 2 and NMS 3) replicates its database to the
corresponding backup server. (For details, see NMS Database Replication on a Distributed
NMS in the NMS Database Replication chapter of the Technical Reference Guide for iDX
Release 3.3)
If a Primary or an Auxiliary NMS server fails, it is possible to switch to the corresponding
Backup NMS server by disabling the failed server and reconfiguring the Backup server to take
its place. This procedure is identical to the procedure for a Single NMS failure. Therefore,
follow the procedure in Configuring Single NMS Failover on page 4.
NOTE: Do not forget to execute Step 5 of Section 1.2.1, Configuring the Network
Scripts on page 5 to disable Database Replication feature on the Backup NMS
server before bringing it up as the active server.
NOTE: If NMS Database Replication feature is enable on the Distributed NMS

servers, then the remaining procedures in this chapter are not applicable.

iDX Release 3.3
For a complete description of the NMS Database Replication feature see the Technical
Reference Guide for iDX Release 3.3.
2.2 Configuring Distributed NMS Redundancy

This section describes how to configure redundancy in a distributed NMS network.
CAUTION: Do not perform this procedure if Database Replication feature is enable

on the NMS servers. See Distributed NMS Redundancy and Replication on page 9.
2.2.1 Configuring the CRONTAB on the Servers

The crontab file must be reconfigured in order to correctly run the idsBackup and idsRestore
scripts.
Configuring the crontab on the Primary NMS

On the Primary NMS, perform the following:
1. Log on to the Primary NMS and switch to the root account.
crontab -e
The crontab file opens in the vi editor.
3. On the Primary NMS, remove the comment symbol (#) from the beginning of all lines in
keep 2
4. Add the following line to the crontab file, replacing the IP address shown with the Backup
NMS servers address, and, if desired, substituting the filename tokens shown with your
own:
NOTE: The idsBackup command creates a local backup, and when using the
inst option, it creates a remote backup and installs it on the specified
NMS.
NOTE: A forward slash character (\) is required before the token %

character for the crontab to execute properly. For example, use \%ip and
not %ip
5. Optionally, change the time and frequency with which backups are performed by
modifying the string 30 0 * * * (highlighted above). This is a five-field string set by
default to run a backup every night at 12:30am. The time and frequency of the backups
can be modified by changing these fields according to the following definitions. The fields
below are numbered left to right.
Field 1 (00 by default): Minute of the specified Hour (0 - 59)

iDX Release 3.3
Field 2 (1 by default): Hour of the Day (0 - 23)

Field 3 (* by default): Day of the Month (1 - 31)
Field 4 (* by default): Month (1 - 12)
Field 5 (* by default): Day of the week (0 - 6, with Sunday = 0)
6. Optionally, the number of local backup copies kept can be changed by modifying --keep
2, for example, to --keep 1 to save disk space.
:wq!
An example of an edited crontab file for the Primary NMS is shown below.
keep 2
Configuring the crontab on NMS 2 and NMS 3

A distributed NMS has a special backup configuration that must be configured in the crontab
file.
Perform the following steps on NMS 2 and NMS 3 of the Distributed NMS:
1. Log on to the Auxiliary NMS (NMS 2 or NMS 3) as idirect and switch to the root account.
crontab -e
3. On the Auxiliary NMS, remove the comment symbol (#) from the beginning of all lines in
keep 2
4. Add the following line to the crontab file, replacing the IP address shown with the NMS
servers address, and, if desired, substituting the filename tokens shown with your own:
30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --bkup
5. Save changes to the file and exit by pressing Esc entering:
:wq!
An example of an edited crontab file is shown below.

iDX Release 3.3

keep 2
30 2 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --bkup
Configuring the crontab on the Backup NMS

1. Log on to the Backup NMS and switch to the root account.
crontab -e
3. Remove the comment symbol (#) from the following line:
This command performs a recommended, daily maintenance check on the database.
An example of an edited crontab file for the Backup NMS is shown below.
keep 2
2.2.2 Verify That the System Backup is Working

This procedure provides all necessary steps to verify the system backup script is working and
that the appropriate backups are made on the Backup NMS server.
The time in crontab can be modified to run the commands 1-3 minutes after the current time,
and then check the idsBackup.log to see if the backup and install is running properly.
1. Log on to the Primary NMS and switch to the root account.
2. Enter the following command to push SSH keys to the Backup NMS:
pushSSHKeys <Backup NMS IP address>
The following sample output displays:
Pushing SSH key for user root to idirect@172.20.4.253:
The authenticity of host 172.20.4.253 (172.20.4.253) cant be
established.
RSA key fingerprint is 3e:28:f2:a5:50:22:a5:2f:51:33:69:39:ca:a4:8d:63.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 172.20.4.253 (RSA) to the list of known hosts.
idirect@172.20.4.253s password: remote_users_password
SSH key for user root successfully pushed to idirect@172.20.4.253

iDX Release 3.3
3. Get the current time.

date
Tue Aug 31 15:21:45 UTC 2010
4. In the crontab file, set the idsBackup command to run two minutes from the current
time.
crontab -e
keep 2
23 15 * * * /usr/bin/idsBackup --inst
5. Check the idsBackup.log to monitor if the commands executed successfully.
tail f /var/idirect/log/idsBackup.log
6. Restore the original settings in the crontab file.
7. In a Distributed NMS configuration, repeat Step 1 through Step 6 on each Auxiliary NMS.
NOTE: On the Auxiliary NMS servers, the line to edit ends in --bkup rather than
--inst as shown below.
23 15 * * * /usr/bin/idsBackup --bkup
2.3 Configuring Distributed NMS Failover

This section describes how to have the Backup NMS server takes over for either the Primary
NMS Server or for an Auxiliary NMS Server.
CAUTION: Do not perform this procedure if Database Replication feature is enable

on the NMS servers. See Distributed NMS Redundancy and Replication on page 9.
2.3.1 Preparing the Backup NMS Server to Take Over

Perform the following to prepare for the Backup NMS to take over:
1. If the primary NMS is still accessible, reconfigure the primary NMS eth0 interface with
another unused IP address and physically unplug the eth0 interface. If the interface has
been reconfigured, the NMS server must be rebooted. Additionally, the network services
can be restarted by entering the following command at the prompt:
service network restart
2. If the primary NMS remains connected to the network, the iDirect NMS services MUST be
shut down by entering the following command and the IP address must be different.
Additionally, these services MUST remain down while the other new (backup) NMS server
is online.
service idirect_nms stop
service idirect_nms stop nms_config

iDX Release 3.3
cd /etc/init.d
chkconfig --level 2345 idirect_nms off
3. Verify that the routing configuration on the Backup NMS server is correct. If RIPv2 was
running on the Primary NMS server, it has to be running on the backup. Any static routes
that were configured on the Primary NMS Server need to be configured on the Backup NMS
server. Make sure they are configured as persistent routes so they remain in effect after a
reboot.
2.3.2 Configuring Backup to Take Over for Primary

To bring the Backup NMS server on-line to replace the Primary server, perform the following
procedures.
Procedure 1. Edit the eth0 Configuration

Configure the Backup NMS Server eth0 interface with the primary NMS original eth0 interface
address. The reason for this is to match the original primary NMS IP address that is configured
in the configuration (options) files of the iDirect network elements.
2. Change the IP address and hostname, and configure the NMS services to start
automatically at boot time by performing Configuring the Network Scripts on page 5.
3. Rebuild the Distributed NMS configuration. Refer to Appendix A of the iBuilder User Guide
(Section A.4 Setting UP a Distributed NMS Environment through Section A.6, Granting
Read Permissions to NMS 2 and NMS 3).
4. The network should now be running on the Backup NMS Server. The status on the Line
Cards and the Remotes should be normal in iMonitor.
5. If needed, apply the Protocol Processor level and network level configuration to all
networks via iBuilder.
2.3.3 Backup NMS Server Takes Over for an Auxiliary Server

To bring the backup NMS server on-line in as an Auxiliary server, perform the following:
2. Find the restore file by entering the following commands to change to the backup
directory and list the files:
cd /var/idirect/backup
ll
This example shows a list of files in the backup directory:
-rw-r--r-- 4 idirect idirect 49519268 Feb 27 00:31 192.168.76.81-
2012.02.27-00.30.01
2012.02.28-00.30.01
2012.02.27-00.30.01

iDX Release 3.3

2012.02.28-00.30.01
2012.02.27-00.30.01
2012.02.28-00.30.01
3. Identify the backup file to restore, and enter the following command:
/usr/bin/idsRestore --bkup /var/idirect/backup/<filename>
Where, <filename> is the name of the backup file.

When this commands executes, the idsRestore script runs and the Backup NMS server
becomes the Auxiliary server. In this example, it becomes NMS2. During the takeover
process, the IP address is changed to that of the Auxiliary server. It may take a few
minutes for the Upstream router ARP tables to update. When the update is complete, IP
connectivity to the server is restored.
4. Configure the NMS services to start automatically at boot time by entering the following
commands:
cd /etc/init.d
chkconfig idirect_nms on
5. Reboot the Backup NMS.
6. If needed, apply the Protocol Processor level and network level configuration to all
networks via iBuilder.
7. Ensure that the backup is working by performing Verify That the System Backup is
Working on page 12.

iDX Release 3.3

iDX Release 3.3
13861 Sunrise Valley Drive, Suite 300
Herndon, VA 20171
+1 703.648.8000
+1 866.345.0983
www.idirect.net
Advancing a Connected World

NMS Redudancy and Failover - IDX 3.3 Rev B

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

NMS Redudancy and Failover - IDX 3.3 Rev B

Uploaded by

Copyright:

Available Formats

NMS Redundancy and Failover

iDX Release 3.3

November 20, 2014

A Family of iDirect Companies

iDirect Government Technologies

iDirect Asia Pte Ltd

Document Name: TN_NMS_Redundancy_3.3_Rev B_112014.pdf

ii NMS Redundancy and Failover

Revision Date Updates

1 Single NMS Redundancy and Failover . . . . . . . . . . . . . . . . . . . . . . . . . . .1

2 Distributed NMS Redundancy and Failover . . . . . . . . . . . . . . . . . . . . . . . 9

NMS Redundancy and Failover v

2.2.1 Configuring the crontab on the Primary NMS . . . . . . . . . . . . . . . . . . . . . 10

vi NMS Redundancy and Failover

Figure 1-1. ifcfg-eth0 Results Display. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

NMS Redundancy and Failover vii

Table 1-1. ifcfg-eth0 File Entries ........................................................................... 5

NMS Redundancy and Failover viii

Convention Description Example

NMS Redundancy and Failover ix

Convention Description Example

WARNING: A Warning highlights an essential operating or maintenance

CAUTION: A Caution highlights an essential operating or maintenance procedure,

NOTE: A Note is a statement or other notification that adds, emphasizes, or

iDirect Contact Information:

x NMS Redundancy and Failover

iGT Contact Information:

NMS Redundancy and Failover xi

xii NMS Redundancy and Failover

1.1 Configuring Single NMS Server Redundancy

NOTE: This procedure is not applicable if NMS Database Replication feature is

1.1.1 Configuring the CRONTAB on the Servers

NMS Redundancy and Failover 1

Configuring the CRONTAB on the Primary NMS

NOTE: A forward slash character (\) is required before the token %

2 NMS Redundancy and Failover

30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --inst

Configuring the crontab on the Backup NMS

1.1.2 Verify That the System Backup is Working

NMS Redundancy and Failover 3

The authenticity of host 172.20.4.253 (172.20.4.253) cant be

1.2 Configuring Single NMS Failover

NOTE: This procedure is not applicable if NMS Database Replication feature is

4 NMS Redundancy and Failover

service idirect_nms stop nms_config

1.2.1 Configuring the Network Scripts

NOTE: Command lines are case-sensitive.

Table 1-1. ifcfg-eth0 File Entries

NMS Redundancy and Failover 5

Figure 1-1. ifcfg-eth0 Results Display

4. Record the hardware address (HWADDR) as displayed in Figure 1-1.

Figure 1-2. Network File Display

6 NMS Redundancy and Failover

Table 1-2. Network File Commands

Figure 1-3. Etc/Hosts File Display

NMS Redundancy and Failover 7

8 NMS Redundancy and Failover

2.1 Distributed NMS Redundancy and Replication

NOTE: If NMS Database Replication feature is enable on the Distributed NMS

NMS Redundancy and Failover 9

2.2 Configuring Distributed NMS Redundancy

CAUTION: Do not perform this procedure if Database Replication feature is enable

2.2.1 Configuring the CRONTAB on the Servers

Configuring the crontab on the Primary NMS