Pete Soto Jr

08/10/09
Rev. 1_2
Whitepaper

mon_msg.sh documentation

This paper explains the function of the mon_msg.sh script, requirements,

operation, how to implement and the procedures for its use to facilitate
monitoring of Tivoli Workload Scheduler Windows .msg files and file system disk
space availability.

Utility overview
The mon_msg.sh utility script is designed to run from the Tivoli Workload
Scheduler Version 8.2.0 or greater workstation with the following UNIX operating
systems:
򐂰 AIX
򐂰 HPUX
򐂰 Linux_i386
򐂰 Solaris

The mon_msg.sh is used to monitor the size of TWSHome/*.msg,

TWSHome/pobox/*.msg files and TWS filesystem disk space availability. The
script has ratios that may be set for the .msg files. When ratio is exceeded script
may be configured to auto increase size of .msg file, send notification that upper
limit was exceeded and that .msg file upper limit was increased. The amount to
increase is a defined percentage of current .msg file maximum size.

© Copyright IBM Corp. 2008. All rights reserved. 1

 The increase of .msg file is performed by TWS evtsize binary. Use of this binary
permits increase of .msg file while TWS is still active.

Script will monitor available space for TWS filesystem and will send notification if
it drops below defined value.

The mon_msg.sh should be run periodically to monitor size of the .msg files and
available TWS filesystem disk space.

Implementation will require editing the mon_msg.sh script and adjusting the
specific variables that will determine the operation. The script may be executed
from UNIX prompt as TWS user for workstation or may be defined as a TWS job.
Ideally the mon_msg.sh script should be a defined job in a jobstream with an
“every” dependency.

mon_msg.sh Requirements
The /tmp directory must exist on workstation where mon_msg.sh will execute
since /tmp directory is where temporary files will be generated,

Script may exist in any directory and executed by TWS user for the TWS
instance. If script will be executed for multiple TWS instances that exist on
workstation then access permissions will need to be adjusted accordingly. The
mon_msg.sh must have execute permissions for user that will execute the script.

If script is defined as a TWS job then script path must be specified in the job
definition. Entries in the TWSRegistry.dat and “thiscpu” variable in
TWSHome/localopts file must be valid.

Important: The script options must be reviewed to determine if default

variables are appropriate and customized, if necessary, before the script is
executed.

mon_msg Variables
The mon_msg.sh has seven variables and one module that may be customized.
Review script and adjust variables (see below) accordingly.

2 Mon_msg.sh Utility for Monitoring Tivoli Workload Scheduler Message Files and Filesystem
Variable: FILE_SYS_AVAIL_SPACE_LIMIT
Minimun low limit setting. This number will be used as base to determine if
available disk space of the TWS filesystem has dropped below the specified
value. Use whole numbers, the value should be in K-bytes. The value
specified should allow TWS/UNIX administrator sufficient time to make
adjustments beforeTWS runs out of disk space, causing TWS processes to
terminate and possibly compromising the Symphony file.
FILE_SYS_AVAIL_SPACE_LIMIT=<kbytes>.
Default value (1500000):
FILE_SYS_AVAIL_SPACE_LIMIT=15000000

Note: 15000000 KB is about 14648.4 MB or 14.3 GB

Variable: FILE_SYS_INVOKE_NOTIFY
Value to determine if notification is required when available disk space for
TWS filesystem has dropped below specified limit. Set value to "yes" to
invoke notification, any other value will default to "no".
FILE_SYS_INVOKE_NOTIFY=<yes or no>.
Default value:
FILE_SYS_INVOKE_NOTIFY=yes

Variable: FILE_SYS_NOTIFY_CONTACT
Value to identify contact to receive notification when TWS filesystem drops
below the free disk space value specified in variable
FILE_SYS_AVAIL_SPACE_LIMIT. Default action is to send email to root user
if variable FILE_SYS_INVOKE_NOTIFY is set to yes. An email list could be
specified if several users should be notified by separating users with a comma
",". FILE_SYS_NOTIFY_CONTACT=<email_id> or
<cell_phone_number@carrier> (see Figure 1).
Default value:
FILE_SYS_NOTIFY_CONTACT=root

mon_msg.sh documentation 3
Variable: THRESH_PER
Upper limit percentage setting. This number will be used as ratio to
determine when a .msg file has exceeded its upper limit. Use whole numbers,
do not use a decimal point. The calculation will convert to decimal value.
THRESH_PER=<two digit value>.
Default value (85%):
THRESH_PER=85

Variable: AUTO_INCREASE_MSG
Value to determine if .msg file should be increased automatically when value
of a .msg file is at or greater than upper limit percentage specified in variable
THRESH_PER. The default is AUTO_INCREASE_MSG=no. Set value to
"yes" to invoke auto adjust. Any other value will default to "no". Increasing of
.msg file will permit .msg file to increase in size as more records are added
which means that it will require additional disk space. TWS filesystem must
be able to handle increased size. AUTO_INCREASE_MSG=<yes or no>
Default value:
AUTO_INCREASE_MSG=no

Variable: AUTOMSG_INCR_PER
Percentage of increase for .msg files that exceed the upper limit percentage
specified in variable THRESH_PER. AUTOMSG_INCR_PER. The increase of
.msg file is performed by TWS evtsize binary. The use of this binary permits
the increase while TWS is still active. Use whole numbers, do not use a
decimal point. The calculation will convert to decimal value.
AUTOMSG_INCR_PER = < 1 or 2 digit value>
Default value (20%):
AUTOMSG_INCR_PER=20

Variable: MSG_INVOKE_NOTIFY
Value to determine if notification is required when of upper limit of .msg file
has been exceeded. Set value to "yes" to invoke notification, any other value
will default to "no". MSG_INVOKE_NOTIFY=<yes or no>.
Default value:
MSG_INVOKE_NOTIFY=yes

4 Mon_msg.sh Utility for Monitoring Tivoli Workload Scheduler Message Files and Filesystem
Variable: MSG_NOTIFY_CONTACT
Value to identify contact to receive notification when upper limit of .msg file
has been exceeded. Default action is to send email to root user if variable
MSG_INVOKE_NOTIFY is set to yes. An email list could be specified if
several users should be notified by separating users with a comma ",".
MSG_NOTIFY_CONTACT=<email_id> or <cell_phone_number@carrier>
(see below).
Default value:
MSG_NOTIFY_CONTACT=root

Cell Phone SMS Text Messaging

Major US cellular carriers use the 10_digit_number@cell.carrier_domain.com
format for SMS to text capable cell phones, with a limit of 160 characters in the
subject and message body (total). You will need to determine limitations for
subject and body of email for the respective carrier. Refer to Figure 1.

Note: Cell phone must be able to receive text messages and carrier may have
limitations.

Carrier------> Send Email to phonenumber@....

Alltel-------> @message.alltel.com
Cingular-----> @mobile.mycingular.com
Nextel ------> @messaging.nextel.com
Sprint-------> @messaging.sprintpcs.com
SunCom-------> @tms.suncom.com
T-mobile-----> @tmomail.net
Verizon------> @vtext.com
Virgin Mobile> @vmobl.com
VoiceStream--> @voicestream.net

Figure 1 Carrier email domain information

Notification Modules
The notification modules may be customized to perform an additional task (or
multiple tasks) or a task other than sending out an email. The sections in bold for
following two figures may be customized by the user, refer to Figure 2 and
Figure 3.

mon_msg.sh documentation 5
 Attention: User customization of notification tasks will not be supported by
TWS L2 support.

#MSG Notification Module

#
if [ "${MSG_INVOKE_NOTIFY}" = "yes" ]
then
#This section may be customized to perform another task
#other than sending out an email

#The following message will be sent. If script is being executed

#from within a TWS job the ${UNISON_STDLIST} ${UNISON_JOB}
#information will be included.

#The message generated will depend it script was executed via TWS.
if [ "${tws_job}" = "y" ]
then

mail ${MSG_NOTIFY_CONTACT} <<-!

One or more .msg files have exceeded their upper limit
of ${THRESH_PER}% on ${TWSCPUNAME}. Please review stdlist
${UNISON_STDLIST} for job ${UNISON_JOB}.

Files that exceeded threshold:

${MSG_MAX}
${MSG_INCREASE_MESSAGE}
!
else
mail ${MSG_NOTIFY_CONTACT} <<-!
One or more .msg files have exceeded their upper limit
of ${THRESH_PER}% on ${TWSCPUNAME}.

Files that exceeded threshold:

${MSG_MAX}
${MSG_INCREASE_MESSAGE}
!
fi
fi

Figure 2 MSG Notification Module

6 Mon_msg.sh Utility for Monitoring Tivoli Workload Scheduler Message Files and Filesystem
 #Filesystem Notification Module
#Added 02/08/08 PSJ
#Added checking if TWS file system disk space was exceeded
test -f /tmp/msg_file_sys_exceeded.$$
if [ "$?" = "0" ]
then

if [ "${FILE_SYS_INVOKE_NOTIFY}" = "yes" ]
then
#This section may be customized to perform another task
#other than sending out an email

#The following message will be sent. If script is being executed

#from within a TWS job the ${UNISON_STDLIST} ${UNISON_JOB}
#information will be included.

#The message generated will depend it script was executed via TWS.
if [ "${tws_job}" = "y" ]
then

mail ${NOTIFY_CONTACT} <<-!

WARNING: ${TWSCPUNAME} filesystem has exceeded their upper limit.
Available disk space of file system ${mounted_file_system}
for TWS Home directory ${TWSPATH} is ${available_disk} Kbytes
which is less that the defined which is less than the defined
threshold limit of ${FILE_SYS_AVAIL_SPACE_LIMIT} Kbytes.

Please increase size of file system ${mounted_file_system}

or TWS Home directory ${TWSPATH} before TWS file
system runs out disk space otherwise TWS operations
may be compromised.

TWS file system info

${TWS_file_sys}
!
else
mail ${NOTIFY_CONTACT} <<-!
WARNING: ${TWSCPUNAME} filesystem for TWS directory has exceeded specified upper
limit.

WARMING:
${TWSCPUNAME} file system ${mounted_file_system} for TWS Home directory
${TWSPATH} is ${available_disk} Kbytes which is less than the
defined threshold limit of ${FILE_SYS_AVAIL_SPACE_LIMIT} Kbytes.

TWS file system info

${TWS_file_sys}
!
fi
fi
fi

Figure 3 Filesystem Notification Module

The notification task can include generation of a trouble ticket, invoking of a third
party application, issuing a page to a pager, or sending out a SMS text message
to a cell phone.

The notification task will send a message via email. Message sent will depend on
how the script was invoked. If script is being executed from within a TWS job the
${UNISON_STDLIST} ${UNISON_JOB} information will be included in the email

mon_msg.sh documentation 7
 otherwise a generic message without these variables will be generated and sent.
Refer to Figure 4, Figure 5, Figure 6 and Figure 7 for sample emails. UNIX
workstation must be able to send out email since current task uses UNIX mail to
send out email.

Note: The recipient of notification must be monitored regularly. If script is

configured to auto resize the .msg file, verification may be necesary to
determine that TWS filesystem has sufficient disk space.

Date: Wed, 5 Sep 2007 10:56:08 -0500 (CDT)

From: maestro <m821a@taos.ibm.com>
Message-Id: <200709051556.l85Fu8Ct022126@taos.ibm.com>
Status: R

WARNING:
One or more .msg files have exceeded their upper limit
of 85% on TWS workstation m821a.

Files that exceeded threshold:

Current warning level is set at 85% of 30000 records.

Message file /tws/m821a/pobox/M82A.msg on m821a is at 28896 records ie.. 96%.
AWSDEK702I File /tws/m821a/pobox/M82A.msg maximum size changed to 36000 bytes
/tws/m821a/pobox/M82A.msg was increased by 20%

TWS file system info:

/dev/vg00/lvol9 16384000 15760050 609180 96% /tws

Figure 4 Sample email for .msg ratio execeded and configured for autoincrease .msg

Date: Wed, 5 Sep 2007 10:56:08 -0500 (CDT)

From: maestro <m821a@taos.ibm.com>
Message-Id: <200709051556.l85Fu8Ct022126@taos.ibm.com>
Status: R

WARNING:
One or more .msg files have exceeded their upper limit
of 85% on TWS workstation m821a.

Files that exceeded threshold:

Current warning level is set at 85% of 30000 records.

Message file /tws/m821a/pobox/M82A.msg on m821a is at 28896 records ie.. 96%.
File limit size should be increased with /tws/m821a/bin/evtsize
command before TWS operations are compromised.

TWS file system info:

/dev/vg00/lvol9 16384000 15760413 608839 96% /tws

Figure 5 Sample email for .msg ratio execeded and not cofigured for autoincrease of
.msg

8 Mon_msg.sh Utility for Monitoring Tivoli Workload Scheduler Message Files and Filesystem
 Date: Wed, 13 Feb 2008 11:31:32 -0600 (CST)
From: root@taos.tivlab.austin.ibm.com
Message-Id: <200802131731.m1DHVWlu015152@taos.tivlab.austin.ibm.com>
WARNING: TAOS filesystem for TWS directory has exceeded specified upper limit.
Status: R

WARNING:
m821a file system /tws for TWS Home directory
/tws/m821a is 609165 kbytes which is less than the
defined threshold limit of 1500000 kbytes.

Please increase size of file system /tws

for TWS Home directory /tws/m821a before TWS file
system runs out disk space otherwise TWS operations
may be compromised.

Figure 6 Sample email when available TWS filesystem drops below defined threshold.

Date: Mon, 17 Nov 2008 17:18:58 -0600 (CST)

From: root@taos.tivlab.austin.ibm.com
Message-Id: <200811172318.mAHNIw99017275@taos.tivlab.austin.ibm.com>
WARNING:
Status: R

WARNING:
One or more .msg files have exceeded their upper limit
of 85% on TWS worksation m821a. Please review stdlist
/tws/m821a/stdlist/2008.11.17/O17097.1718 for job M821A#JOBS.MON5.
Files that exceeded threshold:

Current warning level is set at 85% of 30000 records.

Message file /tws/m821a/pobox/M82A.msg on m821a is at 28896 records ie.. 96%.
File limit size should be increased with /tws/m821a/bin/evtsize
command before TWS operations are compromised.

TWS file system info:

/dev/vg00/lvol9 16384000 15760498 608758 96% /tws

Figure 7 Sample of email sent from script as a defined job

mon_msg Implementation Procedures

Script may be executed from UNIX prompt or as a TWS defined job. User will
need to determine how often script will launch and set that as the “every”
dependency for the job.

mon_msg.sh documentation 9
To Execute From UNIX Prompt:
1. Edit mon_msg.sh and customize variables.
2. Change permission for script to include execute permissions.
3. Execute the mon_msg.sh script
mon_msg.sh <tws_user_name>
Where: tws_user_name is TWS user for workstation.

To Execute as a TWS Job.

1. Edit mon_msg.sh and customize variables.
2. Change permission for script to include execute permissions.
3. Define a TWS job that executes the mon_msg.sh script. The logon user
should be same user specified with script execution.
4. The path must be provided for the mon_msg.sh script. See Figure 8
5. Create a job stream and include the job with an “every” dependency. The
“every” dependency will determine how often the job will execute. The job
stream should be scheduled to run on a daily basis (on everyday). Refer to
Figure 9.

M821A#MON_MSG SCRIPTNAME "/tws/m821a/scripts/mon_msg.sh m821a"

STREAMLOGON "m821a"
DESCRIPTION ".msg monitor script"
RECOVERY STOP

Figure 8 Sample mon_msg job definition

SCHEDULE M821A#MON_MSG_SCH
ON EVERYDAY
:
M821A#MON_MSG EVERY 60
END
Figure 9 Sample mon_msg_sch job stream definition

Script Operation
The script will verify that the specified user exists before proceeding with the
execution of the script

10 Mon_msg.sh Utility for Monitoring Tivoli Workload Scheduler Message Files and Filesystem
The values specified in mon_msg.sh script will determine operation and tasks
performed by the script.

The script will terminate immediately if specified user is not valid.

The script will generate temporary files in the /tmp directory. Generated files will
have a prefix of msg_ and include the process id/pid as an extension.These
generated files will be removed as soon as the script completes.

The increase of the .msg file is performed by the TWS evtsize binary. The use of
this binary permits the increase of .msg file while TWS is still active.

The script will display information for .msg files that exceed the defined upper
limit ratio of .msg file. Displayed information will include information about
filesystem where TWS instance is located. Display will be written to stdout. If
script is executed from within a TWS job, output will be displayed in the job stdlist.
Refer to Figure 10.

Current warning level is set at 85% of 10000000

Message file /tws/m821a/Courier.msg on m821a is at 9000000 ie.. 90%.
AWSDEK702I File /tws/m821a/Courier.msg maximum size changed to 12000000
bytes

Current warning level is set at 85% of 10000000

Message file /tws/m821a/Intercom.msg on m821a is at 9000000 ie.. 90%.
AWSDEK702I File /tws/m821a/Intercom.msg maximum size changed to 12000000
bytes

Current warning level is set at 85% of 10000000

Message file /tws/m821a/pobox/serverA.msg on m821a is at 9000000 ie.. 90%.
AWSDEK702I File /tws/m821a/pobox/serverA.msg maximum size changed to
12000000 byte

TWS file system info:

/tws (/dev/vg00/lvol9 ) : 16303138 total allocated
Kb
1239430 free allocated Kb
15063708 used allocated Kb
92 % allocation used

Figure 10 Sample .msg and TWS filesystem information

The script will generate an email notification depending on how it was executed.

mon_msg.sh documentation 11
Troubleshooting
The script has specific requirements for execution. Review/verify the following:
1. Script mon_msg.sh must be executable.
2. Valid TWSuser must be specified.
3. The /tmp directory must exist.
4. The /etc/TWSRegistry.dat file must exist and be valid/current.
5. The “thiscpu” variable in the TWSHome/localopts file must have correct TWS
workstation name.
6. Verify that TWS filesystem has enough disk space.
7. UNIX mail must be able to send out emails.

12 Mon_msg.sh Utility for Monitoring Tivoli Workload Scheduler Message Files and Filesystem