Getting more done with less: Using scripts and utilities

to run commands across all your IBM AIX servers

quickly
Brian K. Smith

08 July 2013

System administrators are increasingly required to manage more servers, decreasing the
amount of time they are able to dedicate to each server. Time-saving utilities are crucial in
ensuring efficiency in managing these servers. Simple, serially run one-line Secure Shell (SSH)
loops and parallel-run commands using utilities, such as dsh and parallel SSH, drastically
improve efficiency, especially in large-enterprise environments.

Introduction
In most companies, system administrators are being asked to do more with less. The number
of IBM AIX instances that a system administrator must manage has grown drastically as
virtualization technologies, such as IBM PowerVM, have taken hold. PowerVM allows a single
piece of hardware to run dozens or even hundreds of IBM AIX logical partitions (LPARs). Years
ago, system administrators might have been responsible to manage only a couple of dozen AIX
instances, while in today's environments it is common to find system administrators with hundreds
of AIX LPARs to manage.
If you are in a large AIX environment with many LPARs, it isn't practical to log in to each one
when changes are needed or when you need to collect information from each server. In these
large environments, the best option is to carefully use technology that allows you to quickly run a
command or script across many servers. This article covers several options for running commands
and scripts across a large number of servers.

SSH key overview

All the options discussed in this article rely on SSH keys being set up between a central
management server and the rest of the servers.
The first step is to pick a central management server to use. You should pick a server that is as
secure as possible and has network access to all other servers in your environment. One option is
Copyright IBM Corporation 2013
Getting more done with less: Using scripts and utilities to run
commands across all your IBM AIX servers quickly

Trademarks
Page 1 of 11

developerWorks

ibm.com/developerWorks/

to create a new LPAR just for this purpose, and lock it down as much as possible. Other potential
options are a Network Installation Manager (NIM) server or a system monitoring server.
The next step is to pick a user account to use. Many tasks that you might likely want to automate
require root privileges. However, it is not recommended to directly use the root account due to
potential security issues. A good alternative is to create a normal user account and grant it root
access with a utility, such as sudo.
After you have identified which server you will use as your central management server and have
identified a user account, the next step is to generate an SSH key. There are many tutorials
available for SSH keys, so this article will not elaborate on the topic. Essentially, you need to use
the ssh-keygen command to create a private and public key on the central management server.
The private key is only on the central management server, while the public key can be copied to all
other servers.
You need to establish a new user account on each server, create a ".ssh/authorized_keys" file
in the user's home directory and put the public key in the file. If you plan to run commands with
root-level access, you should also set up sudo or something similar to grant the user additional
access. If you use sudo, use the "NOPASSWD" option so that sudo does not prompt the user for a
password.
To verify that the SSH keys are working, you should attempt to use ssh to run a remote command
such as hostname on one of the servers. For example, if the server name is "server1", you should
be able to type ssh server1 hostname, and the response should be server1. Using sudo, you
should be able to validate this by running the ssh server1 sudo su -c whoami, command which
should return root.

Running a single command on multiple servers using a simple for

loop
One method for running a command remotely is a simple one-line loop command. The names
of the servers on which the command will be run are read out of a file. Each line of the file has a
single server name. Listing 1 shows an example of the for loop command line to run a command
on multiple servers.

Listing 1. Example of an ssh for loop

for server in `cat serverlist`; do printf "%-20s" $server;
ssh -q -o "BatchMode yes" $server oslevel -s 2>&1; echo; done | grep -v ^$

Let's break this down in detail:

for server in `cat serverlist`; do Creates a loop such that every server in the
serverlist file will be processed by the loop and the server name will be set as the $server
variable.
printf "%-20s" $server; Displays the server's name, left justified, with padded spaces.
This will cause the server names and output to all line up properly.
Getting more done with less: Using scripts and utilities to run
commands across all your IBM AIX servers quickly

Page 2 of 11

ibm.com/developerWorks/

developerWorks

ssh -q -o "BatchMode yes" $server oslevel -s 2>&1; Calls the ssh command with
a couple of options. The -q option prevents the login banner from being displayed. The o "BatchMode yes" option causes SSH to run in a batch mode and not prompt the user for
anything. The $server will be changed to the current server's name by the for loop. In this
example, the oslevel -s is the command that we need to run on the remote server. The 2>&1
option is used to redirect the standard error to the standard output. Without this, the output
might be out of order as the standard error is not buffered.
echo; Ensures that whether or not the command produces an output, the cursor is moved
to the following line in order for the next server name in the loop to line up in its correct place.
It does this by using a carriage return at the end of each line, setting up the correct format if
there is no output returned following the command. If the command did produce an output,
this causes an extra blank line, however, these lines will be removed in the next step.
done | grep -v ^$ Ends the for loop, then filters the output for any blank lines and removes
them.
Figure 1 shows an example of this command line and the output. The file serverlist has a list of
server names (one per line) and the command that will be run on each server is oslevel -s.

Figure 1. An example of a for loop command line and its output

Figure 2 shows an example of a for loop command line using sudo and its output.

Figure 2. An example of a for loop command line using sudo for root access
and its output

In Figure 2, a user account is deleted on multiple servers. Root-level access to run rmuser is
granted by sudo. There is no output on aixtest1 and aixtest3, which means that the user was
deleted. On aixtest2, aixtest4, and aixtest5 the rmuser reports an error because the user does
not exist.
The downside to this for loop method is that each command is run on one server at a time
(serially). It generally takes a minimum of of a second to establish an SSH connection and run
Getting more done with less: Using scripts and utilities to run
commands across all your IBM AIX servers quickly

Page 3 of 11

developerWorks

ibm.com/developerWorks/

a quick command. If there is network latency or the command being run takes longer, it might take
multiple seconds per server or longer. If you are dealing with hundreds or thousands of servers,
the for loop approach may take too long. However, it is very simple and effective for smaller scale
cases.

Running a script on multiple servers using a simple for loop

Many times, the task at hand requires more than just running a single command across all of the
servers. In these cases, you might need to run a script on the remote servers.
One way to do this is to copy the script to each server [using Secure Copy Protocol (SCP) or
another file protocol], remotely run the script with SSH, then optionally delete the script with
another SSH connection. However, this is very inefficient as it requires multiple connections to
each server and requires copying the script file to each server.
A much better way to do this is to remotely run either a shell interpretor or a script interpretor (such
as Perl) on the remote server and redirect the script you want to run from the local server. Using
this method, the script is run on the remote server without ever having to copy the script file to the
remote server.
Listing 2 shows the contents of the info.sh script that exists on the central management server.
This is a simple script that detects whether the server is AIX or not. If it is AIX, the OS level, CPU
mode, entitled CPU capacity, and the amount of memory are displayed.

Listing 2. The info.sh script that exists on the central management server
#!/usr/bin/ksh
if [ "`uname`" = "AIX" ]; then
printf "This server is running AIX level
oslevel -s
echo "CPU and Memory info: "
lparstat -i | egrep "^Mode|^Entitled Capacity
echo "------------------------------"
else
echo "This is not an AIX server"
fi

: "
|^Online Memory"

Listing 3 shows a for loop similar to the first one we covered. However, in this for loop, the remote
command we are running is the Korn shell (ksh) interpretor, and we are redirecting the input to
ksh from the info.sh file. This causes the info.sh script to be run on each remote server without
actually having to copy the script to the servers.

Listing 3. An example of a for loop command to run local info.sh script on

each server listed in the servelist file
for server in `cat serverlist`; do printf "%-20s\n" $server;
ssh -q -o "BatchMode yes" $server 'ksh 2>&1' < ./info.sh ; echo; done | grep -v ^$

Listing 3 shows an example of a for loop command to run local info.sh script on each server
listed in the servelist file without having to copy script to each server.
Getting more done with less: Using scripts and utilities to run
commands across all your IBM AIX servers quickly

Page 4 of 11

ibm.com/developerWorks/

developerWorks

Figure 3 shows that when this command is run, the info.sh script is run on each server specified in
the serverlist file.

Figure 3. Using a simple for loop to run a script on remote servers using input
redirection

This is an efficient way to run scripts on a large number of servers without ever actually copying
the script to each server. However, the script is run serially so it might not be practical if the script
takes longer time to complete or you have a large number of servers that the script will be run on.

Comparing serial and parallel running of scrips

The previously covered for loops run commands on the remote servers serially, which means
one after another. There are also options to run commands in parallel, meaning the central
management server starts commands on multiple servers simultaneously. For example,
suppose you need to run a command on 35 servers and each server takes 1 second for the
SSH connection to be established and the command to be run. If run serially, it would take 35
seconds for this to complete. If run in parallel running 16 servers simultaneously, it would take
approximately 3 seconds (35 servers / 16 run simultaneously = approximately 3 seconds). There is
a significant speedup when running commands in parallel on a large number of servers. Figure 4
shows a diagram that compares serial and parallel running of commands.
Getting more done with less: Using scripts and utilities to run
commands across all your IBM AIX servers quickly

Page 5 of 11

developerWorks

ibm.com/developerWorks/

Figure 4. Diagram showing difference between serial and parallel execution of

scripts.

Running a command on multiple servers using the dsh command

AIX includes a dsh command that is designed to run a command on multiple servers in parallel. By
default, dsh can run up to 64 commands in parallel (this is customizable using the -f flag).
The first step in using dsh is to make sure that it is installed on your central management server.
This can be checked by running which dsh; . If a path is returned to dsh, the server has it installed.
If not, you need to either install the csm.dsh file set for AIX 6.1 or the dsm.dsh and dsm.core
file sets for AIX 7.1. The appropriate file set for AIX 6.1 or AIX 7.1 should be on your AIX product
installation media.
The next step is to set some environment variables that instruct dsh how to operate.
Getting more done with less: Using scripts and utilities to run
commands across all your IBM AIX servers quickly

Page 6 of 11

ibm.com/developerWorks/

developerWorks

Listing 4. Environment variables to set for dsh

export DSH_NODE_LIST="/home/user/serverlist"
export DSH_NODE_RSH=/usr/bin/ssh
export DSH_NODE_OPTS="-q -o BatchMode=yes"

Listing 4 shows examples of the variables that must be set before using dsh.
The DSH_NODE_LIST variable should be set to a file name that specifies the servers on which you
need to run the commands, one server per line.
The DSH_NODE_RSH variable is set to /usr/bin/ssh to instruct dsh to use SSH as the connection
protocol.
The DSH_NODE_OPTS variable is set to -q -o BatchMode=yes to instruct the SSH client to not display
the login banner and not prompt the user for anything.
After these variables are set, a command can be run on multiple servers by running dsh , followed
by the command line that should be run on each server. Figure 5 shows the oslevel command
being run on several servers using dsh.

Figure 5. "oslevel" command being run on several servers using "dsh"

Another useful functionality of dsh is the dshbak command. This can format the output of dsh in an
easier to use format if the command returns a lot of output. To use dshbak, simply run the normal
dsh command and pipe the output to dshbak as shown in Figure 6.

Getting more done with less: Using scripts and utilities to run
commands across all your IBM AIX servers quickly

Page 7 of 11

developerWorks

ibm.com/developerWorks/

Figure 6. Using "dshbak" to format the output of "dsh"

Using parallel SSH to run commands and scripts on multiple servers

Parallel SSH is a utility that is designed to run SSH commands on multiple servers in parallel and
works similarly to the AIX dsh utility. However, parallel SSH has several advantages including:
being cross platform (same utility works on Linux and other UNIX variants), being an open
source, and having the capability to run scripts using redirection.
Parallel SSH is available in most Linux distribution repositories for easy installation.
To use parallel SSH on AIX you will first need to install version 2.4 or later of the Python scripting
language. I found that the parallel SSH setup.py installation program can be difficult to run on AIX,
and it is not necessary. Instead, you can manually copy the parallel SSH binaries easily to any
location from which you wantto run them.
To use parallel SSH to run a command on several servers, use a command similar to the one
listed in listing 5.
Listing 5. Example command to have pssh to run a command on several servers
# pssh -h serverlist -i oslevel -s

The -h serverlist command specifies the list of servers in the serverlist file(one server name
per line). The -i option specifies that the output must be inline on the screen. oslevel -s is the
name of the command that should be run on each of the servers.
Getting more done with less: Using scripts and utilities to run
commands across all your IBM AIX servers quickly

Page 8 of 11

ibm.com/developerWorks/

developerWorks

Figure 7 shows an example output of running the oslevel command on several servers using
pssh.

Figure 7. Running the "oslevel" command on several servers using "pssh"

Using pssh, it is also possible to quickly and easily run a script on multiple servers. This can be
done without copying the script to each server using input direction. This is basically the same
method covered earlier in the article under the Running a script on multiple servers using a simple
for loop section. However, in this case, it is done in parallel and thus it is much faster.
To run the info.sh script (shown in Listing 2) using pssh, use the command line shown in Listing 6.
Listing 6. Using "pssh" to run a local script on several servers
# pssh -h serverlist -i -I < info.sh

The -I switch tells pssh to read the command to be run from the standard input. The < info.sh
option redirects the info.sh script to the standard input of pssh, which in effect runs this script on
each server without actually having to copy the script to each server.
Figure 8 shows the output of running the info.sh script on several servers using pssh.

Getting more done with less: Using scripts and utilities to run
commands across all your IBM AIX servers quickly

Page 9 of 11

developerWorks

ibm.com/developerWorks/

Figure 8. Running the info.sh script on several servers using "pssh"

Going further
The AIX dsh suite also includes a command named dcp that allows files to be copied to or from
multiple servers in parallel. This can make pushing out files to many LPAR's a very quick and easy
process.
Parallel SSH also includes similar copy commands, which both run in parallel. The pscp command
is used to copy files to servers, while pslurp is used to copy files from servers.
Figure 9 shows the pscp command in use to copy the info.sh file into the /tmp directory on several
servers.

Getting more done with less: Using scripts and utilities to run
commands across all your IBM AIX servers quickly

Page 10 of 11

ibm.com/developerWorks/

developerWorks

Figure 9. Using "pscp" to copy info.sh to the /tmp directory on several servers

Conclusion
As demonstrated, there are several methods to run commands or scripts on multiple servers.
System administrators can use both simple one-line loops that serially run the command or script
on each server, and also utilities that run commands in parallel (such as dsh and parallel SSH).
Using these tools and methods can allow system administrators to be more efficient, getting more
done in less time.

Resources
The man page for the dsh command
The man page for the dcpcommand
The parallel SSH home page
Copyright IBM Corporation 2013
(www.ibm.com/legal/copytrade.shtml)
Trademarks
(www.ibm.com/developerworks/ibm/trademarks/)

Getting more done with less: Using scripts and utilities to run
commands across all your IBM AIX servers quickly

Page 11 of 11