Linux Shell - Ibm

System Administration Toolkit: Process administration tricks
Martin Brown (mc@mcslp.com), Freelance Writer, Freelance Developer
Summary: Discover how to get the information you want on UNIX processes. Knowing what is running on your UNIX system is one of the most basic requirements of any system administrator. The standard process list is useful, but often the information that it provides is not in the right format or doesn't contain exactly the processes or information you need. Being able to select specific types of data from the process list makes administration a lot easier. In this article, you'll examine how to extend that process further to improve the readability of the information, or provide summaries and information that are not easily obtainable elsewhere. You'll also look at methods for standardizing how to obtain process information across different UNIX platforms. View more content in this series
Tag this! Update My dW interests (Log in | What's this?) Skip to help for Update My dW interests
Date: 21 Feb 2006 Level: Introductory PDF: A4 and Letter (57KB)Get Adobe Reader Also available in: Russian Activity: 12024 views Comments: 0 (View | Add comment - Sign in)
Average rating (86 votes) Rate this article
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment.
Using ps The ps command line tool lists running processes. The tool exists on all variants of UNIX and, on the whole, works in the same basic fashion -- asking the kernel for a list of running processes and then reporting on the list of processes and their properties, such as memory usage, running time, and other details. The ps tool is actually a very powerful utility, although many administrators probably rely on only one or two of the available options and, in their eyes, to drill down to the information they want. You can get more information out of the command by making use of the built-in command line options, and even more if you combine ps with commands through pipes to get access to exactly the information you want. Listing all processes The standard output from ps only lists your running processes, even when you are logged in as root. Depending on whether your UNIX environment is based on the BSD or AT&T, SysV UNIX base alters the primary command line options that you use to get information on other processes on the system, or changes the information that is displayed. Within a BSD-based UNIX environment, the output incorporates the process ID, terminal, status, time (duration of execution in CPU seconds used, rather than an indication of when the process was started), and the command executed, as shown in Listing 1.
Listing 1. Listing process on BSD UNIX variants

$ ps PID 391 9165 476 9299 9319 9423 9513 9301 9302 9321 9322 TT p5 p5 p6 p6 p6 p6 p6 p7 p8 p9 pa STAT S S+ S S S S R+ Ss+ Ss+ Ss+ Ss+ TIME 0:00.24 0:00.50 0:01.03 0:00.09 0:00.07 0:00.12 0:00.01 0:00.01 0:00.03 0:00.01 0:00.02 COMMAND /bin/bash emacs /bin/bash xterm xterm ftp atuin ps /usr/X11R6/bin/luit bash /usr/X11R6/bin/luit bash
Under an SVR4 environment, fewer columns are provided (you no longer get a process status), as seen in Listing 2 below.
Listing 2. Listing process on BSD UNIX variants

$ ps
PID 19915 29145 32256 26986 31303 31358
TTY pts/3 pts/3 pts/3 pts/3 pts/3 pts/3
TIME 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00
CMD bash emacs emacs xterm ftp ps
To get a list of all the processes running on the system, you need to use different command line options, according to the UNIX variant in use. Within a BSD UNIX, the command line option -a lists all user processes, including your own. However, the list will still not include processes without a controlling terminal (for example, those started at boot time, daemons, and those executing as part of a cron job). To get a list of all running processes, you must use the -A command line option (see Listing 3).
Listing 3. Listing all processes on BSD systems

$ ps -A PID TT STAT TIME 1 ?? S<s 0:15.47 23 ?? Ss 0:00.02 /private/var/vm/swapfile 27 ?? Ss 0:00.95 49 ?? Ss 0:05.17 50 ?? Ss 0:01.89 51 ?? Ss 0:04.40 52 ?? Ss 0:00.08 53 ?? Ss 0:02.80 55 ?? Ss 11:03.59 57 ?? Ss 0:01.13 ... 8051 p2 S+ 0:00.61 292 p3 Ss 0:00.02 372 p3 S+ 0:00.42 312 p4 Ss+ 0:00.03 332 p5 Ss 0:00.03 391 p5 S 0:00.24 9165 p5 S+ 0:00.50 352 p6 Ss 0:00.04 476 p6 S 0:01.04 9299 p6 S 0:00.09 9319 p6 S 0:00.07 9423 p6 S 0:00.14 9520 p6 R+ 0:00.01 9301 p7 Ss+ 0:00.01 9302 p8 Ss+ 0:00.03 9321 p9 Ss+ 0:00.01 9322 pa Ss+ 0:00.02 COMMAND /sbin/launchd /sbin/dynamic_pager -F kextd /usr/sbin/configd /usr/sbin/coreaudiod /usr/sbin/diskarbitrationd /usr/sbin/memberd -x /usr/sbin/securityd /usr/sbin/notifyd /usr/sbin/DirectoryService ssh root@bear bash ssh admin@atuin bash bash /bin/bash emacs bash /bin/bash xterm xterm ftp atuin ps -A /usr/X11R6/bin/luit bash /usr/X11R6/bin/luit bash
The -A command line option is equivalent to using both -a and -x options, where -a shows processes with controlling terminals and -x shows processes without controlling terminals. Under SVR4 variants, it is the -e command line option that shows you all running processes, irrespective of whether or not they have a controlling terminal. It is equivalent, in terms of the processes displayed, to the BSD -A option. You can see a sample of the output in Listing 4.
Listing 4. Process listings in SVR4 environments

$ ps -e PID TTY 0 ? 1 ? 2 ? 3 ? 308 ? 7 ? 9 ? 506 ? 260 ? 259 ? 52 ? 282 console 267 ? 264 ? 90 ? ... 462 ? 464 ? 463 ? 473 ? 552 ? 527 ? 548 ? TIME CMD 15:24 sched 0:00 init 0:00 pageout 0:00 fsflush 0:00 devfsadm 0:06 svc.star 0:10 svc.conf 0:00 htt_serv 0:00 rpcbind 0:00 cron 0:00 dhcpagen 0:00 ttymon 0:00 lockd 0:00 statd 0:00 sysevent 0:00 0:00 0:00 0:00 0:00 0:00 0:01 smcboot smcboot smcboot htt in.telne dmispd snmpd
Where the output differs is in the columns of information that are displayed, but you can fix that by specifically selecting the columns that you want. Listing specific information The ps tool includes a number of standard sets of columns that are displayed. For example, within SVR4, it is common to use ps -ef for more extensive information about the processes listed, including the parent process ID, processor utilization, start time, and a more detailed command line, as shown here in Listing 5.
Listing 5. Extending the output

ps -ef UID PID PPID C root 0 0 0 root 1 0 0 root 2 0 0 root 3 0 0 root 308 1 0 root 7 1 0 /lib/svc/bin/svc.startd ... root 562 1 0 bd -q15m root 576 555 1 root 416 1 0 smmsp 561 1 0 Ac -q15m ... root 552 283 0 root 527 1 0 root 548 1 0 STIME 15:56:26 15:56:26 15:56:26 15:56:26 15:57:09 15:56:29 TTY ? ? ? ? ? ? TIME 15:24 0:00 0:00 0:00 0:00 0:06 CMD sched /sbin/init pageout fsflush devfsadmd
15:58:17 ? 16:01:47 pts/1 15:57:14 ? 15:58:17 ? 15:57:47 ? 15:57:22 ? 15:57:24 ?
0:00 /usr/lib/sendmail 0:00 ps -ef 0:00 /usr/sbin/syslogd 0:00 /usr/lib/sendmail 0:00 /usr/sbin/in.telnetd 0:00 /usr/lib/dmi/dmispd 0:01 /usr/sfw/sbin/snmpd
For BSD environments, you normally add the -l option, which produces "long" output from each process, such as Listing 6.
Listing 6. Showing more process detail in BSD

$ ps -al UID PID PPID CPU PRI NI VSZ RSS WCHAN 0 9165 391 0 31 0 57896 6376 501 352 349 0 31 0 27784 52 0 476 352 0 31 0 27784 600 0 9299 476 0 31 0 44988 1880 0 9319 476 0 31 0 44988 1888 0 9423 476 0 31 0 27504 488 0 9540 476 0 31 0 27384 504 0 9301 9299 0 31 0 27332 452 /usr/X11R6/bin/luit 0 9302 9301 0 31 0 27784 888 0 9321 9319 0 31 0 27332 452 /usr/X11R6/bin/luit 0 9322 9321 0 31 0 27784 888 STAT S+ Ss S S S S R+ Ss+ Ss+ Ss+ Ss+ TT p5 p6 p6 p6 p6 p6 p6 p7 TIME 0:00.50 0:00.04 0:01.05 0:00.09 0:00.07 0:00.15 0:00.01 0:00.01 COMMAND emacs bash /bin/bash xterm xterm ftp atuin ps -axl
p8 0:00.03 bash p9 0:00.01 pa 0:00.02 bash
The problem with these options is that, although they show more information, the information is not always particularly useful, or it contains information that you don't want cluttering up the display when you are looking for a particular process.
Fortunately, all versions of ps also include the ability to specify what columns are displayed. You can use this to great effect, either only picking out the information you want or, in a heterogeneous environment, you can use it to create a standardized output across a range of different UNIX environments. To use this feature, use the -o command line option and list the individual columns that you want to display as a comma-separated list. Although there are differences between the ranges of columns available, many of them are standard across different UNIX versions. For example, pid, ppid (parent pid), command, RSS (resident set size or memory usage), and priority are available in all variants. In use, you can use -o to pick the columns and the display order. For example, to get the pid, ppid, and command, you would use -opid,ppid,command, as shown within BSD in Listing 7, or -opid,ppid,comm within SVR4, as seen in Listing 8.
Listing 7. Selecting specific columns in BSD

$ ps -o pid,ppid,command PID PPID COMMAND 391 332 /bin/bash 9165 391 emacs
Listing 8. Selecting specific columns in SVR4

$ ps -opid,ppid,comm PID PPID COMMAND 555 552 -sh 622 555 ps
Once you have selected the columns you like, you might then want to select how the information is ordered. The default for ps is to sort the list of processes by the process ID, but this can have the effect of hiding the very information you are looking for. When looking for memory hungry processes, it is more useful to display the output listed in order of the memory used, rather than process ID. Some ps variants support this option by using a command line option. BSD variants sort by memory usage when the -m option is used, or by CPU usage when using -r. Many SVR4 variants don't have a specific solution for this, but you can produce similar results in both environments by combining ps with sort. For example, to get a list of processes sorted by the amount of CPU they are using, you might use the command in Listing 9 within BSD.
Listing 9. Tracking usage by CPU in BSD
$ ps -A -o pid,%cpu,command|sort -n +1 ... 358 0.1 ftp 11425 0.1 /bin/bash 28684 0.3 trivial-rewrite -n rewrite -t unix -u 356 0.4 ssh 354 0.5 as 23988 1.1 emacs 136 14.6 cc1plus 26306 23.6 cpp
Within SVR4, you need to change %cpu for pcpu, but otherwise the command is identical (see Listing 10).
Listing 10. Tracking usage by CPU in SVR4

$ ps -e -opid,pcpu,comm|sort -n +1 ... 3 0.1 fsflush 555 0.1 -sh 627 0.2 sort 628 0.2 ps
This command chain works, because you have specifically ordered your process list columns and can therefore sort on those columns to get the information you really want. If you want to look for processes by a different criteria, there are other solutions available. Listing specific processes Obviously, once you get a list of running processes, you might want to list specific processes. An obvious way to do this is to combine the output of ps with grep to extract the information you want. On some UNIX variants, you might have access to a specific utility for doing this, such as pgrep, but often it can be just as effective to use grep if you are looking for specific commands:
$ ps -ef|grep bash
The ps command also supports showing processes according to more specific criteria, such as user ID, parent process, or controlling terminal. For example, the -U command line option limits the list of processes shown to those with the specified user name. For example, to show the processes currently owned by the root user, see Listing 11.
Listing 11. Listing processes by user
$ ps -U root PID TTY 0 ? 1 ? 2 ? 3 ? 308 ? 7 ? ... 552 ? 527 ? 629 pts/1 548 ?
TIME CMD 15:24 sched 0:00 init 0:00 pageout 0:02 fsflush 0:00 devfsadm 0:06 svc.star 0:00 0:00 0:00 0:01 in.telne dmispd ps snmpd
To get all of the processes on a specified terminal, use -t, as shown here in Listing 12.
Listing 12. Listing by terminal

$ ps -t 3 PID TTY 19915 pts/3 29145 pts/3 32256 pts/3 TIME CMD 00:00:00 bash 00:00:00 emacs 00:00:00 emacs
Once you have this information, you might want to use it to perform some operations on the processes. Signaling multiple processes One of the most common commands to run once you have found the process you are looking for is kill. It sends one or more processes a specific signal. In the case of a daemon that has started multiple threads or child processes, you can also try sending a signal to the parent process to signal all processes. However, this doesn't work with all daemons and applications. Obviously, you want to avoid having to manually pick out the processes. Some UNIX variants come with a tool called pkill, which can send the same signal to processes matching a particular pattern or other criteria, such as those on a particular terminal, process group, user ID, or group ID list. You can simulate the basic operation of sending a signal to multiple processes matching a particular command pattern by chaining together the ps, grep, awk, xargs and kill commands. For example, to send the kill signal to all of the processes with "httpd" in their command, you might use:
$ ps -e -opid,command |grep httpd|awk '{print $1}'|xargs kill -9
It is easier to understand if you dissect the individual elements:

$ ps -e -opid,command
This shows a list of all the running processes (in an SVR4 system, use -A for BSD). It shows only the process ID and command that was being executed. You don't need any other information, and using more detailed output might introduce text that would otherwise match your search:
$ ps -e -opid,command |grep httpd
This extracts only processes that have httpd in the command (since the only other information being generated from the process list is the process ID):
$ ps -e -opid,command |grep httpd|awk '{print $1}'
By using awk, you filter the output printing out of only the first argument, the process ID:
$ ps -e -opid,command |grep httpd|awk '{print $1}'|xargs kill -9
The xargs command takes a list of space-delimited items (spaces that include returns, linefeeds, tabs, and one or more spaces) and formats it as a list of arguments supplied to the given command -- in this case, the kill command. It is best to place this into a script with a suitable name (pkill or killbyname). You can set the script to accept two arguments, the signal and the text to be matched, and even account for OS differences, as shown in Listing 13.
Listing 13. Killing processes by command string

#!/bin/sh HOSTTYPE=`uname -s` SIGNAL=$1 STRING=$2 if [ -z "$1" -o -z "$2" ] then echo Usage: $0 signal string exit 1
fi case $HOSTTYPE in Darwin|BSD) ps -a -opid,command | grep $STRING | awk '{ print $1; }' | xargs kill $SIGNAL ;; Linux|Solaris|AIX|HP-UX) ps -e -opid,command | grep $STRING | awk '{ print $1; }' | xargs kill $SIGNAL ;; esac
The same basic technique shown here can be used for other similar collations. Calculating memory usage The ps tool provides two columns that haven't been covered yet. The RSS column provides the "resident set size" of a process; this is the amount of physical memory used by the process and a good indication of how much real memory a given process is using. The VSZ column details the total amount of memory being used by a process, including what is allocated for internal storage, but often swapped to disk. Both of these columns are common to the majority of ps variants. Determining the two figures gives a good idea of memory usage. If you combine the output from ps with grep to select specific processes and awk to calculate the totals, you can get a good idea of how much physical and virtual memory is being used up by single application, or an application and its children. For example, to determine the amount of real and virtual memory being used by the bash process, you could use the command in Listing 14.
Listing 14. Calculating memory usage with ps and awk

$ ps -A -o rss,vsz,command|grep bash | \ awk '{rss += $1; vsz += $2 } END { print "Real: ",rss, "Virtual: ",vsz }' Real: 4004 Virtual: 305624
This can be particularly useful when you are diagnosing memory and swap usage problems. Using a job control compatible shell It is fairly common for a typical system administrator to have more than one or two specific tasks on the go at any one time. Although you might have more than one
connection open to a server at any time -- whether that is multiple terminal windows (through xterm, for example) or other terminals, or remotely through SSH or Telnet -there are times when within your active shell or environment you want to control or monitor multiple processes. All shells support the option to run a command automatically in the background by appending the ampersand (&) to the end of the command. But sometimes you want to put an interactive application, such as an editor into the background so that you can run a shell command and then return to your editor session. The ability to control background processes in this way is called job control and a standard feature on both the Korn and C shells, and also open source shells, such as bash and zsh. For basic job control within the shell every time you start a command to run in the background, the command (which can be any valid command line, even inline scripts) is given a job reference ID.
$ find / -name "core" >/tmp/corelist 2>&1 & [3] 11957
You can get a list of the running background jobs using the jobs command, as shown in Listing 15.
Listing 15. Using the jobs command

$ jobs [1]- Stopped [2]+ Stopped [3] Running emacs MCSLP/Intranet/News.pm emacs MCSLP/Intranet/Media.pm find / -name "core" >/tmp/corelist 2>&1 &
In the listing, the second emacs command is denoted with the + sign. This indicates that the shell considers it to be the currently active job. The find command you started previously is not the active job, because it does not require interaction (although it generates output, it doesn't require input to continue), and therefore is not the active process. The first emacs process is denoted with -- to indicate that the shell considers this the previously active command. You can reference these jobs using the %+ and %strings, respectively. You can switch any of these running jobs to be the foreground process by typing fg followed by the job number, or one of the job strings (%+, %-). If you omit a reference, the shell switches to the currently active job.
To suspend the currently running process, press Control-Z. You can reconfigure this using:
ftp> [3]+ Stopped ftp atuin
This works with many different commands and applications. It should also work for the majority of simple commands you run in the shell, such as ls or find. Note that the job is marked as Stopped. This means that the execution of the command has been paused. To switch the command into a background process, use the bg command. Like fg, bg either takes a job reference or defaults to the currently active job with no arguments. If the command is one that requires input (an editor, FTP, and so forth), then the next time you press return after the bg command, you will be warned that the process has been temporarily suspended (see Listing 16).
Listing 16. Warning that process has been temporarily suspended

$ bg [3]+ ftp atuin & $ [3]+ Stopped
Note that if your background command generates output and you haven't redirected it, it will be output to the screen, even if you have placed the command in the background through job control . In a busy environment, job control becomes one of the easiest ways to manage and control multiple background jobs, or to quickly drop out of editors and other commands back to your active shell (as opposed to running a new shell).
If you want to have access to a job control shell, or anything other than the standard shell when you are in single user or a recovery mode on a UNIX system, then you must make sure that a statically linked copy of the shell is available within one of the standard binary directories, such as /bin or /sbin (this directory is specifically for statically linked binaries of applications). This will ensure that the shell is available when secondary filesystems have not been loaded.
Running processes reliably in the background You will occasionally want to run a script, utility, or command line in the background. However, most systems terminate a command running in the background if the user is disconnected, or they log out, which can be opposite of your intentions if you want to log in, start a command, and log out again.
Such termination can be particularly frustrating if you have to restart or otherwise reinitialize background or daemon processes that do not automatically daemonize themselves, or that rely on a separate script to start and manage the execution of the of a daemon application. The MySQL mysqld_safe script is a good example of a script that works in this way. To prevent the application from automatically terminating when you log out, use the nohup command as a prefix to the command line or tool you want to run, like this:
$ nohup find/ -name core
Unless you specifically redirect the output of your commands, nohup automatically writes the standard output and standard error to a file called nohup.out in the current directory. You can output to your own file just by using standard redirection, but remember to redirect both the output and error streams, for example:
$ nohup find/ -name core >/tmp/corefind.out 2>&1
I find that I almost automatically use nohup to run any command that I think will last longer than two or three minutes, even if I am running the command on the console. This probably has as much to do with the automatic redirection of the output than its ability to prevent termination during a connection failure. Back to top Summary Using the techniques shown in this article, you should be able to quickly find the UNIX processes you want, and the information about them. If you are in a heterogeneous environment, then you can also use the command line options to standardize the output you generate, making it easier for you to find the processes you want.
System Administration Toolkit: Migrating and moving UNIX filesystems

Summary: Learn how to transfer an entire file system on a live system, including how to create, copy, and re-enable the new file system. If you have a UNIX disk or system failure or simply fill up your file system, then you need to create a new partition and file system and copy over the contents. You might even need to mount the new partition in place to preserve the location of vital files and components. To add further complications, you need to do this on a live system, where you'd need to preserve file permissions, ownership, and possibly named pipes and other components. Effectively transferring these components and retaining all of this information is a vital part of the migration process. View more content in this series
Date: 03 Jul 2006 Level: Intermediate Also available in: Chinese Russian Activity: 20457 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different
UNIX environments, including methods of simplifying administration in a heterogeneous environment. Moving a UNIX directory or filesystem There are many occasions when you might need to move a UNIX filesystem from one device or hard disk partition to another, or when you need to move content from a file system to free up space, creating a new file system in the process. There might be an impending fault with the device, or you might simply be running out of space. You can do this on a system that is running in single-user mode, or on a live and running system where you need to ensure that the files are available during the move. In the case of the latter situation -- migrating a live file system -- it could be that you need to move an application while the application is running. In either case, you might have to move a system-related file system (such as /usr or /var), and whether you are running in single-user mode or not, the file system might actually be in active use. In any file system movement operation, you must ensure that the data has been correctly copied to the new destination. As a typical example, imagine that you have a system with a file/disk layout and usage like that in Listing 1.
Listing 1. Example file/disk layout and usage

Filesystem /dev/sda3 udev /dev/hda1 /dev/hdb1 /dev/hde1 /dev/hde4 none 1K-blocks 7692908 517560 115377640 115380192 96132940 22960280 517560 Used Available Use% Mounted on 6467756 1225152 84% / 184 517376 1% /dev 1743668 107773060 2% /var/lib/mysql 14604460 94914696 14% /export/data 3962940 87286644 5% /export/home 133484 21660480 1% /tmp 0 517560 0% /dev/shm
As you can see in Listing 1, the root file system is 84 percent full and you should probably move a directory from the root file system that is not already on a different partition to a new partition, or device, to ensure that the root file system does not fill up. It is best to move a single directory with a specific purpose that has the most significant impact on the file system that you want to free up space on. For example, in this case, you might want to put the /var or /usr directory onto its own file system. Moving multiple, smaller, directories is not as efficient. You want a single and easy to move directory to free up space. In some situations, you will be moving or copying an existing partition to a new location (for example, during a potential device failure). Here, the choice of the directory or existing file system already has been made for you.
Throughout this article, I assume that you have already added a new hard disk device, or have a spare disk device or partition to use as the destination for the directory or file system. Back to top Quick guide to moving data The basic sequence for moving a file system, or for placing an existing directory onto a new file system, is as follows:

Choose the data to be copied. Create a new partition. Create a new file system on the partition. Mount the partition on a temporary directory. Copy the data to the temporary directory. Rename the original directory. Create the new mount point. Mount the file system.
Ideally, at the end of this, you should reboot, although this isn't always necessary or possible; you should certainly try to reboot at the earliest opportunity. Let's take a closer look at each step of the sequence. Back to top Creating a new partition and file system Before you create a new partition and file system, first ensure that the available size and space matches the size of the files/directory that you are going to be placing on the new file system. The easiest way to do this is to use the du tool to determine the current file/directory size. For example, if you are going to move the /usr directory from the example system, you would use the command shown in Listing 2.
Listing 2. Moving the /usr directory from the example system

$ du -sk /usr 3923068 /usr
The -s option ensures that it generates a summary of the whole directory, rather than the contents, and the -k option ensures that you get the directory size in kilobytes. From the above, you can determine that you need a new partition that is at least 4GB in size.
If you are moving an existing file system to a larger partition, (rather than a single directory within an existing file system) then use the output from df to determine the current file system size. Once you know the size, you should create a partition that is ideally 25 percent larger than the size you need, and certainly no less than 10 percent larger. For the example file system above, you need a new partition that is at least 5GB in size. To create a new partition and file system, you need to use the tools that are appropriate for your operating system. For example:
Under Linux, you need to use fdisk to configure the partitions on a physical hard drive and one of the mke2fs or mkfs.* creation commands to create a suitable file system on the new partition. On a Solaris SPARC box, use format to create or configure the partition. On a Solaris x86 box, you might also need to use fdisk to configure the disk partition before using format to configure the Sun partition table. You can then use newfs (or one of the file system specific commands) to create the file system. Under AIX, use mkvg, chpv, or some other volume management tool to create a suitable partition or volume group for your new partition. You can then use crfs to create the file system.
With the file system in place, create a new directory under which you can mount the new file system while the data is copied. For example, I prefer to create a new file system within a similar location and with a name that identifies the file system as temporary. With the /usr example above, I would create a new directory, /mnt/usr.tmp, where the new file system would be mounted. Back to top Copying the information Actually copying the information is probably the simplest part of the process, but choosing the right tool can make a big difference to how effectively and efficiently the information is transferred. You should, however, take care to ensure that you are copying files to the correct location; you want to avoid overwriting existing data. There are many different methods available, but the two primary solutions are to use the basic cp tool or the tar tool. The former is a very quick method but, on some operating systems, you might experience problems with non-standard files, such as pipes and devices. The tar tool can be a slower alternative, but it is more reliable when transferring other file types and provides more visual feedback about the copy process. Whichever method you use, you should ensure that any files on the source file system are not actively being updated. Remember that any copy you make only contains the data that was in the individual files at the point in time the copy was made. If files are
being updated on the file system at the time, the files on the new file system might be incomplete and corrupt. Back to top Using cp The cp tool supports a recursive copying option (-r) that copies all the files, directories, and the files within those directories to the destination. You can also use the -p option to preserve ownership and permissions on the copied files. If you want to ensure the new file system retains the security details of the source, this is vital. There are few situations when this is not a requirement. To copy the files from an existing directory to the new location, follow these steps:
1. Change to the new target directory and confirm that you are in the right place (see Listing 3).
Listing 3. Changing the target directory

$ cd /mnt/usr.tmp $ pwd /mnt/usr.tmp
2. Copy the files from the old directory to the current (new) directory, as shown in Listing 4.
Listing 4. Copying the files to the new directory

$ cp -pr /usr/* .
You should perform a quick verification that the files have copied over. A simple ls will give an indication (see Listing 5). You'll examine more detailed checking methods later.
Listing 5. Verifying that the files have copied over

$ ls -l /usr total 238 drwxr-xr-x 2 root lrwxrwxrwx 1 root
bin root
1024 Apr 20 13:11 4lib/ 5 Apr 20 12:40 5bin -> ./bin/
lrwxrwxrwx 1 root drwxr-xr-x 6 root lrwxrwxrwx 1 root lrwxrwxrwx 1 root ../var/adm/ drwxr-xr-x 10 root drwxr-xr-x 8 root drwxr-xr-x 8 root drwx-----8 root drwxr-xr-x 4 root drwxr-xr-x 4 root ... lrwxrwxrwx 1 root ../var/tmp/ drwxr-xr-x 4 root drwxr-xr-x 4 root drwxr-xr-x 3 root drwxr-xr-x 7 root drwxr-xr-x 5 root drwxr-xr-x 3 root
root bin root root bin bin bin bin bin bin root bin bin bin bin bin bin
9 512 3 10 512 512 512 512 16384 512
Apr Apr Apr Apr Apr Apr Apr Apr Apr Apr
20 20 20 20 20 20 20 20 20 20
12:34 12:42 12:41 13:03 12:59 12:47 12:53 12:53 13:17 12:33
X -> ./openwin/ X11/ X11R6 -> X11/ adm -> apache/ apache2/ appserver/ aset/ bin/ ccs/
10 Apr 20 12:32 tmp -> 2048 512 1024 512 512 512 Apr Apr Apr Apr Apr Apr 20 20 20 20 20 20 13:00 13:17 13:17 13:03 12:44 12:40 ucb/ ucbinclude/ ucblib/ vmsys/ xpg4/ xpg6
If you discover that some files have not been copied over correctly or if symbolic links and other special filetypes are not copying properly, then you might want to try tar. Back to top Using tar The tar tool offers a number of benefits over cp. First and foremost, it tends to be more reliable for non-standard file types. Second, because it can provide visual feedback on the files being copied, it can be a more comforting method of copying the files across to the new system. Last, but certainly not least, you can use tar to create an archive file of the file system,which can act as a backup of your source directory/file system in event of a problem. On a user file system with a computer in single-user mode, the tar file method can be used to create the new file system content without mounting the new destination at a temporary directory mount point. The best way to use tar for copying is to pipe the file created by tar through another tar in the new location that is extracting the files. An example is shown in Listing 6.
Listing 6. Using tar for copying

$ cd /usr $ tar cfp - ./* |(cd /mnt/usr.tmp; tar xvfp -)
The c argument in the first tar tells tar to create an archive. The v, f, and p options specify verbose output (prints the files being added/extracted), writes/reads from a file (instead of a tape device), and retains permissions and ownership, respectively. The line in Listing 6 works, because the second half of the pipe first changes to the target directory before reading the .tar file that is being created in the first half of the pipe from standard input. If you want to create a .tar file and use this, rather than performing the direct copy, you must have a file system capable of holding all the files from your source directory. Listing 7 shows the process for copying.
Listing 7. Process for copying

$ $ $ $ cd /usr tar cfp /tmp/usr.tar ./* cd /mnt/usr.tmp tar xvfp /tmp/usr.tar
Whichever solution you use, you should get a report as each file is copied into, or out of, the archive, providing you use the v command-line option (see Listing 8).
Listing 8. Printout of copy process

a ./4lib/ 0K a ./4lib/libX.so.1.0 symbolic link to ./libX11.so.4.3 a ./4lib/libX11.so.4.3 216K a ./4lib/libXaw.so.4.0 208K a ./4lib/libXmu.so.4.0 72K a ./4lib/libXol.so.3.1 1056K a ./4lib/libXt.so.4.1 264K a ./4lib/libce.so.0.0 48K a ./4lib/libdeskset.so.0.1 64K a ./4lib/libdga.so.1.0 40K a ./4lib/libhelp.so.1.0 24K a ./4lib/libolgx.so.3.1 56K a ./4lib/libtt.so.1.1 848K a ./4lib/libttstub.so.1.1 32K a ./4lib/libxview.so.3.73 1328K a ./4lib/libdl.so.1.0 symbolic link to ../../lib/libdl.so.1 a ./4lib/libc.so.1.9 403K a ./4lib/libc.so.2.9 402K ...
Back to top
Using tar and direct file system exchange To copy the contents of a file system to a new location without creating a temporary mount point, you must first have the space to hold all of the files from your source directory. If replacing a directory with a new file system:
1. Create the new partition and file system as before. 2. Create a .tar file of the source directory (see Listing 9).
Listing 9. Creating a .tar file

$ cd /home $ tar cfvp /tmp/home.tar
3. Rename the source directory (see Listing 10).
Listing 10. Renaming the source directory

$ cd .. $ mv home home.old
4. Create the directory and set the same permissions and ownership as the original directory. 5. Mount the new file system on the new directory. 6. Extract the .tar file (see Listing 11).
Listing 11. Extracting the .tar file

$ cd home $ tar xvfp /tmp/home.tar
If you are changing the partition for an existing file system (shown in Listing 10), then the sequence is similar, but you unmount the existing file system rather than renaming the directory (see Listing 12).
Listing 12. Changing the partition for an existing file system

$ cd .. $ umount /home
This option is still safe, because you have a complete copy of the source file system on the old partition. You haven't deleted the contents or source material in either solution. Back to top Verifying the copy Whether you use cp or tar, you should always verify the copy has completed successfully. Although tar confirms the files that are copied (with the v option), you should ensure that the files are correctly recreated on the new file system. First, check the output of du on both the old and the new system (see Listing 13).
Listing 13. Checking the output of du

$ du -sk /usr 3923068 /usr $ du -sk /mnt/usr.tmp 3923068 /mnt/usr.tmp
The two numbers should be identical; however, depending on the type of the new file system and file and directory allocation sizes on the old file system/directory and the new one, you might notice a slight discrepancy with the size. Another good test is just to compare the number of files/directories on the source and destination. You can do this with a simple find command, as shown in Listing 14.
Listing 14. Comparing the number of files/directories on source and destination

$ find /usr |wc -l 347001 $ find /mnt/usr.tmp |wc -l 347001
Another good test, if you are copying file system to file system, is to compare inode numbers; these show how many inodes have been allocated. If you are copying from one file system to another of exactly the same type, the number of inodes used should be identical. Use df, with the -i command-line option, to get the inode statistics or, on traditional UNIX systems, the bare df output shows the number of 'files' created. The example in Listing 15 is from a Linux system.
Listing 15. Comparing inode numbers

$ df -i /usr Filesystem /dev/sda3
Inodes 977280
IUsed 411959
IFree IUse% Mounted on 565321 43% /
The example in Listing 16 is on Solaris.
Listing 16. Comparing inode numbers on Solaris

$ /bin/df /usr /usr
(/dev/dsk/c0t0d0s3 ): 9076010 blocks
863695 files
Back to top Updating your system to reflect the new organization By this stage, you should have a new file system that contains a copy of the file system or directory that you are moving to a new file system. You should now update your system files (particularly the file system mount information) to reflect the new structure. This information is stored in /etc/fstab, /etc/vfstab, or might be available through a specific administration tool, such as SAM on HP-UX. If you have been migrating a directory within an existing file system:
1. Rename the original directory. 2. Create a new directory. 3. Use chown and chmod to set the ownership and permissions on the new directory.
Ideally, you should now reboot your system to ensure that that the new layout is used. You must reboot if you are migrating a file system from one device to another. It is unlikely you will be able to unmount the existing file system, especially if it is a system directory (in other words, one under /var or /usr).
If a reboot is not possible, manually mount the new system in the new mountpoint and then reboot as soon as possible. Once you have rebooted and confirmed everything is working properly, you can delete the old directory or reuse the old partition. Back to top Summary There are many situations where copying a UNIX filesystem, live or otherwise, is required. It could be because you are running out of space, requiring a larger partition for your file system to enable installation of software, or even an impending hardware fault. In any of these situations, you need to use the techniques covered in this article to copy over the existing files to the file system. Such a copy is not without its traps -- copying a live file system can be risky, especially if there are open files. You should also be careful to ensure that you don't accidentally overwrite a partition, or existing files, with the files you are trying to copy. However, as you've seen here with some careful thought, you can effectively migrate files reliably to take advantage of more space, even on a live system.
Systems Administration Toolkit: Using SNMP data

Martin Brown (mc@mcslp.com), Professional writer, Freelance
Summary: The Simple Network Management Protocol (SNMP) is built in to many devices, but often the tools and software that can read and parse this information are too large and complicated when you only want to check a quick statistic or track a particular device or issue. This article looks at some simplified methods for getting SNMP information from your devices and how to integrate this information into the rest of your network's data map. View more content in this series Tags for this article: aix, aix_and_unix, protocol, snmp, toolkit, unix
Date: 15 Apr 2008 Level: Intermediate PDF: A4 and Letter (47KB)Get Adobe Reader Also available in: Chinese Activity: 21678 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Back to top SNMP basics There are many ways you can monitor your UNIX server. See the Resources for some examples of the type of monitoring available. Monitoring a single server is not a problem, but monitoring the same information across a number of servers can present problems. If one of the servers you are in charge of runs out of disk space, you want to know about it before it starts to affect your users and clients. Monitoring multiple servers in this way, especially if they use a variety of different operating systems, can be a problem. The differences in command line tools, output formats, values, and other information all complicate what should otherwise be a simple process. What is required is a solution that provides a generic interface to the information that works, irrespective of the UNIX variant you are using. The Simple Network Management Protocol (SNMP) provides a method for managing information about different systems. An agent runs on each system and reports information using SNMP to different managing systems. SNMP is often a built-in component for network devices such as routers and switches, and is the only method available for retrieving statistics and status information remotely (without logging in to some sort of interface). On most hosts you will need to explicitly run SNMP software to expose information about the host over the SNMP protocol.
Information can be retrieved from an agent either explicitly, by requesting the information using a GET request, or the agent can broadcast information to management systems using the TRAP or INFORM messages. In addition, managing systems can set information and parameters on the agent, but this is usually only used to change the network configuration. The types of information that can be shared can be quite varied. It can be everything from network settings, statistics, and metric data for network interfaces, through to monitoring CPU load and disk space. The SNMP standard does not define what information the agent returns; instead, the available information is defined by Management Information Bases (MIBs). The MIB defines the structure of the information that is returned, and are organized into a hierarchical structure using object identifiers (OID). You access information within an agent by requesting data using a specific location within the MIB structure. For example, some of the more common IDs are shown in Listing 1.
Listing 1. SNMP object IDs

sysDescr.0 sysObjectId.0 sysUpTime.0 sysContact.0 sysName.0 sysLocation.0 sysServices.0 ifNumber.0 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.2.0 1.3.6.1.2.1.1.3.0 1.3.6.1.2.1.1.4.0 1.3.6.1.2.1.1.5.0 1.3.6.1.2.1.1.6.0 1.3.6.1.2.1.1.7.0 1.3.6.1.2.1.2.1.0
You can see from this list that the MIBs are numerical and, effectively, in sequence. When obtaining information you can use a GET request to obtain a specific value, or GETNEXT to get the next property from the last one you read. You can also use the names. The names shown above are all part of the system tree, so you can read the value by getting using the OID 'system.sysUpTime.0'. The values that you read are also of specific types. You can read integer, floating point, and string values that are all defined as 'scalar' objects. Within these objects are types that are identified with specific significance. For example, time interval values are reported as 'timeticks,' or hundredths of a second. These values need to be converted into a more readable human form before being displayed. There are also MIB objects that return tabular data. This is handled by returning additional OID instances that can be grouped together to make an SNMP table. From a security perspective, SNMP agents can be associated with a specific community, and managing systems access information by using the community as a
method of validating their access to the agent. In Version 1 of the SNMP standard, the community string was the only method of securing or restricting access. With Version 2 of the SNMP standard, the security was improved, but could be complex to handle. With Version 3, considered the current version since 2004, the standard was improved with explicit authentication and access control systems. Back to top Getting SNMP statistics There are many different ways of obtaining information from SNMP systems, including using professional management tools, programming interfaces, and command line tools. Of the latter, probably the best known and easiest to use is the snmpwalk command, which is part of a larger suite of SNMP tools that allow you to obtain information from SNMP agents directly from the command line. This command will walk the entire subtree of a given management value and return all the information about the system contained within the subtree. For example, Listing 2 shows the output when querying a local system for all the information within the 'system' tree.
Listing 2. 'Walking' an SNMP tree

$ snmpwalk -Os -c MCSLP -v 1 localhost system sysDescr.0 = STRING: Linux tweedledum 2.6.23-gentoo-r8 #1 SMP Tue Feb 12 16:32:14 GMT 2008 x86_64 sysObjectID.0 = OID: netSnmpAgentOIDs.10 sysUpTimeInstance = Timeticks: (34145553) 3 days, 22:50:55.53 sysContact.0 = STRING: root@Unknown sysName.0 = STRING: tweedledum sysLocation.0 = STRING: serverroom sysORLastChange.0 = Timeticks: (0) 0:00:00.00 sysORID.1 = OID: snmpFrameworkMIBCompliance sysORID.2 = OID: snmpMPDCompliance sysORID.3 = OID: usmMIBCompliance sysORID.4 = OID: snmpMIB sysORID.5 = OID: tcpMIB sysORID.6 = OID: ip sysORID.7 = OID: udpMIB sysORID.8 = OID: vacmBasicGroup sysORDescr.1 = STRING: The SNMP Management Architecture MIB. sysORDescr.2 = STRING: The MIB for Message Processing and Dispatching. sysORDescr.3 = STRING: The management information definitions for the SNMP User-based Security Model. sysORDescr.4 = STRING: The MIB module for SNMPv2 entities sysORDescr.5 = STRING: The MIB module for managing TCP implementations sysORDescr.6 = STRING: The MIB module for managing IP and ICMP
implementations sysORDescr.7 = STRING: The MIB module for managing UDP implementations sysORDescr.8 = STRING: View-based Access Control Model for SNMP. sysORUpTime.1 = Timeticks: (0) 0:00:00.00 sysORUpTime.2 = Timeticks: (0) 0:00:00.00 sysORUpTime.3 = Timeticks: (0) 0:00:00.00 sysORUpTime.4 = Timeticks: (0) 0:00:00.00 sysORUpTime.5 = Timeticks: (0) 0:00:00.00 sysORUpTime.6 = Timeticks: (0) 0:00:00.00 sysORUpTime.7 = Timeticks: (0) 0:00:00.00 sysORUpTime.8 = Timeticks: (0) 0:00:00.00
You can see here a range of information about the host, including the operating system (in sysDescr.0), the amount of time that the system has been available (sysUpTimeInstance), and the location of the machine. The interval time here is shown in both its original value (timeticks) and the converted, human-readable days, hours:minutes:seconds. The uptime or availability of a machine is a very common use for SNMP, as it provides probably the most convenient and efficient method for determine whether a machine is up and processing requests. Other solutions that have been described in past parts of the series include ping or using rwho and ruptime. These latter two solutions are very CPU and network intensive and not very friendly in terms of their resource utilization. Note, however, the limitation of the uptime described here, which is the information shown in the uptime of the SNMP agent, not the uptime of the entire machine. In most situations the two are same, especially for devices with built-in SNMP monitoring, such as network routers and switches. For computers that expose their status through SNMP, there may be a discrepancy between system and SNMP agent uptime. You can get a quicker idea of the status of a machine through SNMP using snmpstatus. This obtains a number of data points from a specified SNMP agent, including the IP address, description, uptime, and network statistics (packets sent/received, and IP packets sent/received). For example, if we look at a Solaris host, you can see the simplified information, as shown in Listing 3.
Listing 3. Simplified information

$ snmpstatus -v1 -c public t1000 [192.168.0.26]=>[SunOS t1000 5.11 snv_81 sun4v] Up: 2:12:10.20 Interfaces: 4, Recv/Trans packets: 643/160 | IP: 456/60 2 interfaces are down!
This machine has recently been rebooted (hence the low uptime and packet statistics). The snmpstatus command has also determined that two of the interfaces on the
machine (which has four Ethernet ports) are down. This is a good example of the sort of warning information that SNMP can provide to help notify you of an issue that requires further investigating. For obtaining a specific piece of information, you can use the snmpget command, which reads one or more OIDs directly and reports their value. For special types, it will also convert to a human-readable format. For example, to get the system description and uptime, use the following command (in Listing 4).
Listing 4. Getting system description and uptime information

$ snmpget -v1 -c public t1000 system.sysUpTime.0 system.sysContact.0 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (867411) 2:24:34.11 SNMPv2-MIB::sysContact.0 = STRING: "System administrator"
In isolation, all of these methods are useful, but in reality, you need to be able to monitor and track multiple machines and multiple OIDs to get a full picture of what is going on. We can do this by using one of the many programmable interfaces to SNMP. Back to top Getting SNMP data programmatically The Net::SNMP module for Perl obtains information from one or more agents using SNMP. Other, similar, interfaces are available for other languages, including Python, Ruby, and PHP (see Resources). The interface works by you creating a session that communicates (and if necessary authenticates) with the SNMP agent on the desired host. Once you have an active and valid session, you can request data from the agent directly for one or more OIDs. The information is returned in the form of a hash of information, tied between the OID and the corresponding value. Listing 5 shows a very simple script that will obtain the system uptime for each of the hosts supplied on the command line.
Listing 5. Getting a single SNMP agent property with Perl and Net::SNMP
#! /usr/local/bin/perl use strict; use Net::SNMP; my $uptimeOID = '1.3.6.1.2.1.1.3.0';
foreach my $host (@ARGV) { my ($session, $error) = Net::SNMP->session( -hostname => $host, -community => 'public', -port => 161 ); warn ("ERROR for $host: $error\n") unless (defined($session)); my $result = $session->get_request( -varbindlist => [$uptimeOID] ); if (!defined($result)) { warn ("ERROR: " . $session->error . "\n"); } else { printf("Uptime for %s: %s\n",$host, $result->{$uptimeOID}); } $session->close; }
In the script, we've provided the full numerical OID for the system, sysUpTime property. You have to supply the list of OIDs to obtain when using the get_request() method as a reference to an array, and then pull the information back out from the hash that is returned. In Listing 5 we build the array reference dynamically during the call, and then use the OID as the hash key when printing out the result. Using the script, we can get a list of the uptimes for each host supplied on the command line (see Listing 6).
Listing 6. List of uptimes for each host

$ perl uptime.pl tweedledum t1000 Uptime for tweedledum: 4 minutes, 52.52 Uptime for t1000: 6 minutes, 26.12
Of course, watching this information manually is hardly efficient. Back to top Tracking SNMP data over time
Viewing a single instance of an SNMP OID property at one time is not always very useful. Often you want to monitor something over time (for example, availability), or you want to monitor for changes in particular values. A good example is disk space. SNMP can be configured to record all sorts of information, and disk space is a common system to want to monitor so that you can identify not only when the disk space reaches a particular level, but also when there is a significant change to the disk space, which might signify a problem. For example, Listing 7 shows a callback-based solution to constantly monitor the diskspace. In the script, we output a running total, but it could be configured to only output the warning message that is triggered when there is a reduction in the diskspace.
Listing 7. Getting a running view of SNMP properties

#! /usr/local/bin/perl use strict; use warnings; use Net::SNMP qw(snmp_dispatcher); my $diskspaceOID = '1.3.6.1.4.1.2021.9.1.7.1'; foreach my $host (@ARGV) { my ($session, $error) = Net::SNMP->session( -hostname => $host, -nonblocking => 0x1, ); if (!defined($session)) { warn "ERROR: $host produced $error - not monitoring\n" } else { my ($last_poll) = (0); $session->get_request( -varbindlist => [$diskspaceOID], -callback => [ \&diskspace_cb, \$last_poll ] ); } } snmp_dispatcher(); exit 0; sub diskspace_cb
{ my ($session, $last_poll) = @_; if (!defined($session->var_bind_list)) { printf("%-15s ERROR: %s\n", $session->hostname, $session->error); } else { my $space = $session->var_bind_list->{$diskspaceOID}; if ($space < ${$last_poll}) { my $diff = ((${$last_poll}-$space)/${$last_poll})*100; printf("WARNING: %s has lost %0.2f%% diskspace)\n", $session->hostname,$diff); } printf("%-15s Ok (%s)\n", $session->hostname, $space ); ${$last_poll} = $space; } $session->get_request( -delay => 60, -varbindlist => [$diskspaceOID] ); }
The script is in two parts, and uses some functionality within the Net::SNMP module that allows you to call a function when an SNMP value is obtained from a host, coupled with the ability to continually monitor hosts and SNMP objects in a simple, but efficient, loop. The first part sets up each host to monitor the information. We are only monitoring one piece of information, but we could monitor others as part of the solution. The object is configured as 'non-blocking,' so that the script will not wait if the host cannot be reached, but simply move on to the next host. Finally, in the call to get_request(), we submit the callback information. The first argument here is the name of the function to be called when the response is received from the agent. The second is an argument that will be supplied to the function when it is called. We'll use this argument to be able to record and track the previous value returned by the SNMP call. Within the callback function, we compare the newly returned value and the previous value. If there's a reduction, we calculate the percentage reduction and then report a warning. The final part of the callback is to specify that another retrieval should occur, here specifying that the next retrieval should be delayed by 60 seconds. The existing
callback information is retained. In effect, the script obtains the value from the SNMP agent, calls the callback function, which then queues up another retrieval in the future. Because the same callback is already defined, the process repeats in an endless loop. Incidentally, the script uses the dskAvail OID value, and calculates the percentage difference based on the last and new values. The dskTable tree that this property is part of actually has a disk percentage property that we could have queried, instead of calculating it manually. However, the value returned is probably not finely grained enough to be useful. You can see this property and current values by using snmpwalk to output the dskTable tree, which itself is part of the UCD MIB (Listing 8).
Listing 8. Getting a dump of available MIB data

$ snmpwalk -v 1 localhost -c public UCD-SNMP-MIB::dskTable UCD-SNMP-MIB::dskIndex.1 = INTEGER: 1 UCD-SNMP-MIB::dskPath.1 = STRING: / UCD-SNMP-MIB::dskDevice.1 = STRING: /dev/sda3 UCD-SNMP-MIB::dskMinimum.1 = INTEGER: 100000 UCD-SNMP-MIB::dskMinPercent.1 = INTEGER: -1 UCD-SNMP-MIB::dskTotal.1 = INTEGER: 72793272 UCD-SNMP-MIB::dskAvail.1 = INTEGER: 62024000 UCD-SNMP-MIB::dskUsed.1 = INTEGER: 7071512 UCD-SNMP-MIB::dskPercent.1 = INTEGER: 10 UCD-SNMP-MIB::dskPercentNode.1 = INTEGER: 3 UCD-SNMP-MIB::dskErrorFlag.1 = INTEGER: noError(0) UCD-SNMP-MIB::dskErrorMsg.1 = STRING:
To find the property in the first place, you can dump all the known properties by using snmptranslate. By filtering this with grep we can see the information we want: $ snmptranslate -Ts |grep dsk. To get a numerical value, use snmptranslate and provide the name with the -On option (see Listing 9).
Listing 9. Using snmptranslate

$ snmptranslate -On UCD-SNMP-MIB::dskAvail .1.3.6.1.4.1.2021.9.1.7
Running the script, we get a running commentary (and warnings) for the disk space usage on the specified host. See Listing 10.
Listing 10. Monitoring disk space automatically

$ perl diskspace-auto.pl tweedledum tweedledum Ok (50319024) WARNING: tweedledum has lost 2.67% diskspace) tweedledum Ok (48976392) WARNING: tweedledum has lost 1.65% diskspace) tweedledum Ok (48166292) tweedledum Ok (48166292) tweedledum Ok (48166292) tweedledum Ok (48166292)
You can see from this output that we have lost some significant space out of the space available on this disk on the specified host. To monitor more hosts, just add more hostnames on the command line. Back to top Publishing information through an SNMP agent The SNMP package includes a daemon, snmpd, which can be configured to expose a variety of information using the SNMP protocol. The configuration for the information to be exposed is controlled using the /etc/snmpd.conf file. For example, Listing 11 shows the snmpd.conf file on the host used in the earlier examples in this article.
Listing 11. Sample snmpd.conf file

syslocation serverroom proc imapd 20 10 disk / 100000 load 5 10 10
Each of these lines populates different information. In the example, we set the location of the machine, and then configure some specific items to monitor. The proc section monitors a specific process, shown here as a monitor for the IMAP daemons for a mail service. The numbers following the option specify the maximum number of processes allowed to be running, and the minimum number that should be running. You can use this to make sure that a particular service is running, and that you haven't exceeded capacity that might indicate a fault. When the process count goes above the MAX value, an SNMP trap is generated.
For the disk, you specify the path to the directory to be monitored and the minimum size (in kilobytes) that the disk should have free. Again, an SNMP trap is triggered if the disk space dips below this value. Finally, the load information shows the maximum CPU load for 1, 5, and 15 minutes that should be reported. This is equivalent to the output of the uptime command that shows the process loading for these intervals. Like the other configured limits, a trap is raised when these limits are exceeded. Manually setting this information is not difficult, but also not ideal. A simple menu-based solution, snmpconf, is available if you want a more straightforward method of setting the configuration. Back to top Summary Monitoring your servers and devices is a process that can be very complex, especially as the number of devices in your network increases. SNMP is an efficient, and extensible, method for exposing and reporting this information. Because the interface is consistent across all the devices, you can get uptime, network statistics, disk space, and even process monitoring using the same methods across multiple hosts. In this article we've looked both at the basics of SNMP and also how to read specific values from different hosts. Using the Net::SNMP perl module we have also examined methods for reading information, using both one-hit and continual monitoring-based solutions. Finally, we examined the methods for configuration additional information to be exposed on a system so that you can customize and monitor the systems you need for your network when using the snmpd daemon.
Systems Administration Toolkit: Log file basics

Martin Brown (mc@mcslp.com), Professional Writer, Freelance
Summary: A typical UNIX or Linux machine creates many log files during the course of its operation. Some of these contain useful information; others can be used to help you with capacity and resource planning. This article looks at the fundamental information recorded within the different log files, their location, and how that information can be used to your benefit to work out what is going on within your system. Tags for this article: aix, aix_and_unix, dns, log_file, systems_admin, toolkit
Date: 26 Feb 2008 Level: Intermediate PDF: A4 and Letter (63KB)Get Adobe Reader Also available in: Chinese Japanese Activity: 27526 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Back to top Log files All systems generate a varying quantity of log files that track and record different information about your machine. The content and utility of these files varies from system to system, but the core information provided by the files is often consistent. For example, all UNIX and Linux machines use the syslog, a generic logging system that is used by the operating system and applications and services to log information. The syslog records a whole host of data, including logins, performance information, and failures reported by different hardware and systems. In addition to the syslog, systems also have a variety of service, environment, and application logs that record information about the machine and its operation. Although parsing and extracting the content of the log files for information can be time consuming and sometimes complex, the wealth of information in those logs is difficult to ignore. The log file can provide hints on potential problems, faults, security lapses and, if used correctly, can even help provide warnings on load and capacity of your servers.
Back to top Log locations The location of the various log files varies from system to system. On most UNIX and Linux systems the majority of the logs are located in /var/log. For example, Listing 1 shows a list of logs located on a Gentoo Linux system.
Listing 1. Linux /var/log directory contents

$ ll /var/log total 3312 -rw-r----- 1 root root 8218 2007-1103 06:21 dmesg -rw-rw---- 1 portage portage 650111 200802-02 13:01 emerge.log -rw------- 1 root root 24024 2007-1105 07:26 faillog -rw-r--r-- 1 root root 386032 200709-28 14:39 genkernel.log drwxr-xr-x 2 root root 4096 2007-1103 06:47 iptraf/ -rw-r--r-- 1 root root 292292 200802-03 08:07 lastlog -rw------- 1 root root 1346931 200802-03 08:50 messages drwxr-xr-x 2 root root 4096 2006-0830 17:04 news/ drwxr-xr-x 3
root root 4096 2007-0928 13:22 portage/ drwxrwx--- 2 root portage 4096 2007-1103 06:40 sandbox/ drwxrwx--- 2 snort snort 4096 2007-1013 11:34 snort/ -rw-rw-r-- 1 root utmp 496896 200802-03 08:07 wtmp -rw-rw-rw- 1 root mc 61189 2007-0610 11:37 Xorg.0.log -rw-rw-rw- 1 root root 61189 2007-0610 10:40 Xorg.0.log.old
On Solaris, IBM AIX, and HP-UX, the main syslog and most of the other logs are written to files within the /var/adm directory (Listing 2).
Listing 2. Traditional UNIX /var/adm contents

$ ls -al /var/adm total 230 drwxrwxr-x 9 root sys 512 Feb 3 15:30 . drwxr-xr-x 48 root sys 1024 Feb 3 15:32 .. drwxrwxr-x 5 adm adm
512 Feb 2 16:13 acct -rw------1 uucp bin 0 Jan 12 18:49 aculog drwxr-xr-x 2 adm adm 512 Feb 2 16:03 exacct -r--r--r-1 root root 2856 Feb 3 16:10 lastlog drwxr-xr-x 2 adm adm 512 Feb 2 16:03 log -rw-r--r-1 root root 69065 Feb 3 16:08 messages drwxr-xr-x 2 root sys 512 Feb 2 16:09 pool drwxrwxr-x 2 adm sys 512 Feb 2 16:13 sa drwxr-xr-x 2 root sys 512 Feb 2 17:03 sm.bin -rw-rw-rw1 root bin 0 Jan 12 18:47 spellhist drwxr-xr-x 2 root sys 512 Feb 2 16:03 streams -rw-------
1 root root 93 Feb 3 16:08 sulog -rw-r--r-1 root bin 3720 Feb 3 16:14 utmpx -rw-r--r-1 adm adm 29760 Feb 3 16:10 wtmpx
In addition, some non-system-level messages and information are written into logs located within /var/log (Listing 3). For example, on Solaris, by default, mail debug entries are written into /var/log/syslog.
Listing 3. Additional logs in /var/log on Solaris

$ ls -al /var/log/ total 48158 drwxr-xr-x 7 root sys 512 Feb 3 16:07 . drwxr-xr-x 48 root sys 1024 Feb 3 15:32 .. -rw------1 root sys 0 Jan 12 18:48 authlog -rw-r--r-1 root other 27 Feb 2 16:17 brlog drwxr-xr-x 2 root root 512 Feb 2 16:39 gdm drwxr-xr-x 2 root sys 512 Feb 2 16:09 pool -rw-r--r-1 root sys 24480410 Feb 3 12:51
postrun.log drwxr-xr-x 2 root sys 512 Feb 2 16:41 swupas -rw-r--r-1 root other 635 Feb 2 17:25 sysidconfig.log -rw-r--r-1 root sys 3967 Feb 3 16:08 syslog drwxr-xr-x 3 root sys 512 Feb 2 17:25 webconsole drwxr-xr-x 2 root sys 512 Feb 2 16:37 xen -rw-r--r-1 root root 66171 Feb 3 16:07 Xorg.0.log -rw-r--r-1 root root 66256 Feb 3 16:06 Xorg.0.log.old
Of course finding the files is the least of the issues. You need to know what the files contain for the information to be of any use. Depending on the UNIX variants, some logs may be littered about in other places, but there has been a significant attempt to standardize on log file locations to one of the directories already mentioned. Back to top Log types and data Log types fall into two categories, text log files that contain messages and information in a simple text format, and files that are encoded in a binary format. The former is used for most of the logs in your typical system as they are easy to write and, perhaps more importantly, easy to read. The issue with text files is that they can sometimes be difficult to extract information from in a structured way, because the text format of the files allows the information to be written in any way or structure.
The latter format is more practical for very structured information, or for information that needs to be written in a particular way or format. For the example, the utmp and wtmp data is written to a file in fixed blocks of binary data so that the information can read and be written in a quick and efficient format. Unfortunately, this means that the information is difficult to read without using special tools. System logs (syslog) The syslog service is a daemon that runs the background and accepts log entries and writes them to one or more individual files. All messages reported to syslog are tagged with the date, time, and hostname, and it's possible to have a single host that accepts all of the log messages from a number of hosts, writing out the information to a single file. Messages are also identified by the service that raise the issue (for example, mail, dhcp, kernel), and a class indicating the severity of the message. The severity can be marked as info (purely for information), warning, error, critical (a serious problem that needs addressing), and even emergency (the system needs urgent help). The service is highly configurable (generally through /etc/syslog.conf, or the equivalent), and allows you to select what classes of information to log, and where to log the information. For example, you can write all the standard information out to a file. But for critical messages, where administrators need the information right away, these messages can be sent immediately to the console. Listing 4 shows the main configuration content of the default syslog.conf file from a Solaris 10 installation.
Listing 4. Sample syslog.conf file

*.err;kern.notice;auth.notice /dev/sysmsg *.err;kern.debug;daemon.notice;mail.crit /var/adm/messages *.alert;kern.err;daemon.err operator *.alert root *.emerg * ... mail.debug ifdef('LOGHOST', /var/log/syslog, @loghost) ...
ifdef('LOGHOST', , user.err /dev/sysmsg user.err /var/adm/messages user.alert 'root, operator' user.emerg * )
Because syslog is a standard logging mechanism within UNIX/Linux it is used to record a massive array of different information. This includes boot messages, login and authorization information, and service startup/shutdown. In addition, syslog is often used to record e-mail messages delivery, filesystem issues, and even DHCP leases, DNS issues, and NFS problems. Because syslog can write the data to different areas, it's not always obvious that syslog is writing the information. The main destination for the on-disk copy of the syslog differs between UNIX variants. Many Linux solutions write the information to /var/log/messages. On AIX, Solaris, and HP-UX, the syslog is written to /var/adm/messages. You can see a sample of /var/adm/messages from a Solaris machine in Listing 5.
Listing 5. Sample system log output

Feb 3 16:06:58 solaris2 ata: [ID 496167 kern.info] cmdk2 at ata1 target 0 lun 0 Feb 3 16:06:58 solaris2 genunix: [ID 936769 kern.info] cmdk2 is /pci@0,0/pciide@1f,1/ide@1/cmdk@0,0 Feb 3 16:06:59 solaris2 asy: [ID 267298 kern.notice] asy0: UART @ 3f8 scratch register: expected 0x5a, g ot 0xff Feb 3 16:06:59 solaris2 asy: [ID 702181 kern.notice] Cannot identify UART chip at 3f8 Feb 3 16:06:59 solaris2 asy: [ID 267298 kern.notice] asy1: UART @ 2f8 scratch register:
expected 0x5a, got 0xff Feb 3 16:06:59 solaris2 asy: [ID 702181 kern.notice] Cannot identify UART chip at 2f8 Feb 3 16:07:01 solaris2 genunix: [ID 314293 kern.info] device pciclass,030000@2(display#0) keeps up device sd@1,0(sd#1), but the latter is not power managed Feb 3 16:07:01 solaris2 /usr/lib/power/powerd: [ID 387247 daemon.error] Able to open /dev/srn Feb 3 16:07:08 solaris2 /sbin/dhcpagent[164]: [ID 778557 daemon.warning] configure_v4_lease: no IP broadcast specified for ni0, making best guess Feb 3 16:07:31 solaris2 sendmail[503]: [ID 702911 mail.crit] My unqualified host name (solaris2) unknown; sleeping for retry Feb 3 16:07:32 solaris2 sendmail[507]: [ID 702911 mail.crit] My unqualified host name (solaris2) unknown; sleeping for retry Feb 3 16:07:48 solaris2 svc.startd[7]: [ID 652011 daemon.warning] svc:/system/webconsole:console: Method "/lib/svc/method/svcwebconsole start" failed with exit status 95. Feb 3 16:07:48 solaris2 svc.startd[7]: [ID 748625 daemon.error] system/webconsole:console failed fatally: transitioned to maintenance (see 'svcs -xv' for details) Feb 3 16:07:55 solaris2
pseudo: [ID 129642 kern.info] pseudo-device: devinfo0 Feb 3 16:07:55 solaris2 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0 Feb 3 16:08:31 solaris2 sendmail[503]: [ID 702911 mail.alert] unable to qualify my own domain name (solaris2) - using short name Feb 3 16:08:32 solaris2 sendmail[507]: [ID 702911 mail.alert] unable to qualify my own domain name (solaris2) -using short name
You can see in the sample output that there is a wide range of information here, from problems and issues with hardware devices through to issues with the current configuration of the mail service. The format of the file is quite straightforward: it contains the date, hostname, service name, a unique ID (to enable the system to log multi-line messages and have them identified), and the identifier and class of the entry. The remaining text on each line is just free-form text from the system logging the error message. The format of the file makes it easy to pull out the information you want. All the lines in the file are tagged with a unique ID and all lines are tagged with the identifier and class of the error message. For example, you can pull out information on critical issues with the mail system by using grep to pick out the entries tagged with mail.crit: $ grep mail.crit /var/adm/messages. To process the detail of the individual lines within the log is more complex. Although the first few columns within the file are standardized (they are written by the syslog daemon), the format of the remainder of the line is entirely dependent on the component reporting the error message. This can make it complex to read and parse the contents of the file, as you will need to treat each line according to the identifier and reporter. Even then, some lines will not follow a format. Kernel log (dmesg and alog)
All UNIX and Linux systems have a log that is actually part of the kernel. In practice the log is actually a section of memory in the kernel used to record information about the kernel that may be impossible to write to disk because the information is generated before the filesystems are loaded. For example, during the boot process, the filesystems are not accessible for writing (most kernels boot with the filesystem in read mode until the system is considered safe enough to switch to read/write mode). The data in this log contains information about the devices connected to the system and any faults and problems recorded by the system during the boot and operational process. On some systems the information is periodically dumped into a file (/var/log/dmesg); on others it is only available by using the alog command (AIX) or dmesg (all other UNIX/Linux variants). The information generated by the kernel is not always written out to another file, such as syslog. This can mean that certain pieces of information, such as internal data on devices and hardware, is only available through the dmesg log. For example, Listing 6 shows some sample output from dmesg on a Gentoo Linux system. Here it is showing the main boot information, trimmed for brevity.
Listing 6. The dmesg log contents

$ dmesg Linux version 2.6.22-gentoo-r8 (root@gentoo2.vm) (gcc version 4.1.2 (Gentoo 4.1.2 p1.0.1)) #1 SMP Fri Sep 28 14:22:07 GMT 2007 BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 000000000009fc00 (usable) BIOS-e820: 0000000000100000 0000000020000000 (usable) 0MB HIGHMEM available. 512MB LOWMEM available. Entering
add_active_range(0, 0, 131072) 0 entries of 256 used Zone PFN ranges: DMA 0 -> 4096 Normal 4096 -> 131072 HighMem 131072 -> 131072 early_node_map[1] active PFN ranges 0: 0 -> 131072 On node 0 totalpages: 131072 DMA zone: 32 pages used for memmap DMA zone: 0 pages reserved DMA zone: 4064 pages, LIFO batch:0 Normal zone: 992 pages used for memmap Normal zone: 125984 pages, LIFO batch:31 HighMem zone: 0 pages used for memmap DMI not present or invalid. Allocating PCI resources starting at 30000000 (gap: 20000000:e0000000) Built 1 zonelists. Total pages: 130048 Kernel command line: root=/dev/ram0 init=/linuxrc ramdisk=8192 real_root=/dev/hda3 udev Local APIC disabled by BIOS -- you can enable it with "lapic" mapped APIC to ffffd000 (0140c000) Enabling fast FPU save and restore... done. Enabling unmasked
SIMD FPU exception support... done. Initializing CPU#0 CPU 0 irqstacks, hard=c054e000 soft=c052e000 PID hash table entries: 2048 (order: 11, 8192 bytes) Detected 2295.874 MHz processor. Console: colour VGA+ 80x25 Dentry cache hash table entries: 65536 (order: 6, 262144 bytes) Inode-cache hash table entries: 32768 (order: 5, 131072 bytes) Memory: 511616k/524288k available (3150k kernel code, 12100k reserved, 818k data, 264k init, 0k highmem) virtual kernel memory layout: fixmap : 0xffe17000 0xfffff000 (1952 kB) pkmap : 0xff800000 0xffc00000 (4096 kB) vmalloc : 0xe0800000 0xff7fe000 ( 495 MB) lowmem : 0xc0000000 0xe0000000 ( 512 MB) .init : 0xc04e7000 0xc0529000 ( 264 kB) .data : 0xc0413884 0xc04e0364 ( 818 kB) .text :
0xc0100000 0xc0413884 (3150 kB) Checking if this processor honours the WP bit even in supervisor mode... Ok. Calibrating delay using timer specific routine.. 4674.89 BogoMIPS (lpj=23374475) Mount-cache hash table entries: 512 CPU: After generic identify, caps: 0f80b9b9 00000000 00000000 00000000 00000001 00000000 00000000 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L3 cache: 4096K CPU: After all inits, caps: 0f80b9b9 00000000 00000000 00000140 00000001 00000000 00000000 ...
Listing 7 shows the output from another machine running Gentoo Linux, and in this example you can see some faults being reported by a running filesystem.
Listing 7. Disk error from dmesg

EXT3-fs: mounted filesystem with ordered data mode. sd 7:0:1:0: [sdf] Result: hostbyte=0x00 driverbyte=0x08 sd 7:0:1:0: [sdf] Sense Key : 0x3 [current] sd 7:0:1:0: [sdf]
ASC=0x4b ASCQ=0x0 end_request: I/O error, dev sdf, sector 894959703 EXT3-fs error (device sdf1): ext3_get_inode_loc: unable to read inode block inode=55935010, block=111869955 sd 7:0:1:0: [sdf] Result: hostbyte=0x00 driverbyte=0x08 sd 7:0:1:0: [sdf] Sense Key : 0x3 [current] sd 7:0:1:0: [sdf] ASC=0x4b ASCQ=0x0 end_request: I/O error, dev sdf, sector 894959703
From Listing 7, you can see that you probably need to check the filesystem, as there appears to be a fault on the filesystem or disk. In this instance, the information was also reported in the syslog (Listing 8).
Listing 8. Disk error in syslog
messages:Feb 3 12:17:53 bear sd 7:0:1:0: [sdf] Result: hostbyte=0x00 driverbyte=0x08 messages:Feb 3 12:17:53 bear sd 7:0:1:0: [sdf] Sense Key : 0x3 [current] messages:Feb 3 12:17:53 bear sd 7:0:1:0: [sdf] ASC=0x4b ASCQ=0x0 messages:Feb 3 12:17:53 bear end_request: I/O
error, dev sdf, sector 894959703 messages:Feb 3 12:17:53 bear EXT3fs error (device sdf1): ext3_get_inode_loc: unable to read inode block - inode=55935014, block=111869955
But in the case of a serious fault or failure, dmesg can sometimes be your only good source of information on what is happening on your system. Back to top User records (utmp/x, wtmp/x, lastlog) These files contain the user login and system data logs. Information in these files is written in the special utmp format and so you will need special tools to extract the information. The data held within these logs records login times and system startup/shutdown times, both for a historical record of the logins and for quick access to the last boot or login time used during login. See Resources for another article within the System Administration Toolkit that contains information on how to parse these files. The cron logs The cron time daemon, which is responsible for running many services in the background at periodic intervals, generates its own logs of information. On some systems, the cron log is recorded using syslog, but on Solaris and some traditional UNIX variants, the information is written to the file /var/cron/log. The information contained in the log includes the details of the command executed and when the job started and stopped. For an example of the log contents, see Listing 9.
Listing 9. The log of cron activity
! *** cron started *** pid = 283 Sun Feb 3 16:07:10 2008 > CMD: /usr/local/bin/logmanage >/dev/null 2>&1 > root 946 c Sun Feb 3 17:10:00 2008 < root 946 c Sun Feb 3 17:10:00 2008 > CMD: /usr/local/bin/backup >/dev/null 2>&1 > root 949 c Sun Feb 3 17:11:00 2008 < root 949 c Sun Feb 3 17:11:01 2008
Parsing the contents of the log can be an effective way to determine any problems with jobs that don't seem to execute properly. It can also be a good way to check the execution time of a job. Long-running jobs, or jobs that never seem to have finished, probably indicate a problem that should be investigated. Back to top Log file management You should make sure that you manage the logs on your systems. Log files can grow very large and in many cases you will want to keep an historical record of events on your machine for problems. For example, a phantom reboot or shutdown of a system should be investigated, and often the system logs are the only source of information. Although it cannot tell you everything that was taking place at the time the failure occurred, you may get information that helps, such as the precise time of the failure, or information about events that led up to the problem. Potential security problems and login attempts may indicate that your machine was being hacked and that may have led to or even been the cause of the problem. Keeping months and months of logs is probably not necessary (although it may under some circumstances be a legal requirement). On a busy system you can easily record 25MB or more information each day to the system logs, and logs are frequently the cause of insufficient disk space errors. Some UNIX/Linux variants include an automatic log management process (Solaris includes the /usr/sbin/logadm command), but it is not that difficult to create your own. A typical arrangement is to keep individual logs for a short period of time (for example, four weeks) and number them sequentially. For example, if you have the file messages,
last week's file is in messages.1, the two-week-old file is in messages.2, and so on. This makes migration of the files very easy. You must, however, be careful that you can successfully copy and recreate the file so that you do not lose any significant amount of information in the migration and archiving process. For the old files, to save space, you can also archive the content. Listing 10 shows a simple script that will copy and archive individual files into a suitably named directory within the original location.
Listing 10. Simple log-archiving facility

#!/bin/bash # Manage logs and archive them if necessary # Keeps 4 copies of logs cd /var/log for type in cyrus dmesg emerge.log faillog genkernel.log messages do mkdir -p $type.d cp $type.d/$type.3.bz2 $type.d/$type.4.bz2 cp $type.d/$type.2.bz2 $type.d/$type.3.bz2 cp $type.d/$type.1.bz2 $type.d/$type.2.bz2 cp $type $type.d/$type.1 && cat </dev/null >$type bzip2 -vf9 $type.d/$type.1 done
Running the script copies, recreates, and archives the log files. Note how the files are migrated; we just move the current to the week older in each case. Then finally we archive and recreate the original file.
Back to top Summary Log files can contain a whole wealth of information, but understanding the depth of the information and the format of the files helps immensely when you are trying to diagnose and resolve problems. This article has looked at the basics of log files, their location, and the details of the contents of those files and how they can help you diagnose problems and identify issues before they become problems. The article also examined the format of the different files and the relationships between different files and their contents.
Systems Administration Toolkit: Testing system validity

Summary: Examine methods of storing and later checking the validity of your configuration files. Despite all the security systems you have in place, it is still possible that somebody has accessed your system and changed your configuration or security settings. Tags for this article: aix, aix_and_unix, configuration, operating_system, security, system_administration, system_p, system_validity, systems_admin, toolkit... more tags
tools, tools_and_utilities, unix Tag this! Update My dW interests (Log in | What's this?) Skip to help for Update My dW interests
Date: 11 Sep 2007 Level: Intermediate Also available in: Chinese Russian Activity: 15075 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Verifying file content Establishing that the configuration files (and if necessary the application files) that make up your UNIX system are the ones you expect can go a long way to ensure your system is up and running without problems. It can also form a key part of your security processnot only by safeguarding that your system is valid, but also by making certain that the security configuration of your system is also secure. Although your systems should be secure enough that nobody can get in and make any modifications to your configuration files, you need to protect yourself, or at least have the ability to confirm that if someone got in you could verify whether the configuration files (or indeed any file) had been altered by an unauthorized individual. So how do you verify file content? Well, there are a number of different parameters that you need to consider. All of the following items have the potential to cause problems if they are changed in a working configuration:

File contents File owner Group owner File permissions Modification time Creation time
One simple way would be to keep a copy of the configuration files (and their associated parameters) on a different machine, and then run a regular comparison on the local and remote files. The problem with this method is that it requires a considerable amount of space and, more importantly, a considerable amount of time to record and compare this amount of information. This method can also have a detrimental effect on your ability to compare the information in a timely fashion.
Another often proposed technique is to simply record the file size, modification time, mode, and ownership information. Because the information is much shorter, it is much easier to store and quicker to check and verify compared to the full file contents. The problem is that the file size provides no real indication that the contents have changed. Consider this one line file:
location:/home/mcbrown
And the same file, with a line changed:

location:/home/slbrown
Both files would be 22 characters long, but the content is different, and such a relatively innocent change could have serious consequences, even though a check on the file size would return no difference. As to the other parameters, modification times, file mode, and other information can all be modified. You can alter modification times using the touch command. Even the file creation times can be faked by changing the time on the machine and recreating the file. One item that is far more difficult to change on a UNIX system is the file inode number. The inode number is a unique ID given to a file when the file is first created. The inode number is used by the underlying file system drivers to identify the file on the file system. The inode often changes when a file has been edited, because most editors create a new file and write the new contents to that file before removing the old file and renaming the new file to the original name. This makes inode comparison a good indicator of whether the file has been edited. Recording these pieces of information is still not enough; you need an efficient method of comparing the file content. Probably the best method of this is to use the file checksum. The file checksum Creating a checksum for a file is a classic way of comparing whether the content of the file is different without having to physically compare every byte of each file. The way a checksum works is to use an algorithm over the file contents. The algorithm generates an almost unique fingerprint of the file contents. You can accomplish this task in many different ways. For example, you can add the value of each byte together to use an algorithm that applies a complex calculation to the individual bits or bit groups of
a given file. The exact methods used are beyond the scope of this article and are dependent on which checksum tool you use. UNIX includes a simple checksum command, sum. This command is very basic, but it provides a reasonably unique number that you can use to identify the difference in most files. However, there are limits to this algorithm. Many modern solutions provide the md5 command. The latter produces a 128-bit fingerprint of a file and can theoretically generate a unique signature for any file of any size. The md5 algorithm for generating checksum information was originally developed to generate a unique fingerprint for a file before the file was encrypted so that the validity of the decrypted file could be guaranteed. Checksums generated by md5 can be represented either as a binary string, hex string, or base64 encoded string. The latter format is used in MIME email messages to ensure that different attachments in a file are uniquely identified. Creating checksums for files Because there are command-line solutions for creating checksum information, you can create a checksum of any file right on the command line. A good demonstration of how unique the checksum information can be is to use the file samples demonstrated earlier that had the same physical length and content that differed by only characters. You can obtain the checksum for both files in one command, as shown here in Listing 1.
Listing 1. Obtaining the checksum for both files in one command

$ sum old new 50093 1 old 62381 1 new
Even though only two characters are different in Listing 1, you still get significantly different checksum figures. Listing 2 shows the same files, this time checked with md5.
Listing 2. Checking the files with md5

$ md5 old new MD5 (old) = 602f604720d3b57925e99bcaa7d931a4 MD5 (new) = c3f06c217a0f26c16f8d030837d8718b
Here the checksums are significantly different and there should be no doubt that the files in question differ in some way.
Another solution for creating checksums is to use Perl to generate the checksum information. There is a module available for Perl, Digest::MD5, which can generate MD5 checksums from any string of data or from a supplied file. Listing 3 shows a simple script that returns the MD5 checksum for a file supplied on the command line as a hex string (identical to the format shown in Listing 2).
Listing 3. Script that returns the MD5 checksum

use Digest::MD5; use IO::File; my $chk = Digest::MD5->new(); foreach my $file (@ARGV) { $chk->addfile(IO::File->new($file)); print "$file -> ",$chk->hexdigest,"\n"; }
You can run the script on the same files as before, and you should get identical information, as shown here in Listing 4.
Listing 4. Running Digest::MD5 on the same files

$ simpmd5.pl old new old -> 602f604720d3b57925e99bcaa7d931a4 new -> c3f06c217a0f26c16f8d030837d8718b
To make this process useful, you need to record the information into a file so that you can compare the information again later. Before you do that, add the other information you want to compare (modification times, file sizes, ownership, inodes, and so forth) into the stored data. Adding other data to your report The Perl stat() function can obtain a whole heap of information from a given file, most of which you can use. The list of information that you can obtain from the file is shown here in Listing 5.
Listing 5. Perl stat() function
0 1 2 3 4 5 6 7 8 9 10 11 12
dev ino mode nlink uid gid rdev size atime mtime ctime blksize blocks
device number of filesystem inode number file mode (type and permissions) number of (hard) links to the file numeric user ID of file's owner numeric group ID of file's owner the device identifier (special files only) total size of file, in bytes last access time in seconds since the epoch last modify time in seconds since the epoch inode change time in seconds since the epoch (*) preferred block size for file system I/O actual number of blocks allocated
You can record nearly all of this information, but some of it is useless to use because it either changes too regularly, or it is not consistent across reboots. The following fields should probably be ignored:

rdevBecause this is unique to special files only (usually devices or pipes), it can probably be ignored. atimeThe last access time of the file changes each time a files is accessed. This means that the file is likely to change even though the file has never been modified in any way. Recording that information could lead to false positives in the difference report. blksizeThe block size used for file system I/O. Although this is unlikely to change, other factors than a file modification could lead to a change in this value, so recording it on a file-by-file basis is pointless. blocksThe number of blocks allocated for the file on the file system. This information is specific to a file, but if you are also recording the file size, then recording both is probably overkill.
These fields are useful to record for some specific reasons:
devThe device number of the file system should be consistent across reboots, providing you do not regularly mount and unmount the file systems. If the file systems are mounted in the same order on each reboot, then the device number should be consistent. nlinkThe number of hard links to a file can help to identify whether someone has created a hard link to the file in a location where they can overwrite the file and bypass the permissions of the original. You cannot have a hard link to a file with different ownership and permissions than the original. ctimeThe inode change time will be altered either based on when the file was created or when the ownership or mode information was altered. If this value has changed, then it might indicate that these values were altered, even if they were later returned to their normal values.
Listing 6 shows a script that writes out the file path, checksum, and other data to the standard output, separating each field of information with a colon. For the checksum, you not only checksum the file content, but you also add the other information into the
checksum data so that just by comparing the checksum alone you can determine if there was a difference.
Listing 6. Writing out the file path, checksum and other data to standard output
#!/usr/local/bin/perl use use use use Digest::MD5; IO::File; strict; File::Find ();
my $chksumfile = 'chksums.dat'; use vars qw/*name *dir *prune/; *name = *File::Find::name; *dir = *File::Find::dir; *prune = *File::Find::prune; File::Find::find({wanted => \&wanted}, $ARGV[0]); sub wanted { next unless (-f $name); my $fileinfo = genchksuminfo($name); printf ("%s\n",$fileinfo); } sub genchksuminfo { my ($file) = @_; my $chk = Digest::MD5->new(); my (@statinfo) = stat($file); $chk->add(@statinfo[0,1,2,3,4,5,7,9,10]); $chk->addfile(IO::File->new($file)); return sprintf("%s:%s:%s", $file,$chk->hexdigest, join(':',@statinfo[0,1,2,3,4,5,9,10])); }
The script uses the File::Find module in Perl, which traverses a directory and finds every file and directory from the base point. For each file, the wanted() function is called and, in that function for each file, the genchksuminfo() function is called. That gets the information with stat() and creates the file path, checksum, and other information as a line and returns it. In this script, that information is just printed out to the standard output.
The command accepts the directory to be scanned, so you can generate the checksum information. For the /etc, you would use the command shown in Listing 7.
Listing 7. Scanning /etc

$ perl savemd5.pl /etc /private/etc/6to4.conf:e6b1ba3e7683a0df9be21c9e9f5d1f6a:234881026:46788: 33188:1:0:0:1152674600:1155914028 /private/etc/afpovertcp.cfg:dc7c89b0626d6e603131902d387816f7:234881026:30152: 33188:1:0:0:1151780398:1166194017 /private/etc/aliases:de483c306c03f35dcbd45d609f8e68ce:234881026:47440: 33188:1:0:0:1151828538:1155914028 /private/etc/aliases.db:aa95ae673dcb6ba89684a6f4bbe3dba5:234881026:47437: 33188:1:0:0:1151828588:1155914028 /private/etc/authorization:39f7938ae1df629d422b27ec1a17f3dd:234881026:950752: 33188:1:0:0:1162503594:1162503594 /private/etc/auto.mnt:3da7579cdc03c529059a42de51c6679e:234881026:1013554: 33188:1:0:0:1162728759:1162728759 /private/etc/auto.mnt~:54d856aa344d03a6084d63c9dd7e1d9c:234881026:1013530: 33188:1:0:0:1162728576:1162728576 /private/etc/bashrc:fb23bdcacf23f69f1ce92e3b910c03b9:234881026:42880: 33188:1:0:0:1151805563:1155914028 /private/etc/compilers:363c62792a79df85cd0c8d71ff274495:234881026:821586: 33188:1:0:0:1159026690:1162503150 /private/etc/crontab:b9af1eb506bd68a43465789174bfe5e1:234881026:29678: 33188:1:0:0:1151800085:1166193736 ...
The final stage to the process is to store the information and provide a way of comparing the current situation with the stored one. Validating checksum information The final script builds on the script in Listing 6. The script expands significantly from the original script, incorporating a number of new features:
A command-line option that parses using the Getopt::Long module. This enables you to support specifying the chksumfile (storage of checksums and other information you calculate), whether or not you compare the new information with the old (by reading the contents of the chksumfile), and the ability to specify the base directory to be searched. If you compare the file, the data will be updated and only the differences will be reported. A loadchksumdata() function to load and parse an existing data file in a way that allows you to easily compare the new information with the old. A gendiff report() function that actually compares the individual fields of the stored information and the current information to tell you what has changed. This function is only called if you determine that there has been a difference of some kind.
Listing 8. Final script

#!/usr/local/bin/perl use use use use use Digest::MD5; IO::File; strict; File::Find (); Getopt::Long;
my $chksumfile = 'chksums.dat'; my $compare = 0; my $basedir = '/etc'; use vars qw/*name *dir *prune/; *name = *File::Find::name; *dir = *File::Find::dir; *prune = *File::Find::prune; GetOptions("chksumfile=s" => \$chksumfile, "compare" => \$compare, "basedir=s" => \$basedir); my $chksumdata = {}; if ($compare) { loadchksumdata($chksumfile); } my $outfile = ''; if (!$compare) { $outfile = IO::File->new($chksumfile,"w"); } File::Find::find({wanted => \&wanted}, $basedir); if ($compare) { foreach my $file (keys %{$chksumdata}) { print STDERR "Couldn't find $file, but have the info on record\n"; } } sub loadchksumdata { my ($file) = @_; open(DATA,$file) or die "Cannot open check sum file $file: $!\n"; while(<DATA>)
{ chomp; my ($filename,$rest) = split(/:/,$_,2); $chksumdata->{$filename} = $_; } close(DATA); } sub wanted { next unless (-f $name); my $fileinfo = genchksuminfo($name); if ($compare) { if (exists($chksumdata->{$name})) { if ($chksumdata->{$name} ne $fileinfo) { print STDERR "Warning: $name differs from that on record\n"; gendiffreport($chksumdata->{$name}, $fileinfo); } delete($chksumdata->{$name}); } else { print STDERR "Warning: Couldn't find $name in existing records\n"; } } else { printf $outfile ("%s\n",$fileinfo); } } sub gendiffreport { my ($orig,$curr) = @_; my @fields = qw/filename chksum device inode mode nlink uid gid size mtime ctime/; my @origfields = split(/:/,$orig); my @currfields = split(/:/,$curr); for(my $i=0;$i<scalar @origfields;$i++) { if ($origfields[$i] ne $currfields[$i]) { print STDERR "\t$fields[$i] differ; was $origfields[$i], now $currfields[$i]\n"; } } }
sub genchksuminfo { my ($file) = @_; my $chk = Digest::MD5->new(); my (@statinfo) = stat($file); $chk->add(@statinfo[0,1,2,3,4,5,7,9,10]); $chk->addfile(IO::File->new($file)); return sprintf("%s:%s:%s", $file,$chk->hexdigest, join(':',@statinfo[0,1,2,3,4,5,9,10])); }
To use the script, you first need to generate a file with the base checksum and other data that will act as your base comparison file. For example, to create a checksum data file for the /etc directory, you might use the following command line:
$ genmd5.pl --basedir=/etc --chksumfile=etc-chksum.dat
Now that you have the information, if you edit a file and then re-run the script, you should get a report of the differences. In Listing 9, you can see the results when the /etc/hosts file is edited.
Listing 9. Results when the /etc/hosts file is edited

$ genmd5.pl --basedir /private/etc --compare Warning: /private/etc/hosts differs from that on record chksum differ; was d4a23fcdaa835d98ede1875503273ce6, now beb50782b3fd998f35786b1e6f503d1b inode differ; was 4879566, now 4879581 size differ; was 1186929905, now 1186930065 mtime differ; was 1186929905, now 1186930065 Couldn't find /private/etc/hosts~, but have the info on record
Note that you report both the differences in an individual file and the fact that a file has been deleted. If a new file had been created, the difference would have been reported as well. Using the checksum data Using the script in Listing 6 generates a file that you can use to test and verify the validity of a system. Of course, the very fact that the file exists means you have to store
that information securely, otherwise anybody could update the information, including any unauthorized individual that happens to use your machine and alter the files you want to secure. There is no hard and fast rule for this information, but it should be clear that storing the file you create on the same machine that you generated it on is probably a bad idea once it has been located, it could be altered. The same is true of storing the file on another machine on the same network. Once the file has been located, it could be subverted and altered. The best solution is probably to write the file to a CD or DVD that can be kept completely separate from the machine. The problem with this solution is that you must keep the information up to date. Each time you legitimately update or alter the files that you are monitoring, you must update the checksum file. Although this makes the process somewhat more difficult, the benefits of the security information that the file provides can be incalculable. Summary In this article, you developed a script that you can use to generate information that checks the validity of a file or directory full of files. The recorded information includes the file path, a checksum of the file so that you can compare the file contents, and unique information about the file (inode, permissions, ownership information) so that you can identify differences should they occur. How you make use of this script is up to you. You could store the information and run the script regularly to identify problems as soon as they happened, or you could use the file as a post-mortem tool in the event of a problem to find out which files had been changed so that you have a list of files to be examined.
System Administration Toolkit: Swap space management and tricks

Martin Brown (mc@mcslp.com), Freelance Writer, Author
Summary: Configure your swap space (including adding space in an emergency) to get the most out of your system. In this article, you'll learn how to monitor your system to determine an effective swap space figure as well as examine methods for using swap space for more than just secondary random access memory (RAM).
View more content in this series Tags for this article: de, espacio, info, space, swap, tuning, verificacion
Date: 31 Oct 2006 Level: Intermediate Also available in: Chinese Activity: 62032 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Back to top How UNIX uses swap space All UNIX operating systems support a system of virtual memory (VM). With a VM system, you have two primary areas that store information about the active applications and the data it uses -- the physical random access memory (RAM) and space allocated on hard drives. The amount of applications and data that can be loaded into memory can be larger than the physical RAM in the machine with VM, because applications and information that are not actively being used are swapped to disk (hence the term swap space). The benefit of a VM system is that physical RAM is more effectively used -- this happens because it only holds executing applications and the data being used. Because RAM is faster than hard disk space, you get the performance you need from
the actively loaded applications while also benefiting from the ability to run more applications than would normally be supported. The result is that you gain a much larger addressable memory space (4GB on 32-bit systems and significantly larger on 64-bit systems), which makes application development easier while still providing high performance. Back to top Setting swap space values The amount of swap space that you configure on your machine is largely dependent on the application and operating system. You might be unable to run all of the applications that you want with too little swap space, and you might be wasting disk space that is never used with too much swap space. You might also be indirectly supporting the ability to overload your system, since too much swap space compared to too little RAM leads to bad performance, as the system spends more time swapping pages in and out than it spends actually processing. Different UNIX variants use their swap space in different ways. Some only use swap space when the physical memory has already been used. Others use swap space for all inactive applications and unused pages. Some keep a copy of the data in all applications -- both memory and swap space. By default, some avoid using swap space and, where possible, provide a mechanism to let you tune how swap space is used. Many use a combination of these different arrangements to make the best of their swap space. For example, within Linux, you can adjust the swappiness of the system when using the 2.6 kernel by setting the vm.swappiness value in /etc/sysctl.conf. Higher values lead to more pages being swapped and lower values lead to more applications being kept in memory, even if they are idle. Beyond these basics, there are a number of different points of view and rules on how you should configure VM, and how much disk space you should allocate. Some of these approaches have been summarized below:
Some administrators believe you should never have less swap space than physical RAM. This was a requirement with older versions of UNIX (particularly SunOS) because of the way in which VM was used. In theory, having more VM than physical RAM enables the operating system to swap out all current processes to disk when the system becomes idle, and it can speed up performance for systems because new processes can be loaded directly into RAM without having to swap applications beforehand. For development systems, many administrators agree that you should have as much VM as you have RAM. The reason for this approach is that swap space is also used during a system failure as the location where the active memory is dumped; the stored memory dump is then reloaded during boot and saved as a kernel core file. If the configured swap space is less than the physical RAM, such dumps are not possible.
In high-performance deployment environments, you can configure as little VM as possible, on the basis that you want to keep the applications active in memory and don't need a lot of swap. Crash recovery is not required, and the ability to identify a potential lack of physical RAM (through running out of swap space) provides a warning that your application environment is not optimized, or that you need to upgrade your RAM. For desktop environments, having a very large VM allocation can be to your advantage, as it enables you to run a large number of applications (many of which will probably be idle and easily swapped) making more RAM available for the active applications.
Choosing the right amount of swap space is highly dependent on the platform you are configuring -- it's intended use and how you want to cope with a lack of available VM. For some conservative values:

On a desktop system, allocate at least as much swap space as physical RAM. On a server, allocate at least 50 percent of your available RAM as swap space. Once you have a figure in mind, you need to configure the swap space.
Back to top Configuring swap space All UNIX systems have multiple types of swap space. The primary swap space is that of which is normally configured during boot time and usually set up as part of the installation of your operating system. All UNIX systems usually have one allocation of primary swap space. How the swap space is created and configured differs from system to system. Under Linux, a disk partition used for swap is identified through the partition ID, and the partition must be initialized by the mkswap command:
$ mkswap /dev/sda2
On most traditional UNIX environments, you can use any available partition without any preparation. On HP-UX, AIX, and Linux, you use the swapon command to add a partition to the available swap space:
$ swapon /dev/sda2
On Solaris, you use the swap command to add the swap to the system:
$ swap -a /dev/dsk/c0t0d0s2
Note that swap space added in this way will not be retained during a reboot. To add the swap space permanently to your system, you must configure one of the boot configuration files to enable the swap allocation automatically. Back to top Adding boot time swap space Under most UNIX variants, swap is added automatically at boot time as part of the kernel initialization. Most use lines in the file system mount file (/etc/fstab, /etc/vfstab or /etc/filesystems) to specify the partitions to be used as swap space. You can see an example of this within the /etc/vfstab file from Solaris (see Listing 1).
Listing 1. Specifying the partitions to be used as swap space in Solaris

#device mount #to mount options # fd /proc /dev/dsk/c0d0s1 /dev/dsk/c0d0s0 /dev/dsk/c0d0s7 /devices ctfs objfs swap device to fsck /dev/rdsk/c0d0s0 /dev/rdsk/c0d0s7 mount point /dev/fd /proc / /export/home /devices /system/contract /system/object /tmp FS type fd proc swap ufs ufs devfs ctfs objfs tmpfs fsck mount pass at boot 1 2 no no no no yes no no no yes -
Listing 2 shows an example in Linux for the /etc/fstab file.
Listing 2. Specifying the partitions to be used as swap space in Linux

# <fs> <dump/pass> /dev/sda1 /dev/sda3 /dev/sda2 /dev/hda1 /dev/hdb1 /dev/hde1 /dev/hde2 /dev/hde3 /dev/hde4 <mountpoint> <type> <opts> noauto,noatime noatime sw noatime noatime noatime noatime ext3 noatime noatime noauto,ro 1 0 0 1 1 1 1 1 1 1 0 0 2 2 2 2 2 2
/boot ext2 / ext3 none swap /var/lib/mysql ext3 /export/data ext3 /export/home ext3 /var/tmp ext3 /usr/portage/distfiles /tmp ext3 iso9660
/dev/cdroms/cdrom0 /mnt/cdrom
0 0
none
/proc
proc
defaults
0 0
Finally, you use the /etc/swapspaces file to configure swap space partitions in AIX (see Listing 3).
Listing 3. Using the /etc/swappspaces file to configure swap space partitions

* This file is modified by the chps, mkps and rmps commands and referenced * by the lsps and swapon commands. hd6: dev = /dev/hd6 paging00: dev = /dev/paging00 paging01: dev = /dev/paging01
Rather than editing this file directly, you should use the corresponding commands or the Smitty tool to modify the file contents. To delete additional swap, you can use the corresponding swapoff or swap -d command to disable the swap partition. However, note that you can only disable swap that is not currently in use, and it is not possible to disable the primary swap partition. Back to top Adding temporary swap capacity There are times when you can run out of configured swap space partitions and, for a short period, require more VM until you can organize a more permanent solution. If you have a partition available, then you should prepare it as outlined above in the Configuring swap space section. If you don't have a spare partition available, then you can create one or more files, which you can temporarily add into the available swap pool. Most systems even let you use swap files on NFS mounted filesystems, which can be used when you have no local disk space available. To use a file for swap space, first create a suitable file. The mkfile> command is most appropriate here, as it provides a simple way of creating large files. You specify the filename and filesize on the command line:
$ mkfile 512m /export/data/swapfile
You can then add the file as you would a partition:

$ swap -a /export/data/swapfile or $ swapon /export/data/swapfile
Swap files used in this way are never generally a long-term solution to the problem for swap space. You should configure a dedicated partition for swap space as soon as possible. Back to top Monitoring swap space usage There are two aspects to monitoring swap space and how it is used: current usage and active usage. You can find an array of different tools across the UNIX variants that report available free memory and VM usage with current usage. Under Solaris, the swap tool provides most of the swap interface functionality, including reporting usage and available devices. For example, to get a list of configured swap devices, see Listing 4.
Listing 4. Monitoring swap space usage under Solaris

$ swap -l swapfile /dev/dsk/c0d0s1
dev swaplo blocks free 102,1 8 1220928 1220928
In Listing 4, you can see that you have one swap device that is completely unused. Under AIX, you use the lsps tool to get the same information (see Listing 5).
Listing 5. Using lsps under AIX to monitor swap space

$ lsps -a Page Space hd6 Physical Volume hdisk0 Volume Group rootvg Size %Used Active Auto 2048MB 3 yes yes
Under Linux, you use the swapon tool, as shown in Listing 6.
Listing 6. Using swapon tool under Linux

$ swapon -s Filename /dev/sda2
Type partition
Size Used 1004052 319744
Priority -1
With HP-UX, you use swapinfo, as shown in Listing 7.
Listing 7. Using swapinfo to monitor swap space

swapinfo -t Kb Kb Kb PCT START/ Kb TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME dev 2097152 0 2097152 0% 0 1 /dev/vg00/lvol2 reserve - 1468224 -1468224 memory 3299740 1220268 2079472 37% total 5396892 2688492 2708400 50% 0 -
In each of the above examples, you can see the list of available swap space locations, their use, and get a reasonable idea of how much (or how little) the swap is being used. These snapshot views are useful, but they only show your current usage. There is little information here on how regularly or extensively your swap space is being used. For performance reasons or when diagnosing a problem that you suspect might be swap space related, you often need a more interactive view of the memory and paging usage. Back to top Active VM monitoring The vmstat command provides a long-term view of virtual usage, including providing information on the amount of memory being actively swapped between swap space and RAM. On the command line, you need to specify the interval (in seconds) to use when reporting statistics. The resulting output shows a wealth of information, and it can be used to make decisions about your VM configuration and allocation. Listing 8 shows a sample of the output for a Solaris machine (an interval of five seconds).
Listing 8. Sample output for a Solaris machine

kthr memory cpu r b w swap free us sy id 0 0 0 945932 385632 0 36 63 0 0 0 944616 384068 0 1 99 0 0 0 944612 384064 0 1 99 0 0 0 944612 384064 0 1 99 0 0 0 944612 384064 0 1 99 0 0 0 944612 384064 0 1 99 0 0 0 944500 383988 1 5 94 0 0 0 942060 382444 0 2 98 0 0 0 942048 382432 0 2 98 0 0 0 946188 388188 6 18 76 0 0 0 944488 386308 8 29 63 0 0 0 943128 385104 8 32 60 0 0 0 937692 377628 0 2 97 0 0 0 937228 377208 0 1 99 0 0 0 937220 377200 1 6 93 0 0 0 937348 377320 1 7 93 0 0 0 937428 377408 1 5 94 0 0 0 937436 377404 1 6 93 page re 9 0 0 0 0 0 disk faults in 413 654 596 694 658 576 sy 172 21 21 12 11 18 cs 87 51 52 48 50 52 120 146 188
mf pi po fr de sr cd cd cd cd 36 13 8 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 100 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 5 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
115 850 1 6 62 0 0
612 1656 669 699 256 414
9 105
481 2400 10 10 10 0 0 769 4753 1 8 8 0 0
595 12613 6082 625 15705 6285 579 16077 6304 654 647 403 38 131 63 218 228 180 295
839 5194 176 9 9 0 32 209 0 0 2 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
0 14 0 0 0 0 0 2 0 0 3 0
121 980 0
623 1246 667 1266 702 890
126 1044 2 1 57 686 72 861 0 0 0 0
0 54
689 1113
There are a number of columns here -- some more important than others. The key ones are:

swap -- shows the total of swap available free -- shows the amount of free swap space pi -- kilobytes paged in po -- kilobytes paged out fr -- kilobytes freed
A very low amount of free space and a high number for pages in and out probably indicates a lack of VM and, with very high pages in and out, probably a lack of physical RAM. Back to top Using swap space for temporary storage Under some UNIX variants, notably Solaris and Linux, you can use the tmpfs filesystem; this uses VM as storage space. When you have ample RAM, this provides an excellent way of supporting very fast disk access; unused files will be swapped on to disk. The result is a highly efficient solution for fast access to the files you use most often, and it can be a significant speed boost for applications, such as software development, where a large number of temporary files are created during the compilation process. Back to top Summary Getting your swap space configuration right can make a big difference to the efficiency of your machine. Most people set and forget their swap space settings but, in this article, you've examined several ways to add and adjust your swap space. You've also seen how you can permanently add swap to your configuration and how to add swap in an emergency when you run out of memory.
Systems Administration Toolkit: Understanding DNS

Summary: The Domain Name System (DNS) is the service that converts hostnames and domain details into the IP addresses required for application to communicate. Under UNIX, the primary DNS service is based on BIND, and DNS itself is a key part of most UNIX installations. This article looks at the basics of DNS setup, how servers and requests are distributed, and exchanged and how to set up and keep a DNS environment running smoothly. Tags for this article: aix, aix_and_unix, dns, nis, systems_admin, toolkit
Date: 04 Mar 2008 Level: Intermediate PDF: A4 and Letter (66KB)Get Adobe Reader Also available in: Chinese Activity: 19751 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Back to top DNS basics As a general rule, humans are not particularly good at remembering numbers. Unfortunately, the IP addressing system uses numbers to identify individual hosts. Remembering numbers for all of the machines that you might want to use both inside and outside of your network is obviously impractical. Instead, it's easier to remember the name of your server as 'bear' or the address of a Web site on the Internet as www.mcslp.com. Historically, UNIX has used three main locations for this type of information. The built-in /etc/hosts file is the main source and a useful backup, but keeping it up to date across multiple machines requires a lot of copying of data. A distributed system, where individual machines and services can contact a central point, is the better method and two systems, the Network Information System (NIS) and DNS, were developed with different goals in mind. NIS is a general-purpose database-sharing service and can be used to share host, password, and other normally text-based databases. Unfortunately, NIS is not as effective across systems, and certainly not in the distributed format of the Internet. The Domain Name System (DNS) resolves that by providing a mechanism so that names can be resolved to the appropriate IP address. Addresses within the domain
name system are organized in a tree structure. Although you can organize any private DNS in the format that you want, it is easiest to understand the DNS system by looking at the structure used on the Internet. Looking at a typical DNS address, www.mcslp.com, the name can be divided by splitting up the contents by a period, starting with the fragment on the far right (com). From our example, www.mcslp.com, the 'com' is example of a top-level domain (TLD) name that organizes the names by organizational or regional structure. For example, valid TLDs included com (commercial), edu (educational) and net (network related). On the Internet, regional organization provides for country-specific TLDs, such uk for the United Kingdom or fr for France. Each fragment to the left of the TLD is a subdivision of the original domain. For example, mcslp.com is a subdivision of the com TLD. Further subdivisions can occur, up to a maximum of 127 levels. Each subdivision is known as a subdomain. From the example, mcslp.com is a subdomain of the com TLD, and www.mcslp.com is a subdomain of the mcslp.com subdomain. The use of subdomains has two main purposes; one is for ease of identification and the other is for the purposes of delegation. From an identification perspective, it is much easier to understand and identify that www.mcslp.com and bear.mcslp.com are both hosts within the mcslp.com domain. Similarly, it is easy to identify that www.google.com and mail.google.com are both hosts within the google.com domain. The delegation relates to the way in which administrators (and computers) are responsible for the configuration and contents of the various domains. The mclsp.com domain is a subdomain of the com TLD. The administrators of the com TLD have given permission for the domain and subdomains of the mcslp.com domain to be the responsibility of another administrator. This delegation is what enables administrators of services on the Internet to 'own' a domain and in turn assign their own IP addresses, configure their own names, and organize their own structure to help them identify the machines within their network. Internally, the domain name system can also be used as a method for identifying and locating machines on your network, and you can use the organization and delegation features of DNS to help organize your network. For example, mcslp.com is the public domain for a company, while mcslp.pri is the private domain used internally to identify machines within the network. The address www.mcslp.pri relates to an internal Web server, and mail.mcslp.pri relates to an internal mail server. In addition, a subdomain, vm.mcslp.pri, is used to provide a logical grouping of hosts (in this case, virtual machines). Back to top
How domain name resolution works Domain name resolution works by a client contacting a server and requesting that a particular address be resolved to the corresponding part (i.e., name to IP address, or IP address to name). Because name servers can hold different types of information, the exact response and process depends on the query, but for the examples we will assume a simple name lookup that is expecting an IP address in response. In general, the client asks the server to resolve an address and receives the resolved address back from the server. Queries can be of two types: recursive and nonrecursive. In a recursive query, the client expects the server to respond the full answer (i.e., to resolve a name to the IP address). Non-recursive queries allow the server to return a partial response, such as returning the name or address of the server that may know the answer (or may have more information). The reason for this is that the DNS systems work on a system with an analogy similar to Chinese whispers. If the server doesn't know the answer itself (because it is not responsible for the domain), then it will find out which server is responsible for the domain and ask that server instead. In a recursive query, the server does the asking and returns the final result. Non-recursive queries are generally used by servers when talking to each other to resolve a query on behalf of a client. You can see this more clearly by looking at Figure 1. The numbers in the diagram relate to the same steps within the list below.
1. The client checks its own cache to see if it has resolved the name recently but doesn't find anything. 2. The client sends a recursive query to its primary name server. 3. The name server checks its database and cache, doesn't find anything, and sends an iterative request to a root server for the .com top-level domain.
Figure 1. Recursive and non-recursive queries
4. The root server responds to the DNS server with the IP address of the DNS server that is authoritative for the mcslp.com domain. 5. The DNS server sends an iterative request to the DNS server for the mcslp.com domain to resolve the address www.mcslp.com. 6. It uses an iterative request because it already knows that the mcslp.com DNS server is authoritative for the domain. 7. The mcslp.com DNS server returns the IP address to the DNS server. 8. The DNS server returns the IP address to the client.
Caching of DNS requests

Because it's likely that the same hostname can be requested multiple times within a short period of time (for example, when loading all of the content of a typical Web page), DNS information is cached, both locally by the client and on the server. Caching on the client occurs to prevent making multiple requests when the information is already known. Caching on DNS servers occurs within a controllable limit called the Time To Live (TTL). This sets the period that a DNS entry will be retained in the cache before a query to resolve the entry is requested again from a server with responsibility for the domain being queries. The downside of the TTL and caching is that different users may see different information, based on which DNS server they requested a name resolution from, and when the original request took place.
Setting up a DNS server Earlier in this article, I discussed the role of delegation and the subdivision of domains and subdomains to allow individual administrators to be responsible for their own domains and the information and hosts defined within that domain. Setting up your own domain server can be useful both if you are hosting a public DNS domain available on the Internet, and if you want to configure your own DNS information to resolve the addresses and other information within your private network.
To set up a DNS server within a UNIX server or client, you need a DNS daemon, which answer queries about one or more domains. The most common daemon is BIND, and most implementations of a DNS server use BIND or a derivative of the BIND code as the basis for their DNS service. The core of the BIND service is a daemon called named (or in.named), which handles all of the queries. The configuration of a typical BIND environment involves the main named configuration file, usually /etc/named.conf (or /etc/bind/named.conf) and a number of 'zone' files, which hold the individual information about each DNS domain that the server is responsible for. The main configuration file sets up the directories used for the zone files, security information, and also the additional servers that should be queried if the DNS server is unable to resolve the address itself. You need this setting if you want your server to resolve addresses outside of your network (for example, on the Internet), in addition to those inside your network for the configured zones. You can see the header portion of a simple configuration file in Listing 1.
Listing 1. Simple configuration file

options { directory "/var/bind"; forwarders { 212.23.3.100; 212.23.6.100; }; listen-on-v6 { none; }; listen-on { 127.0.0.1; }; allow-query { 192.168.0.*; 192.168.1.*; 192.168.2.*; }; pid-file "/var/run/named/named.pid"; };
Note that within the configuration file you need to use the double-slash (//) to introduce a comment. You should also be aware that the semicolon is used to split up individual configuration options of the file, and that braces are used to group configuration options
together. Most problems with the content of the configuration file are introduced when a semicolon or brace is missing. The individual configurable options show in this example are as follows:

directory -- This sets the base directory where the zone files can be found. forwarders -- This sets the list of DNS hosts that will be queried when a recursive query is sent to the DNS server that the server is unable to answer. Usually you should set this to the next highest DNS server in the hierarchy. For example, if the server was configured for the subdomain servers.mcslp.pri, then you would set the forwarding to point to the DNS servers of mcslp.pri for resolution. For most situations, you should set the forwarding addresses to those of your ISP to allow Internet addresses to be resolved. listen-on-v6 -- Sets whether the server should listen on an IPv6 address. The 'none' setting disables IPv6 support. listen-on -- Sets the IP addresses on which the server should listen for resolution requests. In the example, the server will only listen to requests from localhost (i.e., the machine itself). If you omit this configuration line, the server will listen on all configured IP interfaces. You can use this to limit requests on certain interfaces if you are using the server as a router where you only want certain interfaces to support DNS. allow-query -- This configures the hosts that are allowed to request DNS resolution. In a network that may span multiple IP subnets, you may want to limit which servers are responsible for handling queries to a specific subnet to allow for the load to be spread among multiple servers. pid-file -- Sets the location of the file to hold the process ID of the running named daemon. This is useful if you want to be able to kill or reload the zone information because you can locate the PID of the server using this file.
The remainder of the configuration file contains the zone file data that will be used by your server to share DNS information. Back to top Setting up the DNS data DNS information is stored within a series of 'zone' files. Typically you have one zone file for each domain that you host.
Forward zones -- These primarily map domain names to IP addresses. In addition, other domainlevel information, such as valid nameservers, mail server priorities for the domain, and so on is stored in these files. Normally you would use a single zone file for a forward domain. For example, mcslp.com or an internal mcslp.pri zone. Reverse zones -- These map IP addresses back to names. For security reasons, many services, including NFS, OpenSSH, and Web servers can be configured to check whether the IP address of a request resolves back to a name, and, where relevant, that the request host and IP/address match both forwards and backwards. For reverse zones, you only create records recording the relationship between the IP address and the host.
Root zones -- These provide the list of 'root' servers that hold information about subdomains on the Internet and their delegation records. By providing a root zone file, you can let your DNS server query the root Internet servers for information about other hosts on the internet.
All configured zones within your configuration are designated either as master or secondary. Master zones are those for which the server is a master host (i.e., the delegated server for the domain). Only one server can be the master for a domain; additional servers that can be authoritative for a domain (that is, they provide qualified responses) are classed as 'secondary,' or slave, servers. You can use the slave servers to help spread the load and provide resilience in the event of a failure. We'll return to this issue later in this article. For each zone file that you host, you must add a zone portion to the named.conf file (see Listing 2).
Listing 2. Adding a zone portion to the named.conf file

zone "mcslp.pri" in { file "pri/mcslp.pri.zone"; type master; };
This configures the zone mcslp.pri (an internal subdomain used to identify computers only on the local network), the file in which the zone data is located (relative to the directory option), and the type of the file. In this case, the actual file will be located in /var/bind/pri/mcslp.pri.zone. An example of the contents of the file, in this case a 'forward' zone, primarily mapping names to IP addresses, are shown in Listing 3.
Listing 3. Zone file

$TTL 86400 mcslp.pri. IN SOA bear.mcslp.pri. admin.mcslp.pri. ( 2008021401; serial 3h ;
refresh 1h retry ;
1w ; expiry 1h ) ; minimum mcslp.pri. IN NS bear.mcslp.pri. mcslp.pri. IN A 192.168.0.2 bear.mcslp.pri. IN A 192.168.0.2 airong.wireless IN A 192.168.0.210 ... mail IN CNAME bear www IN CNAME bear mcslp.pri. IN MX 10 bear
The header block of the zone file specifies the information about the domain. In this case, we specify the domain name and the SOA (Start of authority) information. The SOA configures the data about the domain to define which servers are classed as authoritative (that is, then give qualified responses to queries for this domain). The remainder of the SOA record configures a serial number (used by other DNS servers to determine whether their copy of the domain data is up to date), and the refresh and retry intervals (in seconds) for updating the information. From our example, there is only one server qualified to give responses about hosts and other data within the domain, bear.mcslp.pri. The hostname, rather than IP address, is provided so that you can change the host IP address without having to update the DNS SOA record. The remainder of the information within the zone file then configures the specific DNS records. DNS is capable of holding much more information than simple name to address data, and therefore the DNS records are organized into different types. A list of the main types includes:
A (address) -- Specifies the IP address for a given name. NS (nameserver) -- Specifies the nameserver for a given domain. CNAME (canonical name) -- An alias to an existing host. MX (mail exchanger) -- The hostname of a server responsible for handling e-mail for the domain.
For names within the domain, you can specify a name that is implied to be within the domain simply by specifying the name, for example: bear IN A 192.168.0.2. Because the DNS knows that this file contains information for the mcslp.pri domain, the full hostname is expanded to bear.mcslp.pri. You can also specify the fully qualified domain name (FQDN) by setting the full hostname and appending a period to the end of the name: amazon IN CNAME
www.amazon.com
The above will configure the amazon.mcslp.pri hostname to act as an alias to www.amazon.com. Returning to the original list of hosts, the hosts are configured as follows in Listing 4.
Listing 4. Configuring the hosts

bear.mcslp.pri. A 192.168.0.2 IN
Defines the IP address of bear.mcslp.pri as 192.168.0.2 airong.wireless IN A 192.168.0.210 Defines the IP address of airong.wireless.mcslp.pri to 192.168.0.210. This is an example of setting up a subdomain within our original mcslp.pri domain for organizational purposes. In this case, the wireless.mcslp.pri is used to hold the details of wireless devices. mail IN www CNAME CNAME bear bear IN
Creates aliases to bear.mcslp.pri. These are examples of utility aliases that can be used to identify different services. By using an alias, you can change the host ultimately responsible for these services without having to reconfigure clients with the updated host. mcslp.pri. MX 10 IN bear
The mail exchanger records are used by mail transport agents like sendmail and postfix when looking up where to send e-mail. In the above example, bear.mcslp.pri is configured to accept any e-mail where the domain within the e-mail address is mcslp.pri. For 'reverse' lookups, that is, when you want to resolve an IP address to a name, you should set up an additional zone. Reverse zones have a specific domain name that identifies them as reverse zones. You can see an example of this in Listing 5.
Listing 5. Reverse zones

$TTL 86400 0.168.192.inaddr.arpa. IN SOA bear.mcslp.pri. admin.mcslp.pri. ( 2007111900 ; serial 3h ; refresh 1h ; retry 1w ; expiry 1h )
; minimum 0.168.192.inaddr.arpa. IN NS bear.mcslp.pri. 2.0.168.192.inaddr.arpa. IN PTR bear.mcslp.pri.
The domain is the portions of your IP address class, in reverse order, and then placed within the in-addr.arpa domain. For example, IP addresses in the class C address 192.168.0.x are defined within the 0.168.192.in-addr.arpa domain, as shown in Listing 4. The header and NS (name server) records are as before. IP address to hostname mappings are defined use the PTR (pointer) record. You should have one of these records for each corresponding name to IP address entry within your domain. To add your reverse domain mapping to your named config, specify the domain and domain file in the same way within your named.conf file (see Listing 6).
Listing 6. Adding reverse domain mapping

zone "0.168.192.inaddr.arpa" IN { file "pri/db.192.168.0"; type master; };
You should also configure the 'hint' zone, which refers to the root servers and a localhost forward and reverse domain so that resolution of localhost and 127.0.0.1 (the localhost IP address) operate correctly. You can find examples of these in the bind source, and usually also provided with your bind installation. Listing 7 shows the named.conf zone entries.
Listing 7. named.conf zone entries

zone "." IN { type hint; file "named.ca";
}; zone "localhost" IN { type master; file "pri/localhost.zone"; allow-update { none; }; notify no; }; zone "127.inaddr.arpa" IN { type master; file "pri/127.zone"; allow-update { none; }; notify no; };
Once the zones are configured and you have added the zone file, you can start up the named daemon. For most UNIX/Linux variants, you can do this using the script in /etc/init.d: # /etc/init.d/named start. For Solaris 10, use svcadm: # svcadm enable bind. Once the named daemon is running, make sure you edit your /etc/resolv.conf file (see Listing 8) to configure the domain and DNS servers that provide responses:
Listing 8. Edit your /etc/resolv.conf file to configure the domain and DNS servers
domain mcslp.pri nameserver 192.168.0.2
You may also need to edit your /etc/nsswitch.conf file so that hostname lookups are redirected to the DNS service. For example, to use the local files first and then resort to DNS: hosts: files dns. You should now be able to look up hosts from the DNS using dig, a tool that queries DNS servers for information. Listing 9 shows the result when looking up bear.mcslp.pri.
Listing 9. Result when looking up bear.mcslop.pri
# dig bear.mcslp.pri ; <<>> DiG 9.3.4-P1 <<>> bear.mcslp.pri ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 858 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;bear.mcslp.pri. IN A ;; ANSWER SECTION: bear.mcslp.pri. 86400 IN A 192.168.0.2 ;; AUTHORITY SECTION: mcslp.pri. 86400 IN NS bear.mcslp.pri. ;; Query time: 1 msec ;; SERVER: 192.168.0.2#53(192.168.0.2) ;; WHEN: Wed Feb 20 18:08:34 2008 ;; MSG SIZE rcvd: 62
The output shows the server that was queried, and the full DNS record returned. Back to top Spreading the load DNS is a distributed service, in that more than one server is capable of responding to a given query, providing it knows who to ask for the information. In addition to forwarding DNS requests and the recursive nature, you can also configure more than one server to be 'authoritative' for a particular domain. Secondary or slave servers are delegated through the domains NS records as being qualified to give responses. Secondary DNS servers transfer their zone information directly from the primary DNS server and keep a copy locally. Clients can then be configured to use one or more of the DNS servers responsible for a domain. This can help to reduce the load on requests
and also ensures that if the primary server (or another slave) is down when a query is made, another server can provide an answer to the client resolution query. To add more DNS servers to your configuration, first edit your zone files and add NS records for each slave that will also be hosting the domain. This ensures that your slaves will return authoritative answers to queries for that domain. Then, add an entry to your named.conf file that specifies the domain name with a type of 'slave' and then providing a list of qualified 'masters' that can transfer a copy of the domain to the slave. For an example, setting up a slave for the mcslp.pri domain (see Listing 10).
Listing 10. Setting up a slave for the mcslp.pri domain

zone "mcslp.pri" IN { type slave; file "sec/mcslp.pri"; masters { 192.168.0.2; }; };
Now reload or start the named service on your slave server and check the logs. You should see entries reporting the successful transfer of the domain to your slave (see Listing 11).
Listing 11. Reloading the named service on your slave server

Feb 20 18:17:41 tweedledum named[27142]: running Feb 20 18:17:41 tweedledum named[27142]: zone mcslp.pri/IN: Transfer started. Feb 20 18:17:41 tweedledum named[27142]: transfer of 'mcslp.pri/IN' from
192.168.0.2#53: connected using 192.168.0.6#53905 Feb 20 18:17:41 tweedledum named[27142]: zone mcslp.pri/IN: transferred serial 2008022000 Feb 20 18:17:41 tweedledum named[27142]: transfer of 'mcslp.pri/IN' from 192.168.0.2#53: end of transfer
Now you can add the IP address of your new slave server to the list of valid IP addresses to query. Back to top Summary The DNS system, at its heart, resolves the names with which we are all familiar on the Internet and locally on our networks to an IP address. Although we take the system for granted, it is in fact a well-designed service that distributes and delegates queries and zones around the Internet and enables you to create custom zones within your own network. In this article we've seen how the fundamentals of the DNS service work, and how that relates not only to the internal process of resolve local addresses but also to addresses on the Internet. We've also examined how to create a simple DNS setup for use on your local network, and how that integrates with Internet name resolution.
Systems Administration Toolkit: Monitoring mail usage

Martin Brown (mc@mcslp.com), Freelance Writer, Consultant
Summary: Examine methods for reporting on e-mail usage and getting detailed statistics on mailbox use. E-mail logs can provide a wealth of information about your
users and their e-mail habits, including how often and frequently they use their e-mail. Depending on your UNIX systems, you might even be able to glean more detailed information, such as identifying which users are using their folders effectively and who could benefit from further training and advice. View more content in this series Tags for this article: aix, aix_and_unix, log_file, monitoring, systems_admin, toolkit
Date: 18 Dec 2007 Level: Intermediate Also available in: Chinese Russian Activity: 12493 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Examining e-mail logs The logs generated by the different e-mail processes can contain a lot of useful information. What information is contained in the logs depends largely on the mail system you are using (sendmail, postfix, qmail, and so forth), but some examples of the information that you can extract include:

E-mail rates and quantities Per-user e-mail rates Common hosts and domains sending and receiving e-mail Spam and virus quantities and filtering
Examining the logs can give you valuable information about how e-mail is used in your organization. Parsing the logs can be a complicated process, and the quality of the results largely depend on the format of the log and the complexity involved in parsing the file for information. For example, if you look at the log produced by the postfix mail transfer agent (MTA), you can see that the information for a single message is spread over a number of lines (see Listing 1).
Listing 1. Extract from a raw e-mail log

Nov 17 03:17:34 narcissus postfix/smtpd[14281]: connect from localhost[127.0.0.1] Nov 17 03:17:34 narcissus postfix/smtpd[14281]: 4F4CB1109404: client=localhost[127.0.0.1] Nov 17 03:17:34 narcissus postfix/cleanup[14278]: 4F4CB1109404: message-id=<Pine.SOC.4.64.0711162015280.16833@logical.logicalapproach.com> Nov 17 03:17:34 narcissus postfix/qmgr[104]: 4F4CB1109404: from=<zfs-discuss-bounces@opensolaris.org>, size=7632, nrcpt=1 (queue active) Nov 17 03:17:34 narcissus postfix/smtpd[14281]: disconnect from localhost[127.0.0.1] Nov 17 03:17:34 narcissus postfix/smtp[14279]: DBA5B11093FD: to=<dev@gendarme.example.com>, relay=127.0.0.1[127.0.0.1], delay=11, status=sent (250 2.6.0 Ok, i d=08640-07, from MTA([127.0.0.1]:10025): 250 Ok: queued as 4F4CB1109404) Nov 17 03:17:34 narcissus postfix/qmgr[104]: DBA5B11093FD: removed Nov 17 03:17:34 narcissus postfix/pipe[14283]: 4F4CB1109404: to=<dev@gendarme.example.com>, relay=cyrus, delay=0, status=sent (gendarme.example.com) Nov 17 03:17:34 narcissus postfix/qmgr[104]: 4F4CB1109404: removed Nov 17 03:20:07 narcissus postfix/smtpd[14355]: connect from narcissus.example.com[192.168.0.110] Nov 17 03:20:07 narcissus postfix/smtpd[14355]: disconnect from narcissus.example.com[192.168.0.110] Nov 17 03:20:07 narcissus postfix/smtpd[14355]: connect from narcissus.example.com[192.168.0.110] Nov 17 03:20:07 narcissus postfix/smtpd[14355]: disconnect from narcissus.example.com[192.168.0.110] Nov 17 03:23:16 narcissus postfix/smtpd[14410]: connect from f048226119.adsl.alicedsl.de[78.48.226.119] Nov 17 03:23:17 narcissus postfix/smtpd[14410]: 6CAAE1109461: client=f048226119.adsl.alicedsl.de[78.48.226.119] Nov 17 03:23:17 narcissus postfix/cleanup[14411]: 6CAAE1109461: message-id=<000901c828c9$037c1e48$b9cc3eb7@ggpxddd> Nov 17 03:23:17 narcissus postfix/qmgr[104]: 6CAAE1109461: from=<maxwell@research.nokia.com>, size=2051, nrcpt=1 (queue active) Nov 17 03:23:18 narcissus postfix/smtpd[14410]: disconnect from f048226119.adsl.alicedsl.de[78.48.226.119] Nov 17 03:23:30 narcissus postfix/smtpd[14414]: connect from localhost[127.0.0.1] Nov 17 03:23:30 narcissus postfix/smtpd[14414]: 62E941109473: client=localhost[127.0.0.1]
Nov 17 03:23:30 narcissus postfix/cleanup[14411]: 62E941109473: message-id=<000901c828c9$037c1e48$b9cc3eb7@ggpxddd> Nov 17 03:23:30 narcissus postfix/qmgr[104]: 62E941109473: from=<>, size=3220, nrcpt=1 (queue active) Nov 17 03:23:30 narcissus postfix/smtpd[14414]: disconnect from localhost[127.0.0.1]
Fortunately, you can identify the process of a single message through the system by using the unique reference number that each e-mail is given. For example, Listing 2 shows an e-mail with the reference number 4F4CB1109404, which you can use to pull out relevant information to determine what to extract.
Listing 2. Extracting the information from a single mail transfer

Nov 17 03:17:34 narcissus postfix/smtpd[14281]: 4F4CB1109404: client=localhost[127.0.0.1] Nov 17 03:17:34 narcissus postfix/cleanup[14278]: 4F4CB1109404: message-id=<Pine.SOC.4.64.0711162015280.16833@logical.logicalapproach.com> Nov 17 03:17:34 narcissus postfix/qmgr[104]: 4F4CB1109404: from=<zfs-discuss-bounces@opensolaris.org>, size=7632, nrcpt=1 (queue active) Nov 17 03:17:34 narcissus postfix/pipe[14283]: 4F4CB1109404: to=<dev@gendarme.example.com>, relay=cyrus, delay=0, status=sent (gendarme.example.com) Nov 17 03:17:34 narcissus postfix/qmgr[104]: 4F4CB1109404: removed
From the extract, you can see the different types of information that you can remove from the logs, such as:

Date and time Sender Recipient Message size Message count
By parsing the log file contents, you can create some interesting statistics on combinations of these different elements to gain a better idea of how e-mail is being used. Parsing the log file for information To parse the contents, you need to be able to identify each e-mail individually. This is possible because the MTA embeds a unique ID with each e-mail, which is often inserted into the log file output. For example, in this line:
Nov 17 03:17:34 narcissus postfix/pipe[14283]: 4F4CB1109404: to=<dev@gendarme.example.com>, \ relay=cyrus, delay=0, status=sent (gendarme.example.com)
the unique ID is the hex value 4F4CB1109404, and you can also determine from this line the address of the recipient. The sender of this e-mail can be found in another line with the same embedded ID:
Nov 17 03:17:34 narcissus postfix/qmgr[104]: 4F4CB1109404: from=\ <zfs-discuss-bounces@opensolaris.org>, size=7632, nrcpt=1 (queue active)
The information is not always sequential, as the MTA might be handling multiple e-mails at the same time, and the information all gets written to the log as different parts of the process are completed. You can also see from the line above the total size of the e-mail (7632 bytes) and the number of recipients (one). Listing 3 shows a Perl script that collates the information and then outputs some summary statistics of the number of e-mails and their total size.
Listing 3. Parsing the log into some useful statistics

#!/usr/bin/perl # # Script to extract email statistics from log files # Time::ParseDate will be used parse the time into an epoch # value, and then DateTime can be used to reformat the date # again use Time::ParseDate; use DateTime; # Parse the first file on the command line open(MAIL,$ARGV[0]) or die "Couldn't open $ARGV[0]: $!\n"; # Create a structure to hold the stats my $mails = {}; # Parse each line of the file while(<MAIL>) {
chomp; my $mailid = 0; # Look for the 12 digit hex mail ID if (m/: ([A-Z0-9]{12}):/) { $mailid = $1; } # Extract the date and parse it into an Epoch value if (m/(\S+ \d+ \d{2}:\d{2}:\d{2}) .*? $mailid/) { $mails->{$mailid}->{date} = parsedate($1); } # Extract the sender address and email size if (m/$mailid: from=<(.*?)>, size=(\d+),/) { $mails->{$mailid}->{from} = $1; $mails->{$mailid}->{size} = $2; } # Extract the recipient if (m/$mailid: to=<(.*?)>/) { $mails->{$mailid}->{to} = $1; } } close(MAIL); # Compile together the stats by parsing the formatted # information into another summary structure my $mailstats = {}; foreach my $mailid (keys %{$mails}) { # Don't create a summary entry if we don't have enough information # (sender/recipient is empty) if (!defined($mails->{$mailid}->{to}) || !defined($mails->{$mailid}->{from}) || $mails->{$mailid}->{to} !~ m/[a-z]/ || $mails->{$mailid}->{from} !~ m/[a-z]/) { next; } # Count the number of emails to each recipient $mailstats->{$mails->{$mailid}->{to}}->{count}++; # Sum up the email size to each recipient
$mailstats->{$mails->{$mailid}->{to}}->{size} += $mails->{$mailid}->{size}; # Count the number of emails from each sender $mailstats->{$mails->{$mailid}->{from}}->{count}++; # Sum up the email size from each sender $mailstats->{$mails->{$mailid}->{from}}->{size} += $mails->{$mailid}->{size}; # Sum up the same information, but organized on a date by date basis if (defined($mails->{$mailid}->{date})) { my $dt = DateTime->from_epoch( epoch => $mails->{$mailid}->{date})->ymd(''); my $mailto = $mails->{$mailid}->{to}; my $mailfrom = $mails->{$mailid}->{from}; $mailstats->{$mailto}->{_date}->{$dt}->{count}++; $mailstats->{$mailto}->{_date}->{$dt}->{size} += $mails->{$mailid}->{size}; $mailstats->{$mailfrom}->{_date}->{$dt}->{count}++; $mailstats->{$mailfrom}->{_date}->{$dt}->{size} += $mails->{$mailid}->{size}; } } # Dump out the information show mail counts and mail sizes # on a mail address basis foreach my $address (sort keys %{$mailstats}) { # Only show information from email addresses that are # local if ($address =~ m/@.*example\.com$/) { printf('%-40s %5d %9d', $address, $mailstats->{$address}->{count}, $mailstats->{$address}->{size}); print("\n"); } }
Running the script on a sample log file produces some statistics showing the mail counts and sizes for the domain specified (see Listing 4). The actual information that you print is significantly smaller than the amount of information you could produce. The statistics compiled can also output the information on a day-by-day basis, and that could help to indicate mail counts for each day so that you can get an idea of the level of load on the mail server.
Listing 4. Generated statistics

023649@gendarme.example.com admin@gendarme.example.com answersquad@gendarme.example.com buy@gendarme.example.com buy@example.com dev@gendarme.example.com dev@example.com finance@gendarme.example.com flummox@gendarme.example.com klhxs@example.com dj@gendarme.example.com dj@example.com mcwhoever@gendarme.example.com mlists@gendarme.example.com play@gendarme.example.com postmaster@gendarme.example.com fhf@gendarme.example.com fhffilter@gendarme.example.com spam@gendarme.example.com tmbg-org-dj@gendarme.example.com ybxmbjw@gendarme.example.com 5 5 50 79 2 165 8 4 203 4 96 2 7 38 26 5 101 14 49 16 1 8922 13141 348712 806097 2911 1010980 94430 14493 858775 8080 703754 2063 137185 473165 155220 15561 836863 225081 349750 126144 1342
You can see from this example data that certain users are receiving large amounts of mail while others are comparatively light users. In this case, the information tells you that the users receiving large amounts of mail probably merit closer attention to determine whether they are using their e-mail effectively. If you were using a single mail server that you knew was reaching an overload point, you might consider splitting out the users so that a small number of heavy users are moved to a single server, and the users with "lighter" usage are placed on to another server to split the load effectively. For now, let's use the information as a guide to look at some of the heavier users more closely and see how they are actually storing and organizing their e-mail. Mailbox storage formats One of the more extensive (and possibly controversial) methods for monitoring your mail usage is to look at how individual users are using and examining their e-mail. The problem with this approach is that some users see such examination as an intrusion, and others feel that how they organize and archive their e-mail is up to them.
The reality is that knowing how your users are organizing their e-mail can have an impact on performance, backup strategies for e-mail, and capacity planning by enabling you to determine which parts of the system are being used the most. Actually, reading the e-mail of individuals is a bad idea, but you can still glean a lot of information from mailboxes of users by using fairly simple techniques that do not require you to look at individual messages. Also bear in mind that the e-mail solution that you are using also affects the information that can be gleaned. Most modern e-mail solutions use the Internet Mail Access Protocol (IMAP) system for storing e-mail. Here the e-mail is stored on the server and the client accesses the e-mail folder to read the contents. This system allows different clients to connect to the server and access the information, which is an advantage. Because all the e-mail is on a single server, backups and access to e-mail while traveling (either directly or from a hotel) are much easier. The back-end of the IMAP server is where the individual e-mail is stored. The majority of systems use one of two storage methods, single file or Maildir. The former method uses a single file to hold all of the e-mail, occasionally changing this to a system where a single file holds the mail for each folder. Using a single file is problematic, especially in a multi-user system because of issues of file locking and the performance problems associated with working with a single, large file. The best solution is the Maildir format, where standard directories on the file system are used to refer to mail folders and individual files are used to refer to each individual email message. This provides for quicker and more efficient access. However, there are potential problems with large amounts of mail in a single location with both systems. With the single-file solution, performance and possibility of corruption are large. With the Maildir format, some systems suffer when you have a large number of files in a single directory. This is a file system, not mail system, issue, but it still needs to be addressed. You can use the mailbox usage report to determine whether a new server would alleviate the problems or whether a user needs to organize their e-mail to better improve the overall performance of the server. Getting some basic mailbox usage statistics One of the simplest ways to examine the way your users are organizing their e-mail is to run a simple disk usage report. This helps to highlight whether some users are storing large quantities of e-mail on the server and presenting a larger proportion of the heavy load that needs to be addressed.
The mail server used for these examples is Cyrus IMAP, which uses the Maildir format, but the same basic principles apply to solutions that use a single file for their mail store. Listing 5 shows a simple disk usage report for some of the users.
Listing 5. Getting disk usage statistics

$ du -sk * 20 admin 1020 appleby 45828 applicants 13264 buy 11704 dev 11200 finance 52 gordon 25720 mc 472 mcantrip 4340 mlists 120952 myeling 12 nutshell 1812 play 1104 postmaster 1468 quarantine 36 questions 78940 fhf 4156 fhfarc 2444 spam
If you want to enforce mail quotas, you could compare the information output at this stage with a global or per-user quota, and provide the user with some kind of warning. Quotas, however, only tell the user if they are using up disk space. They do not mention if there is a better way of organizing their e-mail or making better use of the space available. To find out whether a user is making use of the ability to store e-mail in different folders, use find to perform some more detailed examinations. For example, you can request a disk usage report on a directory-by-directory basis with the Maildir format. find searches for directories, and sort ensures that the information is returned in directory name order (see Listing 6). The output has been trimmed slightly to highlight specific areas.
Listing 6. Getting disk usage stats on a folder-by-folder basis

$ find . -type d -exec du -sk {} \; |sort -t. +1 ... 45828 ./applicants 1964 ./brady
616 1304 ... 11200 4160 12 1192 5476 296 ... 78940 ... 460 3444 1484 40920 1264 1984 ...
./brady/Article Comments ./brady/Editors ./finance ./finance/Banking ./finance/Insurance ./finance/Other ./finance/Rewards ./finance/Surveys ./sandra ./sandra/Nutshell ./sandra/Orders ./sandra/SANDRA-general ./sandra/Sent Mail ./sandra/Tracey ./sandra/eBay
Here you have three users at sensible extremes. applicants is using a single mail folder to store all their mail, and finance has sensibly split their e-mail into different folders for easy organization. Meanwhile, sandra seems to be ordering the e-mail correctly, but has one massive folder with all her sent e-mail. Both applicants and sandra would probably benefit from some sage advice on how to better organize their e-mail. But is there anyway you could help them? Filing e-mail automatically There are many solutions for filing and automatically organizing your e-mail. Some of them are built into your client and some are built into the mail server. For example, the Cyrus IMAP server comes with the Sieve system (see Resources) for automatically filtering e-mail when it's delivered to the mailbox of the user. While this obviously helps new e-mail, existing e-mail needs to be organized by hand. One way to do this is to use a script that accesses the mailbox just like any other client, and then organizes and filters your e-mail based on some rules built into the script. Listing 7 shows a Perl script that accesses the e-mail folder as a client and processes the contents. In this example, the script creates a new sub-folder structure under the original folder that organizes the e-mail by date, with a folder for each year, and a folder for each month within that year.
Listing 7. Automatically reorganizing e-mail in a golder by date

#! /usr/bin/perl
# Mail filter to file mail on a date basis use use use use use Mail::IMAPClient; Date::Parse; Data::Dumper; strict; warnings;
# The IMAP Server my $Server = 'imap.example.com'; # The Mailbox we want to filter my $INBOX = "Sent-Mail"; # Open the server connection my $IMAP = Mail::IMAPClient -> new (Server => $Server, User => 'user', Password => 'password',); # Open the mailbox we want to filter $IMAP->select($INBOX) or die "Couldn't select $INBOX"; # We want to filter every message, so obtain a list of every # message by the message ID my @msgids = $IMAP->search("ALL"); # Don't do anything if there's nothing to process exit(0) if (scalar @msgids == 0); # Now parse the message contents to determine # the From, To, Subject and Address of each message my $parsed = $IMAP->parse_headers( $IMAP->Range(\@msgids), "From", "To", "CC", "Subject", "Date", ); # Set up some message counters my $toprocess = scalar @msgids; my $processed = 0; my $counter = 0; # Process each message foreach my $msgid (keys %{$parsed}) { $processed++; # Extract the date, and build a new folder path
# The new path will split up emails first by # year and then by month, all as subfolders # of the current folder my ($ss,$mm,$hh,$day,$month,$year,$zone) = strptime($parsed->{$msgid}->{Date}->[0]); # Try another date if the first one couldn't be identified if (!defined($year)) { ($ss,$mm,$hh,$day,$month,$year,$zone) = strptime($parsed->{$msgid}->{Date}->[1]); } # default to 2004 if we can't find a year if (!defined($year)) { $year = 2004; } # Make some assumptions about the year # Occasionally a date will contain only two digits # So assume it's either in the year 2000, or 1990+ $year += 2000 if ($year <10); $year += 1900 if (($year >90) && ($year <= 200)); $year = 2004 if $year <= 1990; $month += 1; # Construct the new folder path my $destfolder = sprintf('%s/%s/%02d',$INBOX,$year,$month); # Grab the entire message my $Message = $IMAP -> message_string($msgid); # Try to change to the destination folder, # or create it if we couldn't select the folder my $selectstat = $IMAP->select($destfolder); unless ($selectstat) { $IMAP->create($destfolder); } # Go back to the Inbox so that we select the right message # next time round $IMAP->select($INBOX); # Add the original message to the new folder my $AppendStatus = $IMAP -> append_string($destfolder,$Message); # When you add a message to a folder, the message # is marked as unread, so mark all the messages # in the folder as read by reading them $IMAP->select($destfolder);
my @unseenMIDs = $IMAP->unseen(); foreach my $MID (@unseenMIDs) { $IMAP->message_string($MID); } # Go back to the original folder, and delete the message # if it was successfully moved $IMAP->select($INBOX); if ($AppendStatus) { $IMAP -> delete_message($msgid); } $counter++; } # Print out a summary of what we achieved printf("Processed %5d out of %5d msgs\r",$counter,$toprocess); # Make sure we clean out the folder where we deleted messages # and then disconnect $IMAP->expunge(); $IMAP->disconnect();
The script relies on you having the user's login and password, but you could easily adapt the script to accept this information on the command line where user could execute the script when needed. Running this script on a folder, such as the Sent Mail folder, nicely organizes the information, as you can see here in Listing 8 when you run the script again.
Listing 8. A filtered mailbox shows a better structure

28932 15288 416 812 684 2068 2576 772 708 1104 3308 2840 13632 908 884 ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent Mail Mail/2006 Mail/2006/03 Mail/2006/04 Mail/2006/05 Mail/2006/06 Mail/2006/07 Mail/2006/08 Mail/2006/09 Mail/2006/10 Mail/2006/11 Mail/2006/12 Mail/2007 Mail/2007/01 Mail/2007/02
1596 864 1532 1448 1116 2092 2220 972
./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent ./martin/Sent
Mail/2007/03 Mail/2007/04 Mail/2007/05 Mail/2007/06 Mail/2007/07 Mail/2007/08 Mail/2007/09 Mail/2007/10
A modification of the original script is to parse the message contents and file the mail according to the e-mail sender or subject. The main benefit of this solution over the typical server or client mail filing systems is that you can use it on existing mailboxes. Many server and client mail systems only file mail on "incoming" or "new" e-mail and become useless once the e-mail reaches the user's mailbox. In some situations, you can even use it as an automatic process that just becomes part of the e-mail system. You can set up the various auto-filers to execute automatically, where the user drops the e-mail on to an e-mail folder and each night the e-mail is automatically moved to a predetermined location. Other uses for this technique include warning the user when they have a high level of email in a folder, particularly the inbox, or simply for automatically "archiving" e-mail away into a folder so that the user only has their recent inbox contents available. If you use global mailboxes (for example, in support or queries from customers), then the autofiling approach can be an effective way of organizing and filtering the e-mail so that it can be located again later. When using an automated customer relationship management (CRM) system, the issue is often tracked with a number. You can easily set up an auto filer, such as the one demonstrated above, to filter the e-mail by the issue number into an appropriate folder. Note that this doesn't replace the need for server-side or client-side filtering, but it can be an extension to that process that provides more flexibility and additional information and support. Summary In this article, you've examined some of the main ways in which you can extract more information about the e-mail usage of the server at a user level. This information is not about determining whether the server as a whole is running efficiently, but whether your users are getting the best out the services available and how effectively they are using the systems. Parsing the e-mail logs to determine high-volume users is a good way to start that process. You can find out who is sending and receiving the most e-mail, and determine
whether there are users who should move to other servers or who need more attention to determine their e-mail use. For the latter, a closer inspection of the user's mailbox and mail filing is a good idea, as it can help. Even better, you can re-filter the contents of the mailbox by parsing the information with a script and moving mail messages into a more suitable structure.
Systems Administration Toolkit: Network scanning

Summary: Discover how to scan your network for services and how to regularly monitor your services to keep uptimes to a maximum. A key way of ensuring the security of your network is to know what is on your network and what services individual machines are at risk of exposure. Unauthorized services, such as Web servers or file sharing solutions, not only degrade performance, but others can use these services as routes into your network. In this article, learn how to use these same techniques to ensure that genuine services remain available. Tags for this article: aix, aix_and_unix, network, network_scanning, scanning, systems_admin, toolkit
Date: 04 Dec 2007 Level: Intermediate Also available in: Chinese Russian Activity: 22688 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through
years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Types of network scanning Scanning your network enables you to build a quick picture of exactly what is going on, configuration options, and what is available on your network. Different network scans look for and record different information. And you can also employ them in different ways to determine various types of information about your network. One of the key elements of any network environment is to be able to identify what is on your network. Network hacks, unauthorized equipment, and bad or misconfigured hosts can all lead to potential problems on your network. The first type of network scanning you should use is one that finds and locates the Internet Protocol (IP) addresses in use on the network. Since an IP address is comparatively useless without also having the network port on which you communicate (for example, the ports that Hypertext Transfer Protocol (HTTP) and Simple Mail Transfer Protocol (SMTP) protocols use), you must find out which of these ports others actively listened to for connections. The other type of scanning involves actually looking at the information contained within the raw packets that go past on your network. Raw network scanning provides a wealth of information, such as understanding the precise nature of exchanged information over the network and the performance of the network as a whole. By combining the information you know about hosts and ports on the network with the information you gain from the raw network scanning, you can get a much better understanding of what is actually going on within the network. For example, during a network scan, you might find that a particular host is listening on a non-standard port, but it can be difficult to determine the use of the host and port without raw network scanning to understand what type of date and information. Is someone using the host and port for a genuine purpose, such as HTTP, or is it something more subversive, such as a network hack, an open SSH, or other port that one can use for a security breach? Scanning for hosts Determining what hosts exist on your network is the most basic of network scanning tasks, and it can be the greatest way of understanding what is going on within your network. The information on its own can be useful, but it can be even more beneficial to combine the information that you have determined and discovered with the information
that you should technically already knowthat is, the list of hosts configured on your network and their IP address. When scanning for network IPs, there are a few things that you should keep in mind:

You can change the IP addresses and, if you use Dynamic Host Configuration Protocol (DHCP) on your network, then the list of IP addresses currently in use is probably incomplete. A single scan is unlikely to tell you everything you need to know. When scanning for IP addresses, you can only identify the IP address of a host that is on and running an operating system. Systems that are off or asleep do not show up. Therefore, you often need multiple scans to be able to identify all the hosts on the network that you know are valid. The fact that you can find a host indicates that it is configured on your network, but it doesn't tell you if it is authorized. You need to compare the information with data that you know to ensure that the host is valid on your network.
The nmap tool provides the ability to scan for hosts across your network. It also identifies which hosts are accessible, their IP and MAC addresses, and scans and checks the open ports, protocols used, and the host operating system. To perform a basic ping test and identify all of the hosts configured on the network, use the -sP command-line option and provide the IP address specification that you want to scan. The IP address specification supports a number of different formats, including the IP address and netmask, network specification, and even IP address ranges. For example, to scan all of the hosts in the 192.168.0.0 network, you could use Listing 1.
Listing 1. Scanning all the hosts in the 192.168.0.0 network

$ nmap -sP 192.168.0.0/24 Starting Nmap 4.20 ( http://insecure.org ) at 2007-10-14 19:15 GMT Host gendarme.mcslp.pri (192.168.0.1) appears to be up. MAC Address: 00:0F:B5:30:42:60 (Netgear) Host 192.168.0.31 appears to be up. MAC Address: 00:02:11:90:15:31 (Nature Worldwide Technology) Host 192.168.0.50 appears to be up. MAC Address: 00:61:CA:28:0C:23 (Unknown) Host 192.168.0.66 appears to be up. MAC Address: 00:08:7B:05:D1:52 (RTX Telecom A/S) Host 192.168.0.68 appears to be up. MAC Address: 00:18:4D:59:AB:B5 (Netgear) Host 192.168.0.101 appears to be up. MAC Address: 00:16:CB:A0:3B:CB (Apple Computer) Host 192.168.0.103 appears to be up. MAC Address: 00:0D:93:4E:61:2A (Apple Computer) Host nautilus.mcslp.pri (192.168.0.108) appears to be up. MAC Address: 00:16:CB:8F:04:0B (Apple Computer)
Host narcissus.mcslp.pri (192.168.0.110) appears to be up. MAC Address: 00:16:CB:85:2D:15 (Apple Computer) Host airong.wireless.mcslp.pri (192.168.0.210) appears to be up. MAC Address: 00:0F:B5:0D:82:FC (Netgear) Host gentoo2.vm.mcslp.pri (192.168.0.230) appears to be up. Nmap finished: 256 IP addresses (11 hosts up) scanned in 20.758 seconds
The output gives you a very clear list of the hosts discovered, the IP and MAC addresses registered, and a brief determination of the probable host provider. Since individual manufacturers are given blocks of MAC addresses, you can obtain this information. What this information doesn't tell you is what ports these hosts are potentially listening for connections on. Listening ports can tell you what sort of traffic the host is using and, if the host is not one you are aware of, what that host might be doing so that you can identify it. Scanning for ports When a machine is running a particular service, for example, a Web server, e-mail server, or peer-to-peer service, the machine must listen on a specific port. Some of these ports are well-known (for example, port 80 is used for HTTP). You can do a port scan on all hosts that you check or just an individual host. For example, if you perform a port scan on a single host, then you can get a very good idea of what the machine is doing. You can see here in Listing 2 when you scan the ports for a server.
Listing 2. Scanning the ports for a server

$ nmap -sT narcissus Starting Nmap 4.20 ( http://insecure.org ) at 2007-10-14 19:45 GMT Interesting ports on narcissus.mcslp.pri (192.168.0.110): Not shown: 1674 closed ports PORT STATE SERVICE 21/tcp open ftp 22/tcp open ssh 25/tcp open smtp 53/tcp open domain 80/tcp open http 106/tcp open pop3pw 110/tcp open pop3 111/tcp open rpcbind 143/tcp open imap 311/tcp open asip-webadmin 389/tcp open ldap 427/tcp open svrloc
443/tcp open https 548/tcp open afpovertcp 625/tcp open unknown 749/tcp open kerberos-adm 1011/tcp open unknown 1014/tcp open unknown 1015/tcp open unknown 2049/tcp open nfs 3306/tcp open mysql 3689/tcp open rendezvous 5900/tcp open vnc MAC Address: 00:16:CB:85:2D:15 (Apple Computer) Nmap finished: 1 IP address (1 host up) scanned in 22.399 seconds
The scan output only shows the active Transmission Control Protocol (TCP) and IP ports. If you also want to scan the User Datagram Protocol (UDP) ports, you must specify the -sU options, as shown here in Listing 3.
Listing 3. Specifying the -sU options

$ nmap -sU -sT narcissus Starting Nmap 4.20 ( http://insecure.org ) at 2007-10-14 20:05 GMT Interesting ports on narcissus.mcslp.pri (192.168.0.110): Not shown: 3150 closed ports PORT STATE SERVICE 21/tcp open ftp 22/tcp open ssh 25/tcp open smtp 53/tcp open domain 80/tcp open http 106/tcp open pop3pw 110/tcp open pop3 111/tcp open rpcbind 143/tcp open imap 311/tcp open asip-webadmin 389/tcp open ldap 427/tcp open svrloc 443/tcp open https 548/tcp open afpovertcp 625/tcp open unknown 749/tcp open kerberos-adm 1011/tcp open unknown 1014/tcp open unknown 1015/tcp open unknown 2049/tcp open nfs 3306/tcp open mysql 3689/tcp open rendezvous 5900/tcp open vnc 53/udp open|filtered domain 67/udp open|filtered dhcps
88/udp open|filtered kerberos-sec 111/udp open|filtered rpcbind 123/udp open|filtered ntp 427/udp open|filtered svrloc 464/udp open kpasswd5 989/udp open|filtered unknown 990/udp open|filtered unknown 1008/udp open|filtered ufsd 1021/udp open|filtered unknown 5353/udp open|filtered zeroconf MAC Address: 00:16:CB:85:2D:15 (Apple Computer) Nmap finished: 1 IP address (1 host up) scanned in 78.605 seconds
You can repeat this process across all the hosts in your network. Recording the information As described earlier, a single pass of either the host or port scanning information is only useful for the instant in time when the scan was executed. Hosts might be down, services might be temporarily unavailable and, in a very busy network, the scanning packets simply might not have gone through. You should be running the scan regularly and then comparing (and even combining) the output from multiple executions. The easiest way to do this is to run nmap, record the output, and then use a tool like diff to compare the output and report the changes. You can see an example of this in Listing 4, where you are comparing the output of two executions of nmap on the same host.
Listing 4. Comparing the output of two executions of nmap on the same host
$ diff narcissus.20070526 narcissus.20070924 26c26 < 2049/tcp open nfs --> 3306/tcp open mysql 27a28 > 5900/tcp open vnc
Lines begin with a '<' or '>', according to which file the line is different. In this case, you can see that the Network File System (NFS) protocol is in the first file but not the second, while both the mysql and vnc ports have been open during the latter scan. Scanning raw packets
The nmap tool is great for finding IP hosts and open hosts; however, nmap doesn't tell you the content of the actual traffic running over the network. Capturing raw packets enables you to see which hosts are communicating with a given host, what information is being exchanged, and which are being used. You can get the raw packet information and find out what information is being transferred with the right settings (for example, which users are logging in to a particular mail server from which host). You can even use it to identify the type of exchanged traffic on an otherwise unidentified port. If you want to scan the raw data information that goes past on the network, you need to use a tool other than nmap, which only scans, hosts, and ports. There are many different tools available, including tools that come with different operating systems and third-party solutions. See the Resources section for some examples. On AIX, you can use the iptrace tool to scan the network. On Solaris, the snoop tool performs a similar purpose, while third-party tools like Ethereal also provide the scanning ability. Each tool uses the same premise. For example, with the Ethernet, network standard packets are echoed to all physical ports and, by scanning all the packets that go past on the network, you can examine the content. With Ethernet switches, packets are not echoed to all ports, but many modern switches provide a management port where the information is echoed. Even on switched networks, you can gain some valuable information by running the packet-scanning tool on the host you want to investigate. The methods for using these tools and the information they can provide differ slightly, but the fundamentals are the same. With AIX, the iptrace is a background daemon, so you must explicitly start and stop the tool to switch the process on and off. For example, to start, type:
# startsrc -s iptrace -a "-i tr0 /home/user/iptrace/log1"
To stop, type:
# stopsrc -s iptrace
On Solaris, the snoop tool is an application that you can execute at will:
# snoop
Because the quantity of information passing by on the network can be considerable, you can select to only obtain a limited set of information by limiting the host, ports, or
protocols that you want to scan. For example, to only examine the exchanged data to or from a particular host, try:
# snoop narcissus
Furthermore, because most display devices are unable to keep up with the display of the raw data information, it is best to export the information directly to a file:
# snoop -o /tmp/netdata narcissus.mcslp.pri
To read back the information from the file, use the -i command-line option to specify the input source:
# snoop -v -i /tmp/netdata
In the above example, verbose mode has also been switched this ensures that the full contents of the raw packets are recorded, instead of just the summary information that snoop provides by default. When you examine the raw data, snoop automatically formats and parses the content of the packet for you, even down to the individual protocol level. For example, Listing 5 shows the output during an NFS data exchange.
Listing 5. Output during an NFS data exchange

ETHER: ----- Ether Header ----ETHER: ETHER: Packet 31 arrived at 10:00:2.70371 ETHER: Packet size = 174 bytes ETHER: Destination = 0:2:11:90:15:31, ETHER: Source = 0:16:cb:85:2d:15, ETHER: Ethertype = 0800 (IP) ETHER: IP: ----- IP Header ----IP: IP: Version = 4 IP: Header length = 20 bytes IP: Type of service = 0x00 IP: xxx. .... = 0 (precedence) IP: ...0 .... = normal delay IP: .... 0... = normal throughput IP: .... .0.. = normal reliability IP: .... ..0. = not ECN capable transport IP: .... ...0 = no ECN congestion experienced IP: Total length = 160 bytes
IP: IP: IP: IP: IP: IP: IP: IP: IP: IP: IP: IP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: TCP: RPC: RPC: RPC: RPC: RPC: RPC: RPC: RPC: RPC: RPC: NFS: NFS: NFS: NFS: NFS: NFS: NFS: NFS: NFS: NFS: NFS: NFS: NFS: NFS:
Identification = 14053 Flags = 0x4 .1.. .... = do not fragment ..0. .... = last fragment Fragment offset = 0 bytes Time to live = 64 seconds/hops Protocol = 6 (TCP) Header checksum = 8195 Source address = 192.168.0.110, narcissus.mcslp.pri Destination address = 192.168.0.31, ultra3 No options ----- TCP Header ----Source port = 2049 Destination port = 1022 (Sun RPC) Sequence number = 1812746020 Acknowledgement number = 1237063652 Data offset = 20 bytes Flags = 0x18 0... .... = No ECN congestion window reduced .0.. .... = No ECN echo ..0. .... = No urgent pointer ...1 .... = Acknowledgement .... 1... = Push .... .0.. = No reset .... ..0. = No Syn .... ...0 = No Fin Window = 65535 Checksum = 0x48da Urgent pointer = 0 No options ----- SUN RPC Header ----Record Mark: last fragment, length = 116 Transaction id = 2490902676 Type = 1 (Reply) This is a reply to frame 29 Status = 0 (Accepted) Verifier : Flavor = 0 (None), len = 0 bytes Accept status = 0 (Success) ----- Sun NFS ----Proc = 3 (Look up file name) Status = 2 (No such file or directory) Post-operation attributes: (directory) File type = 2 (Directory) Mode = 0770 Setuid = 0, Setgid = 0, Sticky = 0 Owner's permissions = rwx Group's permissions = rwx Other's permissions = --Link count = 13, User ID = 1025, Group ID = 1026 File size = 442, Used = 4096 Special: Major = 0, Minor = 0
NFS: NFS: NFS: NFS: NFS: NFS:
File system id = 234881046, File id = 22 Last access time = 14-Oct-07 16:45:50.000000000 GMT Modification time = 11-Oct-07 12:48:53.000000000 GMT Attribute change time = 11-Oct-07 12:48:53.000000000 GMT
Each line is prefixed with the protocol level within the packet that was scanned. In this case, you can see the raw Ethernet, IP, and TCP packet data, including the source and destination host information and packet options. At the Remote Procedure Call (RPC) level, you get the record information for the NFS request, and then at the NFS file, you get the actual data that was exchanged. In this case, you are looking at the examination of a file (during a directory listing, most likely), and you can see the file's type, permissions, and modification time information as well. Listing 6 shows the output during a Domain Name Server (DNS) lookup for a hostname.
Listing 6. Output during a DNS lookup for a hostname

ETHER: ----- Ether Header ----ETHER: ETHER: Packet 2 arrived at 10:05:19.05284 ETHER: Packet size = 155 bytes ETHER: Destination = 0:2:11:90:15:31, ETHER: Source = 0:16:cb:85:2d:15, ETHER: Ethertype = 0800 (IP) ETHER: IP: ----- IP Header ----IP: IP: Version = 4 IP: Header length = 20 bytes IP: Type of service = 0x00 IP: xxx. .... = 0 (precedence) IP: ...0 .... = normal delay IP: .... 0... = normal throughput IP: .... .0.. = normal reliability IP: .... ..0. = not ECN capable transport IP: .... ...0 = no ECN congestion experienced IP: Total length = 141 bytes IP: Identification = 53767 IP: Flags = 0x0 IP: .0.. .... = may fragment IP: ..0. .... = last fragment IP: Fragment offset = 0 bytes IP: Time to live = 64 seconds/hops IP: Protocol = 17 (UDP) IP: Header checksum = 267b IP: Source address = 192.168.0.110, narcissus.mcslp.pri IP: Destination address = 192.168.0.31, ultra3
IP: IP: UDP: UDP: UDP: UDP: UDP: UDP: UDP: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS: DNS:
No options ----- UDP Header ----Source port = 53 Destination port = 46232 Length = 121 Checksum = 0C31 ----- DNS Header ----Response ID = 55625 AA (Authoritative Answer) RA (Recursion Available) Response Code: 0 (OK) Reply to 1 question(s) Domain Name: www.mcslp.pri. Class: 1 (Internet) Type: 28 (IPv6 Address) 2 answer(s) Domain Name: www.mcslp.pri. Class: 1 (Internet) Type: 5 (Canonical Name) TTL (Time To Live): 86400 Canonical Name: narcissus.mcslp.pri. Domain Name: narcissus.mcslp.pri. Class: 1 (Internet) Type: 28 (IPv6 Address) TTL (Time To Live): 86400 IPv6 Address: fe80::216:cbff:fe85:2d15 1 name server resource(s) Domain Name: mcslp.pri. Class: 1 (Internet) Type: 2 (Authoritative Name Server) TTL (Time To Live): 86400 Authoritative Name Server: narcissus.mcslp.pri. 1 additional record(s) Domain Name: narcissus.mcslp.pri. Class: 1 (Internet) Type: 1 (Address) TTL (Time To Live): 86400 Address: 192.168.0.110
You can see in this example that snoop has decoded the contents of the DNS protocol to show the raw data returned by the lookup. In Listing 7, you can see the packets transferred when accessing a Web site; in this case, it is the header information returned by the Web site but, by looking at other packets, you can also see the original request information and even the data returned.
Listing 7. Packets transferred when accessing a Web site

ETHER: ----- Ether Header ----ETHER: ETHER: Packet 10 arrived at 10:05:19.05636 ETHER: Packet size = 563 bytes ETHER: Destination = 0:2:11:90:15:31, ETHER: Source = 0:16:cb:85:2d:15, ETHER: Ethertype = 0800 (IP) ETHER: IP: ----- IP Header ----IP: IP: Version = 4 IP: Header length = 20 bytes IP: Type of service = 0x00 IP: xxx. .... = 0 (precedence) IP: ...0 .... = normal delay IP: .... 0... = normal throughput IP: .... .0.. = normal reliability IP: .... ..0. = not ECN capable transport IP: .... ...0 = no ECN congestion experienced IP: Total length = 549 bytes IP: Identification = 53771 IP: Flags = 0x4 IP: .1.. .... = do not fragment IP: ..0. .... = last fragment IP: Fragment offset = 0 bytes IP: Time to live = 64 seconds/hops IP: Protocol = 6 (TCP) IP: Header checksum = e4e9 IP: Source address = 192.168.0.110, narcissus.mcslp.pri IP: Destination address = 192.168.0.31, ultra3 IP: No options IP: TCP: ----- TCP Header ----TCP: TCP: Source port = 80 TCP: Destination port = 48591 TCP: Sequence number = 150680669 TCP: Acknowledgement number = 1315156336 TCP: Data offset = 20 bytes TCP: Flags = 0x18 TCP: 0... .... = No ECN congestion window reduced TCP: .0.. .... = No ECN echo TCP: ..0. .... = No urgent pointer TCP: ...1 .... = Acknowledgement TCP: .... 1... = Push TCP: .... .0.. = No reset TCP: .... ..0. = No Syn TCP: .... ...0 = No Fin TCP: Window = 65535 TCP: Checksum = 0x7ca4 TCP: Urgent pointer = 0 TCP: No options
TCP: HTTP: ----- HyperText Transfer Protocol ----HTTP: HTTP: HTTP/1.1 200 OK HTTP: Date: Mon, 15 Oct 2007 09:05:02 GMT HTTP: Server: Apache/2.2.4 (Unix) DAV/2 PHP/5.2.1 SVN/1.4.3 mod_ssl/2.2.4 OpenSSL/0.9.7l mod_auth_tkt/2.0.0rc2 mod_fastcgi/2.4.2 HTTP: Accept-Ranges: bytes HTTP: Connection: close HTTP: [...] HTTP:
Remember that the information exchanged is encrypted (for example, the Secure Sockets Layer (SSL) protocol used with HTTP and other protocols) so that the system cannot decrypt the data. Summary Knowing what is on your network and what exchanged information is on your network can go a long way to resolving a number of issues, including potential security issues and network performance problems. In this article, you've examined methods for scanning your network to find hosts and open ports and how to record and compare that information over time. As you've seen though, network scanning can only do so much. For real detail, you need to examine the content of the exchanged packets. You also learned that a variety of tools are available for the different platforms to get detailed information about the exchanged data. Using these tools collectively is a good habit to get into, as it ensures that you know what is on your network and what it is doing, and it helps you to isolate problems and potentially malicious or unauthorized hosts and practices.
Systems Administration Toolkit: Monitor user usage

Summary: Explore new ways to record UNIX logins and other system activities in a number of different logs, and take advantage of this information to monitor user usage. This can be helpful from a number of perspectives, either to use for chargeback
reporting or just to get an idea of how busy and active individual users are on the system to help when planning and allocating resources. View more content in this series Tags for this article: aix, aix_and_unix, chargeback, monitor, systems_admin, toolkit, unix
Date: 23 Oct 2007 Level: Intermediate Also available in: Chinese Russian Activity: 20729 views Comments: 1 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Getting current user information Getting a list of users currently using the UNIX system can be achieved in a number of different ways. The most obvious and straightforward is to use the who command. The who command returns a list of users currently logged in, the terminal they are connected to, the date they logged in and, if they are a remote user, the IP address of the hostname that they logged in from. You can see an example in Listing 1 below.
Listing 1. Using the who command to return a list of users currently logged in
$ who mc mcbrown
pts/2 pts/3
Sep 12 14:29 Sep 12 14:37
(sulaco.mcslp.pri) (nautilus.mcslp.pri)
On some systems, there is also a quicker format available using the -q command-line option, as shown here in Listing 2.
Listing 2. The who command with the -q option

$ who -q mc mcbrown # users=2
This output is similar to another command available on some systems called users, which just outputs the list of users with no count (see Listing 3).
Listing 3. The users command

$ users mc mcbrown
You can often find out more detailed information by adding the -a and -H command-line options. The -a option includes all of the most recent information from the /var/adm/utmpx file, which is used to record login information. The same file also records other events, such as the date and time of the boot and the last run level change. You'll be taking a closer look at the information stored in that file later in this article. The -H option adds header information to the columns of output. You can see an example of this in Listing 4.
Listing 4. The who command with the -a and -H options

$ who -aH NAME LINE . system boot . run-level 3 zsmon . LOGIN console LOGIN console mc + pts/2 (sulaco.mcslp.pri)
TIME Sep 12 Sep 12 Sep 12 Sep 12 Sep 12 Sep 12
IDLE 11:35 11:35 11:35 11:35 11:35 14:29 3 3:14 0:20 0:20 .
PID 0 215 221 510 569
COMMENTS S (:0)
mcbrown + pts/3 (nautilus.mcslp.pri)
Sep 12 14:37
0:12
675
The first two lines give you the information about when the system was last booted and the date and details of the last run level change. The machine shown here is running Solaris, and it monitors the serial ports for logins using the zsmon daemon. Then you have two logins tagged to the console, one of these is the login for the active user, and the other is attached to an X Server (the :0 indicates the X Windows System screen reference). The final two lines are remote logins using Secure Shell (SSH). For all active processes, you get a process ID, and that means you can identify the original user within the ps list by their ID. The LINE column is the terminal the user is connected on; the console is obviously the main keyboard and monitor of the machine. The pts lines refer to pseudo-terminals created automatically to handle the remote SSH connections. Of course, knowing who is on the local machine is interesting, but how about on other machines in your network? Getting remote user information There are two background services that provide remote user information, rusers and rwho, which are backed up by two daemons, rusersd (more commonly referred to as in.rusersd) and rwhod (also known as in.rwhod). These both rely on the Remote Procedure Call (RPC) protocol to share information across the network. Traditionally, enabling rwhod and rusersd was seen as a wasteful use of network resources, because these daemons broadcast information over the network continually, using up bandwidth. On a network with a large number of hosts, the amount of information generated can be considerable. Today, the quantity of information generated is small in comparison to the speed of the network. Neither rwhod or rusersd are available on all systems but, for those that provide the tool, it can be an easy way of getting the information you want. If the tools are not already running, you need to start up the in.rwhod and in.rusersd daemons. Once the daemons are running, you have a number of tools available for outputting the information about all of the machines. The rwho tool is the remote and network-enabled version of the who tool. See Listing 5 for an example.
Listing 5. The rwho tool
$ rwho -a mc solaris-desktop:console mc solaris-desktop:pts/1 mc ultra3:pts/2 mcbrown ultra3:pts/3 M
Sep Sep Sep Sep
12 12 12 12
11:29 11:32 14:29 14:37
3:41 :06 :37
The rwhod tool broadcasts information, and it captures the broadcast information from other hosts and collates it into the /var/spool/rwhod directory. The rusers tool sends a broadcast over the network for the rusersd daemon on the other hosts to return the information about the users currently logged in, as seen here in Listing 6.
Listing 6. The rusers tool

$ rusers Sending broadcast for rusersd protocol version 3... 192.168.0.31 mc mcbrown solaris-desktop. mc mc
Another tool available once the rwhod daemon is running is ruptime, which returns the loading and uptime information for all servers running rwhod in the local network. Listing 7 shows an example. This makes a really great tool for determining the status of your machines without having to log in to them individually.
Listing 7. The ruptime tool

$ ruptime solaris-desktop up ultra3 up
3:53, 3:46,
1 user, load 0.00, 0.00, 0.00 2 users, load 0.00, 0.00, 0.00
For who and rwho, the information about who that is currently connected to your machine is recorded in the /var/adm/utmp file, and historical information about users that have logged in (and logged out) is in the wtmp machine. Both of these files contain a wealth of information, but the files are not available in an immediate format. Let's take a closer look at these files and how you can display that information. Log files used for user activity There are a number of files used to record user logins and activity. The three main files for recording this information are:
utmpThis file records the information about users currently logged in. It should contain one record for each logged in user. wtmpThis file records all logins (and logouts) by users. On a busy system, this file is quite large, as it contains one login and one logout record. This file also contains some system-related log information, such as restart, shutdown, and date changes. lastlogThis file records the last login time for each user. This file contains only one record for each user.
Once you know how and when the files are written, the relationship between these files is quite straightforward. The basic sequence is as follows:
When a user logs in, the lastlog file is opened, and their record with the login date and time is updated. Then the utmp is opened and their current login information is recorded. The login record (generally a copy of the information added to lastlog) is written to the utmp log to record the login information. During logout, the login record written to utmp is deleted (since the user is no longer logged in), but a new record is written to wtmp to record the fact that the user has logged out. The format of these files is entirely binary; there is no way to report the information without using a separate tool that reads and parses the information. In general, the basic format of the wtmp and lastlog files is the same, and they use a simple format to record the login time, the login line, and the hostname line information, as shown here in Listing 8, which shows the Solaris format.
Listing 8. Solaris format from wtmp

struct lastlog { time_t ll_time; char ll_line[UT_LINESIZE]; char ll_host[UT_HOSTSIZE]; };
The format is not the same on different operating systems, and you have to be careful to extract the right information. On AIX, the utmp definition is shown in Listing 9.
Listing 9. utmp definition

struct utmp { char ut_user[8]; char ut_id[14] char ut_line[12]; short ut_type; pid_t ut_pid; struct exit_status {
short short }
e_termination; e_exit;
Listing 10 shows the definition of the value of the ut_type field from the record.
Listing 10. Definition of the value of the ut_type field

#define #define #define #define #define #define #define #define #define #define #define EMPTY RUN_LVL BOOT_TIME OLD_TIME NEW_TIME INIT_PROCESS LOGIN_PROCESS 0 1 2 3 4 5 6
USER_PROCESS 7 DEAD_PROCESS 8 ACCOUNTING 9 UTMAXTYPE ACCOUNTING
The username information is not included in the record because it is encoded according to the users ID. If the ID of the user that logs in is 1000, then the lastlog record can be located by accessing the record written at 1000 x sizeof (struct lastlog). The utmp file structure is similar but, because it is a sequential log of login and logout and other event entries, the record includes the login information in Listing 11.
Listing 11. utmp file structure definition

struct utmp { char char char time_t };
ut_line[UT_LINESIZE]; ut_name[UT_NAMESIZE]; ut_host[UT_HOSTSIZE]; ut_time;
The Linux ut_type has similar value definitions, as shown here in Listing 12.
Listing 12. Linux ut_type definition
#define #define #define #define #define #define #define #define #define #define
UT_UNKNOWN RUN_LVL BOOT_TIME NEW_TIME OLD_TIME INIT_PROCESS LOGIN_PROCESS USER_PROCESS DEAD_PROCESS ACCOUNTING
0 1 2 3 4 5 6 7 8 9
And a much more complex structure definition, as shown here in Listing 13.
Listing 13. More complex utmp structure definition

struct utmp { short ut_type; pid_t ut_pid; char ut_line[UT_LINESIZE]; char ut_id[4]; char ut_user[UT_NAMESIZE]; char ut_host[UT_HOSTSIZE]; struct exit_status ut_exit;
/* /* /* /* /* /* /*
type of login */ PID of login process */ device name of tty - "/dev/" */ init id or abbrev. ttyname */ user name */ hostname for remote login */ The exit status of a process marked as DEAD_PROCESS */
/* The ut_session and ut_tv fields must be the same size when compiled 32- and 64-bit. This allows data files and shared memory to be shared between 32- and 64-bit applications */ #if __WORDSIZE == 64 && defined __WORDSIZE_COMPAT32 int32_t ut_session; /* Session ID, used for windowing */ struct { int32_t tv_sec; /* Seconds */ int32_t tv_usec; /* Microseconds */ } ut_tv; /* Time entry was made */ #else long int ut_session; /* Session ID, used for windowing */ struct timeval ut_tv; /* Time entry was made */ #endif int32_t ut_addr_v6[4]; char __unused[20]; }; /* IP address of remote host */ /* Reserved for future use */
Most systems come with simple tools to extract the information for you. To dump out the contents of the wtmp file, use the last command. This simply dumps out all the information (see Listing 14).
Listing 14. Dumping out the contents of the wtmp file using the last command
$ last statmon statmon statmon statmon statmon statmon statmon
ftp ftp ftp ftp ftp ftp ftp
nautilus.mcslp.p narcissus.mcslp. nostromo.mcslp.p sulaco.mcslp.pri nautilus.mcslp.p nostromo.mcslp.p narcissus.mcslp.
Wed Wed Wed Wed Wed Wed Wed
Sep Sep Sep Sep Sep Sep Sep
12 12 12 12 12 12 12
15:50 15:50 15:50 15:49 15:45 15:45 15:45
15:50 15:50 15:50 15:49 15:45 15:45 15:45
(00:00) (00:00) (00:00) (00:00) (00:00) (00:00) (00:00)
You can often filter the information by user, host, or tty if you need to. For example, to get all of the logins by root, use what is shown in Listing 15.
Listing 15. Getting all logins by root

$ last root root console root console root console root console root console root console root console root console root console root console root console root console root console root console root console root console
:0 :0
:0
:0
Mon Mon Sat Sat Sun Sat Sat Fri Sun Thu Thu Thu Wed Wed Wed Wed
Sep Sep Sep Sep Sep Sep Sep Aug Aug Jul Jul Jul Jul Jul Jul Jul
25 25 9 9 3 2 2 25 20 20 20 20 19 19 19 19
11:32 11:27 13:17 10:47 06:52 14:24 08:02 17:16 16:04 07:23 07:22 02:57 12:22 12:10 12:05 11:47
11:32 (00:00) 11:27 (00:00) 13:28 (00:11) 13:14 (02:26) down (6+03:54) down (16:27) down (06:13) down (00:39) 16:04 (00:00) 07:31 (00:07) 07:23 (00:00) 02:57 (00:00) down (05:38) 12:19 (00:08) 12:09 (00:04) 11:55 (00:07)
wtmp begins Wed Jul 19 09:54
Because the wtmp also contains information about reboots and shutdowns, you can list that information, too (see Listing 16).
Listing 16. Getting information about reboots and shutdowns

$ last reboot reboot system reboot system reboot system reboot system reboot system reboot system
boot down boot down boot down
Wed Wed Mon Thu Thu Thu
Sep 12 11:28 Sep 5 12:16 Sep 3 13:03 Mar 1 11:33 Mar 1 09:57 Mar 1 08:12
reboot reboot reboot reboot reboot reboot reboot reboot reboot
system system system system system system system system system
boot down boot down boot down boot down boot
Thu Thu Thu Thu Thu Sun Sun Mon Mon
Mar Mar Mar Mar Mar Dec Dec Sep Sep
1 1 1 1 1 17 17 25 25
08:05 08:12 08:05 08:03 08:02 10:04 10:02 11:44 10:2
The previous tools are fine, but what if you want to parse the information yourself? You could write a simple C program to extract the information but, in fact, you can also do it with Perl by using the unpack() function. Parsing the binary log files Using the unpack function within Perl requires creating a suitable packstring, which defines the datatypes that you are going to extract from the binary record data that you actually read from the raw binary utmp file. Listing 17 shows a very simple Perl script that does a direct dump of the data in wtmp.
Listing 17. Perl script that dumps the data in wtmp

my $packstring = "a8a8a8ssssl"; my $reclength = length(pack($packstring)); my @ut_types = qw(EMPTY RUN_LVL BOOT_TIME OLD_TIME NEW_TIME INIT_PROCESS LOGIN_PROCESS USER_PROCESS DEAD_PROCESS ACCOUNTING); open(D,"</var/log/wtmp") or die "Couldn't open wtmp, $!"; while(sysread(D,my $rec,$reclength)) { my ($user,$userid,$line,$pid,$type,$eterm,$eexit,$time) = unpack($packstring,$rec); print("$user, $userid, $line, $pid, $ut_types[$type], ", "$eterm, $eexit, ", scalar localtime($time),"\n"); } close(D) or die "Couldn't close wtmp, $!";
Note that you have to read the data using sysread because you are reading raw binary packets (not lines). The packstring you use gets the username, userid, ttyline, PID, utmp type, terminal exit code, and time.
The first three fields are the important ones, as they contain the line, name, and host information. The size of these fields is conveniently defined in utmp.h (see Listing 18).
Listing 18. Size of the first three fields defined in utmp.h

#define UT_NAMESIZE #define UT_LINESIZE #define UT_HOSTSIZE 8 8 16
Note that you must get the size right, as it affects the information you read. If you have a Solaris or BSD system, it uses the simpler structure that was shown in Listing 8, so you need a different packstring and extracted fields, as shown here in Listing 19.
Listing 19. Data dump with Solaris or BSD system

my $packstring = "a8a8a16l"; my $reclength = length(pack($packstring)); my @ut_types = qw(EMPTY RUN_LVL BOOT_TIME OLD_TIME NEW_TIME INIT_PROCESS LOGIN_PROCESS USER_PROCESS DEAD_PROCESS ACCOUNTING); open(D,"</var/log/wtmp") or die "Couldn't open wtmp, $!"; while(sysread(D,my $rec,$reclength)) { my ($line,$name,$host,$time) = unpack($packstring,$rec); print("$line, $name, $host,", scalar localtime($time),"\n"); } close(D) or die "Couldn't close wtmp, $!";
Running this script gives you the login information for wtmp. The lines where you get only the line information are logouts rather than logins, as shown here in this fragment in Listing 20.
Listing 20. Getting login information for wtmp

ftp599, statmon, nautilus.mcslp.p,Wed Sep 12 16:00:13 2007 ftp599, , ,Wed Sep 12 16:00:14 2007 ftp4003, statmon, sulaco.mcslp.pri,Wed Sep 12 16:04:35 2007 ftp4003, , ,Wed Sep 12 16:04:35 2007 ftp4035, statmon, narcissus.mcslp.,Wed Sep 12 16:05:00 2007
ftp4035, ftp4037, ftp4037, ftp4057, ftp4057,
, ,Wed Sep 12 16:05:00 2007 statmon, nostromo.mcslp.p,Wed Sep 12 16:05:01 2007 , ,Wed Sep 12 16:05:02 2007 statmon, nautilus.mcslp.p,Wed Sep 12 16:05:14 2007 , ,Wed Sep 12 16:05:14 2007
Summary In this article, you've taken a quick look at what happens when you log in, how the logins are recorded into the UNIX system, and how you can use that information to determine who is logged on currently and who has been logged on in the past. You could use a modified version of the Perl script, for example, to provide total user-time information and charge it back to a user or department.
System Administration Toolkit: Distributed administration using SSH

Summary: Use Secure Shell (SSH) to run commands on remote UNIX systems and, with some simple scripts, put together a system that enables you to manage many systems simultaneously from one machine without having to log in directly to the machines themselves. Also examine the basics of a distributed management system and some scripts and solutions using the technique. View more content in this series Tags for this article: administrators, aix, aix_and_unix, commands, ssh, systems_admin, toolkit
Date: 14 Aug 2007 Level: Intermediate Also available in: Chinese Russian Activity: 59240 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Simplifying remote login Secure Shell (SSH) tools provide a secure method for logging in and exchanging information with a remote host. A number of different tools are provided, including the general-purpose SSH tool (which provides a remote terminal connection), SCP (a secure, host-to-host, copy solution), and SFTP, a secure file copy solution that works in a similar fashion to the standard FTP tools. All of these tools are secure in that the information that is exchanged is encrypted. In addition, the authentication of connections is secured using a public or private key mechanism. One of the main benefits of SSH is that you can bypass the normal login and password exchange by copying your public key to a remote machine. Although this is useful when using SSH to log in to a remote machine (as it means you don't have to provide a password), it is even more useful when performing remote administration. Having to type in a password can also make automated remote administration (for example, running a command through cron) impossible, because in an automated script, you won't be around to type in the password! When using SSH to run commands across multiple machines without exchanging your public key, you need to type in your password for each machine. A quick and simple way of setting this up is to create a public key:
$ ssh-keygen -t rsa
Follow the on-screen instructions, but don't set a password when prompted, as you will then need to enter the password each time you want to use the key. This creates a private and a public key file. Now you just need to append the contents of the public key file in .ssh/id_rsa.pub, and append it to the .ssh/authorized_keys file on the remote host
and user you want to use when logging in. You need to append the public key file contents to each machine you want to log in to automatically. Running a remote command There are many ways in which you can run a remote command. You can run a single remote command by adding the command you want to run to SSH after the login or host information. For example, to get the disk information for a remote host, you might use the command and get the output in Listing 1 below.
Listing 1. Running a simple command through SSH

$ ssh mc@gentoo.vm df Filesystem /dev/hda3 udev /dev/hdc1 /dev/hdc2 /dev/hdc3 shm 1K-blocks 14544820 128044 1968872 1968904 1968904 128044 Used Available Use% Mounted on 3611520 10194464 27% / 564 127480 1% /dev 50340 1818516 3% /var/tmp 1482220 386668 80% /usr/portage 35760 1833128 2% /home/build 0 128044 0% /dev/shm
Bear in mind that the sequence in Listing 1 requires you to enter a password if you haven't already exchanged your public key with the remote host. You can also execute a sequence of commands by separating each command with a semicolon and then placing the entire sequence of commands into quotes so that it is identified as a single argument. An example of executing both a disk check and an uptime check is shown in Listing 2.
Listing 2. Executing a disk and an uptime check

$ ssh mc@gentoo.vm "df;uptime" Filesystem 1K-blocks /dev/hda3 14544820 udev 128044 /dev/hdc1 1968872 /dev/hdc2 1968904 /dev/hdc3 1968904 shm 128044 14:31:27 up 12 min, 2 users,
Used Available Use% Mounted on 3611520 10194464 27% / 564 127480 1% /dev 50340 1818516 3% /var/tmp 1488100 380788 80% /usr/portage 35760 1833128 2% /home/build 0 128044 0% /dev/shm load average: 0.01, 0.05, 0.06
You can string as many commands as you like into this operation. Filtering, for example, using grep or other tools, is also possible, but you need to make sure to embed the entire remote command expression into the quotes (see Listing 3).
Listing 3. Filtering using grep

$ ssh root@gentoo.vm "cat Dec 17 18:05:37 localhost denied Dec 17 18:05:37 localhost Dec 17 18:05:37 localhost Dec 17 18:06:31 localhost denied Dec 17 18:06:31 localhost Dec 17 18:06:31 localhost Dec 17 18:06:40 localhost denied Dec 17 18:06:40 localhost ... /var/log/messages|grep 'su\['" su[19218]: pam_authenticate: Permission su[19218]: FAILED su for root by mc su[19218]: - pts/1 mc:root su[19221]: pam_authenticate: Permission su[19221]: FAILED su for root by mc su[19221]: - pts/1 mc:root su[19222]: pam_authenticate: Permission su[19222]: FAILED su for root by mc
The first item to note about Listing 3 is that you are logging in directly to the remote machine as root. This is because the file you want to view is only accessible to the superuser. You must ensure that your system is configured to allow remote root logins for this to work. The second important note about this example is that you've performed the grep operation remotely. In actual fact, you don't need to do this. The standard input and output of the remote host are replicated to the local machine, so the output from the command can be filtered locally, as shown here in Listing 4.
Listing 4. Output filtered locally

$ ssh root@gentoo.vm "cat Dec 17 18:05:37 localhost denied Dec 17 18:05:37 localhost Dec 17 18:05:37 localhost Dec 17 18:06:31 localhost denied Dec 17 18:06:31 localhost Dec 17 18:06:31 localhost Dec 17 18:06:40 localhost denied Dec 17 18:06:40 localhost Dec 17 18:06:40 localhost /var/log/messages" | grep 'su\[' su[19218]: pam_authenticate: Permission su[19218]: FAILED su for root by mc su[19218]: - pts/1 mc:root su[19221]: pam_authenticate: Permission su[19221]: FAILED su for root by mc su[19221]: - pts/1 mc:root su[19222]: pam_authenticate: Permission su[19222]: FAILED su for root by mc su[19222]: - pts/1 mc:root
Of course, the effect is essentially the same. Using the remote pipe method, though, is useful when the information or command that you want to pipe with is remote. For example, you can use ls in combination with du to determine the disk usage of different directories with the command shown in Listing 5.
Listing 5. Determining disk usage of different commands

ssh root@gentoo.vm "ls -d /usr/local/* |xargs du -sh " Password: 4.0K /usr/local/bin 4.0K /usr/local/games 4.0K /usr/local/lib 0 /usr/local/man 4.0K /usr/local/sbin 12K /usr/local/share 4.0K /usr/local/src
Before moving on to redistributing these techniques to multiple machines, there's a quick trick for running remote interactive sessions directly without having to log in first. Direct interactive sessions As shown previously, you can directly run a number of different commands and chains of commands. One of the benefits of the SSH solution is that although the command itself is executed remotely, the input and output of the command are sourced from the calling machine. You can use this as a method for exchanging information between the two machines relating to the commands that you want to execute. The commands that you execute can cover almost anything from a range of different commands. However, because you are running commands directly from the command line, there are limits to what you can execute directly with this method. For example, trying to edit a remote file with an editor using this method and techniques shown above usually fail (see Listing 6).
Listing 6. Edit a remote file fails

$ ssh root@gentoo.vm "emacs /etc/amavisd.conf" emacs: standard input is not a tty
You can resolve this by forcing SSH to allocate a pseudo-tty device so that you can interact directly with the remote application.
Running a remote command across multiple machines So far, you have concentrated on running a single command or command string on a single remote machine. Although the interactive session trick is useful when performing remote administration directly with SSH, it is likely that you will want to automate the process, which means that the interactive element is unlikely to be of very much use. To run the same command remotely across a number of machines, you need to build a simple wrapper around the SSH command and the remote command that you want to run so that the process is repeated on each remote machine. You can do this with a very simple for loop, as demonstrated in Listing 7 below.
Listing 7. for loop to run the command remotely

for remote in mc@gentoo.vm mc@redhat; do echo $remote; ssh $remote 'df -h'; done mc@gentoo.vm Filesystem Size Used Avail Use% Mounted on /dev/hda3 14G 4.1G 9.2G 31% / udev 126M 564K 125M 1% /dev /dev/hdc1 1.9G 56M 1.8G 4% /var/tmp /dev/hdc2 1.9G 1.3G 558M 70% /usr/portage /dev/hdc3 1.9G 35M 1.8G 2% /home/build shm 126M 0 126M 0% /dev/shm mc@redhat Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 7.1G 5.5G 1.3G 82% / /dev/hda1 99M 13M 82M 14% /boot none 125M 0 125M 0% /dev/shm
You can easily turn this into a simple script, as shown here in Listing 8.
Listing 8. Reducing the for loop to simple command

#!/bin/bash # Script to run a command across multiple machines # Global options TIMEOUT=10 ERRLOG=/tmp/remote-err-$$.log OUTLOG=/tmp/remote-out-$$.log
# Extract the command line MACHINES=$1;shift COMMAND=$1;shift for machine in $MACHINES do echo $machine ssh -oConnectTimeout=$TIMEOUT $machine $COMMAND >>$OUTLOG 2 >>$ERRLOG done cat $OUTLOG cat $ERRLOG >&2 rm -f $OUTLOG $ERRLOG
The MACHINES and COMMAND are "as-is" as you extract them from the command line. When using the script, you must put the user or host combinations and the command into double quotes to ensure they are identified as a single argument. The only other addition is the TIMEOUT option. This sets the ConnectTimout option to SSH to ensure that when running a command you don't needlessly wait to connect to a host that might not be available. The default is set at the head of the script and should ensure you don't wait too long. When running the commands, you send the output to a couple of log files, one for standard output and the other for standard error. Then you output these individually to the appropriate location. This highlights one of the benefits of SSHthe remote machine redirects to the same location (standard output, standard error), so you can redirect locally while retaining the meaning of the output. For example, you can repeat the df check using this script:
$ runremote.sh "gentoo redhat" "df -h"
Because you redirected the standard output and error, you can even generate a log of the whole process:
$ runremote.sh "gentoo redhat" "df -h" 2>/tmp/error.log
Using remote execution for performance monitoring When using runremote.sh, you might want to play with the exact value used for the timeout value, and you might even want to change the value, depending on what you are doing. For example, if you were using this script to get a snapshot of the current
status by using uptime across a bunch of machines, you wouldn't want to wait too long for the connection and command to take place, otherwise the snapshot would be imprecise. Also, the script, as it stands, runs the command sequentially. Not only does this take a long time if you have a large number of machines, but the time delay between the first machine and the last executing the chosen command might be so significant that correlation across machines might be impossible. A slightly adjusted script, runremote2.sh, is shown in Listing 9. This executes the remote command almost simultaneously (by running it in the background), and then also pipes the output to individual log files.
Listing 9. Script that executes the remote command almost simultaneously

#!/bin/bash # Script to run a command across multiple machines # Global options TIMEOUT=10 ERRLOG=/tmp/remote-err-$$.log OUTLOG=/tmp/remote-out-$$.log # Extract the command line MACHINES=$1;shift COMMAND=$1;shift for machine in $MACHINES do echo $machine >>$OUTLOG.$machine ssh -oConnectTimeout=$TIMEOUT $machine $COMMAND >>$OUTLOG.$machine 2>>$ERRLOG.$machine & done # Wait for children to finish wait cat $OUTLOG.* cat $ERRLOG.* >&2 rm -f $OUTLOG.* $ERRLOG.*
In this script, you also echo the machine name out to the command log (unique for each machine supplied). To ensure that the script doesn't exit before all the remote commands have executed, you need to add a wait command to wait for the children of the script to finish.
Now you can use the script to check multiple machines simultaneously (see Listing 10).
Listing 10. Using the script to check multiple machines simultaneously

$ runremote2.sh "narcissus gentoo.vm droopy@nostromo mcbrown@nautilus" 'uptime' droopy@nostromo 19:15 up 9 days, 23:42, 1 user, load averages: 0.01 0.03 0.00 gentoo.vm 18:10:23 up 1 day, 10:02, 2 users, load average: 1.72, 1.84, 1.79 mcbrown@nautilus 19:15 up 10:08, 4 users, load averages: 0.40 0.37 0.29 narcissus 19:15 up 8 days, 7:04, 4 users, load averages: 0.53 0.54 0.57
This kind of monitoring can be useful when you want to get a whole network picture for example, to check a problem with a group or cluster of machines when running Web or database services and want to identify potential spikes or issues simultaneously across that group of machines. Be aware, however, that there will still be delays, especially if a machine is particularly busythe time for the connection to be made and the command to be executed could leave some significant time delays across different machines. Running the same operation across multiple machines Creating users across a number of machines can be a pain. There are obviously plenty of solutions for trying to resolve the difficulty from the use of single-sign on utilities, such as the Network Information Service (NIS) or LDAP-based solutions, but you don't always have to synchronize the users in this way. You could use SSH to do this for you by running the adduser command across multiple machines. But under Solaris, the name of the command is useradd. The command-line options are largely the same, so you could use run-remote.sh twice (see Listing 11).
Listing 11. Running run-remote.sh twice

$ runremote.sh "gentoo redhat" "adduser -u 1000 -G sales,marketing mcbrown" $ runremote.sh "solaris solaris-x86" "useradd -u 1000 -G sales,marketing mcbrown"
You've now created the same user across a number of machines with the same groups and the same user ID, but this is hardly practical. A much better way would be to use the tips demonstrated in the "System Administration Toolkit: Standardizing your UNIX command-line tools" article (see Resources) to use the same command across multiple machines:
$ runremote.sh "gentoo solaris" "adduser.sh -u 1000 -G sales,marketing mcbrown"
Summary In this article, you've examined a simple, but powerful, method to run commands on a remote machine. Although the basics of the process are straightforward, you can also create additional functionality to complete some robust, automated remote administration tasks (for example, the ability to redirect and pipe remote local input together). By implementing some simple shell script tricks, you can even use the system to remotely administer a number of machines simultaneously, simplifying many of the repetitive tasks and performance monitoring.
System Administration Toolkit: Build intelligent, unattended scripts

Summary: Look at how to create scripts that are able to record their output, trap and identify errors, and recover from errors and problems so that they either run correctly or fail with a suitable error message and report. Building scripts and running them automatically is a task that every good administrator has to handle, but how do you handle the error output and make intelligent decisions about how the script should handle these errors? This article addresses these issues. View more content in this series Tags for this article: aix, aix_and_unix, errors, logging, scipts, systems_admin, toolkit, unattended_scripts
Date: 03 Jul 2007 Level: Intermediate Also available in: Chinese Russian Activity: 37347 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. The unattended script problem There are many issues around executing unattended scriptsthat is, scripts that you run either automatically through a service like cron or at commands. The default mode of cron and at commands, for example, is for the output of the script to be captured and then emailed to the user that ran the script. You don't always want the user to get the email that cron sends by default (especially if everything ran fine) sometimes the user who ran the script and the person actually responsible for monitoring that output are different. Therefore, you need better methods for trapping and identifying errors within the script, better methods for communicating problems, and optional successes to the appropriate person. Getting the scripts set up correctly is vital; you need to ensure that the script is configured in such a way that it's easy to maintain and that the script runs effectively. You also need to be able to trap errors and output from programs and ensure the security and validity of the environment in which the script executes. Read along to find out how to do all of this. Setting up the environment Before getting into the uses of unattended scripts, you need to make sure that you have set up your environment properly. There are various elements that need to be explicitly
configured as part of your script, and taking the time to do this not only ensures that your script runs properly, but it also makes the script easier to maintain. Some things you might need to think about include:

Search path for applications Search path for libraries Directory locations Creating directories or paths Common files
Some of these elements are straightforward enough to organize. For example, you can set the path using the following in most Bourne-compatible shells (sh, Bash, ksh, and zsh):
PATH=/usr/bin:/bin:/usr/sbin
For directory and file locations, just set a variable at the header of the script. You can then use the variable in each place where you would have used the filename. For example, when writing to a log file, you might use Listing 1.
Listing 1. Writing a log file

LOGFILE=/tmp/output.log do_something >>$LOGFILE do_another >>$LOGFILE
By setting the name once and then using the variable, you ensure that you don't get the filename wrong, and if you need to change the filename name, you only need to change the name once. Using a single filename and variable also makes it very easy to create a complex filename. For example, adding a date to your log filename is made easier by using the date command with a format specification:
DATE='date +%Y%m%d.%H%M'
The above command creates a string containing the date in the format YYYYMMDD.HHMM, for example, 20070524.2359. You can insert that date variable into a filename so that your log file is tagged according to the date it was created.
If you are not using a date/time unique identifier in the log filename, it's a good idea to insert some other unique identifier in case two scripts are run simultaneously. If your script is writing to the same file from two different processes, you will end up either with corrupted information or missing information. All shells support a unique shell ID, based on the shell process ID, and are accessible through the special $$ variable name. By using a global log variable, you can easily create a unique file to be used for logging:
LOGFILE=/tmp/$$.err
You can also apply the same global variable principles to directories:
LOGDIR=/var/log/my_app
To ensure that the directories are created, use the -p option for mkdir to create the entire path of the directory you want to use:
mkdir -p $LOGDIR
Fortunately, this format won't complain if the directories already exist, which makes it ideal for running in an unattended script. Finally, it is generally a good idea to use full path names rather than localized paths in your unattended scripts so that you can use the previous principles together.
Listing 2. Using full path names in unattended scripts

DATE='date +%Y%m%d.%H%M' LOGDIR=/usr/local/mcslp/logs/rsynclog mkdir -p $LOGDIR LOGNAME=$LOGDIR/$DATE.log
Now that you've set up the environment, let's look at how you can use these principles to help with the general, unattended scripts. Writing a log file Probably the simplest improvement you can make to your scripts is to write the output from your script to a log file. You might not think this is necessary, but the default
operation of cron is to save the output from the script or command that was executed, and then email it to the user who owned the crontab or at job. This is less than perfect for a number of reasons. First of all, the configured user that might be running the script might not be the same as the real person that needs to handle the output. You might be running the script as root, even though the output of the script or command when run needs to go to somebody else. Setting up a general filter or redirection won't work if you want to send the output of different commands to different users. The second reason is a more fundamental one. Unless something goes wrong, it's not necessary to receive the output from a script . The cron daemon sends you the output from stdout and stderr, which means that you get a copy of the output, even if the script executed successfully. The final reason is about the management and organization of the information and output generated. Email is not always an efficient way of recording and tracking the output from the scripts that are run automatically. Maybe you just want to keep an archive of the log file that was a success or email a copy of the error log in the event of a problem. Writing out to a log file can be handled in a number of different ways. The most straightforward way is to redirect output to a file for each command (see Listing 3).
Listing 3. Redirecting output to a file

cd /shared rsync --delete --recursive . /backups/shared >$LOGFILE
If you want to combine error and standard output into a single file, use numbered redirection (see Listing 4).
Listing 4. Combining error and standard output into a single file

cd /shared rsync --delete --recursive . /backups/shared >$LOGFILE 2>&1
Listing 4 writes out the information to the same log file. You might also want to write out the information to separate files (see Listing 5).
Listing 5. Writing out information to separate files

cd /shared rsync --delete --recursive . /backups/shared >$LOGFILE 2>$ERRFILE
For multiple commands, the redirections can get complex and repetitive. You must ensure, for example, that you are appending, not overwriting, information to the log file (see Listing 6).
Listing 6. Appending information to the log file

cd /etc rsync --delete --recursive . /backups/etc >>$LOGFILE >>$ERRFILE
A simpler solution, if your shell supports it, is to use an inline block for a group of commands, and then to redirect the output from the block as a whole. The result is that you can rewrite the lines in Listing 7 using the structure in Listing 8.
Listing 7. Logging in long form

cd /shared rsync --delete --recursive . /backups/shared >$LOGFILE 2>$ERRFILE cd /etc rsync --delete --recursive . /backups/etc >>$LOGFILE 2>>$ERRFILE
Listing 8 shows an inline block for grouping commands.
Listing 8. Logging using a block

{ cd /shared rsync --delete --recursive . /backups/shared cd /etc rsync --delete --recursive . /backups/etc } >$LOGFILE 2>$ERRFILE
The enclosing braces imply a subshell so that all the commands in the block are executed as if part of a separate process (although no secondary shell is created, the enclosing block is just treated as a different logical environment). Using the subshell, you can collectively redirect their standard and error output for the entire block instead of for each individual command. Trapping errors and reporting them One of the main advantages of the subshell is that you can place a wrapper around the main content of the script, redirect the errors, and then send a formatted email with the status of the script execution. For example, Listing 9 shows a more complete script that sets up the environment, executes the actual commands and bulk of the process, traps the output, and then sends an email with the output and error information.
Listing 9. Using a subshell for emailing a more useful log

LOGFILE=/tmp/$$.log ERRFILE=/tmp/$$.err ERRORFMT=/tmp/$$.fmt { set -e cd /shared rsync --delete --recursive . /backups/shared cd /etc rsync --delete --recursive . /backups/etc } >$LOGFILE 2>$ERRFILE { echo "Reported output" echo cat /tmp/$$.log echo "Error output" echo cat /tmp/$$.err } >$ERRORFMT 2>&1 mailx -s 'Log output for backup' root <$ERRORFMT rm -f $LOGFILE $ERRFILE $ERRORFMT
If you use the subshell trick and your shell supports shell options (Bash, ksh, and zsh), then you might want to optionally set some shell options to ensure that the block is
terminated correctly on an error. For example, the -e (errexit) option within Bash ensures that the shell terminates when a simple command (for example, any external command called through the script) causes immediate termination of the shell. In Listing 9, for example, if the first rsync failed, then the subshell would just continue and run the next command. However, there are times when you want to stop the moment a command fails because continuing could be more damaging. By setting errexit, the subshell immediately terminates when the first command stops. Setting options and ensuring security Another issue with automated scripts is ensuring the security of the script and, in particular, ensuring that script does not fail because of bad configuration. You can use shell options for this process. Other options you might want to set in a shell-independent manner (and the richer the shell, the better, as a rule, at trapping these instances). In the Bash shell, for example, u ensures that any unset variables are treated as an error. This can be useful to ensure that an unattended script does not try to execute when a required variable has not been configured correctly. The -C option (noclobber) ensures that files are not overwritten if they already exist, and it can prevent the script from overwriting files it shouldn't have access too (for example, the system files), unless the script has the correct commands to delete the original file first. Each of these options can be set using the set command (see Listing 10).
Listing 10. Using the set command to set options

set -e set -C
You can use a plus sign before the option to disable it. Another area where you might want to improve the security and environment of your script is to use resource limits. Resource limits can be set by the ulimit command, which is generally specific to the shell, and enable you to limit the size of files, cores, memory use, and even the duration of the script to ensure that the script does not run away with itself. For example, you can set CPU time in seconds using the following command:
ulimit -t 600
Although ulimit does not offer complete protection, it helps in those scripts where the potential for the script to run away with itself, or a program to suddenly use a large amount of memory, might become a problem. Capturing faults You have already seen how to trap errors, output, and create logs that can be emailed to the appropriate person when they occur, but what if you want to be more specific about the errors and responses? Two tools are useful here. The first is the return status from a command, and the second is the trap command within your shell. The return status from a command can be used to identify whether a particular command ran correctly, or whether it generated some sort of error. The exact meaning for a specific return status code is unique to a particular command (check the man pages), but a generally accepted principle is that an error code of zero means that the command executed correctly. For example, imagine that you want to trap an error when trying to create a directory. You can check the $? variable after mkdir and then email the output, as shown in Listing 11.
Listing 11. Trapping return status

ERRLOG=/tmp/$$.err mkdir /tmp 2>>$ERRLOG if [ $? -ne 0 ] then mailx -s "Script failed when making directory" admin <$ERRLOG exit 1 fi
Incidentally, you can use the return status code information inline by chaining commands with the && or || symbols to act as an and, or, or type statement. For example, say you want to ensure that the directory gets created and the command gets executed but, if the directory is not created, the command does not get executed. You could do that using an if statement (see Listing 12).
Listing 12. Ensuring that a directory is created before executing a command
mkdir /tmp/out if [ $? -eq 0 ] then do_something fi
You can modify Listing 12 into a single line:

mkdir /tmp/out && do_something
The above statement basically reads, "Make a directory and, if it completes successfully, also run the command." In essence, only do the second command if the first completes correctly. The || symbol works in the opposite way; if the first command does not complete successfully, then execute the second. This can be useful for trapping situations where a command would raise an error, but instead provides an alternative solution. For example, when changing to a directory, you might use the line:
cd /tmp/out || mkdir /tmp/out
This line of code tries to change the directory and, if it fails, (probably because the directory does not exist), you make it. Furthermore, you can combine these statements together. In the previous example, of course, what you want to do is change to the directory, or create it and then change to that directory if it doesn't already exist. You can write that in one line as:
cd /tmp/out || mkdir /tmp/out && cd /tmp/out
The trap command is a more generalized solution for trapping more serious errors based on the signals raised when a command fails, such as core dump, memory error, or when a command has been forcibly terminated by a kill command. To use trap, you specify the command or function to be executed when the signal is trapped, and the signal number or numbers that you want to trap, as shown here in Listing 13.
Listing 13. Trapping signals

function catch_trap {
echo "killed" mailx -s "Signal trapped" admin } trap catch_trap 1 2 3 4 5 6 7 8 9 10 11 sleep 9000
You can trap any signal in this way and it can be a good way of ensuring that a program that crashes out is caught and trapped effectively and reported. Identifying reportable errors Throughout this article, you've looked at ways of trapping errors, saving the output, and recording issues so that they can be dealt with and reported. However, what if the script or commands that you are using naturally output error information that you want to be able to use and report on but that you don't always want to know about? There is no easy solution to this problem, but you can use a combination of the techniques shown in this article to log errors and information, read or filter the information, and mail and report or display it accordingly. A simple way to do this is to choose which parts of the command that you output and report to the logs. Alternatively, you can post-process the logs to select or filter out the output that you need. For example, say you have a script that builds a document in the background using the Formatting Objects Processor (FOP) system from Apache to generate a PDF version of the document. Unfortunately in the process, a number of errors are generated about hyphenation. These are errors that you know about, but they don't affect the output quality. In the script that generates the file, just filter out these lines from the error log:
sed -e '/hyphenation/d' <error.log >mailerror.log
If there were no other errors, the mailerror.log file will be empty, and email is sent with the error information. Summary In this article, you've looked at how to run commands in an unattended script, captured their output, and monitored the execution of different commands in the script. You can log the information in many ways, for example, on a command-by-command or global basis, and check and report on the progress. For error trapping, you can monitor output and result codes, and you can even set up global traps that identify problems and trap them during execution for reporting
purposes. The result is a range of options that handle and report problems for scripts that are running on their own and where their ability to recover from errors and problems is critical.
System Administration Toolkit: Set up remote access in UNIX through OpenSSH

Martin Brown (mc@mcslp.com), Freelance Writer, Author
Summary: Use OpenSSH to provide a secure environment for running a remote terminal. The basics of OpenSSH and terminal usage are quite simple but, in this article, examine additional elements that allow automatic login to remote hosts, methods for running remote applications, and how to securely copy files between hosts. View more content in this series Tags for this article: aix, aix_and_unix, install, openssh, remote_access, system, systems_admin, toolkit
Date: 13 Feb 2007 Level: Intermediate Also available in: Chinese Russian Activity: 23099 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of
this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Back to top Why use OpenSSH? The standard networking services that you use every day, such as FTP, Telnet, RCP, remote shell (rsh), and so forth, are fine within a closed environment, but the information that you transfer over the network with any of these services is not encrypted. Anybody with a packet sniffer on your network or on a remote machine can view the information as it is exchanged, sometimes even password information. Furthermore, with all of these services, the options for auto-login during the process are limited, and often rely on embedding the plain text password into the command line to execute a statement, making the process even more insecure. The Secure Shell (SSH) protocol was developed to get around these limitations. SSH provides for encryption of the entire communication channel, including the login and password credential exchange, and it can be used with public and private keys to provide automatic authentication for logins. You can also use SSH as an underlying transport protocol. Using SSH in this way means that once you have opened a secure connection, the encrypted channel can exchange all types of information, even HTTP and SMTP, using the same, secure communication mechanism. OpenSSH is a free implementation of the SSH 1 and SSH 2 protocols. It was originally developed as part of the OpenBSD (Berkeley Software Distribution) operating system and is now released as a generic solution for UNIX or Linux and similar operating systems. Back to top Installing OpenSSH OpenSSH is free software and can be downloaded from the main OpenSSH Web site (see Resources). The OpenSSH system can be built from source code on a range of systems including Linux, HP-UX, AIX, Solaris, Mac OS X, and many others. You can often find a precompiled binary for your chosen platform and version. Some vendors even provide the OpenSSH kit as part of the operating system. To build, you need the following:

C compiler (GNU C compiler (gcc) or similar) Zlib -- the compression library OpenSSL -- the Secure Sockets Layer (SSL) security library
If you want to work with the default configuration settings, then you use the common build sequence, as shown in Listing 1 below.
Listing 1. Using the common build sequence

$ ./configure $ make $ make install
This installs the binaries, libraries, and configuration files into the /usr/local directory, for example, /usr/local/bin for binary and /usr/local/etc for configuration files. If you want to integrate the tools into your main environment, then you might want to specify the -prefix option, which sets the base directory, and the --sysconfdir option, which sets the location of the configuration files:
$ ./configure --prefix=/usr --sysconfidir=/etc/ssh
Some other common options you might specify include:

--with-tcp-wrappers -- This option is required if you want integration with the TCP wrapper
security system.
--with-ssl-dir=DIR -- This option specifies the location of the OpenSSL libraries. --with-pid-dir=DIR -- This option specifies the location of the PID file that stores the
process ID for the sshd daemon. --with-xauth=DIR -- This option specifies the location of the xauth command used for X authentication.
Once configured, build as normal with make. With the build and installation process completed, you need to configure your system, starting with creating the SSH keys that uniquely identify your system and enable secure communication between clients and your host. You can run:
$ make host-key
Alternatively, you can perform the steps manually on the command line. You need to create three keys (one for each of the main encryption algorithms: rsa1, rsa, and dsa). For example, Listing 2 shows you how to create an rsa1 key.
Listing 2. Creating an rsa1 key
$ ssh-keygen -t rsa1 -f /etc/ssh/ssh_host_key Generating public/private rsa1 key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /etc/ssh//ssh_host_key. Your public key has been saved in /etc/ssh//ssh_host_key.pub. The key fingerprint is: 43:aa:58:3c:d8:30:de:43:af:66:2a:b2:8d:02:08:86 root@remotehost
The system prompts you for a passphrase. For host keys, you probably don't want a passphrase for the key, so you can just press Return to use a blank (empty) passphrase. Alternatively, you can speed up the process by using the -N option on the command line (see Listing 3).
Listing 3. Using the -N option to speed up the process

$ ssh-keygen -t rsa1 -f /etc/ssh/ssh_host_key -N "" Generating public/private rsa1 key pair. Your identification has been saved in /etc/ssh/ssh_host_key. Your public key has been saved in /etc/ssh/ssh_host_key.pub. The key fingerprint is: a3:e3:21:4f:b5:9f:ff:05:46:66:bc:36:a1:47:a0:64 root@remotehost
Now repeat the process to create the rsa and dsa keys (see Listing 4).
Listing 4. Creating the rsa and dsa keys

$ ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N "" $ ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N ""
Two files are created for each key type: the public key (in the file ending .pub) and the private key. You should ensure that the private keys are readable only by root and the SSH processes -- this should be configured automatically. You might want to copy the public keys to a central location on a Network File System (NFS) share to enable people to add them to the known host keys list. Finally, you need to start the sshd process and configure it to execute at boot time. For Linux hosts, you can find a suitable init script that you can add to /etc/init.d in contrib/redhat/sshd.init. Back to top
Using SSH for basic terminal access The primary role for OpenSSH is the SSH tool, which provides a secure alternative to the Telnet protocol for logging into a UNIX or Linux host remotely. To connect to a remote host using the standard shell, you can just type the name of the host:
$ ssh remotehost
The system, by default, tries to use your current username as the login name. To use a different login, prefix the hostname and separate the two with the @ sign. For example:
$ ssh mc@remotehost
The system prompts you for the user's password -- this is like Telnet. The first time you connect to a host, the system asks if you want to keep a copy of the remote hosts public key in your 'known hosts' file (see Listing 5).
Listing 5. Known hosts file

$ ssh root@remotehost The authenticity of host 'remotehost (10.211.55.3)' can't be established. RSA key fingerprint is cc:c8:8b:75:3d:b6:00:2f:a9:9c:53:4c:03:0f:3d:1b. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'remotehost' (RSA) to the list of known hosts.
In the future, you won't get this warning, but a warning is sent if the system detects a difference in the public key returned by the remote host and the public key that you have on file for that host, as this might indicate a potential hacking attempt. It can also mean that the administrator has simply regenerated the host keys. In essence, there are no differences between an SSH and a Telnet session, except that the SSH session is encrypted, making it almost impossible for somebody to snoop on your session and determine your password or the commands and operations you are executing. You can also use SSH to run a command directly on a remote host without having to use the shell. For example, see Listing 6 to run the who command on a remote host.
Listing 6. Running the who command on a remote host

$ ssh mc@remotehost who admin console Nov 23 mc ttyp1 Dec 2 mc ttyp2 Dec 10 admin ttyp3 Dec 12 mc ttyp4 Dec 15
14:04 10:53 (sulaco.mcslp.pri) 06:50 (sulaco.mcslp.pri) 13:33 12:38 (nautilus.mcslp.p)
Remote execution also emulates the standard input, output, and error of the client host. This means that you can redirect output to a remote command. For example, you can append information directly to a remote file by piping the output from the command to an SSH on a remote host (see Listing 7).
Listing 7. Appending information directly to a remote file

$ echo "Hello World" |ssh mc@remotehost 'cat >> helloworlds.txt'
You can use this to your advantage when you simplify the login process when using SSH. Back to top Using SFTP to exchange files The sft command is an FTP-like alternative that uses the secure communication channel provided by the SSH protocol. To open an SFTP connection, specify the hostname on the command line:
$ sftp remotehost
Remember, the previous command assumes you want to use the same login as on the current host. To use a different login, prefix the username to the hostname:
$ sftp mc@remotehost
Although SFTP works in a similar fashion to FTP, there are some limitations and differences. For example, dir in FTP provides a long listing of files (see Listing 8).
Listing 8. dir provides a long listing of files in FTP

ftp> dir 502 'EPSV': command not understood. 227 Entering Passive Mode (192,168,0,110,150,159) 150 Opening ASCII mode data connection for directory listing. total 1472 drwx-----3 mc staff 102 Nov 4 11:17 Desktop drwx-----3 mc staff 102 Nov 4 11:17 Documents drwx------ 18 mc staff 612 Nov 5 18:01 Library drwx-----3 mc staff 102 Nov 4 11:17 Movies drwx-----3 mc staff 102 Nov 4 11:17 Music drwx-----4 mc staff 136 Nov 4 11:17 Pictures drwxr-xr-x 4 mc staff 136 Nov 4 11:17 Public drwxr-xr-x 6 mc staff 204 Nov 4 11:17 Sites drwxrwxrwx 3 root staff 102 Dec 24 07:30 tmp drwxr-xr-x 7 root staff 238 Dec 11 08:39 trial 226 Transfer complete.
Within SFTP, dir works as an alias for the hosts directory listing command, which on UNIX or Linux is ls. By default, dir provides only a short listing (see Listing 9).
Listing 9. dir provides only a short listing in SFTP

sftp> dir Desktop Public Sites
Documents tmp
Library trial
Movies
Music
Pictures
For a long listing, use the same options as you would with ls (see Listing 10).
Listing 10. Long listing

sftp> dir -l drwx-----3 mc drwx-----3 mc drwx-----18 mc drwx-----3 mc drwx-----3 mc drwx-----4 mc drwxr-xr-x 4 mc drwxr-xr-x 6 mc drwxrwxrwx 3 root drwxr-xr-x 7 root
staff staff staff staff staff staff staff staff staff staff
102 102 612 102 102 136 136 204 102 238
Nov 4 11:17 Nov 4 11:17 Nov 5 18:01 Nov 4 11:17 Nov 4 11:17 Nov 4 11:17 Nov 4 11:17 Nov 4 11:17 Dec 24 07:30 Dec 11 08:39
Desktop Documents Library Movies Music Pictures Public Sites tmp trial
Other commands, changing directories (cd and lcd for local), creating directories (mkdir), and sending (put) and receiving (get) files remain the same. For the latter commands, put and get both accept wildcards (like mput and mget in FTP), but be careful when transferring multiple files without a wildcard within SFTP. For example, sftp> mget file1 file2 file3 is identified as an attempt to obtain file1 and file2 and place them into the local directory file3, which probably won't exist. Back to top Using scp to copy files between hosts The scp command works like the rc command, except that files are transferred using the SSH protocol. This makes scp a much better option when transferring contentsensitive files, or when exchanging files automatically over the Internet. The format is similar to rcp; you specify the file paths copy between incorporating the hostname, if necessary, into the path. For example, to copy the .bashrc file from remotehost to the local machine, use:
$ scp remotehost:/users/mc/.bashrc ~/.bashrc
As before, to specify the username to use, prefix the host with the username and the @ sign:
$ scp mc@remotehost:/users/mc/.bashrc ~/.bashrc
The ~ meta-character should also work to access information from a home directory, providing the user you are using to connect has rights to read it:
$ scp mc@remotehost:~mc/.bashrc ~/.bashrc
To copy from the home directory of the user you are logging in as, use:
$ scp mc@remotehost:.bashrc ~/.bashrc
The scp command also supports standard expansion rules. So to copy all of the .bash* files, you might use:
$ scp mc@remotehost:.bash* ~
Or you can be even more specific and select individual files using brace ({}) expansion:
$ scp mc@remotehost:".bash{rc,_path,_aliases,_vars}" ~
Note that the brace expansion of the file path, not the full remote path expression, has been enclosed in double quotes. In all of the above cases, the system prompts you for a password for the remote host. You can avoid this by providing the host with the public portion of your own personal key. Back to top Enabling automatic login using public keys When you log in to a remote system with ssh, sftp, or scp, you still need to use your password to complete the login process. Once you have exchanged a valid key with a remote site by creating a public or private key and providing the public portion of the key into the ~/.ssh/authorized_keys file, you can eliminate this requirement and allow automatic logins. To create the public or private key, you need to use ssh-keygen, specifying the type of key encryption. The rsa key type is used in the demonstration, but other key types are also valid. See Listing 11 to create the key.
Listing 11. Creating the key

$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa):
You should enter the location of the file where you want to save the key (both the public and private components). Using the default (within the .ssh directory in your home directory) is usually fine (see Listing 12).
Listing 12. Prompt to enter a passphrase

Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase):
If you enter a passphrase at this stage, you create a secure keyfile, but you also have to enter the passphrase each time you use the key. Pressing Return means that no password is required (see Listing 13).
Listing 13. Bypassing the password requirement by pressing the Return key
Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 98:da:8d:48:a8:09:44:b1:b3:62:51:2d:a9:6b:61:ba root@remotehost
A public key (id_rsa.pub) and the corresponding private key (id_rsa) have been created. To enable automatic login, you must copy the contents of the public key into the authorized_keys file within the ~/.ssh directory of the remote host. You can do this automatically using SSH (see Listing 14).
Listing 14. Enabling automatic login

$ cat ./.ssh/id_rsa.pub | ssh mc@remotehost 'cat >> .ssh/authorized_keys';
Better still, if this is something that you do regularly across a range of hosts, you can use a small script or shell function that performs all of the necessary steps for you, as shown here in Listing 15.
Listing 15. Using a shell script to enable automatic login

OLDDIR='pwd'; if [ -z "$1" ]; then echo Need user@host info; exit; fi; cd $HOME; if [ -e "./.ssh/id_rsa.pub" ]; then cat ./.ssh/id_rsa.pub | ssh $1 'cat >> .ssh/authorized_keys'; else ssh-keygen -t rsa; cat ./.ssh/id_rsa.pub | ssh $1 'cat >> .ssh/authorized_keys'; fi; cd $OLDDIR
Using the setremotekey script, you can copy an existing key or, if it doesn't already exist, create one before copying:
$ setremotekey mc@remotehost
Now, whenever you need to log in to a remote host with your public key, you can use the script of your personal key with the list of accepted keys for the user on the remote host. Back to top Summary OpenSSH is an important tool that keeps the communication and transfer of information between machines secure. Not only does it provide a secure alternative to common tools like Telnet, FTP, and RCP, it can also act as a transport protocol for other services like Subversion, X Windows System, and rsync. In this article, you've had a look at the basic steps required to get up and running, how to get the best out of the main tools provided by OpenSSH, and how to use the key exchange facility to simplify your login and connectivity issues.
System Administration Toolkit: Backing up key information

Summary: Most UNIX administrators have processes in place to back up the data and information on their UNIX machines, but what about the configuration files and other elements that provide the configuration data your machines need to operate? This article provides detailed information on techniques for achieving an effective and efficient backup system for these key files. View more content in this series Tags for this article: admin, administrator, aix, backup, system, tar, toolkit, unix
Date: 15 Aug 2006 Level: Intermediate Also available in: Chinese Activity: 7375 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Back to top Identifying key files Your top priority when organizing backups of your UNIX system is the data that they contain. Whether it is databases, development source files, or any other type of socalled user generated information, it is vital to back up this data to ensure that, if there is a failure or other problem, you can restore the data and get back to work. There are, however, a larger number of files and information that exist on your system that technically aren't user data, but these files might take a considerable amount of time to recreate, or reconfigure. How long, for example, would it take you to reconfigure a server or recreate the Domain Name System (DNS) files for your domain? A full backup -- in other words, one where you copy all of the files from your system -would obviously capture everything, but it can be an expensive way of backing up your information. You should be able to create an effective backup by picking and choosing specific files that configure, generate, or support the information and applications. On a UNIX or Linux system, the vast majority of configuration files for the system are located in the /etc directory, but you should consider the full list of potential files (and probably locations) for backup, including:

Main configuration directory (/etc) DNS domain information (/var/bind)
NIS/NIS+ files and configuration (/var/yp) Apache or other Web server configuration (/var/apache, /etc/apache or /usr/local/apache) Mail files or folders (/var/mail and /usr/mail) Lightweight Directory Access Protocol (LDAP) server data (/var/ldap or /usr/local/ldap) Security certificates Custom kernel drivers Kernel configuration or build configuration and parameters License keys and serial numbers Custom scripts and applications User/root login scripts Mail configuration; particularly if you use a solution, such as Cyrus Internet Message Access Protocol (IMAP), where user mail folders are specially recorded and indexed
Other files and sources will be dependent on the system and environment, but it shouldn't take too long to develop a list of the key configuration files that would seriously affect your company, or system, if they were lost. Back to top Collating data for storage Although it is tempting to simply back up the data directly from its source location, copying the backup data to a separate directory before it is backed up enables you to be more selective about the files that are copied, and it also gives you the flexibility to choose a suitable backup method. With the files in one location, you can back up to tape, disk, or copy the contents to another machine without having to reorganize the source files. Reconfiguration of the files that are backed up to any destination only requires changing the script that collates and copies the files to the backup preparation directory. Because you have a local and immediate copy of the information, restoring data in the event of a failure can be quick and straightforward, and you still retain the ability to back up the information to tape, disk, or another system. How information is collated is important, because different solutions imply different storage requirements, techniques, and the facilities available for restoration. Back to top Methods for recording the information You have a vast array of methods for actually backing up and storing your information. Obvious choices are to back up the files to a traditional medium, such as tape. A simpler solution, one that offers a number of benefits and pitfalls, is to copy the relevant information to another machine on your network. The critical element to any successful backup solution is to have a copy of the important information in another location. It is
largely irrelevant whether that is another physical device, a removable storage solution, or another machine altogether. The removable storage solution (tape, disk, or even USB) is the most reliable backup from a disaster recovery standpoint, in that the storage can be kept offsite in a different location. This protects you from a catastrophe, such as fire or theft, at the location where your computers are kept. Using storage on another machine means that the backup data is basically online and available. Restoring from a backup in this situation can be as straightforward as copying the files back to your server in the event of a failure, or copying them over to a replacement system in the event of a system failure. When using a second computer to store your backups, it's a good idea to have not only a copy of the files locally but, if possible, use a directly available offsite location for the files. That machine can either be another computer on the Internet, a machine on your WAN, or at another site. This provides the necessary redundancy and safety. Using a professional or commercial backup solution requires you to reinstall the software before restoring from a backup, and certain configuration information and key files might be required or useful before that software is installed. Back to top Using tar to store backup data Some of the most straightforward methods for storing the information can be a tar, cpio, or other archive file type. When using this method, it is a good idea to date the files and create a simple backup script that generates suitably named files. As a counter to this, you also need a method of deleting backups that you no longer need (for example, those older than a specific period). Listing 1 shows a simple script that creates tarred and compressed (using bzip2) backups of individual directories. The backup files are created on a Network File System (NFS) share to a remote system that holds a copy of the backup.
Listing 1. Creating tarred and compressed backups of individual directories

#!/bin/bash DATE=`date +%Y%m%d.%H%M` HOST=`hostname` TEMP=/mnt/backupprepare echo "Preparing backup..."
cd $TEMP files=`/usr/local/mcslp/filesbydate.pl notlast5days $HOST*` if [ -n "$files" ] then echo "Deleting old files: $files" rm $files fi cd /etc tar cf - ./* |bzip2 -9 - >$TEMP/$HOST-etc.$DATE.tar.bz2 cd /var/bind tar cf - ./* |bzip2 -9 - >$TEMP/$HOST-bind.$DATE.tar.bz2 cd /export/home/webs tar cf - ./* |bzip2 -9 - >$TEMP/$HOST-webs.$DATE.tar.bz2 cd /etc/apache2 tar cf - ./* |bzip2 -9 - >$TEMP/$HOST-webconfig.$DATE.tar.bz2
The DATE variable is generated using the date command and creates a filename of the form 20060627.2200, or 10PM on the 27th of June 2006. To make the backup script portable, all files are created with a prefix containing the name of the host on which the files were created so that you can easily backup multiple hosts to the same location, and the TEMP directory is the destination for each backup. A separate Perl script is used to determine what files in the backup preparation directory can be deleted. You will examine that script shortly. In this script, you specify that you want to keep files for the last five days -- in other words, the script selects files not created in the last five days, based on the date specification you use in the filenames for the backup files. The actual backup process is a simple tar command, combined with bzip2 to compress the files. Because the files generated could be quite large, you might want to adapt this to only choose files that might have changed within a certain period. You can enable this by using find to select the files you want (see Listing 2).
Listing 2. tar command with bzip2 to compress files

tar cf - `find . -type f -mtime -1` | bzip2 -9 >$TEMP/$HOST-webconfig.$DATE.tar.bz2
You specify that only files are selected in the find statement to prevent changes to directories triggering the inclusion of files that have not changed into the tar file. This includes files with reference to the current directory, as shown in Listing 3.
Listing 3. Referencing the current directory

$ cd /etc $ tar cf etc.tar ./*
This avoids specifying the directory explicitly, as shown in Listing 4.
Listing 4. Avoid specifying the directory explicitly

$ tar cf etc.tar /etc
This ensures that recovered files can be put into a spare directory, and not the live location. The script for deleting old files uses the filename, extracts the embedded date and time, and works out whether the files are within a specified limit, for example, within or not within a specific number of days (see Listing 5).
Listing 5. Script for deleting old files

#!/usr/local/bin/perl my $choice = shift; my @files = @ARGV; my @selection; if ($choice =~ /thismonth/) { my ($day,$mon,$year) = dateaslist(); my $match = sprintf('%04d%02d',$year,$mon); foreach my $file (@files) { if ($file =~ m/$match/ && $choice eq 'thismonth') { push @selection,$file; } elsif ($file !~ m/$match/ && $choice eq 'notthismonth') {
push @selection,$file; } } } elsif ($choice =~ /today/) { my ($day,$mon,$year) = dateaslist(); my $match = sprintf('%04d%02d%02d',$year,$mon,$day); foreach my $file (@files) { if ($file =~ m/$match/ && $choice eq 'today') { push @selection,$file; } elsif ($file !~ m/$match/ && $choice eq 'nottoday') { push @selection,$file; } } } elsif ($choice =~ /last(\d+)days/) { my $days = $1; my ($day,$mon,$year) = dateaslist(time()-($1*24*3600)); my $match = sprintf('%04d%02d%02d',$year,$mon,$day); my $spec = sprintf('last%ddays',$days); my $notspec = sprintf('notlast%ddays',$days); foreach my $file (@files) { my ($date) = ($file =~ m/(\d{8})/); push @selection,$file if ($date >= $match && $choice eq $spec); push @selection,$file if ($date < $match && $choice eq $notspec); } } print join ' ',@selection; sub dateaslist { my ($time) = @_; $time = time() unless defined($time); my ($day,$mon,$year) = (localtime($time))[3..5]; $mon++; $year+= 1900; return($day,$mon,$year); }
Using the script, you can pick backup files by a variety of methods (see Listing 6).
Listing 6. Picking backup files
$ filesbydate.pl last5days $ filesbydate.pl notlast14days $ filesbydate.pl nothismonth
# Files created in the last 5 days # Files 15 days or older # Files not created this month
Remember that the comparison works on the filename, not the file system creation or modification date, and so the script is able to work with files that might have been created overnight. Back to top Using rsync to store backup data The rsync tool copies an entire directory structure between locations, or machines, using a special algorithm that only transfers changes in the files. This makes it a very efficient method of copying files, particularly between machines, and that means that the backup process completes quickly. There are two ways you can use rsync, either as a simple synchronization method, in which you can copy all of the critical files to a new drive or system, or as a backup method by copying entire directory trees on a date-by-date basis, in the same way as you created tar backup files in the previous examples. The former method is quick and easy, but you cannot go back to a specific date in the event of a failure. The latter method provides a date-by-date option, but it requires more management (especially because you need to delete older versions that you no longer need) and obviously a significant amount of space, because the files are not compressed. You do, however, get easier and more direct access. Setting up rsync is beyond the scope of this article, but once configured, transferring and synchronizing the information is straightforward. Listing 7 shows a script to synchronize files using rsync.
Listing 7. Script to synchronize files using rsync

#!/bin/bash DESTBASE=admin@atuin:/mnt/backupprepare HOST=`hostname` cd /export/data/svn rsync --stats --rsh=/usr/bin/ssh --delete --recursive --times -og --links . $DESTBASE/$HOST/svn cd /export/home/webs rsync --stats --rsh=/usr/bin/ssh --delete --recursive --times -og
--links . $DESTBASE/$HOST/webs cd /var/bind rsync --stats --rsh=/usr/bin/ssh --delete --recursive --times -og --links . $DESTBASE/$HOST/bind cd /etc rsync --stats --rsh=/usr/bin/ssh --delete --recursive --times -og --links . $DESTBASE/$HOST/etc
The rsync command options specified are as follows:

--stats shows the statistics of the synchronization. --rsh tells rsync to use Secure Shell (SSH) to copy the files (for security). --delete deletes files on the destination that do not exist in the local directory. --recursive ensures the entire directory is examined. --times preserves the file creation and modification times. -og preserves ownership and group information. --links copies links as links, instead of the files they link to.
The DESTBASE variable specifies the base location (in this case, a file system on a remote host), and the HOST variable holds the hostname information so that you can use the same script across multiple hosts for backup. Back to top Backing up the collated data In the previous sections, you've used tar and rsync to create backups in a separate folder. You could use this collated information as your main backup, especially if the files were located on another machine. Ideally, however, you should also backup these files to another location. The above scripts, whether tar or rsync, collate the information from multiple directories on multiple hosts into a single location. From this base, you can backup the information further using whatever techniques you prefer, including copying to another machine or device, or by copying to tape or disk. Back to top Keeping a longer-term record Most backup solutions, however well managed, often rely on rotating and recycling the media, or destination, used to store the data. There are, however, some types of key data where you might want to keep a longer record of the information, and might even want to be able to record information on changes and modifications to the data as part of the backup process.
The period over which you record the information is really limited only by the amount of disk space, or storage, that you have available. Using the date methods, particularly with tar or cpio, means you can keep a longer-term record with little impact on storage. By using a regular full backup (in other words, all the files) and a diff backup where you backup only changes, you can extend the duration still further. Back to top Summary Backing up key files is about a combination of identifying the files and finding a suitable method for storing and backing up that information. There are many locations for files to be stored, and you should take great care to ensure you cover important, but often forgotten, areas like kernel drivers and libraries and configuration. Making an effective backup of that information can then be achieved in whatever method is appropriate. Unlike user data, key files often need to be recovered more urgently to get the machine into the correct configuration before the rest of the restoration continues. Having ready access to that information, by using tar or rsync, can often be more effective.
System Administration Toolkit: Get the most out of bash

Summary: Ease your system administration tasks by taking advantage of key parts of the Bourne-again shell (bash) and its features. Bash is a popular alternative to the original Bourne and Korn shells. It provides an impressive range of additional functionality that includes improvements to the scripting environment, extensive aliasing techniques, and improved methods for automatically completing different commands, files, and paths. View more content in this series Tags for this article: aix, aix_and_unix, bash, linux, systems_admin, toolkit, unix
Date: 12 Dec 2006 Level: Intermediate
Also available in: Chinese Russian Activity: 21413 views Comments: 1 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Back to top Bash background Shells under UNIX and Linux typically fall into one of two categories based on the original shells included with the earliest versions of UNIX. The two types are the Bourne shell and the C shell, the latter being distinctive because its format and structure is like that of the C programming language. The Bourne shell is easier to use and understand than the C shell, but it is less practical for the more complex script programming that you might want to achieve within the shell programming environment. The Korn shell provides ease of use of the Bourne shell and added extensions for job control (allowing you to easily manage multiple background jobs), command-line editing and history, and added elements of the C shell to make programming easier. The Bourne-again shell (bash) is an open source project that combines the principles of the Bourne shell, programming environment of the C shell, extended functionality of the Korn shell, and a number of extensions of its own to provide a rich environment not only for programming shell scripts, but also as an interactive shell environment for controlling and interacting with your machine. Back to top Command-line editing and key bindings
The main command prompt within bash provides both the ability to edit the command line and a history function, remembering individual command lines so that you can execute them again. The editing functionality means that you can go forward and backward through the command line currently displayed to make changes and correct typos. In bash's standard form, you can use the cursor keys for basic movement. More extensive commands, such as going backward and forward by words, are controlled by the Readline library that supports both vi and emacs bindings by default. To set the editing mode, specify your preferred mode either on the command line or in a bootstrap file: $ set editing-mode emacs. For example, using the emacs editing mode, the following key bindings are in effect:

Control-A -- This key binding takes you to the beginning of the line. Control-E -- This key binding takes you to the end of the line. Control-K -- This key binding deletes everything to the end of the line. Meta-B -- This key binding goes back by one word. Meta-F -- This key binding goes forward by one word. Meta-D -- This key binding deletes the current word.
You can in fact bind any key or combination you like to a particular operation using the internal bind bash command. To start, you can get a list of the available commands by using the -P option (see Listing 1).
Listing 1. Using the -P option to get a list of available commands

$ bind -P abort can be found on "\C-g", "\C-x\C-g", "\M-\C-g". accept-line can be found on "\C-j", "\C-m". alias-expand-line is not bound to any keys arrow-key-prefix is not bound to any keys backward-byte is not bound to any keys ... yank can be found on "\C-y". yank-last-arg can be found on "\M-.", "\M-_". yank-nth-arg can be found on "\M-\C-y". yank-pop can be found on "\M-y".
The \C refers to the control key. The \M sequence refers to the 'meta' key (special on some keyboards, or usually the Alt key or the Escape key). To set a binding, you must specify the key sequence and the command to be executed, separated by a colon, with the key sequence escaped by a double quote (in extreme circumstances, you might need to escape this again with a single quote). For example,
to change Control-B to go backwards word by word, use $ bind "\C-b":backwardword. You can even use the binding to execute a shell command (for example, to run an application). To do this, you must add the -x option, and this is an example of where the escaping of both is required. For example, to set Control-E to run emacs, you would use the following: $ bind -x '"\C-e"':emacs. To have key bindings in bash enabled every time, you can either set the information in the .inputrc file (which then affects all Readline-enable d applications), or you can place specific bash bindings in your startup scripts, which will be covered later in this article. Back to top Command history Every command that you enter is recorded so that you go back to the command and either run it again verbatim, or edit it and run the edited version. You can go backwards and forwards through your command history (in reverse chronological order) using Control-P (previous command) and Control-N (next). You can only go next if you have already gone back through your previous commands. If you know the contents of the command but can't remember where it was and don't want to go through the command list by hand, you can use Control-R to do a reverse intelligent search. This goes back to the first command (in reverse order) that matches the first letter you typed; letters typed successively match more and more specifically. Once you've found the command you want, press Return to run it verbatim. To edit the command you've found, use the cursor keys (or key bindings) to move to the point you want to edit. Back to top Prompt customization All shells enable you to customize the prompt presented to you on the command line, usually by setting the value of the PS1 variable. Normally this is limited to being able to set a static string or, in case of the Korn shell, you can normally set a dynamic value, such as the current directory. Bash extends this functionality even further to allow the inclusion of username, hostname, and even hidden characters (for example, the escapes required for colorization or setting the title of windows and Xterms). The string specified is evaluated each time just before the prompt is printed so that it's always up to date.
Custom expansions of specific dynamic values are achieved with a series of backslash characters. For example, to set a typical prompt consisting of the username and last part of the current directory, use the following: PS1="\u \W$". A more typical solution is to display the username, hostname, directory, and the dollar or hash sign, depending on the UID of the current user. The latter option displays a hash sign when the effective user ID is zero (root): PS1='\u@\h:\W \$ ". The full list of the available expansions is shown in Table 1.
Table 1. List of available expansions Escape character \t \d \n \s \w \W \u \h \# \! \nnn \$ \\ \[ Description \t is the time, in HH:MM:SS format. \d is the date, in Dayname Month Day format (for example, Fri Oct 13). \n is a new line. \s is the name of the shell (typically bash, or '-bash' if the shell is a login shell). \w is the full path of the current directory. \W is the final part of the current directory (in other words, 'mc' when in '/home/mc'). \u is your username. \h is the hostname. \# is the command number of this command (the value is incremented for each line). \! is the unique history number for this command. \nnn is the character that is identified by the octal nnn. \$ is the # character if the effective UID is zero, otherwise it is the $ character. \\ is a backslash. \[ begins an escape sequence; generally used for terminal control characters for colorization and titles.
\]
\] ends an escape sequence.
The escape sequence characters can be used to send terminal control sequences, which in turn can be used to set colors and terminal titles. Because bash updates this information each time the prompt is shown, this automatically updates window titles accordingly. For example, use the following sequence in Listing 2 to set the terminal window and Xterm title to user@hostname:fullpath, and set the prompt to hostname:endpath with an appropriate termination character (dollar or hash).
Listing 2. Setting the terminal title and the prompt

PS1=" \[\033]0;\u@\h: \w\007\]\[\033[31m\]\h \[\033[34m\]\W \$ \[\033[00m\]"
In addition, the hostname is colored in red when it is root, green when it is a normal user, and the remainder of the prompt is colored in cyan. Back to top File and directory completion To help speed up your shell interaction, bash includes pathname completion -- that is, you can get bash to either fill in the remaining parts of a pathname or provide you with a list of potential expansions. For example, if you look at the current directory, there are a range of different subdirectories (see Listing 3).
Listing 3. Subdirectories in the current directory

$ ls -f back/ cheffyhack/ build/ cvs/ calc/ dbdumps/
edin/ install/ logs/
logstomerge/ lost+found/ my.cnf
mysql-binlogs/ mysqlsizer statmon/
svn/ vmware/ webs/
To change into the edin directory, you could type: $ cd edin. Or, using completion within bash, you could type: $ cd e and then press the TAB key. By default, this will attempt to expand the pathname beginning with 'e'; the result in this directory should be: $ cd edin.
If there is more than one match, then the first press of TAB fails; pressing TAB a second time returns a list of matching paths (see Listing 4).
Listing 4. Returning a list of matching paths

$ cd my my.cnf
mysql-binlogs/ mysqlsizer
The expansion continues to work, replacing what it can with unique components, either until there are either no further expansions, or only one. For example, consider the following sequence shown in Listing 5.
Listing 5. Continuing the expansion

$ cd my <TAB> my.cnf mysql-binlogs/ mysqlsizer $ cd mys <TAB> $ cd mysql <TAB> <TAB> mysql-binlogs/ mysqlsizer $ cd mysqls <TAB> $ cd mysqlsizer
If the pathname is a directory, then you can continue to expand each part of the pathname. Back to top Aliases Aliases are a simple mechanism that enable you to provide an expansion for a given sequence. Because the alias is an expansion and not a replacement, it enables you to continue adding options and arguments to an aliased command. For example, it's common to set an alias ('ll'), which expands to 'ls -l', for a long listing of a directory or list of files. You can create the alias using the following command line: alias ll='ls -l'. To use, just type the alias at the prompt: $ ll. Because the alias is just that, an alias to the real command, you can add further options, for example, to list all the files in the directory, use: $ ll -a, which effectively expand to: $ ls -l -a.
Aliases are an effective way to run commands quickly, or run your favorite commands with the command-line options you commonly use. Back to top Using the directory stack The directory stack enables you to store one or more directories into a temporary area, and then bring them back again. The list of directories acts like a stack -- you push a directory onto the top of the stack, and pop a directory off the top again to get it back. To push a directory onto the stack, use pushd and the directory you want to place on the stack -- for example, you can push the current directory, which can be identified by the single period (see Listing 6).
Listing 6. Pushing the current directory

$ pushd . /export/data /export/data
Bash responds with the directory you pushed and a list of all the directories currently in the stack separated by a space. You can see this more clearly by changing the directory and pushing it again (see Listing 7).
Listing 7. Pushing a different directory

$ cd /etc $ pushd . /etc /etc /export/data
Using popd takes the last directory added to the stack off and changes to that directory (see Listing 8).
Listing 8. Using popd

$ cd /usr/local/bin $ popd /etc /export/data $ pwd
The popd command returns the list of directories before changing to the new directory. The directory functionality is most often used when you want to temporarily change your directory and then go back to the original directory. To make this kind of situation easier, bash is often configured with two aliases, pu and po, as shown in Listing 9.
Listing 9. Pu and po aliases

$ alias pu='pushd .' $ alias po='popd'
For example, imagine you are working in a directory and suddenly realize you need to create a tar archive in another directory (see Listing 10).
Listing 10. Use pu and po to create a tar archive

$ pwd /usr/local/bin $ pu /usr/local/bin /usr/local/bin $ cd /export/data $ tar cf webs.tar ./webs $ po $ pwd /usr/local/bin
Although the directory stack system supports multiple directories, it frequently gets used for only one or two while you temporarily pop somewhere else. Back to top Bash run time configuration files When logging in, bash supports the standard .profile file (as used by Bourne shell), in addition to its own specific .bash_profile. The file to use is chosen, as follows:

If ~/.bash_profile exists, use it, otherwise: If ~/.bash_login exists, use it, otherwise: If ~/.profile exists, use it.
For bash shells not started as a login shell (for example, when using Xterm or another application that starts a sub shell, bash looks for the ~/.bashrc file, if it exists.
Since the contents of .bashrc and .bash_profile are unlikely to change, it is not uncommon to find the .bash_profile containing the following in Listing 11.
Listing 11. .bash_profile contents

if [ -f ~/.bashrc ]; then source ~/.bashrc fi
Beyond these rules, the contents of the files is entirely up to you. Because so much customization is available within the bash environment, it is also not uncommon to find that .bashrc/.bash_profile contents are simply wrappers that source a number of other ~/.bash_* scripts. For example, you might split up your configuration into:

~/.bash_aliases -- to store all your custom aliases and functions ~/.bash_path -- to store all your path specifications ~/.bash_vars -- to store all your bash variables
These are then sourced within .bashrc (see Listing 12).
Listing 12. .bashrc contents

if [ -f ~/.bash_path ]; then source ~/.bash_path fi if [ -f ~/.bash_vars ]; then source ~/.bash_vars fi if [ -f ~/.bash_aliases ]; then source ~/.bash_aliases fi
You should be careful to ensure that the files are sourced in the right order. Back to top Scripting improvements The last area in which bash provides some considerable improvement is in the scripting functionality. One of the commonly quoted limitations of most shells is that they have
loose variable typing, no support for arrays of data, and no built-in functionality to perform basic math or expressions. All of these have been resolved within bash to different degrees. Variables within bash can be declared before they are used, and the declaration can include a type. For example, to declare a variable as an integer type (and therefore to always be identified as a valid number), use: $ declare -i myint. To set a value at the same time, use: $ declare -i myint=235. To perform basic arithmetic, you can embed the expression into $(( )) (see Listing 13).
Listing 13. Embedding the expression to perform basic arithmetic

$ echo $((4+3*12)) 40
You can also include variables (see Listing 14).
Listing 14. Including variables

$ echo $((myint+3*12)) 63
To declare a variable as an array type, use $ declare -a myarray. You can add values by specifying a parenthesized list of values: $ declare -a myarray=(tom dick harry). To get a value out of the array, specify the array reference (see Listing 15).
Listing 15. Specifying the array reference

$ echo ${myarray[1]} dick
You can use the same system to populate the array, for example, from a list of files (see Listing 16).
Listing 16. Populating the array from a list of files

$ declare -a files=`ls` $ echo $files back/ build/ calc/ cheffyhack/ cvs/ dbdumps/ edin/ install/ \ logs/ logstomerge/ lost+found/ my. cnf mysql-binlogs/ mysqlsizer statmon/ svn/ vmware/ webs/
And you can use the entire contents of the array by using the @ symbol as the element specification (see Listing 17).
Listing 17. Using the @ symbol as the element specification

for file in ${files[@]} do echo $file done
This manipulation and variable support makes many aspects of programming with a shell script significantly easier. Back to top Summary Bash provides a number of significant improvements over the traditional Bourne, Korn, and C shells. The bulk of these extensions improve the experience for the interactive user -- that is, the user who is using bash as their main method of interacting with their system. Some systems become a habit, for example, pathname completion and aliases, while others are vital on individual occasions. The directory stack functionality is a good example; it becomes invaluable when you want a quick method of visiting a directory without losing your train of thought. Whatever elements you use within the bash shell, you will find a rich environment and an almost limitless array of customizations that you can use to improve your interaction and environment.
System Administration Toolkit: Monitoring a slow system

Summary: When your UNIX system runs slow, it is vital that you discover what the problem is as quickly as possible so you can get your system back into the normal operating mode. There are many causes for a slow system, but actually identifying the problem can be exceedingly difficult. In this article, study examples of how to identify and diagnose the cause of your slow running UNIX system to get your machine running properly again. View more content in this series Tags for this article: aix, performance
Date: 07 Jun 2006 Level: Intermediate Also available in: Russian Activity: 35259 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Back to top
Causes of slow systems There are many different potential causes of a slow system, but they can generally be slotted into the following areas:
Too many processes. Your system is merely running too many applications at the same time, or running a small number of very CPU-intensive operations. Either you are overloading the server, or there is a runaway process that is sapping your system resources. Too much active memory. If your processes are using a lot of memory, then your system might be swapping a lot of pages in and out to disk and, that means, your system is spending more time swapping the memory than it is actually using it. Hardware fault. Occasionally you will come across a hardware fault that causes slow downs. A bad network card, hard drive, or memory might result in your system spending a long time waiting for information.
To diagnose the problem, you need to use a number of available tools to examine your UNIX system. Back to top Choosing a connection method If your machine is running particularly slow, then the first issue will be how you connect to your machine so that you can start the monitoring process. A slow machine might not accept connections through Telnet or over a remote shell protocol, such as ssh. If you are not already logged in, getting access might be impossible. Instead, consider using the console, either directly or through a separate hardware solution, such as a network or serial based console monitor. The console is more likely to allow you to log in, because there will already be a login process (which will be replaced with your shell) running. If, once you log in, you are unable to run any processes through your shell, it indicates that your system has run out of process space; a reboot will probably be the only way of returning your system to normal. To reboot your system, use init or telinit to adjust the run level; run level 6 is usually synonymous with a reboot. Using init/telinit is more likely to reboot the system, as there is only one process involved to force the reboot. Once the system is back up and running, you need to use some of the techniques in this article to monitor the running state of the system and record the output. If the slow system event happens again, you can then perform a post-mortem debug and examine the reason why your system runs so slowly. Back to top
Using uptime If you suspect that your machine is running slowly, then the first command you should run is uptime. Uptime reports the current time, the amount of time the machine has been up and running (in other words, the time since the machine has booted), and the current number of users. It then provides three figures that show the load average in the last minute, five minutes, and fifteen minutes. For example:
$ uptime 18:28:54 up 10 days, 8:38, 2 users, load average: 2.24, 5.34, 3.42
In this example, the machine has had a load average of more than two in the last minute, more than five in the last five minutes, and more than three in the last fifteen minutes. The definition of load average is complex and driven by the status of the processes that are executing. In general, each process that is running, waiting for CPU, or waiting on I/O would add one to the load average. These figures are calculated and then averaged over time. In a single-CPU system, a load average of over one means that the CPU is under powered for the type of load you are supplying. But because of the multi-process nature of UNIX, it is generally acceptable for a load average of up to two over the long term (in other words, the fifteen minute figure) before you should be concerned. In a multi-CPU (or multi-core) system, divide the load average by the number of CPUs. To determine if the machine is over stressed, use the same principles as above. An alternative way of looking at the figures is to treat them as a percentage; in other words, if the figures above were from a single-CPU system, then the machine would be able to cope with the load if it was 224 percent faster. Within a multi-CPU system, you should use the number of CPUs plus one to determine the maximum load. For example, a four-CPU system would have a maximum load average of 5. It is quite common to find a machine that has a load average that is significantly higher than its maximum for a short period. For example, when building or compiling an application or performing a very disk intensive task, your load averages can shoot up. This is why the output includes one, five, and fifteen minute averages, because it helps to smooth out any transient extreme loads. Any long-term, or unexpected, high values probably signify a problem and need further investigation. If the numbers are low but your system is slow, it might point to a swap space issue.
Back to top Using ruptime If you are managing a large network of systems, then there is a simple way of monitoring the load and usage of all the machines in your network. The ruptime tool collects data that is broadcast by all the machines on the network and collects it into a local file so that the current status of all the machines can be examined. For example, Listing 1 shows the output from a small network:
Listing 1. Output from a small network

$ ruptime bear ultra3 atuin
up 10+09:13, up 6+01:16, down 4+00:52
2 users, 1 user,
load 0.66, 0.68, 0.50 load 0.00, 0.00, 0.00
The last machine has not reported any data for eleven minutes, so it is officially listed as down. To generate the information, the rwhod daemon (sometimes in.rwhod) needs to be running on each machine in your local network. This daemon broadcasts the information for the local machine and collects the broadcast data from all the other machines. Because of the way the rwho/ruptime system works, there can be performance issues, especially in very large networks where the number of systems reporting and the network traffic they generate can be considered to be detrimental. In very busy systems, the need to broadcast the data can also mean that the information is never reported, the data can be out of date, or the system could be reported as down when it is just busy. Back to top Tracking large processes If you suspect that the problem is due to a large or overly busy process, then you should check the output of the ps tool, look for the process size, the percentage of memory, and CPU being utilized. On an SVR4 system (Solaris and AIX), you can use the following command to get a list of processes (see Listing 2).
Listing 2. Command to get a list of processes
$ ps -A -o pcpu,pmem,rss,vsz,comm %CPU %MEM RSS VSZ COMMAND 0.2 0.0 0 0 fsflush 0.1 0.2 1464 8288 /usr/lib/ssh/sshd 0.1 0.1 1032 1320 ps 0.0 1.0 9536 47608 /usr/openwin/bin/Xsun 0.0 0.7 6312 10720 dtgreet 0.0 0.6 6136 9352 /usr/sfw/sbin/snmpd 0.0 0.4 3208 5720 /usr/lib/fm/fmd/fmd 0.0 0.3 2808 8512 /usr/lib/ssh/sshd 0.0 0.3 2800 8504 /usr/lib/ssh/sshd 0.0 0.3 2768 8512 /usr/lib/ssh/sshd 0.0 0.3 2368 4056 /usr/sbin/nscd 0.0 0.2 2096 9176 /usr/dt/bin/dtlogin ...
Listing 3 shows how it looks on a BSD-derived system.
Listing 3. Getting a list of processes on a BSD system

$ ps -A -o pcpu,pmem,rss,vsz,command|sort -n +3 %CPU %MEM RSS VSZ COMMAND 0.0 0.0 152 27236 nfsd-server 0.0 0.0 152 27236 nfsd-server 0.0 0.0 152 27236 nfsd-server 0.0 0.0 152 27236 nfsd-server 0.0 0.0 152 27236 nfsd-server 0.0 0.0 152 27236 nfsd-server 0.0 0.0 152 27236 nfsd-server 0.0 0.0 152 27236 nfsd-server 0.0 0.0 164 27236 nfsd-master 0.0 0.0 224 27240 /usr/sbin/update 0.0 0.3 4364 29196 /usr/sbin/securityd 0.0 0.2 2760 29288 jabberd -c /etc/jabber/jabber.xml -H /private/var/jabber/ -U jabber 0.0 0.0 184 29300 nfsiod -n 4 0.0 0.2 3544 29712 /usr/sbin/configd 0.0 0.0 500 30628 /usr/sbin/sshd -i 0.0 0.0 260 30648 /usr/sbin/smbd -D 0.0 0.0 736 30648 /usr/sbin/smbd -D 0.0 0.1 1216 30700 /usr/sbin/sshd -i ... 0.0 0.1 2180 50664 imapd: narcissus.mcslp.pri [192.168.0.110] mc user.mc 0.0 0.1 2184 50664 imapd: sulaco.mcslp.pri [192.168.0.101] mc user.mc 0.0 0.1 2204 50720 imapd: narcissus.mcslp.pri [192.168.0.110] buy user.buy 0.0 0.1 2264 50720 imapd: sulaco.mcslp.pri [192.168.0.101] buy user.buy 0.0 0.1 2272 50984 imapd: kernel.mcslp.pri [192.168.0.106] slp
user.slp 0.0 1.2 18348 54368 servermgrd -x 0.0 0.2 3200 85920 /usr/sbin/named -f 0.0 1.1 16820 122240 /usr/libexec/mysqld --basedir=/usr --datadir=/var/mysql --user=mysql --pid-file=/var/mysq 0.0 0.5 8572 158164 /usr/libexec/slapd -d 0 -h ldap:/// ldapi://%2Fvar%2Frun%2Fldapi 0.0 0.0 204 289396 rpc.statd
In both cases, the CPU and memory usage percentages have been displayed in the process list so that you can get a good idea of the loading on the system. The 's' and 'stat' columns (for SVR4 and BSD, respectively) show the current status of the process. A large number of running processes (status 'R') indicate that the process is currently running. By using a combination of the state, CPU, and memory percentage, you should be able to identify if there is a process that is running away and over using your system resources. Back to top Using iostat The iostat tool provides information about the terminal, disk activity, and CPU utilization. You can specify a single numerical argument to set the report interval; a second numerical argument sets the number of reports. For example, Listing 4 shows how to report the statistics every five seconds.
Listing 4. Reporting statistics every five seconds

$ iostat 5 tty dad1 tin tout kps tps serv 0 7 440 39 14 0 39 2 0 0 0 13 4 3 0 0 13 0 0 0
sd1 kps tps serv 0 0 3 0 0 0 0 0 0 0 0 0
nfs1 kps tps serv 0 0 0 0 0 0 0 0 0 0 0 0
cpu us sy wt 5 18 0 0 0 0 0 0 0 0 0 0
id 77 100 100 100
The exact information, shown by default, varies from system to system; Listing 4 was from a Solaris system. The example in Listing 5 comes from a BSD environment.
Listing 5. iostat in a BSD environment
disk1 disk0 KB/t tps MB/s KB/t tps MB/s 167.67 0 0.02 20.70 5 0.09 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 14.33 24 0.33 0.00 0 0.00 2.83 1 0.00
cpu us sy id 6 3 90 15 3 82 16 2 82 18 4 79 23 4 73
Dealing with the CPU statistics first, the columns show user (us), system (sy), and idle (id) percentages. The user time shows how much time was spent on user processes. The system time shows the time in system processes (including, when wait time is not shown, the time the system is waiting for I/O). The idle time shows the percentage of time that the CPU(s) were idle. Disk output shows how busy the individual physical disks (including NFS mounts, if applicable) are, usually in transactions per second and MB or KB transferred per second. Large figures here, especially if combined with high wait/system times, might indicate a disk that is too slow for the system. You can try to spread your application so that it uses different disks you might be able to improve performance. If the disk is the same as the one used for virtual memory, then it might be a problem with lack of memory and too much swapping. Back to top Using vmstat You can monitor virtual memory statistics with the vmstat tool. Like iostat, it accepts a numerical interval (see Listing 6).
Listing 6. Monitoring memory statistics with vmstat

$ vmstat 5 kthr memory page disk faults r b w swap free re mf pi po fr de sr dd s1 -- in sy cs us sy id 0 0 0 2820888 809552 94 525 121 69 50 0 26 16 0 0 297 1342 272 9 4 87 0 0 0 2824752 778872 2 7 0 0 0 0 0 0 0 0 229 34 109 0 1 99 0 0 0 2824752 778872 0 0 0 0 0 0 0 2 0 0 233 28 116 0 0 100 0 0 0 2824752 778872 0 0 0 0 0 0 0 0 0 0 228 26 110 0 0 100 0 0 0 2824752 778872 0 0 0 0 0 0 0 0 0 0 229 28 111 0 0 100
cpu
The vmstat tool outputs thread/process information, memory/swap usage, page ins/outs, disk I/O, page faults, and CPU statistics. The CPU/thread block shows the processes/threads in the run queue (r), blocked processes waiting for I/O resources (b), and those that were swapped. High numbers in the blocked processes column indicates slow disks. High numbers in the swapped column indicate that there are two many processes using too much memory that need to be swapped in and out. Swapping is an expensive process and will significantly degrade your system. The memory shows the amount of swap currently available and the size of the free list (the number of pages that could be swapped if the RAM were requested). Low swap values indicate that you are running out of swap space, which doesn't necessarily show a problem, as long as you have enough RAM to run the applications. Low free list values might indicate that you have a lot of active RAM in use, which might trigger swap space use if you add more processes to the system. The page columns show the pages of memory swapped in and out to disk. The key columns are the pi/po (page in/page out), which indicate how many pages have been exchanged. High paging indicates a lack of RAM; a high scan rate (the sr column) shows a potential memory bottleneck. Back to top Using top The top tool can provide a useful way to monitor a live system and the active processes, loading, and memory statistics. There are many different types of top, some of which are installed by default on some systems and also the latest open source version of the tool. The information provided is like a combination of the uptime, swap space, and ps tools. For example, the following is from a Solaris system running V3.5.1 of the top tool (see Listing 7).
Listing 7. Using top

last pid: 9385; load averages: 7.14, 2.98, 1.21 61 processes: 55 sleeping, 4 running, 1 zombie, 1 on cpu CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap Memory: 1024M real, 712M free, 125M swap in use, 2705M swap free PID 9313 9349 9385 9384 9145 USERNAME LWP PRI NICE SIZE RES STATE root 1 22 0 35M 34M run root 1 22 0 21M 20M run root 1 39 0 4320K 3904K run root 1 29 0 3888K 3424K run root 1 59 0 3736K 2144K cpu TIME 0:03 0:01 0:00 0:00 0:00 CPU 8.87% 5.47% 0.38% 0.30% 0.11% COMMAND cc1 cc1 as as top
9180 root 486 root 548 root 553 mc 9345 root 9348 root 9325 root 599 mc 9312 root 9 root svc.configd
1 1 1 1 1 1 1 1 1 16
59 59 59 49 49 59 49 59 59 59
0 0 0 0 0 0 0 0 0 0
1808K 46M 10M 8288K 1328K 1328K 1328K 8288K 1328K 9464K
1472K 9536K 6360K 1472K 928K 928K 928K 1488K 928K 2016K
sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep
0:00 0:00 0:00 0:01 0:00 0:00 0:00 0:00 0:00 0:06
0.10% 0.03% 0.03% 0.02% 0.01% 0.01% 0.01% 0.00% 0.00% 0.00%
make Xsun dtgreet sshd gcc gcc gcc sshd gcc
The top tool shows the CPU usage of individual processes; for example, in the previous sample, you can see that you are compiling a number of files and how much CPU they are using. You should also keep an eye on the process state: high numbers of running processes might indicate an over-busy system (compare the running processes with the CPU states and load averages for the system). Top itself can be a CPU hog; it is best to run it at a relatively large update interval, to prevent the monitoring having a detrimental effect on the performance of your system. You can specify the update interval in seconds using either -s or -d command-line option (depending on your platform). Back to top Using SAR In situations where you cannot monitor the status of your server live but you want to be able to monitor the status of a machine after it has caused a problem, you can use the SAR (system activity reporter) tool. This works by recording information at specific intervals into a global file that can then be post processed to display information about the machine. Because the recording process continues in the background, it can be used to plot the system performance over time and might help you identify the cause of the problem. Information is generally recorded day by day, for up to a month, at intervals that you specify. Logs are written into /var/log/sa/saDD or /usr/adm/sa/saDD, where DD is the day of the month. Enabling SAR is specific to the system, and generally you will need to set up a cron job that automatically runs the collection script (sa1). Another script, sa2, can create a daily report that you can study. For example, the crontab below shows the default system performance statistics recording on a Solaris system:
0 * * * 0-6 /usr/lib/sa/sa1 20,40 8-17 * * 1-5 /usr/lib/sa/sa1 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A
Once the information has been collected, you can extract data using the sar command. The amount of information, which is recorded by the system, can be voluminous, and the level of detail that can be selected and extracted from that data is similarly large. However, you can get a feel for the quantity and quality of the data by using the -A command-line argument to SAR, which reports all the currently recorded information.
Listing 8. Output generated with the sar command, using the -A argument
11:49:38 13:20:00 13:40:01 14:00:00 14:20:00 14:40:01 15:00:00 15:20:00 Average 11:49:38 ... Average %usr 1 19 0 0 0 0 0 3 device dad1 dad1,a dad1,b dad1,c dad1,d dad1,e dad1,h nfs1 nfs2 sd1 %sys 1 5 0 0 0 0 0 1 %busy 1 0 0 0 1 0 0 0 0 0 %wio 0 0 0 0 0 0 0 0 avque 0.3 0.0 0.0 0.0 0.2 0.0 0.0 0.0 0.0 0.0 %idle 99 76 100 100 100 100 100 96 r+w/s 5 0 0 0 3 0 1 0 0 0 blks/s 365 4 0 0 143 39 178 0 31 0 avwait 47.3 15.4 0.0 0.0 53.0 117.3 29.0 0.0 0.5 0.0 avserv 4.5 8.6 13.8 0.0 3.9 5.9 4.6 0.0 14.5 3.3
11:49:38 runq-sz %runocc swpq-sz %swpocc 13:20:00 2.0 2 0.0 0 13:40:01 5.3 15 0.0 0 14:00:00 0.0 0 0.0 0 14:20:00 0.0 0 0.0 0 14:40:01 1.5 0 0.0 0 15:00:00 0.0 0 0.0 0 15:20:00 0.0 0 0.0 0 Average 5.0 2 0.0 0
11:49:38 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 13:20:00 0 11 97 0 1 89 0 0 13:40:01 0 803 100 4 381 99 0 0 14:00:00 0 0 100 0 0 39 0 0 14:20:00 0 0 100 0 0 56 0
0 14:40:01 0 15:00:00 0 15:20:00 0 Average 0
0 0 0 0
0 0 0 120
100 100 100 100
0 0 0 1
0 0 0 56
61 48 32 99
0 0 0 0
11:49:38 swpin/s bswin/s swpot/s bswot/s pswch/s 13:20:00 0.00 0.0 0.00 0.0 305 13:40:01 0.00 0.0 0.00 0.0 223 14:00:00 0.00 0.0 0.00 0.0 111 14:20:00 0.00 0.0 0.00 0.0 112 14:40:01 0.00 0.0 0.00 0.0 112 15:00:00 0.00 0.0 0.00 0.0 114 15:20:00 0.00 0.0 0.00 0.0 114 Average 0.00 0.0 0.00 0.0 fork/s 0.64 9.31 0.01 0.00 0.01 0.01 0.02 1.46 152 exec/s rchar/s wchar/s 0.59 38118 25779 6.53 773352 1558934 0.01 342 186 0.00 150 128 0.00 153 128 0.02 326 167 0.03 641 272 1.04 118615 232791
11:49:38 scall/s sread/s swrit/s 13:20:00 526 39 26 13:40:01 2288 803 320 14:00:00 22 2 2 14:20:00 20 2 2 14:40:01 20 2 2 15:00:00 26 3 3 15:20:00 29 3 3 Average 11:49:38 13:20:00 13:40:01 14:00:00 14:20:00 14:40:01 15:00:00 15:20:00 Average 416 125 52
iget/s namei/s dirbk/s 2 31 3 29 385 25 0 1 0 0 0 0 0 0 0 0 1 0 0 2 0 5 61 4
11:49:38 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s 13:20:00 0 0 39 0 0 0 13:40:01 1 0 397 0 0 0 14:00:00 0 0 9 0 0 0 14:20:00 0 0 0 0 0 0 14:40:01 0 0 0 0 0 0 15:00:00 0 0 16 0 0 0 15:20:00 0 0 38 0 0 0 Average 11:49:38 0 proc-sz ov 0 72 inod-sz ov 0 0 file-sz ov 0 lock-sz
13:20:00 13:40:01 14:00:00 14:20:00 14:40:01 15:00:00 15:20:00 11:49:38 13:20:00 13:40:01 14:00:00 14:20:00 14:40:01 15:00:00 15:20:00 Average 11:49:38 13:20:00 13:40:01 14:00:00 14:20:00 14:40:01 15:00:00 15:20:00 Average 11:49:38 13:20:00 13:40:01 14:00:00 14:20:00 14:40:01 15:00:00 15:20:00 Average
53/16154 54/16154 57/16154 57/16154 57/16154 57/16154 57/16154 msg/s 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 atch/s 13.39 188.44 0.30 0.16 0.20 0.72 0.89 29.66
0 0 0 0 0 0 0
1732/69661 15118/69661 15120/69661 15120/69661 15120/69661 15121/69661 15127/69661
0 0 0 0 0 0 0
358/358 358/358 359/359 359/359 359/359 359/359 359/359
0 0 0 0 0 0 0
0/0 0/0 0/0 0/0 0/0 0/0 0/0
sema/s 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 pgin/s ppgin/s 3.67 5.05 9.91 25.61 0.05 0.06 0.00 0.00 0.00 0.00 0.01 0.01 0.02 0.02 1.90 4.38 pflt/s vflt/s slock/s 41.14 77.09 0.00 373.73 1086.42 0.00 0.61 1.59 0.00 0.34 0.76 0.00 0.48 1.01 0.00 0.98 2.37 0.00 1.43 3.47 0.00 60.43 170.40 0.00
pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf 0.03 0.06 0.06 0.00 0.00 6.41 19.18 13.84 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.95 2.83 2.05 0.00 0.00
11:49:38 freemem freeswap 13:20:00 109186 5736615 13:40:01 95816 5614822 14:00:00 97408 5649849 14:20:00 97311 5647409 14:40:01 97418 5653711 15:00:00 97338 5648982 15:20:00 97333 5648993 Average 98516 5654784
11:49:38 sml_mem alloc fail lg_mem alloc fail ovsz_alloc fail 13:20:00 4178176 3572465 0 38477824 32137880 0 14663680 0 13:40:01 16572672 10204085 0 99106816 80782488 0 15310848
0 14:00:00 0 14:20:00 0 14:40:01 0 15:00:00 0 15:20:00 0 Average 0
16589056 10261693 16589056 10259613 16589056 10260061 16589056 10267477 16589056 10274757 14813733 9300022
0 99106816 80797968 0 99106816 80736600 0 99106816 80820088 0 99106816 80902432 0 99106816 80864920 0 90445528 73863192 0
0 0 0 0 0
15343616 15343616 15343616 15343616 15343616 15241801
Where possible, the output above has been trimmed to limit the amount of data shown (not all disk stats are shown, for example). For more information on SAR, check the Resources section and the manual page for your system. Back to top Summary While there is not always a direct correlation between a slow UNIX system and the statistical information that you can extract, your first job when discovering a slow system should be to collect as much information as possible. Whether you do this actively (through ps, uptime and other tools) or passively (through SAR or top) depends on the situation. Armed with that information, you should be able to tell whether your UNIX system is slow because it is overworked (CPU over usage), low on physical memory (large amounts of swapping), or there is a problem with a runaway process (high single process CPU time).
System Administration Toolkit: Problems and pitfalls

It's a trap
Chris Herborth (chrish@pobox.com), Freelance Writer, Author
Summary: Avoid common pitfalls and traps to help keep your systems running smoothly. Knowing the right way of dealing with full disks, or a crippled system, is nearly as important as having tools in your arsenal to make sure you're prepared to react quickly to missing files or an insecure system. This article focuses on some of the most common problems and issues facing UNIX administrators and ways to achieve a safe and effective resolution. View more content in this series Tags for this article: administrators, aix, aix_and_unix, problems, systems_admin, toolkit
Date: 14 Nov 2006 Level: Intermediate Also available in: Chinese Russian Activity: 8550 views Comments: 0 (View | Add comment - Sign in)
About this series The typical UNIX administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment. Back to top Deleting open log files In the course of your administrative duties, you might notice a system starting to run low on disk space. If it's a critical system, it won't be easy to take the machine down and add more storage, and you're probably already using the quota system to keep individual users from soaking up too much disk space. It's only natural to start looking
around for things that can be deleted, archived to another system, or some offline storage. Log files are frequently an early target in this process, as the busy /tmp and /var file systems often have limited space. (Listing 1 shows you /tmp and /var on my iBook, which isn't running any busy services.) Certain services, such as Web servers, Java2 Enterprise Edition (Java EE) Web applications, and databases, can create enormous logs, especially if someone has configured them in debug mode.
Listing 1. /tmp and /var can accumulate a lot of data, even on a personal workstation
chrish@Bender [530]$ sudo du -sh /tmp/ /var/ 44K /tmp/ 1.0G /var/
After verifying that nobody needs the log data, you fire off a quick rm command and blow them away. But you don't recover any disk space. If you're not familiar with the semantics of the UNIX filesystem, you might think the machine needs a reboot and a potentially time consuming file system integrity check (using the fsck command in single-user mode). On a standard UNIX filesystem, you can delete a file while it's still open for reading or writing. The name of the file is removed from the file system, and its storage space is recovered by the operating system when the program using it closes the file. This feature is often used by programs creating temporary files; they create the file, open it, and delete it. The file auto deletes if the program crashes or exits normally, so the programmer doesn't need to worry about closing the file or removing it later. This matters because your space-consuming log file is being held open by the server that writes to it. Deleting it just removes the name from the file system, and it won't recover any drive space until the process exits or closes the file. To get around this problem, you could restart the service that owns the log file, although this service interruption might cause a rebellion among your users. Another option is to rename the log file, and then tell the process to reload its configuration files. Any existing processing continues until normal completion using the open log file, and any new requests will be logged to a new log file created using the old name. By convention, most server processes (all of the useful ones) reset themselves and reload their configuration files when you send a hangup signal (signal 1 or HUP). Listing 2 shows one way of sending the hangup signal to all of the Web server processes currently running.
Listing 2. Telling the Web server to reload its configuration file and reset its files
chrish@Bender [507]$ ps -A | grep httpd | grep -v grep | \ awk '{ print $1; }' | xargs -L 1 sudo kill -HUP Password:
That's a bit of a mouth full, so let's take a look at each part of that pipe. The ps and grep commands search through all processes for the httpd (and skip the grep process that's looking for the httpd processes). Next, awk reduces the output to the process ID, which you feed into xargs. The xargs command then takes each process ID (since you used L 1 to grab one line at a time), and uses sudo kill -HUP to send each one a hangup signal. Back to top Deleting critical files A sure-fire way to mess up a working system is to accidentally delete some critical files. Shared libraries, executables, or vital system configuration files are especially vulnerable to this sort of accident. One way to avoid this problem is to never log in to the system as the root user (see the Logging in as root section). Regular users can't destroy vital system files unless you've been mucking with the standard permissions. Another way is to make the directories read-only by removing the write bit (see Listing 3).
Listing 3. Making important directories read-only

chrish@Bender [541]$ cd /etc chrish@Bender [542]$ sudo find -d . -type d | xargs sudo chmod -w
You can use the find -d option to do a depth-first search of directories (-type d is also specified), and then use xargs and chmod to remove the write bit, rendering each directory read-only. This stops everyone from creating new files and, more importantly, from deleting existing files. It won't stop people with the appropriate permissions (that is, you) from editing existing files. Be very careful with this! If you have a poorly designed application that requires a writable directory, it could start failing with surprising error messages. Most programs
confine their automatic file creation and deletion to /tmp and /var. Keep in mind that you'll have to put the write-bit back on (same process, but use u+w instead of -w in the command from Listing 3) when installing new software that needs to drop a configuration file or something into the read-only directory. Back to top Coping with a crippled system There are many ways to cripple a system, but most of them are going to require access to the system console to help repair things. If the system has been crippled by run-away processes (see the "Monitoring a Slow System" article in this series; there's a link in the Resources section) consuming all of the available process slots or eating up so much memory that the machine is spending all of its time swapping to disk, you'll either need to kill the offending process or processes or, if you can't even log in and execute a kill command, restart the machine. If you can access the system but can't kill the offending process for whatever reason, shutting down to single-user mode stops all non-essential services and any user-run processes. To send the machine into single-user mode, you'll use the telinit for System V-based UNIXes (sudo telinit 1) or the shutdown command for operating systems that grew from Berkeley Software Distribution (BSD) (sudo shutdown now). When you're done fixing things in single-user mode, the easiest way to get back to normal is to restart the machine; again, depending on your system's heritage, you'll use either telinit (telinit 6) or shutdown (shutdown -r now) to reboot. In the worst case, your system might be damaged enough that you'll have to boot from the operating system installation media or a rescue disc. These almost always provide a minimal single-user environment that you can use to run disk checks (fsck), check for system compromises, or restore damaged files from backup. You do keep backups, right? A good backup strategy saves you a lot of work when things get messed up, and it makes clumsy users (who can't seem to stop deleting their important files) very happy to see you. Back to top Keeping sequenced files and archives Having a file sometimes just isn't good enough; you need the last version or the version from last week. This could be something easy, such as someone in human resources
overwriting the only copy of the paycheck processing file, or it might be something more exciting, such as vital system configuration files. Keeping incremental backups of the system's important (and user) files is an important way of preventing this sort of disaster. Have you ever deleted or overwritten an important file? No problem, all you need to do is grab the version from last night from the incremental backup. Listing 4 (which I've called newer-archive.sh on my system) shows you a simple shell script that creates an archive of files that are newer than a specified file. You can use this to create incremental backups of files that are newer than the last incremental backup.
Listing 4. A simple incremental archive script

#!/bin/sh # # Make an incremental archive containing files that have been # modified since the last archive was created. # # Usage: # # newer-archive.sh -o new-file.tar -nt old-file files old_file="" new_file="" files="" archiver="tar -T - -czpsSf" while [ "$1" != "" ] ; do case $1 in -o) new_file=$2 shift ;; -nt) old_file=$2 shift ;; *) files="$files $1" ;; esac shift done for path in $files ; do find $path -newer $old_file done | $archiver $new_file
The -o option specifies the output file, and the -nt option specifies the file that you should use as a baseline; any files that are newer than this one will be added to the archive. After the options, files, or directories you want to archive can be listed, you'll need to add them all to the output file. You can modify this to work with any sort of archiver, assuming you can find a way to feed it a list of files to archive through a pipe. You might also need to tweak the tar options specified here if your system doesn't have GNU tar installed. You can combine this script with the date command (see Listing 5) to create an archive with the current date and time in it.
Listing 5. Using date to specify a backup archive name

chrish@Bender [525]$ sudo ~/bin/newer-archive.sh -o incremental-$(date +%Y-%m-%d-%H.%M.%S).tar.gz -nt incremental-2006-09-06-11.15.03.tar.gz /Users
By using the + option from date to specify a different output format (year-month-dayhour.minute.second), you can create a filename that incorporates the current date and time, use the last incremental backup (which is a month out of date that's too long between backups) as the reference old file, and back up all the new or modified user data. Another option is to use RCS's ci and co commands to create a change history for each file. You can also use ci to check in a file. This creates a history file (the ci filename creates another filename that contains the file's history and its older revisions), and it sets the file to read-only. Use co -l to check out the file and make it writable again. After you're done making changes, check the file back in with a meaningful change log message (see Listing 6).
Listing 6. Using RCS to keep track of file versions

chrish@Bender [536]$ ci -u points.txt points.txt,v <-- points.txt enter description, terminated with single '.' or end of file: NOTE: This is NOT the log message! >> important points to cover in the article >> . initial revision: 1.1 done chrish@Bender [537]$ dir points.txt -r--r--r-1 chrish chrish 170 Oct 6 14:34 points.txt chrish@Bender [538]$ co -l points.txt
points.txt,v --> points.txt revision 1.1 (locked) done chrish@Bender [539]$ vi points.txt chrish@Bender [540]$ ci -u points.txt points.txt,v <-- points.txt new revision: 1.2; previous revision: 1.1 enter log message, terminated with single '.' or end of file: >> added another important point >> . done
The ci command's -u option automatically checks out a read-only version of the file when you check it in. The co command's -l option locks the file so that you (and only you) can edit it. RCS only works well with plain text files; if you need to keep older versions of binary files, look into something more powerful, such as Subversion (see Resources). Back to top Creating users or groups On most systems, adding a new user or group seems like a simple matter of editing the /etc/passwd file (and possibly the shadow password file, which actually contains the passwords) or /etc/group file using your favorite text editor. It's easy to remember, the file format isn't that challenging, and it'll be quick. There are a number of reasons why you want to avoid doing this, and they're the reasons why most UNIX systems, especially modern ones, ship with a tool for creating new users and groups. Editing these vital system files can cause havoc. Sure they're simple, but it's still easy to get distracted and mess something up. Maybe your editor of choice locks files while you're working, which could prevent anyone from logging in while you're editing. You're also left with a pile of work after adding a user by hand. You need to create a new home directory, fill it with the standard home directory goodies, add them to all of the appropriate groups, and create system-level things, such as a mail spool for the new user. Why give yourself more work? The user and group creation tools are there to save you time and effort (and make sure nothing gets messed up, which helps you maintain your guru-like reputation). Most standard UNIX systems have adduser (or useradd) and addgroup (or groupadd) commands available to the administrator. Many Linux distributions have handy
graphical tools (such as Fedora Core's User Manager), and FreeBSD's comprehensive sysinstall utility also handles user and group creation. On Mac OS X, you'll use the Accounts preferences to create users, and you'll use the NetInfo Manager to create new groups. Back to top Logging in as root As you know, root has all the power in a UNIX system. The root user can do anything and, as they say, "with great power comes great responsibility." And yet, some people insist on logging in as root all the time, even if they're not doing anything that requires all of this power. Always, always create (and use!) a regular user account for yourself on any system where you have root access when you need to do something that requires root privileges. Use the system's su (see Listing 7) or sudo (see Listing 8) command (whichever is available on your system) to temporarily become root.
Listing 7. Temporarily becoming root with the su command

chrish@Bender [514]$ su Password: #
Listing 8 uses the sudo command to run a command as root.
Listing 8. Running a command as root with the sudo command

chrish@Bender [517]$ sudo id Password: uid=0(root) gid=0(wheel) groups=0(wheel), 1(daemon), 2(kmem), 3(sys), 4(tty), 29(certusers), 5(operator), 80(admin), 20(staff)
Why avoid running as root? One errant rm command, or accidentally unpacking a tarball into the wrong spot, and your system might be damaged enough to require some major repair work. Back to top Securing systems
The systems on your network need to be secured; there's no doubt about that. Leaving a server or router with the default passwords is practically an invitation for unscrupulous (or even just curious) people to start poking around. This could result in a damaged system, either intentionally or accidentally messed up by the intruder, or worse, a compromised system, secretly modified to distribute spam, stolen software, or who knows what. A good policy for securing your systems is to start by denying everything instead of allowing everything. Specifically, turn off every network service you don't actually need, block every incoming network port except for services you really want exposed to the random populace of the Internet (or your local area network (LAN) if you're behind a firewall), and remove all users who aren't actually using the system. From there, you can re-deploy services, open network ports, and add users as necessary. This might seem like extra work, but it lets you know exactly what's going on with the system. Having a reasonable password policy, if you can, also enhances security. A lot of corporate environments have password policies that actually encourage bad password behavior by requiring frequent password changes. If a user has a strong password, making them change their password too often increases the likelihood that they'll either forget their new password (thus creating work for your helpdesk), or that they'll write the password down and keep it near the machine. You'd be amazed at the number of people who keep a list of their highly-secure current passwords on a sticky note under the keyboard or mouse pad. Tools, such as the powerful crack -- it's a dictionary-based password tester -- (see Resources for a Wikipedia entry), can also help you weed out weak passwords by testing a password file or other password store against words, common misspellings of words (such as l33t speak), and other word-mangling algorithms (see Listing 9).
Listing 9. Using crack to check for weak passwords

chrish@Bender [503]$ cp /etc/passwd . chrish@Bender [504]$ sudo crack passwd
Note that you're operating on a copy of the password file (your systems passwords might be in /etc/shadow or a centralized store on the network), not the live file. You never know when things might go wrong! Back to top Summary
This article showed you several ways of dealing with a range of system administration traps, and it showed you how the most obvious solution to a problem might not be the right one. You should now be able to deal with common disasters without losing your cool, and you'll be able to prepare in advance for problems by keeping incremental backups and having a secure system. You'll also save time by taking advantage of the tools available on your UNIX system of choice.

Linux Shell - Ibm

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Linux Shell - Ibm

Uploaded by

Copyright:

Available Formats

System Administration Toolkit: Process administration tricks

Martin Brown (mc@mcslp.com), Freelance Writer, Freelance Developer

Listing 1. Listing process on BSD UNIX variants

Listing 2. Listing process on BSD UNIX variants

PID 19915 29145 32256 26986 31303 31358

TTY pts/3 pts/3 pts/3 pts/3 pts/3 pts/3

TIME 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00

CMD bash emacs emacs xterm ftp ps

Listing 3. Listing all processes on BSD systems

Listing 4. Process listings in SVR4 environments

Listing 5. Extending the output

15:58:17 ? 16:01:47 pts/1 15:57:14 ? 15:58:17 ? 15:57:47 ? 15:57:22 ? 15:57:24 ?

Listing 6. Showing more process detail in BSD

p8 0:00.03 bash p9 0:00.01 pa 0:00.02 bash

Listing 7. Selecting specific columns in BSD

Listing 8. Selecting specific columns in SVR4

Listing 9. Tracking usage by CPU in BSD

Listing 10. Tracking usage by CPU in SVR4

Listing 11. Listing processes by user

Listing 12. Listing by terminal

It is easier to understand if you dissect the individual elements:

Listing 13. Killing processes by command string

Listing 14. Calculating memory usage with ps and awk

Listing 15. Using the jobs command

Listing 16. Warning that process has been temporarily suspended

System Administration Toolkit: Migrating and moving UNIX filesystems

Listing 1. Example file/disk layout and usage

Listing 2. Moving the /usr directory from the example system

Listing 3. Changing the target directory

Listing 4. Copying the files to the new directory

Listing 5. Verifying that the files have copied over

1024 Apr 20 13:11 4lib/ 5 Apr 20 12:40 5bin -> ./bin/

9 512 3 10 512 512 512 512 16384 512

Listing 6. Using tar for copying

Listing 7. Process for copying

Listing 8. Printout of copy process

Listing 9. Creating a .tar file

3. Rename the source directory (see Listing 10).

Listing 10. Renaming the source directory

Listing 11. Extracting the .tar file

Listing 12. Changing the partition for an existing file system

Listing 13. Checking the output of du

Listing 14. Comparing the number of files/directories on source and destination

Listing 15. Comparing inode numbers

IFree IUse% Mounted on 565321 43% /

The example in Listing 16 is on Solaris.

Listing 16. Comparing inode numbers on Solaris

(/dev/dsk/c0t0d0s3 ): 9076010 blocks

Systems Administration Toolkit: Using SNMP data

Listing 1. SNMP object IDs

Listing 2. 'Walking' an SNMP tree

Listing 3. Simplified information

Listing 4. Getting system description and uptime information

Listing 6. List of uptimes for each host

Listing 7. Getting a running view of SNMP properties

Listing 8. Getting a dump of available MIB data

Listing 9. Using snmptranslate

Listing 10. Monitoring disk space automatically

Listing 11. Sample snmpd.conf file

Systems Administration Toolkit: Log file basics

Listing 1. Linux /var/log directory contents

Listing 2. Traditional UNIX /var/adm contents

Listing 3. Additional logs in /var/log on Solaris

Listing 4. Sample syslog.conf file