Tutorial to use a Computing Cluster with Sun Grid

Engine software (SGE)

Alvaro Sebastian Yague
based on the notes written by
Biostatistics Information Technology (BIT) Committee
Johns Hopkins Bloomberg School of Public Health, Dept. of Biostatistics

17 September 2013

Contents
1 Introduction

2 Connecting to the cluster (ssh)

2.1 Installing and executing an SSH client . . . . . . . . . . . . . . . . . . . . . . .
2.2 Access Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2
3
3

3 Transferring files to/from the cluster (scp)

3.1 SCP protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4
4

4 Running jobs on cluster nodes

4.1 Running regular jobs (qrsh) . . . . . . .
4.2 Batch jobs . . . . . . . . . . . . . . . . .
4.3 Submitting batch jobs (qsub) . . . . . .
4.4 Checking job status (qstat) . . . . . . . .
4.5 Deleting jobs (qdel) . . . . . . . . . . . .
4.6 Errors and output from batch jobs . . .
4.7 Specifying memory needs . . . . . . . . .
4.8 Checking your jobs memory usage . . .
4.9 Requesting multiple slots for a multi-core
4.10 Using a default specification file . . . . .

. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
job .
. . .

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

5
5
6
7
9
10
11
11
12
13
13

Introduction

This tutorial is mostly based in the documentation1 written by the Biostatistics Information
Technology (BIT) Committee of the Johns Hopkins Bloomberg School of Public Health, Dept.
of Biostatistics.
1

http://www.biostat.jhsph.edu/bit/cluster-usage.html

The first step to be able to use a Computing Cluster is to ask for access to the system administrators. Usually you will have to formally request to your research center or faculty for
it. After that you will be provided with an user, password and the domain name or IP of the
cluster. Read section 2 Connecting to the cluster (ssh) to learn how to use the login data.
Computer clusters are useful to perform high-performance calculations, that means calculations that will require a normal computer running during several days, weeks or months. If a
calculation last only several hours it is not worth to use a computer cluster. Also it is important to highlight that it is required some advanced knowledge of computer science to use the
cluster infraestructure, average computer users without previous training will be not able to
use properly the cluster neither to take advantage of it.
Computer clusters can be used for research in physics, chemistry, computer science, statistics,
engineering, aeronautics, computational biology and bioinformatics, among other topics.
It is important to know how many nodes, cores (or slots) and RAM memory has the cluster.
Each node is like an individual computer that will have several cores (commonly called processors or slots) and some amount of RAM memory that should be shared among all the cores. For
example, a cluster with 20 nodes of 40 cores and 60GB per node will have in total 800 cores and
1.2TB of RAM. In this example each core will be able to use as maximum 60GB, but if we have
the 40 cores working simultaneously, each core will be allowed to use only 1.5GB. Furthermore,
clusters are not homogeneous and they can have nodes with different amounts of cores and
memory. This topic will be explained later in subsection 4.7 Specifying memory needs when
we will have to assign memory limits to our processes. In subsection 4.9 Requesting multiple
slots for a multi-core job it will be explained how to request many cores for our jobs. We
will call job to every single process, task, program, script or interactive session that we run or
execute in the cluster.
In this tutorial we will deal with everything related to job submission, scheduling, and execution
on a cluster under the control of Sun Grid Engine software2 (SGE). The Grid Engine project
was sponsored by Sun Microsystems 3 as an open source community effort to facilitate the
adoption of distributed computing solutions. Among other things, SGE can be used to limit
the total number of slots each user is allowed to run simultaneously on the cluster. By default
in SGE, one job uses one slot, but see Requesting multiple SGE slots for a multi-core job. You
may submit jobs which request more than the slot limit and they will all be queued to run as
your other jobs finish. When the cluster nodes are all at maximum capacity, jobs waiting to
run may be subject to a functional share priority algorithm as we have defined it using SGE.

Connecting to the cluster (ssh)

Most of the computing clusters use the Secure Shell protocol to connect from a remote computer
(from example at work or home).
Secure Shell (SSH) is a cryptographic network protocol for secure data communication, remote
command-line login, remote command execution, and other secure network services between
two networked computers that connects, via a secure channel over an insecure network, a
server and a client (running SSH server and SSH client programs, respectively). The bestknown application of the protocol is for access to shell accounts on Unix-like operating systems,
2
3

http://gridengine.sunsource.net
http://www.sun.com/gridware

but it can also be used in a similar fashion for accounts on Windows. It was designed as a
replacement for Telnet and other insecure remote shell protocols such as the Berkeley rsh
and rexec protocols, which send information, notably passwords, in plaintext, rendering them
susceptible to interception and disclosure using packet analysis. The encryption used by SSH
is intended to provide confidentiality and integrity of data over an unsecured network, such as
the Internet.

2.1

Installing and executing an SSH client

To be able to use this protocol you need to install a SSH client in your computer. There are
many options for Windows, perhaps the most popular and free is Putty 4 . For Linux systems
OpenSSH5 is the prefered one, installing it in an Ubuntu system is very easy (other distributions
have similar ways to install):
$ sudo aptg e t i n s t a l l opensshc l i e n t

To access into the cluster you have to execute the SSH client indicating your username and the
IP or Domain Name of the SSH server (the cluster), also the connection port if different than
default 22:
$ s s h p 2222 u s e r n a m e @ c l u s t e r . domain . com

The first time that you connect, the following message will be shown, you should answer yes
and push intro key:
The authenticity of host cluster.domain.com (123.4.56.789) cant be established.
RSA key fingerprint is 3c:6d:5c:99:5d:b5:c6:25:5a:d3:78:8e:d2:f5:7a:01.
Are you sure you want to continue connecting (yes/no)?

You will be prompted to enter your password. You should enter the password provided by the
cluster administrator.
If you have any problem following the former instructions, you can try to connect in verbose
mode to try to find more info about errors:
$ s s h vvv p 2222 u s e r n a m e @ c l u s t e r . domain . com

2.2

Access Node

After login into the remote machine you will enter in a remote command line like this:
[ username@headnode ] $

That means that you are logged into the cluster as username and that you are in the shell of the
headnode (or access node). But the access node is only used for login, DO NOT RUN JOBS
IN ACCESS NODE!. This machine is not for doing any sort of computation. Rather, it is
ONLY for managing files, text editing and for submitting jobs to the whole cluster. Usually
long-running jobs found on access node (R BATCH jobs, for example) are killed without notice
by the system administrator and you will lose any data and/or computations associated with
the running job. So how to run any program in the cluster? This answer will be answered in
4
5

http://www.putty.org
http://www.openssh.com

the following section Running jobs on cluster nodes. Now we will talk about how to copy
files between our local machine and the cluster.

Transferring files to/from the cluster (scp)

There are two widely used data transfer protocols to copy files between remote computers, they
are SCP (Secure Copy Protocol) and FTP (File Transfer Protocol). There are some differences
between them:
SCP: encrypted; login with username/password; uses port 22 by default; allows file transfers
to any folder with privilegies.
FTP: plaintext; login with username/password; uses ports 20, 21 by default; is usually restricted to a FTP folder on server.
To be able to use one of them, it must be installed in the remote machine as server and you
have to install in your computer the client utility (in a similar way than SSH client).

3.1

SCP protocol

SCP protocol is included in the OpenSSH6 client explained in the subsection 2.1. Copying files
between computers with SCP is similar to the SSH execution (subsection 2.1). You must specify
username and the IP or Domain Name of the SSH server (the cluster), also the connection port
if different than default 22. Local and remote folder are the scp command arguments and they
must be in the right order, first origin and then destination folder for the file copy:
$ s c p P 2222 u s e r n a m e @ c l u s t e r . domain . com : [ ORIGIN FOLDER] [ DESTINATION
FOLDER]

If you desire to copy a file from your local home ( /) to the remote cluster home directory:
$ s c p P 2222 /home/ l o c a l u s e r / f i l e t o c o p y
u s e r n a m e @ c l u s t e r . domain . com : / home/ username /

or easier (because home dir is the default copy dir):

$ s c p P 2222 / f i l e t o c o p y u s e r n a m e @ c l u s t e r . domain . com

If you desire to copy a file from remote to local home dirs:

[ username@headnode ] $ s c p P 2222
u s e r n a m e @ c l u s t e r . domain . com : / home/ username / f i l e t o c o p y /home/ l o c a l u s e r /

or:
[ username@headnode ] $ s c p P 2222
u s e r n a m e @ c l u s t e r . domain . com : / home/ username / f i l e t o c o p y /

Finally it exists an interesting option (-r) that allows to copy files and folders recursively between
the machines:
6

http://www.openssh.com

$ s c p P 2222 r /home/ l o c a l u s e r / f o l d e r t o c o p y
u s e r n a m e @ c l u s t e r . domain . com : / home/ username /

or:
$ s c p P 2222 r / f o l d e r t o c o p y u s e r n a m e @ c l u s t e r . domain . com : /

The whole content of the local folder to copy, files and subfolders, will be copied to the
remote home dir.

Running jobs on cluster nodes

SGE comes with a number of command line tools to help you manage your jobs on the cluster.
Most of the time you can get away with just knowing a few commands. The most immediately
relevant ones are:
qrsh: run a program or open an interactive session on the cluster.
qsub: submit a batch job to the cluster.
qstat: query the status of your jobs on the cluster (or look at all jobs running on the cluster).
qdel: delete a job from the cluster/queue.

4.1

Running regular jobs (qrsh)

While you should not run computations on the access node as explained before (subsection 2.2),
you can run regular programs and interactive jobs on a cluster node. This can be done using
the SGE command qrsh. This command essentially opens up a remote shell or runs a program
on a cluster node.
qrsh command is an equivalent of ssh but it only can be used inside the cluster and it does not
need any parameters. qrsh command execution implies to jump to a cluster node shell to run
or commands there without disturbing the access node. By default qrsh will use only one slot
(core) in some of the configured nodes for this task.
qrsh execution is very easy, just type the command in the access node shell:
[ username@headnode ] $ q r s h

The command prompt will change to something like this:

[ username@n30 ] $

That means that now you are using one slot in the node 30. In this new remote shell you can
run a regular program or even an interactive one like an R session:
[ username@n30 ] $ R

Now we can run programs, interactive jobs and short computations without affecting the access
node (If dozens of users were running programs at the same time in access node it would collapse

easily). Now you can run whatever program you want. For example, you can run R interactive
shell or compiling some code.
However, you must remember to logout (exit or CTRL D). Otherwise, you will be taking up
a slot in the queue which will not be available to others. To come back to the access node, just
type:
[ username@n30 ] $ e x i t

qrsh is also able to run a program without opening a shell:

[ username@headnode ] $ q r s h cwd python s c r i p t . py

The -cwd option tells SGE to execute the batch.sh script on the cluster from the current
working directory (otherwise, it will run from your home directory, which is probably not what
you want).
NOTE: You may also specify memory requirements, multi-core or special queues on your qrsh
command just as you do on the qsub command (subsection 4.7) and subsection 4.9).
NOTE: DO NOT run background or nohup jobs while using qrsh. Sun Grid Engine (SGE)
must know about your job/session so that it can manage and account for cluster resources. By
default, SGE assumes one slot (corresponding to one CPU core) for each qrsh session. If you
still have running programs and no session appears for you in qstat (subsection 4.4), then you
have done something that is not appropriate for the way the HPSCC Cluster is managed.
NOTE: If you encounter an error while running a program interactively on a cluster node and
your program crashes, it still might be in the clusters process queue. If you dont quit out of
your program normally, make sure to check the cluster queue (via qstat, (subsection 4.4)) and
see if your (interactive) job is still there. If it is, get the job-ID and kill the job using qdel.

4.2

Batch jobs

Batch processing or job scheduling is execution of a series of programs (jobs) on a computer

without manual intervention. So, all input data are preselected through scripts, command-line
parameters, or job control language. This is in contrast to interactive jobs which prompt the
user for such input. In batch processing a program takes a set of data files as input, processes
the data, and produces a set of output data files.
Batch processing has some benefits:
It can shift the time of job processing to when the computing resources are less busy.
It avoids idling the computing resources with minute-by-minute manual intervention and
supervision.
By keeping high overall rate of utilization, it amortizes the computer, especially an expensive one.
It allows the system to use different priorities for batch and interactive work.
As we have indicated, the cluster uses Suns Grid Engine (SGE) to control scheduling of jobs
(section 1). SGE comes with a number of command line tools to help you manage your jobs
6

on the cluster. Most of the time you can get away with just knowing a few commands. The
most immediately relevant ones are:
qsub: submit a batch job to the cluster.
qstat: query the status of your jobs on the cluster (or look at all jobs running on the cluster).
qdel: delete a job from the cluster/queue.

4.3

Submitting batch jobs (qsub)

qsub is the command to submit batch jobs. If you specify the option -b y qsub runs binaries
like qrsh does by default (subsection 4.1), the only condition is that the program should not be
interactive:
[ username@headnode ] $ qsub cwd b y python s c r i p t . py
[ username@headnode ] $ qsub cwd b y R CMD BATCH s c r i p t .R

The -cwd option tells SGE to execute the batch.sh script on the cluster from the current
working directory (otherwise, it will run from your home directory, which is probably not what
you want).
By default (without -b y) qsub cannot run binaries and requires that every job that you submit
to the cluster must be wrapped in a shell script. Not to worry, wrapping your program with
a shell script is not as difficult as it might sound! Below are instructions for how to run a R
batch job on the cluster.
There are two basic steps to submit a R BATCH job to the cluster:
First, assuming you have an R program in a file named script.R , you need to create a shell
script file that will invoke and run your R program when it is submitted to the cluster. Lets
call this new file batch.sh. If batch.sh is in the same directory as script.R, batch.sh should
content this line:
R CMD BATCH s c r i p t .R

The file might have other lines in it to specify SGE job options or commands to run before or
after R CMD BATCH script.R line.
Once youve written your short batch.sh file you could submit it to the cluster via the command:
$ qsub cwd batch . sh

After submitting your job with qsub, you can use the qstat command to see the status of your
job(s) (subsection 4.4).
The command qsub can be followed my additional options, the most important are listed here:
-help: Prints a listing of all options.
-N name: The name of the job.
-a date time: Defines or redefines the time and date at which a job is eligible for execution.
-cwd: Execute the job from the current working directory.
7

-wd working dir: Execute the job from the directory specified in working dir.
-p priority: Defines or redefines the priority of the job relative to other jobs.Priority is an
integer in the range -1023 to 1024. The default priority value for jobs is 0.
-l resource=value,...: Specific resource request list. Will be explained in subsection 4.7.
-pe parallel environment: Will be explained in subsection 4.9.
For example, the following command will submit a job named myjob to be executed on Dec
24 at 12:00 from the directory /home/username/qsub:
$ qsub N jobname a 12241200 wd /home/ username / qsub batch . sh

Qsub command options can be also embedded in the shell script. The following shell script is
intended to run a python script, but before it contains additional lines that specify additional
batch running options, notice that there is one option per line and they are preceded by #$,
lines startint in # are comments:
#! / b i n / sh
# Which a c c o u n t t o be c h a r g e d cpu time
#$ A username
# date time t o run , f o r m a t [ [ CC] yy ]MMDDhhmm[ . SS ]
#$ a 12241200
# 6 or more p a r a l l e l p r o c e s s e s t h a t r e q u i r e 128M o f memory
#$ pe pvm 6 l mem=128
# Send m a i l t o t h e s e u s e r s
#$ M username1@host , username1@host
# Mail a t b e g i n n i n g / end /on s u s p e n s i o n
#$ m b e s
# Export t h e s e e n v i r o n m e n t a l v a r i a b l e s
#$ v PVM ROOT,FOOBAR=BAR
# The j o b i s l o c a t e d i n t h e c u r r e n t w o r k i n g d i r e c t o r y .
#$ cwd
python s c r i p t . py

You can look at the manual page for qsub (type man qsub at the prompt) to get more
information on qsub additional parameters.
NOTE: It is important to remember that the qsub command MAY NOT BE directly followed
by an executable program (a binary file). For example: qsub python script.py. Doing so will
result in your program not running. Your job will be assigned a job id and will be appear if you
immediately do a qstat command. But, after a minute or so, the job will cancel and a message
will be written in an error file. The correct command would be qsub -b y python script.py.
NOTE: You do not have to put an & at the end of the line. qsub automatically sends your
job to the cluster and returns to your command line prompt so that you can do other things.

4.4

Checking job status (qstat)

qstat is the command to query the status of the batch jobs on the cluster.
While you are logged into a cluster node via qrsh, if you start another session and run qstat
form the access node:
q s t a t u username

you will obtain something like the following:

job-ID prior
name
user
state submit/start at
queue
slots
----------------------------------------------------------------------------------------141 0.55500 QRLOGIN
username
r
09/17/2013 14:12:16 all.q@n29.acid
1

The job labeled QRLOGIN is the qrsh interactive session.

But the main utility of qstat is checking the status of jobs submitted via qsub. By default, qstat
with no arguments shows the status of your current cluster jos:
qstat

If you have no jobs running on the cluster qstat will print nothing, but if you do, the results
will look something like:
job-ID prior
name
user
state submit/start at
queue
slots
----------------------------------------------------------------------------------------143 0.00000 jobname2
username
qw
09/17/2013 14:25:10
1
142 0.00000 jobname1
username
r
09/17/2013 14:23:24
1
141 0.55500 QRLOGIN
username
r
09/17/2013 14:12:16 all.q@n31.acid
1

Under the state column you can see the status of your job. Some of the codes are:
r: the job is running.
t: the job is being transferred to a cluster node.
qw: the job is queued (and not running yet).
Eqw: an error occurred with the job .
You can look at the manual page for qstat (type man qstat at the prompt) to get more
information on the state codes.
Another important thing to note is the job-ID for your job. You need to know this if you ever
want to make changes to your job. For example, to delete your job from the cluster as will be
explained in the next section.
Running qstat with the option -j job-ID we can obtain more information about a submitted
job
q s t a t j 142
==============================================================
job_number:
142
exec_file:
job_scripts/142
submission_time:
Tue Sep 17 14:23:24 2013
owner:
username
uid:
5032

group:
gid:
sge_o_home:
sge_o_log_name:
sge_o_path:
sge_o_shell:
sge_o_workdir:
sge_o_host:
account:
hard resource_list:
mail_list:
notify:
job_name:
jobshare:
hard_queue_list:
env_list:
script_file:

Ldap
2000
/home/username
asebastian
/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin
/bin/bash
/home/username/qsub
headnode
sge
h_vmem=2G,mem_free=1536M
username@host
FALSE
jobname1
0
all.q
batch.sh

The command qstat -f will show a detailed information about all the cluster nodes (total and
used cores per node, load average, architechture) and the assignations of our jobs.
q s t a t f
queuename
qtype resv/used/tot. load_avg arch
states
--------------------------------------------------------------------------------all.q@n29.acid
BIP
0/0/64
0.00
linux-x64
--------------------------------------------------------------------------------all.q@n30.acid
BIP
0/32/64
0.51
linux-x64
--------------------------------------------------------------------------------all.q@n31.acid
BIP
0/1/64
0.01
linux-x64
142 0.55500 jobname1 username
r
09/17/2013 14:23:24
1
--------------------------------------------------------------------------------all.q@n32.acid
BIP
0/0/64
0.02
linux-x64
---------------------------------------------------------------------------------

You can also be notified via email when your jobs status changes. This is done with the options
-m and -M, for example to receive an email when the job ends:
qsub m e M username@domain . com batch . sh

There are more notifying options available, you can find them in the qsub manual page (man
qsub)

4.5

Deleting jobs (qdel)

qdel is the command to delete a job from the cluster/queue. To delete a job from the cluster,
you can obtain the job-ID running qstat (subsection 4.4) and execute:
q d e l 142

Now it should not appear in the list of jobs retrieved with qstat (subsection 4.4):
job-ID prior
name
user
state submit/start at
queue
slots
----------------------------------------------------------------------------------------143 0.00000 jobname2
username
qw
09/17/2013 14:25:10
1
141 0.55500 QRLOGIN
username
r
09/17/2013 14:12:16 all.q@n29.acid
1

10

4.6

Errors and output from batch jobs

By default the standard output and the standard error stream of the job are stored in two files
in the home directorory or in the path specified by the qsub options -cwd and -wd:
[jobname].o[job-ID]: standard output (E.g. jobname.o142)
[jobname].e[job-ID]: standard error (E.g. jobname.e142)
The paths used for the standard output and error files can be chosen with the qsub options -o
and -e respectively.

4.7

Specifying memory needs

When submitting your job(s), if you did not specify any memory requirements, SGE would
choose the cluster node(s) with the lowest CPU load WITHOUT REGARD TO MEMORY
AVAILABILITY (subject to other scheduling parameters which we have defined). It is, therefore, IMPORTANT to specify your expected memory requirements when submitting cluster
jobs. After calculating approximately how much memory your job will need, you should add a
memory resource requirement to your qsub (or qrsh) command.
Here is one example explaining why you should specify memory requeriment even if you expect
your job to use as little as 3G of memory. It is possible that a user has requested 18G of
memory and is using all of that or more on a 20G node. That user may be the only one on that
node and if it has 8 cpu-cores there could be 7 slots open. If you send in that moment a job
without memory requirement, your job might go to that node since it looked lightly loaded as
far as jobs. It will happily accept your job and begin swapping and slow down both jobs now
on the node.
To solve this problem system administrators usually define a default minimum memory for each
submitted job. But if you know your job will consume less memory, you should specify a lower
limit to allow more jobs to be ran at the same time.
There are three main parameters to specify resource requirements for a cluster job, they follow
the command option -l and can be grouped together separated by commas:
mem free=MEMORY: is the minimum free RAM memory available on a node required
to send the job. You should use approx what you think your job will need (or a little
more) on the mem free request. This does not reserve memory for your job. It simply
puts your job on a node with that amount of memory currently available. Sometimes
a node will have enough free memory available but it will be assigned to another job
requirements and will not be able for ours.
h vmem=MEMORY: is the maximum amount of memory your job is allowed to use. To
avoid running away at the high end use the h vmem parameter to limit your jobs total
memory use. (It is advisable that all users use this parameter to stop a runaway job from
crashing the node, so as a rule is also set as default by the system administrator).
h fsize=SIZE: is the maximum size of files that can be created by the job. It is useful to
avoid the consequences of any bug in their program that might, under certain conditions,
cause a file to grow without bounds.

11

MEMORY and SIZE parameters can be specified in megabytes M, or gigabytes G.

In the following example, it is required a minimum of 4G of RAM memory on a node to start
the job, the job will be stopped if it consumes more than 6G of memory or create files bigger
than 1 GB. It would behave similarly on a qrsh command.
qsub cwd l mem free=4G, h vmem=6G, h f s i z e =1G batch . sh

A similar example to run an interactive R shell with qrsh in a cluster node with 5G of RAM
available would be:
q r s h l mf=4G, h vmem=6G

To assign qsub memory requirements you must check previously how much memory needs your
job, in the next subsection 4.8 Checking your jobs memory usage it will be explained.
After submitting your job with qsub, remember to use the qstat command to check which queue
(node) your job actually went to (see subsection 4.4).

4.8

Checking your jobs memory usage

To decide the memory requirements for our job it is important to have some tools to check the
memory usage of our job. We should prepare a faster example of our job and do some running
tests with or without qsub.
The simplest tool is the famous Linux command time, it does not only measures how long a
program or script takes to run, but also it can measure the amount of memory used with the
verbose option -v:
/ u s r / b i n / time v batch . sh

Remember to run the script using a qrsh shell in a node different than the main one. The
full path /usr/bin/time must be specified to run verbose mode, and the output will show the
maximum memory required as Maximum resident set size:
Command being timed: "batch.sh"
User time (seconds): 7.56
System time (seconds): 0.66
Percent of CPU this job got: 31%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:26.45
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 806880
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 26
Minor (reclaiming a frame) page faults: 51354
Voluntary context switches: 13921
Involuntary context switches: 1661
Swaps: 0
File system inputs: 2256
File system outputs: 40
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096

12

Exit status: 0

The second option is to run the job using qsub, and after its completion check the memory
usage with the gacct command specifying the job-ID with the -j option:
q a c c t j 142 | g r e p vmem
maxvmem

768.039M

While your qsub job is running you can see its memory usage using the qstat command:
q s t a t j 142 | g r e p vmem

NOTE: qrsh sessions will not report memory usage.

4.9

Requesting multiple slots for a multi-core job

If you have a program (job) that runs in a multi-threaded fashion, using multiple CPU-cores,
then you need to request a parallel environment with multiple slots (where one slot corresponds
to one CPU-core). With the -pe local N qsub option it is possible to request N slots on a
single cluster node for a multi-threaded job or session. At the same time it is recommended to
continue using mem free to select a node that currently has memory available for your job.
The mem free value should be for the total amount memory that you expect your job will
need, but specify the h vmem memory limit as memory limit/N since SGE multiplies it by
N when you ask for multiple slots. In the following example SGE will allocate the job 40 slots
on a cluster node with 40G avaliable and effectively set h vmem to 80G (40x2G).
qsub cwd pe make 40 l mem free=40G, h vmem=2G batch . sh

Note: The local parallel environment is a construct name particular to our cluster ... and
implies that your request is for that many slots on a single cluster node.

4.10

Using a default specification file

One easy way to always have a default limit on your jobs and sessions is to store some default
settings in a file named .sge request in your home directory, for example:
#
# Set defaults for mem_free and h_vmem
-l mem_free=5G,h_vmem=6G
#
# Set the standard value for stack size limit
# (needed for some programs to run properly when h_vmem is set)
-l h_stack=256M
#
# Set a default maximum file size that an SGE job can create
-l h_fsize=10G
#
# Send email when your job ends, for every cluster job that you start (including a qrsh
-M username@domain.com
-m e
13

Other options you use all of the time can also go in this defaults file.
Then, if you do not specify those parameters on your command line, the defaults from the
file .sge request will be used. Obviously, if you need higher limits on a given job you need
to override those defaults with appropriate command line options. You could use -m n on
individual qsub job command lines to suppress email notification for certain jobs. Or better
yet, you might only put the -M your email@jhsph.edu in the .sge request file and simply use
the -m e option on jobs for which you want email notification.
NOTE: You may also invoke the options shown above (and others) by including special lines
at the top of your job shell scripts. Lines beginning with #$ are interpreted as qsub options
for that job, see subsection 4.3.

Acknowledgements
The author would like to thank Tomasz K. Jarzynka, Network Information Services administrator at International Institute of Molecular and Cell Biology, for his valuable comments and
support.

14