Running serial jobs on a parallel queue

The HPCF has traditionally been set up for running parallel jobs. Despite this it is perfectly easy to run a number of serial jobs on a queue set up for parallel ones. When you submit a job to the queuing system you generally have a script which contains everything you want to be run. When the queuing system runs your job it allocates the number of cpus you asked for and runs the script on the first one of these cpus. With a system with two cpus per node such as maxwell if we want to use more than two cpus we need to know which other machines have been allocated to us and connect to them to run jobs on those nodes aswell.

Gridengine as set up on maxwell sets a variable $TMPDIR which is local to each job and holds the location of a working directory used by the queuing system for that job.
One of the files in this directory is $TMPDIR/machines which holds a list of the machines allocated to this job. It will have an entry for each cpu we have asked for. Since maxwell is a dual cpu machine each machine name will appear twice. This file is used by mpi to know which machines it should run on. If we want to run a number of serial jobs then we can use the file in a very similar way. Once we know which machines to run jobs on we need to use ssh to connect to that machine and run the commands we want.

Mpi generally runs the same executable with the same input files on each cpu. We may want to run the same executable but we will probably want to use different input files.
Probably the easiest way for us to keep track of this is to number our input files.
So if we want to run a program analysis input file input.1 input.2 input.3 input.4 we could do it with the following script:

#!/bin/sh
j=1
for i in `cat  $TMPDIR/machines`
do
ssh $i "cd $PWD;  analysis input.$j > output.$j" &
let j=$j+1
done

The problem with the above script is that it will simply return once it has started all the jobs and they will then be killed.
We need to develop it a little further so that it keeps track of the jobs it has spawned and waits for them to finish. The following script does this.

#!/bin/sh
j=1
for i in `cat  $TMPDIR/machines`
do
ssh $i "cd $PWD;  analysis input.$j > output.$j" &
# Get the PID of the SSH
let x[$j]=$!
let j=$j+1
done
# Wait for the ssh processes to finish
k=1
while [ $k -lt $j ]
do
wait ${x[$k]}
let k=$k+1
done

The above script if submitted will run as many jobs as cpus of the queue you have submitted to.
Various modifications could be made to this scheme. If we wanted to run a number of different programs then you could package these up
into a number of scripts.
For example

script.1

#!/bin/sh
prog1 < inputfile1 > outputfile1

and script.2

#!/bin/sh
prog2 < inputfile2 > outputfile2

The script we submit would then look like

#!/bin/sh
j=1
for i in `cat  $TMPDIR/machines`
do
ssh $i "cd $PWD;  script.$j" &
let x[$j]=$!
let j=$j+1
done
k=1
while [ $k -lt $j ]
do
wait ${x[$k]}
let k=$k+1
done

One would need to make sure all the scripts are executable with

chmod +x script.*

These scripts do not need to contain just one program. The default run time for the S queue on maxwell is 24 hours and so it is to your advantage if your jobs take as close to 24 hours as you can. So if each of your programs took 7 hours then you could put 3 of these into each of your scripts. If a job is running when the 24 hours are up the queuing system will kill it. If your program needs to run for more than 24 hours then you may need to look at ways of saving the state of it and resuming from where it was up to.