'UNSW - Science

UNSW - Science - HPC

Making use of Local Scratch

The best way of copying data to and from local scratch is to use the prologue and epilogue scripts that run immediately before and after your job. These scripts can run for a maximum of 5 minutes and run even if the job runs out of time.

For example, you might have the following prologue and epilogue scripts in /srv/scratch/z1234567.

Prologue

#! /bin/bash
 
echo "prologue: copy from working directory to local scratch"
echo "    ${PBS_O_WORKDIR}"
echo "    ${TMPDIR}"
echo
 
rm -rf ${PBS_O_WORKDIR}/out
rsync -a ${PBS_O_WORKDIR}/ ${TMPDIR}

Epilogue

#! /bin/bash
 
echo "epilogue: copy from local scratch to working directory"
echo "    ${TMPDIR}"
echo "    ${PBS_O_WORKDIR}"
echo
 
rsync -a ${TMPDIR}/ ${PBS_O_WORKDIR}/out

These scripts will copy everything from the job's working directory to local scratch, and then copy everything from local scratch to a directory called out within the job's working directory.

To instruct the scheduler to run the scripts ensure that the scripts are executable and then add the following lines
to your job script.

#PBS -l prologue=/srv/scratch/z1234567/prologue
#PBS -l epilogue=/srv/scratch/z1234567/epilogue

Array Jobs

If you are running an array job $PBS_ARRAYID is not available directly within prologue and epilogue scripts, but it can be obtained indirectly via $TMPDIR.

if [[ "${TMPDIR}" =~ \[([0-9]+)\]\.katana ]]; then
     PBS_ARRAYID=${BASH_REMATCH[1]}
fi