'UNSW - Science

UNSW - Science - HPC

Storage

On most clusters storage is split into 3 different types each of which has different advantages and limitations. Additionally source code can be stored on one of the version control servers run by the Faculty of Science Computing Unit.

Name Location Can also be called
Cluster Home Drive

/home/z1234567

$HOME

Global Scratch

/srv/scratch/z1234567

/srv/scratch/$USER

UNSW Home Drive

/home/z1234567/hdrive

$HOME/hdrive

Cluster Home Directories

Your home directory $HOME (e.g. /home/z1234567) is your personal storage area on the cluster and it is available on every node in the cluster. Any files stored in your home directory are regularly backed up, but a quota is imposed on each user. You can check your current usage with the quota command. The home directory is suitable for storing relatively small but important files such as documents, scripts and source code.

Your Cluster home directory is available on all of the nodes in the cluster.

Global Scratch

Each user is also provided with a global scratch directory (e.g. /srv/scratch/z1234567). It is available on every node in the cluster, however, unlike home directories it is not backed up and there is no quota. The only limit imposed on global scratch is the physical capacity of the storage device. It is a shared resource and care must be taken by each user to avoid exceeding reasonable levels of usage. Everytime a user logs in they are presented with a usage summary to help monitor personal and total usage. Global scratch is not for long term storage of data; it is ideally suited to the temporary storage of data for running jobs.

Global scratch is available on all the nodes in the cluster and should be used for any large files.

UNSW H-Drive and Shared Drives

When you log on to a cluster your UNSW H-Drive and Shared Drives are not available. If you run the following command

z1234567@katana:~$ network

then your drives will be available at $HOME/hdrive and $HOME/maths (for example) respectively.

Your UNSW Network drives are NOT available on the cluster nodes.

CCRC data storage

If you are in the CCRC and have been given access to one of the data directories then you will find the storage under /srv/ccrc.

These drives are NOT available on cluster nodes.

Local Scratch

Every compute node provides an amount of disk space as local scratch. The total capacity of local scratch is typically 200GB. When a job starts on a compute node, a directory within local scratch is automatically created for the sole use of the job, and it is automatically removed when the job finishes. The location of the local scratch directory is available via the $TMPDIR environment variable within the PBS job script. Local scratch space is ideally suited for I/O intensive jobs that read and write intermediate files. However, any files required beyond the life of the job must be copied elsewhere (e.g. global scratch) before the job finishes otherwise the data will be lost. If a job requires local scratch space then it is highly recommended to declare the local storage requirements in a resource request within the job script.

If you have I/O intensive jobs then you can automate the transfer of files to and from local scratch using the prologue and epilogue commands.