'UNSW - Science

UNSW - Science - HPC

Copying Files To and From a Cluster

The method of transferring files to and from clusters depends on your local machine. If you are a Linux user then you should use rsync and if you are a Windows user then you should download and install WinSCP or the more powerful FileZilla which has all of the functionality of WinSCP as well as a number of additional features such as remote file editing and advanced folder synchronization. FileZilla is available for most operating systems and is easily downloadable.

To provide the best experience files should be transferred to via the Katana Data Mover (KDM) server kdm.science.unsw.edu.au to the appropriate location. For example if you are a Linux user then you can use the following commands on your local computer.

If you are a Windows user then you can use the graphical interface in WinSCP to select the same directories and copy files.

Copying files into global scratch

Global scratch is located in

/srv/scratch/z1234567

so to copy the directory fred from your local Linux desktop using rsync to global scratch you would use

z1234567@mymachine:~$ rsync -avh fred z1234567@kdm.science.unsw.edu.au:/srv/scratch/z1234567

If you want to copy the directory john from the cluster home drive to scratch then you can log on to the KDM server kdm.science.unsw.edu.au and use

z1234567@kdm.science.unsw.edu.au:~$ rsync -avh john /srv/scratch/z1234567

Copying job scripts onto a cluster home drive

Your cluster home drive is located in

/home/z1234567

so to copy the directory fred using rsync to your cluster home drive under Linux you would use

rsync -avh fred z1234567@kdm.science.unsw.edu.au:/home/z1234567

Copying results from a cluster to your local machine

If you have saved your results into a directory called myresults under global scratch then under Linux you would use

rsync -avh z1234567@kdm.science.unsw.edu.au:/srv/scratch/z1234567/myresults .

to copy files to your local machine.

Some advanced options

Once you are familiar with using rsync for these commands you can add the --delete option which deletes any file on the destination that is not on the source. This is used to make sure that the 2 directories are EXACTLY the same.