'UNSW - Science

UNSW - Science - HPC

Moving Files Around A Cluster

Once you have files in your home directory or Global Scratch you can move or copy them using either Linux commands such as cpmv and rsync or a local installed program such as FileZilla r WinSCP.

If you are dealing with large amounts of data (>100Gb at one time) then this may place a load on the Katana head node and rather than conecting directly to Katana to move files you should connect to the Katana Data Mover instead.

Using the Katana Data Mover (KDM) Server

If you are familiar with using Linux commands to copy or move files then you can do that directly by logging on to kdm.science.unsw.edu.au via SSH in the same way that you would log in to Katana and then use the cp, mv and rsync commands that you would normally use under Linux.

If you are not familiar with using the Linux command line for moving or copying files then the easiest way to move files around is to use client software such as FileZilla (https://filezilla-project.org). Once you have connected to kdm.science.unsw.edu.au using your zID and zPass you should see a remote view which corresponds to the files sitting on Katana. You can then use the FileZilla interface to move files and folders around.

Moving Files Around Using RSYNC on kdm.science.unsw.edu.au

In Linux there is a great application that is used to copy files and directories between directories and computers that is called rsync. The examples below show you how you can use RSYNC after connecting to the Katana Data Mover using SSH (using PuTTY or the ssh command).

We start by getting a connection to the server kdm.science.unsw.edu.au

z1234567@kdm.science.unsw.edu.au:~$ screen -S zID

In order to make sure that your copy will keep running even if you are disconnected you should use the 'screen' command. To start a new screen we use the command screen -S ID so we start a new screen by typing

z1234567@kdm.science.unsw.edu.au:~$ screen -S zID

and then you can start the import by going to the directory containing the files and using one of the commands below:

Copy the directory called mydir and its' contents to the existing directory /path/to/destination.

z1234567@kdm.science.unsw.edu.au:~$ rsync -avh /path/to/source/directory/mydir /path/to/destination

Copy the contents of the directory called mydir (but not the directory itself) to the existing directory /path/to/destination. Note the slash at the end of the source directory name.

z1234567@kdm.science.unsw.edu.au:~$ rsync -avh /path/to/source/directory/mydir/ /path/to/destination

At any time you can detach the screen by typing Control a then Control d and log out. When you log back in you can check your progress by typing

z1234567@kdm.science.unsw.edu.au:~$ screen -R

to re-attach the screen.

When you are finished with the screen you can close it in the same manner that you would use to log out.