'UNSW - Science

UNSW - Science - HPC

Copying files to the UNSW Data Archive

The UNSW Data Archive is the primary research storage facility provided by UNSW. The Data Archive gives UNSW researchers a free, safe and secure storage service to store and access research data well beyond the life of the project that collected that data.

To help researchers make use of this system the Katana Data Mover server (KDM) has a script that you can use to copy files from Katana into a project on the Data Archive system.

Note: To use this script you must have access to the UNSW Data Archive which requires setting up a Research Data Management Plan.

GETTING THE SCRIPT

As well as being written here the instructions for obtaining and using the Data Archive script are available as part of the module command on the KDM server. To see what versions of the Data Archive script are available log on to kdm.science.unsw.edu.au via SSH and then type

module avail

and see what version you should use. For example as of the time of writing unswdataarchive/2015-09-10 is the latest version. You can then type

module help unswdataarchive/2015-09-10

to get instructions if you require.

To use the Data Archive you need to set up a configuration file. To obtain the file type the command

module add unswdataarchive/2015-09-10
get-config-file

after loading the module which will copy a file called

config.cfg

into the current directory.

Note: This script will only work on the KDM server kdm.science.unsw.edu.au. You should not try to run it on any other server including Katana. If you want to run the script on any other Linux, Windows or Mac computer then you can so by copying the file /share/apps/unswdataarchive/2015-09-10/script.zip.

INITIAL SETUP

Before you use the script for the first time you will need to generate a token for uploading data to the archive then edit the configuration file config.cfg and to change the following line. To generate a token send an email to the IT Service Centre (ITServiceCentre@unsw.edu.au) asking for a Data Archive token to be generated.

token=

If you haven't generated a token you can also upload content using your zID and zPass by adding the following line to the file config.cfg and you will be asked for your zPass when you start the upload.

user=z1234567

NOTE: If you wish to automate the script or just reduce the number of steps required to upload data you should generate a token rather than using your zID and zPass.

RUNNING THE COMMAND

 

So that you can log out and leave your upload running you should use the 'screen' command. Start a new screen by typing

screen -S zID

and then you can start the import by going to the directory containing the files and typing

upload.sh “/path/to/your/local/directory” “/UNSW_RDS/D0000000/your/collection/name”

to upload files and directories and

download.sh “/UNSW_RDS/D0000000/your/collection/name” “/path/to/your/local/directory”

to download files and directories from the UNSW Data Archive.

Once it has started to run you can detach the screen by typing Control a then Control d and log out. When you log back in you can check your progress by typing

screen -R

to re-attach the screen.

When you are finished with the screen you can close it in the same manner that you would use to log out.