The UNSW Data Archive is the primary research storage facility provided by UNSW. The Data Archive gives UNSW researchers a free, safe and secure storage service to store and access research data well beyond the life of the project that collected that data.
To help researchers make use of this system the Katana Data Mover server (KDM) has a script that you can use to copy files from Katana into a project on the Data Archive system.
Note: To use this script you must have access to the UNSW Data Archive which requires setting up a Research Data Management Plan.
GETTING THE SCRIPT
As well as being written here, the instructions for obtaining and using the Data Archive script are available as part of the module command on the KDM server. To see what versions of the Data Archive script are available log on to kdm.science.unsw.edu.au via SSH and then type
and see what version you should use. For example as of the time of writing
unswdataarchive/2015-09-10 is the latest version. You can then type
module help unswdataarchive/2015-09-10
to get instructions if you require.
To use the Data Archive you need to set up a configuration file. To obtain the file type the command
module add unswdataarchive/2015-09-10 get-config-file
after loading the module which will copy a file called
into the current directory.
Note: This script will only work on the KDM server kdm.science.unsw.edu.au. You should not try to run it on any other server including Katana. If you want to run the script on any other Linux, Windows or Mac computer then you can so by copying the file
You can also download the latest version here.
Before you use the script for the first time you will need to generate a token for uploading data to the archive then edit the configuration file config.cfg and to change the following line. To generate a token send an email to the IT Service Centre (ITServiceCentre@unsw.edu.au) asking for a Data Archive token to be generated.
If you haven't generated a token you can also upload content using your zID and zPass by adding the following line to the file config.cfg and you will be asked for your zPass when you start the upload.
NOTE: If you wish to automate the script or just reduce the number of steps required to upload data you should generate a token rather than using your zID and zPass.
If you want to use sftp then load the sftp module and then run
sftp -oUser=UNSW_RDS:z1234567 -P 8022 rds.unsw.edu.au
RUNNING THE COMMAND
So that you can log out and leave your upload running you should use the 'screen' command. Start a new screen by typing
screen -S zID
and then you can start the import by going to the directory containing the files and typing
upload.sh “/path/to/your/local/directory” “/UNSW_RDS/D0000000/your/collection/name”
to upload files and directories and
download.sh “/UNSW_RDS/D0000000/your/collection/name” “/path/to/your/local/directory”
to download files and directories from the UNSW Data Archive.
Once it has started to run you can detach the screen by typing
Control a then
Control d and log out. When you log back in you can check your progress by typing
to re-attach the screen.
When you are finished with the screen you can close it in the same manner that you would use to log out.