Page History

Gadi has 6 six dedicated data-mover nodes that handle exactly that, moving data to and from the system at a high speed. 

These data-mover nodes have the domain name 'gadi-dm.nci.org.au', as seen below, and you can use this when moving data to and from the system.

• SCP (Secure Copy Protocol) 
• rsync
• 3rd party clients
• wget
• File Compression

From local system →  Gadi 

To start a scp transfer through Gadi's data-mover nodes, you can run the command,

$ scp <filename> <user>@gadi-dm.nci.org.au: <destination>

Notice that the domain is no longer gadi.nci.org.au, but gadi-dm.nci.org.au, as you are logging into a data-mover node instead of a login node. 

Replace the filename with the file you wish to transfer, including the file extension e.g. .sh, .pdf, and use your username as the login. The destination should be replaced with location that you want the file to be placed in. for example  

$ scp testfile.sh <user>@gadi-dm.nci.org.au:/home/900/<user>

You will then be prompted for your password, once entered, your file transfer will begin. 

From Gadi → local system

If you want to do the opposite, move a file from gadi to your local system, you simply need to reverse the prompt to reflect this, like so

$ scp <user>@gadi-dm.nci.org.au:/home/900/<user>/testfile.sh ./<destination>

If you are transferring a larger file and want to add a layer of protection to the transfer, using rsync will give you the ability to resume a transfer that has been interrupted. As with SCP, reversing your pathways will change which direction the transfer is happening, either uploading from your local device or downloading from Gadi. 

rsync is also allows greater control over the transfer and exactly what happens with the file by placing options into the command line, such as 

$ rsync -vP  testfile.sh <user>@gadi-dm.nci.org.au:/home/900/<user>

the -vP in the command line means that the output file will be verbose and preserve all of its attributes such as timestamps. -P is a great way to get an exact copy of your file. 

To find a listing of all the options available to you with rsync use the command

$ man rsync

to print a list of all of the variables that you can use. 

If something happens and your transfer doesn't complete, simply run the exact command again and the transfer will pick up where it left off.

Third party applications can be used as a way of visualising file transfers instead of using command line. 

FileZilla, WinSCP, MobaXterm, and many more will allow you to drag and drop you files, negating the need to remember and use command lines. These can be a very handy tool to add to your HPC toolbox and we recommend that you become acquainted with them.

Please refer to the specific programs user manual to learn how to utilise their transfer features.