Panel | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||
Gadi has 6 six dedicated data-mover nodes that handle exactly that, moving data to and from the system at a high speed. These data-mover nodes have the domain name 'gadi-dm.nci.org.au', as seen below, and you can use this when moving data to and from the system. |
Panel | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||
Using these data-movers ensures that large data transfers are kept off the login nodes, reducing impact for other users who are accessing them.
There are several ways to initiate a transfer and which one you use will depend on your operating system and the size of the file you wish to transfer.
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
SCP (Secure Copy Protocol)
|
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
rsync
|
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
3rd party clients Clients such as FileZilla, WinSCP, and MobaXterm have the ability to complete transfers simply by dragging and dropping, negating the need to use command lines. |
Anchor | ||||
---|---|---|---|---|
|
Panel | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||
From local system → Gadi To start a
Notice that the domain is no longer gadi.nci.org.au, but gadi-dm.nci.org.au, as you are logging into a data-mover node instead of a login node. Replace the filename with the file you wish to transfer, including the file extension e.g. .sh, .pdf, and use your username as the login. The destination should be replaced with location that you want the file to be placed in. for example , like the following for an example:
You will then be prompted for your password, once entered, your file transfer will begin. From Gadi → local system If you want to do the opposite, move a file from gadi Gadi to your local system, you simply need to reverse the prompt to reflect this, like so
To find a listing of all the options available to you with
to print a list of all of the variables that you can use. |
Anchor | ||||
---|---|---|---|---|
|
Panel | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||
If you are transferring a larger file and want to add a layer of protection to the transfer, using
the To find a listing of all the options available to you with
to print a list of all of the variables that you can use. If something happens and your transfer doesn't complete, simply run the exact command again and the transfer will pick up where it left off. |
Anchor | ||||
---|---|---|---|---|
|
Panel | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||
Third party applications can be used as a way of visualising file transfers instead of using command line. FileZilla, WinSCP, MobaXterm, and many more will allow you to drag and drop you files, negating the need to remember and use command lines. These can be a very handy tool to add to your HPC toolbox and we recommend that you become acquainted with them. Please refer to the specific programs user manual to learn how to utilise their transfer features. |
Anchor | ||||
---|---|---|---|---|
|
Panel | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||
If you need a file that is already hosted online, the To do this, first locate the url of the file you need, you can usually do this by right clicking the file and selecting 'Copy link address'. Once you've copied the link, run the command
Simply replace <url> with the link that you copied and the system will immediately start downloading your file and place it into your home directory. |
Panel | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||
copyq queue. To read the process of how to achieve this, follow this link. |
Anchor | ||||
---|---|---|---|---|
|
Panel | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||
A useful way of minimising transfer time is compressing files so that the overall size of the package is smaller. This is also done when you have a lot of small files that can be compressed into one larger file and transferred with less effort. These files can't be used in their current state and need to be unpacked first. Depending on what type of file you are trying to access, you will need to unpack them in different ways. Use the list below to work out what command you need to run to unpack your file.
|