In terms of backups, /home
is the only filesystem on Gadi that provides any sort of failsafe in case of accidental deletion.
Every /home
folder has a hidden directory that doesn't show up in a contents listing. This directory, ~/.snapshot
, contains various snapshots of your home directory.
If you've accidentally deleted a file and want to revert back to a snapshot prior the this unfortunate event, you can run,
$ ls ~/.snapshot
to get a print out similar to this image on the right.
Each of these folders is a snapshot of your entire /home
folder at the time that is stamped in the suffix. from here you can choose the one that closest resembles the time you want to revert to and get your file back. Let's say you wanted to revert to the top most hourly backup, you would run,
$ cp -r ~/.snapshot/hourly.2023-08-18_0105/lost_folder ~/recovered_folder
These snapshots aren't included in your 10 GiB quota on /home
, but the recovered file will be so you need to be aware of how much space the file will take up and how much room is left in your quota.
To check this, you can run the command below, which gives you the size of the folder you are copying and storage left in /home
.
$ cd ~/.snapshot/hourly.2023-08-18_0105/ $ du -sh lost_folder 306M lost_folder $ du -sh $HOME 3.1G /home/<111>/<aaa111>
/scratch isn't intended to be a long term storage area, it is intended as an area to run your jobs and then shift the data away. As a result, /scratch is scanned regularly for files that haven't been accessed in 100 days.
Any files that aren't accessed in this time period are then placed into quarantine, files placed in quarantine will be permanently deleted after 14 days unless they are recovered.
To help with this, NCI has provided a tool that
To run this tool, use the command
$ nci-file-expiry -h
Which will return:
From here you can choose to dive even deeper into these options by adding --help
to the command line, for example:
$ nci-file-expiry recover --help
This gives a breakdown of what commands need to be entered for each tool. In this case, we can see below that to recover a file we need the Universally Unique IDentifier, or UUID, or the quarantined file we wish to recover.
If the target directory does not exist, you may have to create that manually and also need to specify the filename it should be saved as. Example:
$ nci-file-expiry recover ca7547b7-2066-4c24-9feb-864b23c038de /scratch/xxy/ab1234/dir1/dirTwo/geometry.py
In case there are lots of files to be retrieved, consider using the batch-recover
option.
This, however, involves a few steps to setup:
$ nci-file-expiry list-quarantined > list_of_files.txt
Use a combination of Unix utilities like awk,
grep
, to the output file to contain:
<UUID> <path/to/quarantined/file>.
Example: awk '{print $1,$6}' < list_of_files.txt > final_list_of_files.txt
Final entries in this file should be something similar to:
395087c4-0555-4521-8fe4-6899796aef93 /scratch/xxy/ab1234/abc/xyz/FDS6.7.7.tar.gz397465c4-0532-4957-8af4-6899678eaf39 /scratch/xxy/ab1234/dec/a3f/random.txt ...
nci-file-expiry
will not create the path for you.The best thing of recovering lots of quarantined files is to recreate all the directories before submitting the recovery requests. Then you can recover all of the objects without issues.
The --json
flag to list-quarantined will generate JSON output which will have "type" field which is "d" for directories and "-" for files. You will need to parse the output to select the directory and file names. For each directory name, simply run
$ mkdir -p <directory name>
to create the missing directories.
The --json
flag to `list-quarantined` can be used for processing in a scripting language like Python
.
When this file is ready, just submit it for processing:
$ nci-file-expiry batch-recover final_list_of_files.txt
Depending on the number of files in the request, it may take some time to recover all of them. You can keep checking the status of the recovery by occasionally running:
$ nci-file-expiry status