Updated: – DRAFT
To meet the increasing demand for more space on /scratch from our researchers and to reduce the amount of 'forgotten' files, NCI are introducing a new file management policy for the Gadi scratch file system. This new policy will automatically clean up files older than 100 days and so create more space available to research projects.
As a consequence of this policy change, it will facilitate greater fairness in the use of temporary scratch storage for all NCI projects.
The removal of old files is a three stage process.
Step 1 - Files older than 100 days are moved from project directories on /scratch into a quarantine space. Once a file has been moved to quarantine it will no longer be accessible for modification to as input into to any HPC job.
Step 2 - Files in quarantine remain available for 14 days. During this time files maybe recovered by members of the project and brought back into active use if needed.
Step 3 - Any files remaining in quarantine at the end of 14 days will be deleted. Once a file is deleted, it can not be recovered. Remember the /scratch filespace is for a temporary files system for working files only. Data that researchers or projects wish to keep for an extended period of time must be copied of the /scratch filesystem to the projects /gdata space, archived to mass data or downloaded to local storage.
This is a significant shift in the way the /scratch filesystem is managed so it will be progressively introduced in May-June 2022, with full implementation from 1 July 2022.
The timing of this implementation is important to be completed before the July downtime as it will support essential tuning and reconfiguration of the /scratch file system in a full production, peak performance capacity.
The introduction of policy may lead to a large number of files that will need to be retired having built up since Gadi started operations. To make this process more manageable for researchers, it will be staged in a few phases according to the following schedule:
Users have the option to identify and restore files from quarantine using the nci-file-expiry command. See the attached document for more information about the nci-file-expiry utility.
Important points to note about the new /scratch file management process: