Updated: 19 Apr 2022 – DRAFT

To meet the increasing demand  for more space on /scratch from our researchers and to reduce the amount of 'forgotten' files, NCI  are introducing a new file management policy for the Gadi scratch file system. This new policy will automatically clean up files older than 100 days and so create more space available to research projects.

As a consequence of this policy change,  it will facilitate greater fairness in the use of temporary scratch storage for all NCI projects.

The removal of old  files is a three stage process.

Step 1  - Files older than 100 days are moved from project directories on /scratch into a quarantine space. Once a file has been moved to quarantine it will no longer be accessible for modification to as input into to any  HPC  job.

Step  2 - Files in quarantine remain available for 14 days.  During  this time files maybe recovered by members  of the project and brought back into active use if needed.

Step 3 - Any files remaining in quarantine at the end of 14 days will be deleted. Once a file is deleted, it can not be recovered.  Remember the /scratch filespace is for a temporary files system for working files only.   Data that researchers or projects wish  to keep for an extended period of time must be copied of the /scratch filesystem to the projects /gdata   space, archived to  mass data or downloaded to local  storage.

This is a significant shift in the way the /scratch filesystem is managed so it will be progressively introduced in May-June 2022, with full implementation from 1 July 2022.

The timing of this implementation is important to be completed before the July downtime as it will support essential tuning and reconfiguration of the /scratch file system in a full production, peak performance capacity.  

The introduction of policy may lead to a large number of files that will need to be retired having built up since Gadi started operations.  To  make this process more manageable for  researchers,  it will be staged in a few phases according to the following schedule: 

1. 17 May 2022 : Files within /scratch project directories with a atime greater than 365 days will be quarantined before being deleted on the 31st May ?
2. 07 Jun 2022 : Files with atime greater than 100 days will be retired and moved into quarantine. These files will then be deleted 14 days later.
3. From  , files with atime greater than 100 days will be retired on a continuous basis.  This means that a script will run everyday and add files into  the the quarantine space if the atime is  greater than 100 days. It  will also mean that files that have aged for 14  days in quarantine will be automatically be deleted everyday by  the same script.

Users have the option to identify and restore files from quarantine using the nci-file-expiry command. See the attached document for more information about the nci-file-expiry utility.

Important points to note about the new /scratch file management process:

• If you don't need the files in /scratch please delete them asap. You don't need to wait for the script to be run. 
• The /scratch file system is intended for temporary, working storage. For persistent storage, use the /g/data, massdata systems  or  download to your local filesystem.
• All projects with active NCMAS allocations now have /g/data directories. Default allocation is 2.5 GB/KSU.
• Stakeholder projects will get /g/data allocations per entitlements and demand. For /g/data allocations, please contact your scheme allocation manager. NCI (help@nci.org.au ) can help put you in touch with the appropriate scheme managers if needed.
• Project default scratch quotas will be raised at the time of July quarterly maintenance. (Note that default quotas are still necessary for file system safety.)
• Projects expecting to use large amounts of scratch capacity will still need to request appropriate quotas. Consultation with NCI HPC and Storage groups may be needed for projects using peak-scale scratch capacity.
• Large scratch requests (e.g. >= 10 TB) from projects with compute allocations of less than 1 MSU/year, or without demonstrated track records, will be accommodated in phases with advice from NCI Storage and HPC groups.
• In general, exceptions to the scratch file expiry policy will not permitted. If you need advice or assistance to prepare for the full implementation in 05 Jul 2022 (TBC) contact NCI user support: help@nci.org.au .