The following email was sent to all members of project ua6 on 18/8/2017 by Clare Richards.
Dear ua6 member,
As you are hopefully aware, ua6 is currently scheduled for migration to NCI’s new /g/data1 filesystem on Tuesday, 22 August. You will not be able to access data while the move is in progress.
The schedule and information about the migration process can be found here https://opus.nci.org.au/display/NDM/NCI+Data+Migration (use your NCI login).
In the lead-up to migration, the ua6 data managers have identified a number of files which were duplicated during CMIP5 data download. Paola Petrelli of the Climate CoE CMS team has compiled a list of directories which are identical copies on the system, that is, they contain files that have the same tracking ID but are a superseded version, or even exactly the same files downloaded from different ESGF servers. We plan to remove these duplicates at the same time as the migration outage.
To see the list of ~9,000 identified directories, please visit this page: https://opus.nci.org.au/display/CMIP/CMIP5+replica+clean+up
The logic behind duplicate identification, and the final list of candidates for deletion, has been reviewed separately by data managers from the Bureau of Meteorology and CSIRO, and NCI have approved the data removal.
What do you need to do?
- If you are currently using the ARCCSSive database to access CMIP5 data, there will be no impact on you, the database has already been updated to point to the “to keep” versions of each file.
- If you use the DRSv2 structure, please be advised that the DRS tree will need to be rebuilt following the deletion. In the interim it will be moved to “DRSv2_legacy” and a new DRSv2 tree built from scratch. We will notify you when the new DRS tree structure is ready in the days following clean up and migration.
- If you access data directly on the filesystem using absolute paths, we recommend you review the list of deletion candidates attached to the above wiki page
We anticipate these changes will have minimal impact on the community but please provide any feedback or concerns by 5pm on Monday 21/8/2017 to Clare.Richards@anu.edu.au and Paola.Petrelli@utas.edu.au.
Please note that ua6 access points in THREDDS and the VDI (CWSLab), will be unavailable from the evening of August 21st. Services will return overnight the day migration and clean up is completed, i.e. 24-48 hours later. This includes ua6_4 NRM data.
Thank you for your patience during this process. We will notify you when the data is available again.
The ua6 data managers