18 Oct 2023

Systems Impacted: /g/data

Filesystem Faults

Dear NCI Users,

   Due to a fault in a Lustre filesystem, access to gdata, and logins to Gadi may hang. A large number of nodes have also killed running jobs due to the fault.

Update 16:00: The issue has been resolved.

Regards,

NCI User Services

2 May 2023

Systems Impacted: Massdata Tape Archive System, mdss utility on Gadi

Massdata Tape Archive System Outage – Update 4

Dear NCI Users,

NCI staff have carried out extended testing on the repaired Tape Library system over the weekend. The tests have successfully completed, with no robotics or tape handling faults observed. 

We have re-enabled the mdss put command, and normal operations have resumed. We will continue to closely monitor the operation of the massdata archival system and the tape library robotics.

Thank you for your patience as we worked to resolve this fault over recent weeks.

If you require further assistance, please contact NCI User Services via the Helpdesk at https://help.nci.org.au or help@nci.org.au.

Reminder – Scheduled Maintenance Today, Tuesday 2 May

A reminder to all users that NCI’s scheduled quarterly maintenance will be taking place today, with access to Gadi, /g/data, the NCI network and other services unavailable throughout the day. View a detailed list of affected services here.

Regards,

NCI User Services

28 Apr 2023

Systems Impacted: Massdata Tape Archive System, mdss utility on Gadi

Massdata Tape Archive System Outage – Update 3

Dear NCI Users,

NCI staff have continued to work with the vendor support team to resolve the mechanical fault in one of NCI’s Tape Library systems supporting Massdata.

Unfortunately we have experienced extensive customs clearance delays and delivery problems with some parts. The parts were delivered earlier this week, and a Field Engineer has been on site for two days to work through the robotics faults.

We have performed a set of initial tests this afternoon and they have completed successfully. An extended test will run over the weekend to verify the robotics are operating correctly. If this is successful, mdss put operations will be re-enabled on Monday 1 May.

We understand that the reduced functionality of the mdss utility has significantly impacted some projects and apologise for the inconvenience.

If this is causing issues for your project or workflow, please contact NCI User Services via the Helpdesk at https://help.nci.org.au or help@nci.org.au . We will work with you to identify a temporary workaround or solution.

Regards,

NCI User Services

5 Apr 2023

Systems Impacted: Massdata Tape Archive System, mdss utility on Gadi

Massdata Tape Archive System Outage – Update 2
 
Dear NCI Users,
 
NCI staff are continuing to work with the vendor support team to resolve the mechanical fault in one of NCI’s Tape Library systems supporting Massdata.
 
The critical parts we have been waiting on have arrived in Sydney, and are being cleared by Australian Customs. NCI has been advised that delivery to our datacenter could occur as soon as tomorrow. Additional components have been sourced from an Australian warehouse and are scheduled to be installed by the vendor’s Field Technician tomorrow. 
 
We are aiming to resolve the issue this week. Unfortunately if there are further complications or delivery delays, this may extend the full return to service until Tuesday April 11 or Wednesday April 12 due to the upcoming Easter Public Holidays. 
 
If the ongoing reduced functionality of the mdss utility is significantly impacting your project or worklow, please contact NCI User Services via the Helpdesk at https://help.nci.org.au or help@nci.org.au. We will work with you to identify a temporary workaround or solution.

Easter Public Holidays – 7 and 10 April

NCI systems will operate as normal on the upcoming public holidays.

• 7 and 10 April: Good Friday and Easter Monday

Please note that support services will not be staffed on these days. Support requests received on these public holidays will be addressed as soon as possible on the next business day.

Regards,
NCI User Services

28 Mar 2023

Systems Impacted: Massdata Tape Archive System, mdss utility on Gadi

Update 1
 
Dear NCI Users,
 
NCI staff, in conjunction with the vendor support team, have continued to diagnose the mechanical fault in one of the NCI Tape Library systems supporting Massdata.
 
We are currently awaiting the international delivery of parts which will allow us to bring the systems back into a fully operational state. Where possible, parts have been used from our onsite stock and local warehousing.
 
Unfortunately, we are not yet in a position to re-enable mdss put requests, however we are working towards this as promptly as possible.
 
Recall operations (e.g. mdss get) remain available, and additional throughput is being brought online progressively. Users are encouraged to evaluate if a recall is urgent or required in the short term and defer it if possible.
 
If the reduced functionality of the mdss utility is significantly impacting your project or worklow, please contact NCI User Services via the Helpdesk at https://help.nci.org.au or help@nci.org.au . We will work with you to identify a temporary workaround or solution.
 
An additional update will be provided in the coming days.

Regards,
NCI User Services

21 Apr 2023

Systems Impacted: NCI Network 

Dear NCI Users,
 
The NCI Network has been experiencing occasional connectivity issues in recent weeks, which has resulted in intermittent loss of connection to Gadi (HPC), Nirin (Cloud) and Data Services.

The root cause of the instability has been identified, and the NCI network team has implemented some changes to the network which have stabilised the environment. A full solution has been tested and will be implemented during the next maintenance day on 2 May 2023.

For support or enquiries, please contact NCI User Services via the Helpdesk at https://help.nci.org.au or help@nci.org.au.

Regards,
NCI User Services

6 Apr 2023

Systems Impacted: NCI Network 

Dear NCI Users,
 
Over the last week the NCI Network has been experiencing occasional connectivity issues which has resulted in intermittent loss of connection to Gadi (HPC), Nirin (Cloud) and Data Services.

The NCI Network team are aware of the issue and are working on a resolution. We apologise for any inconvenience this is causing and will provide an update next week on the status of this issue.
 
For support or enquiries, please contact NCI User Services via the Helpdesk at https://help.nci.org.au or help@nci.org.au.

Regards,
NCI User Services

23 Mar 2023

Dear NCI Users,
 
The recent issue regarding the correct receipt and logging of NCI Helpdesk emails has now been resolved and tickets are being correctly lodged.

Users are able to submit a help request either by logging into the NCI Help Portal at help.nci.org.au or emailing help@nci.org.au. 

We apologise again for the disruption this issue may have caused.

Regards,
NCI User Services