Update - We are watching Mahuika get busy again with over 1000 jobs actively running and approximately 2000 completed! We want to thank everyone for their patience and understanding.
OnDemand and Freezer will remain unavailable over the weekend.
We have arranged limited weekend coverage to support critical issues related to the incident on the Mahuika platform. Our support and platform teams will be monitoring and responding to urgent issues during scheduled weekend coverage periods. While we will make every effort to respond as quickly as possible, please note that all requests will be triaged and prioritised based on severity and business impact.
Our support team will resume normal business operations by 9am on Monday morning when standard support levels will return.
If you experience a critical issue related to the current situation, please raise it by emailing support@nesi.org.nz.
We will provide our next update on Monday at 10:00.
Jul 31, 2026 - 17:00 NZST
OnDemand and Freezer will remain unavailable over the weekend.
We have arranged limited weekend coverage to support critical issues related to the incident on the Mahuika platform. Our support and platform teams will be monitoring and responding to urgent issues during scheduled weekend coverage periods. While we will make every effort to respond as quickly as possible, please note that all requests will be triaged and prioritised based on severity and business impact.
Our support team will resume normal business operations by 9am on Monday morning when standard support levels will return.
If you experience a critical issue related to the current situation, please raise it by emailing support@nesi.org.nz.
We will provide our next update on Monday at 10:00.
Jul 31, 2026 - 17:00 NZST
Update - We will shortly begin restoring some compute capacity and are continuing to work in the background to enable the rest of the nodes. We are taking a staggered approach at restoring service to ensure we can keep thermal and other environmental metrics in check.
OnDemand and Freezer remain unavailable, and we do not expect this to change before next week.
my.nesi.org.nz is available again for project and allocation requests and renewals.
We plan to provide our next update around 17:00.
Jul 31, 2026 - 15:01 NZST
OnDemand and Freezer remain unavailable, and we do not expect this to change before next week.
my.nesi.org.nz is available again for project and allocation requests and renewals.
We plan to provide our next update around 17:00.
Jul 31, 2026 - 15:01 NZST
Update - You can now log in via ssh to check on SLURM job and data management tasks, however OnDemand services are not available yet.
You can access the contents of your home, nobackup and project directories. As a reminder, login nodes should not be used for compute processes.
We are continuing to work bringing compute nodes online. They will be made accessible once we have completed health checks on them. Most jobs which were running when the nodes failed should have been automatically requeued, all queued jobs are still be in the queue.
Access to the REANNZ Mahuika Globus endpoint is also now available for transferring data to/from Mahuika.
There is still no access to Freezer storage via any method.
We plan to provide our next update at 15:00.
Jul 31, 2026 - 13:02 NZST
You can access the contents of your home, nobackup and project directories. As a reminder, login nodes should not be used for compute processes.
We are continuing to work bringing compute nodes online. They will be made accessible once we have completed health checks on them. Most jobs which were running when the nodes failed should have been automatically requeued, all queued jobs are still be in the queue.
Access to the REANNZ Mahuika Globus endpoint is also now available for transferring data to/from Mahuika.
There is still no access to Freezer storage via any method.
We plan to provide our next update at 15:00.
Jul 31, 2026 - 13:02 NZST
Update - We are continuing to bring the system online and test various services. The Research Support Team has begun more specific testing to ensure that Slurm, MPI and other aspects of jobs are working.
We will not be proceeding with the regularly scheduled autocleaner this week.
Service restoration efforts are ongoing. We do not yet have enough information to provide an accurate recovery estimate, but we will continue to share regular updates. Unless the situation materially changes, we expect to provide our next update around 13:00.
Jul 31, 2026 - 09:58 NZST
We will not be proceeding with the regularly scheduled autocleaner this week.
Service restoration efforts are ongoing. We do not yet have enough information to provide an accurate recovery estimate, but we will continue to share regular updates. Unless the situation materially changes, we expect to provide our next update around 13:00.
Jul 31, 2026 - 09:58 NZST
Update - Storage services -other than Freezer- are operational.
Recovery of login services is progressing well, however this work is ongoing. The compute nodes are not online yet, and batch jobs are not running.
We expect to provide an update at approximately 10:00am tomorrow.
Thank you for your patience thus far.
Jul 30, 2026 - 17:05 NZST
Recovery of login services is progressing well, however this work is ongoing. The compute nodes are not online yet, and batch jobs are not running.
We expect to provide an update at approximately 10:00am tomorrow.
Thank you for your patience thus far.
Jul 30, 2026 - 17:05 NZST
Update - HPC storage services appear to be in good health, although final checks are ongoing.
In parallel, we are working to get login services up and running however, at this stage we do not expect to have compute nodes online today.
Freezer and tape-based storage is expected to remain offline for longer, while we perform drive and media consistency checks. We will provide further information on this as we know more.
We plan to provide another update at approximately 17:00 today.
Jul 30, 2026 - 15:00 NZST
In parallel, we are working to get login services up and running however, at this stage we do not expect to have compute nodes online today.
Freezer and tape-based storage is expected to remain offline for longer, while we perform drive and media consistency checks. We will provide further information on this as we know more.
We plan to provide another update at approximately 17:00 today.
Jul 30, 2026 - 15:00 NZST
Update - We have powered on essential hardware to begin bringing systems online. We are actively working on ramping up services starting with HPC storage and validating the status of storage. We are bringing up the minimum number of virtual instances to manage the rest of the fleet once storage is online.
We are keeping a very close eye on the environmental metrics at the data centre to prevent any additional fluctuations that might impact our tape media.
We plan to provide another update at approximately 15:00 today.
Jul 30, 2026 - 12:55 NZST
We are keeping a very close eye on the environmental metrics at the data centre to prevent any additional fluctuations that might impact our tape media.
We plan to provide another update at approximately 15:00 today.
Jul 30, 2026 - 12:55 NZST
Update - Tamaki Data Centre (TDC) has confirmed that the cooling system is back online and they are performing environmental checks etc.
We need to do a slow and controlled resumption of business as usual and perform health checks on the systems as we go.
We plan to provide another update about the progress and status by 13:00 today.
Jul 30, 2026 - 10:05 NZST
We need to do a slow and controlled resumption of business as usual and perform health checks on the systems as we go.
We plan to provide another update about the progress and status by 13:00 today.
Jul 30, 2026 - 10:05 NZST
Identified - The current status is that we have the two working chillers at the data centre - TDC and a team is working on restoring the full chiller capacity.
Jul 30, 2026 - 10:00 NZST
Jul 30, 2026 - 10:00 NZST
Update - Tamaki Data Centre has identified issues affecting two of the site's three chillers, resulting in reduced cooling capacity.
Teams are actively investigating the root cause and monitoring site temperatures and cooling performance. Our priority is the safe restoration of cooling services and maintaining the integrity of the platform.
Once temperatures have stabilised and cooling capacity has been validated, we will commence our recovery and startup plan. We do not plan to proceed with any start up until at least tomorrow morning. Further updates will be provided as the investigation progresses.
Jul 29, 2026 - 17:16 NZST
Teams are actively investigating the root cause and monitoring site temperatures and cooling performance. Our priority is the safe restoration of cooling services and maintaining the integrity of the platform.
Once temperatures have stabilised and cooling capacity has been validated, we will commence our recovery and startup plan. We do not plan to proceed with any start up until at least tomorrow morning. Further updates will be provided as the investigation progresses.
Jul 29, 2026 - 17:16 NZST
Update - Cooling at the Tamaki Data Centre has not yet been restored. All services and hardware remains down. We currently don't have any ETA.
Jul 29, 2026 - 16:25 NZST
Jul 29, 2026 - 16:25 NZST
Update - We are continuing to investigate this issue.
Jul 29, 2026 - 15:10 NZST
Jul 29, 2026 - 15:10 NZST
Update - This incident has continued to escalate so we are shutting down the HPC to avoid hardware damage.
Jul 29, 2026 - 14:57 NZST
Jul 29, 2026 - 14:57 NZST
Update - We are continuing to investigate this issue.
Jul 29, 2026 - 13:24 NZST
Jul 29, 2026 - 13:24 NZST
Update - For jobs already running on impacted nodes, users will see some jobs ended early with state NODE_FAIL and then requeued.
New jobs can still be submitted to the queue, but they will not start.
Jul 29, 2026 - 13:23 NZST
New jobs can still be submitted to the queue, but they will not start.
Jul 29, 2026 - 13:23 NZST
Update - We are stopping new jobs from starting while the investigation continues. Submitted jobs will remain in the queue and start once we have resolved the incident.
Jul 29, 2026 - 13:21 NZST
Jul 29, 2026 - 13:21 NZST
Investigating - We have had some nodes go down due to a serious environmental (cooling) issue at the data centre. We are working to mitigate impacts.
Jul 29, 2026 - 13:15 NZST
Jul 29, 2026 - 13:15 NZST
About This Site
This page shares the system status of REANNZ's advanced computing platform and storage services, including impacts of any known outages or planned maintenance work.
Please note that these services are currently only supported within business hours, so there may be delays in communicating outages outside of these hours despite best efforts.
To view the status of REANNZ's network services, visit: https://reannz.status.io
Apply for Access
Operational
Data Transfer
Operational
Submit new HPC Jobs
Operational
Jobs running on HPC
Degraded Performance
NeSI OnDemand
Major Outage
HPC Storage
Operational
User Support System
Operational
Support Documentation
Operational
Flexible High Performance Cloud
Degraded Performance
Long-term Storage (Freezer)
Major Outage
Flexible High Performance Cloud Services
Degraded Performance
Virtual Compute Service
Degraded Performance
Bare Metal Compute Service
Degraded Performance
FlexiHPC Dashboard (web interface)
Degraded Performance
FlexiHPC CLI interface
Degraded Performance
Public API of the FlexiHPC Service
Degraded Performance
Login to Mahuika
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.