Firmware upgrades on OpenStack Production Controller nodes to address HPE AMD Server Vulnerabilities and Preparation for OpenStack Upgrade March 2026
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 17, 2026 - 10:00 NZDT
Scheduled - Firmware upgrades on OpenStack Production Controller nodes to address HPE AMD Server Vulnerabilities and Prep for OpenStack Upgrade March 2026

Description:
- Firmware upgrade on OpenStack Production Controller nodes a01cc01,a01cc02 and a01cc03
- This will also address recent HPE AMD Server Vulnerabilities and prepare for OpenStack Upgrades which will be done in March 2026
- The upgrade will be performed in a rolling fashion, so there will be 2 nodes always available to provide OpenStack controller services

Schedule:
-We will start with a01cc03,a01cc02 and a01cc01 last as there are application settings run exclusively out of this host (PDU metrics scraping)
- a01cc03 - Tuesday, Mar 17, 2026 from 10AM
- a01cc02 - Wednesday, Mar 18, 2026 from 10AM
- a01cc01 - Tuesday, Mar 19, 2026 from 10AM

Possible Impact:
- Possible intermittent or no access to FlexHPC Dashboard (web interface), FlexiHPC CLI interface and Public API of the FlexiHPC Service while each controller is under maintenance
- APC and HPE PDU metrics scraping will stop during the time a01cc01 will be worked on as its currently configured to scrape only from this controller. However, we will test with configuring a01cc02 during the upgrade of a01cc01 so there is no break in the metric scraping for those PDUs.
Mar 17, 2026 10:00 - Mar 19, 2026 22:00 NZDT

About This Site

This page shares the system status of REANNZ's advanced computing platform and storage services.
To view the status of REANNZ's network services, visit: https://reannz.status.io

Uptime over the past 90 days. View historical uptime.
Apply for Access Operational
Data Transfer Operational
Submit new HPC Jobs Operational
Jobs running on HPC Operational
NeSI OnDemand Operational
90 days ago
98.82 % uptime
Today
HPC Storage Operational
User Support System Operational
Support Documentation Operational
Flexible High Performance Cloud Operational
Long-term Storage (Freezer) Operational
90 days ago
99.99 % uptime
Today
Flexible High Performance Cloud Services Under Maintenance
90 days ago
100.0 % uptime
Today
Virtual Compute Service Operational
Bare Metal Compute Service Operational
FlexiHPC Dashboard (web interface) Under Maintenance
90 days ago
100.0 % uptime
Today
FlexiHPC CLI interface Under Maintenance
90 days ago
100.0 % uptime
Today
Public API of the FlexiHPC Service Under Maintenance
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Mar 17, 2026

Unresolved incident: Firmware upgrades on OpenStack Production Controller nodes to address HPE AMD Server Vulnerabilities and Preparation for OpenStack Upgrade March 2026.

Mar 16, 2026
Automatic cleaning launched for scratch filesystem (/nesi/nobackup) - Announcement
Completed - Automatic cleaning of the scratch filesystem now continues as a BAU process.
Mar 16, 11:28 NZDT
Update - The scratch autocleaner is now underway, doing its first round of file deletions. The next deletion run will start on Dec 17th,, once that is completed we will suspend the autocleaner over the Christmas holiday period. Then it will start again with a scan on Jan 14th, followed by the file deletions two weeks later. From there it will run every two weeks. If you have any issues please contact support @nesi.org.nz.
Dec 5, 10:43 NZDT
Scheduled - As announced earlier, we’ve re-activated an automatic cleaning process for temporary data stored on our scratch filesystem (/nesi/nobackup). Project members with files scheduled for deletion were notified by email (Subject to their my.nesi.org.nz Notification preferences. To confirm or adjust what notifications you receive, follow these instructions: https://docs.nesi.org.nz/Getting_Started/my-nesi-org-nz/Managing_notification_preferences/)
Files will be deleted on Wednesday 03 December (two weeks from yesterday's email notification). For more details on how the auto-deletion process works, visit:
https://docs.nesi.org.nz/Storage/File_Systems_and_Quotas/Automatic_cleaning_of_nobackup_file_system/
If you have files identified as candidates for deletion that you need to keep beyond the scheduled expiry date, you can move them to your project directory or to Freezer, our long-term storage service. However, if you plan to move more than 2 TB of data or if you need to increase your project directory quota, email support@nesi.org.nz so that we can discuss your storage needs and assist you.
Nov 20, 08:44 NZDT
OnDemand is down
Resolved - This incident has been resolved.
Mar 16, 10:33 NZDT
Monitoring - A fix has been implemented and we are monitoring the results.
Mar 15, 13:12 NZDT
Update - We are continuing to investigate this issue.
Mar 15, 10:16 NZDT
Investigating - OnDemand is currently down with a "Home Directory not Found" error. The issue will be resolved once staff are available to investigate.

Access to Mahuika via a terminal and ssh is unaffected.
Mar 15, 10:12 NZDT
Mar 15, 2026
Mar 14, 2026

No incidents reported.

Mar 13, 2026
my.nesi.org.nz and its databases system update
Completed - The scheduled maintenance has been completed.
Mar 13, 15:14 NZDT
Verifying - Verification is currently underway for the maintenance items.
Mar 13, 15:13 NZDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 13, 14:00 NZDT
Update - We will be performing a database upgrade affecting our authentication system. During the maintenance window, users will be unable to log in. Existing sessions are expected to continue working.
Mar 11, 14:43 NZDT
Scheduled - We will be undergoing scheduled maintenance during this time.
Mar 11, 14:15 NZDT
Mar 12, 2026
Minor maintenance on network switches
Completed - The scheduled maintenance has been completed.
Mar 12, 15:57 NZDT
Update - We are continuing to verify the maintenance items.
Mar 12, 15:42 NZDT
Verifying - Verification is currently underway for the maintenance items.
Mar 12, 14:37 NZDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 12, 12:00 NZDT
Scheduled - We are making a small change to our network switches to restrict ssh access to the switches themselves. No impact is expected, this is just a hazard warning.
Mar 12, 11:15 NZDT
Mar 11, 2026

No incidents reported.

Mar 10, 2026
Support request tickets automatically transitioned causing unwanted notifications
Resolved - Atlassian advised: The issue has now been resolved, and the service is operating normally for all affected customers.
Mar 10, 14:46 NZDT
Investigating - Our support system is currently affected by an issue with Atlassian infrastructure.
> We’re currently investigating a known incident impacting automation events for some customers in the APAC region. You can follow live updates and the latest status here:
https://jira-service-management.status.atlassian.com/incidents/cbnm86l1ss2k
While the support system is functional, some tickets are automatically transitioned causing unwanted notifications.
Mar 10, 14:01 NZDT
Mar 9, 2026

No incidents reported.

Mar 8, 2026

No incidents reported.

Mar 7, 2026

No incidents reported.

Mar 6, 2026

No incidents reported.

Mar 5, 2026

No incidents reported.

Mar 4, 2026
Slow I/O Performance
Resolved - Additional SSD storage has been added to HPC3/Mahuika's WEKA filesystems earlier in February. This has almost completely eliminated capacity related back-pressure events, which were previously having an adverse impact on ongoing read and write performance. The additional SSD capacity also adds to the peak performance capability of our filesystems, nearly doubling their total throughput to 376 GB/s reads and 71 GB/s writes, and so giving us plenty of headroom for more data-intensive workloads, particularly of the AI variety.

We are still working on additional tuning to improve performance for tiered reads from underlying object storage, but the overall situation should already be much improved. If you continue to experience any IO related issues or performance inconsistencies please reach out to support so we can investigate.
Mar 4, 23:10 NZDT
Monitoring - With the implementation of a new autocleaning process on the scratch/nobackup filesystem last month we have now resolved the associated storage capacity issues related to this incident.

Unfortunately, some workloads will still be experiencing reduced read performance depending on other activity on the system - if you notice highly variable or poor IO performance then please report it to support as we may be able to assist with workarounds.

There are several pieces of work in progress to improve IO performance on the new platform based on learnings from the first 6 months of operations, including an expansion of SSD/NVMe capacity. We will make an announcement soon providing some additional details on this and also update documentation to reflect important considerations for users.
Jan 15, 11:57 NZDT
Update - There has been a period of IO stalls this morning as we've dealt with some storage hardware failures. That issue is now resolved, however space reclamation continues in the backend and is having a detrimental impact on read performance. We are working with WEKA support to look at mitigation options. Apologies for the performance impact - if your jobs are impacted and need a runtime limit extension then please reach out to support.
Nov 24, 12:44 NZDT
Identified - Our storage system is currently very full and this is forcing the backend object storage to undertake some urgent administration by way of defragmentation. This increased load is having a detrimental affect on I/O performance, especially read I/O, and this is likely to continue for some days. We are urgently looking at ways to mitigate this. Researchers could help alleviate this in the short term by cleaning up any unwanted files and data as soon as possible please.
Nov 21, 12:29 NZDT
NeSI OnDemand Update
Completed - The scheduled maintenance has been completed.
Mar 4, 12:52 NZDT
Verifying - We have completed the required updates to OnDemand and its support services.

During this time users would have noticed that they were asked to log back in and all previous sessions were gone, this is expected as we have changed some under lying supporting services that required this.

All work should have saved to the shared file system.

We thank you for patience and if you are having any issues at all please reach out to the team.
Mar 3, 13:34 NZDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 3, 09:32 NZDT
Scheduled - We will be doing an update to OnDemand and its supporting services.

During this time you will experience some intermittent connection issues with a high possibility of current active sessions being closed or dropped.

We apologize for the inconvenience caused at during this time
Mar 2, 10:09 NZDT
Mar 3, 2026
← Incident History Powered by Atlassian Statuspage