After working with support partners overnight we've identified a configuration regression related to an earlier incident. This regression appears to have caused performance issues for networking and virtual machine shared storage, which as a result impacted many workloads across the cloud. The problematic nodes have been removed from production, this seems to have stabilised the situation and services have now been restored. Users are encouraged to login and check services/workflows/jobs etc and please get in touch if you are experiencing any ongoing issues. Our apologies for the issues and unplanned downtime!
Posted Feb 23, 2024 - 11:33 NZDT
Update
We are continuing to remove the problematic nodes from the storage cluster.
Connections to running instances will still be intermittent.
We apologise for any inconvenience cause during this time.
Posted Feb 22, 2024 - 18:27 NZDT
Update
We are in the process of removing problematic nodes, however this is not a quick process.
Connections to instances may be slow or intermittent.
We apologise for any inconvenience cause during this time.
Posted Feb 22, 2024 - 16:09 NZDT
Identified
We have found an issue with the underlying storage cluster that could be attributing to network issues users are experiencing.
Running Instances look to be fine at this stage, however external connections may be slow or intermittent.
We apologise for any inconvenience cause during this time, If you are experiencing any issues please reach out to support so we can advise
Posted Feb 22, 2024 - 12:16 NZDT
Update
We are continuing to monitor for any further issues.
Posted Feb 22, 2024 - 10:06 NZDT
Monitoring
We are currently monitoring an possible network issue within the Flexi/RDC Space.
If you are experiencing any issues please reach out to support so we can advise
Posted Feb 21, 2024 - 17:26 NZDT
Investigating
We are currently investigating an possible network issue within the Flexi/RDC Space.
Running Instances look to be fine at this stage.
If you are experiencing any issues please reach out to support so we can advise