24th September 2020

Intercapital link issues

14:15 AEST - Resolving incident, Post Incident Review is available upon request.

11:47 AEST - We will continue to monitor this closely until 2pm before closing the incident.

11:33 AEST - All services are restored and working as expected. We are now lodging a fault for our backup link to determine the cause of packet loss. This only occurs under load so we will conduct testing early morning as scheduled maintenance.

11:09 AEST - Primary link has been restored, our backup link is experiencing packet loss under load. It has been decided due to the instability of the backup link currently we are going to turn on the primary link again at 11:30 AEST. Once primary has been restored we will work on the issues surrounding the back up link.

10:47 AEST - Our back up link experienced a short failure in packet loss causing connectivity loss between Canberra and Sydney. This is separate to the issue this morning. We are investigating this also, updates to follow.

10:10 AEST - We have disabled the failed link to prevent further interruptions during business hours when it is restored. Once the link has been restored we will fail this back over and conduct testing. Some high availability services did not experience a clean fail over and we are investigating the cause of this also.

09:04 AEST - Carrier is onsite replacing faulty line card, ETA 1 hour but please report any services that are down and we can manually clear them.

08:14 AEST - Carrier has been contacted and fault has been lodged, we are currently running on back up services. It appears that some of our high availability networks have been affected with hung connections. We are are identifying these network manually but please contact us

07:46 AEST - Services were interrupted during a 10min window but appear be stable. Updates to come.

07:30 AEST - We have identified issues on our network regarding connectivity between Canberra Sydney causing a large disruption. We are investigating the issue and will provide updates on this page.