Vostron Status

Both feeds are now operational - Normal operation resumed

Resolved

Resolved

We can now confirm resolution to this network incident.

Date Created - 15/09/2020 06:10:41
Incident Duration - 0 day(s) 9 hour(s) 14 minute(s).
Service Affecting Status - Incident is not Service Affecting.
Service Affecting Duration - 0 day(s) 8 hour(s) 59 minute(s).

Due to planned works the supplier platform in Telehouse North was out of service . All customer services have been brought back up and traffic levels across our platform have been confirmed as stable since approximately 13:19hrs. This incident was caused due to planned works within supplier network. The associated software upgrades as part of the change were rolled back to restore service. The incident will now be resolved as all customer services are available and stable with no further issues reported or identified.

SSE has completed the maintenance. We have seen stable power to all Southampton Science Park buildings from 4 PM.

Service has been restored at 10:28 following our engineers restarting a process on one of the VoIP servers. Engineers will be investigating the crash log in an attempt to ascertain the cause. We apologise for any inconvenience.

Engineers will be investigating the crash log in an attempt to ascertain the cause. We apologise for any inconvenience.

Incident report:
This report is a breakdown of events for the service outage experienced on the 26th February 2018 affecting all customers using our Voice-over-IP telephony platform.

Breakdown of events:
09:42 - Our Network Operations Centre received multiple alerts relating to our phone platform specifically relating to a large number of phones deregistering from the platform. Engineers immediately started the investigation.
09:48 - Engineers identified the reason for instability of the platform as increasing request queue that is not being served.
09:51 - Engineers rebooted the registration backend.
09:52 - Issue was resolved and service was fully restored.

Root Cause
We identified that the root cause as a bug in the maintenance script that was run around 9:30. This script was run and interrupted mid way by human interaction, locking the call records table. Our system was backloging the requests to the backend database but finally exhausted the resources and started to drop new ones.
We are putting controls in place to try and mitigate this set of circumstances. During last week our vendor has prepared a fix for the maintenance script and this has now been implemented.
We apologise again for the inconvenience caused by this incident.

As of 11:41 all Telephony services were resumed and customers should not experience any more issues.

The cause of this is still under investigation, and a full report will be sent out in due-course.

Going forward, we are closely monitoring our Telephony platform and have engineers on stand-by should any complications re-occur.

If any problems persist, please restart/reboot/power cycle your handset or contact support on 02380 111 222.

We sincerely apologise for any inconvenience this disruption may have caused, and assure you that we will be doing our upmost to ensure that such issues are not experienced in the future.

Vostron Support Team

The fault is now fully resolved.

We believe this incident is now by and large resolved.

The Science Park contractors have completed their maintenance.