Status detail: Hosted Telephony

Initial update: 2018-02-26 10:02:32

Latest update: 2018-03-06 13:53:02

Current status: Resolved

6th March 2018 at 13:53: Resolved: Hosted Telephony
Incident report:
This report is a breakdown of events for the service outage experienced on the 26th February 2018 affecting all customers using our Voice-over-IP telephony platform.

Breakdown of events:
09:42 - Our Network Operations Centre received multiple alerts relating to our phone platform specifically relating to a large number of phones deregistering from the platform. Engineers immediately started the investigation.
09:48 - Engineers identified the reason for instability of the platform as increasing request queue that is not being served.
09:51 - Engineers rebooted the registration backend.
09:52 - Issue was resolved and service was fully restored.

Root Cause
We identified that the root cause as a bug in the maintenance script that was run around 9:30. This script was run and interrupted mid way by human interaction, locking the call records table. Our system was backloging the requests to the backend database but finally exhausted the resources and started to drop new ones.
We are putting controls in place to try and mitigate this set of circumstances. During last week our vendor has prepared a fix for the maintenance script and this has now been implemented.
We apologise again for the inconvenience caused by this incident.
26th February 2018 at 10:10: Monitoring: Hosted Telephony
Our engineers are monitoring the system.
If any of the phones are still in unregistered state - please power cycle/ reboot the phone to speed up the process
26th February 2018 at 10:07: Resolving: Hosted Telephony
Issue is now fixed - all phones are re-registering.
26th February 2018 at 10:05: Identified: Hosted Telephony
The issue with one of the processes on the system has been found and it is being rectified
26th February 2018 at 10:02: Investigating: Hosted Telephony
Phones unregistering - unable to make or receive calls