Post Mortem | Service Disruption Notification resolved, 01/23/2019 09:15AM UTC Dubai, United Arab Emirates

Notification history

01/23/2019 09:15AM UTC

01/23/2019 08:00AM UTC
Cato's Engineering has identified application fault within Dubai's PoP causing service disruption.

The net effect of this was that certain flows/connections which were being processed by the relevant POP were dropped leading to disconnections, packet loss and sessions terminations while being migrated to another PoP.

The PoP had connectivity issues and our automated system decided to place it in maintenance mode in order to migrate tunnels to the nearest PoP, due to a timing bug it entered maintenance too late after it was already recovered, and so stayed in maintenance.

Resolution and Remediation
Issue was mitigated, bug was fixed and fully patched in all relevant PoPs.
implemented to prevent any future re-occurrence.