PoP Infrastructure downtime resolved, 06/04/2019 01:36PM UTC Stockholm, Sweden Prague, Czech Republic Beijing, China Columbus, OH Seattle, WA Boston, MA Paris, France Vancouver, Canada Singapore, Singapore Amsterdam, The Netherlands New York, NY Denver, CO San Jose, CA Atlanta, GA London, United Kingdom Frankfurt, Germany Montreal, Canada Chicago, IL Los Angeles, CA Dublin, Ireland Cato Management Application Miami, FL Toronto, Canada Ashburn, VA Dallas, TX Calgary, Canada Madrid, Spain Portland, OR Dubai, United Arab Emirates Johannesburg, South Africa Guadalajara, Mexico São Paulo, Brazil Sydney, Australia Melbourne, Australia Ho Chi Minh City, Vietnam Hanoi, Vietnam Bangkok, Thailand Kuala Lumpur, Malaysia Mumbai, India Tokyo, Japan Seoul, Republic of Korea Shanghai, China Hong Kong, China Tel Aviv, Israel


Notification history

06/04/2019 01:36PM UTC
We have identified that our service is currently down due to a infrastructure outage. Cato's engineering are working to restore the service ASAP.
If you have any further questions, please contact our Support team at support@catonetworks.com
We are sorry for the inconvenience caused.

RCA details:
Incident Details: 
On June 4th at 12:51 UTC 2019 Cato experienced a global disruption to its service. The 
disruption happened as an administrator inadvertently and manually bypassed a production 
failsafe, initiating a global propagation of non-recoverable configuration change. This caused 
all tunnels to disconnect. 
Service was fully restored at 13:32 UTC. 
Immediate Actions: 
In order to restore the service, the following actions have been taken 
1. We have rolled back software update on all Cato PoPs to the previously stable 
configuration before the incident
2. We have tested extensively and confirmed with customers that the service has been 
restored
Preventive Steps: 
 
1. We have immediately disabled admin privileges on production, to be reintroduced in 
a controlled manner once the following steps are complete.
2. We will be conducting a review of all access to our production environment.
3. We will be adding additional approval steps and failsafe mechanisms to the 
deployment of software in our production environment to prevent such incidents in 
the future
06/04/2019 01:12PM UTC
We have identified that our service is currently down due to a infrastructure outage. Cato's engineering are working to restore the service ASAP.
If you have any further questions, please contact our Support team at support@catonetworks.com
We are sorry for the inconvenience caused.
06/04/2019 01:00PM UTC

06/04/2019 01:00PM UTC
We have identified that our service is currently down due to a infrastructure outage. Cato's engineering are working to restore the service ASAP.
If you have any further questions, please contact our Support team at support@catonetworks.com
We are sorry for the inconvenience caused.
06/04/2019 01:00PM UTC
We have identified that our service is currently down due to a infrastructure outage. Cato's engineering are working to restore the service ASAP.
If you have any further questions, please contact our Support team at support@catonetworks.com
We are sorry for the inconvenience caused.