PoP Infrastructure downtime resolved, 06/04/2019 01:36PM UTC Cato Management Application Ashburn, VA Atlanta, GA Boston, MA Chicago, IL Columbus, OH Dallas, TX Denver, CO Los Angeles, CA Miami, FL New York, NY Portland, OR San Jose, CA Seattle, WA Montreal, Canada Vancouver, Canada Toronto, Canada Calgary, Canada Amsterdam, The Netherlands Dublin, Ireland Frankfurt, Germany London, United Kingdom Madrid, Spain Paris, France Prague, Czech Republic Stockholm, Sweden Tel Aviv, Israel Hong Kong, China Beijing, China Shanghai, China Seoul, Republic of Korea Singapore, Singapore Tokyo, Japan Mumbai, India Kuala Lumpur, Malaysia Bangkok, Thailand Hanoi, Vietnam Ho Chi Minh City, Vietnam Melbourne, Australia Sydney, Australia São Paulo, Brazil Mexico City, Mexico Johannesburg, South Africa Dubai, United Arab Emirates


Notification history

06/04/2019 01:36PM UTC
We have identified that our service is currently down due to a infrastructure outage. Cato's engineering are working to restore the service ASAP.
If you have any further questions, please contact our Support team at support@catonetworks.com
We are sorry for the inconvenience caused.

RCA details:
Incident Details: 
On June 4th at 12:51 UTC 2019 Cato experienced a global disruption to its service. The 
disruption happened as an administrator inadvertently and manually bypassed a production 
failsafe, initiating a global propagation of non-recoverable configuration change. This caused 
all tunnels to disconnect. 
Service was fully restored at 13:32 UTC. 
Immediate Actions: 
In order to restore the service, the following actions have been taken 
1. We have rolled back software update on all Cato PoPs to the previously stable 
configuration before the incident
2. We have tested extensively and confirmed with customers that the service has been 
restored
Preventive Steps: 
 
1. We have immediately disabled admin privileges on production, to be reintroduced in 
a controlled manner once the following steps are complete.
2. We will be conducting a review of all access to our production environment.
3. We will be adding additional approval steps and failsafe mechanisms to the 
deployment of software in our production environment to prevent such incidents in 
the future
06/04/2019 01:12PM UTC
We have identified that our service is currently down due to a infrastructure outage. Cato's engineering are working to restore the service ASAP.
If you have any further questions, please contact our Support team at support@catonetworks.com
We are sorry for the inconvenience caused.
06/04/2019 01:00PM UTC

06/04/2019 01:00PM UTC
We have identified that our service is currently down due to a infrastructure outage. Cato's engineering are working to restore the service ASAP.
If you have any further questions, please contact our Support team at support@catonetworks.com
We are sorry for the inconvenience caused.
06/04/2019 01:00PM UTC
We have identified that our service is currently down due to a infrastructure outage. Cato's engineering are working to restore the service ASAP.
If you have any further questions, please contact our Support team at support@catonetworks.com
We are sorry for the inconvenience caused.