{"id":5790,"date":"2015-03-20T13:48:13","date_gmt":"2015-03-20T13:48:13","guid":{"rendered":"https:\/\/www.gosquared.com\/blog\/?p=5790"},"modified":"2019-11-28T11:53:03","modified_gmt":"2019-11-28T11:53:03","slug":"outage-2015-03-20","status":"publish","type":"post","link":"https:\/\/www.gosquared.com\/blog\/outage-2015-03-20","title":{"rendered":"Outage on Friday 20th March"},"content":{"rendered":"

As you may have noticed from our status updates<\/a>, we suffered a lengthy interruption in service this morning.<\/p>\n

To begin with, I want to apologise for the frustration this has caused you. As users of our own product, we know how disruptive issues can be. And as users, we’d also\u00a0expect to know what went wrong, and what’s being done to prevent it happening again. That’s what this post is all about.<\/p>\n

What was affected?<\/h2>\n

Between 10:32 and 12:09 UTC on Friday 20th March, our tracking API was not functioning correctly. This means that GoSquared stopped tracking new data from all sources during this period.<\/p>\n

Additionally, during this period, the APIs powering GoSquared’s web apps were taken out of service in response to failing health checks related to the tracking issues.<\/p>\n

Why did this happen?<\/h2>\n

We process billions of requests a day. To deliver our fast, real-time service at such scale, we run clusters of numerous instances on Amazon EC2, the service we use to run our servers.<\/p>\n

Unfortunately, one of the EC2 instances completely stopped functioning without any notice or warning signs. All health checks and metrics were normal prior to the fault, otherwise we would have taken evasive action to prevent\u00a0the issue.<\/p>\n

The EC2 instance was running a node that was part of the database cluster we use for ingesting data into GoSquared.\u00a0Losing this node put the cluster into an unhealthy state, making it unable to handle data reliably.\u00a0This meant the system could no longer process data. Frustratingly, our failover plan did not prevent an outage.<\/p>\n

We tried to recover the node, a process that should only take minutes, but the EC2 instance was entirely unresponsive – refusing to even shut down or restart. Instead, we had to migrate over to a new cluster which is non-trivial and accounted for the length of the outage.<\/p>\n

That sounds scary. Was any existing data lost?<\/h4>\n

No. Only new data tracked during the outage period was missed. All existing data already saved in GoSquared is\u00a0safe and sound.<\/p>\n

Why did the node stop working?<\/h4>\n

We’re confident it’s due to a faulty EC2 instance for reasons outside of our control.<\/p>\n

Future prevention<\/h2>\n

People sometimes warn about the stability of EC2, but in our experience, instance instability like this is rare in\u00a06 years of running on EC2. Nothing is 100% reliable though, and unfortunately this affected\u00a0us today.<\/p>\n

Fortunately, we already know what to do to prevent this happening again:<\/p>\n