As you might have noticed we had a server outage for about 7 hours last night (or day depending on where you live!). The root cause was a database server failure, and the reason why it lasted so long is because it happened just after I went to bed . We are still unclear as to what caused the failure, but we’re continuing to look into it to try and prevent whatever happened from happening again.
The database server is in a manually patched state right now, and as we mentioned we’ll be working on figuring out what exactly when wrong so we can implement a proper fix. We will likely have a scheduled maintenance within the next week or two to rollout a proper fix for the database system.
I’m also going to be working on some implementing some better monitoring and alerting functionality so I have a better chance of being woken up when there’s a huge outage like this.
Sorry to all affected