Important Downtime

You may have just experienced the downtime that occurred, but I thought I'd go a bit in-depth on what happened here. As of right now we're not sure exactly what happened, but it seems that our Varnish server (caching) had a "bit" of a hiccup. What exactly /did/ happen to it is, at this point in time, unknown. However, we experienced the following;

At about 23:09 GMT +0, our forums went cold. I wasn't by my computer at this point and couldn't look into it, but received a couple of messages from kurozael. At about 23:25 I arrive by my computer to a plethora of messages saying the site is down, and thus the investigation begins. Around five minutes later (and testing different things), I get to the conclusion that the caching server had a hiccup. kurozael had already redirected the domain towards our webserver, but it'd take too long for the local caches to update, so I disabled Varnish and had the frontserver avoid it.

To actually understand anything of what I just said - the new environment for our sites is getting a bit more extensive. At the front of everything web-based we have a single server (that can failover to a different server if needed, almost instantly) acting as a load balancer. Behind that we have # servers (as in we're not telling you) taking care of web requests. Behind that again we have database server(s) that handle all database requests.

Essentially, there's always at least three servers involved in every request to this forum. That is not counting any external API or third party involved. The load balancer itself caused no issues per se, but something on one of the web servers did. Until I figure out exactly why the outage occurred, the caching server will be disabled.

Sorry, everyone. Shouldn't happen again.

tl;dr: Forums were down from 23:09 to 23:38 GMT +0.

 

Spoiler: dank meme

Keep up the good work [HASHTAG]#transparency[/HASHTAG]

 

Have an image!