As many of our regular readers might have noticed, along with the additional content we seemed to have lost some stability.
Waring: Technical information spoiler!
This is just a quick note to let you all know that the problem has finally been identified (a connection loop triggered by multiple simultaneous article views originating from Firefox 3.6.12 that caused no error codes to be thrown but that would critically max out the server memory in a matter of seconds.)
It’s taken us this long to track it down because there was no direct link to the event in the server logs because technically there was no error until the web server went down. When the logs were checked we were given every indication that it was some kind of a distributed but targeted denial of service attack because the only thing it showed were several hundred connections requests in the space of a second. Also, any single event wouldn’t crash the server. It took many people all using Firefox, to view the same article at the same time. We don’t believe this to be a problem with Firefox, it only served as the catalyst to the actual problem.
So, although there was no direct link to the offending code, through trial and error and an obsessive drive to know not just that the problem was being addressed, but to know exactly what was the cause, Quiet Speculation is now stable again.
Technical spoiler complete.
We thank you very much for your patience (even if you weren’t patient, if you’re reading this you came back to us despite the issues). It’s been a frustrating ride for us as well, but please know that no matter the scope of the issues, we won’t rest until your experience is as perfect as we can make it.