January 8th downtime

[Tweak]

Firstly we wish to extend our apologies for the downtime today and explain the reasoning behind its cause.

As many of you will be aware the forums have had some hiccups lately caused by what I had assumed was another site on the server. However, the same hiccup occurred last night at roughly 5am GMT, the server notified me and I was able to watch it go down and discovered the cause is actually these forums. Now this has been confirmed I can change the configuration and make changes to try and mitigate the issues. This is the good news.

Unfortunately as the server rebooted one of the hard disks failed and caused the server to hang on it's File System Check. This resulted in all remote access failing and I had to contact the data centre technicians to take a look. At roughly 4pm we were able to get the server booted and could explore the extent of the damage. It has come to light that the secondary disk drive failed, taking with it all backups. At these stage this is not an urgent matter but it does mean we are currently running without backups.

No other damage appears to have occurred and the forums are running fine, expect a few patchy times whilst I get them configured to run smoothly. At this time we are still awaiting the hardware replacement, when this is scheduled the forums will be taken offline for approximately 15 minutes. More details on this will be given once they are known.

Once again, our apologies for the disruption caused.

[Garth]

Thanks for sorting it all out Tweak, I know you've been working hard to get the forums and IRC back up again.  You deserve a huge thank-you from the entire forum, and a good rest now too!  n.n

[White Wolf Guardian]

I hope that hard drive gets to you as soon as possible, thanks for keeping the place up and running x.x

[Alexei]

Thanks for getting it up and running so quick!

We all appreciate the work you do.

[Vee Katame: His Wolfy]

Thanks for getting things going again. I know how hard you were working on it.
It was still fun hanging out in the IRC for a bit this morning while the site was down.

[Ares the Ram]

Thank you for getting the forums up again Tweak. Youre awesome!

[✡]

thanks for getting the forums back up tweak, you're a real hero

[Tweak]

I hope that hard drive gets to you as soon as possible, thanks for keeping the place up and running x.x

The hard drive is in stores, it's just getting in a technician as our setup isn't standard and the monkey techs aren't  attuned to it.

Thanks for getting it up and running so quick!

We all appreciate the work you do.

Quick? It was down for 12 hours

Thanks for getting things going again. I know how hard you were working on it.
It was still fun hanging out in the IRC for a bit this morning while the site was down.

IRC is always fun, you should spend more time there!

thanks for getting the forums back up tweak, you're a real hero

Not like the forums were the only things down.


A couple of changes have been made to the forums and server to try and reduce the crashing we have been experiencing. The forums now have load sensors built in to disable features based on server load and completely disable the forums when load reaches a high amount - a quick test yesterday showed this worked.

The server itself has also been tweaked to handle connections better which so far is looking pretty good, further changes will be made as necessary.

A backup system of some sort is also in the pipeline, whether this be merely a way for giving official announcements, a "BRB" page, or a full server is to be decided.

[iKero-chu™]

Thanks Tweak.

We all love you.

[Especially me.]
;3

[Tweak]

Thanks Tweak.

We all love you.

[Especially me.]
;3

LIES!


The hardware replacement has been scheduled within a window of 10:15 - 10:30 GMT (4:15 - 4:30 Central), during the replacement the server will be offline for approximately 15 minutes whilst the switch is made.

As I'm away from my desk at this time there may be a slight delay with all services coming online, they will be brought up ASAP.

Apologies in advance for the inconvenience this may cause.

[White Wolf Guardian]

Will the IRC link be up again ?

[Tweak]

IRC runs off the same server, so no. To be honest the services that are likely to be delayed are IRC and the Search daemon.

[Tweak]

Hard drive was successfully replaced by 10:50am GMT. A backup is currently running.

[Tweak]

Apologies for today's downtime.

First, to quash rumours - no we were not attacked.

Turns out we had a second hardware failure which took time to diagnose and fix. It was nothing critical and during this time the server was still running correctly, just no outside connection could be made.