Server / Network Updates

[Dessum]

Thanks again Tweak and good luck

[Tweak]

In case anyone noticed the short outage earlier. It only affected us for a few seconds while the network redistributed it's load. Report from my datacentre is below.

Between 14.58 and 15.02 this afternoon there was a brief network outage on some parts of the network. This was caused by a denial of service attack on one of our customers. Engineers enabled the DOS protection system within a few minutes.

Also in regards to the crashing issues, it seems the server is stable again, I have issued some limits on another site on the server which seemed to be causing them a few months ago and since then everything has appeared stable. A forced cache has also been introduced to cut the number of calls for images on the forums here, as we rarely change the images they are now cached on your system. This should improve load time to some extent.

Additionally a bug seems to have been squashed which was causing some issues with the server. Although they tended not to become too apparent as I managed to jump in and reboot the web server before it became an issue there were a couple of incidents where connections were refused. This was caused by a bug in a module within the server software causing processes to freeze and use slots, this slowly built up until the software thought it had hit its limit. This appears to now have been rectified.

[Tweak]

Apologies for the outage we experienced earlier, report from the datacentre is below.

We have just recovered from a very serious Distributed Denial of Service attack which took out parts of our network. Many customers will have seen very serious connectivity problems starting at about 16.30 BST this afternoon. The attack is ongoing but all services are now back, and we will have more details shortly.

All services have been back to normal since 17.13 BST and although the attack continues it is being completely mitigated by the DDOS solution. We have identified the target of the attack and removed it from the network. (The target was a server purchased earlier today which seems to have been purchased with the express intent of damaging our network.)

Once again apologies but this issue was outside our control but was dealt with in cooperation with our technicians.

[White Wolf Guardian]

It's good to see the attackers are gone

[Tweak]

They aren't, they just aren't causing a disruption now.

[Tweak]

IRC connectivity should now be available again.

IRC has been online all afternoon but due to DoS attacks on the servers hosting the web-based client we use all network access to it was disabled. We worked with their technical team to ensure all was working before switching back on the link a few moments ago.

Apologies for anyone who was unable to get into IRC, all should be fine now.

[Tweak]

Received from my datacentre:

On the night of Sunday 23rd May we are going to be completing a network upgrade. You will experience a short period of downtime sometime between midnight and 3am BST. The downtime will be no longer than 10 minutes.

Additionally to this improved network a solution should go live to completely nullify any future DDoS attacks.

Apologies for any issues this may cause.

[jakob_furreault]

yay! i didnt even know you could nullify dos attacks

[flames]

If it's to give us further protection from DDoS attacks then any inconveniences it could cause should be more than worth it. It's a pain that the network is targeted so frequently.

[Tweak]

The work above was completed at 4am BST.

[Tweak]

Apologies for the extended downtime last night, unfortunately I was stuck in Chicago for 4 hours and have only just had a chance to sit down and solve the issue, everything should be fine now.

[Asia Kali Yusufzai]

Good work *claps*

[Timmy Fox]

Pawsome,  but what was the issue if i may ask?

[Tweak]

No idea, was too tired to check.

[TechRoo]

No idea, was too tired to check.

Just rebooted the server?