Server / Network Updates

[Tweak]

! Made some changes to some security systems to reduce the amount of false error-reports.

This will have a knock-on effect due to less IO usage.

! Also fixed an error with the main index page for the root domain, this will have the same effect as above - though to a lesser degree.

[Tweak]

Today's downtime was caused by a data centre failure.

At approximately 11am Central time a HVAC failure occurred causing a major switch to overheat and shut down for safety reasons. On repair of the HVAC system it failed again and was subsequently replaced.

The server was not affected in anyway other than a loss of network connectivity, as soon as the ambient temperature was reduced and the networking infrastructure was restarted the systems came back online immediately.

Apologies for the downtime.

[Tweak]

I wish to extend my apologies for the extended downtime experienced over the past couple of days.

Since our infrastructure changes we have had much less downtime and a much more stable service for all customers here including those with high-activity accounts.

Unfortunately we experienced a large scale Denial of Service attack, again our infrastructure is built to withstand most DoS attacks without a degraded service but due to the scale of this attack our upstream providers were forced to set the main server IP to a null-route (basically make it point nowhere) to reduce the huge stream of traffic from the attack.

As the attack is still ongoing and to protect the general network infrastructure the IP is to remain offline and so the need to migrate to another IP and refresh licenses caused some delay in getting everything online.

This is one of the rare force majeur events that occur in which we cannot predict or do anything but work as swiftly as possible to restore service.

I once again extend my apologies for the downtime and as always extend my gratitude for your continued business.

[Tweak]

! Security rule signatures updated.

Please report any issues with 403/406 errors should they occur.

[Furpaws]

! Security rule signatures updated.

Please report any issues with 403/406 errors should they occur.

it says connection reset when I post, it was working yesterday but not today. is it related?
also it asked me to answer math questions when logging in earlier,it didn't do it before. 

pretty sure I have no viruses, this is a mac and I check it for viruses.
I tried different browsers & they all say connection reset
I can post this with a proxy but without it I can't post anything not even a bug ticket. 

[Tweak]

Is this still an issue? It shouldn't have been caused by my changes.

[Tweak]

ANNOUNCEMENT :: NETWORK MAINTENANCE
13th March 2013 :: 02:00 - 04:00 GMT

I have just received the following notice from the data centre.

To improve network performance and resilience we will be conducting network upgrade maintenance on the morning of Wednesday 13th March 2013 betweek 02:00 and 04:00 GMT.

Whilst no outage is planned, connectivity to all services should be considered at risk during this time.

[Tweak]

Unfortunately due to a database failure some data has been lost.

As a result, all posts, PMs, changes, etc made within the last 24 hours should be deemed lost. Unfortunately only daily backups are taken and as such no newer backup can be restored.

We apologise for this inconvenience.

[WingedZephyr]

We had another database failure yesterday which resulted in some data loss. Tweak is doing his best to make sure this sort of thing won't keep happening, but for now, I'm afraid that any posts and messages that were made in the past 24 hours have been lost.

We apologize if this has caused any inconvenience.

[Tweak]

Apologies for the bad service as of late.

A few changes have been made and we'll be checking to ensure the infrastructure is once again sound.

[anoni]

Was there another Database failure? I can't seem to find some posts I made last night

[Tweak]

There was, 99% of the database was recovered due to the changes made but it's possible some were unrecoverable.

[Tweak]

It's 1am on the 4th day of only minimal sleep so I'll keep this short.

Unfortunately we were hit with a file system corruption leading to us having to rebuild to another machine - this was then met by the same error - resulting in another rebuild. At this time all services are online but may be inaccessible whilst everything updates.

Service may be a little slower than usual whilst we investigate the possibility of our software causing the file system issues. Please bare with us whilst we get back up to speed.

I want to apologise for the slowness of this fix, even with only 4-5 hours a night of sleep I lost a lot of time due to work.

[Nikolai the Lynx]

I haven't been able to even reach the site since I was last on, good to finally get back.

Sounds like you are working your butt off trying to get this fixed, so I wouldn't worry, we appreciate your efforts.

[MrRazot]

Thanks Tweak, you're a legend