Jason Speck's dashboard

We had the chance to fully review the outage last night with our system admin. The main cause of the problem was a limit set in our virtual environment for disk space. After the initial migration period of moving to SoftLayer, everything has been very smooth and the servers have been handling our fast growth very well.

The reason we hit the limit was pretty embarrassing, but I guess it can happen to all of us. We have more than enough storage on our servers, including iSCSI which allows us to quickly expand, but a backup service that we setup in the beginning as a test was processing backups on the server and eating up disk space each time it ran. At a certain point, we hit the limit and it killed our sites.

We do have backups on S3 and SoftLayer's remote datacenter, this was just a result of some initial processes from our migration running on the server.

The problem was trivial and should have been resolved much faster. I admit that our response time to fix the issue was awful as well as our ability to communicate the issue. We are updating our monitoring services to provide more redundancy and we are working on some ways that we can communicate with you if this ever happens again. Possibly a Twitter feed or having our blog hosted in a remote datacenter. Once it is ready, we will email everyone.

We apologize for any problems this may have caused. Please get in touch if you have any further concerns.

To receive updates, please subscribe to our RSS feed on our blog:

http://www.wildbit.com/blog

Thanks,
Chris

crayolalettuce: We'll check this out and email you.

I'm still getting:

svn: Commit failed (details follow):
svn: MKACTIVITY of '/lacense_rails/!svn/act/6b41b6da-1e46-0410-81e5-a65acb6fdc1c': 400 Bad Request (http://svn.olive.beanstalkapp.com)

when I try to update or commit from the command line.

We've fixed the issue and both the web and svn servers are back online. I will follow up shortly with an explanation.

Thanks,
Chris

We are looking into this right now. Should be fixed soon.

Chris

Thanks Randy. It's a bug, we'll take care of it.

Chris