The Problem with Web-based Everything
So, I've been tinkering with the free version of Toodledoo - a web-based GTD task manager, and I was thinking about upgrading to a "pro" account. Unfortunately, they had a storm run through last night, which engaged the generators. When the generators kicked in, something didn't work right, and power was lost. This in turn caused a database crash, which caused it to corrupt, and they are still down now.
Here's what their homepage states right now:
So, here's the story. A big storm went through the city where our datacenter is located. The datacenter decided to proactively switch to generators. During the switch, something got screwed up, and the power went off for a few minutes. As (bad) luck would have it, this caused our database to get corrupted. We are currently working to bring it back online and restored from the live backup. The crack team at Rackspace is on the job. Thanks Rackspace! Unfortunately, the database is so large, that it will take some time to transfer and verify all the data. Hopefuly not more than a few hours. We know that this is very bad, and we apologize for any inconvience that this will cause. Please check the forums when we are back online for a full report.
Update: Its obviously taking longer than we expected and we are really sorry for that.
Now, I'm not paying anything for the service, and I'm fine with the downtime. However, I don't think I'll be upgrading anytime soon - this outage tells me a few things:
- They don't use UPS's.
- They don't use more than one data center.
- They likely don't manage their own servers.
Again, all of this is fine - it costs money to do all these things, and I understand the decision to not do it. However, when I pay for software as a service, I expect the software and the service to be highly available.

Comments
I've got all of my production
I've got all of my production servers colocated into Tier 1 facilities, and the only UPSes I've got are attached to the local SAN storage. When you're in a facility like that, you really feel like you don't need the UPSes, because the power plant is so solid, and really, introducing a battery backup is just another part to break at that point.
On the other hand, my production facility used to be located in a Tier 2 colo, which is nearly as good. During the course of a nightly upgrade, someone who worked for the colo kicked our plug out of the socket in the sub-floor space and we lost power to one side of the rack.
I called them about it, and the checked. Yep, it was unplugged. So they plugged it back in...to the other circuit belonging to the other side of the rack. So I've got a 20 amp circuit with somewhere around 35 amps of power tring to go through it. So of course, the breaker trips and we lose both sides of the rack.
This is why we're in 2 tier 1 colocations now ;-)
UPS's in Colo facilities
Agreed on the UPS thing - we don't have them either. The key is that our (and I'm guessing yours does too) colo has huge batteries that can supply power while the generators are being fired up.
I have no experience with RackSpace, but either they don't have that setup, or it wasn't tested very well.
Post new comment