May 20: Twitter Downtime

Glad that there is some sort of updates on here, as the problem with the site being down is that you can't post service updates on it.....

Bad news: Twitter is down. Good news: Users *need* it back up. Bad news: Eventually, users *won't* need it back up.

Why Twitter Why! What will all of us Zappos people do?

life in high-tech is hard. My own network is crawling at snail pace today without a visible reason, I had a dsl outage, yesterday, e-mail didn't work. I'm actually pretty happy that those things seem to cluster this time.

This is going to sound super dorky but...when Twitter is down I don't find myself angry or frustrated, I find myself thinking about how easy it would be for governments to shut down free flow of information. Twitter serves as a real time portal for ideas and up to the minute info. and the more i twitter the more i realize that so many members of the 'community' have brilliant and subversive and creative ways of changing their RL communities and passing that info on...I'm not paranoid or a weirdo (ok I *might* be a weirdo :p ) but it really does make me think.

I won't ditch twitter this time, but I doubt whether I will ditch the service next time.

Same old broken record....

Mass exodus to Pownce, anyone? Will Pownce have scaling issues? Frustrated Twitter users will find out. I'd sell Twitter to Yahoo and enjoy the reliability.

twitterless

I didn't realize I was a Twitter Addict. I'm a Twiddict going through withdrawal.

---Quote---

"The issue that we experienced today is actually not related to the problem we experienced yesterday. In turn, these two problems do not share the same cause with the downtime we experienced last week. At root is the database, but the cause is different."

---Quote---

You do realize how incompetent the statement above makes you look right?

The "growing pains" excuse is getting tired. Don't get me wrong... I love me some Twitter. Hope it's fixed soon!

Ugh, this must be very frustrating for you! Here's to getting it fixed soon.

Come on men! i miss twitter a lot

...too bad twitter can't scale...

I assume you're doing a great job. Awaiting a long-term stable service, go for it :-)

i just signed up this morning so this isn't exactly a good first impression. does this happen often?

Wow, I had no idea Twitter was down. I live at BrightKite now. http://brightkite.com/people/acompute... It's like a Twitter that doesn't break. Oh sure, there's no twooshing there yet, but a man can dream. P.S. Vote for David Cook!

stuff like this in a large ever growing database will happen, and can at the most inconvenient times.
Thanks for the information!

I can't...I just can't live w/o Twitter...where are you friend?

i may have been the straw that broke the camel's back on this one. I just joined yesterday.

twitter needs to get its act together and to have error messages that actually say what's going on when the site is down.

twitter is a lifeline for a lot of us, and it can't remain one if it doesn't work.

if you need a new host, get one. if you need new servers, move to them. if you need more employees, hire them.

you're begging another company to come along and do twitter right.

These things happen - hope you can get the service back up soon. As a former sysadmin, I feel your pain.

Barbara

http://www.youtube.com/watch?v=YuyYMk...

Twitters concept is great and Brightkite is not the same at all. They do need to do a large scale upgrade or someone else will using the same concept.

With each downtime, I get closer and closer to moving everything to FriendFeed. If all 950 of my followers were on FriendFeed there would be no contest between the two. Twitter is an embarrassment.

http://friendfeed.com/ericabiz/

Oh cheer up everyone. this stuff is harder than rocket surgery. why not go outside and enjoy the nice weather for a minute or two. Come back and you'll have something delightful to twitter away about.

I want my money back! Oh Wait . . . Never mind.

Twitterzzzzzz....you go down too much!

A more detailed answer would be nice, but sheesh, everyone, didn't you ever watch "Bambi"? As Thumper says, "If you can't say anything nice, then don't say anything at all."

It's freaking free!!! It's free... You are all whiners. Thanks twitter for making this thing that is up more than not free to us.

Now how will I let millions of people I don't know what it is I am doing *right now*?? I am being factious, of course.

Finally some folks at the end agree with me... It's free!!!

stand up a parallel twitter beta with a new architecture asap. I know zero about web site dev and scaling. Cannot be easy with following wars like between kevin rose, Leo laporte.etc. Get fixed before a celeb gets on it and you have a millions problem and not a thousands problem!

it seems that almost every time I look at Twitter there's something not working as it should. It's most disappointing.

.

While it may be fun to complain, essentially we're looking at a currently free service that is at the forefront in a communications revolution. It's like complaining about the reliability of your Model A car 100 years ago, except you paid big bucks for that.

We're at a critical juncture, however, and stability & reliability obviously need to be improved quickly, or Twitter will be a footnote - Web2.0's Edsel.

So the real question is: How can the Twitter community help???

Every day there's a new conference or blog re: the power of social networks, open source communities, mining the cognitive surplus, ad infinitum. How about practicing what we preach?

Twitter.com, what do you need?

Twitter users, what can you provide?

Let's solve the problem rather than bitching at what I guess is a rather stressed out tech crew.

I am new to Twitter and having a great time.. so thanks for the update!

I am new to twitter as well. Thank you for the update

Hey everyone, Twitter is not really down. Go check. (This comment only applies when Twitter is up.)

All I know is that this site makes me want cupcakes in a bad way.

I can post updates, but is there a reason my rss feed might be still down?

Thought it was just me...

well hurrryyy i have lots (140 char) to say.

gah! i was having american idol moments with others on twitter. we heart twitter. promoted twitter in a BIG way at an event today. long live twitter!

Seems to be down again. 2nd time today.

I'd hate to see the downtime issue cited in the future as a reason for Twitters ultimate failure, and the reason some other similar but stabler service's usurpation.

yep. down hard.

I was really looking forward to getting my election coverage from the people of twitter, rather than the inane talking heads of corporate TV.

I've already investing time in building a network to do just that for me. I'd really rather not do it again on another site.

Well, that`s great, but basicly teh thing is that twitter works really bad, it's always down or working terribly wrong,. that's actually the fact you should try to work out on and fix.

Well, thanks for the feedback at least. I thought it was just me for a while, good of you guys to keep the users up to speed.

And yes, I really hope you guys DO keep it together in the long term.

Oh hi. That doesn't tell me what is happening with the database. Are you creating indexes? Are you distributing the database to more nodes? Etc.

As an aside, you guys could use less bandwidth if you made the icons on web smaller in file size.

Will there be any data loss?

a lot of us have a lot of time invested in twitter. we'd like to know your plans for preventing it from going down again tomorrow and the day after. this is getting a little silly.

One of the first rules in customer service is admitting to the customer that you screwed up. Part of that, for Twitter, should be a snappy page that's put up when the service tanks. Let us know what's happening immediately, without having to go to blog.twitter.com, techcrunch.com, DaringFireball, getsatisfaction, or some other site to learn the status.

what database? what was the failure specifically?

start selling ads already and beef up the infrastructure. twitter might be down now. but it's slow as mud most of the day. call it, "api drag."

But seriously... I have a question. No I don't. I'm just wasting bandwidth. But YOU enjoy reading what I write, don't you. Good job today TwitterTechs getting things back up and running. I knew ya could. Also, it's nice to know Goldtoe has a real name. All and all, it's been a Tweetin' good day. Time for beer.

I've been on Twitter since they started providing their service (before they were flooded with new users). They have had performance and scaling issues from the beginning. I doubt money is an issue considering the amount of funding they have received. If they did need money, I'm sure it would be easy to get considering the value of such a large social network.

That said, most of us are too busy to deal with repeated outages and sluggish performance. The time I spend here is valuable, and the fact that I (and the rest of you) remain, adds value to this service...so in a way, we are "paying for this service." In addition, I think that charging a small fee in exchange for reliability and improved performance would go over well with regular users (such as the model used by Flickr).

I can't understand why they are not building a more robust parallel system or mirroring the current system such that situations like this (extended downtimes) could be avoided. Making changes to code in a live database..the only live database you have..is highly risky and IMHO a sign of inexperience/hubris. If they simply mirrored their existing system, they could make changes to the code, observe for unusual behavior, tune the database, and back up the valuable data that the system holds without any evidence of that activity to the end-user. Then, in an instant, they could deploy the changes and we'd be none-the-wiser.

As others have said, its only a matter of time before people become fed up and move to another free service.

Twitter trials remind me of ICQ in the early 90's, please just don't sell out to AOL!

Well, you ALMOST lasted 24 hours before you went down again.

Unfortunately down again.

any idea when twitter will be up again? its been down 2 days now

Hi-

Since joining Twitter I have not been able to login (weekdays) after 4pm ct.

Is this coincidence, or a technical issue??

Mark

its apparently based on some shitty code and a poorly designed, non-replicated database. Yes, its free, but people use it.

Grrrrrrrrrrrrrr.
Twitter is down most of the time this week and last. I'm kinda over it.

back up it looks like

ya its back up now. They need some kind of backup here. 1000s use it and rely on it. So if they need to, charge money.

Can we say backup servers?

Or perhaps redundant failover clustering of the servers. Now that they just got more investments

Twitter very slow even when up.

Is twitter still down. I havent been able to get onto homepage since 4-ish EST

Very hit and miss here. This will all make for a better twitter in the long run

Thanks, as always, for the updates, Jason. Love 'em.

Read the Blog. Thanks! This obviously more in depth than simply needing more server space or network programmers, right? If it's $$ related, can we donate somewhere? :)

Thanks for finally providing a wee little bit o insight into what's going on. Now maybe we can help instead of feeling powerless and lashing out.

Those spikes on your graphs could be bots. Have you noticed the activity on Summize? do a search for Ansel Adams or John F. Buckley & you'll notice one cloned bot's ability to influence the top 10 tweet words list. Could those correlate with your spikes?

Then there's what I call the mass stalker bots. I posted about one on the 5/19 twitter down thread on getsatisfaction. It was adding about a dozen followers per minute, and is now following over 32,000 ppl. I've got at least 2-3 of them (and I've only got 75 followers.) Have you considered capping the number of people a user can follow at something ridiculous like 5,000? having 32,000 followers times an unknown # of stalker bots has got to be a tremendous extra load on your servers.

I bet if you think this might be part of the problem, there's a whole army of users willing to help track them down. Even if a small portion of your user base spent a little bit of time helping to isolate your problems instead of twitting about the their lunch or lashing out, that's a mind-bogglingly massive trouble-shooting force, and maybe your employees can actually enjoy their pizza.

And what about @Perliux look at this: http://twitter.com/Perliux?page=5 something was going on with Anarquia.sms. Do a summize search on anarquia and it looks like there were other users that had problems too.

Do these correlate with spikes on your hitherto secretive graph?

I'm no techie, but I'm pretty familiar with standard troubleshooting procedures. you know, like when stuff starts going wrong, simplify simplify simplify. The fastest way to trouble-shoot a problem is cut it in half. How about when you see your next spike, cut SMS. If the spike goes away, you know the problem is there. If not, bring back SMS and cut API (like I know what that is). And then SHARE the information.

You've got a large, tech-saavy, nearly addicted user base that is willing to help out if you're willing to provide the leadership to take advantage of it.

well, switching to pownce or friendfeed doesn't look that bad now...

Twitter is now on "downtime"

i have joined pownce as well, just so I have a foot in both camps

The more technically inclined may find this post about Twitter's architecture interesting: http://dev.twitter.com/2008/05/twitte...

that was an interesting link, sms traffic is not something you can rely on, one sms from my cell phone company arrived 20 days after it was sent, is there any update on twitter today as it appears to be running slow.

it was about one hour behind on twirl for long time, but now it is updating in a timely fashion.

Do you want today's (5/23) Twitter: "Help! I've fallen and I can't get up!" problem report here or elsewhere?

yeah I just woke up and checking the overnight twitter stream to find that I can't tweet replies, this has not been a good week for twitter, they have to be feeling the pressure by now

Is this some of that awesome RoR scalability I've heard so much about?

Not sure why people keep hammering on the "Twitter is free" bit. Yes it is, and I'm VERY grateful that they've put it out here for us to use. And I have no problem believing that running Twitter is a *very* complicated undertaking and that Twitter staffers are working their tails off trying to get it stable. But here's the thing: When it doesn't work, particularly for days on end, then it's not useful to me, whether it's free or not. I'm not trying to blame or shame Twitter; that's just a fact. If it's repeatedly not useful to me, eventually, I'll just quit coming by to use it--not because I'm pissed, just because I'm busy and Twitter's no longer a useful tool. I assume that others will do the same. I don't want that to happen!

does anyone know why twitter is down again

Suggestion... add 'Beta' next to the Twitter logo and you'll be communicating the correct 'operating stage' message for the company.

Twitter needs to port from a CMS to a Messaging framework, this does not happen overnight and in fact is a complete rewrite of the underlying system (kinda like having to do a engine rebuild on a car while its going 75 miles an hour... in heavy traffic - if not impossible - very hard).

Call it "Re-Beta" if you like.

I believe that Twitterers will appreciate the honest repositioning and understand that they are in FACT actually helping to 'create' the utility - not just 'use' it.

In the old days it was called a barn raising; when your neighbor helped you build your barn because they knew you'd help them. Since Twitter has been responsible for many already building their businesses, reputations, networks. It might be a good time to understand that they need some help now. The cool thing is that all you have to bring is 'understanding'.

Hope it gets recover soon
I use twitter frequently i love it

You're right to some extent. If you follow certain people/organizations than you can be updated when they send a tweet or mechanic jobs details. More and more organizations are now using twitter to deliver information. Just search for major news agencies and you'll find the ones who have twitter accounts.