TweetTrak is offline indefinately

FYI, TweetTrak is still running, and you may get intermittent messages from it as it will continue trying to send messages. The alternative is to stop it completely and when it starts back up to not send any of the missing messages.

Unless popular sentiment is to just stop it altogether pending a resolution of this issue... its probably the best course of action.

Apparently things are still going through atm.. perhaps email was just a warning and something else was causing errors (like if search was unavailable).. or perhaps was only temporary.

Regardless, I won't be able to make such a change in a timely manner, and hopefully we can manually address issues as they arise, but that is in Twitter's hands.

Guess I'll just wait and see when Twitter responds (probably sometime tomorrow) and take things from there.

Got an email from two people at twitter, the API Lead Developer and the Head of Operations below. I, of course, think I make my case pretty well, but ultimately leave the decision to the community. Perhaps i am biased and unable to see their justification.

So, get a cup of coffee, sit back, relax, and make yourself comfortable... its a bit long (i can get longwinded), but I think there are many good points that were worth making. If you would like to comment, please add it as a response to this comment, rather then as a response to the original problem.

With any further ado... my email response, including the email exchange back to the beginning:

Please see my comments to both Jeremy and Adam's email below...

On Tue, November 4, 2008 12:03 am, Jeremy LaTrasse wrote:
> Ryan,
>
> I'm very sorry we had to disable the white-listing of your app. Our email
> bounce rate was skyrocketing (we usually don't bounce more than 1% of
> outgoing mail each day, the *USERNAME OMITTED* incident sent that number to almost
> 10%), and we are diligently trying to keep our bounce rates down so that
> ISP's don't block legitimate mail to users in their domain. I am really
> glad
> that you are going to take steps on your end to limit potential runaways
> like that one.

Then it seems to me that the problem should be addressed on the email notification side of things, since that is where the problem lies. That way, the source of the issue wouldn't matter. Consider that what would happen if I, like *USERNAME OMITTED*, had an email address configured that would bounce mail messages to it, and I was featured in an article on Slashdot or some Mainstream Media outlet, and 20,000 people decide to follow me. Since I have New Follower Emails turned on, I would get flooded with 20,000 emails, all of which would bounce back and cause the same exact problem.

So, why not implement what most people do... a filter on your 'no-reply@tweettrak.com' address to process that. In actuality, I noticed that the no-reply@tweettrak had changed last week to be forumlaic to allow you to parse the bounce emails so that you can disable them. (My DM messages are coming from twitter-dm-*EMAILADDRESS OMITTED*@postmaster.twitter.com now )

It would be one thing if the Request to send the DM was what the cause of problem (ie, Twitter's infrastructure couldn't handle an uptick in the API usage). And don't confuse the fact that the originating action was the DM from @TweetTrak with the true source of your concern... sending emails to bad email addresses.

>
> It was quite a scare for the operations crew, on the eve of such an
> historic
> U.S. election, to realize that the applications defenses to prevent such
> runaways were not behaving as planned. We pushed the panic button and had
> Alex disable your whitelist access.

So, it *was* a knee jerk reaction. And it was of the worst kind... a delayed over-reaction after the problem was no longer an issue as I had already noticed it before twitter had, and stopped the offending user's request to track very common terms.

If you doubt the veracity of my claims, check TweetTrak's public timeline, and compare it to its DM inbox.

This problem isn't unique to twitter. If we ask ourselves, "What would Google do" (bad pun intended, i'm sorry) when someone sets up an alert on common words and has a notification method that bounces. I honestly don't know what they do, but i'm pretty sure they would not disable their entire alert service. (and yes, i realize that TweetTrak is an API application while google alerts is an internal function at google, but I doubt average users see the distinction).

>
> We are working to improving ways that we can approach the issue that
> forced
> the de-listing today in the near term. I can assure you that Alex and I,
> and
> everyone here at Twitter will be discussing those options, and keep in
> constant communications about them with all developers and users too.
> We'll
> need some time to sort out how best to approach the various aspects to
> ensure that we don't repeat the *USERNAME OMITTED* incident.

So, in the meantime, the mere potential for a problem that took nearly 2 months to surface and is no longer causing problems is removing functionality that some users desperately miss and see as vital to using Twitter. (see http://tinyurl.com/63nh5d and http://tinyurl.com/5rz3yd for some user comments)

I am a software architect and design systems like this for a living. If I were to design things, the solution is simple. Require verification of ownership of the email address if twitter doesn't already do that. That is, twitter will disable all notifications to an unverified email. Then, send all mails from a user identifiable address so that all the bounces can be processed automatically to identify the user and the initiator of the email. That's already done apparently. Finally, process the bounce messages and once a threshold is met (20 bounces in a day perhaps?) that email address losses its verified status and is marked as bouncing. The user would then have to re-verify the email address or update the address to a correct one. Simple, and effective, and there's no burden on API developers to ensure that they aren't sending too many messages.

>
> I can tell you that neither engineers, nor operations staff here want to
> limit developers use of our services, the API's are near and dear to all
> of
> our hearts in all seriousness, and we want the developer community to
> continue to flourish.

Then I would expect that Twitter would treat the developers with respect and actually work with them rather then throw them in a penalty box with exagerated claims (see below). Indeed, the lifeblood of twitter *IS* its API as it is what allows twitter to function as a messaging framework and allow for innovate uses of Twitter.

>
> Please feel free to contact me via email if necessary, I'm sure that we'll
> be in touch with you very soon about getting you the type of access your
> application requires.
>
> Jeremy
>
> On Mon, Nov 3, 2008 at 9:21 PM, Alex Payne <*EMAIL ADDRESS OMITED*> wrote:
>
>> > To my knowledge, there has only been one user (*USERNAME OMITTED*) over this past
>> > weekend that tracked several very common terms, so they were sent 20
>> DMs
>> > per term per minute for about 14 hours before i noticed the load.
>>
>> *USERNAME OMITTED* was the account that our operations team brought to my
>> attention.

So, it was just one isolated account that had misconfigured their own use of a legitimate service that caused a large number of bounced emails that caused concern? Then why present the problem as "a handful of users getting thousands of direct messages from TweetTrak"? It wasn't "a handful" and the problem wasn't that they received direct messages, but rather that they chose to have those messages be emailed, and in turn, those emails were directed to a bouncing email address.

>>
>> > I am planning on incorporating statistics to track which terms are
>> getting
>> > sent at what rate, and i would be able to add in some protection to
>> > prevent a large number of DMs to be received by a particular user.
>> > However, I won't be able to add that until the weekend at the
>> earliest,
>> > and i don't believe that the problem is an issue going forward.
>>
>> Okay. Once you have a rate limit in place, we'll re-enable your
>> whitelisted status.

As i said, i was planning on adding those features, i didn't have an immediate plan for it, nor would i be able to get it done until at least after this weekend. There was much more important features to add that actually provide a benefit to users, such as allowing them to ignore updates from certain users or only send a single notifaction when there are two tracked terms that match a tweet. The latter feature is what will provide the statistics that will allow me to provide the data to determine when someone is receiving too many messages.

However, I would argue that limiting the number of messages is the WRONG approach, since those limits are arbitrary and like some of my users have said... they appreciated being able to receive a spike in notifications for some thing like #sandiegofire.. if they were limited to 200DMs a day... when they most need the service, they would be shut out.

>>
>> > It seems as if you are punishing everyone for the actions of one (or a
>> > few) users, and I implore you to monitor the usage from TweetTrak
>> rather
>> > then disable its usefulness
>>
>> It's not fun for me to temporarily remove you from the whitelist; my
>> job is supporting developers, not making their lives harder. That
>> said, we're responsible for the consequences of API apps like yours.
>> Every one of those messages to brain02 resulted in a bounce message
>> from his email provider, as his account settings were to deliver
>> direct messages to email. We now have to smooth things over with said
>> email provider. That's why we had to take you off the whitelist in a
>> hurry.

Again, I believe the proper place to handle that is in processing your bounce messages and disable (temporarily or permanantly) emails that are bouncing. And the "smoothing things over" conversation is a simple one: "One of your users, configured an application that uses our API to receive a large number of notifications and he had his email set to send emails to your domain. We are apologize for the large number of messages and are working on a system to detect these bounces and disable mail to them, and expect to have that implemented very soon". Nearly every ISP is willing to work with people on a case-by-case basis, unless you're a small player (Twitter isn't, IMO) and you're dealing with AOL (I hope for your sakes it wasn't AOL).

>>
>> > Also, I would think that it would make sense for these limits to be
>> > enforced from within the API. I understand that requires more
>> processing
>> > within Twitter, but the time and effort spent on that would reduce the
>> > amount of time spent chasing down abuses of the API. I would think
>> that
>> > having the limits implemented once correctly within the API would save
>> > many developers a lot of time to also have to build in code to track
>> > usage, and many will probably not implement it correctly.
>>
>> We do limit standard usage of the API, including direct messages per
>> day and status updates per day. However, you were on the whitelist,
>> and we lift those limits for whitelisted users.

Well, the problem is NOT the volume of the DMs, so the limits seem arbitrary and silly for legitimate uses. Its in MY interest to limit silly abuses of my service as it impacts other users and that's why I'll be implementing a blacklist within my system so that I can see when people search for things like "the". This probably needs to be a mostly manual process since some three letter acronyms may be useful to search for.

But, I would hope that you would see that there is a benefit to having per user limits as well, even if they are only soft limits to raise warnings that can be investigated manually. I would also allow granularity in the whitelist so that something like TweetTrak can send unlimited DMs per day, but be limited to 100 per user (soft cap), 200 per user hard cap. Then, the policy should be to investigate (perhaps on the developer's request when the hard cap is hit, with automated warnings to the developer when they exceed the soft cap) each service on a case by case basis and raise the limits if they make sense, or completely eliminate them.

>>
>> > One question though to help me understand the need for this... is this
>> a
>> > cost to send text messages issue? an infrastructure issue in dealing
>> with
>> > the extra traffic? or is this more of an issue of being friendly to
>> users?
>> > 200 DMs seems like a relatively arbitrary limit especially if someone
>> is
>> > tracking something like "san diego wildfire". Normally its activity is
>> > quite low, but just when a fire breaks out is exactly when they would
>> want
>> > to see all those messages, and that is why they will hit limits.
>>
>> It's not about cost, it's about abuse issues like the above.
>>
>> > It appears as if you have already made this change, and i have already
>> hit
>> > the limit. Sadly, i am simply stopping my application and posting
>> about
>> > Twitter's decision to limit the functionality of my application. I
>> would
>> > have appreciated a warning about this rather then just shutting off
>> the
>> > service.
>>
>> We did give you a warning, and we took you off the whitelist. This
>> doesn't prevent your application from functioning, it simply
>> re-imposes the limits most API clients are subject to. You are, of
>> course, free to blog about your experience, but I think any reasonable
>> reader would understand that we (Twitter, Inc.) have responsibilities
>> to other entities on the Internet not to allow abusive uses of our
>> service. Our behavior is in keeping with that of most any API
>> provider, none of whom allow abusive and unlimited uses of their
>> service, even if those abuses were unintentional, as yours were.

I believe you are mistaken on this, unless You have a funny definition of a warning or some email was not delivered. I checked all my email, including my spam filtered email, and the previous contact from a human at twitter was on 9/26 after I spent the nearly all of September trying to get TweetTrak onto the whitelist in the first place.

A warning to me goes something like, "You are doing foo, which causes problems for us. If you don't change your system within 24 hours to no longer behave that way, we will be forced to disable your account". You see, a warning implies that there is an impending consequence that may be prevented. Had that been the email I received (where foo="sending a large number of messages to a user (*USERNAME OMITTED*) that is resulting in bounced email messages") then I would have replied explaining that the problem had been remedied. For it truly isn't an issue unless someone again tracks some very common terms. Once there's a re-occurance, then perhaps it becomes appropriate to suggest (or force) the developer to change their application to not allow that.

You see, that is how you respect and work with your developers, by openly and precisely disclosing information, not exaggerating the issue, and also allowing them to repair issues before apply repercussions.

Also, I'll take your permission to blog about this as permission to include this email response. I believe in transparency in most things and don't believe there to be any sensitive information contained within (Of course I will remove email addresses and user account information). You may be right that most readers will see that Twitter has responsibilities to manage how they use resources (emails to ISPs), however I think that those same readers will agree that the burden of handling bounced mail should fall on Twitter and not force developers to impose arbitrary limits on their users. Perhaps I'm wrong, but users will be able to decide for themselves.

>>
>> Please let me know when you've implemented rate limits on your side
>> and I'll re-enable your whitelisting.

It won't be for a while, however... I still have TweetTrak running.. and despite the search API being unreliable, it is still running along sending more then 200 DMs per day. So, there is either a problem in your limiting code or perhaps the change to remove me from the whitelist has not yet taken place.

>>
>> > On Mon, November 3, 2008 7:02 pm, Alex Payne wrote:
>> >> Hi,
>> >>
>> >> We've noticed a handful of users getting thousands of direct messages
>> >> from TweetTrak. Until you throttle this back, we've been forced to
>> >> take the TweetTrak account off our API Whitelist. Some reasonable
>> >> limit, like 200 direct messages per user per day, would be great.
>> Let
>> >> me know if I can help. Thanks!
>> >>
>> >> --
>> >> Alex Payne - API Lead, Twitter, Inc.
>> >> http://twitter.com/al3x
>> >>
>> >
>> >
>> > --
>> >
>> >
>> >
>>
>>
>>
>> --
>> Alex Payne - API Lead, Twitter, Inc.
>> http://twitter.com/al3x
>>
>

--