RyanK's dashboard

That's great.. but why on every page is there a blob of text in the side bar that says:

If you have questions or experience problems, please report them to the BarCamp support forum. For issues related to this wiki, please see the PBWiki Support Forum.

Both "Support forums" are links to Get Satisfaction... The first being getsatisfaction.com/barcamp while the later being getsatisfaction.com/pbwiki

If this isn't their official support area, they should remove that link. Also, the point of Get Satisfaction is for open support. I'm sure there isn't anything special about my account that this should be happening just for me.

Finally, the Help area appears to be more for owners of the wiki. To be honest, I don't want to bother much with this and if I have to jump through hoops to get their attention, I'll just delete my account and forget trying to use their services.

Thanks for the reply though... I know you're trying to be helpful. – RyanK, on July 01, 2009 20:08

Hmmm, no... but I suspect that they are working on something since it has failed in a new and unexpected way.

I have it running very cautiously so that I don't anger anyone at Twitter. When something unexpected happens it shuts down so that I can look at it. Thanks for the heads up!

I've restarted it and allowed the last few hours of Tweets to go out. I really need to follow up with the Twitter people to find out if they are doing anything about this as this has been going on for a couple weeks now! =/

Thanks again for reporting this and for your patience!

The items that get 10-20 hits a week will probably scale out to about the check every 5-10 minute range, which feels about right to me for responsiveness. Obviously there's a tradeoff that needs to be made to not pummel the search API and I hope to strike a balance that Twitter and Users can live with.

Priorities are an interesting idea, and would definitely be very useful if i have to limit the number of searches i can execute for a user or the number of messages i can send. If something has to give, it can be the low priority items.

TweetTrak is actually scaling quite well. It takes very little CPU time to manage all of these requests. It actually spends most of its time waiting for a search to be due to execute, or waiting on the result of a search or sending some DMs to users.

The real scalability issues will be if Twitter's APIs will allow me to execute a large number of searches (responsibly, of course) and to send a large number of DMs (again, within limits and only that the User has requested). Only time will tell if twitter will welcome the feature and be able to handle the load. I'm not too optimistic with the way they have handled whitelisting so far (taking nearly a month to get on it, then getting bumped off it with no warning and the expectation i rewrite a lot of code to get back on it, and now hitting a search limit when their isn't a publicly stated limit)

Let's hope TweetTrak has a future. – RyanK, on November 12, 2008 02:45

I have reset things as it is possible that your search got queued up twice as it appears as if twitter's search servers are a bit overloaded.

The changes i am working on would prevent this from happening, however I have to redesign some of those things if i need to do some rate limiting from my side.

I should probably also point out that currently if a tweet matches more then one term that you are tracking, you will receive one DM for each search. For example, if i was tracking "Obama" and "McCain" then i would receive two messages about the same tweet if it contains both Obama and McCain. (Workaround for that is tracking "Obama OR McCain")

Please mark this as solved if the issue has been fixed. If the problem persists, I will look into it further.

Thanks!

I'd agree if this was still an issue, but it was resolved by Sunday afternoon. That means there was a full 24 hours of business as usual and they are still making the decision to block things.

Also, the ony impact, according to them, is that they have to explain why they sent many emails to a bouncing address. The worst case would be if that ISP blocked twitter from delivering emails and maybe reported them to a blacklist.

Fail whales were not mentioned at all, and as I said in the email, i would understand if there was an infrastructure impact and they needed to take drastic action.

So, I stand by my claim that it is a knee-jerk over-reaction that's completely unnecessary. But who am I to judge! – RyanK, on November 04, 2008 15:02

Thank you for taking the time to report this as an issue to Twitter.

I'm using a topic on TweetTrak's Get Satisfaction (http://tinyurl.com/63nh5d) to report on the status of this issue.

Got an email from two people at twitter, the API Lead Developer and the Head of Operations below. I, of course, think I make my case pretty well, but ultimately leave the decision to the community. Perhaps i am biased and unable to see their justification.

So, get a cup of coffee, sit back, relax, and make yourself comfortable... its a bit long (i can get longwinded), but I think there are many good points that were worth making. If you would like to comment, please add it as a response to this comment, rather then as a response to the original problem.

With any further ado... my email response, including the email exchange back to the beginning:

Please see my comments to both Jeremy and Adam's email below...

On Tue, November 4, 2008 12:03 am, Jeremy LaTrasse wrote:
> Ryan,
>
> I'm very sorry we had to disable the white-listing of your app. Our email
> bounce rate was skyrocketing (we usually don't bounce more than 1% of
> outgoing mail each day, the *USERNAME OMITTED* incident sent that number to almost
> 10%), and we are diligently trying to keep our bounce rates down so that
> ISP's don't block legitimate mail to users in their domain. I am really
> glad
> that you are going to take steps on your end to limit potential runaways
> like that one.

Then it seems to me that the problem should be addressed on the email notification side of things, since that is where the problem lies. That way, the source of the issue wouldn't matter. Consider that what would happen if I, like *USERNAME OMITTED*, had an email address configured that would bounce mail messages to it, and I was featured in an article on Slashdot or some Mainstream Media outlet, and 20,000 people decide to follow me. Since I have New Follower Emails turned on, I would get flooded with 20,000 emails, all of which would bounce back and cause the same exact problem.

So, why not implement what most people do... a filter on your 'no-reply@tweettrak.com' address to process that. In actuality, I noticed that the no-reply@tweettrak had changed last week to be forumlaic to allow you to parse the bounce emails so that you can disable them. (My DM messages are coming from twitter-dm-*EMAILADDRESS OMITTED*@postmaster.twitter.com now )

It would be one thing if the Request to send the DM was what the cause of problem (ie, Twitter's infrastructure couldn't handle an uptick in the API usage). And don't confuse the fact that the originating action was the DM from @TweetTrak with the true source of your concern... sending emails to bad email addresses.

>
> It was quite a scare for the operations crew, on the eve of such an
> historic
> U.S. election, to realize that the applications defenses to prevent such
> runaways were not behaving as planned. We pushed the panic button and had
> Alex disable your whitelist access.

So, it *was* a knee jerk reaction. And it was of the worst kind... a delayed over-reaction after the problem was no longer an issue as I had already noticed it before twitter had, and stopped the offending user's request to track very common terms.

If you doubt the veracity of my claims, check TweetTrak's public timeline, and compare it to its DM inbox.

This problem isn't unique to twitter. If we ask ourselves, "What would Google do" (bad pun intended, i'm sorry) when someone sets up an alert on common words and has a notification method that bounces. I honestly don't know what they do, but i'm pretty sure they would not disable their entire alert service. (and yes, i realize that TweetTrak is an API application while google alerts is an internal function at google, but I doubt average users see the distinction).

>
> We are working to improving ways that we can approach the issue that
> forced
> the de-listing today in the near term. I can assure you that Alex and I,
> and
> everyone here at Twitter will be discussing those options, and keep in
> constant communications about them with all developers and users too.
> We'll
> need some time to sort out how best to approach the various aspects to
> ensure that we don't repeat the *USERNAME OMITTED* incident.

So, in the meantime, the mere potential for a problem that took nearly 2 months to surface and is no longer causing problems is removing functionality that some users desperately miss and see as vital to using Twitter. (see http://tinyurl.com/63nh5d and http://tinyurl.com/5rz3yd for some user comments)

I am a software architect and design systems like this for a living. If I were to design things, the solution is simple. Require verification of ownership of the email address if twitter doesn't already do that. That is, twitter will disable all notifications to an unverified email. Then, send all mails from a user identifiable address so that all the bounces can be processed automatically to identify the user and the initiator of the email. That's already done apparently. Finally, process the bounce messages and once a threshold is met (20 bounces in a day perhaps?) that email address losses its verified status and is marked as bouncing. The user would then have to re-verify the email address or update the address to a correct one. Simple, and effective, and there's no burden on API developers to ensure that they aren't sending too many messages.

>
> I can tell you that neither engineers, nor operations staff here want to
> limit developers use of our services, the API's are near and dear to all
> of
> our hearts in all seriousness, and we want the developer community to
> continue to flourish.

Then I would expect that Twitter would treat the developers with respect and actually work with them rather then throw them in a penalty box with exagerated claims (see below). Indeed, the lifeblood of twitter *IS* its API as it is what allows twitter to function as a messaging framework and allow for innovate uses of Twitter.

>
> Please feel free to contact me via email if necessary, I'm sure that we'll
> be in touch with you very soon about getting you the type of access your
> application requires.
>
> Jeremy
>
> On Mon, Nov 3, 2008 at 9:21 PM, Alex Payne <*EMAIL ADDRESS OMITED*> wrote:
>
>> > To my knowledge, there has only been one user (*USERNAME OMITTED*) over this past
>> > weekend that tracked several very common terms, so they were sent 20
>> DMs
>> > per term per minute for about 14 hours before i noticed the load.
>>
>> *USERNAME OMITTED* was the account that our operations team brought to my
>> attention.

So, it was just one isolated account that had misconfigured their own use of a legitimate service that caused a large number of bounced emails that caused concern? Then why present the problem as "a handful of users getting thousands of direct messages from TweetTrak"? It wasn't "a handful" and the problem wasn't that they received direct messages, but rather that they chose to have those messages be emailed, and in turn, those emails were directed to a bouncing email address.

>>
>> > I am planning on incorporating statistics to track which terms are
>> getting
>> > sent at what rate, and i would be able to add in some protection to
>> > prevent a large number of DMs to be received by a particular user.
>> > However, I won't be able to add that until the weekend at the
>> earliest,
>> > and i don't believe that the problem is an issue going forward.
>>
>> Okay. Once you have a rate limit in place, we'll re-enable your
>> whitelisted status.

As i said, i was planning on adding those features, i didn't have an immediate plan for it, nor would i be able to get it done until at least after this weekend. There was much more important features to add that actually provide a benefit to users, such as allowing them to ignore updates from certain users or only send a single notifaction when there are two tracked terms that match a tweet. The latter feature is what will provide the statistics that will allow me to provide the data to determine when someone is receiving too many messages.

However, I would argue that limiting the number of messages is the WRONG approach, since those limits are arbitrary and like some of my users have said... they appreciated being able to receive a spike in notifications for some thing like #sandiegofire.. if they were limited to 200DMs a day... when they most need the service, they would be shut out.

>>
>> > It seems as if you are punishing everyone for the actions of one (or a
>> > few) users, and I implore you to monitor the usage from TweetTrak
>> rather
>> > then disable its usefulness
>>
>> It's not fun for me to temporarily remove you from the whitelist; my
>> job is supporting developers, not making their lives harder. That
>> said, we're responsible for the consequences of API apps like yours.
>> Every one of those messages to brain02 resulted in a bounce message
>> from his email provider, as his account settings were to deliver
>> direct messages to email. We now have to smooth things over with said
>> email provider. That's why we had to take you off the whitelist in a
>> hurry.

Again, I believe the proper place to handle that is in processing your bounce messages and disable (temporarily or permanantly) emails that are bouncing. And the "smoothing things over" conversation is a simple one: "One of your users, configured an application that uses our API to receive a large number of notifications and he had his email set to send emails to your domain. We are apologize for the large number of messages and are working on a system to detect these bounces and disable mail to them, and expect to have that implemented very soon". Nearly every ISP is willing to work with people on a case-by-case basis, unless you're a small player (Twitter isn't, IMO) and you're dealing with AOL (I hope for your sakes it wasn't AOL).

>>
>> > Also, I would think that it would make sense for these limits to be
>> > enforced from within the API. I understand that requires more
>> processing
>> > within Twitter, but the time and effort spent on that would reduce the
>> > amount of time spent chasing down abuses of the API. I would think
>> that
>> > having the limits implemented once correctly within the API would save
>> > many developers a lot of time to also have to build in code to track
>> > usage, and many will probably not implement it correctly.
>>
>> We do limit standard usage of the API, including direct messages per
>> day and status updates per day. However, you were on the whitelist,
>> and we lift those limits for whitelisted users.

Well, the problem is NOT the volume of the DMs, so the limits seem arbitrary and silly for legitimate uses. Its in MY interest to limit silly abuses of my service as it impacts other users and that's why I'll be implementing a blacklist within my system so that I can see when people search for things like "the". This probably needs to be a mostly manual process since some three letter acronyms may be useful to search for.

But, I would hope that you would see that there is a benefit to having per user limits as well, even if they are only soft limits to raise warnings that can be investigated manually. I would also allow granularity in the whitelist so that something like TweetTrak can send unlimited DMs per day, but be limited to 100 per user (soft cap), 200 per user hard cap. Then, the policy should be to investigate (perhaps on the developer's request when the hard cap is hit, with automated warnings to the developer when they exceed the soft cap) each service on a case by case basis and raise the limits if they make sense, or completely eliminate them.

>>
>> > One question though to help me understand the need for this... is this
>> a
>> > cost to send text messages issue? an infrastructure issue in dealing
>> with
>> > the extra traffic? or is this more of an issue of being friendly to
>> users?
>> > 200 DMs seems like a relatively arbitrary limit especially if someone
>> is
>> > tracking something like "san diego wildfire". Normally its activity is
>> > quite low, but just when a fire breaks out is exactly when they would
>> want
>> > to see all those messages, and that is why they will hit limits.
>>
>> It's not about cost, it's about abuse issues like the above.
>>
>> > It appears as if you have already made this change, and i have already
>> hit
>> > the limit. Sadly, i am simply stopping my application and posting
>> about
>> > Twitter's decision to limit the functionality of my application. I
>> would
>> > have appreciated a warning about this rather then just shutting off
>> the
>> > service.
>>
>> We did give you a warning, and we took you off the whitelist. This
>> doesn't prevent your application from functioning, it simply
>> re-imposes the limits most API clients are subject to. You are, of
>> course, free to blog about your experience, but I think any reasonable
>> reader would understand that we (Twitter, Inc.) have responsibilities
>> to other entities on the Internet not to allow abusive uses of our
>> service. Our behavior is in keeping with that of most any API
>> provider, none of whom allow abusive and unlimited uses of their
>> service, even if those abuses were unintentional, as yours were.

I believe you are mistaken on this, unless You have a funny definition of a warning or some email was not delivered. I checked all my email, including my spam filtered email, and the previous contact from a human at twitter was on 9/26 after I spent the nearly all of September trying to get TweetTrak onto the whitelist in the first place.

A warning to me goes something like, "You are doing foo, which causes problems for us. If you don't change your system within 24 hours to no longer behave that way, we will be forced to disable your account". You see, a warning implies that there is an impending consequence that may be prevented. Had that been the email I received (where foo="sending a large number of messages to a user (*USERNAME OMITTED*) that is resulting in bounced email messages") then I would have replied explaining that the problem had been remedied. For it truly isn't an issue unless someone again tracks some very common terms. Once there's a re-occurance, then perhaps it becomes appropriate to suggest (or force) the developer to change their application to not allow that.

You see, that is how you respect and work with your developers, by openly and precisely disclosing information, not exaggerating the issue, and also allowing them to repair issues before apply repercussions.

Also, I'll take your permission to blog about this as permission to include this email response. I believe in transparency in most things and don't believe there to be any sensitive information contained within (Of course I will remove email addresses and user account information). You may be right that most readers will see that Twitter has responsibilities to manage how they use resources (emails to ISPs), however I think that those same readers will agree that the burden of handling bounced mail should fall on Twitter and not force developers to impose arbitrary limits on their users. Perhaps I'm wrong, but users will be able to decide for themselves.

>>
>> Please let me know when you've implemented rate limits on your side
>> and I'll re-enable your whitelisting.

It won't be for a while, however... I still have TweetTrak running.. and despite the search API being unreliable, it is still running along sending more then 200 DMs per day. So, there is either a problem in your limiting code or perhaps the change to remove me from the whitelist has not yet taken place.

>>
>> > On Mon, November 3, 2008 7:02 pm, Alex Payne wrote:
>> >> Hi,
>> >>
>> >> We've noticed a handful of users getting thousands of direct messages
>> >> from TweetTrak. Until you throttle this back, we've been forced to
>> >> take the TweetTrak account off our API Whitelist. Some reasonable
>> >> limit, like 200 direct messages per user per day, would be great.
>> Let
>> >> me know if I can help. Thanks!
>> >>
>> >> --
>> >> Alex Payne - API Lead, Twitter, Inc.
>> >> http://twitter.com/al3x
>> >>
>> >
>> >
>> > --
>> >
>> >
>> >
>>
>>
>>
>> --
>> Alex Payne - API Lead, Twitter, Inc.
>> http://twitter.com/al3x
>>
>

--

Apparently things are still going through atm.. perhaps email was just a warning and something else was causing errors (like if search was unavailable).. or perhaps was only temporary.

Regardless, I won't be able to make such a change in a timely manner, and hopefully we can manually address issues as they arise, but that is in Twitter's hands.

Guess I'll just wait and see when Twitter responds (probably sometime tomorrow) and take things from there.

FYI, TweetTrak is still running, and you may get intermittent messages from it as it will continue trying to send messages. The alternative is to stop it completely and when it starts back up to not send any of the missing messages.

Unless popular sentiment is to just stop it altogether pending a resolution of this issue... its probably the best course of action.

Yes, however.. tracking "love" is probably not a good idea. Very recently, someone tracked quite a few common words which impacted the rest of people using TweetTrak, as it was trying to forward 20 tweets per term every minute.

"Grand Haven" should do the trick!

You can always test your search term by going to http://search.twitter.com and trying out things there. TweetTrak is really just running searches for you on your behalf and sending you DMs with the new results.

Hope this helps!

ack, found another place where i am connecting to twitter where i am not trying to catch any unexpected behavior.

Sadly, most of this code was written in a weekend and hasn't been put under heavy load. There are only two methods that actually need to connect, and i had only added the error handling code to one of them.

i am in the process of rewriting everything so that it is object oriented (nerd speak so that it is more elegantly designed to reduce code duplication and make it easier to find, fix, and prevent these sorts of problems).

TweetTrak now gives me useful error messages, and if it goes down again in the near future, it will be in a unique and unexpected way.

There was about 6 hours of downtime this time around. (I like to be honest and transparent about the project). Its working now and has sent out all of the delayed notifications.

Sorry for the trouble, but Thank You for reporting it!

looking at http://search.twitter.com/operators it doesn't appear as if you can specify a language to limit your results to.

However, presuming your search terms are english words, its very likely that only english tweets will match.

If there are terms you want to exclude, you can start a word with dash (their example is "beer -root" which would give you beer related tweets that are not about root beer). This method obviously isn't perfect since if the word root is used elsewhere it would prevent you from getting a tweet you may be interested in.

Do you have an example of something you're trying to search for that doesn't work well? I'm sure we could probably come up with some suggestions.