Loading Profile...
Hitting Rate Limit on searches through the API
From the API FAQ in the TOS Section:
4. We do not rate limit the search API under ordinary circumstances, however we have put measures in place to limit the abuse of our API. If you find yourself encountering these limits, please contact us and describe your app's requirements.
Sadly, my application (@TweetTrak) relies heavily on Twitter's search engine, and I would like to work with the search team to determine what type of load I can throw at it.
A brief description of the application... it takes DM requests from users and executes searches at regular intervals on their behalf. These searches include the since_id parameter of the last match i found, and runs at a maximum rate of once a minute, and slows down after 15 minutes of no matches to every 2 minutes, every 3 at an hour... and continues on until a couple weeks pass with no matches, at which point it searches once every 6 hours.
The goal of the delay is so that popular terms will be searched often enough to have conversations via my tracking functionality, but be nice enough for rare terms that are 'dormant'. Currently there are over 500 terms I'm tracking for people, and i don't yet have statistics for the number of searches executed per minute.
However, i expect the service to grow and would hope that the search team can Whitelist my API as well as provide some guidelines for how much activity they would like me to limit things too. I would like to be able to support many thousands of users and searches.
Hopefully your team agrees with TweetTrak's users that this is an invaluable service and can provide the level of access that is required for this. I am happy to adjust the frequency and rate of decay curves to satisfy Twitter's concerns and hope to maintain a short window of 60 second interval to allow for quick replies.
4. We do not rate limit the search API under ordinary circumstances, however we have put measures in place to limit the abuse of our API. If you find yourself encountering these limits, please contact us and describe your app's requirements.
Sadly, my application (@TweetTrak) relies heavily on Twitter's search engine, and I would like to work with the search team to determine what type of load I can throw at it.
A brief description of the application... it takes DM requests from users and executes searches at regular intervals on their behalf. These searches include the since_id parameter of the last match i found, and runs at a maximum rate of once a minute, and slows down after 15 minutes of no matches to every 2 minutes, every 3 at an hour... and continues on until a couple weeks pass with no matches, at which point it searches once every 6 hours.
The goal of the delay is so that popular terms will be searched often enough to have conversations via my tracking functionality, but be nice enough for rare terms that are 'dormant'. Currently there are over 500 terms I'm tracking for people, and i don't yet have statistics for the number of searches executed per minute.
However, i expect the service to grow and would hope that the search team can Whitelist my API as well as provide some guidelines for how much activity they would like me to limit things too. I would like to be able to support many thousands of users and searches.
Hopefully your team agrees with TweetTrak's users that this is an invaluable service and can provide the level of access that is required for this. I am happy to adjust the frequency and rate of decay curves to satisfy Twitter's concerns and hope to maintain a short window of 60 second interval to allow for quick replies.
I’m undecided
5
people have this problem
I have this problem, too!
Tell me when someone solves it.
The more people who report this problem, the more it gets noticed.
The more people who report this problem, the more it gets noticed.
-
Inappropriate?RyanK:
I like your since_id decay curve method.
Any User tracking @User to check for unexpected @replies from people not followed will certainly appreciate the initial once-a-minute search.
However, I currently use TweetTrak to track a couple of keywords that turn up maybe 10 - 20 hits a week, so there's little point in ever checking more often than hourly.
So you might allow users to define search priorities. Thus urgent (the default value) would begin searching once a minute at the top of the decay curve; but low might begin at the bottom of the decay curve, and search only 4 times a day (or even less).
I've no idea how you'd include urgent, low etc. in the command line, however.
In the long term I'm a little worried about TweetTrak's scaleability: I suspect that if TweetTrak ever reached the popularity of Twitter's original track facility, you'd run into the same sort of load limits that induced Twitter to turn off their track in the first place.
As the number of TweetTrak'd terms increases, I imagine you'll have to concentrate more and more on avoiding redundant searches (e.g. @replies from followed users) and stupid searches (for and etc.).
I’m a TweetTrak fan.
1 Comment
Add a comment
This solves the problem
-
The items that get 10-20 hits a week will probably scale out to about the check every 5-10 minute range, which feels about right to me for responsiveness. Obviously there's a tradeoff that needs to be made to not pummel the search API and I hope to strike a balance that Twitter and Users can live with.
Priorities are an interesting idea, and would definitely be very useful if i have to limit the number of searches i can execute for a user or the number of messages i can send. If something has to give, it can be the low priority items.
TweetTrak is actually scaling quite well. It takes very little CPU time to manage all of these requests. It actually spends most of its time waiting for a search to be due to execute, or waiting on the result of a search or sending some DMs to users.
The real scalability issues will be if Twitter's APIs will allow me to execute a large number of searches (responsibly, of course) and to send a large number of DMs (again, within limits and only that the User has requested). Only time will tell if twitter will welcome the feature and be able to handle the load. I'm not too optimistic with the way they have handled whitelisting so far (taking nearly a month to get on it, then getting bumped off it with no warning and the expectation i rewrite a lot of code to get back on it, and now hitting a search limit when their isn't a publicly stated limit)
Let's hope TweetTrak has a future.
