Page 1 of 1

Rate limiting

Posted: 13 May 2011, 11:28
by gouxim
Hey,

In order to have a scalable API, we are soon going to implement rate limiting on the API. It means that you will be allowed to make only a certain amount of requests per hour (the value is yet to be defined, but it shoud be around 200-300 requests per hour). In case you reach the limit, you will receive à 400 Bad Request error, and you'll have to wait the begining of the next hour to make requests again.

In the future we will handle special demands so that a popular application (eg. sites like tm-ladder.com) can have a higer rate limit.

Rate limiting is a fairly standard features on public web services APIs. You can check the twitter page for tips about how to avoid beeing rate limited: http://dev.twitter.com/pages/rate-limiting#tips

Re: Rate limiting

Posted: 14 May 2011, 08:51
by fastforza
Interesting, nice move. 8-) You seem to like specifications set by social networks. :P

Re: Rate limiting

Posted: 14 May 2011, 12:47
by gouxim
It's not so much that I like them, it's more than those are good practices when it comes to web services. And they've been set by social network, but also by companies like google or amazon (with aws). :geek:

Re: Rate limiting

Posted: 14 May 2011, 17:56
by destroflyer
Hm... maybe it would be better to cache some of the requests instead of "forbidding" or deny them after a several amount?

Re: Rate limiting

Posted: 14 May 2011, 21:34
by Knutselmaaster
Like with the "old" datafetcher, you can always cache locally i suppose.

Re: Rate limiting

Posted: 15 May 2011, 13:40
by gouxim
destroflyer wrote:Hm... maybe it would be better to cache some of the requests instead of "forbidding" or deny them after a several amount?
Caching is already done on the server, and as Knutselmaaster pointed out it's a good pratice to cache locally in your application.

Rate limiting however allows us to have a more scalable infrastucture because we can more easilly do server dimensioning.

Re: Rate limiting

Posted: 16 May 2011, 22:59
by w1lla
The formula in the old tmf datafetcher was this±

Code: Select all

$cachetime = 86400;
		$midnight = floor(time()/86400)*86400-3600;
		$secondssincemidnight = time()-$midnight;
		$nextupdate = floor($secondssincemidnight / $cachetime)*$cachetime + $midnight;
So its useful to use this for the new providers.

Also in a old version fo tmndatafetcher there was no cache holder so its better to use the code that i justed posted.

Re: Rate limiting

Posted: 06 Jul 2011, 10:57
by gouxim
Rate limiting is now effective on the API. Whenever you make a request on the API, there's a X-Rate-Limit header with an x-www-form-urlencoded string (just like an URI query string, you can decoded it with the http://www.php.net/manual/en/function.parse-str.php php function).

Here's how it looks like:

Code: Select all

X-Rate-Limit: current=6&limit=2000&expires_in=3122
When the rate limit is reached, you get a "400 Bad Request" error, and the Retry-After header is included in the response:

Code: Select all

Retry-After: 2888
In the next update of the SDK I'll provide methods to easilly access those values.