Rate limiting

[gouxim]

Hey,

In order to have a scalable API, we are soon going to implement rate limiting on the API. It means that you will be allowed to make only a certain amount of requests per hour (the value is yet to be defined, but it shoud be around 200-300 requests per hour). In case you reach the limit, you will receive à 400 Bad Request error, and you'll have to wait the begining of the next hour to make requests again.

In the future we will handle special demands so that a popular application (eg. sites like tm-ladder.com) can have a higer rate limit.

Rate limiting is a fairly standard features on public web services APIs. You can check the twitter page for tips about how to avoid beeing rate limited: http://dev.twitter.com/pages/rate-limiting#tips

[fastforza]

Interesting, nice move. You seem to like specifications set by social networks.

[gouxim]

It's not so much that I like them, it's more than those are good practices when it comes to web services. And they've been set by social network, but also by companies like google or amazon (with aws).

[destroflyer]

Hm... maybe it would be better to cache some of the requests instead of "forbidding" or deny them after a several amount?

[Knutselmaaster]

Like with the "old" datafetcher, you can always cache locally i suppose.

[gouxim]

Hm... maybe it would be better to cache some of the requests instead of "forbidding" or deny them after a several amount?

Caching is already done on the server, and as Knutselmaaster pointed out it's a good pratice to cache locally in your application.

Rate limiting however allows us to have a more scalable infrastucture because we can more easilly do server dimensioning.

[w1lla]

The formula in the old tmf datafetcher was this±

Code: Select all

$cachetime = 86400;
		$midnight = floor(time()/86400)*86400-3600;
		$secondssincemidnight = time()-$midnight;
		$nextupdate = floor($secondssincemidnight / $cachetime)*$cachetime + $midnight;

So its useful to use this for the new providers.

Also in a old version fo tmndatafetcher there was no cache holder so its better to use the code that i justed posted.

[gouxim]

Rate limiting is now effective on the API. Whenever you make a request on the API, there's a X-Rate-Limit header with an x-www-form-urlencoded string (just like an URI query string, you can decoded it with the http://www.php.net/manual/en/function.parse-str.php php function).

Here's how it looks like:

Code: Select all

X-Rate-Limit: current=6&limit=2000&expires_in=3122

When the rate limit is reached, you get a "400 Bad Request" error, and the Retry-After header is included in the response:

Code: Select all

Retry-After: 2888

In the next update of the SDK I'll provide methods to easilly access those values.