This morning over coffee, Leo (a.k.a. thinkberg) and I discussed this post on the TwentyPeople blog on their problems with the Twitter API.
The story is that they started fruji, some kind of Twitter analytics project, as a side project, but it has since become so successful that they’re running into issues with Twitter’s API limits. As you probably know, Twitter only allows a certain number of requests to their API per hour. For Fruji, they have to get the list of followers for a user from Twitter. The biggest problem is that request often time out after 5 seconds, but still cost you one API call. They’ve build all kinds of workarounds, like backing off (requesting 50, 25, 10 followers, and so on, until they get a result, and then ramping up the queries, until the next timeout), but it is still a huge nuisance. For users which have hundreds of thousands of followers, getting all followers for the analysis takes a long time, even days or weeks. The problem is that Twitter is not doing anything about this issue.
This reminds me a lot of when we were using the search API to look for retweets for our twimpact analysis. That was back before the streaming API existed. We were periodically searching for the latest tweets containing “RT”, and then paging through the results. Every now and then we would hit a server which was apparently down and get a time-out, or even a response which said there were no results. Whenever that happened, we would also start to perform all kinds of workarounds, back-offs, and so on. But this often took so long that on the next round, we’d have to step through even more pages, resulting in potentially more errors, and so on.
Obviously, Twitter is not really doing a great service to all those companies and startups who try to build services around Twitter, and they probably also don’t have much incentive, because they’re making no money of the API access. They’ve started to sell their streams through gnip, but both Leo and I agree that the offerings are too big and just insanely expensive. They don’t seem to have the current numbers on the website, but half the Twitter stream costs a few hundred thousand dollars per year. Even Google eventually decided that this is just not worth it.
What they should have done is to provide a way to monetize the standard API with moderate rates. How about you get 100 API requests per hour for free, and the next 1000 for 5$ a month? That would be some model which gave startups a model to work with.
We suspect that there are two reasons why Twitter didn’t followed that path: They’re afraid that people will just steal their data to clone another social network, and they couldn’t sell that model to their investors. Investors probably love monetizing through ads. Google’s build their whole company on ads, so let’s do ads!
But I think they’re wrong. As Leo notices, once you’re in an actual business relationship with someone (and not simply someone clicking the “Yeah, I’ve read the ToS” button), you should be in a better position to sue them if they really use the data in some way which you don’t like.
In any case, I think the case for Twitter’s lost. App.net anyone?
Posted by Mikio L. Braun at 2012-11-06 13:10:00 +0000
blog comments powered by Disqus