random: (Default)
Random ([personal profile] random) wrote in [site community profile] dw_dev 2009-05-27 02:24 am (UTC)

Continuing to reply to myself, so that I don't forget stuff and I have it all in one place.

Some things in this are made fairly easy - Net::Twitter has been around for quite a while in CPAN, and Net::Twitter::OAuth is under fairly active development right now to add oauth authentication to the Net::Twitter package, so there's a bunch of tools that make this not a matter of re-inventing the wheel.

However, while until about a year ago it was possible to query the Twitter API using the form http://twitter.com/status/friends_timeline/username.xml to get a given username's unprotected friends entries (exactly what we're looking for), they disabled that feature to reduce server load during the period when Twitter was having a lot of server outages. And while one of the last posts in this thread is about a year ago, with one of the developers saying they were looking for how to bring it back, it's still not possible to do that without authentication, which makes the whole project impossible. In theory, I suppose, we could have it show whichever tweets could be read by any logged-in DW user who had logged in to oauth THEMSELVES and had an oauth token stored attached to their profile - the Twitter REST API does allow that - but there's no code for doing that in Net::Twitter, and it's kludgey at best anyways.

(Also, if we were going to use oauth to show protected entries on the logged-in user's reading page, it leaves the somewhat tricky question of how/when they get redirected to the oauth login page. Presumably the first time they go to their reading page after enabling the module, but that still feels unelegant. So all things consider, I'm not at all sure the reading page part of this is at all viable right now, which is a pity, because I really liked the idea.)

On the upside, using the Twitter API to just show the user's own tweets on their own pages is fairly simple, since user_timeline still doesn't require any authentication at all, dodging that whole can of worms. To do that, all we'd need is to store the Twitter username in a preferences table somewhere (I haven't looked at DW's database schema yet), use the user_timeline method to get the most recent tweets, and then generate the properly formatted sidebar block.

Another thing we'd probably need to take into account is that Twitter has IP-based rate limiting for queries. By default, it's limited to 100 queries per hour - but they have a process where you can ask for whitelisting up to 20,000 queries per hour. Even with that kind of a limit, though, as Dreamwidth grows, we're going to need to be aware of it, and in any case we want to reduce unnecessary load on both DW and Twitter. Some kind of caching (maybe store for 5 or 10 minutes, maybe even a bit longer) is probably essential, and it's probably good long-term practice (even if it adds a little load) to make sure that whenever we go to refresh the cached feed, we use the rate_limit_status call first to make sure that we still have queries that can be used, and extend the cache until the hour limit ends if we're out of queries, rather than querying for the results, triggering the limit, and getting an error back (the time at which the number of queries allowed resets is in the response to the rate_limit_status call).

And now I'm going to walk away from my computer. :)

Post a comment in response:

If you don't have an account you can create one now.
HTML doesn't work in the subject.
More info about formatting

If you are unable to use this captcha for any reason, please contact us by email at support@dreamwidth.org