On clients and APIs

[darael]

Dreamwidth's APIs are poorly documented (people basically have to work off docs for old versions of LJ's APIs). They're also missing key features, like comment handling for more than backups.

I've been told there have been "some internal conversations about deprecating the XML-RPC API -- keeping it for backwards compatability, but moving to a much more modern second-gen API", but that nobody has had both the time and the inclination to work on designing such a thing.

Well, this is me, volunteering. To that end, I'm looking for input on what exactly such a new API needs to provide, and whether there's a preferred underlying technology to build on (exempli gratia, stick with XML-RPC? Change to SOAP? Use JSON? RESTful or not? et cetera). What I'm getting at here is that I'm entirely happy to take point, as it were, and to make decisions (especially where there's little or no consensus and someone has to make the call), draw up specs, write docs, and so forth, but the result is highly unlikely to be a really useful API unless I get input from more sources than my own experience and looks at the code.

At this stage, therefore, I want everything you, the reader, have to say on the subject. Use cases especially.

Go.

Comments

[denise]

I am so exceptionally glad that you are interested in taking up the challenge, you have no idea. :D (And I am dead serious about the socks.) (For those who didn't see the start of this, it stemmed from this GH issue.)

The conversations we've had have mostly been in-person, so we don't really have anything to point anyone to. I think the best thing is to get the conversation going from the ground up. I do remember the consensus was RESTful and JSON, but I am not the technical person here! (mark and fu will likely want to weigh in, but Mark's swamped at his day job and Fu is on vacation right now.) There might be a better option. I honestly don't know.

The rest of this comment is going to be a complete and total braindump in absolutely no particular order. /me cracks knuckles

The XML/RPC API that we inherited from LJ is really divided into four conceptual chunks:

* Functions to update your own journal.
* Functions to manage your "friends list". (Which we don't have, per se -- when we were splitting "friend" into "I want to read you" (subscription) vs "I want you to be able to read my locked stuff" (access), we ran into a load of questions with the API as to whether it should affect access or subscription, and we never decided on a good answer -- we wanted to keep backwards compatability for LJ clients, but we couldn't tell enough about what client authors were using that API call for to be able to decide if it should be access or subscription. Any new API will be able to make the distinction from the beginning, so it will be a lot more relevant.)
* Functions to make it easy to download your data for offline backup
* Functions necessary to handle logging into the site and authenticating the person who wants to do stuff.

Stuff the existing API does as part of logging in/authenticating/etc, so needs to be present in some form:

getchallenge — Generate a server challenge string for authentication.
login — validate user's password and get base information needed for client to function
sessionexpire — Expires session cookies.
sessiongenerate — Generate a session cookie.

Core functions of the site that can be done through the existing API, and a new API should also be able to do (name of the existing API call is in parens):

* Post an entry (postevent)
* Edit an entry (editevent)

"Friend"-related stuff (see above for caveats):

* Check for updates on the reading list (checkfriends)
* Manage the user's circle: subscribing to someone, giving someone access, unsubbing from someone, removing someone's access (editfriends)
* Manage the user's access filters (editfriendgroups)
* Get a list of people who are on the account's subscriptions/access list (getfriends) -- I am fairly sure, if I don't misremember, that as part of the friend=>subscription/access split, we just threw up our hands and had this API function return blank all the time, because we couldn't decide if it should return "people this account subscribes to" or "people this account gives access to" for better backwards compatability.
* Get a list of what other accounts have listed the account you're logged in as as a friend (friendof) -- if I remember correctly (and I'm pretty sure I do), this was added to the API on LJ to enable people to make silly little toys, since people were scraping profiles in order to get that data. (Stuff like "six degrees of separation" tools, etc.) Like 'getfriends', I'm pretty sure we just threw up our hands and have this return blank now, because we couldn't decide if it should run off access or subscription.

Downloading/backing up data:

* getdaycounts — This mode retrieves the number of journal entries per day.
* getevents — Download parts of the user's journal. See also syncitems mode.
* getfriendgroups — Retrieves a list of the user's defined groups of friends.
* getusertags — Retrieves a list of the user's defined tags.
* syncitems — Returns a list of all the items that have been created or updated for a user.

And then there's the one lone API call that doesn't really fit into any of the above, 'consolecommand'. It lets people, well, run admin console commands. This is really less of an issue for APIv2, since we've been moving away from the console as something non-admin type people should ever have to touch.

I think the biggest thing to keep in mind, conceptually speaking, is that the existing API serves two functions: "we added this to the APIs so people can access data about accounts to make useful toys without screen-scraping" (aka, data that is public and does not require authentication to be able to see, but the API is there to lighten the load on the servers from third-party tools) and "we added this to the APIs because it's something people want to do to interact with the site and their account, so it requires authentication".

Examples of the former: "look up an interest and see who lists that interest", "who does this account subscribe to", "when was the last time this account was updated". Examples of the latter: "post an entry", "read my reading list" (logged-in, so you can see locked entries you have access to), "reply to this comment", "edit an entry", "make changes to my account preferences". Sort of the distinction between "show me all the userpics Account X has uploaded" and "allow me to upload, keyword, and delete my userpics", so to speak.

For an APIv2, I would mostly want to concentrate on the "stuff that requires authentication to allow a user to interact with their account" at first, especially since we'd want to keep the old XML-RPC API around (for backwards compatability with existing "LiveJournal" clients) for at least a little while and definitely until we had a wide range of clients (for desktop and for mobile, and for multiple computing environments).

So, for a "minimum viable prototype" of an APIv2 (we need a snazzier name for that) I'd say I should be able to, through the API (and thus through any downloaded client that uses the API):

* Write an entry and post it to my journal or to a community I have posting access to
* Edit an entry I've already posted (including the metadata such as tags and icons) (which of course requires a way to a, get the text of past entries, b, get lists of tags I've previously used, c, create new tags if necessary, d, get a list of icons I've uploaded plus their keywords, etc, etc -- this is kind of a rabbit hole of "other dependencies" here)
* Read my reading page (with authentication so I can read locked entries I'm authorized to see)
* Read entries other people have posted, including comments
* Post comments, and edit comments I've posted
* Download my whole journal (entries and comments) for offline backup
* Upload icons, delete icons, or edit icons' keywords/descriptions/comments

Things I'd look for in the next iteration:

* Subscribe to people, grant access to people, unsubscribe from people, and revoke access from people
* Manage my subscription filters and my access filters
* Read my reading page with particular subscription filters
* Ban and unban people from my journal
* Change my privacy settings (or, really, change any of my account settings)
* Manage my communities: add members, remove members, handle community moderation tasks (entries in a community's moderation queue, I mean), change community settings

The "look up public data about any account" part of the API -- those bits that aren't necessary for the above, I mean -- can probably wait until after all of the above has been handled.

darael, does that braindump help focus your thinking at all? I can probably keep braindumping for a while, but I want to give you the chance to ask questions first. ;)

(Also, I don't know if you know this, but we have an irc channel -- irc.freenode.net, channels #dreamwidth for general social hangout and chat, #dreamwidth-dev for development-focused discussion -- that you might want to come join us in. It's even odds at any given moment as to whether there's anybody awake and chatting in #dreamwidth-dev, but if you want to bounce ideas off people in realtime, #dreamwidth almost always has at least a few people alive and willing to answer questions about things like how they use the site, etc.)

[andrewducker]

Also: OAuth.

At the moment, for the users who use my web-app to post to DW, I have to store their actual password. This is...not good.

[darael]

I am entirely behind this. Caching passwords is bad. A good API should enable an app to store an application-specific and account-specific auth token, and OAuth is an extant, well-defined protocol for doing just that.

[fu]

Agreed.

[denise]

Someone else is working on implementing OAuth!

[darael]

Good. It can be the preferred authentication mechanism for YAAPI (I am using that name now and the only way to stop me is to come up with a better one).

[denise]

*snerk* I am not going to fight you on it! We are Perl people, after all ;)

[pauamma]

TMTOAPITDI! (Does DW still support the flat API?)

[denise]

I think so! At least I can't remember ever killing it ;)

[andrewducker]

Yay!

[darael]

In point of fact is it viable to require OAuth for YAPI authentication, or is there a compelling reason to support other authentication mechanisms? If the latter, what else needs to be supported?

This is an open question; it is not directed specifically at denise.

Oh, and it would take a lot of persuasion to get me to spec support for plaintext username-and-password. Especially if it was going to be supported over plain HTTP (no TLS). But that should be obvious.

[fu]

Let's go with OAuth only -- it *may* be possible that we decide we want to support something else in the future, but plenty of APIs support purely OAuth with no problems.

I'd be happy to be able to move away from plaintext username and password!

[schilling_klaus]

Will this Oauth thing be mandatory for clients afte rthe API change? I only see it used in proprietary services such as Twitter and Facebook. Is there any Free Software oauth tool for the GNU/Linux command line that can be invoked from a bash script or within the GNU Emacs editor in order to upload the content of a buffer to dreamwidth after the API changes?

[denise]

OAuth isn't a proprietary service, it's an open standard. There are dozens of implementations.

[darael]

Braindump is good. Braindump is helpful.

Expect a series of smaller comment-replies, both to make following design-thought-threads easier and because I'll be writing them as they occur.

For example: "Snazzier name": APING (NG=next generation) or YAAPI/YAPI (Yet Another (Application) Programming Interface) (both from free-software time-for-another-implementation naming conventions). The latter seems particularly appropriate given the prior existence of three different APIs of varying levels of functionality.

[darael]

Regarding multiple iterations: I really need to get a list up somewhere, and keep it up-to-date, of what features would be for what iteration. Thus far I can see a "ready-to-start-implementing" (at least authentication and posting and editing entries. Call it feature-parity with the Blogger API.), a "viable prototype" (enough features to actually be useful, so that's fetching comments, basic full-dump style backup, reading page), "idealised basic" (add posting and editing comments, treating them as first-class objects, icon management), "solid" (account settings, moderation tasks, &c), "polished" (everything one can do through a browser, even if it takes multiple calls under-the-hood), and what one might call "wishlist" (things that exist purely to lighten server load and allow people to make little toys).

I may be subdividing a little too finely, here, but I think the principle's sound.

I should get on that.

[denise]

I like iterations! Iterations help to prevent total burnout :)

[alexbayleaf]

I would be delighted to start working with the API as soon as it has a simple authentication/posting mechanism.

I'm already starting to brainstorm apps I'd build with just that, starting with take-a-pic-on-your-phone-and-post-it-really-easily (speaking of things that need a snappier name).

[jewelfox]

I personally wanted to build "read your friends list on your phone."

[alexbayleaf]

I occasionally think this, then wonder why it can't be more easily done by making existing site styles more responsive/mobile friendly?

afuna and I fiddled a little bit with some CSS stuff to make site scheme readable on a phone a while back. Nothing ever came of it, but if it were workable, I would actually love to be able to read DW -- just plain old web based DW -- on my phone.

[denise]

It's in progress as part of the Foundation/SCSS conversion, but there are a lot of pages to convert.

[jewelfox]

I occasionally think this, then wonder why it can't be more easily done by making existing site styles more responsive/mobile friendly?

More easily done by whom? >_>

Plus a native Windows / Windows Phone app would have live tile support, and the ability to easily share non-access-locked content.

[fu]

I like the idea of chopping things into smaller chunks.

Sooooo a suggestion for where to put up the list: let's have an issue on github for each separate feature, grouped under a milestone for each iteration -- that is vs having one issue called "implement API" with everything listed as a comment there ;)

[darael]

This is a good plan, but I don't want to clutter up the dreamwidth/dw-free github section with bugs to the effect of "this hasn't been specced yet". As such, I'm going to publish a new repository (Darael/dw-yaapi) on which the milestones and the feature-spec bugs can go, as well as the specifications to close said bugs. Then, when a milestone's complete, it becomes viable to create a corresponding milestone on dw-free and appropriate "this spec isn't implemented" bugs to go with it. That link's dead at the time of writing, but expected to go live in, ooh, five minutes? I probably don't have time tonight to actually enter the first milestone's bugs, though.

[fu]

Whoo, okay!

I don't think anybody would be bothered by having the discussion in dreamwidth/dw-free, but that also looks like a good place to start :) i don't want to step all over your thought process, but I'll keep an eye on the issues repo for updates.