![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
![[site community profile]](https://www.dreamwidth.org/img/comm_staff.png){kind=small}
On clients and APIs
Dreamwidth's APIs are poorly documented (people basically have to work off docs for old versions of LJ's APIs). They're also missing key features, like comment handling for more than backups.
I've been told there have been "some internal conversations about deprecating the XML-RPC API -- keeping it for backwards compatability, but moving to a much more modern second-gen API", but that nobody has had both the time and the inclination to work on designing such a thing.
Well, this is me, volunteering. To that end, I'm looking for input on what exactly such a new API needs to provide, and whether there's a preferred underlying technology to build on (exempli gratia, stick with XML-RPC? Change to SOAP? Use JSON? RESTful or not? et cetera). What I'm getting at here is that I'm entirely happy to take point, as it were, and to make decisions (especially where there's little or no consensus and someone has to make the call), draw up specs, write docs, and so forth, but the result is highly unlikely to be a really useful API unless I get input from more sources than my own experience and looks at the code.
At this stage, therefore, I want everything you, the reader, have to say on the subject. Use cases especially.
Go.
I've been told there have been "some internal conversations about deprecating the XML-RPC API -- keeping it for backwards compatability, but moving to a much more modern second-gen API", but that nobody has had both the time and the inclination to work on designing such a thing.
Well, this is me, volunteering. To that end, I'm looking for input on what exactly such a new API needs to provide, and whether there's a preferred underlying technology to build on (exempli gratia, stick with XML-RPC? Change to SOAP? Use JSON? RESTful or not? et cetera). What I'm getting at here is that I'm entirely happy to take point, as it were, and to make decisions (especially where there's little or no consensus and someone has to make the call), draw up specs, write docs, and so forth, but the result is highly unlikely to be a really useful API unless I get input from more sources than my own experience and looks at the code.
At this stage, therefore, I want everything you, the reader, have to say on the subject. Use cases especially.
Go.
no subject
Fast login switching
Something that tolerates network latency -- when I'm in the subway and I have no signal, I want to be able to hit send on an action and have the app realize that there's no connection and queue it for later, and in context with the fast login switching, if I'm working as User A and then as User B when the train comes out of the tunnel, I don't want my queued actions for User A to accidentally be sent as the current logged in user (B) instead.
Easy retrieval of related comments: given sufficient information to identify a comment, what is its direct parent, direct children, and whole thread? (Bonus points for easy pagination of the whole thread, because some RP threads get hella long.)
With any given comment, have I replied to it yet? And under which identity?
Some other things which are currently painful through the web involve entry management, like re-doing tags.
Retrieve all entries with a given tag.
Apply a given tag to an entry.
Retrieve the current tags for an entry.
Retrieve the current security settings for an entry. (Must play well with custom settings, including multiple custom security groups.)
Apply these security settings to an entry.
Retrieving the userpic for a given entry.
Retrieving the userpic for a given comment.
*pauses a bit to catch breath* I hope this is a good starting point.
no subject
I think a lot of those use cases you mention are things that someone would want to build a client to do, but are not necessarily use cases for an API in particular. Which is a very tiny and perhaps useless distinction (since obviously for a client to be able to do something, the API needs to have provisions for it) but I think is perhaps worth making at this point?
Braindump comment to follow shortly.
(no subject)
no subject
The conversations we've had have mostly been in-person, so we don't really have anything to point anyone to. I think the best thing is to get the conversation going from the ground up. I do remember the consensus was RESTful and JSON, but I am not the technical person here! (
The rest of this comment is going to be a complete and total braindump in absolutely no particular order. /me cracks knuckles
The XML/RPC API that we inherited from LJ is really divided into four conceptual chunks:
* Functions to update your own journal.
* Functions to manage your "friends list". (Which we don't have, per se -- when we were splitting "friend" into "I want to read you" (subscription) vs "I want you to be able to read my locked stuff" (access), we ran into a load of questions with the API as to whether it should affect access or subscription, and we never decided on a good answer -- we wanted to keep backwards compatability for LJ clients, but we couldn't tell enough about what client authors were using that API call for to be able to decide if it should be access or subscription. Any new API will be able to make the distinction from the beginning, so it will be a lot more relevant.)
* Functions to make it easy to download your data for offline backup
* Functions necessary to handle logging into the site and authenticating the person who wants to do stuff.
Stuff the existing API does as part of logging in/authenticating/etc, so needs to be present in some form:
getchallenge — Generate a server challenge string for authentication.
login — validate user's password and get base information needed for client to function
sessionexpire — Expires session cookies.
sessiongenerate — Generate a session cookie.
Core functions of the site that can be done through the existing API, and a new API should also be able to do (name of the existing API call is in parens):
* Post an entry (postevent)
* Edit an entry (editevent)
"Friend"-related stuff (see above for caveats):
* Check for updates on the reading list (checkfriends)
* Manage the user's circle: subscribing to someone, giving someone access, unsubbing from someone, removing someone's access (editfriends)
* Manage the user's access filters (editfriendgroups)
* Get a list of people who are on the account's subscriptions/access list (getfriends) -- I am fairly sure, if I don't misremember, that as part of the friend=>subscription/access split, we just threw up our hands and had this API function return blank all the time, because we couldn't decide if it should return "people this account subscribes to" or "people this account gives access to" for better backwards compatability.
* Get a list of what other accounts have listed the account you're logged in as as a friend (friendof) -- if I remember correctly (and I'm pretty sure I do), this was added to the API on LJ to enable people to make silly little toys, since people were scraping profiles in order to get that data. (Stuff like "six degrees of separation" tools, etc.) Like 'getfriends', I'm pretty sure we just threw up our hands and have this return blank now, because we couldn't decide if it should run off access or subscription.
Downloading/backing up data:
* getdaycounts — This mode retrieves the number of journal entries per day.
* getevents — Download parts of the user's journal. See also syncitems mode.
* getfriendgroups — Retrieves a list of the user's defined groups of friends.
* getusertags — Retrieves a list of the user's defined tags.
* syncitems — Returns a list of all the items that have been created or updated for a user.
And then there's the one lone API call that doesn't really fit into any of the above, 'consolecommand'. It lets people, well, run admin console commands. This is really less of an issue for APIv2, since we've been moving away from the console as something non-admin type people should ever have to touch.
I think the biggest thing to keep in mind, conceptually speaking, is that the existing API serves two functions: "we added this to the APIs so people can access data about accounts to make useful toys without screen-scraping" (aka, data that is public and does not require authentication to be able to see, but the API is there to lighten the load on the servers from third-party tools) and "we added this to the APIs because it's something people want to do to interact with the site and their account, so it requires authentication".
Examples of the former: "look up an interest and see who lists that interest", "who does this account subscribe to", "when was the last time this account was updated". Examples of the latter: "post an entry", "read my reading list" (logged-in, so you can see locked entries you have access to), "reply to this comment", "edit an entry", "make changes to my account preferences". Sort of the distinction between "show me all the userpics Account X has uploaded" and "allow me to upload, keyword, and delete my userpics", so to speak.
For an APIv2, I would mostly want to concentrate on the "stuff that requires authentication to allow a user to interact with their account" at first, especially since we'd want to keep the old XML-RPC API around (for backwards compatability with existing "LiveJournal" clients) for at least a little while and definitely until we had a wide range of clients (for desktop and for mobile, and for multiple computing environments).
So, for a "minimum viable prototype" of an APIv2 (we need a snazzier name for that) I'd say I should be able to, through the API (and thus through any downloaded client that uses the API):
* Write an entry and post it to my journal or to a community I have posting access to
* Edit an entry I've already posted (including the metadata such as tags and icons) (which of course requires a way to a, get the text of past entries, b, get lists of tags I've previously used, c, create new tags if necessary, d, get a list of icons I've uploaded plus their keywords, etc, etc -- this is kind of a rabbit hole of "other dependencies" here)
* Read my reading page (with authentication so I can read locked entries I'm authorized to see)
* Read entries other people have posted, including comments
* Post comments, and edit comments I've posted
* Download my whole journal (entries and comments) for offline backup
* Upload icons, delete icons, or edit icons' keywords/descriptions/comments
Things I'd look for in the next iteration:
* Subscribe to people, grant access to people, unsubscribe from people, and revoke access from people
* Manage my subscription filters and my access filters
* Read my reading page with particular subscription filters
* Ban and unban people from my journal
* Change my privacy settings (or, really, change any of my account settings)
* Manage my communities: add members, remove members, handle community moderation tasks (entries in a community's moderation queue, I mean), change community settings
The "look up public data about any account" part of the API -- those bits that aren't necessary for the above, I mean -- can probably wait until after all of the above has been handled.
(Also, I don't know if you know this, but we have an irc channel -- irc.freenode.net, channels #dreamwidth for general social hangout and chat, #dreamwidth-dev for development-focused discussion -- that you might want to come join us in. It's even odds at any given moment as to whether there's anybody awake and chatting in #dreamwidth-dev, but if you want to bounce ideas off people in realtime, #dreamwidth almost always has at least a few people alive and willing to answer questions about things like how they use the site, etc.)
no subject
At the moment, for the users who use my web-app to post to DW, I have to store their actual password. This is...not good.
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
no subject
Expect a series of smaller comment-replies, both to make following design-thought-threads easier and because I'll be writing them as they occur.
For example: "Snazzier name": APING (NG=next generation) or YAAPI/YAPI (Yet Another (Application) Programming Interface) (both from free-software time-for-another-implementation naming conventions). The latter seems particularly appropriate given the prior existence of three different APIs of varying levels of functionality.
no subject
I may be subdividing a little too finely, here, but I think the principle's sound.
I should get on that.
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
no subject
As someone who still wants to make that Dreamwidth app, I am 100 percent in favour of this and would be happy to give you feedback as needed.
no subject
Given those things, I would almost certainly develop apps/tools/something using them, and perhaps contribute to a Ruby gem to wrap it all nicely.
Ask and ye shall receive.
It shall be done.
no subject
Yes, yes, and yes.
book recommendation/offer
Re: book recommendation/offer
Accordingly, I'll have a read of the older version and be in touch if I'm finding it helpful.
no subject
no subject
no subject
Various fiddly details:
* OAuth / JSON / REST as everyone else above me has said :)
* We should have some form of explicit versioning in the API url for future-proofing, e.g., /api/v1/blah
* strongly in favor of multiple iterations, and I like the break down you already have
* please don't feel the need to copy the existing API functionality exactly! (I don't think you will but I want to emphasize that)
no subject
NNTP :-p
Well, I'm only half-trolling. Consider where I patch my news client to talk to DW, and so each journal is a newsgroup, and each post the head of a usenet thread. Then I can say "Any new posts or comments in
If we're not going to present DW as if it were usenet (shame!), then I think the features I'd like are around making it easier to track new comments as well as articles, for instance:
"What comments on this post are new?"
"What posts in this DW / readinglist have new comments?"
[similar web UI improvements would be grand, too, but that's not what we're talking about here]
no subject
However. If YAAPI is done right, it would be entirely possible for someone to develop a third-party pseudo-news-server that acted as a Dreamwith gateway. If they were feeling particularly clever, they could make messages with a Supersedes header edit the appropriate entry or comment iff it belonged to the person sending the message.
Such a gateway would have to be either run as its own instance per-user or associate an OAuth token with an account on the gateway.
It would be a big project, but it would be a cool one.
Read/unread state can be done in the client if there is (for example) a call to return IDs of posts and comments newer than $TIME in $JOURNAL
no subject
So I'm aware how much easier they make something that is very difficult with the LJ way of doing things: extended discussions involving more than two people. As it is, unless you track a post, you only get notifications to your comments, not for any others, and unless you actively go back to a post to look, you don't see them. Tracking comments is crucial.
When I was pondering doing an OLR for LJ, I looked at the LJ API, but the thing that it was missing was indeed a call that said 'give me all the posts and comments in journal x since time t'.
Doing it via NNTP would save a pile of work, because of how many news readers exist.
(no subject)
(no subject)
(no subject)
(no subject)
no subject
I say this because I've worked in large organisations where one group writes an API that happens to have this property, and another group, later, does something the first group never imagined. A formal requirements-listing process would have been entirely futile; the second group didn't know the API was being written, and didn't decide in advance what they wanted to do - they got the API and then thought about what they could do with it.
no subject
As for "anything one can do through the ordinary human interface", it should (in a final version) do both more and less than that. The ordinary human interface does a whole lot of post-processing that's better left to a client, so in that sense YAAPI should do less, and it doesn't provide direct ways to access certain things, bundling them into larger units, in which sense YAAPI should do more.
no subject
Reasoning: 1. Future code simplification / avoidance of duplication / cleaner front/back end seperation; 2. Enforces the "API clients should be able to do anything the web interface can" ambition that was mentioned above (although of course we might choose not to allow certain actions from anything but the web client)
Of course, I have little experience of such things, and so this might be a silly idea ;-)
no subject
If we were just starting out that would be a great goal, absolutely, but the major argument against doing that now is the same argument explaining why we went with forking LJ to begin with vs starting over from scratch: the epic amount of rewriting it would entail, including the inevitable introduction of a number of bugs. The core functionality of the site (updating, commenting, etc) that the YAAPI will be targeting are the oldest and most hardened areas of the site, with in most cases ... *counts on fingers* 15 or so years of bugfixes and security fixes, etc. You don't throw that out unless there's a very, very, very compelling reason to do so.
Code duplication is less of an argument here (since, again, those areas of the code don't change very frequently so there are fewer opportunities to get out of sync). And really, the amount of work necessary to rewrite the site frontend to use the API as backend is huge, and we just don't have the resources for that. We have enough huge sprawling maintenance tasks outstanding already; we don't need more.
(no subject)
OK, so I've been silent for a while
Quick query: Even with the assumption that YAAPI would be primarily JSON-based there are two major ways it could be implemented: The entire action could be encapsulated in an HTTP POST, or as much as possible could be placed in HTTP headers (exempli gratia, a JSON object containing OAuth tokens as the content of the HTTP Authorization: header, and use of (again, exempli gratia) things like the HTTP DELETE method if deleting an entry or comment rather than overloading POST).
My inclination is to go with the latter, but there may be good reasons to do everything with POST requests.
Thoughts?
Re: OK, so I've been silent for a while
Hmm. I don't want to get bogged down in details, but these two examples feel like separate things, and the answer to one doesn't necessarily affect the other.
e.g., auth -- I'm strongly in favor of keeping it as simple as possible. Maybe even as simple as /v1/foo/1234?token=xxxx
The difference would be most obvious when testing GET requests, but even with POST requests, sometimes lining things up just right can get fiddly, I'd like to avoid that as much as possible.
HTTP DELETE sounds right. I don't feel very strongly about that vs an empty HTTP POST -- other than that I can imagine a hypothetical where someone accidentally does the latter ;) So using DELETE sounds reasonable.
I guess PATCH vs POST is another possibility?
Re: OK, so I've been silent for a while
Might be a good idea to post new questions in a followup entry -- dunno how many people will still be watching this one!