![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
![[site community profile]](https://www.dreamwidth.org/img/comm_staff.png){kind=small}
On clients and APIs
Dreamwidth's APIs are poorly documented (people basically have to work off docs for old versions of LJ's APIs). They're also missing key features, like comment handling for more than backups.
I've been told there have been "some internal conversations about deprecating the XML-RPC API -- keeping it for backwards compatability, but moving to a much more modern second-gen API", but that nobody has had both the time and the inclination to work on designing such a thing.
Well, this is me, volunteering. To that end, I'm looking for input on what exactly such a new API needs to provide, and whether there's a preferred underlying technology to build on (exempli gratia, stick with XML-RPC? Change to SOAP? Use JSON? RESTful or not? et cetera). What I'm getting at here is that I'm entirely happy to take point, as it were, and to make decisions (especially where there's little or no consensus and someone has to make the call), draw up specs, write docs, and so forth, but the result is highly unlikely to be a really useful API unless I get input from more sources than my own experience and looks at the code.
At this stage, therefore, I want everything you, the reader, have to say on the subject. Use cases especially.
Go.
I've been told there have been "some internal conversations about deprecating the XML-RPC API -- keeping it for backwards compatability, but moving to a much more modern second-gen API", but that nobody has had both the time and the inclination to work on designing such a thing.
Well, this is me, volunteering. To that end, I'm looking for input on what exactly such a new API needs to provide, and whether there's a preferred underlying technology to build on (exempli gratia, stick with XML-RPC? Change to SOAP? Use JSON? RESTful or not? et cetera). What I'm getting at here is that I'm entirely happy to take point, as it were, and to make decisions (especially where there's little or no consensus and someone has to make the call), draw up specs, write docs, and so forth, but the result is highly unlikely to be a really useful API unless I get input from more sources than my own experience and looks at the code.
At this stage, therefore, I want everything you, the reader, have to say on the subject. Use cases especially.
Go.
no subject
Fast login switching
Something that tolerates network latency -- when I'm in the subway and I have no signal, I want to be able to hit send on an action and have the app realize that there's no connection and queue it for later, and in context with the fast login switching, if I'm working as User A and then as User B when the train comes out of the tunnel, I don't want my queued actions for User A to accidentally be sent as the current logged in user (B) instead.
Easy retrieval of related comments: given sufficient information to identify a comment, what is its direct parent, direct children, and whole thread? (Bonus points for easy pagination of the whole thread, because some RP threads get hella long.)
With any given comment, have I replied to it yet? And under which identity?
Some other things which are currently painful through the web involve entry management, like re-doing tags.
Retrieve all entries with a given tag.
Apply a given tag to an entry.
Retrieve the current tags for an entry.
Retrieve the current security settings for an entry. (Must play well with custom settings, including multiple custom security groups.)
Apply these security settings to an entry.
Retrieving the userpic for a given entry.
Retrieving the userpic for a given comment.
*pauses a bit to catch breath* I hope this is a good starting point.
(no subject)
(no subject)
no subject
The conversations we've had have mostly been in-person, so we don't really have anything to point anyone to. I think the best thing is to get the conversation going from the ground up. I do remember the consensus was RESTful and JSON, but I am not the technical person here! (
The rest of this comment is going to be a complete and total braindump in absolutely no particular order. /me cracks knuckles
The XML/RPC API that we inherited from LJ is really divided into four conceptual chunks:
* Functions to update your own journal.
* Functions to manage your "friends list". (Which we don't have, per se -- when we were splitting "friend" into "I want to read you" (subscription) vs "I want you to be able to read my locked stuff" (access), we ran into a load of questions with the API as to whether it should affect access or subscription, and we never decided on a good answer -- we wanted to keep backwards compatability for LJ clients, but we couldn't tell enough about what client authors were using that API call for to be able to decide if it should be access or subscription. Any new API will be able to make the distinction from the beginning, so it will be a lot more relevant.)
* Functions to make it easy to download your data for offline backup
* Functions necessary to handle logging into the site and authenticating the person who wants to do stuff.
Stuff the existing API does as part of logging in/authenticating/etc, so needs to be present in some form:
getchallenge — Generate a server challenge string for authentication.
login — validate user's password and get base information needed for client to function
sessionexpire — Expires session cookies.
sessiongenerate — Generate a session cookie.
Core functions of the site that can be done through the existing API, and a new API should also be able to do (name of the existing API call is in parens):
* Post an entry (postevent)
* Edit an entry (editevent)
"Friend"-related stuff (see above for caveats):
* Check for updates on the reading list (checkfriends)
* Manage the user's circle: subscribing to someone, giving someone access, unsubbing from someone, removing someone's access (editfriends)
* Manage the user's access filters (editfriendgroups)
* Get a list of people who are on the account's subscriptions/access list (getfriends) -- I am fairly sure, if I don't misremember, that as part of the friend=>subscription/access split, we just threw up our hands and had this API function return blank all the time, because we couldn't decide if it should return "people this account subscribes to" or "people this account gives access to" for better backwards compatability.
* Get a list of what other accounts have listed the account you're logged in as as a friend (friendof) -- if I remember correctly (and I'm pretty sure I do), this was added to the API on LJ to enable people to make silly little toys, since people were scraping profiles in order to get that data. (Stuff like "six degrees of separation" tools, etc.) Like 'getfriends', I'm pretty sure we just threw up our hands and have this return blank now, because we couldn't decide if it should run off access or subscription.
Downloading/backing up data:
* getdaycounts — This mode retrieves the number of journal entries per day.
* getevents — Download parts of the user's journal. See also syncitems mode.
* getfriendgroups — Retrieves a list of the user's defined groups of friends.
* getusertags — Retrieves a list of the user's defined tags.
* syncitems — Returns a list of all the items that have been created or updated for a user.
And then there's the one lone API call that doesn't really fit into any of the above, 'consolecommand'. It lets people, well, run admin console commands. This is really less of an issue for APIv2, since we've been moving away from the console as something non-admin type people should ever have to touch.
I think the biggest thing to keep in mind, conceptually speaking, is that the existing API serves two functions: "we added this to the APIs so people can access data about accounts to make useful toys without screen-scraping" (aka, data that is public and does not require authentication to be able to see, but the API is there to lighten the load on the servers from third-party tools) and "we added this to the APIs because it's something people want to do to interact with the site and their account, so it requires authentication".
Examples of the former: "look up an interest and see who lists that interest", "who does this account subscribe to", "when was the last time this account was updated". Examples of the latter: "post an entry", "read my reading list" (logged-in, so you can see locked entries you have access to), "reply to this comment", "edit an entry", "make changes to my account preferences". Sort of the distinction between "show me all the userpics Account X has uploaded" and "allow me to upload, keyword, and delete my userpics", so to speak.
For an APIv2, I would mostly want to concentrate on the "stuff that requires authentication to allow a user to interact with their account" at first, especially since we'd want to keep the old XML-RPC API around (for backwards compatability with existing "LiveJournal" clients) for at least a little while and definitely until we had a wide range of clients (for desktop and for mobile, and for multiple computing environments).
So, for a "minimum viable prototype" of an APIv2 (we need a snazzier name for that) I'd say I should be able to, through the API (and thus through any downloaded client that uses the API):
* Write an entry and post it to my journal or to a community I have posting access to
* Edit an entry I've already posted (including the metadata such as tags and icons) (which of course requires a way to a, get the text of past entries, b, get lists of tags I've previously used, c, create new tags if necessary, d, get a list of icons I've uploaded plus their keywords, etc, etc -- this is kind of a rabbit hole of "other dependencies" here)
* Read my reading page (with authentication so I can read locked entries I'm authorized to see)
* Read entries other people have posted, including comments
* Post comments, and edit comments I've posted
* Download my whole journal (entries and comments) for offline backup
* Upload icons, delete icons, or edit icons' keywords/descriptions/comments
Things I'd look for in the next iteration:
* Subscribe to people, grant access to people, unsubscribe from people, and revoke access from people
* Manage my subscription filters and my access filters
* Read my reading page with particular subscription filters
* Ban and unban people from my journal
* Change my privacy settings (or, really, change any of my account settings)
* Manage my communities: add members, remove members, handle community moderation tasks (entries in a community's moderation queue, I mean), change community settings
The "look up public data about any account" part of the API -- those bits that aren't necessary for the above, I mean -- can probably wait until after all of the above has been handled.
(Also, I don't know if you know this, but we have an irc channel -- irc.freenode.net, channels #dreamwidth for general social hangout and chat, #dreamwidth-dev for development-focused discussion -- that you might want to come join us in. It's even odds at any given moment as to whether there's anybody awake and chatting in #dreamwidth-dev, but if you want to bounce ideas off people in realtime, #dreamwidth almost always has at least a few people alive and willing to answer questions about things like how they use the site, etc.)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
no subject
As someone who still wants to make that Dreamwidth app, I am 100 percent in favour of this and would be happy to give you feedback as needed.
no subject
Given those things, I would almost certainly develop apps/tools/something using them, and perhaps contribute to a Ruby gem to wrap it all nicely.
Ask and ye shall receive.
(no subject)
book recommendation/offer
Re: book recommendation/offer
no subject
(no subject)
no subject
Various fiddly details:
* OAuth / JSON / REST as everyone else above me has said :)
* We should have some form of explicit versioning in the API url for future-proofing, e.g., /api/v1/blah
* strongly in favor of multiple iterations, and I like the break down you already have
* please don't feel the need to copy the existing API functionality exactly! (I don't think you will but I want to emphasize that)
no subject
NNTP :-p
Well, I'm only half-trolling. Consider where I patch my news client to talk to DW, and so each journal is a newsgroup, and each post the head of a usenet thread. Then I can say "Any new posts or comments in
If we're not going to present DW as if it were usenet (shame!), then I think the features I'd like are around making it easier to track new comments as well as articles, for instance:
"What comments on this post are new?"
"What posts in this DW / readinglist have new comments?"
[similar web UI improvements would be grand, too, but that's not what we're talking about here]
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
no subject
I say this because I've worked in large organisations where one group writes an API that happens to have this property, and another group, later, does something the first group never imagined. A formal requirements-listing process would have been entirely futile; the second group didn't know the API was being written, and didn't decide in advance what they wanted to do - they got the API and then thought about what they could do with it.
(no subject)
no subject
Reasoning: 1. Future code simplification / avoidance of duplication / cleaner front/back end seperation; 2. Enforces the "API clients should be able to do anything the web interface can" ambition that was mentioned above (although of course we might choose not to allow certain actions from anything but the web client)
Of course, I have little experience of such things, and so this might be a silly idea ;-)
(no subject)
(no subject)
OK, so I've been silent for a while
Quick query: Even with the assumption that YAAPI would be primarily JSON-based there are two major ways it could be implemented: The entire action could be encapsulated in an HTTP POST, or as much as possible could be placed in HTTP headers (exempli gratia, a JSON object containing OAuth tokens as the content of the HTTP Authorization: header, and use of (again, exempli gratia) things like the HTTP DELETE method if deleting an entry or comment rather than overloading POST).
My inclination is to go with the latter, but there may be good reasons to do everything with POST requests.
Thoughts?
Re: OK, so I've been silent for a while
Re: OK, so I've been silent for a while