darael: Black lines on a white background.  A circle, with twenty-one straight lines connecting seven equally spaced points. (Default)
Darael ([personal profile] darael) wrote in [site community profile] dw_dev2014-04-20 01:53 am
Entry tags:

On clients and APIs

Dreamwidth's APIs are poorly documented (people basically have to work off docs for old versions of LJ's APIs). They're also missing key features, like comment handling for more than backups.

I've been told there have been "some internal conversations about deprecating the XML-RPC API -- keeping it for backwards compatability, but moving to a much more modern second-gen API", but that nobody has had both the time and the inclination to work on designing such a thing.

Well, this is me, volunteering. To that end, I'm looking for input on what exactly such a new API needs to provide, and whether there's a preferred underlying technology to build on (exempli gratia, stick with XML-RPC? Change to SOAP? Use JSON? RESTful or not? et cetera). What I'm getting at here is that I'm entirely happy to take point, as it were, and to make decisions (especially where there's little or no consensus and someone has to make the call), draw up specs, write docs, and so forth, but the result is highly unlikely to be a really useful API unless I get input from more sources than my own experience and looks at the code.

At this stage, therefore, I want everything you, the reader, have to say on the subject. Use cases especially.

Go.
azurelunatic: Vivid pink Alaskan wild rose. (Default)

[personal profile] azurelunatic 2014-04-20 01:43 am (UTC)(link)
I think one of the models that would happily build stuff using an API would be role players. So some things that they'd need in applications, and I'm not sure how all of this translates into things the API needs:

Fast login switching
Something that tolerates network latency -- when I'm in the subway and I have no signal, I want to be able to hit send on an action and have the app realize that there's no connection and queue it for later, and in context with the fast login switching, if I'm working as User A and then as User B when the train comes out of the tunnel, I don't want my queued actions for User A to accidentally be sent as the current logged in user (B) instead.
Easy retrieval of related comments: given sufficient information to identify a comment, what is its direct parent, direct children, and whole thread? (Bonus points for easy pagination of the whole thread, because some RP threads get hella long.)
With any given comment, have I replied to it yet? And under which identity?


Some other things which are currently painful through the web involve entry management, like re-doing tags.

Retrieve all entries with a given tag.
Apply a given tag to an entry.
Retrieve the current tags for an entry.
Retrieve the current security settings for an entry. (Must play well with custom settings, including multiple custom security groups.)
Apply these security settings to an entry.
Retrieving the userpic for a given entry.
Retrieving the userpic for a given comment.


*pauses a bit to catch breath* I hope this is a good starting point.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2014-04-20 03:04 am (UTC)(link)

I think a lot of those use cases you mention are things that someone would want to build a client to do, but are not necessarily use cases for an API in particular. Which is a very tiny and perhaps useless distinction (since obviously for a client to be able to do something, the API needs to have provisions for it) but I think is perhaps worth making at this point?

Braindump comment to follow shortly.

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2014-04-20 03:26 am (UTC)(link)
I am so exceptionally glad that you are interested in taking up the challenge, you have no idea. :D (And I am dead serious about the socks.) (For those who didn't see the start of this, it stemmed from this GH issue.)

The conversations we've had have mostly been in-person, so we don't really have anything to point anyone to. I think the best thing is to get the conversation going from the ground up. I do remember the consensus was RESTful and JSON, but I am not the technical person here! ([staff profile] mark and [personal profile] fu will likely want to weigh in, but Mark's swamped at his day job and Fu is on vacation right now.) There might be a better option. I honestly don't know.

The rest of this comment is going to be a complete and total braindump in absolutely no particular order. /me cracks knuckles

The XML/RPC API that we inherited from LJ is really divided into four conceptual chunks:

* Functions to update your own journal.
* Functions to manage your "friends list". (Which we don't have, per se -- when we were splitting "friend" into "I want to read you" (subscription) vs "I want you to be able to read my locked stuff" (access), we ran into a load of questions with the API as to whether it should affect access or subscription, and we never decided on a good answer -- we wanted to keep backwards compatability for LJ clients, but we couldn't tell enough about what client authors were using that API call for to be able to decide if it should be access or subscription. Any new API will be able to make the distinction from the beginning, so it will be a lot more relevant.)
* Functions to make it easy to download your data for offline backup
* Functions necessary to handle logging into the site and authenticating the person who wants to do stuff.

Stuff the existing API does as part of logging in/authenticating/etc, so needs to be present in some form:

getchallenge — Generate a server challenge string for authentication.
login — validate user's password and get base information needed for client to function
sessionexpire — Expires session cookies.
sessiongenerate — Generate a session cookie.

Core functions of the site that can be done through the existing API, and a new API should also be able to do (name of the existing API call is in parens):

* Post an entry (postevent)
* Edit an entry (editevent)

"Friend"-related stuff (see above for caveats):

* Check for updates on the reading list (checkfriends)
* Manage the user's circle: subscribing to someone, giving someone access, unsubbing from someone, removing someone's access (editfriends)
* Manage the user's access filters (editfriendgroups)
* Get a list of people who are on the account's subscriptions/access list (getfriends) -- I am fairly sure, if I don't misremember, that as part of the friend=>subscription/access split, we just threw up our hands and had this API function return blank all the time, because we couldn't decide if it should return "people this account subscribes to" or "people this account gives access to" for better backwards compatability.
* Get a list of what other accounts have listed the account you're logged in as as a friend (friendof) -- if I remember correctly (and I'm pretty sure I do), this was added to the API on LJ to enable people to make silly little toys, since people were scraping profiles in order to get that data. (Stuff like "six degrees of separation" tools, etc.) Like 'getfriends', I'm pretty sure we just threw up our hands and have this return blank now, because we couldn't decide if it should run off access or subscription.

Downloading/backing up data:

* getdaycounts — This mode retrieves the number of journal entries per day.
* getevents — Download parts of the user's journal. See also syncitems mode.
* getfriendgroups — Retrieves a list of the user's defined groups of friends.
* getusertags — Retrieves a list of the user's defined tags.
* syncitems — Returns a list of all the items that have been created or updated for a user.

And then there's the one lone API call that doesn't really fit into any of the above, 'consolecommand'. It lets people, well, run admin console commands. This is really less of an issue for APIv2, since we've been moving away from the console as something non-admin type people should ever have to touch.

I think the biggest thing to keep in mind, conceptually speaking, is that the existing API serves two functions: "we added this to the APIs so people can access data about accounts to make useful toys without screen-scraping" (aka, data that is public and does not require authentication to be able to see, but the API is there to lighten the load on the servers from third-party tools) and "we added this to the APIs because it's something people want to do to interact with the site and their account, so it requires authentication".

Examples of the former: "look up an interest and see who lists that interest", "who does this account subscribe to", "when was the last time this account was updated". Examples of the latter: "post an entry", "read my reading list" (logged-in, so you can see locked entries you have access to), "reply to this comment", "edit an entry", "make changes to my account preferences". Sort of the distinction between "show me all the userpics Account X has uploaded" and "allow me to upload, keyword, and delete my userpics", so to speak.

For an APIv2, I would mostly want to concentrate on the "stuff that requires authentication to allow a user to interact with their account" at first, especially since we'd want to keep the old XML-RPC API around (for backwards compatability with existing "LiveJournal" clients) for at least a little while and definitely until we had a wide range of clients (for desktop and for mobile, and for multiple computing environments).

So, for a "minimum viable prototype" of an APIv2 (we need a snazzier name for that) I'd say I should be able to, through the API (and thus through any downloaded client that uses the API):

* Write an entry and post it to my journal or to a community I have posting access to
* Edit an entry I've already posted (including the metadata such as tags and icons) (which of course requires a way to a, get the text of past entries, b, get lists of tags I've previously used, c, create new tags if necessary, d, get a list of icons I've uploaded plus their keywords, etc, etc -- this is kind of a rabbit hole of "other dependencies" here)
* Read my reading page (with authentication so I can read locked entries I'm authorized to see)
* Read entries other people have posted, including comments
* Post comments, and edit comments I've posted
* Download my whole journal (entries and comments) for offline backup
* Upload icons, delete icons, or edit icons' keywords/descriptions/comments

Things I'd look for in the next iteration:

* Subscribe to people, grant access to people, unsubscribe from people, and revoke access from people
* Manage my subscription filters and my access filters
* Read my reading page with particular subscription filters
* Ban and unban people from my journal
* Change my privacy settings (or, really, change any of my account settings)
* Manage my communities: add members, remove members, handle community moderation tasks (entries in a community's moderation queue, I mean), change community settings

The "look up public data about any account" part of the API -- those bits that aren't necessary for the above, I mean -- can probably wait until after all of the above has been handled.

[personal profile] darael, does that braindump help focus your thinking at all? I can probably keep braindumping for a while, but I want to give you the chance to ask questions first. ;)

(Also, I don't know if you know this, but we have an irc channel -- irc.freenode.net, channels #dreamwidth for general social hangout and chat, #dreamwidth-dev for development-focused discussion -- that you might want to come join us in. It's even odds at any given moment as to whether there's anybody awake and chatting in #dreamwidth-dev, but if you want to bounce ideas off people in realtime, #dreamwidth almost always has at least a few people alive and willing to answer questions about things like how they use the site, etc.)
andrewducker: (Default)

[personal profile] andrewducker 2014-04-20 10:52 am (UTC)(link)
Also: OAuth.

At the moment, for the users who use my web-app to post to DW, I have to store their actual password. This is...not good.

(no subject)

[personal profile] fu - 2014-04-22 07:33 (UTC) - Expand

(no subject)

[staff profile] denise - 2014-04-20 20:03 (UTC) - Expand

(no subject)

[staff profile] denise - 2014-04-20 20:28 (UTC) - Expand

(no subject)

[personal profile] pauamma - 2014-04-20 21:30 (UTC) - Expand

(no subject)

[staff profile] denise - 2014-04-20 21:33 (UTC) - Expand

(no subject)

[personal profile] andrewducker - 2014-04-20 20:27 (UTC) - Expand

(no subject)

[personal profile] fu - 2014-04-22 07:35 (UTC) - Expand

(no subject)

[personal profile] schilling_klaus - 2014-08-08 06:19 (UTC) - Expand

(no subject)

[staff profile] denise - 2014-08-08 07:02 (UTC) - Expand

(no subject)

[staff profile] denise - 2014-04-20 20:05 (UTC) - Expand

(no subject)

[personal profile] alexbayleaf - 2014-04-21 10:04 (UTC) - Expand

(no subject)

[personal profile] jewelfox - 2014-04-21 12:43 (UTC) - Expand

(no subject)

[personal profile] alexbayleaf - 2014-04-21 12:56 (UTC) - Expand

(no subject)

[staff profile] denise - 2014-04-21 20:55 (UTC) - Expand

(no subject)

[personal profile] jewelfox - 2014-04-22 00:57 (UTC) - Expand

(no subject)

[personal profile] fu - 2014-04-22 07:42 (UTC) - Expand

(no subject)

[personal profile] fu - 2014-04-23 08:57 (UTC) - Expand

[personal profile] jewelfox 2014-04-20 07:18 am (UTC)(link)
I tried to find out how to make a DW app for Windows Phone, only to find that I'd have to use those pretty much undocumented APIs in order to so much as read my friends' posts. Apparently LJ had "authenticated RSS" or something, but DW doesn't.

As someone who still wants to make that Dreamwidth app, I am 100 percent in favour of this and would be happy to give you feedback as needed.

[personal profile] alexbayleaf 2014-04-21 10:00 am (UTC)(link)
As a potential developer/user of this API, I would like to say: OAuth, REST, JSON.

Given those things, I would almost certainly develop apps/tools/something using them, and perhaps contribute to a Ruby gem to wrap it all nicely.
fu: Close-up of Fu, bringing a scoop of water to her mouth (Default)

[personal profile] fu 2014-04-22 07:45 am (UTC)(link)
As a potential developer/user of this API, I would like to say: OAuth, REST, JSON.

Yes, yes, and yes.
brainwane: My smiling face, including a small gold bindi (Default)

book recommendation/offer

[personal profile] brainwane 2014-04-21 03:35 pm (UTC)(link)
My spouse wrote "RESTful Web APIs" which has some useful conceptual stuff and a decision procedure for designing a web API. It's well-regarded and I've found it useful. I'd be happy to send you a copy (ebook or paper) in case it'd be helpful. (There's an older version of the book, now deprecated, which you can read for free online.)
quartzpebble: (tea or death)

[personal profile] quartzpebble 2014-04-22 07:01 am (UTC)(link)
I'm going to be doing some work around client libraries for the MediaWiki API this summer (https://www.mediawiki.org/wiki/Evaluating_and_Improving_MediaWiki_web_API_client_libraries) so I'll be watching this with interest. If there are any bits that you can hand off to a new dev, I'd be interested in helping out with YAAPI. I'd also be interested in working on the documentation.

[personal profile] jewelfox 2014-04-22 07:38 am (UTC)(link)
I might be able to help out with documentation also ... my OPW stint was for writing developer docs, and I'll need to have all this information organized in order to make anything based on it.
fu: Close-up of Fu, bringing a scoop of water to her mouth (Default)

[personal profile] fu 2014-04-22 08:03 am (UTC)(link)
I don't think that there's much I can add in terms of use case! But I'm pretty excited that you're taking point on this and I'm happy to do what I can to enable you :)

Various fiddly details:

* OAuth / JSON / REST as everyone else above me has said :)
* We should have some form of explicit versioning in the API url for future-proofing, e.g., /api/v1/blah
* strongly in favor of multiple iterations, and I like the break down you already have
* please don't feel the need to copy the existing API functionality exactly! (I don't think you will but I want to emphasize that)

[staff profile] mark and I talked some time back about a new API -- there's some code for uploading files over in DW::Controller::API::Media -- but that doesn't cover authentication yet, because we've only been using it on-site (for AJAX calls).
emperor: (Default)

[personal profile] emperor 2014-04-22 05:33 pm (UTC)(link)
Random ill-formed Things I Would Like:

NNTP :-p

Well, I'm only half-trolling. Consider where I patch my news client to talk to DW, and so each journal is a newsgroup, and each post the head of a usenet thread. Then I can say "Any new posts or comments in [personal profile] emperor?" ; I can visit a particular post/thread and see which comments/articles are new and, if necessary, the already-read context (/comments/articles) surrounding them. This would be Very Cool.

If we're not going to present DW as if it were usenet (shame!), then I think the features I'd like are around making it easier to track new comments as well as articles, for instance:

"What comments on this post are new?"
"What posts in this DW / readinglist have new comments?"

[similar web UI improvements would be grand, too, but that's not what we're talking about here]
lovingboth: (Default)

[personal profile] lovingboth 2014-04-23 03:12 pm (UTC)(link)
It's a while ago, but for many years I was a member of CIX, a conferencing system using the CoSy software. Like BIX, but UK-based. Because it charged by the minute and we didn't have any free phone calls, offline readers, like news readers, became popular.

So I'm aware how much easier they make something that is very difficult with the LJ way of doing things: extended discussions involving more than two people. As it is, unless you track a post, you only get notifications to your comments, not for any others, and unless you actively go back to a post to look, you don't see them. Tracking comments is crucial.

When I was pondering doing an OLR for LJ, I looked at the LJ API, but the thing that it was missing was indeed a call that said 'give me all the posts and comments in journal x since time t'.

Doing it via NNTP would save a pile of work, because of how many news readers exist.

(no subject)

[staff profile] denise - 2014-04-23 22:42 (UTC) - Expand

(no subject)

[personal profile] lovingboth - 2014-05-28 22:47 (UTC) - Expand

(no subject)

[personal profile] schilling_klaus - 2014-08-07 21:37 (UTC) - Expand
damerell: (computers)

[personal profile] damerell 2014-04-23 03:14 pm (UTC)(link)
I realise this is going to be a bit moon-on-a-stick, but in my experience, it's a bad mistake to try and work out what an API needs to do; the API should make it straightforward to do anything one can do through the ordinary human-facing interface.

I say this because I've worked in large organisations where one group writes an API that happens to have this property, and another group, later, does something the first group never imagined. A formal requirements-listing process would have been entirely futile; the second group didn't know the API was being written, and didn't decide in advance what they wanted to do - they got the API and then thought about what they could do with it.

[personal profile] swaldman 2014-04-24 05:46 am (UTC)(link)
A slightly higher-level thought / suggestion: Unless there are major performance or other reasons not to, make it an eventual goal for DW's main web interface to use the same new API.

Reasoning: 1. Future code simplification / avoidance of duplication / cleaner front/back end seperation; 2. Enforces the "API clients should be able to do anything the web interface can" ambition that was mentioned above (although of course we might choose not to allow certain actions from anything but the web client)

Of course, I have little experience of such things, and so this might be a silly idea ;-)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2014-04-24 06:00 am (UTC)(link)

If we were just starting out that would be a great goal, absolutely, but the major argument against doing that now is the same argument explaining why we went with forking LJ to begin with vs starting over from scratch: the epic amount of rewriting it would entail, including the inevitable introduction of a number of bugs. The core functionality of the site (updating, commenting, etc) that the YAAPI will be targeting are the oldest and most hardened areas of the site, with in most cases ... *counts on fingers* 15 or so years of bugfixes and security fixes, etc. You don't throw that out unless there's a very, very, very compelling reason to do so.

Code duplication is less of an argument here (since, again, those areas of the code don't change very frequently so there are fewer opportunities to get out of sync). And really, the amount of work necessary to rewrite the site frontend to use the API as backend is huge, and we just don't have the resources for that. We have enough huge sprawling maintenance tasks outstanding already; we don't need more.

(no subject)

[personal profile] swaldman - 2014-04-24 09:32 (UTC) - Expand
fu: Close-up of Fu, bringing a scoop of water to her mouth (Default)

Re: OK, so I've been silent for a while

[personal profile] fu 2014-05-06 07:29 pm (UTC)(link)

Hmm. I don't want to get bogged down in details, but these two examples feel like separate things, and the answer to one doesn't necessarily affect the other.

e.g., auth -- I'm strongly in favor of keeping it as simple as possible. Maybe even as simple as /v1/foo/1234?token=xxxx

The difference would be most obvious when testing GET requests, but even with POST requests, sometimes lining things up just right can get fiddly, I'd like to avoid that as much as possible.

HTTP DELETE sounds right. I don't feel very strongly about that vs an empty HTTP POST -- other than that I can imagine a hypothetical where someone accidentally does the latter ;) So using DELETE sounds reasonable.

I guess PATCH vs POST is another possibility?

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

Re: OK, so I've been silent for a while

[staff profile] denise 2014-05-06 08:16 pm (UTC)(link)

Might be a good idea to post new questions in a followup entry -- dunno how many people will still be watching this one!