On clients and APIs

[darael]

Dreamwidth's APIs are poorly documented (people basically have to work off docs for old versions of LJ's APIs). They're also missing key features, like comment handling for more than backups.

I've been told there have been "some internal conversations about deprecating the XML-RPC API -- keeping it for backwards compatability, but moving to a much more modern second-gen API", but that nobody has had both the time and the inclination to work on designing such a thing.

Well, this is me, volunteering. To that end, I'm looking for input on what exactly such a new API needs to provide, and whether there's a preferred underlying technology to build on (exempli gratia, stick with XML-RPC? Change to SOAP? Use JSON? RESTful or not? et cetera). What I'm getting at here is that I'm entirely happy to take point, as it were, and to make decisions (especially where there's little or no consensus and someone has to make the call), draw up specs, write docs, and so forth, but the result is highly unlikely to be a really useful API unless I get input from more sources than my own experience and looks at the code.

At this stage, therefore, I want everything you, the reader, have to say on the subject. Use cases especially.

Go.

Comments

[jewelfox]

I occasionally think this, then wonder why it can't be more easily done by making existing site styles more responsive/mobile friendly?

More easily done by whom? >_>

Plus a native Windows / Windows Phone app would have live tile support, and the ability to easily share non-access-locked content.

[quartzpebble]

I'm going to be doing some work around client libraries for the MediaWiki API this summer (https://www.mediawiki.org/wiki/Evaluating_and_Improving_MediaWiki_web_API_client_libraries) so I'll be watching this with interest. If there are any bits that you can hand off to a new dev, I'd be interested in helping out with YAAPI. I'd also be interested in working on the documentation.

[fu]

Agreed.

[fu]

Let's go with OAuth only -- it *may* be possible that we decide we want to support something else in the future, but plenty of APIs support purely OAuth with no problems.

I'd be happy to be able to move away from plaintext username and password!

[jewelfox]

I might be able to help out with documentation also ... my OPW stint was for writing developer docs, and I'll need to have all this information organized in order to make anything based on it.

[fu]

I like the idea of chopping things into smaller chunks.

Sooooo a suggestion for where to put up the list: let's have an issue on github for each separate feature, grouped under a milestone for each iteration -- that is vs having one issue called "implement API" with everything listed as a comment there ;)

[fu]

As a potential developer/user of this API, I would like to say: OAuth, REST, JSON.

Yes, yes, and yes.

[fu]

I don't think that there's much I can add in terms of use case! But I'm pretty excited that you're taking point on this and I'm happy to do what I can to enable you :)

Various fiddly details:

* OAuth / JSON / REST as everyone else above me has said :)
* We should have some form of explicit versioning in the API url for future-proofing, e.g., /api/v1/blah
* strongly in favor of multiple iterations, and I like the break down you already have
* please don't feel the need to copy the existing API functionality exactly! (I don't think you will but I want to emphasize that)

mark and I talked some time back about a new API -- there's some code for uploading files over in DW::Controller::API::Media -- but that doesn't cover authentication yet, because we've only been using it on-site (for AJAX calls).

[emperor]

Random ill-formed Things I Would Like:

NNTP :-p

Well, I'm only half-trolling. Consider where I patch my news client to talk to DW, and so each journal is a newsgroup, and each post the head of a usenet thread. Then I can say "Any new posts or comments in emperor?" ; I can visit a particular post/thread and see which comments/articles are new and, if necessary, the already-read context (/comments/articles) surrounding them. This would be Very Cool.

If we're not going to present DW as if it were usenet (shame!), then I think the features I'd like are around making it easier to track new comments as well as articles, for instance:

"What comments on this post are new?"
"What posts in this DW / readinglist have new comments?"

[similar web UI improvements would be grand, too, but that's not what we're talking about here]

[darael]

NNTP is acknowledged as something onto which Dreamwidth maps rather well. It's only not going to happen because there isn't someone with the knowledge and expertise to implement and secure it, or a large enough userbase to be likely to find someone from it (AIUI).

However. If YAAPI is done right, it would be entirely possible for someone to develop a third-party pseudo-news-server that acted as a Dreamwith gateway. If they were feeling particularly clever, they could make messages with a Supersedes header edit the appropriate entry or comment iff it belonged to the person sending the message.

Such a gateway would have to be either run as its own instance per-user or associate an OAuth token with an account on the gateway.

It would be a big project, but it would be a cool one.

Read/unread state can be done in the client if there is (for example) a call to return IDs of posts and comments newer than $TIME in $JOURNAL

[darael]

This is a good plan, but I don't want to clutter up the dreamwidth/dw-free github section with bugs to the effect of "this hasn't been specced yet". As such, I'm going to publish a new repository (Darael/dw-yaapi) on which the milestones and the feature-spec bugs can go, as well as the specifications to close said bugs. Then, when a milestone's complete, it becomes viable to create a corresponding milestone on dw-free and appropriate "this spec isn't implemented" bugs to go with it. That link's dead at the time of writing, but expected to go live in, ooh, five minutes? I probably don't have time tonight to actually enter the first milestone's bugs, though.

[fu]

Whoo, okay!

I don't think anybody would be bothered by having the discussion in dreamwidth/dw-free, but that also looks like a good place to start :) i don't want to step all over your thought process, but I'll keep an eye on the issues repo for updates.

[lovingboth]

It's a while ago, but for many years I was a member of CIX, a conferencing system using the CoSy software. Like BIX, but UK-based. Because it charged by the minute and we didn't have any free phone calls, offline readers, like news readers, became popular.

So I'm aware how much easier they make something that is very difficult with the LJ way of doing things: extended discussions involving more than two people. As it is, unless you track a post, you only get notifications to your comments, not for any others, and unless you actively go back to a post to look, you don't see them. Tracking comments is crucial.

When I was pondering doing an OLR for LJ, I looked at the LJ API, but the thing that it was missing was indeed a call that said 'give me all the posts and comments in journal x since time t'.

Doing it via NNTP would save a pile of work, because of how many news readers exist.

[damerell]

I realise this is going to be a bit moon-on-a-stick, but in my experience, it's a bad mistake to try and work out what an API needs to do; the API should make it straightforward to do anything one can do through the ordinary human-facing interface.

I say this because I've worked in large organisations where one group writes an API that happens to have this property, and another group, later, does something the first group never imagined. A formal requirements-listing process would have been entirely futile; the second group didn't know the API was being written, and didn't decide in advance what they wanted to do - they got the API and then thought about what they could do with it.

[darael]

Native NNTP has been rejected even when there was a patch supplied, because the people who spend most time working on Dreamwidth aren't willing to learn, secure, and maintain another service, even if it happens just to be a gateway to an existing one.

The best I can hope for is to spec YAAPI such that implementing an NNTP gateway using it is pretty easy. Such a gateway would be most easily (but least usefully) done as a single-user affair, in the style of sn or leafnode, rather than a full multi-user NNTP server á la INN or papercut, simply because the latter would need a way to associate gateway accounts with DW OAuth tokens.

And yes, I've thought about it. I mean, I came at this because places like the wiki still suggest that NNTP is a maybe-eventually DW feature, and the only place it was documented as a not-going-to-happen was on Bugzilla, which has of course gone.

In any case, building such a dreamwidth-to-news gateway would require a "give me all posts and comments in journal x since time t" call, and I'm certain that there are other reasons to have it in YAAPI as well. For non-NNTP purposes, there should probably be two other calls for just entries and for comments on an entry.

[darael]

I don't want to write an API to do specific things, but I do want use-cases for the simple reason that if I ask for them people are more likely to point out ones that would require something I've neglected to think of. As I'm also wanting to do this iteratively, so as to have something usable at the most basic level of functionality as soon as possible, these things also help me decide what goes in from the start and what should be deferred to the second or later iteration.

As for "anything one can do through the ordinary human interface", it should (in a final version) do both more and less than that. The ordinary human interface does a whole lot of post-processing that's better left to a client, so in that sense YAAPI should do less, and it doesn't provide direct ways to access certain things, bundling them into larger units, in which sense YAAPI should do more.

[denise]

And also because people who characterize NNTP as "widely known" or "well supported" haven't taken a good look around in the past decade. There are very, very, very few clients left for Mac or Windows that are actively supported (and they're mostly all shareware or commercialware) and if you asked 100 DW users what NNTP was or if they'd ever heard of a newsreader, you'd get 1 "yes" and it'd be someone who's still reading Usenet with their old copy of tin.

(I mean, I love tin. But the Average User can't run tin.)

[swaldman]

A slightly higher-level thought / suggestion: Unless there are major performance or other reasons not to, make it an eventual goal for DW's main web interface to use the same new API.

Reasoning: 1. Future code simplification / avoidance of duplication / cleaner front/back end seperation; 2. Enforces the "API clients should be able to do anything the web interface can" ambition that was mentioned above (although of course we might choose not to allow certain actions from anything but the web client)

Of course, I have little experience of such things, and so this might be a silly idea ;-)

[denise]

If we were just starting out that would be a great goal, absolutely, but the major argument against doing that now is the same argument explaining why we went with forking LJ to begin with vs starting over from scratch: the epic amount of rewriting it would entail, including the inevitable introduction of a number of bugs. The core functionality of the site (updating, commenting, etc) that the YAAPI will be targeting are the oldest and most hardened areas of the site, with in most cases ... *counts on fingers* 15 or so years of bugfixes and security fixes, etc. You don't throw that out unless there's a very, very, very compelling reason to do so.

Code duplication is less of an argument here (since, again, those areas of the code don't change very frequently so there are fewer opportunities to get out of sync). And really, the amount of work necessary to rewrite the site frontend to use the API as backend is huge, and we just don't have the resources for that. We have enough huge sprawling maintenance tasks outstanding already; we don't need more.

[swaldman]

nod fair enough.

OK, so I've been silent for a while

[darael]

I haven't abandoned this; things have just been a bit full-on.

Quick query: Even with the assumption that YAAPI would be primarily JSON-based there are two major ways it could be implemented: The entire action could be encapsulated in an HTTP POST, or as much as possible could be placed in HTTP headers (exempli gratia, a JSON object containing OAuth tokens as the content of the HTTP Authorization: header, and use of (again, exempli gratia) things like the HTTP DELETE method if deleting an entry or comment rather than overloading POST).

My inclination is to go with the latter, but there may be good reasons to do everything with POST requests.

Thoughts?

Re: OK, so I've been silent for a while

[fu]

Hmm. I don't want to get bogged down in details, but these two examples feel like separate things, and the answer to one doesn't necessarily affect the other.

e.g., auth -- I'm strongly in favor of keeping it as simple as possible. Maybe even as simple as /v1/foo/1234?token=xxxx

The difference would be most obvious when testing GET requests, but even with POST requests, sometimes lining things up just right can get fiddly, I'd like to avoid that as much as possible.

HTTP DELETE sounds right. I don't feel very strongly about that vs an empty HTTP POST -- other than that I can imagine a hypothetical where someone accidentally does the latter ;) So using DELETE sounds reasonable.

I guess PATCH vs POST is another possibility?

Re: OK, so I've been silent for a while

[denise]

Might be a good idea to post new questions in a followup entry -- dunno how many people will still be watching this one!

[lovingboth]

It is true that there are fewer news-only programs, but that's probably because mail programs like Thunderbird do NNTP too.

Hmm, track everything of interest and use email to get all comments, hmm.

[schilling_klaus]

Gnus, the nntp client built into the GNU Emacs, is cross-platform.