dw_dev | API wishlist?

Entry tags:

speccing

API wishlist?

So my workplace gives us a chance twice a year to work on anything we want for a few days, work-related or otherwise, and I thought I'd use some of my time to actually get the new API into a state where we can make it live. With that in mind! What endpoints would you like to see implemented? We're making this Swagger/OpenAPI-compliant, so everything takes JSON as arguments and return JSON formatted data. Stuff already on the list to do or partially done:

/api/v1/users/{username} - GET, shows info about the user (specific info TBD; subset of profile, probably?)
/api/v1/users/{username}/icons - GET, lists icons
/api/v1/users/{username}/icons/{iconid} - GET, returns icon data
/api/v1/users/{username}/journals - GET, returns list of journals with write access.
-- for now, returns only user's primary journal
/api/v1/journals/{username}/accesslists - GET, list of access lists for journal
/api/v1/journals/{username}/tags - GET, list of tags for journal
/api/v1/journals/{username}/xpostaccounts - GET list of xpost accounts
/api/v1/moods - GET list of moods
/api/v1/commentsettings - GET list of allowed comment settings
/api/v1/journals/{username}/entries - POST new entry, GET list of recent entries
/api/v1/journals/{username}/entries/{id} - POST edit entry, GET specific entry
/api/v1/journals/{username}/files - POST new file
/api/v1/spec - GET, returns a description of the API

(Mark and D, as usual, get the final say - some stuff people want may end up being security/privacy risks in non-obvious ways)

Flat | Top-Level Comments Only

Most obvious (to me) things missing are lack of anything for logging in or out, commenting, or retrieving a user's reading page entries.

Ah, yeah, I forgot to put login/logout there because it's likely to be a slightly different beast - what I've got written already works with regular auth cookies, but obviously that's not the best option for an API :)

*starts hyperventilating*

As it happens now's kinda bad for me – this is interrupting my procrastination of other things. What's your timeline for comments being useful? Can you give us a deadline for getting back to you?

My big push is going to be end of this week/this weekend, but I doubt I'll get EVERYTHING done so - whenever you've got time?

Okay, I'm back.

You have:

/api/v1/journals/{username}/accesslists - GET, list of access lists for journal

How about allowing manipulation of accesslists by API? In parallel with your other URLs, we could have:

/api/v1/journals/{username}/accesslist/{accesslistid} - GET, list of members of access list; POST, save list of members of access list.

But honestly, I would feel better, as a developer, being able to add or subtract individual users from an accesslist, rather than having to overwrite the whole thing every time I made an edit. The latter seems like a recipe for really awful bugs.

Use cases: GET: being able to retreive a list of members means I can back up my access lists, something that is important to me, and actually super hard. I've also kicked around an idea of combining such a feature with OpenID, such that I could run a website where if users authenticate there with their DW OpenID, they can have access to things based on my access lists – basically API access to my accesslist memberships lets me use DW's ACLs elsewhere. So I could, for instance, set up sound or PDF or video sharing just for people on my access lists. POST: remote/client management of membership.

We should probably think through how all this will work with communities, to make sure nothing is incompatible with them, later on when the API is extended to them.

I mean everything looks fine to me at the moment, but it's 2:21am local time.

/api/v1/journals/{username}/entries/defaults - GET, returns the key-value pairs of all the default settings for a journal's entries (i.e. security, commenting, screening, ?).

Use case: so posting clients can correctly apprise the user of what the default behavior will be when they hit the Post button.

I'd kind of like to be able to do the sorts of navigational things via the API one can do via the standard web interface: return a batch of entries all of the same:
• tag
• security level (including custom levels)
• year
• month
• day
in chronological or reverse chronological order.

(If it would be possible to intersections of any of {tag, security, date}, that would be grand.)

Usecase: that would be a step in the direction of being able to have an entry-management tool, that allows bulk changes to batches of entries, like "set all entries older than X to No Comments" and "set all entries tagged 'my sex life' to Private".

Edited 2017-10-12 06:39 (UTC)

/api/v1/journals/{username}/files - POST new file

And can we have GET list of files?

(A way to back up files!!)

I want to acknowledge I'm dumping a lot out here, and want to apologize if I'm being overwhelming. I was taking that "wishlist" thing at its word.

As a general principle, I'm personally far more interested in the power of third-party apps (mobile, desktop, or even server) to provide functionality to DW that it doesn't have, rather than to provide alternative platforms for access that do what the website (or a subset of the website) already does. I think there's a tendency for APIs for web-based services to be designed to focus on the core functionality of the web user interface; but by opening up the non-core functionality, an API allows third parties to solve problems for the service and provide novel features.

Backup utilities is a classic example, and what with the Great Russian Migration from LJ earlier this year, I think it's on a lot of people's minds. There was no reasonable way to back up one's images from LJ, because the API didn't cover it.

Similarly, providing API access to images allows hypothetical client developers to do things like make phone clients that interoperate with cameras, to make it super easy to post photos to DW. That would probably be a popular feature – it could even make DW more competitive as a platform – but not so certain a thing that DW would want to invest in developing such a client. An API allows third parties to take a stab at it, and DW to benefit by it if they do.

Heh, the reason I started working on the API was to be able to provide utilities for RPers (like generating a list of all your comment threads in a comm, or counting how many comments or entries you made in a month), so I definitely hear you on opening up stuff for other devs to implement non-core functionality like that.

And no worries about the amount of stuff! It may not all get done this push, but it's good info to have going forward.

♥!

So, I'll just add this, while asking for the moon:

/api/v1/journals/{username}/grants - GET returns the list of usernames that username (actually {journalname}) grants access to – and if {username} has that public on their profile, you don't have to be authenticated to the API to get it.

Because I think – not sure, but I've thought cursorily through this – that's the last technical obstacle between here and DW Phone Posts being a viable, commerical third-party service. Not that's I'm promising to start a DW collateral business in my non-existent spare time, but. Just sayin'.

(If there is an unemployed dev ops/sys admin/developer who always wanted her own business, and thinks this might be it, she should shoot me a DM.)

Edited (Tyop.) 2017-10-13 19:00 (UTC)

Oh, and! Speaking of asking for the moon:

It would be awesome – and way beyond the functionality of a mere API as presently being conceived – if DW had a settings page that allowed users to create API access keys that granted permissions to an arbitrary subset of DW functionality, so then they could hand the API access key over to a third party, to let that third party act as an agent, with less-than-full control of one's journal.

• A backup service could be allowed to read one's locked posts, but not one's grantors' locked posts, and have no write access at all.

• An automated posting service, e.g. IFTTT, could be allowed to post, but not read or comment at all.

• A design service could be allowed to change one's journal's style, without having any access to locked content, and no other write or read permissions.

This is excellent news.

My suggestion is: 'list of posts and comments that have been created, modified or deleted since <time>', where <time> is something that can be used to reliably avoid gaps; it might be a sequence number rather than an actual time. Could be multiple queries for different kinds of object, with the caveat that nobody wants to poll every entry in a journal to see if there are changes to comments.

Obviously what I'm interested in here is incremental journal backups l-)

Hm, I'll look into it! This may end up being better-handled in other ways, though - I know we have at least partial code for 'back up my whole journal' already done somewhere, and I'm guessing it may use DB flags like the importer does, which aren't exposed outside of the server backend.

FWIW, I'd want that for a "show me which of my friends' posts have new comments" functionality. Which goes with what

siderea said above about enabling functionality that DW doesn't have.

I once (years ago) considered writing a mobile app as a learning project for school to give me notifications of new posts, but realized that the api really should support a couple of functions first.

- Number of new entries on reading page since time x
- Number of new entries on subscription filter y since time x
- List of subscription filters for my account

Those would allow displaying the number of new posts on mobile's front screen easily and setting up which number should be displayed on the mobile app.

Using push notifications for that would be extra nice, but I know nothing about them, being a noob. :)

Edited 2017-10-11 08:11 (UTC)

I will add those to the list! I'm not sure push notifs would be possible under this framework, because that requires the server to make outbound contact all the time, instead of waiting on requests, and I could see that getting expensive quickly, but the others are definitely possible.

I guess it would depend on how many users are using the app and how often they are set to poll the numbers (how real-time the users want it), and how often the numbers change.. If you have a relatively rarely changing numbers, then polling them often would cause more load than pushing them. Also it might be a good idea to consider that there will probably be clients out there who (somewhat understandably) want to poll way too often to keep the notifications closer to real time. Real time does not matter to me personally, I don't know if it matters to other people.

Edited 2017-10-13 20:39 (UTC)

Hmm, true! I think push notifications may be out of my skill set, honestly, but I will definitely investigate.

Investigation is good, and if it turns out that there's a need for this, then just recognizing that the need exists and writing it down is also a step forward.

Edited 2017-10-14 08:00 (UTC)

There are some stand-alone mobile apps which make it very easy to send arbitrary push notifications to a mobile device.

iOS: Prowl https://www.prowlapp.com/
Android: Notify My Android http://www.notifymyandroid.com/

Your code can make a single HTTP request from your server, and these applications (and their back-end-services) do all the rest. Using these services/apps, sending arbitrary push notifications becomes nearly a one-liner.

I don't have personal experience with Notify My Android, but I've used and integrated Prowl (iOS) into a few projects quickly and easily.

Each user who wants notification would have to install the (appropriate) app, and provide DW with their unique notification key, which they obtain from the app. Then DW servers make one quick, light HTTP request with the user's key in it, and poof, notification pops up.

Prowl lets you not only include the notification message, but also an optional URL that the user can click on to go right to the relevant page (e.g., to see a newly posted comment).

I'd be very happy to help out, including doing some prototyping.

UPDATE: turns out you can also just send an email to {apikey}@api.prowlapp.com, and the mobile device will get a push notification.

Edited 2018-02-05 17:29 (UTC)

Get inbox updates.
Track.

AKA as a user I'd like to get real push notifications for tracked items and be able to view new comments.

I'd like to see access to the inbox, too. It would be great to be able to clients that backup and manage the inbox.

Also, I'm concerned that if the API manages to cause clients to happen, and people start DWing from the phones or desktop apps or whatever, and those don't support the inbox, it will become even more impossible to direct-message people, because those users will never see anything about it, unless they have turned on email notification (which apparently is not on by default?)

I don't just want the API to support the inbox, I kinda want inbox support to be mandatory for DW posting/reading clients. Because I'm afraid that otherwise clients will turn the inbox into dead.letter.

ONG access to the inbox, I have 5K messages from importing a big LJ and cannot stand deleting them 10 at a time. I would like to select based on

Your entries have failed to import and will not be retried.
XMLRPC failure: Client error: Invalid text encoding: Cannot display this post. Please see http://www.livejournal.com/support/encodings.bml for more information.

and delete them all!!!

I asked Denise when I reported it but she was more interested in fixing the inbox spam than deleting from my personal inbox.

I completely forgot about inbox, because I don't use it. But yes, I think it would be good to have a call for number of new items in my inbox, because then that could also be set as a notification on a mobile device.

Those options are definitely possible! Though as I mentioned in another comment, I think push notifs may be too computationally expensive - that's a call for Mark to make

Understood. I would happily pay for push notifications.

My wishlist item is 'documentation', especially if you're changing major aspects of how the API works. (I looked into implementing the reading list/access list split for an open-source client a couple of years ago, and had to bow out because there wasn't enough API documentation support. Now that I'm a better developer, I'd like to get back to this - OpenLJ works great with DW otherwise.)

Oh, for sure! Part of the nice thing about OpenAPI + the way I've started implementing it is that it's partially self-documenting - the files that generate the docs are the same ones that generate the actual code. A human still has to add the descriptive text that the server doesn't need, but they can't actually get totally unsync'd, and will always at least have paths/methods/what it takes/what it gives back. The current XML-RPC API and the docs that go with it, uh, leave a lot to be desired.

Most excellent! If you want someone to run an editorial eye over the documentation afterwards, please ping me; I have some free time.

Thanks! I'm definitely guilty of just being like 'it does a thing!' because I've usually been neck-deep in code for hours at that point

This is a good starting point--a very data-driven API. Thanks for doing this! I also get some days like that, and should figure out if this could count.

A second level down might also be nice:
given (auth, user, tag), get a list of entries
given (auth, user, accesslist), get a list of entries.
given (auth, user, entry), get previous or next entries.
Auth here is the identity to query using, since the results of those matter based on who you are.

Some more queries I find myself doing manually that might be nice to have a simpler way to do:

1. Translating between date and entries, in both directions. (It's one-to-many, of course.) Even cooler would be "get entries between START and END", but that's fancier.
2. Something regarding polls. Probably the ability to map entry to poll ID, then querying by poll ID to get poll results/status. (I put a poll in almost every entry, but I am...not typical.)

/api/v1/journals/{username}/subscriptions - GET, list of subscription lists for journal (and POST to update them).

Definitely doable!

(here via

siderea)

I may have missed it, but I don't see anything here for retrieving, posting, or editing comments.

Can't wait to get my hands on this. Also, thanks for pointing me at Swagger/OpenAPI -- that's really useful.

You didn't want! The list wasn't intended to be complete, just 'what's happening first' - about half of it already has working code in dev.

Not sure this was covered, and my geek-speek is kinda impaired lately with crap sleep.

My suggestion:
Force the API to recognize security levels (filters) on posts. As in, if there is no implementation of filter levels, the only posts the script/program can see are public ones.

Reason:
There was some hullabaloo a LONG time ago about one of the various third party LJ apps (I don't remember which one, it was at least a decade ago that I heard about this) being able to *completely* bypass post filters/security levels, allowing anyone using that particular app to see *any* post on a journal, regardless of whether the user was included on a filter for that post or not.

Yikes. (That must have been before my time, so over a decade ago, but that sounds like a spectacular failure.)

It was! Those who knew about it sometimes set up "secret journals" with only trusted people having access/knowing the address.

Yikes is right, that's a disaster. I'm pretty sure the code checks that (a bunch on the entry stuff is already developed) but yeah, robust testing necessary for sure.

Please make sure that's enforced on the side of DW servers rather than the client. :)

With everything in the news, I'm sure people will appreciate feeling more like private/filtered entries will actually STAY that way regardless of how people access the journal.

Oh, absolutely. My day job is in web application security, and while security and privacy are not quite the same thing, there's a lot of overlap in ideas and mindset :) Plus I'm working exclusively on the server-side right now anyway!

Yay, thank you!! :)

Woah, so many comments.

This is great news, since some code I used in the past no longer seems to work for dw-free.

Thank you for taking this initiative. I wish you the best of luck!

Where will you make the code available?

Edited (forogt a word) 2017-10-13 18:26 (UTC)

So the code will be part of the core DW repositories - If you mean more documentation stuff, you can see a sample of what gets generated here: http://www.momiji.hack.dreamwidth.net/api/v1/spec - which can be fed into something to generate a nice webpage view of it all pretty easily

Thanks, Momiji. I added this as a bookmark and will investigate while I'm not studying. I sincerely appreciate the work you're doing!

You're welcome! Just a forewarning, that link is on my dev environment so it may randomly be offline as I work out bugs in my code.

Heh. I've noticed. :) Np.

I was poking around for some other stuff and realized I'd actually installed a nicer UI at some point for the API. http://www.momiji.hack.dreamwidth.net/api_ui/#

This documentation is gorgeous and so useful. Thanks. :)

No problem! Once I get the actual code done I'm going to clean up the docs generator some and maybe see about sorting the routes a bit better so it's more navigable.

Ohhh, that looks good. I do have some wishlist points:

some specific params for GET /api/v1/journals/{username}/entries:
- number of entries to fetch
- offset (so it's possible to get entries in batches)
- from & to timestamps (to get entries from a specific timeframe)
- fetch posts by tag

/api/v1/comments/{post_id} - GET comments on a post, with these params:
- number of comments to fetch
- offset
- from & to timestamps

Generally, I'm hoping for more "frontend" type of endpoints that would make the public data available for automatization/processing - i.e. backups, import/export. The old LJ API is geared toward content-generating apps, I'd love something more akin to Tumblr's API.

/api/v1/comments/{post_id} - GET comments on a post, with these params:
- number of comments to fetch
- offset
- from & to timestamps

Oooh, also a param for fetching just unscreened comments.

Related: some analogue to the &thread= functionality on the main site, getting a comment and all its children. redir_type=threadroot would be a bonus.

Got it!

I see that you've decided to use POST for entry-editing (presumably by way of wholesale replacement). Have you considered allowing PATCH, which is intended for the purpose? I realise you'd have to either figure out how to apply application/json-patch+json to some kind of JSON representation of an entry, or break consistency and use text/x-patch as input, but it might be worth it for small edits on large entries.

Yeah, the idea with POST was basically to mirror the way on-site edit functionality works (with the bonus of being able to co-opt a bunch of the existing pathways for doing that). I hadn't considered PATCH, though since writing this initial spec we've swapped to OpenAPI3.0 which is a lot more flexible about content-type.

Flat | Top-Level Comments Only

API wishlist?

no subject

no subject

no subject

no subject

Access to the accesslist membership lists?

Need to think through how this will work with comms

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

Push notification helpers: Prowl (for iOS), and Notify My Android

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

/api/v1/journals/{username}/entries/{id}

Re: /api/v1/journals/{username}/entries/{id}