dw_dev | API things

Entry tags:

data apis

API things

I kept meaning to write this up but

kaberett's volunteer weekend was a good extra nudge, heh.

what is live
- To generate API keys (and delete them), go here - this can be found via Manage Accounts -> Mobile -> Advanced Options (which is at the bottom of the mobile tab)
- The JSON-formatted spec is here and doesn't require an API key to access - we don't have a more human-friendly version yet, though any client that can parse OpenAPI v3.0 specs can take that - Postman is the one I've tried.
- There are not many routes yet - we wanted to get the backend solid before adding stuff that had potential privacy/security risks if something in the backend was wrong
- To use the API key, you need to add an Authorization header to your HTTP request with the value 'Bearer [api key]'.

what is in progress
- I have a pile of routes mostly-written, in this branch. They probably need some light editing, as we moved from OpenAPI v2.0 to v3.0 after I wrote these, and we also hadn't implemented API keys yet. They also need LOTS AND LOTS of testing, oh god please, I'm so bad at QA.
- I wrote a quick'n'dirty primer to adding moar routes here on the wiki. Happy to answer additional questions on it!

what really needs to be done
- TESTING TESTING TESTING
- image-upload-by-API is near the top of the list of 'most wanted things I haven't written yet'
- I want to add a nice UI for the spec a la redoc but I'm not totally sold on that as a look and could use design input and inspiration
- We really need some intro/'API for beginners' text to go on said doc page, I am probably not the right person to write this because I Know Too Much
- It would be great to have a priv system for API keys (so that you could make a key that can read and not post, or vice versa). There are stubs in the API key files for this, but it needs UI and DB stuff as well as... filling out the stubs.
- 404s under /api should really return a JSON 404 message and not... the entire HTML output of the 404 page.

Flat | Top-Level Comments Only

1) It actually *is* limited in the backend, but not well-described in the API at the moment, in part because I haven't figured out how I want to do pagination yet. I think the max entries it will return at once is either 20 or 50? Because yes, that'd be a terrible idea to return ALL entries.

2) That is a good idea! I hadn't actually considered if we store entries differently depending on the editor format, and I should check if we do.

So even in the old entry page, you can use the hacky way to get markdown, which iirc is to put '!markdown' on it's own line as the very first line of the entry, but you can also turn on the beta entry page at /beta (so http://www.momiji.hack.dreamwidth.net/beta on my dreamhack, for instance).

There is a LOT of code, unfortunately, which makes it hard to kind of give a tour, but for API stuff specifically, the routes are defined by a combo of yaml files in /api, Perl endpoint controllers in /cgi-bin/DW/Controller/API/REST, and TO_JSON methods in various object classes that define the JSON print format for that object. For entries specifically, the TO_JSON method is in /cgi-bin/LJ/Entry.pm

1) It actually is limited in the backend, but not well-described in the API at the moment, in part because I haven't figured out how I want to do pagination yet. I think the max entries it will return at once is either 20 or 50? Because yes, that'd be a terrible idea to return ALL entries.

Ok, let leave it the way it is for now. I hope there would be pagination in future, because it would be good for backuping purposes, at least.

2) That is a good idea! I hadn't actually considered if we store entries differently depending on the editor format, and I should check if we do.

I thought about it more, and what do you think about joining all content representation in one structure. Like:

"body" : {
"content" : "Some *text*",
"type" : "markdown",
"rendered" : "Some <i>text</i>"
}

Or may be a bit more complex structure, if there can be more than two representation of the body.

So even in the old entry page, you can use the hacky way to get markdown, which iirc is to put '!markdown' on it's own line as the very first line of the entry, but you can also turn on the beta entry page at /beta (so http://www.momiji.hack.dreamwidth.net/beta on my dreamhack, for instance).

My beta (http://www.nataraj.hack.dreamwidth.net/beta) is empty:

We'll announce more testing opportunities in our official communities. Thank you.

There are no features currently available for beta testing.

That is why I asked. May be some options should be turned on in the config...

There is a LOT of code, unfortunately, which makes it hard to kind of give a tour, but for API stuff specifically, the routes are defined by a combo of yaml files in /api, Perl endpoint controllers in /cgi-bin/DW/Controller/API/REST, and TO_JSON methods in various object classes that define the JSON print format for that object. For entries specifically, the TO_JSON method is in /cgi-bin/LJ/Entry.pm

I'll try to explore the code. It will be low priority, but at least I will try to look at the code before asking questions :-)

Edited 2022-08-20 18:58 (UTC)

Ok, let leave it the way it is for now. I hope there would be pagination in future, because it would be good for backuping purposes, at least.

Well, you can kind of hack pagination by setting the offset parameter, but it's not the most intuitive way of doing it, I agree.

thought about it more, and what do you think about joining all content representation in one structure.

Hmmm. It might be a possibility, but you'll only get the raw content and the format type for entries you have permission to edit. So if you're grabbing entries you don't have permission to edit, you only get the rendered body, and a body object with only one key might look a little silly? I don't know.

My beta (http://www.nataraj.hack.dreamwidth.net/beta) is empty

Oh huh! This is probably a 'I set my dreamhack up ages ago and forgot everything I tweaked' problem, my bad. Looking through the configs, you can uncomment this section in your config-local.pl file and restart apache, I think!

First I did not manage to find framework that allow to wrap json into Moose object fast and simple. So I decided to write one myself. It took me a while, as I am totaly not Moose expert, but now I have early prototype:

https://gitlab.com/dhyannataraj/libwebservice-dreamwidth-perl/-/blob/master/lib/MooseX/Embody/JSON.pm

And I can create Objects from JSON, just by listing it's attributes, like that:

package WebService::DreamWidth::Entry;

use Moose;
use MooseX::Embody::JSON;
use WebService::DreamWidth::Icon;

j_obj "icon" =>(class => "WebService::DreamWidth::Icon", is =>'ro');

j_attr "security" => (is => 'ro');
j_attr "body_html"=> (is => 'ro');
j_attr "subject_raw"=> (is => 'ro');
j_attr "subject_html"=> (is => 'ro');
j_attr "allowmask"=> (is => 'ro');
j_attr "body_raw"=> (is => 'ro');
j_attr "icon_keyword"=> (is => 'ro');
j_attr "entry_id"=> (is => 'ro');
j_attr "datetime"=> (is => 'ro');
j_attr "url" => (is => 'ro');

https://gitlab.com/dhyannataraj/libwebservice-dreamwidth-perl/-/blob/master/lib/WebService/DreamWidth/Entry.pm

This will allow fast creation of prototypes, and then allow adding complex features using Moose's powerful tools.

Edited 2022-09-03 19:39 (UTC)

I have looked at the json returned by /journals/{username}/entries/{entry_id} and /users/{username}

I would expect poster hash from entry, to be subset of hash returned by /users/{username}. poster should have minimal information, that's right, but I expect it to use same keys that is used in full info from /users/{username}. This will allow to make extra attributes loadable on demand.

So I would expect user entry to have "username" instead of "name", as it is username in all the rest cases. And I would expect consistency with "display_name". It should exist in both cases, or does not exist at all...

Oh, excellent points, I'll add that to my to-do list! I really need to sit down and set up reference handling for the spec file parser, so that we can have reusable definitions for stuff like entries, users, icons, etc - at the moment every endpoint spec has to have that information in it, which leads to a lot of inconsistency.

I also miss some basic info, that can be fetched about journal. Like journal title. It would be very handy for testing...

So I can do

if ($client->user("nataraj")->journals->{'nataraj'}->title) eq "Nataraj's cool journal")

and check that WebService::Dreamwidth::Journal objects hash have been properly created by journals method.

Some more notes about consistency:

/users/{username}/icons/ returns full list of "icons" objects, but you can also access single one via /users/{username}/icons/{picid}

/journals/{username}/tags returns full list of "tags" objects, but you can't get certain "tag" object by /journals/{username}/tags/{tagid} (Though I would like to have this API-call while creating tags method for entry)

/users/{username}/journals just lists journal names.

I am not sure what is the REST API tradition, but I would expect that I can guess what I am going to get, by looking at API URL:

Like: /users/{username}/icons/ will return lists of IDs, /users/{username}/icons/1 will return list icon object with ID==1, /users/{username}/icons/* will return full list of icon objects. Or something like that I am not fixed on exact implementation, and do not know what do current tradition of REST APIs prescribes here... Key point is to see what will be returned in each case

What I should get via journals/{username}/accesslists? I am trying to do

curl -H "Authorization: Bearer UJxroElRIqdhgRVYS7y4h6DrFGPP5Wcz" http://www.nataraj.hack.dreamwidth.net/api/v1/journals/nataraj/accesslists

this auth key belongs to nataraj user. It has mutual access and subscription with nataraj2 user. But I still get empty list. How should it work? What should I do to get non-empty list?

Edited 2022-10-30 17:48 (UTC)

Ah, the implementation on that one is broken I think, because I started writing it and then realized I needed to figure out the bitmask arithmetic that backs access filters. For the future, you can set up access filters here: https://www.dreamwidth.org/manage/circle/editfilters and that response should return information about them (which you need if you want to set one as an entry filter). 'accesslist' is maybe not the best wording - we call them access filters on the editing page for them, and 'access groups' on the newer entry creation page.

Most of the objects returned by API has all "coordinates" that is needed to locate that object.

Like icon has "username" and "picid", and this is all you need, to fetch this object again.

Meanwhile "tag" object does not have "journalname" attribute, so you do not have all you need to fetch this object again.

would be also nice to have journal_name in the entry object

Good point on both of those, will make a note of it!

would be also nice to have journal_name in the enrty object

Pagination:

I have 3 entries in my test journal: "Entry 1", "Entry 2" and "Entry 3"

curl -H "Authorization: Bearer UJxroElRIqdhgRVYS7y4h6DrFGPP5Wcz" http://www.nataraj.hack.dreamwidth.net/api/v1/journals/nataraj/entries

Fetches them in backward order: "Entry 3", "Entry 2", "Entry 1"

curl -H "Authorization: Bearer UJxroElRIqdhgRVYS7y4h6DrFGPP5Wcz" http://www.nataraj.hack.dreamwidth.net/api/v1/journals/nataraj/entries?count=1

Returns "Entry 3"

curl -H "Authorization: Bearer UJxroElRIqdhgRVYS7y4h6DrFGPP5Wcz" http://www.nataraj.hack.dreamwidth.net/api/v1/journals/nataraj/entries?count=1\&offset=0

Also returns "Entry 3"

But if I change offset to 1

curl -H "Authorization: Bearer UJxroElRIqdhgRVYS7y4h6DrFGPP5Wcz" http://www.nataraj.hack.dreamwidth.net/api/v1/journals/nataraj/entries?count=1\&offset=1

I am getting "Entry 1" and I guess, this is totally wrong

Edited 2022-11-03 19:18 (UTC)

And speaking about pagination, I guess this offset story is not the best...

If I am fetching some long community, and meanwhile somebody added a new entry, I will get double entry. And if somebody delete entry from the top, I will miss one entry

I guess that all paginated search results should also have some metainfo telling how to fetch next page of the same query...

For DW it may be "next_id": you get the first page, it says "next_id=xxx" in the result data, you query same query with "&start_from=xxx" and getting the next page. Changing any record (except one with id xxx) will not change search result sanity. If looks like more stable...

I also have seen systems that has "search tocken", each search request have been cached, and you can paginate through it using tocken, you've got from the first request. But this was heavy commercial systems, do not think it is DW's case.

Edited 2022-11-03 21:24 (UTC)

Yeah, this is one of the things that I know other people have come up with solutions for, but I need to sit down and research them some.

And you're right, that return order behavior does seem like a bug.

Flat | Top-Level Comments Only

API things

no subject

no subject

no subject

Let me report some progress

And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback