dw_dev | API things

Entry tags:

data apis

API things

I kept meaning to write this up but

kaberett's volunteer weekend was a good extra nudge, heh.

what is live
- To generate API keys (and delete them), go here - this can be found via Manage Accounts -> Mobile -> Advanced Options (which is at the bottom of the mobile tab)
- The JSON-formatted spec is here and doesn't require an API key to access - we don't have a more human-friendly version yet, though any client that can parse OpenAPI v3.0 specs can take that - Postman is the one I've tried.
- There are not many routes yet - we wanted to get the backend solid before adding stuff that had potential privacy/security risks if something in the backend was wrong
- To use the API key, you need to add an Authorization header to your HTTP request with the value 'Bearer [api key]'.

what is in progress
- I have a pile of routes mostly-written, in this branch. They probably need some light editing, as we moved from OpenAPI v2.0 to v3.0 after I wrote these, and we also hadn't implemented API keys yet. They also need LOTS AND LOTS of testing, oh god please, I'm so bad at QA.
- I wrote a quick'n'dirty primer to adding moar routes here on the wiki. Happy to answer additional questions on it!

what really needs to be done
- TESTING TESTING TESTING
- image-upload-by-API is near the top of the list of 'most wanted things I haven't written yet'
- I want to add a nice UI for the spec a la redoc but I'm not totally sold on that as a look and could use design input and inspiration
- We really need some intro/'API for beginners' text to go on said doc page, I am probably not the right person to write this because I Know Too Much
- It would be great to have a priv system for API keys (so that you could make a key that can read and not post, or vice versa). There are stubs in the API key files for this, but it needs UI and DB stuff as well as... filling out the stubs.
- 404s under /api should really return a JSON 404 message and not... the entire HTML output of the 404 page.

Flat | Top-Level Comments Only

\o/ THANK YOU. :D

\o/

Thanks for the details!

Is there any way to test the API yet? I’d be happy to help write some of it, but I like having tests to support my work. (And if not, I could think about creating one next time I do Dreamwidth dev work.)

Not a really structured one (see: I am really bad at QA) - I assume it's possible somewhere to make HTTP requests in Perl and... hopefully our test suite has mock journals and data? Otherwise it's possible to write test suites in Postman. The journals + data is the bigger issue, honestly, because you can't really test if there's nothing to pull down, and very few of the routes will be returning site-level data that's the same for everyone (the comment screening options is one of that handful)

SO EXCITE!!!

Ooooo, neat!

Hm. I'm seeing some oddities w.r.t. character encoding. I have a couple of userpics that have "smart" single quotation marks in them, and they don't seem to be rendering properly.

JSON is supposed to be UTF-8 encoded, and the header I get back from the /users/{userid}/icons call is "Content-Type: application/json; charset=utf-8", but the content of the comments field sometimes has odd characters in it.

Ooh, thanks for catching that, I'll dig into it some. Also, if you want to poke at the new stuff, it's live on my dev environment - http://www.momiji.hack.dreamwidth.net/ - and there is a partial docs page at http://www.momiji.hack.dreamwidth.net/api (the 'try it' buttons don't do anything yet though, as I haven't written the AJAX backing them yet)

Hi!

I want to write some perl script, that would do posting to dreamwidth, using it's API, but did not find not any ready-to-use perl module, not an API spec. As far as I get from this post some kind of API actually exist. Where should I start.

I guess while solving my task I can do some doc or code writing that would be useful to community, but I need to understand what actually exists, and what should be done.

Can somebody help me with that?

Edited 2022-02-04 20:42 (UTC)

So there's actually two APIs - one is the old, inherited-from-LJ XMLRPC interface, which is partially documented here: http://wiki.dwscoalition.org/wiki/index.php/XMLRPC_Protocol and the other is the new REST interface, which is somewhat self-documenting. Both work via fairly plain HTTP calls, which you can do via any package that allows you to construct HTTP calls, though I think there's some wrapper packages for both XMLRPC and for OpenAPI3 (which the new API spec conforms to). If you want to make something that works right now, XMLRPC is the way to go because that's already live and unchanging, while the new API spec is a work in progress when I have time to deal with it.

As far as I've read, XMLRPC does not allow to do MarkDown posts. And that's what I am need to do. That is why I've started exploring this topic.

// I am trying to write ActivityPub to DW reposter script. For my own blog and blogs of my firends. ActivityPub usually have Markdown in it, so it seems to be wise to make it work with MarkDown from the start.

And second thing is that I am a perl developer (or ex perl developer, if one can become ex when he started with perl), and I guess sooner or later I should join to DW developers as a volunteer, as I have some ideas I'd like to implement. May be it is good time to do it now, if you have API-related tasks that can be passed to newcomer developer. If it will make the whole process faster...

Ah, okay! The new entries API doesn't have a markdown flag yet - it expects entries to come in as HTML - but adding one is definitely in scope.

Honestly at this point the big things are 1) properly implementing ratelimiting, 2) working out what the comments API should look like (which is a design question more than a coding question), and 3) a LOT of testing. (3) is the bit that I could use the most help with - I'm not great at QA, and there's a lot of variables to test.

3) a LOT of testing. (3) is the bit that I could use the most help with - I'm not great at QA, and there's a lot

I am not QA expert either, but I like writing unit tests (if they are applicable in this case), I can play with API and see what is working and what is not (I am sometimes good in finding patterns that does not work), and I also can starting to implement script I need, it would be both, API testing and I will have workable script, as soon as API is commited in prod.

So I guess, first thing I need to do is to request own hack.dreamwidth.net and install api-more branch there...

Update: I managed to create account on http://www.momiji.hack.dreamwidth.net/, so I will not need my own hack for now. I will start writing script that do both REST-API request and XMLRPC one (or pure web-site requests if something does not work in XMLRPC), and make sure that results are same. This script can be later used in some integrity check procedure.

This will not be quick, I will use my hobby-time for it, but sooner or later I will finish it.

Edited 2022-02-07 15:11 (UTC)

Hah, still ahead of me, I'm only very grudgingly coming around to make unit tests a regular part of my coding via my day-job.

Let me know if you run into any weirdness, either via a comment here or a DW private message - I have email notifications turned on for both, even if I am sometimes slow to get around to writing replies. I'm hoping I'll have some energy to work on DW code as the weather warms up and the days get longer, and having people wanting to use and help with it is definitely a motivator :)

Sorry for long pauses... But I have first questions/comments:

http://www.nataraj.hack.dreamwidth.net/api/v1/journals/nataraj/entries Will it really return ALL journal entries? If I do it for my (nataraj) journal. It will be really long list, that will take a lot of resources... I guess pagination is needed here for sure. It can't be put in production the way it is now.
http://www.nataraj.hack.dreamwidth.net/api/v1/journals/nataraj/entries/886

"body_raw" : "This is <b>Entry 3</b>",

I would expect here another attribute that will tell what is the formatting is used in raw text. Even if it is only html available for now, I guess is should be specified explicitly.

Can I turn on markdown posts beta on my dreamhack? How can I do it?

I guess it would be also good if you start showing me around in DW code. I guess if you show me key points in the code, I will be able to file merge request for issues like raw-content-type myself. But I am not insisting on this.

Edited 2022-08-20 17:48 (UTC)

1) It actually *is* limited in the backend, but not well-described in the API at the moment, in part because I haven't figured out how I want to do pagination yet. I think the max entries it will return at once is either 20 or 50? Because yes, that'd be a terrible idea to return ALL entries.

2) That is a good idea! I hadn't actually considered if we store entries differently depending on the editor format, and I should check if we do.

So even in the old entry page, you can use the hacky way to get markdown, which iirc is to put '!markdown' on it's own line as the very first line of the entry, but you can also turn on the beta entry page at /beta (so http://www.momiji.hack.dreamwidth.net/beta on my dreamhack, for instance).

There is a LOT of code, unfortunately, which makes it hard to kind of give a tour, but for API stuff specifically, the routes are defined by a combo of yaml files in /api, Perl endpoint controllers in /cgi-bin/DW/Controller/API/REST, and TO_JSON methods in various object classes that define the JSON print format for that object. For entries specifically, the TO_JSON method is in /cgi-bin/LJ/Entry.pm

1) It actually is limited in the backend, but not well-described in the API at the moment, in part because I haven't figured out how I want to do pagination yet. I think the max entries it will return at once is either 20 or 50? Because yes, that'd be a terrible idea to return ALL entries.

Ok, let leave it the way it is for now. I hope there would be pagination in future, because it would be good for backuping purposes, at least.

2) That is a good idea! I hadn't actually considered if we store entries differently depending on the editor format, and I should check if we do.

I thought about it more, and what do you think about joining all content representation in one structure. Like:

"body" : {
"content" : "Some *text*",
"type" : "markdown",
"rendered" : "Some <i>text</i>"
}

Or may be a bit more complex structure, if there can be more than two representation of the body.

So even in the old entry page, you can use the hacky way to get markdown, which iirc is to put '!markdown' on it's own line as the very first line of the entry, but you can also turn on the beta entry page at /beta (so http://www.momiji.hack.dreamwidth.net/beta on my dreamhack, for instance).

My beta (http://www.nataraj.hack.dreamwidth.net/beta) is empty:

We'll announce more testing opportunities in our official communities. Thank you.

There are no features currently available for beta testing.

That is why I asked. May be some options should be turned on in the config...

There is a LOT of code, unfortunately, which makes it hard to kind of give a tour, but for API stuff specifically, the routes are defined by a combo of yaml files in /api, Perl endpoint controllers in /cgi-bin/DW/Controller/API/REST, and TO_JSON methods in various object classes that define the JSON print format for that object. For entries specifically, the TO_JSON method is in /cgi-bin/LJ/Entry.pm

I'll try to explore the code. It will be low priority, but at least I will try to look at the code before asking questions :-)

Edited 2022-08-20 18:58 (UTC)

Ok, let leave it the way it is for now. I hope there would be pagination in future, because it would be good for backuping purposes, at least.

Well, you can kind of hack pagination by setting the offset parameter, but it's not the most intuitive way of doing it, I agree.

thought about it more, and what do you think about joining all content representation in one structure.

Hmmm. It might be a possibility, but you'll only get the raw content and the format type for entries you have permission to edit. So if you're grabbing entries you don't have permission to edit, you only get the rendered body, and a body object with only one key might look a little silly? I don't know.

My beta (http://www.nataraj.hack.dreamwidth.net/beta) is empty

Oh huh! This is probably a 'I set my dreamhack up ages ago and forgot everything I tweaked' problem, my bad. Looking through the configs, you can uncomment this section in your config-local.pl file and restart apache, I think!

First I did not manage to find framework that allow to wrap json into Moose object fast and simple. So I decided to write one myself. It took me a while, as I am totaly not Moose expert, but now I have early prototype:

https://gitlab.com/dhyannataraj/libwebservice-dreamwidth-perl/-/blob/master/lib/MooseX/Embody/JSON.pm

And I can create Objects from JSON, just by listing it's attributes, like that:

package WebService::DreamWidth::Entry;

use Moose;
use MooseX::Embody::JSON;
use WebService::DreamWidth::Icon;

j_obj "icon" =>(class => "WebService::DreamWidth::Icon", is =>'ro');

j_attr "security" => (is => 'ro');
j_attr "body_html"=> (is => 'ro');
j_attr "subject_raw"=> (is => 'ro');
j_attr "subject_html"=> (is => 'ro');
j_attr "allowmask"=> (is => 'ro');
j_attr "body_raw"=> (is => 'ro');
j_attr "icon_keyword"=> (is => 'ro');
j_attr "entry_id"=> (is => 'ro');
j_attr "datetime"=> (is => 'ro');
j_attr "url" => (is => 'ro');

https://gitlab.com/dhyannataraj/libwebservice-dreamwidth-perl/-/blob/master/lib/WebService/DreamWidth/Entry.pm

This will allow fast creation of prototypes, and then allow adding complex features using Moose's powerful tools.

Edited 2022-09-03 19:39 (UTC)

I have looked at the json returned by /journals/{username}/entries/{entry_id} and /users/{username}

I would expect poster hash from entry, to be subset of hash returned by /users/{username}. poster should have minimal information, that's right, but I expect it to use same keys that is used in full info from /users/{username}. This will allow to make extra attributes loadable on demand.

So I would expect user entry to have "username" instead of "name", as it is username in all the rest cases. And I would expect consistency with "display_name". It should exist in both cases, or does not exist at all...

Oh, excellent points, I'll add that to my to-do list! I really need to sit down and set up reference handling for the spec file parser, so that we can have reusable definitions for stuff like entries, users, icons, etc - at the moment every endpoint spec has to have that information in it, which leads to a lot of inconsistency.

I also miss some basic info, that can be fetched about journal. Like journal title. It would be very handy for testing...

So I can do

if ($client->user("nataraj")->journals->{'nataraj'}->title) eq "Nataraj's cool journal")

and check that WebService::Dreamwidth::Journal objects hash have been properly created by journals method.

Some more notes about consistency:

/users/{username}/icons/ returns full list of "icons" objects, but you can also access single one via /users/{username}/icons/{picid}

/journals/{username}/tags returns full list of "tags" objects, but you can't get certain "tag" object by /journals/{username}/tags/{tagid} (Though I would like to have this API-call while creating tags method for entry)

/users/{username}/journals just lists journal names.

I am not sure what is the REST API tradition, but I would expect that I can guess what I am going to get, by looking at API URL:

Like: /users/{username}/icons/ will return lists of IDs, /users/{username}/icons/1 will return list icon object with ID==1, /users/{username}/icons/* will return full list of icon objects. Or something like that I am not fixed on exact implementation, and do not know what do current tradition of REST APIs prescribes here... Key point is to see what will be returned in each case

What I should get via journals/{username}/accesslists? I am trying to do

curl -H "Authorization: Bearer UJxroElRIqdhgRVYS7y4h6DrFGPP5Wcz" http://www.nataraj.hack.dreamwidth.net/api/v1/journals/nataraj/accesslists

this auth key belongs to nataraj user. It has mutual access and subscription with nataraj2 user. But I still get empty list. How should it work? What should I do to get non-empty list?

Edited 2022-10-30 17:48 (UTC)

Ah, the implementation on that one is broken I think, because I started writing it and then realized I needed to figure out the bitmask arithmetic that backs access filters. For the future, you can set up access filters here: https://www.dreamwidth.org/manage/circle/editfilters and that response should return information about them (which you need if you want to set one as an entry filter). 'accesslist' is maybe not the best wording - we call them access filters on the editing page for them, and 'access groups' on the newer entry creation page.

Most of the objects returned by API has all "coordinates" that is needed to locate that object.

Like icon has "username" and "picid", and this is all you need, to fetch this object again.

Meanwhile "tag" object does not have "journalname" attribute, so you do not have all you need to fetch this object again.

would be also nice to have journal_name in the entry object

Good point on both of those, will make a note of it!

would be also nice to have journal_name in the enrty object

Pagination:

I have 3 entries in my test journal: "Entry 1", "Entry 2" and "Entry 3"

curl -H "Authorization: Bearer UJxroElRIqdhgRVYS7y4h6DrFGPP5Wcz" http://www.nataraj.hack.dreamwidth.net/api/v1/journals/nataraj/entries

Fetches them in backward order: "Entry 3", "Entry 2", "Entry 1"

curl -H "Authorization: Bearer UJxroElRIqdhgRVYS7y4h6DrFGPP5Wcz" http://www.nataraj.hack.dreamwidth.net/api/v1/journals/nataraj/entries?count=1

Returns "Entry 3"

curl -H "Authorization: Bearer UJxroElRIqdhgRVYS7y4h6DrFGPP5Wcz" http://www.nataraj.hack.dreamwidth.net/api/v1/journals/nataraj/entries?count=1\&offset=0

Also returns "Entry 3"

But if I change offset to 1

curl -H "Authorization: Bearer UJxroElRIqdhgRVYS7y4h6DrFGPP5Wcz" http://www.nataraj.hack.dreamwidth.net/api/v1/journals/nataraj/entries?count=1\&offset=1

I am getting "Entry 1" and I guess, this is totally wrong

Edited 2022-11-03 19:18 (UTC)

And speaking about pagination, I guess this offset story is not the best...

If I am fetching some long community, and meanwhile somebody added a new entry, I will get double entry. And if somebody delete entry from the top, I will miss one entry

I guess that all paginated search results should also have some metainfo telling how to fetch next page of the same query...

For DW it may be "next_id": you get the first page, it says "next_id=xxx" in the result data, you query same query with "&start_from=xxx" and getting the next page. Changing any record (except one with id xxx) will not change search result sanity. If looks like more stable...

I also have seen systems that has "search tocken", each search request have been cached, and you can paginate through it using tocken, you've got from the first request. But this was heavy commercial systems, do not think it is DW's case.

Edited 2022-11-03 21:24 (UTC)

Yeah, this is one of the things that I know other people have come up with solutions for, but I need to sit down and research them some.

And you're right, that return order behavior does seem like a bug.

Flat | Top-Level Comments Only

API things

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

Let me report some progress

And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback

Re: And API question/feedback