dw_dev | New XML-RPC Methods (editcircle, getcircle, gettrustgroups)

Entry tags:

xml-rpc

New XML-RPC Methods (editcircle, getcircle, gettrustgroups)

Back in April, and again in June,

catness did a fantastic job of writing new methods for the XML-RPC Protocol that replace the LJ-inherited friends methods. They provide ways of editing user relationships, both on the watch and trust edges, and ways to create/edit/delete access filters and subscription filters.

Not many people commented on the bug, though, so

catness didn't get a ton of feedback on the proposed modifications, and none at all as far as I can tell from other people who actually use the API to develop client applications. I recently began moving over the old LJ Server Manual's XML-RPC chapter into the DW wiki, so I read through the patch in an effort to put together some pages for the new getcircle, editcircle and gettrustgroups methods. When I looked through the code, though, I found myself thinking of a few things that should maybe be changed before the new methods get pushed live. Once an API starts getting used by client applications, it becomes much harder to modify, and I thought it was worth having another look before the new methods got put into use.

I first detailed my thoughts in a comment on the bug, but

denise suggested I open a discussion here, so I'm going to repost it all and hopefully provoke a few more people into looking over the patch and talking about whether anything would benefit from some tweaks.

It is worth noting here, as I did in my bug comment, that I am almost totally new to DW development, so maybe there's stuff going on that I am not fully grasping yet -- if that proves to be the case, please do not hesitate to set me straight!

The editcircle add key seems really awkward to me, as I attempt to document it. For one thing, it is called "add", but it is also the only way to remove watch and trust edges, so the add terminology is counter-intuitive. Probably "edit" is more appropriate. (Actually, I think that the trust group/content filter editing stuff should be moved out of editcircle and into a new editfilters method, then editcircle doesn't need an "add" or "edit" key at all since it will only be used for editing users' watch/trust edges, but that may be too radical a change. Ahahaha, too radical a change, she says! Like I do not proceed to go on and on about radical changes for the next hundred paragraphs.)

Also, though, if I'm reading this right, the way to add/remove the watch and trust edges is coded such that you can't just modify one or the other. If you want to, say, start trusting a user, you can't just add the trust edge -- you first have to find out if you already watch that user, and then supply the edge key with both with the new trust value and the current watch value. But then again, if you want to change the user's trust mask, then you can do that... but you can't also change their watch edge at the same time. Maybe instead of having a single "edge" key for watch/trust, we could have an "edittrust" key and and "editwatch" key?

I wonder a little about the lack of symmetry in terms of how trust groups and content filters are dealt with. I know that on the backend they are very different, but I feel those differences should be hidden as much as possible by the API. Maybe that's just me? Anyway, here's where things are different, from what I can tell:

getfriendgroups is deprecated, and gettrustgroups was added, so why not also have getcontentfilters? catness asked about this upthread, and mark said implementation of methods could be piecemeal, but since getcircle already implements the functionality for returning content filters, it would be minimal effort.

The includecontentfilters key of getcircle is like the includetrustgroups key, except that is also includes the list of users in the filter. Any reason we can't match that when we get trust groups? I know that it's more calls on the backend, but that doesn't seem like a deal-breaker?

The way a user's trust groups are set with the add key but a user's content filters are set with separate keys is probably not going to seem intuitive for devs using the API, even though it makes sense from an implementation perspective. catness brought this up in her initial description of the addtocontentfilters key: 'Maybe it's better not to make a special keyword for it but to have an additional keyword in "add"?' So yeah, that's one option, to move it into add in some fashion. Or maybe it's not even worth trying to bring it more into line with how we edit trust groups, since those use masks and content filters don't... Or maybe we should hide the whole mask aspect of trust groups from the API level anyway, and change the add key accordingly. So then you might have (going with my replacement of the edge key with editwatch and edittrust from (1) above):

username - the user to edit

editwatch - 0 to remove watch, 1 to add watch, or not included at all to leave as is

edittrust - 0 to remove trust, 1 to add trust, or not included at all to leave as is

addtocontentfilters - list of filters to add user to

deletefromcontentfilters - list of filters to remove user from

addtotrustgroups - list of trust groups to add user to

deletefromtrustgroups - list of trust groups to remove user from

Hmm, the groupmask key could also be kept around to facilitate LJ clients supporting DW. I don't know, I'm just spewing out all my thoughts now! Too many thoughts.

When you create a content filter, it is included in the API response, but the same is not true when you add a trust group. Presumably, this is because when you specify a non-existent id to edit_trust_groups, it creates a new one with that id so no return struct is needed to inform the client of the new group's id. In contrast, create_content_filter generates a new id for you, so you need to return that id to the client. Maybe it's not worth messing with that for symmetry's sake? But I want to float the idea of changing it so that if settrustgroups receives a non-existent id, it generates the new group's id for you and returns it in a struct, just like setcontentfilters.

This patch exposes terms like "edge" and "trust groups" but wouldn't it be better for the API to use terms that are closer to the front-end experience? I wouldn't expect every dev using the API to read the backend code, so they'll probably expect to see terminology more like "editaccess", "editsubscription", "setaccessfilters" and "setsubscriptionfilters". Or maybe I'm alone on this?

Threaded | Top-Level Comments Only

Why not removefromcontentfilters instead of deletefromcontentfilters and removefromtrustgroups instead of deletefromtrustgroups? After all you used remove in the description you gave, and delete tends to sound a bit drastic and doesn't fit as well as a mirror for add. create and delete, or add and remove make sense as pairs (at least to me).

I agree! Remove would be a better match for add than delete is. I will make a note of that, in case the consensus is that we go ahead with some changes to the methods.

All this sounds very interesting! I can't wait to see what comes out from this. :D

Maybe nothing will come of it! But I am hoping for some juicy discussion.

As a general note: I believe that the whole spec for that client interface started as an attempt to mimic the field names used on the LJ web form, which was IMO a bad idea in the long term, even without the WTF split on top of it. So I'm all for changing the API spec to something more rational, for DW clients. (Also, if you want to consider DW-LJ compatibility problems and discuss them, you may want to repost this to lj_dev on LJ. You may get a better response there.)

Should it be "access filters" and "subscription filters" rather than "watch", "trust", or "content" filters? Those are the names given in the front end now. Or are those terms purely front-end and easily changed, and we want to use the back-end terms for the sake of clone sites?

For new developers, it would probably help to be consistent, so change the code wherever possible to use the same terms as the site. This may need to be a more general DW bug.

Oh, that history of the protocol is good to know, thank you!

I imagine that we probably should consider DW-LJ compatibility issues, so I'll rework this a bit and post it to lj_dev this weekend sometime.

Thank you so much for the detailed and thoughtful feedback!! this is exactly what I had hoped for when starting to work on it, but yes, it's still not too late :)

There's a reason to have gettrustgroups as a separate method because it's the only one required for posting, so it's needed even by a minimal client which does only posting/editing. All the rest, including getcontentfilters, is required only for circle management, which is used much less frequently than posting, and not all the clients are required to implement it - the web interface is more or less adequate for these purposes. (As for me, I had never used friends' management even in LogJam, and only added it to my client because I run out of ideas what else to add :)

Indeed, the spec is stealing part of the terminology from LJ and part from DW backend. I'm used to think of the functions in terms of "trust" and "watch", but if the consensus is to replace them with "access" and "subscription", it's fine. Never realized there's a difference between remove and delete, but now I see it (hopefully).

I had thought the list of users in includecontentfilters to be a necessary evil, because they are not limited the same way as trusted users, so it's not possible to get them all in one mask. Don't know if it makes sense to include the list of users in gettrustgroups - the corresponding LJ function does not, it works with masks. Or we should not try to keep it compatible with LJ implementation at all, as long as the functions are called differently?

*needs to remember more of the code, and to think of the other comments*

Yeah, I don't have enough experience with the code yet to judge whether the front-end names are appropriate for the API or not. As a user of the API, though, I think the front-end terms are vastly preferable.

As to whether the back end code should reflect the site terms more consistently... that sounds like a big change!

Hi, I'm so glad you commented! It took me three days to write this response, mostly due to extreme amounts of waffling. I deleted entire paragraphs, and then put them back, and then deleted them again. /o\

There's a reason to have gettrustgroups as a separate method because it's the only one required for posting, so it's needed even by a minimal client which does only posting/editing.

Ah, when I look at it like that, it makes total sense as to why it got its own method and getcontentfilters didn't make the cut. (It is also now apparent that I am biased toward extracting information from DW when I use the API, and not at all familiar with using it to, say, make posts!)

Indeed, the spec is stealing part of the terminology from LJ and part from DW backend. I'm used to think of the functions in terms of "trust" and "watch", but if the consensus is to replace them with "access" and "subscription", it's fine.

So far, I'm not feeling that either set of terms has the advantage, consensus-wise.

I had thought the list of users in includecontentfilters to be a necessary evil, because they are not limited the same way as trusted users, so it's not possible to get them all in one mask. Don't know if it makes sense to include the list of users in gettrustgroups - the corresponding LJ function does not, it works with masks.

Oh! I hadn't noticed that fundamental difference between trust groups and content filters: you can look up a trust group to find out all the users in it or you can look up a user and get the trust groups it's in, but when it comes to content filters there doesn't seem to be a method for looking up a single user and getting the content filters it's in, you can only look up each content filter and retrieve all the users in it, and then iterate over those lists to find out which filters a single user is in. For some reason, I think I was under the impression that getcircle returned each user's content filters along with their trust mask, but that is definitely not the case, due to the above limitation! (I guess it made just made sense in my head -- when you retrieve information about a user in your circle you want to know if you trust/watch them, if they trust/watch you, what trust groups you've assigned them to, and what content filters you've assigned them to. It's just that we don't actually provide that last bit of info!)

I see what you mean now by necessary evil. :) My quest for symmetry seems hopeless! Oh well.

Or we should not try to keep it compatible with LJ implementation at all, as long as the functions are called differently?

Yeah, it's hard to know what to preserve for the sake of compatibility and what to let go of in the interest of having an API that makes sense for Dreamwidth. If I had designed these methods, I probably would have kept very closely to what was in the existing protocol, but as I am merely documenting, I feel quite free to suggest anything at all!

So what I am thinking now is that the only thing I would absolutely change is the add key of editcircle: rename the key (make it "edit" -- or maybe "set"? "editusers"? "setusers"?), separate the edge key into watch and trust keys, and put the addto/removefrom content filter keys in there:

editcircle

some key name here that isn't add Array of structs with keys:
username - the user to edit
editwatch - 0 to stop watching, 1 to start watching, or not included at all to leave as is
edittrust - 0 to stop trusting, 1 to start trusting, or not included at all to leave as is
addtocontentfilters - array of filter ids to add user to
removefromcontentfilters - array of filter ids to remove user from
groupmask - the mask to use for this user's trust groups
setcontentfilters - same as before
deletecontentfilters - same as before
settrustgroups - same as before
deletetrustgroups - same as before

Any thoughts on that? Pros/cons? I always feel like I am not seeing all sides of this code or something.

If I were being ambitious, I would also consider switching all back-end terms to front-end terms instead. But only if other people thought that made more sense, which I am not getting a good sense of right now.

Oh, oh, oh, but I forgot to ask in my post: your patch moves filter setting/deleting out of its own methods and into editcircle, which means that LJ clients who used to use editfriendgroups to change users' masks will have to adapt to a slightly different request structure for the DW API. Specifically, editfriendsgroups accepted a struct of friend ids to its groupmasks key (so the struct was { "friend id 1" => "new groupmask", "friend id 2" => "new groupmask" }). In the editcircle method the structure of the request is different (i.e. an array of user structs: [ {"username" => "name 1", "groupmask" => "new groupmask"}, {"username" => "name 2", "groupmask" => "new groupmask"}]). Do you think that will be a problem, LJ-compatibility-wise?

As I hinted in my post, I kind of find editcircle a bit overloaded. What are your thoughts on moving the filter methods out of editcircle and into their own methods and making them more like the LJ protocol, like so?

edittrustgroups

groupmasks Struct containing keys:
userid_x - value of each key is the new groupmask for that user

set Struct of structs
groupbit_x - value of each groupbit key is a struct containing keys:
- name
- public
- sort

delete Array of trust group ids to delete.

No, I, um, haven't considered how content filters would work yet vis-a-vis a parallel to that groupmasks key.

And after taking 3 days to craft my comment, I still forgot to ask: why does providing a groupmask to editcircle's add key mean that you can't simultaneously change the user's watch edge?

Hang on, you said there's no way of taking a user and looking up all the content filters they're in? But you do that every time you go to the edit subscription/access page. Does that mean that page is really inefficient?

Well, I am not exactly an expert in the codebase, having been working with it for... two weeks. :) But! As far as I can see, there is no method for "tell me all the content filters I've put so-and-so into". (I am sort of hoping I'm just not looking for it in the right place, though.)

That said, I don't think the edit subscription page is inefficient, because it doesn't actually let you look up all the content filters that you've put a user into. It lets you pick a single filter and see who is in it and who isn't, which is exactly in line with the backend method that, given a content filter id, looks up which users are in it.

If that method doesn't exist, we should probably create it, yes. *G*

htdocs/manage/circle/add.bml has a bit about getting the content filters a user's in, and it does look like it's looking it up based on the filters, not on the users that are in them. We might want to do something about that maybe.

Oh, so that was the page

cesy was talking about. Yeah, that looks less than ideally efficient.

No, not the general manage subscription filters page - this page.

Oops, yeah,

denise pointed me the the page you meant, and also confirmed that, indeed, it's a bit inefficient. Because there's no method for looking up the filters any given user is in, it looks at each of your filters' lists of users to see if the user you're adding/modifying is in any of them.

Hmm, I think that the way that the filters are stored, you really need to look into each filter, to see whether the user is in the filter. Because of caching, it's not as horrible as it sounds!

However, I would be strongly in favor of a convenience method that abstracts it away, so it can be used in more places *g*

Oh, yes, Storable! Is that what you're talking about? That all the users are packed into that data column and then need to be thawed out before you can even see who's in what?

I would def be in favour of a convenience method. Someone should do that!

Kinda, not quite! They're stored as bitmasks, so you need to do a bitwise operation against the bitmask *g*

Oh god, what? My Perl must be way worse than I thought. I now have no idea what you are even referring to. /o\ Thank god I am only documenting and not actually trying to actually code anything.

(Sorry for being so sluggish with responses, it takes time to get back into the programming mindframe for this project, hopefully it will get better :)

Ok, "edit" sounds like a good name, and I see now that it may be more easy for the programmer to edit watch and trust edges separately. The current implementation came from being too lazy to change my client too much, so it continues to use only one variable to determine the relationship :) but I agree that editwatch and edittrust are more intuitive.

editfriendsgroups accepted a struct of friend ids to its groupmasks key ...

Actually, my structure of request was copied from the LJ method editfriends, because I think groupmasks do not belong to editfriendgroups - if this function is called "edit friend groups", it should do just that, and not *also* optionally edit the user's circle. This is the way I had used it in my client anyway, don't know if the other LJ clients use editfriendgroups instead of editfriends as they're supposed to, but I think there are too many changes to talk of backward compatibility, so it's preferrable to do it logically on our end.

why does providing a groupmask to editcircle's add key mean that you can't simultaneously change the user's watch edge?

I guess it was the same consideration - editing group relationship is "different" from editing user relationship, so they're not supposed to be used together. But it's no problem to include both of these keywords.

switching all back-end terms to front-end terms instead.

It's also that "trust" and "watch" are shorter than "access" and "subscription", both are even the same length - so they sound a bit better in my mind :) But it's no problem to change, so the developers don't have to learn two sets of terms.

I kind of find editcircle a bit overloaded.

It was made symmetrical to "getcircle", because when the client includes the circle management functionality, I believe it makes more sense to get all the available data right away because it's all necessary - so it may just as well be done in one request rather than calling at least 3 functions one after another (trust groups, content filters and watched/trusted users). But then, the client would not edit all of them simultaneously, so splitting the edit part into at least 3 functions also makes sense, but then they're not symmetrical? I can't decide what's better :)

and thanks again!! *goes to think of the rest of the comments about content filters...*

I apparently don't even remember enough about Perl to actually understand the code I'm reading, so please, take all the time you need to get back into the mindset! I kind of sprung this on you all of a sudden, many months after you'd last given it any thought.

I think groupmasks do not belong to editfriendgroups

That's a fair point! I'd been thinking of editfriendsgroups as an edit not just of the name/sorting/etc of the group, but also of who is in the group, so the combination of functionality seemed justified to me, but I can also see your point about keeping them separate. Also, I like the idea of not duplicating functionality all over the place -- if you can edit a user's trustmask with editcircle, then LJ-compatibility alone is not a good enough reason to support that same functionality in a second method using a different request structure.

...editing group relationship is "different" from editing user relationship, so they're not supposed to be used together.

Oh! Interesting take on it. I'm used to thinking of all those things as going together, since on DW itself when you want to edit your relationship to a single user from the manage/circle/add page, you can change their watch status, trust status, trust groups and content filters all in one step.

It's also that "trust" and "watch" are shorter than "access" and "subscription", both are even the same length - so they sound a bit better in my mind :) But it's no problem to change, so the developers don't have to learn two sets of terms.

Bleh, I don't feel qualified to decide. I think if we're getting rid of "edge", most of my issues with the back end terms vanish, since watch and trust are pretty intuitively mapped to the front-end terms. Content filters is a bigger leap, but not an impossible one.

so it may just as well be done in one request rather than calling at least 3 functions one after another (trust groups, content filters and watched/trusted users). But then, the client would not edit all of them simultaneously, so splitting the edit part into at least 3 functions also makes sense, but then they're not symmetrical? I can't decide what's better :)

Yeah, see, I waffled a lot over that part! But then I decided to just go ahead and talk it out with you, because then at least we'll have thought it through from all the angles. Mostly when I was considering writing out the documentation for it I just sat back blinking and thought, wow, that is a lot for one method to do! This page is going to be huge! And have a lot of statements like, "If you set the such and such key, and then if something is changed like so, or something is created, then the return structure will possibly contain the following keys..."

Astonishingly, the lack of symmetry doesn't bother me so much here -- I know! How can that be, when I am all about symmetry everywhere else? But I figure if you add in a quick getcontentfilters method to mirror the gettrustgroups one, then having editcircle, edittrustgroups and editcontentfilters seems quite natural, and it becomes quite clear that the ability to get filters using getcircle is provided simply for convenience.

I am torn, though, between wanting simpler edit methods that have straightforward return structs, and wanting a way to, if the client developer desired, do a bunch of circle editing stuff all in one call, like create a trust group and add some users to it. Since I think a case can be made for both sides, we should just leave it as is. No point changing something if the change does not have a clear advantage over the existing code. The documenter will just have to suck it up and explain how all the different keys affect the possible return structures! :)

Also, more on content filters now that I've actually made one (before, they were sort of... theoretical to me). Why does the includecontentfilters key of getcircle only return the names of the users in the filters, and not any of the other data that might be associated to each user, like the tags or ratings to further filter their entries by? Was that a deliberate omission?

If you left the other stuff out for the purpose of keeping things simple right now but with plans to add expanded content filters support later, then maybe it would be better to leave content filters out entirely right now? That way client apps won't start using a method that's still in flux. Given the way the users in a filter are currently returned as a space-separated string, adding in the extra info will probably require a change to that, and that's the kind of API modification that has an impact on client apps.

Hm, given all this discussion, any thoughts on my disabling this protocol method for one more code push? that gives us time to shake things out.

Sorry, afuna! I went away on a holiday so my internet access is spotty -- just got this comment. After all our discussion, I think any future changes will not be super radical, so if you pushed it that's cool! And if you didn't, that is also cool. :D

Never apologize for taking a vacation! (I hope you had a great time ♥)

Live on the site now :-)

Threaded | Top-Level Comments Only

New XML-RPC Methods (editcircle, getcircle, gettrustgroups)

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject