dw_dev | Protocols discussion

Entry tags:

xml-rpc

Protocols discussion

Somebody brought up the topic of Really Simple Discovery and the Metaweblog API in

dw_news, so now might be a good time to discuss protocols and our plans.

We currently have one bug to update the Atom publishing protocol and some bugs about updating our XML-RPC.

This is a post to collate discussion about:

1. What APIs do we currently support, and what state is our support for them in?
2. What APIs exist and what are their pros and cons? How popular are they with regards to services and clients that support them? How extensible are they, since we have unique posting metadata? Which ones would be a good fit to implement or improve? Where are good resources and documentation about them?
3. Where do we want to head with posting APIs, our overall goal?

Currently supported for posting

* Atom -- old version, needs upgrading desperately. We have a bug for this here. Once updated, we can start extending the protocol to support our custom features.
* Blogger -- had for a while, probably outdated.
* Flat API -- custom LJ flat API, destined for death.
* LJ XML-RPC -- custom LJ XML-RPC API, might be destined for death if we can adequately extend Atom

Current supported and not supported data stuff

* FOAF -- http://foxfirefey.livejournal.com/data/foaf
* Interests data -- http://www.dreamwidth.org/misc/interestdata.bml?user=foxfirefey
* OPML -- http://www.dreamwidth.org/tools/opml.bml?user=foxfirefey
* Icon data -- http://foxfirefey.dreamwidth.org/data/userpics
* Fdata -- Don't have this yet and whether we will is still up in the air; discussion on this here; issues include whether or not we are showing data that isn't shown on the profile

Flat | Top-Level Comments Only

Things I know we support right now:

1) Custom "flat" protocol mode, which is LJ specific, and should die.
2) Custom "XML-RPC" protocol mode, which is LJ specific, and should die.
3) Export comments XML feed, also LJ specific, death would be too good.
4) Export your journal as XML on the web site. WTF?
5) FOAF data for some profile information.
6) RSS 2.0 (I think) data for the journal feeds.
7) Atom (old old old) data for the journal feeds, as well as for writing posts.
8) fdata, intdata, etc?

I am sure we have some other ways of getting data out of the site. Feel free to point out any that I have missed, so we know precisely where to put the scalpel.

As to where we want to go? #1, I want standards compliance for interoperability. I want other sites to be able to access our data, and for us to be able to access theirs. If this means that we will drop the flat and XML-RPC protocols in favor of implementing (and extending as necessary) Atom, then that's what we should do.

I don't know what the state of the art is on protocols. That sounds like something that a guy like David Recordon would know, maybe we can ping him and get a summary. Or someone can do some leg work and research the various states-of-the-art.

I'm right there with you for standards compliance for interoperability.

It permits me to login to and use DW from my favorite web browser, which is Lynx (with SSL support).
Simplicity would be another and I've been seeing that in the subtle changes from LJ to DW code. :0)
XML can result in some gorram serious markup bloat and I'd stay away from it in terms of an API.

Above all, simplicity and interoperability for me, please.

/\__/\
(='.'=)
(")_(")

Music: God is an Astronaut - Remembrance Day

Edited 2009-06-27 23:14 (UTC)

As far as I know, every single major API I know of is XML based--that includes Atom, Metaweblog, Blogger's, WordPress's and Movable Type's.

Don't mind me. I'm just and old command-line junkie from back in the day.

Well, time for me to crawl out from my Unix shell and get with the times.

Please keep it as simple and as "lightweight" as you can.

What I'm seeing in terms of a "front-end" web developer are lots (and maybe lots more) of people using tablet and mobile devices that will call for simpler and "lighter" interfaces.

/\__/\
(='.'=)
(")_(")

Music: Snow Patrol - Lifeboats

Edited 2009-06-28 00:16 (UTC)

There are definitely improvements for making things lighter. For instance, there is a bug for adding handheld stylesheet support to the core of S2--that would mean that handhelds that supported using that stylesheet would use it instead of the regular one.

Coolness to the ghoulness!

I'd assumed that DW development folk would have something like in the works, but that we're still pretty new at making noise on port 80 and working out bugs and stuff, I didn't want to make a lot of noise about a handheld/mobile interface.

The main thing is to get (and keep) DW stable. I'm not about a lot of fancy doo-dads and whatnots, but rather a smaller set of features that work and can be depended upon to do so.

As long as DW remains advertising- and corporate-free, I'd like to contribute to DW development and maintenance as a volunteer. I know my way around a Unix shell and not too shabby at hacking HTML and XHTML with a propensity for doing away with unnecessary markup.

Like most g33ks of my generation, I can teach myself how to do things when pointed to the right documentation (read, "man page"). Is there anything that would be helpful for me to learn in order to become a DW volunteer?

/\__/\
(='.'=)
(")_(")

Music: firefly theme music

It sounds like some of your concerns align with the accessibility group--I'd check out

dw_accessibility with that in mind. The things that concern accessibility are generally the things that also make pages accessible in screen readers!

We have a list of current accessibility bugs here and

dw_dev_training can be a touchstone for starting development. There is a Getting Started wiki page as well.

Are you handy with CSS at all? Once handheld stylesheet abilities are added to the core, I think we'll be needing handheld stylesheets for the current 3 styles--Negatives, Zesty, and Tabula Rasa.

the "flat" protocol isn't. it's pretty awful but is widely supported by clients, including logjam, which is probably the most widely used client on gnu/linux.

LJ's flat protocol isn't a major API, though--I'm talking about APIs used by more than one service. It's one of LJ's main APIs, but it's not going to help us much. Any client that implements LJ's flat protocol would end up having to do a custom implementation of our flat protocol, because we'd have to change and update parts of it anyway with the WTF split.

Doesn't ljlogin use the LJ flat protocol when it logs in/out? No flat protocol on DW would mean no ljlogin on DW.

We have some Blogger support, I think, but no idea how well with whatevertheheck latest protocol.

I made OPML work for us, yay, although I don't know if it's to the latest specification.

We also have an Atom document describing icons a user has: http://foxfirefey.dreamwidth.org/data/userpics

It looks like the ATOM API is extensible, and so maybe that should be the first protocol to work on--figure out what extensions we will need and plan them and implement them.

FOAF can be extended, so I guess that could be a way to go for user information like FData, although I still hanker for a lightweight version for my own nefarious purposes (I'm pretty sure it would make for a faster network browser).

atom (both the feed format and the publishing protocol) is excellent. it's well-specified, flexible, extensible, RESTful, and with appropriate extensions could be used to provide programmatic access to more or less everything that currently has an api. atompub is useful for more than just posting entries -- there is a standards-track feed history extension that, if implemented, would make it easy to write journal backup tools, and imho atompub could be a good replacement for the fotobilder protocol if dw ever provides a photo-hosting service.

This is really encouraging to hear. Not knowing it very well myself, I've been concerned because everybody and their dog seems to roll their own even with standards available, and I was worried that they were doing it because standards like Atom were insufficient. But it looks like Atom might be able to do it all on its own with extensions, and that would be wonderful!

I was just looking at delicous-to-journaling tools, and they do have something that talks XML-RPC. I haven't gotten it to work with LJ yet, and I don't know anyone who has, they're all using scripts. If Dreamwidth can make publishing delicious listings reasonably simple, that would be quite nifty!

Do you have a link to it so I could read up and see if it had more information? Problem is, XML-RPC alone doesn't actually do anything on its own, and I'd need to know what specific type it is.

https://secure.delicious.com/settings/blogging/posting

(I am so glad that my several hours of trying to make this work with LJ happened at a convenient time for DW. Makes it less of a waste.)

Well, they don't say squat about the details, shame on them, but it looks like what they want is the Metaweblog API. Looks like several services do want it, so it might be worth to implement alongside the Atom so that we can have more services work with DW.

I'm nervous about committing to do something like that. I'm more a believer in doing one thing really well (supporting Atom 100%) than having 75% support for Atom, Metaweblog API, Blogger, ...

Of course, if Atom isn't right, that statement could say supporting Metaweblog 100%, and ... you get the idea.

What about supporting Atom 100% for our purposes and then supporting Metaweblog API just plain vanilla--enough for tools that only work with it and not Atom to be able to post? Tools like Delicious automatic posting or Posterous aren't going to know about fancy extensions we have or security extensions anyway; they just need to be able to post. And we can refuse to do anything on it until Atom is totally complete to our satisfaction.

A lot of my friends use Delicious scripts to get their bookmarks onto LJ, and that requires some seriously annoying hacking--getting it onto DW would be great, especially if it can be done without hacking and by converting whatever encoding they're outputting (we think it's western european) into UTF so posts are editable.

It'd be good if, for the simple purpose of allowing a client to post, we supported as many APIs as possible, as sometimes people are restricted on what they can install or use.

But full on editing, exporting and similar can be restricted to getting one decent API that works I suspect, as long as it looks fairly future proof.

Yes, they use the Metaweblog API. I have written many newsletter scripts to map the Del output to LJ's XML-RPC API so as to take advantage of Del's daily blog posting feature.

I've got a script that posts to DW at http://taktix.org/dw.php but the output is in an unsupported encoding and my server (changing next week finally) doesn't support the PHP I need to try to hack it.

Would definitely be good to get delicious output working properly here, but the encoding issue is a problem, really need to chase up my support request to them on that one.

Supporting MetaWeblog would be lovely. Definitely something I'd be in favour of.

How much of Metaweblog would need to be supported, though? I'm looking at it and so far I don't see the same abilities as in Atom for things like introspection. Right now I'm just thinking maybe support it enough to make posts--pretty easy--and Atom is our complete protocol.

Posting would be enough.

Frankly, anything that got rid of the horrible kludge of the delicious reposting PHP scripts would be great.

Flat | Top-Level Comments Only

Protocols discussion

Currently supported for posting

Other posting APIs

Current supported and not supported data stuff

no subject

standards compliance for interoperability

Re: standards compliance for interoperability

that's what I was afraid of

Re: that's what I was afraid of

adding handheld stylesheet support

Re: adding handheld stylesheet support

Re: standards compliance for interoperability

Re: standards compliance for interoperability

Re: standards compliance for interoperability

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject