crawling comments on a community?

[rax]

Hi!

I'm trying to crawl and parse comments on a community for a fandom event (http://hs-worldcup.dreamwidth.org , if you're curious). I've run into a bunch of issues and lack of API documentation, and talked to DW Support a couple of times, and feel like I am further away from successfully doing anything than when I started. Before I say anything else, here is What I Am Really Trying To Do:

• take an individual community post (example: http://hs-worldcup.dreamwidth.org/3493.html#comments) 
• download all of the comments with some sort of threading information --- the data I need in particular is comment subject, comment author, comment content, whether or not it's a reply and if so to what
• parse out that data and do transformations to it and add it to a database (which is not super relevant to this question I don't think but I can go into more detail if necessary)

I looked into the API for getting comments which led me in a roundabout way to www.livejournal.com/developer/exporting.bml . I'm probably missing something obvious here, but I don't actually see how this tells me how to make an API call? It gives me a GET request, but not what to send the GET request to? Also, support told me the only action DW supports here is "Get all comments for a community," not "Get all comments for a page," and I should probably just crawl the pages. Is that what other folks have done when doing this?

If that is what I should do, how do I get around the adult content warning? Is there a flag I can pass with the URL or something? Do I need to do something more complicated than just using curl to grab the pages? Is there something I can pass to say "just give me one piece of HTML with all 5000 comments on it it will be easier for both of us probably?"

Thank you for any suggestions or advice you might have.

Comments

[rax]

Oh! You have to use &auth_as=rax_test_comm but be authenticated elsewhere in your session as rax. That's... is there anywhere someone is working on re-documenting this? Still, that helps a lot, thank you!

[denise]

Check recent entries in this comm -- we're working on deprecating the existing API and replacing it with one that was, uh, conceived in the 21st century.

[rax]

I saw that. It looks awesome, but it's general enough that I have no idea how I would contribute other than to gesticulate wildly and say "the current state is really confusing and bad, and there is sure a lot of text about this new thing here, yup!" What I could do (and will if I have time) after figuring all of this belunkus out is write instructions on how to use what is currently there based on having wrestled with it.

[denise]

Well, my point is that with a bit of luck, the existing APIs will be deprecated within the next year or so and people who want to do things like what you're doing will be able to use the new API. So, while documentation is always a good thing, documentation of the XML-RPC API might be a bit of wasted effort!

Which is not to say "don't do it", just that it may be made obsolete fairly quickly.

[rax]

I'm happy for potential-future-me to have a potential-future-solution, but I want to make sure current-me has something they can use, too. :)

[mark]

I'm not aware of anybody actively working on documenting this better/at all. It's relatively little used; admittedly, possibly because it's mostly undocumented.

We're always happy to have people work on the docs. We've generally been using the wiki for documentation like this, and if you're interested in helping, I can point you places (or rather, I can ask some awesome folks to step in and help point you, since they are awesome and know the current taxonomy etc).