Entry tags:
crawling comments on a community?
Hi!
I'm trying to crawl and parse comments on a community for a fandom event (http://hs-worldcup.dreamwidth.org , if you're curious). I've run into a bunch of issues and lack of API documentation, and talked to DW Support a couple of times, and feel like I am further away from successfully doing anything than when I started. Before I say anything else, here is What I Am Really Trying To Do:
If that is what I should do, how do I get around the adult content warning? Is there a flag I can pass with the URL or something? Do I need to do something more complicated than just using curl to grab the pages? Is there something I can pass to say "just give me one piece of HTML with all 5000 comments on it it will be easier for both of us probably?"
Thank you for any suggestions or advice you might have.
I'm trying to crawl and parse comments on a community for a fandom event (http://hs-worldcup.dreamwidth.org , if you're curious). I've run into a bunch of issues and lack of API documentation, and talked to DW Support a couple of times, and feel like I am further away from successfully doing anything than when I started. Before I say anything else, here is What I Am Really Trying To Do:
- take an individual community post (example: http://hs-worldcup.dreamwidth.org/3493.html#comments)
- download all of the comments with some sort of threading information --- the data I need in particular is comment subject, comment author, comment content, whether or not it's a reply and if so to what
- parse out that data and do transformations to it and add it to a database (which is not super relevant to this question I don't think but I can go into more detail if necessary)
If that is what I should do, how do I get around the adult content warning? Is there a flag I can pass with the URL or something? Do I need to do something more complicated than just using curl to grab the pages? Is there something I can pass to say "just give me one piece of HTML with all 5000 comments on it it will be easier for both of us probably?"
Thank you for any suggestions or advice you might have.
no subject
This API is a very proto-REST style API except it uses XML instead of JSON. You just send HTTP requests like a browser (with the same cookie authentication). There is a reference implementation available:
https://github.com/markpasc/jbackup/blob/master/jbackup.pl
You could also look at the Dreamwidth implementation of the content importer:
https://github.com/dreamwidth/dw-free/blob/develop/cgi-bin/DW/Worker/ContentImporter/LiveJournal/Comments.pm
I hope this is helpful!
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
(no subject)
no subject
https://bitbucket.org/foxfirefey/dwump/src
(no subject)
(no subject)
no subject
An example of that set up is this writing motivation comm: http://verbosity.dreamwidth.org/profile where graphs are automatically posted for their word goal.
And in case it's handy here is a script snakeling set up for having a web server automatically post to dreamwidth bookmarks from pinboard https://github.com/snakeling/PinToDW/blob/master/config.php
(And I've used gmail delay send to have comments / posts go out at specific times as practice for at across multiple time zones bingo sort of game.)
So that might be an alternate path to go down.