DW authentication

[ari_linn]

Hello,

Sorry if my question is offtopic in this community, but it doesn't seem like your dev question thread is very active, so I'll take my chances and ask here. I'm trying to programmatically download DW posts from my journal and have troubles accessing friends-only and private posts. Suppose I have the following request:

GET https://ari-linn.dreamwidth.org/545896.html?format=light&expand_all=1 HTTP/1.1
Host: ari-linn.dreamwidth.org

It results in an error page: "You need to be logged in to see this content."

I thought about logging in programmatically to DW and grabbing cookies to pass with the request, but my attempts are getting me nowhere. I can successfully get challenge and compute response from this:

POST http://www.dreamwidth.org/interface/flat HTTP/1.1
Host: www.dreamwidth.org

mode=getchallenge&ver=1

But I can't login using the challenge-response pair. The following request is invariably giving me an error page: "There was an error processing your request: This account name was not found." Why it can't recognize the user=ari_linn parameter is honestly beyond me.

POST https://www.dreamwidth.org/login?ret=1 HTTP/1.1
Host: www.dreamwidth.org

chal=c0:some:very:complicated:challenge:here&response=md5(challenge_plus_md5(password))&user=ari_linn

Is there any way I can programmatically login to DW to access private posts?

Comments

[mark]

Your best bet is to take an existing client and modify it to do what you want. The other thread you commented on had a link to jbackup.pl client, I believe, which is a good starting point for a simple client that gets all comments/posts.

The /login page is designed for browsers, not programmatic access. You shouldn't be accessing it from code. You should use the XML-RPC method sessiongenerate to create an ljsession cookie which you then pass in through your requests to the comment export page (which is designed for bots to access).

But yeah, I'd definitely recommend taking an existing client and making it work. This protocol is over a decade old and doesn't conform to any modern expectations of what a protocol should be. In other words it's still pretty janky and the easiest path to success is probably just copying someone else's code.

Code link: https://github.com/dreamwidth/dw-free/blob/develop/src/jbackup/jbackup.pl

[ari_linn]

Well, I actually tried to go with ljsm (which, as far as I remember, couldn't download comments) and ljdump (from which I found out how to map comment poster id to poster name). I could successfully authenticate with the API and load posts, but I got totally stuck on mapping comments to their respective posts. I want to know which top-level comment belongs to which post, and there seems to be no way to know. I suspect jitemid in http://www.dreamwidth.org/export_comments?get=comment_body&startid=0 could be the answer, but it is jitemid="0" for all comments, so I don't see how it can be mapped. I tried to abandon the API and turn to actual rendered pages because with them I won't need to go through all the trouble of fetching comments at all, but alas! I can't login to see them.

Update: and in http://www.livejournal.com/export_comments.bml?get=comment_body&startid=0 jitemids are actual meaningful values...

[mark]

So I think the issue is that you imported your journal a long time ago (looks like 4 years ago, originally) and I think there was some sort of error at the time. ~3,000 comments got imported without an entry attachment so they're floating (jitemid=0).

I'd suggest that we clear out the imported content. This means I would take an admin action to purge the imported content (only imported content!) and then you re-initiate the import. This will get fresh copies of the data which should work. (These days the importer is pretty solid.)

What do you think?

[ari_linn]

I think if you can do something like (pseudocode) DELETE FROM comments WHERE user='ari_linn' AND jitemid = 0, then it might not be necessary for me to re-run the importer. I remember filing a support request several years ago after I discovered, much to my dismay, that I have many duplicate posts. The interesting feature of them was that one copy always had all of its comments imported, but the other always lacked comments, and the copy that lacked comments always had its public id (numbersnumbers.html) less than the one with comments. For example 257106.html is commentless, but 515532.html has 10 comments, just like the original entry. I remember support guys failed to figure out my problem and do something about it, so I had to keep all the duplicates in my journal. I've recently identified 500+ of these duplicate commentless entries and manually deleted about 250 of them. What I think is that comments with jitemid="0" are comments of these duplicate posts. Since there's probably no way to link them back to their respective posts, and I've deleted half of these posts anyway, it'll be the best if you could only delete comments with jitemid=0 for me. This way I'll just delete another 250 duplicate posts and go on with my blogging. You can always check if you're irrevocably deleting something useful by matching comment posterids and dates with the query like (in pseudocode):

SELECT * FROM comments jitemid_zeros
LEFT JOIN comments jitemid_normal ON jitemid_zeros.posterid = jitemid_normal.posterid AND jitemid_zeros.createddate = jitemid_normal.createddate
WHERE jitemid_zeros.user = 'ari_linn' AND jitemid_zeros.jitemid = 0 AND jitemid_normal.user = 'ari_linn' AND jitemid_normal.comment_id IS NULL

I'm not familiar with your actual db structure but imagine it to be something like this. If these jitemid_zeros are really just duplicates of successfully imported comments that are properly linked to posts, then you'll get empty query result because all the duplicates will be matched.

[mark]

For better or worse there's quite a bit more that goes into it. At any rate, I've taken care of it -- the "0" comments are gone now.

Also, we've now got pretty effective ability to detect and delete imported duplicates. If you want me to do so, I can. (Too be clear, this is just the lower numbered ones without comments.)

[ari_linn]

Thanks, I indeed stopped seeing comments with jitemid="0".

As for duplicate posts, I've killed them all myself. Thank you for addressing the issue though, hope it'll help you with the importer, and people won't get duplicates anymore (or at least support will have a tool to identify and remove them promptly).