Question thread #122

[pauamma]

It's time for another question thread!

The rules:

- You may ask any dev-related question you have in a comment. (It doesn't even need to be about Dreamwidth, although if it involves a language/library/framework/database Dreamwidth doesn't use, you will probably get answers pointing that out and suggesting a better place to ask.)
- You may also answer any question, using the guidelines given in To Answer, Or Not To Answer and in this comment thread.

Comments

[finalprogramme]

Novice barely-knows-how-to-code question here.

I'm trying to scrape comments from a community with an HTTPS call, using the URL:

https://www.dreamwidth.org/export_comments?get=comment_body&startid=0&authas=COMMNAME

This works if I'm logged in to the browser, no problem. But what I'm struggling with is how to send that request as part of a script. So for example, I put together what I thought would be a simple Python script:

import requests

from requests.auth import HTTPBasicAuth

auth=HTTPBasicAuth('USERNAME', 'APIKEY')

r = requests.get('https://www.dreamwidth.org/export_comments?get=comment_body&startid=0&authas=COMMNAME', auth=auth) print(r.text)

What I get back is the Dreamwidth login page. I'm sure there's something extremely obvious that I'm missing here, but dammit Jim, I'm a docs writer, not a developer.

[denise]

I threw this in our developer channel to see if anyone had any advice for you, but if you don't get any useful replies, maybe try a top-level post to get more attention to it!

[momijizukamori]

So the simplest way is going to just be to grab a logged in session browser cookie - I believe it's the ljuniq cookie, specifically. Then you can set that in the Requests object and it'll act as a logged in cookie session.

There's a way to auth through the old XML-RPC api but I would have to look it up - you have to pass an MD5-hashed version of your username and API key to a specific endpoint and you get an auth cookie back.