Question thread #58

[pauamma]

It's time for another question thread!

The rules:

- You may ask any dev-related question you have in a comment. (It doesn't even need to be about Dreamwidth, although if it involves a language/library/framework/database Dreamwidth doesn't use, you will probably get answers pointing that out and suggesting a better place to ask.)
- You may also answer any question, using the guidelines given in To Answer, Or Not To Answer and in this comment thread.

Comments

[bytebuster]

Is there a way to retrieve cut-tag end-markers in an HTML-rendered post?

Background. I'm writing a small browser script to enable the "Repost" functionality, similar to what the LJ has.
Since this uses no LJ API, I'm solely relying on rendered HTML pages for DW posts.

Problem. I'm having trouble detecting cut-tag markers, when my script is run on the post page (it works fine in journal entries page). Here's the difference:

In Journal entries page it looks like:

<div class="entry-content">
  <!-- (pre-cut body) -->
  <span class="cuttag_container">
    <!-- (expand button) -->
    <div style="display: none;" id="div-cuttag_bytebuster_9782148_1" aria-live="assertive">
      <!-- (here goes the cut body if expanded) -->
    </div>
  </span>
  <!-- (post-cut body) -->
</div>

On a page for specific entry:

<div class="entry-content">
  <!-- (pre-cut body) -->
  <a name="cutid1">
  <!-- (here goes the cut body) -->
  <!-- (post-cut body) -->
</div>

Most obviously, in 2nd case I can't determine where the cut ends.

Question. Is there any way to work this around? What if the DW engine were placed some named anchor element at the cut end?

[pauamma]

What if the DW engine were placed some named anchor element at the cut end?

I think this might work (including for nested cuts), but I have no idea how big a change it would be.

[momijizukamori]

I don't think it'd be a huge change - it'd probably take more time just to track down where that bit is generated from.

Which I may end up doing anyway because this thread made me go 'oh, posts pulled via the API should probably also contain information about where cut start and end too'

[octorok]

Heya! I'm new here and have a fresh install of DW up and going on Linode. I've run make_system.pl once already to create the account and assigned it a random password, but now I want to change it. This is what I get when I try to run make_system.pl a second time:



And as a result I cannot log into the system account on the site at all with either password. I checked every dw_dev community I could find and consulted Google and couldn't find any answers. I hope someone here can help me because I've been trying to figure this out for two days straight. I'm relatively new to dev work so it's probably something really silly/minor but I cannot for the life of me figure it out on my own. Help? :(

[pauamma]

Welcome to Dreamwidth!

This looks normal to me. At a guess, you may have mistyped the password either when trying to log in as system or when entering it at the bin/upgrading/make_system.pl prompt. If you run bin/upgrading/make_system.pl again, you should be able to copy the password as you enter it, then paste it when logging in using your web browser. This should eliminate all mistype problems. If you still can't log in, come back to us and we'll dig in deeper.

Incidentally, you should not run as root at this stage, but as an unprivileged user of your choice. This is explained in http://wiki.dreamwidth.net/wiki/index.php/Dreamwidth_Scratch_Installation#Setting_up_the_DW_user_account .

[octorok]

Thank you for the welcome, and for replying so quickly! And thanks for correcting me on my rookie mistake. I ran make_system.pl again (not as root) and did as you said, and still can't login. I copied and pasted it and everything and still no go.

[pauamma]

OK, the next step is to look at the password as stored in your database, because it looks like somehow, it's storing something different from what you enter. Note that this will reveal the password for user "system". If you're not comfortable doing that for any reason, change it again (using bin/upgrading/make_system.pl) to something innocuous like "password", before proceeding.

At the shell prompt, type:

$LJHOME/bin/ljdb

You should get some verbiage including the words "Welcome to the MySQL monitor.", ending in a "mysql>" command prompt. At the "mysql>" prompt, type:

select id.userid, id.user, pw.password from useridmap id, password pw where id.user = "system" and id.userid = pw.userid;

Paste the output of that command in a comment. Then, type

quit;

to return to the shell prompt.

[octorok]

Here we go. I went ahead and changed it to 'password'.

[pauamma]

OK, that's good news. It looks like what you entered is what was stored. Are you able to log in with "password"? (Assuming you didn't change it again meanwhile.)

[octorok]

Oh no, I'm sorry, I forgot to mention that. Yeah, I tried logging in and it still tells me the password is incorrect.

[pauamma]

OK, let's try to rule out a mysql problem. Run bin/ljdb again, and at the "mysql>" prompt, type:

check table password, user, useridmap;
quit;

and paste the result as usual.

[octorok]

Sorry for the delay! Life is a hell of a thing. Anyway, here's what I get:

[pauamma]

OK, that's good. Do you have memcache set up on your server (or as part of your servers)?

[octorok]

I do not! Should I go set that up now?

[pauamma]

Not at this point. (And actually, you only need it for a production setup.) I was asking to check whether it could enter the picture.

OK, so it looks like you have a healthy database, as far as these 3 tables are concerned at least. The next step is to try figuring out where in the Dreamwidth code the password is getting corrupted, since it looks like it's not corrupted in the database. This will require running a modified version of make_system.pl. Are you familiar enough with git to juggle different versions of a file? (It's OK if you're not. I just don't want to talk down to you if you already know what you're doing.)

[octorok]

Unfortunately, I only have basic knowledge of git. I'm a quick learner though! :)

[pauamma]

Do you, by any chance, have a script-blocking add-on or something similar installed in your browser? If so, try whitelisting your site, force-reloading the login page and see if browser login works with whichever password you have currently set.

[octorok]

I went ahead and just disabled all of my plugins, reloaded chrome, tried to log in and still no go. :(

[pauamma]

OK. Another thing I'd like to try before I start looking pointedly at Dreamwidth code is using http://whatever.example.com/login?nojs=1 (with your domain name substituted for whatever.example.com) to log in. This will shortcircuit the javascript in the login page and do all the work in the back-end.

[octorok]

That didn't seem to work either. Same thing, says it's wrong password.

[pauamma]

OK

Step 1: update your installation as described in http://wiki.dreamwidth.net/wiki/index.php/Dev_Maintenance#Updating_the_Dreamwidth_code_on_your_Dreamhack , taking good note of http://wiki.dreamwidth.net/wiki/index.php/Dev_Maintenance#Additional_Information_for_Non-Dreamhack_Users.

Step 2: Create a branch to keep the modifications you'll make in step 3 away from the main development effort:

cd $LJHOME
git checkout develop
git checkout -b octorok-makesystem-changes

Step 3: Go to https://pastebin.com/raw/DwHfxPek and copy-paste the text into $LJHOME/bin/upgrading/make_system.pl, replacing the existing content. (Specifics depend on your preferred text editor and terminal emulation.)

Step 4: Commit yoour changes to the branch using:

git commit -a -m "Changes to try figuring out system login/password issue"

Step 5: Run the modified $LJHOME/bin/upgrading/make_system.pl and paste the output in a comment.

[octorok]

I got to #4 and get this:



I'm not really sure what to do here.

[pauamma]

Sorry, forgot about that. You need to choose an email address (one that you own) and name that you're willing to share with the world, and run the 2 commands git gave you, replacing you@example.com with the email address you chose and Your Name with the name you chose. Then, redo steps 4 and 5.

[octorok]

Okie dokie! Did all of that, here's what I got :)

[pauamma]

OK. It looks like something went wrong with steps 2 and 3 above, but I'm not sure what offhand. So let's try repairing it first, then we can redo what needs done.

Step a: back up your DW directory. That way, we can return to a known point if something goes wrong(er).

mkdir ~/dw.backup
cd $LJHOME
cp -r . ~/dw.backup

Step b: try repairing the damage.

git checkout develop
git reset --hard origin/develop
git branch -D octorok-makesystem-changes
git checkout -b octorok-makesystem-changes

Step c: retrieve and commit the modified file. (Watch out for wrapped lines in the first command below.)

wget -O $LJHOME/bin/upgrading/make_system.pl https://gist.githubusercontent.com/pauamma/16977a9d0114d1a5e3a6f7417423f22a/raw/8db3e6a81d60055a521bf0b3515d524e0586348f/gistfile1.txt
git commit -a -m "Changes to try figuring out system login/password issue"

Step d: run the modified file and paste the output in a comment.

$LJHOME/bin/upgrading/make_system.pl

[octorok]

I'm learning quite a bit here, thank you for being patient with me. :) Here's the output:



edit: I FORGOT TO WHITE OUT MY PASSWORD IN THE LAST COMMENT!! Eek. Will change it later, lol.

[pauamma]

OK. It looks like there's something wrong with your install, I suspect with the Digest::MD5 perl module. To figure out whether I'm right and how to fix it, I need to check a few things about your installation. First, what Linux distribution (name and version) you're using. Second, the output of the command

perl -v

And third, the output of the command

perl -e 'use strict; use warnings; use Digest::MD5; print "$Digest::MD5::VERSION\n";'

[octorok]

I'm using Ubuntu 17.10 on Linode. Second, here's the output of perl -v:



The third step doesn't look like it did anything, sadly.

[pauamma]

Actually, the single line of output of the 3rd was just what I wanted to know, the version of the Digest::MD5 module you had installed. But I'm still not 100% sure whether there's something wrong with your install or what, so I have more troubleshooting code for you to run.

git checkout octorok-makesystem-changes
wget -O $LJHOME/bin/upgrading/make_system.pl https://gist.githubusercontent.com/pauamma/16977a9d0114d1a5e3a6f7417423f22a/raw/d88f27e2a1d8e9e7b69d4956e326784866244dc6/gistfile1.txt
git commit -a -m "More checking for error conditions"
$LJHOME/bin/upgrading/make_system.pl

And as usual, paste the output in a comment.

[octorok]

Okay cool! Here's the output after following your steps:

[pauamma]

OK, at this point, I'm pretty sure something changed between Ubuntu 16.4 (that the Dreamhack host runs) and 17.10 (that you run) that broke Dreamwidth. I suspect the something that changed is mysql-related. With that in mind, what is the output of

dpkg-query -l '*mysql*'

on your machine?

[octorok]

Interesting. Here's the output this time:

[pauamma]

OK. What's in /etc/mysql/mysql.conf.d/mysqld.cnf ? (Note: it may be several screenfulls long. I'm specifically interested in whether it contains a "sql_mode = ''" line as described in http://wiki.dreamwidth.net/wiki/index.php/Dreamwidth_Scratch_Installation#Configuring_MySQL.2FMariaDB .)

[octorok]

Here we go:

[pauamma]

Just to be clear: did you just add this or was it already there?

[octorok]

It was already there. I accidentally hit a key while searching for it is all. :)

[octorok]

Hey! I think you may have found the problem? I created a new disk and installed 16.04 on it and am able to login! So maybe that was the issue? Either way thank you so much for your help!

[pauamma]

Thanks for your patience while we worked our way through the problem!

ETA: filed bug https://github.com/dreamwidth/dw-free/issues/2248