dw_dev | Code tour: 2017-02-12 to 2017-04-01

No tricks for you today, just a code tour (a day late). Today, we have 41 total issues resolved, with an additional 16 blasts from the past, one duplicate, and one not ready for prime time. Most of these are not live on the site yet, but will be at the next code push.

Of note: If you were trying to log into LiveJournal with your Dreamwidth OpenID, go ahead and try again. That one should be working again already.

Contributors:

afuna,

alierak,

kareila,

rahaeli,

zorkian

And, on with the tour!

Github is complicated. Sometimes, one of our developers retires a very old branch or two that needed retiring, so items that were covered in code tours from years and years past come back to our attention. It's always good for the tour guide to check through and make sure that those were in fact covered by a tour, though!

Issue 447, Issue 448, Issue 457, Issue 459, Issue 460, Issue 462, Issue 464, Issue 538, Issue 630, Issue 764, Issue 857, Issue 913, Issue 916, Issue 918, Issue 934, and Issue 936 were all that kind of blast from the past.

Issue 1328: journal search not picking up some results
Category: Fixing what's broke, site search
Patch by:

kareila and

alierak
Description: Sometimes the site search engine doesn't work like it ought to. Intermittent problems are always hard to diagnose. Originally the answer was something like "let's send out the flying search ~~monkeys~~ naked goblin cats * again!", but then

alierak realized that it takes time for the naked goblin cats to go out and prowl through all the journals and come back with interesting tidbits, and any entries posted in that time should be included the next time the cats are released. In any case, there don't appear to have been any reports of missing search results since 2015, so presumably that fix worked.

Issue 1744: Replying to PMs from OpenID users should Just Work but Really Doesn't (pull request)
Category: Fixing what's broke, making things make sense, OpenID users are people too
Patch by:

kareila
Description:Hitting Reply in the inbox should just work, for everyone. Now it should.
Since Dreamwidth isn't the only website on this great planet of ours, people from places like Wordpress can comment, interact in communities, and send and receive private messages here too, if they sign in with OpenID.
One problem: the Dreamwidth inbox private message system does not know what to do with a username that looks like bradfitzpatrick.wordpress.com. Internally, an "ext_####" type username is used to identify these fine folks, but regular folks shouldn't have to know what that username is in order to send someone a private message.

Issue 1855: Save descriptions can give error message when there are no descriptions (pull request)
Category: Weird user-unfriendly errors, making things make sense, images!!
Patch by:

kareila
Description: Occasionally, some of the little bits of Javascript that are supposed to help with saving image descriptions see the wrong thing and start panicking. That was resulting in some very user-unfriendly errors.
This will put a little playpen around the Javascript, so even when it's very confused, it won't wander off into traffic and cause everyone to panic.

Issue 1907: strip off http/https when checking OpenID logins
Category: OpenID users are people too, making things make sense, let's make sure this never happens again
Patch by:

kareila
Description: If good ol' bradfitzpatrick.wordpress.com wants to log in with OpenID, Dreamwidth shouldn't care if he's using http:// or https:// when he enters his OpenID URL in Dreamwidth's OpenID login box. And now it won't. This bug report brought to us by a very confused OpenID user who now has two Dreamwidth OpenID accounts to contend with.

Issue 1925: add stats collection for resolve-extacct (pull request)
Category: behind-the-scenes, other site usernames are people (and communities) too
Patch by:

kareila
Description: There's some stuff. Now, when that stuff happens, there's a record of it.
So you can link to users on other sites in the <user name="" site=""> tag. For example,

bradfitz. For those folks using visual browsers, you may notice that the little user head next to Brad's name looks like a LiveJournal user head, and

suggestions looks like a community. That's because there's a little thing that goes and checks what type of account each external account happens to be!
And that's what gets logged here, whether that check succeeded or failed, perhaps if the external site was under attack by angry gnomes, or their servers were eaten by a very hungry data center cat. (Don't bring cats in a data center. The techs who run them may love them, but fur does not make a server happy.)

Issue 1931: Backend: deleting an image on /file/edit deletes the original but not the thumbnail (pull request)
Category: Making things make sense, delete means delete!, images!!!
Patch by:

kareila
Description: When you delete an image you've uploaded, you would expect that this image, and all traces of it, would vanish. But there were 200x200 thumbnails of larger images that weren't deleted. Now they will be.

Issue 1957: /support/search isn't linked anywhere (pull request)
Category: Technical support is important
Patch by:

kareila
Description: There are some tools that technical support volunteers use -- or can use, if they know where to find it! This makes a technical support search tool easier to use.

Issue 1972: embeds missing from RSS/Atom feeds? (pull request)
Category: Making things make sense, RSS feeds still exist, embeds
Patch by:

kareila
Description: Every now and then you may include an embedded item, such as a cute cat video on YouTube, in an entry. Your friend Mike who lives in Seattle now and never emails, but still reads your journal from time to time, might be using an RSS feed to keep up with what's going on.
Due to the way Dreamwidth handles embedded content, and the way the rest of the internet works, Mike wouldn't see that cat video unless using the data/rss?unfold_embed=1 argument in the reader. But who has time for any of that? Now the cat videos will show up as expected, even in the RSS feed.

Issue 1975: Live updates (pull request)
Category: potpourii, fixing what's broke, the most interesting man in the world
Patch by:

zorkian
Description: These code changes are already live on site, and needed fixing very, very quickly.
The importer was under heavier use with all our new users; welcome! That meant that the importer was ... surprise surprise, breaking more under the strain! Thanks to Mark, it will break less.
Occasionally, errors in the payment process could break things! Thanks to Mark, it will break less.

Issue 1977: change LF to CRLF in graph labels to display properly (pull request)
Category: Fixing what's broke
Patch by:

alierak
Description: Instead of a nonsense character, this graph will now display properly. (Where is the graph? I'm not actually sure. It must be somewhere.)

Issue 1978: re-enable stats graph printing (pull request)
Category: Fixing what's broke
Patch by:

kareila
Description: Oh! This graph! The graph wasn't actually displaying. Now it'll start to display again.

Issue 1980: fixes for media file access (pull request)
Category: Making things make sense, fixing what's broke, images!!!
Patch by:

kareila
Description: This un-breaks image display when crossposting to other sites, from a previous patch which broke it.

alierak: Well, exact page is kind of vague, it could be in that entry, on anybody's friends page.

mark: My dreams of nice, clean systems :(

karzilla: If you wanted a nice, clean system, you shouldn't have based it on LJ 😆

Issue 1981: HTML cleaner no longer autolinkifying URLs (pull request)
Category: Stand back! I know regular expressions!
Patch by:

kareila
Description: Perfectly fine URLs that were copy-pasted into entries were suddenly failing to auto-linkify.
The problem? Regular expressions (being used to decide where the line would wrap).
The solution? Regular expressions (slightly edited).

Issue 1983: fix uploading of non-factory, non-URL icons (pull request)
Category: Fixing what's broke, icons
Patch by:

kareila
Description: So the updates to the image hosting managed to break icon uploading. You can basically picture a camera as designed by B.S. Johnson.
Volunteer developers, if you happen to like modernizing code that does weird and terrible things, have we got a treat for you...

Issue 1986: Use of $# considered harmful (pull request)
Category: Making things make sense, back-end
Patch by:

zorkian
Description: When you're looking to see if a file even exists, you want to know: does that file exist, yes or no? You do not want to know, say, the location of that file.

Issue 1989: Whitelist Soundcloud embeds
Category: ???
Patch by: Probably not a bug?
Description: Soundcloud has been whitelisted for a while. Unfortunately, sometimes embeds from Soundcloud don't work. This is most likely caused by:
- grabbing the wrong embed code, because Soundcloud's user interface is perplexing
- blocking Flash, because of course the embeds use Flash.

Issue 1990: markdown parser shouldn't transform user tags inside hrefs (pull request)
Category: Making things make sense, Stand back! I know regular expressions!
Patch by:

kareila
Description: As many of us know, but maybe some of us didn't, you can use Markdown in Dreamwidth entries. Unfortunately, when you were using links with the @ symbol in them, the thing that handles Markdown was valiantly trying to stuff a username in the link. Oops.

~code machete break~

Sometimes, there's code sitting around that isn't used, isn't doing anyone any good, and is only an invitation for ~~dust and mice~~ bit-rot and hackers. When we find some, we know what to do: BRING FORTH

karzilla!

Issue 1915: ConfCheck (pull request)
Category: CODE MACHETE, behind-the-scenes
Patch by:

kareila
Description: CODE MACHETE

Issue 1988: remove null-op LJ/EventLogRecord.pm and submodules (pull request)
Category: CODE MACHETE
Patch by:

kareila
Description: CODE MACHETE

Issue 2024: Remove unused code for external_foaf_url (pull request)
Category: CODE MACHETE
Patch by:

kareila
Description: CODE MACHETE

Issue 2025: Remove unused code for mailencoding (pull request)
Category: CODE MACHETE
Patch by:

kareila
Description: CODE MACHETE

Issue 2031: Remove unused code for browselang (pull request)
Category: CODE MACHETE
Patch by:

kareila
Description: CODE MACHETE
Once upon a time on LiveJournal, there was an attempt to offer the site in as many languages as there were translators. This attempt was noble! ... This attempt was riddled with hilarious wrongness, as the features of the site crept ahead of the translations, some things were never translated, and some changes went un-translated for years. In this day of automated translation, sometimes an inexplicable but up-to-date translation is better than a translation that nobody has touched for several years.

~end code machete break~

Issue 1991: Post-push log cleanup (pull request)
Category: Making things make sense, back-end, more logs less garbage
Patch by:

kareila
Description: A system log full of bogus garbage that doesn't really mean very much is not a very useful system log. Enter

karzilla. Less bogus garbage in the logs!

Issue 1992: entry loses slug on edit if poster and journal are on different clusters (pull request)
Category: Fixing what's broke, making things make sense, communities for the people
Patch by:

kareila
Description: In the newspaper universe, the short phrase that's used to describe the contents of a story is called a "slug". This terminology has made its way to the internet.
You can use slugs on Dreamwidth entries. However, if you were using these in community entries, and you and the community were on different server clusters, the server would do exactly the wrong thing with the slug when you edited the entry. Now the server will start doing the right thing.

Issue 2003: additional headers for media file requests (pull request)
Category: Our cloud is the best cloud, images!!!
Patch by:

kareila
Description: Now that images are getting used more, it's time to make sure that our CDN (Cloudflare) does the right thing with them: doesn't make you download the same image a zillion times, but also lets you load the image correctly if you don't have it.

Issue 2009: Manage Circle page should not show accounts you've banned unless you specifically ask it to
Category: Banhammer is a way of life, circle management
Patch by:

afuna
Description:

denise says it better than me:

If you've banned someone who's subscribed to you/given you access, their username won't show in the "subscribes to you" and "has given access to you" sections on your profile, but it will show to you on the Manage Circle page. We don't want to remove that entirely, because it can be a safety/personal-security issue for people to know for sure that someone is definitely reading their public entries. (Yes, it's not possible to say with confidence that someone isn't reading their entries -- there's always the "log out first and then directly visit journal", but if someone's subscribed to you, you know that they're reading your public entries.) Likewise, though, a lot of people ban people so they don't have to see That Failweasel's username staring at them every time they view their profile, and it would be nice if they could load their circle management page without That Failweasel's username staring them right in the face as well.

So, there will now be a place to view all the spammers, failweasels, and other ne'er-do-wells who still read you or give you access, but otherwise you won't be bothered with seeing their name when managing the other folks in your circle.
Note: even if you still give them access and subscribe to them, they will still be in their own separate area. Probably worth checking every few months, to make sure that you've not accidentally banned someone important to you, or haven't accidentally left That Failweasel with access to stuff you'd really rather not show them.

Issue 2010: Remove "ban community" link from hover menu (pull request)
Category: Banhammer is a way of life, communities for the people
Patch by:

rahaeli
Description: The option to ban a community on the hover menu was really, really confusing.
There is a valid case for doing this! If you are a member of the embarrassing_hobby community, and you don't want people looking at your profile to see that, you can ban the community. (You'll still show up on the profile page of the embarrassing_hobby community, of course.) The option is leaving the hover menu, but if you need to hide a community from your profile, you can use the admin console.

Issue 2012: Change 'people' to 'users' on /manage/banusers
Category: Banhammer is a way of life, site text
Patch by:

rahaeli
Description: Phrasing updates on the ban management page.

Issue 2013: Can't log in with your Dreamwidth OpenID on LiveJournal (pull request)
Category: OpenID goes both ways, ignoring the spec in favor of functionality, doing the ~~right~~ wrong thing, Oh LiveJournal
Patch by:

alierak
Description: LiveJournal's handling of OpenID logins is ... not quite right. So, in order to maintain functionality, Dreamwidth will (just for LiveJournal) do ... not quite the right thing, but in a way that works.

Issue 2016: log_event expects 'keys' to return a consistent order (pull request)
Category: Fixing what's broke, oh Perl
Patch by:

kareila
Description: In certain circumstances, Perl used to return some stuff in an easily predictable order. This is no longer the case. Unfortunately, some of our code was depending on that...
Since there are only two things to look for, we can fix this the easy way.

Issue 2019: convert manage/index.bml to TT (pull request)
Category: Back-end, Brad in his dorm room with BML, with a little bit of page update
Patch by:

kareila
Description: While

karzilla was updating the Manage Accounts page to include a link to the Bulk Edit Images page, she went ahead and removed some BML from the page.

Issue 2020: more cleanup (pull request)
Category: Back-end, small fixes, more logs less garbage
Patch by:

kareila
Description: Another month, another set of bogus warnings scooped out of the logs. Plus some extra spaces that didn't need to be around in the text.

Issue 2023: all instances of status_image in DW/Logic/ProfilePage.pm use HTTP requests, plus most seem dead or dying (pull request)
Category: HTTPS everywhere, chat statuses
Patch by:

kareila
Description: Once upon a time, it was very useful to be able to visit your friend's LiveJournal profile and see whether their ICQ account was online. It didn't really matter if that image was HTTP or HTTPS.
These days, your browser objects when you visit your friend's Dreamwidth profile using HTTPS, and all of the AOL, AIM, and other status icons are still HTTP... and a number of those chat services are not functioning the same way they did. So while you can still list your identities across a number of chat services, you'll have to look up your friends' chat statuses elsewhere.
*sprinkles from a 40-oz of sugar-Coke across a whole bunch of broken images*

Issue 2034: HTTPS_UPGRADE_REGEX needs modification (pull request)
Category: HTTPS is for everyone, Stand back! I know regular expressions!
Patch by:

kareila
Description: Photobucket, in its infinite wisdom, has decided to stop serving up HTTPS versions of its images. This has resulted in various chaos: when people try to load HTTPS versions of pages, Dreamwidth helpfully tries to retrieve HTTPS versions of Photobucket images, but Photobucket does not oblige.
This stops trying to squeeze HTTPS out of a HTTP stone.

Issue 2037: Image security settings persisting incorrectly and unhelpfully
Category: Duplicate, images!!!
Patch by: n/a
Description: A fix for this one isn't featured in this code tour. It was obnoxious enough that it got filed twice, though: if you change the image privacy to private and away again on the upload page, it'll stay private until you either upload a new image, or you save it on the Manage Images page. (You just have to hit save on Manage Images; you don't have to actually make any changes there.) So that's the work-around, for anyone bothered by this until it gets fixed. :\

Issue 2039: DW::CleanEmail::nonquoted_text not removing all quoted text (pull request)
Category: Oh Gmail, making things make sense, sometimes we can have nice things anyway
Patch by:

kareila
Description: This one needed a very specific combination of items to reproduce: plain text notification emails, gmail app, and replying by email. Dreamwidth tries to strip out quoted text and quoted email headers, to form a nice clean reply. Sometimes that attempt failed, and there would be stuff like

On Jan 11, 2017 6:18 AM, "DW Comment" < dw_null@dreamwidth.org> wrote:

A user replied to your Dreamwidth entry "test subject" ( http://testuser.dreamwidth.org/7718044.html ) in which you said:

in the reply. This was non-optimal. Turns out in all the reported cases, there was a space between the < and the dw_null@dreamwidth.org>, which the CleanEmail wasn't looking for. Now it's looking for that, too.

Issue 2047: remove %S2_TRUSTED and associated bits in etc/config.pl and cgi-bin/LJ/S2.pm (pull request)
Category: Oh LiveJournal,

whitaker and family, inexplicable things we found cleaning our basements, CODE MACHETE
Patch by:

rahaeli
Description: In the beginning, there was

brad. Then, as is ever the way with small groups, there was shortly Brad's buddy ... Brad. And he became

whitaker. A lot of weird stuff happened, and then S2 (the "new" way of styling journals) came along. If you have custom S2 layers, you may have noticed that if you try to put certain fun things in them (innocent scripts, not-so-innocent scripts), those fun things disappear. By that time,

whitaker was staff. For some reason that has been lost to time, it was deemed reasonable and good that

whitaker should be able to put any old thing in his S2 layers and have them Just Work. But, this isn't LJ, we don't have

whitaker, we've literally never used the feature and it was commented out, but in any case that's bad practice to have something like that lying around. So, out it goes!

Issue 2048: add "this is a RP account" settings flag (pull request)
Category: To the Roleplayers! 🍻
Patch by:

rahaeli
Description: When this goes live, roleplayers will be able to go into Settings and ticky a tickybox that says this is a roleplaying account. Right now, it will display a notice on the profile, to clear up any confusion. Later, it may also give access to features that only make sense for roleplaying accounts. (Roleplayers: this is perhaps a time to start thinking of such features and write them up to submit as suggestions.)

Issue 2050: Hide banned users on manage circle page (pull request)
Category: Github is hard
Patch by:

afuna
Description: This contribution was already included earlier in the code tour. But sometimes Github is hard for everyone to use, including seasoned developers who haven't touched Dreamwidth's code for a while. Welcome back,

fu!

Issue 2051: Cleanup 2017q1 **bonus round** (pull request)
Category: bitty wee cleanups, CODE MACHETE, moar logs fewer errors, 🍌📞
Patch by:

kareila
Description:

karzilla does a lot of tiny fixes that sometimes aren't worth the administrative trouble of submitting one by one. This way, she'll remember to check them in once a quarter. Includes:

@kareila Fix undefined value warning in DW/Controller/Search/Interests.pm
@kareila remove vestiges of phonepost support from LJ/Feed.pm
@kareila remove nonexistent endpoints from %LJ::AJAX_URI_MAP
@kareila remove unused module LJ/Widget/FriendInterests.pm
@kareila remove htdocs/tools/endpoints/extacct_auth.bml.text
@kareila remove siteadmin:entry_redirect from privilege list
@kareila try to fix weird href parse error in logs

zorkian: Phonepost?! Whaaaaat?

rahaeli: lol 2003 called they want their feature back

Issue 2056: Reflect RP account status in site stats
Category: To the Roleplayers! 🍻, not ready for prime time
Patch by: n/a
Description: It is not yet time to add roleplay account statistics to site statistics. This idea will wait.

Issue 2058: add a bunch of requested CSS classes (pull request)
Category: Styles, CSS, making things make sense
Patch by:

rahaeli
Description: It's nice when people making styles have the tools to make everything look the way they want, not just some things. Added to the list: Comment count on month pages, tag usage count, admin-hat regular entries in communities (as opposed to sticky entries, which already had that).

Issue 2060: Small change for filter option when posting an entry
Category: Making things make sense, soon to be superseded by beta
Patch by:

rahaeli
Description: If you're using the beta update page, this doesn't affect you. If you're using the main update page, instead of "Custom" security, you'll see "Access Filter", which should be clearer about what you're selecting.

And that's all for now, folks!

Flat | Top-Level Comments Only

Your code broke on the second cut.

It certainly did! And it's a non-obvious break, so I'm trying to track it down.

Yup! It seems to have been an extended conversation between Notepad++ and the HTML cleaner, where I got in the middle at the wrong point.

I am entertained by the story of Whitaker.

Whitaker was apparently quite the character! I believe that all jokes about Whitaker's Mom have now been stripped from the codebase. (D explained that Whitaker's default brain-404 noise was "Your mom!" which wound up getting turned around on him, and went some of the places that you might expect from a 2001-era tech startup with a lot of dudes...)

Didn't his mother also work for LJ at some point?

I too am entertained and maybe a little appalled by the story of Whitaker.

I should try and see if anybody still has video of a) Whitaker doing the Whitaker dance and/or b) bradfitz doing the Whitaker dance on his Segway and faceplanting...

Did anyone? Asking for a friend.

Thank you so much for doing this.

You're welcome! Code tours are a lot of work but I do like them.

This is a cool code tour. Here's teh page with graphs that need fixing: www.dreamwidth.org/stats/site.

Yay!! (I always get that link backwards.)

It's nice to have a superpower...

It certainly is...

Woohoo! Code tour! So many interesting things I can't list them all. Thanks, Azz!

Thanks for the tour!

Ha, the blast-from-the-past issues were all from me deleting old/no-longer-necessary branches (ie, branches that had been merged) for patches I'd done ages ago. GH apparently considers that 'new activity on the issue'...

also I love that you quote some of our bits of witty banter in the code tour so more people can see what utter dorks we are

If y'all make witty remarks, that means that I don't have to think of additional ones... :D

Code tour: 2017-02-12 to 2017-04-01

no subject

no subject

no subject

no subject

no subject

Stumbled on this looking for something else.

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject