dw_dev | Code tour: 2017-10-28 to 2018-01-25

Entry tags:

code tour

Code tour: 2017-10-28 to 2018-01-25

The majority of these fixes are not yet live on the site, but will be coming to a Dreamwidth near you with the next code push! I've separated out the ones that are in fact already up-and-running. Special shout out to Pau for the epic bug-a-day run!

Issue 993: Detect misspellings of gmail, etc (pull request)
Category:
Patch by:

pauamma
Description: It's quite common for people to get alarmed and upset, when signing up an account, that their confirmation e-mail never arrived... because of a typo in the e-mail address, like "gmal.com", that it's then a pain to fix. This patch means that if an e-mail domain is entered (during sign-up, or at any other point) that is very similar to one of the top six e-mail providers, but isn't quite one of them, there'll now be a chirpy paperclip-reminiscent message that pops up to ask if you're sure you didn't mean gmail. This will hopefully reduce everyone's frustration!

Issue 1590: manage/tags should not silently truncate renamed tags (pull request)
Category:
Patch by:

pauamma
Description: ... because that's just rude: it's not very nice to cheerfully type in a tag that seems to work, save it, and then discover you're missing half a word of the end, usually in the most unhelpful and misleading fashion possible, Because Technology. The system will, going forward, error at the time, so you know it broke. The catch is that without some serious rewriting it was difficult to make it error in this case but not in others -- so you'll now also see errors if there are spaces at the beginning or end of the tag, for example, or if you included upper-case letters. If this causes problems (including annoyance), please do leave a comment or open a support request so thep atch can be revisited!

Issue 1627: Point sending form loses comment field after error (pull request)
Category:
Patch by:

pauamma
Description: Since Dreamwidth's inception, it's been the case that if you try to buy points for a user that doesn't exist -- perhaps, say, because your cat walked across the keyboard at a crucial moment and introduced a stray character to the username you were attempting to buy for -- you wouldn't find out about this until after you'd hit "Add to Order"... at which point the optional note for the recipient would be lost, gone, vanished forever into the wilds of the internet, and you'd have to reconstruct it from scratch. This is, as everyone who has ever lost a post or comment knows, Frustrating. This doesn't alert you to the non-existent username any sooner, but it does mean that if you hit "back" your note will be preserved, so it's an easier fix! This applies across non-shop contexts too, in theory.

Issue 1706: change the 'icons' info in the profile page blurb at top to include bonus icons count to logged-in users with bonus icons (pull request)
Category:
Patch by:

pauamma
Description: There's this thing, right, where Dreamwidth lets you buy bonus icon slots, which will be active while you have any kind of paid account. It's also possible to, by request, transfer these icon slots between accounts -- which is something that a lot of you RPers find pretty handy. However, as a result it was pretty easy to lose track of which account had how many of whichever bonus icons: this fix means that when you're logged in, there will be an explicit statement about how many icons you have in total, how many are active (i.e. you can use them), and how many bonus icon slots are associated with the account in question. This will hopefully reduce general confusion!

Issue 1785: Make member moderation easier to find (pull request)
Category:
Patch by:

pauamma
Description: ... because currently it isn't, and there's no reason to make community management any harder! There will therefore be a bounty of links to the community members moderation page from places other than the There's A New Member Application e-mail! Hurrah for links.

Issue 1818: The "ip" column in the userlog table should be widened to accommodate IPv6 addresses (pull request)
Category:
Patch by:

pauamma
Description: ... which reduces the number of error messages that show up in site logs when somebody deletes or undeletes an account from a (longer) ipv6 ip address, rather than a (shorter) ipv4 one. Fewer unhelpful error messages is good, because it makes it easier to spot and act on the important ones (and people less likely to ignore error messages). Hurrah for site reliability!

Issue 1984: investigate image proxy use for background images in journal layouts (pull request)
Category:
Patch by:

kareila
Description: One of the side-effects of the shift to using HTTPS is that content embedded using HTTP URLs will cause your browser to give a warning about "mixed content" -- about how, even though you're browsing over HTTPS, not everything on the page you're viewing is actually secure. The way Dreamwidth handles this is by feeding all HTTP images through an image proxy, which checks that the provided link is actually more-or-less safe in terms of malware and so on. However, when initially setting the proxy up, background images for journal styles weren't checked, which led to this warning appearing (alarmingly!) when people were viewing their own journals. What will happen now is that safety mechanisms will check for background images embedded using HTTP rather than HTTPS, and if they exist they'll get passed through the proxy, and fingers crossed everything will work as if by magic with an overall massive reduction in warnings.

Issue 2052: remove opt_nomodemail userprop (pull request)
Category:
Patch by:

pauamma
Description: A neatly-trimmed codebase is one it's easier to understand, to extend, and to find problems in! No more hours wasted on the question "what does this code even do?" before eventually finding out that the answer is "precisely nothing"! This is some code that's been hanging around cluttering the place up since before Dreamwidth existed. Some spring-cleaning has taken place.

Issue 2165: allow feeds to disable inclusion of comment count info (pull request)
Category:
Patch by:

pauamma
Description: ... because a lot of people, quite reasonably, want the RSS feed of their journals to be text only, or at least to not include (dynamic!) images that they didn't put in themselves. It will be possible to disabled the small X number of comments image! I... need to read the code and add this feature to the FAQ.

Issue 2169: newly uploaded images on /file/new should reflect correct security settings when user has a more restrictive minimum security setting (pull request)
Category: File/image hosting
Patch by:

pauamma
Description: Let's say you've set the minimum security of posts you make as access-only! This also, as if by magic, applies to photos you upload! ... except that the photos-you've-uploaded page then claims they're "public", even though... they're... access-only. Lying to one's users, even through oversight, is Not Best Practice; the actual visibiliy of images will in future be accurately reflected by the indicator.

Issue 2180: Wrong log in link when journal is NSFW (pull request)
Category:
Patch by:

pauamma
Description: It is nice when links point you to the right place. Broken links make nobody happy (except, possibly, the trolls that lurk beneath the link-bridges and eat the falling detritus). This patch gives us more accurate links.

Issue 2181: Incorrect number of links in Links List for premium/seed accounts (pull request)
Category:
Patch by:

pauamma
Description: It is Not Polite to tell people they get to have X of a thing and then have them discover, having paid for it, that they actually only get X/2 of that thing. Via some frankly heroic testing, premium/seed accounts now actually get the number of links the FAQ says they ought.

Issue 2190: add screening level option to post-by-email options (pull request)
Category:
Patch by:

pauamma
Description: ... such that if you're posting something you think might start a flamewar by e-mail, or indeed if you're just asking for people's addresses for care packages, you can screen all the comments to the post without having to actually visit the site and edit the post manually.

Issue 2200: can't load support requests from D&P users (pull request)
Category:
Patch by:

pauamma
Description: "D&P" is "deleted and purged", that is, the account has been completely removed from the site and the username is up for grabs again. There's lots of good reasons for this! However, volunteers sometimes want to search the Support Board to find pointers on how to answer a question -- for example, if something a bit tricky comes up only once every two years or so, it's not qui-i-i-ite worth adding to the FAQ but confirming how it has previously been answered can be Helpful in getting a (vaguely) timely response. Rather than show requests opened by D&P users in search results but then refusing to actually load them, because... er... of the helpful feature introduced last code push, or thereabouts, where information visible to support volunteers including "how much of the media quota has this account used?" was added. Which then goes a bit wrong if the media quota in question doesn't exist, because code is a Good Puppy and wants to Do What It's Told and freezes up really quite badly when given nonsensical instructions. Now this gets handled sensibly, and those requests load again. ARCHIVISTS REJOICE.

Issue 2201: allow specifying multiple access groups in post-security email header (pull request)
Category:
Patch by:

pauamma
Description: ... because you can show a post to multiple access groups if you're posting from the website, and it's nice to have post-by-email reflect that flexibility. There is the minor problem that there's basically no restrictions on what characters can be included in access group names. So how do you divide up a list of multiple access group names, where normally you'd use a comma, but access group names might themselves contain a comma? Well, er, in this case what happens is you get a site admin to do a quick bit of number crunching on site stats, determine that barely anyone actually uses commas in group names, brief the support volunteers that this might be going to lead to new support requests, and make absolutely sure that (i) if a group name in the middle of a list isn't recognised, it's skipped, but the rest of the list still gets applied; and (ii) if no group names are recognised, the post defaults to private rather than anything else less secure. In summary: overall usability improvement, with small potential for confusion and none for catastrophe!

Issue 2202: better error message when failing to provide birth year on account creation (pull request)
Category:
Patch by:

pauamma
Description: Currently, the sign-up page has a drop-down box for date, a drop-down box for month, and a lightly-outlined easy-to-miss free text box for year, on the grounds that "what year were you born" is not a trivially finite set of options in the same way that "day of month" and "actual month" are. This leads to a lot of rightly-frustrated users coming to support and going "I've ENTERED my birth date, it's VALID, what do you MEAN ' 'you must fill in a valid birth date'". The new error message highlights that the site needs a year too, thank you & apologies.

Issue 2204: Don't allow rename tokens to be bought for d&p accounts (pull request)
Category:
Patch by:

pauamma
Description: "D&P" is, again, "deleted and purged". Somebody whose account has been cast forever into the fiery pits of Mount Doom is not going to be able to use the rename token you've considerately and kindly bought for their account which doesn't exist any more, and this will especially annoy you if you were trying to rename to the D&P account. It can all be sorted out by pestering

denise, but it's simpler all round to prevent it happening in the first place.

Issue 2205: hook admin/renames on payments priv (pull request)
Category:
Patch by:

pauamma
Description: Dreamwidth set up a system in the interests of being potentially maximally helpful to other people using the code (by keeping "can you rename this account?" separate from "can you see what people have bought and how?"). In practice, this makes life more difficult for

jennifer, who has payment-related privileges that don't include, er, being able to rename accounts. It's useful to have someone other than

denise able to do that, for a wide variety of reasons, and so it shall be.

Issue 2217: add console command that gives information on which beta features a user is opted in to testing (pull request)
Category:
Patch by:

pauamma
Description: Last-ish code push, a feature was introduced that allowed support volunteers with certain privileges to see which beta features the person opening a support request had opted into, which was great! ... provided that the person opening the request (a) was doing so on behalf of themself, rather than on behalf of a friend, and (b) they opened it while logged into Dreamwidth rather than by e-mail. Sufficiently senior support volunteers will soon be able to find out which beta features any account is opted in to, which should help a fair bit with troubleshooting and generally helping people use the site more smoothly.

Issue 2219: more accurate text when displaying PM from suspended user (pull request)
Category:
Patch by:

hotlevel4
Description: It is often the case that spammers, surprisingly enough, spam. Which in this case means "send the same spam PM to a bunch of people before they're caught", which means that once they've been caught and their account has been suspended, the spam message shows up in the recipient's Dreamwidth Inbox as "Reply from suspended user"! Which it isn't! Because it was unsolicited spam! It will soon read "Message from suspended user" instead, which will hopefully be less confusing.

Issue 2222: change welcome email from address to the noreply address (pull request)
Category:
Patch by:

pauamma
Description: The Welcome To Dreamwidth e-mail has historically been sent from the "webmaster" e-mail address. If people replied to this e-mail, their response would end up in a private support category... along with a whole bunch of spam, making it very easy to overlook the actual real e-mail, and very likely that it would languish unloved for a very long time. To avoid this, in THE FUTURE e-mails will arrive from noreply@ with a helpful little explanation about the best ways to get in touch with any questions.

Issue 2223: Consider removing userprop dogeartime from bin/upgrading/proplists.dat (pull request)
Category:
Patch by:

pauamma
Description: It is unused and useless code! That means it is time for the CODE MACHETE.

Issue 2227: Remove AOL information from user profile (pull request)
Category:
Patch by:

srukle
Description: ... because AIM is no more, and you don't need your profiles cluttered up with defunct services. A moment of silence, if you will.

Issue 2232: Remove recentactions/actionhistory (pull request)
Category:
Patch by:

zorkian
Description: Coooooooooooooooode macheteeeeeeeeeeeee. (I am indeed using this code tour to procrastinate from the volcanoes I ought to be crunching data on. It has been some very effective procrastination. That doesn't mean I have an infinite capacity to be endlessly humourous about code that has passed through the valley of usefulness and been put gently out to pasture.)

Issue 2247: Incorrect tab order on old Create Entries page (pull request)
Category:
Patch by:

pauamma
Description: ... which meant that you could hit tab as many times as you liked, but you'd never get to change the actual time. This is suboptimal for people who do not use mice, for whatever reason, and has been corrected.

Issue 2255: Get rid of remaining references to S1 memcache keys (pull request)
Category:
Patch by:

pauamma
Description: Cooooooooooooooooooooode macheteeeeeeeeeeeeeee.

Issue 2258: update LJ::User methods in DW::Media to be methods on the DW::Media class instead (pull request)
Category:
Patch by:

pauamma
Description: Makes the codebase a little less hazardous to explore: less "balls of string" and "emergency kittens" and "if you don't hear from me in two hours...", more "actual filing system". How do you find out a thing? You ask the organiser of things! Instead of the counterintuitive orgaiser of the stuff.

Issue 2271: Stock etc/config-private.pl incorrectly refers to %BLOBSTORE instead of @BLOBSTORES (pull request)
Category:
Patch by:

pauamma
Description: ... it is really nice when code works because you've used the right punctuation and pluralisation. This is really only relevant if you were planning on having your own Dreamwidth install, for whatever reason.

Issue 2281: Do not include deleted & purged accounts in the Support High Scores Board (pull request)
Category:
Patch by:

pauamma
Description: ... because people who deleted their accounts sufficiently emphatically that they've subsequently been purged probably don't want that showing up all over the People Who've Spent Most Time Helping Out In Support page, on account of they thought the account didn't exist any more. Plus it's untidy and confusing. So: gratitude for the work, and also letting them get on with their lives!

Issue 2295: Cleanup 2017q2 (pull request)
Category:
Patch by:

kareila
Description: Lots of little corners of code that just needed neatening or dusting or putting back on the shelf THANK YOU CAT YOU HELPED. Everything is subtly better.

The following fixes are already live on the site, mostly as part of the rollout of HTTPS Everywhere, to increase your security while browsing Dreamwidth!

Issue 1119: Turn on HTTPS by default on all site pages
Category: HTTPS Everywhere v1
Patch by:
Description: This is already done, but the issue escaped closing! Hurrah for safer browsing: DW is no longer accessible over HTTP, which means your data is more secure.

Issue 2231: Fix adult interstitial (pull request)
Category:
Patch by:

zorkian
Description: Shortly after switching the site over to HTTPS everywhere, it became apparent that this lead to an infinite redirect loop when you clicked the "yes I'm really over 18" button, and an inability to get to the actual content you first thought of. This was corrected on the live site, because, er, whoops.

Issue 2233: Trying to access a private entry sends the user into a 302 Redirect loop (pull request)
Category:
Patch by:

alierak
Description: ... ye-ep, this too. Now you actually get a useful error message. Hurrah?

Issue 2241: December points bonus
Category:
Patch by:

kareila
Description: You get 10% extra points for any purchases made in the shop during December! December has been and gone. So has this, for the next... ten months.

Issue 2243: In FAQ 3, a sentence is missing a final period
Category:
Patch by:

kaberett
Description: It now has one(.)

Issue 2244: FAQ 103 and FAQ 155 differ on whether h4..h6 tags are allowed
Category:
Patch by:

kaberett
Description: They are! The FAQs now agree, and coincidentally but beneficially reflect reality.

Issue 2257: update copyright year in footer
Category:
Patch by:

denise
Description: HAPPY NEW GREGORIAN YEAR. Fun fact: I got sufficiently Annoyed by volunteers and staff collectively hitting, oooh, July, and only then realising that the year had changed, that I've now got a recurring reminder to myself to file this bug on New Year's Eve.

37 total issues resolved
Contributors:

alierak,

denise,

hotlevel4,

kaberett,

kareila,

pauamma,

srukle,

zorkian

Flat | Top-Level Comments Only

Yay code!

Whee!

Hurrah code, hurrah

kaberett, hurrah

pauamma because look at all of that code that was done by pau! I am quite sure that I am not the only one looking at all of it and thinking, 'I didn't know that I wanted all of that but now that I see it I feel a great satisfaction in my soul'.

Issue 2190: add screening level option to post-by-email options (pull request)
Issue 2201: allow specifying multiple access groups in post-security email header (pull request)

Excellent! I think I may have been one of the people nudging for these. Or maybe the only person nudging for these. How widely is post-via-email used?

When do we expect the next code push? No rush, just would like to know when.

I believe we're planning for sometime in the next week and a halfish! We'll post when we're sure, yadda.

*pours one out for AIM*

Look at all those lovely user experience fixes!! So excited.

I... need to read the code and add this feature to the FAQ.

Let me know if you want (or need) any help.

Thank you for all the bug squashing. :)

I would certainly be open to it -- I'm currently drowning in a sea of spreadsheets and am not doing great at switching headspace to code...

OK. So maybe something like:

By default, there's an image showing the comment counts appended to each entry in the feed. If you wish to remove that image, you can instead use:

https://(mumble).dreamwidth.org/data/rss?no_comment_count=1
https://(mumble).dreamwidth.org/data/atom?no_comment_count=1

Or, combining the two:

https://(mumble).dreamwidth.org/data/rss?tag=cake&no_comment_count=1
https://(mumble).dreamwidth.org/data/atom?tag=cake&no_comment_count=1

Note the use of & instead of ? before the second argument when there's more than one.

before the last sentence, where (mumble) is the magic phrase that renders as the username of whoever is reading the FAQ.

Edited (before != after) 2018-01-26 19:57 (UTC)

I'm so happy to see the HTTPS progress. Thank you all!

I'm currently one of the people working on the EFF's browser extension HTTPS Everywhere. I see

fu added a rule to the extension in 2015 (here's the current rule in the Atlas of rewrite rules). Any changes we ought to make to that rule?

The image proxy-related change reminds me: CloudFlare and Brave use our rules to do automatic rewriting of requests to upgrade from HTTP to HTTPS and keep users safer. (The first item on our architecture roadmap is actually "signed rulesets", a feature to allow EFF to sign rulesets and thus allow users and downstreams to easily download rulesets on their own, outside the extension's updating mechanisms.) Would Dreamwidth be interested in getting https_url to also draw from that set of rewrite rules?

Someday, HSTS! (In v2 or later, I infer.)

Thanks and yay again!

Re rewrite rules: That sounds like the same type of thing we tried with our KNOWN_HTTPS_SITES config (pr #2042) and I think it would be an excellent idea to use your list going forward. Would you mind writing up an implementation spec as a Github issue for further discussion and eventual development?

Sorry, I ran out of time while working on that project, but here is the issue I started!

(+) *goes a-renaming their 29 access lists, all of which used commas to separate some axes of access, so that they use slashes instead* *sigh*

*playful paw-shaking*

(I'm not actually upset, just amused)

*ponders axes (tool) of access… decides this is too silly for now*

... sorryyyyyyyyyyyyyy! This will only affect y'all if you post by e-mail and want to use the post-to-filters feature, but it sounds like you might? So at least I wrote up a good clear description now so that it doesn't come as a nasty surprise... >_>;

(+) We don't actually post-by-email currently, but we might in the future, so it's just futureproofing. I'm just amused that we fell into an edge case. ^..^

For what it's worth, I fell within the same edge case, although for only one access group. :-)

Re #993: ergh. I mean, yes, misspelled domains are a problem, but the reverse problem is that the lexical space for unweighted edit distance is crowded when names are that short. In particular, there are several FastMail domains that are an edit distance 2 or 3 away from “gmail.com” or “hotmail.com” but are not all that perceptually close. “fastmail.com” is itself 3 away from “hotmail.com”, and then there's “xsmail.com”, “150mail.com”, “16mail.com”—basically anything with three characters followed by “mail” or “tmail” will be a hit, or a large number of four-character .com domains, or…

I was about to say that false autocorrect positives for this are not critical but vaguely troubling in terms of turning the nuts on “if you use something that isn't Big and Common, you'll get weird looks and constant pushback from the trying-to-be-helpful”. But then I reread the patch and—is this actually a hard-reject? I hope I'm misreading that. Please be careful not to funnel people into Only Big Email Works any more than you have to (with dim awareness that you probably have to some).

This is categorically not a hard reject! I'm afraid I don't understand a lot of the details, but:

(1) 3 got picked as the magic constant, and
(2) The new line 547 in LJ::User::Message will print "You gave $email as your email address. Are you sure you didn't mean $common_domains[$nearest]?"

For what it's worth one of the e-mails I have associated with DW Stuff In General is my own domain -- I will definitely be glaring at my bosses if we get anywhere close to "you gotta have gmail to sign up a DW account" ;)

(−) Print and then return as an error so that the account creation fails, as far as I could tell—and having just thrown together a dev instance agrees: trying to add an account with a fastmail.com address fails where it succeeds on another domain. “Are you sure you didn't mean hotmail.com? [no ‘continue anyway’ option]” Could we ask that someone who Knows Better Than Us maybe recheck this before it goes live? (Some other code near there is theoretically suspect for the same reason, but this could increase the surface area of false rejects a lot.)

(+) (Meta: we may need to drop out of this thread for Reasons; feel free to reply publicly, but any replies-to-replies from us will likely be via PM.)

Edited (clarified socially ambiguous wording, eep) 2018-01-26 19:18 (UTC)

Confirming that I'm seeing the same results - and I reverted to before the commit and get the same behavior when using an email string that the previous filter matches (which was less of an issue because afaik none of the blacklisted string combos there were actual email providers)

https://github.com/dreamwidth/dw-free/issues/2304

Awesome, that's good news :)

See https://github.com/dreamwidth/dw-free/issues/2304. Thanks for the nudge.

Thank you so much for catching this, and apologies for having been mostly asleep! Am completely happy to relay PMs where they need to go.

error messages are love!

Flat | Top-Level Comments Only

Code tour: 2017-10-28 to 2018-01-25

no subject

no subject

no subject

no subject

no subject

no subject

no subject

Bug 2165

Re: Bug 2165

Re: Bug 2165

Re: Bug 2165

YAY! More HTTPS!

Re: YAY! More HTTPS!

Re: YAY! More HTTPS!

no subject

no subject

no subject

no subject

Email validation of doom?

Re: Email validation of doom?

Re: Email validation of doom?

Re: Email validation of doom?

Re: Email validation of doom?

Re: Email validation of doom?

Re: Email validation of doom?

Re: Email validation of doom?

no subject