dw_dev | Question thread #48

How likely are y'all to need help with the database(s)? I don't know any Perl yet, but I'm in a database development program, and I'd like to help with the skills I do have, if I can.

I'm going to guess most of the major DB stuff is pretty well sorted at this point, BUT I'm getting to the point of actually being able to do something with my new search interface for /customize, and part of that is going to involve tinkering around with the s2 layer tables to allow two (or more) levels of categorization. Right now there's nothing grouping style categories together (like, say, a group of categories, one for each color). My super-alpha proof of concept uses a space in category name as a splitter and does the grouping in the code (so ("Color Red", "Color Pink", "Color Green") becomes ("Color" => {"Red", "Pink", "Green" })) which is incredibly hacky and likely to cause problems no matter what seperator I pick down the line. If you're interested in helping with that, I'm happy to fill you in!

http://wiki.dwscoalition.org/notes/Bot_Policy asks that clients rate-limit their requests.
Is there a suggested request rate, as a default to build into client software?

Apologies if this isn't the right place to ask; if not, where would be better?

I was hoping to find the answer to your question in https://www.dreamwidth.org/site/bot but that page doesn't say much more. I'm going to go on a limb and say once per second is OK. Mark or Denise or anyone who knows can contradict me if they disagree.

Depends on what you're doing, but 1 QPS is probably pretty reasonable for most purposes.

The initial application is backup. So:
- one bulk download the first time round
- subsequent intermittent, (usually) smaller updates
- not much user expectation of responsiveness

But the API library is separated from the user-facing functionality so may support other functions, in a neutral way, in the future.

Has there ever been any further effort toward moving the API toward something that could be usable for someone to develop iOS/Android/WindowsPhone apps? I understand that this is the major hangup in that area, but also that a lot of folks could benefit from a mobile solution.

Hasn't really been progress. I think it's a fantastic idea though and people have talked about it, just needs someone to take up the torch and push it forward. I'm happy to mentor/review.

(And if nobody does do it, it's on my 2017 list. I really would love to see what people can do given a better API!)

Has a design for a new, better API been come up with yet? Or are there any notes anywhere on what people have in mind?

ETA: Ooh, I have been pointed at http://dw-dev.dreamwidth.org/154266.html and http://dw-dev.dreamwidth.org/155046.html .

So a new metaquestion: Is there are place where I should know to go looking for discussion and design documents on DW development projects?

Edited 2017-01-08 01:12 (UTC)

dw_dev for discussion, https://github.com/dreamwidth/dw-free for open projects/issues!

Out of curiosity, is revising the notification system hard because it poses hard-to-solve problems or because it just needs a lot of tedious grinding, or both, or other?

It's really complex and very abstracted out and hard for people to wrap their brains around, plus (for all its failings, and yea verily there are many) the existing system is pretty well battle-tested and we're wary of making huge changes to it (and also we have enough nebulously-defined, open-ended "let's modernize this thing!" projects going on and are really hesitant to add another), plus it's hard to set up a working notification system in a dev environment so the ability to hack on it is limited to people who like banging their head against fiddly little config bits, plus it's one of the most heavily used systems on the site and so it has major performance implications (so often we have to take the non-obvious but better-optimized route instead, which makes for harder-to-read-and-understand code sometimes, and means that changes have to be scrutinized really carefully for performance), plus there's a ton of stuff chucked in there to work around the fact that Email Is A Terrible Protocol, plus all of the above means there's a certain amount of "sod it, I'm just going to go fix something else instead" going on....

Is there something in particular you'd like to see changed? There are a few frequently requested things that we've deliberately chosen not to implement for one reason or another; if it's one of those I can explain why. (Alternately, it's possible that what's bugging you is a possible fix and I can open an issue for it.)

Edited 2017-01-09 19:40 (UTC)

That one has to have inbox notifications on for anything one wants email notifications of.

That, and I'm just curious. I'm glad I asked if only because your answer was interesting. (If anybody doesn't have anything better to do than indulge my curiosity and wants to tell me about the specific tradeoffs that had to be made for performance, I'm interested in a scalability-case-study sort of way. Not because I have any agenda for changing things, but because knowing examples of this sort of thing makes one wiser about development.)

ETA: Also, when I was explaining to someone about how notifications work, I was struck by how the notifications system is both pervasive and extraordinarily customizable. That combination – pervasive + highly user customizable – is often deeply terrible to implement, for a whole host of reasons, including performance. Pretty much any time anybody twitches on DW, a whole cascade of logic has to be checked to see who needs to be told about it. So it occurred to me to wonder how DW does it, and how DW feels about its solution.

Edited 2017-01-09 20:05 (UTC)

Aha! That is one of those "frequently asked for changes that we haven't done yet because Reasons" things -- back in 200-mumble when the notification system was added, it was coded originally to consider the inbox notification the default version and all the other notif methods the copy, and so it counts on there always being an inbox notification object. It would require a vast overhaul to make the change. When LJ changed it a while back, it was a huge changeset, and it was done after we'd diverged enough that we couldn't just adopt it wholesale; it would have been just as much effort to merge it as rewrite it ourselves, especially since they had more resources to throw at performance inefficiencies and so weren't as diligent about optimization as we had to be at the time.

Then, every few years I start thinking "we should go ahead and do that ridiculous amount of work and make email-only notifications possible", and something promptly happens like half the email providers on the internet deciding that our email is spam and stop allowing us to email their accountholders YET AGAIN, at which point I look at the whole thing, am reminded that email is a Terrible Horrible No-Good Very Bad Protocol, and decide that it's not worth doing that ridiculous amount of work if it means losing the on-site backup record of unsent email notifications that people who are missing notifications can consult. (Like, yeah, if it were only a little bit of work I'd say we should go ahead and do it and anyone who turned off inbox notifs would just have to live with missing/delayed email when it happened, but the combination of it being a terrible horrible amount of work plus no longer having the on-site backup for when email is being Terrible again means I keep deciding it's just not worth it. Especially since, again, we have a ton of Modernize This Thing Into Something That Was Created This Millennium projects, and the existing system does work, just not with optimal UX.)

Personally, I just use the delete-all button on my on-site inbox a lot. ;) (Or filter by entry and delete-all for that entry.) There are definitely a few more UI improvements we could do to make it easier to manage large numbers of inbox notifications, we're just hampered by not really having any UI specialists.

There are a lot of things that annoy people about the site where the answer is "because in 200-mumble when it was added, there were no best practices yet and/or the people who made it didn't understand the use case and/or the userbase came up with some new weird way to use it that nobody thought of yet, and to change it now would be a much bigger deal than you'd expect." I blow people's minds sometimes when I point out that our code doesn't date from 2009 when we opened, it dates from 1999 when Brad came up with the first version of LiveJournal, and therefore becomes old enough to be a legal adult in the US this year. (I have a talk I give about working on DW that's called "When Your Code Is Old Enough To Vote" that goes into some of the upsides and downsides of working with a webapp this old.)

There's a shitton of downsides to having used the LJ code when we started, definitely, but there were (and continue to be) enough upsides that I still think it was worth it. I just wish we had more designers and UI specialists and frontend gurus sometimes.

also, re: your edit, which came in while I was replying and I forgot to address in my response BECAUSE I'M RAMBLING TODAY BECAUSE IT'S BETTER THAN DEALING WITH THE UMPTEEN BILLION EMAILS STILL IN MY INBOX: highly customizable was the LJ theory for just about everything for a very long time and we try to keep it the DW theory as much as feasable, and "deeply terrible" is a good description of it. Not only performance issues, but UI/UX issues as well -- LJ/DW is so customizable that it's not intuitive for a lot of people and there are a ton of things that are difficult to explain. Conversely, however, the fact that it is so customizable is a major selling point for a lot of people, and removing or redesigning options/making things less customizable risks angering people or breaking edge-case uses -- and there are a lot of edge-case uses. (Like, most people only use 20% of the site features and options ... but everyone uses a different 20%.) Witness the uproar every time LJ makes radical changes to something...

I'm a lot more willing to support options and under-utilized features than

mark is (he always wants to rip things out, lol) but we've tried to do a lot of simplifying, and there's still a lot we could do.

DEATH TO EVERYTHING RIP IT ALL OUT

The code machete makes life better. It keeps us sane. Removing code removes bugs! Deleting things deletes security holes! It is all better!

Question thread #48

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject