denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Denise ([staff profile] denise) wrote in [site community profile] dw_dev2016-10-02 02:04 am
Entry tags:

Help poke at the new RTE for the beta update page!

[ profile] wohali has been working on issue 664 to add a RTE to the new beta update page, and is ready for more eyeballs to poke at it and try to hit weird/unusual use cases or things that could be improved!

If you have a few, please poke at the new RTE on [ profile] wohali's dreamhack. To test it:

* Go to and create an account.
* Go to to turn on the beta update page and to make a post with it.

The update page should default to RTE on the beta update page. To disable it, go to Settings and check "Completely disable the Rich Text Editor (use plain textarea only)". (Historically speaking, a lot of RTE glitches happened when switching from HTML mode to RTE mode, so it's probably worth testing that a bunch.) Accessibility help is available by pressing Alt+0 (zero) while the editor is focused.

If you run into a problem or want to give feedback, you can comment on this entry. If reporting a problem, please include a link to the entry you made so [ profile] wohali can see the markup!

[personal profile] swaldman 2016-10-02 06:53 am (UTC)(link)

However, think we have a problem with the interaction between the RTE and the "add br tags for linebreaks" functionality, and I don't think it's one that can be solved in a way that pleases absolutely everybody.

The more-elegant way that the RTE now deals with HTML that was entered in raw view means that it is now corrected and prettified, so that a round-trip HTML->RTE->HTML finishes with something different than when it started (paragraph tags are added, new lines and intents are added, and so forth).

But the system that converts newlines into < br > tags has no way to tell between meaningful newlines that were added by the user, and syntactically non-meaningful newlines that were added by the RTE the previous time around. The RTE will add < br > tags for newlines that it itself created, meaning that multiple round-trips can have really unexpected results. See this video for a demonstration:

This is A Problem IMHO, and I can't see any way of making it work intuitively for everybody with the currently desired behaviour, at least without very complex kludges. (this doesn't mean there isn't a way, just that I can't think of one)

A suggestion: Would it be acceptable to only add a < br > tag for a *double* newline, not a single one? This would require a change from those who use the auto-newlines feature (I confess I am not one of them), but it is a new behaviour that would be familiar to some from other markup languages (e.g. LaTeX, Markdown).

Line one

Line Two

would be changed to

Line one< br >
Line Two

or perhaps

Line one
< br >
Line Two

which would still be readable HTML but would remove the "already processed" double newline and avoid having it acted upon again, and I reckon (hopefully?) make multiple roundtrips between RTE and HTML modes a safe thing to do.
Edited 2016-10-02 06:58 (UTC)

[personal profile] swaldman 2016-10-02 04:09 pm (UTC)(link)
Er, actually, I imagine double-line-breaked blocks of text like that would probably end up in seperate < p > tag sets. But presumably people who use the auto-linebreak feature don't mind which tags are used to achieve it...
Edited 2016-10-02 16:09 (UTC)
phidari: (Default)

[personal profile] phidari 2016-10-02 06:17 pm (UTC)(link)
Same issue here. I think some of the problem comes from the fact that if you write a paragraph in the RTE, then switch to HTML, instead of:

it becomes this:

which I'm honestly not a fan of.
wohali: photograph of Joan (Default)

[personal profile] wohali 2016-10-02 07:13 pm (UTC)(link)
The whitespace adding functionality has the ability to independently toggle if newlines are added before open tags, after open tags, before close tags and after open tags. It sounds like you'd prefer before open tags and after close tags, but not for after open tags and before close tags, if I read you correctly? I'd like to have more input on this topic before I change the behaviour based on one comment.

The other issue is that to trigger the conversion behaviour described by Simon in the first post, you have to follow these steps: Switch to (or start in) HTML mode, clear the checkbox, enter some text with newlines, switch to the RTE mode, switch back to the HTML mode, and re-clear the checkbox. I don't understand why you'd re-clear the checkbox to add more newlines after having visited the RTE to convert all the newlines to breaks. Is this something people actually do?
phidari: (Default)

[personal profile] phidari 2016-10-02 07:16 pm (UTC)(link)
Normally I see paragraphs formatted like this in HTML, and I think it makes the most sense:

Is this something people actually do?

People do all sorts of silly things. It's not something I personally would do, but then I don't normally use RTEs when I can help it so I'm probably not the best person to ask.
wohali: photograph of Joan (Default)

[personal profile] wohali 2016-10-02 07:19 pm (UTC)(link)
>> Normally I see paragraphs formatted like this in HTML <<

Thanks for the data point! :) This is a simple change, and if there's more support for it I'm glad to make the alteration.

>> I don't normally use RTEs <<

I really want to hear more from "normal" users who expect to use an RTE more than power users who will find workarounds for (or completely disable) RTE roundtripping concerns. Sadly I'm not sure I'll find those on a dw_dev post :(

[personal profile] swaldman 2016-10-02 09:30 pm (UTC)(link)
FWIW, in normal usage I almost exclusively use the RTE. The only times I switch over are to add tags that the RTE doesn't know about (usually < blockquote >) or to fix things when it gets into a mess.

My instinct is that many of the current text-mode-users might switch over when we have a RTE that works reliably, because why would one bother writing HTML when there's an easier way... but IMBW, and certainly not all will abandon the raw mode :)
wohali: photograph of Joan (Default)

[personal profile] wohali 2016-10-03 12:13 am (UTC)(link)
Good news! The new RTE has a blockquote button. :)

[personal profile] swaldman 2016-10-04 07:34 am (UTC)(link)
Yay ☺️

[personal profile] swaldman 2016-10-02 09:26 pm (UTC)(link)
You don't actually have to do all that to have a rogue-newlines problem, although that's the way to get it from paragraph tags.

Here's a more natural use case:
1. Start writing a post in HTML mode, putting newlines where you want newlines (because that's apparently what people do)
2. After the first part of your post, you switch to RTE mode - maybe to add an image without typing the code, maybe for some other reason. You know that if you leave the box checked, your linebreaks are wiped out, so you uncheck the box. You're probably a bit irritated at having to do this every time, because the box won't remember your preference[1].
3. After adding the image you switch back to HTML mode. The linebreaks have had br tags added. Fair enough.
4. You write another section of your post, then want to add another image (or whatever). You uncheck the box and switch to the RTE. Now all the linebreaks in the first section have become *double* linebreaks.

And so forth - any workflow that involves switching back and forth gives the user a choice between losing all their linebreaks, if they leave the box checked, or producing gaps between lines that grow and grow, if they remember to uncheck it. We can't show a nice easy toggle button to switch that breaks things like this, IMHO.

[1] I'd actually argue that this is a bug in its own right - because how infuriating would it be to write a lengthy post, forget to uncheck a checkbox that keeps rechecking itself, and find that all of your text has been merged into one long paragraph, with no undo facility?
wohali: photograph of Joan (Default)

[personal profile] wohali 2016-10-03 10:31 pm (UTC)(link)
So here's the two issues with what you've said as I see it.

When someone goes from RTE to HTML mode, without special handling by the HTML writer code, all newlines are purged, semantic or not.. This is why I immediately have to convert all newlines to br tags the instant someone transitions from HTML to RTE mode - this protects the semantic newlines by transforming them into tags.

This is also why the box gets checked the instant you move into RTE mode. Because we insert non-original newlines when going back from RTE to HTML mode, if we preserve the state of your checkbox when returning to HTML mode, you'll get newlines where you don't expect them, say around newly-inserted paragraph or div tags. So sure, I can preserve the state of your checkbox but if you roundtrip HTML->RTE->HTML, leaving the checkbox unchecked upon submission will corrupt the result of the output as well.

If we shifted to double-new-lines as paragraph markers, I could preserve the state of the checkbox and we'd be ok, because the HTML writer would never insert two linebreaks in a row; these would only ever be represented by br tags, which I could reverse-translate back into double newlines if desired. But this only works if we change the standard.

But things are not all sunshine and roses with a double-newline-as-br standard. Remember that the DW backend preserves any newlines in entries verbatim and converts them at render time to newlines if the checkbox is not ticked, which is why submitting text with newlines in it in raw/HTML mode with the checkbox checked will preserve those newlines in rendered posts. We could have the new RTE default be to convert double-new-lines to paragraphs, but we'd probably want to still save the post with the checkbox ticked and double-newlines converted to br tags, meaning the backend's auto-newline-to-br-tag processing would never be triggered and the RTE itself would be responsible for ensuring proper HTML is saved into the backend. (We can't change the backend's linebreak processing without breaking the rendering of thousands (millions?) of old posts.)

This is a bit of a braindump, so if anything is unclear please ask away.

[personal profile] swaldman 2016-10-03 10:48 pm (UTC)(link)
Re the first half: Yes, I understand that, and that's why I don't think we can reach what I'd consider to be a good solution while supporting the single-newline-means-linebreak feature[1]. That feature forces you to have the page do user-unfriendly things, such as resetting that check box every time.

Re second half: Damn, I didn't realise that the old system did it at display time rather than submit time. That's awkward :-/ With that said, if we could find a way to make things work right within the RTE, I don't actually see a reason *not* to just add br tags at submit time and never set the backend newline-to-br bit again...
(editing old posts might be awkward, I guess. And there are probably other gotchas that I haven't thought of :-/)

Once again, your work on this is appreciated, and I'm sorry for bringing up problems!

[1] Please bear in mind that I have absolutely no authority around here; I just talk a lot. Just because I don't like soimething doesn't mean that it isn't acceptable to the powers that be!

[personal profile] swaldman 2016-10-04 07:40 am (UTC)(link)
Radical idea: if we have to keep single-newline-means-linebreak, then we don't offer that in conjunction with the RTE. Remove that option from the RTE and add the checkbox, working as it used to in setting a database flag, to the form that is shown if the "disable RTE completely" option is selected. If you want to use the "auto-format" thing, you don't get to use the RTE on the same post.

This does arguably mean removing a feature. I don't know how many used the newline-to-br functionality and round-tripped things in the old RTE. But it's another way to proceed without requiring user-unfriendly kludge. I don't think there's a way to make this work that won't make somebody unhappy...

Now I'm feeling that I'm the only voice on this issue, so I'm going to shut up for a while, and hope that others have opinions.....
Edited 2016-10-04 07:42 (UTC)

[personal profile] pinterface 2016-10-05 03:26 am (UTC)(link)

To use a colloquialism, I don't really have a dog in this fight, but: removing the auto-format checkbox when the RTE loads seems like a pretty reasonable compromise. The newlines-to-br functionality is basically a really low-grade RTE, so when there's a not-terrible RTE available it seems reasonable to let it subsume the functionality of the less-good one.

wohali: photograph of Joan (Default)

[personal profile] wohali 2016-10-02 07:15 pm (UTC)(link)
There was some discussion of this in issue 664 and I think the concept of changing the behaviour is dead for now, especially due to zorkian's comments on the topic. Still, I'm happy to see arguments for doing such, as the current behaviour is...for lack of a better term, problematic.

Also see my follow-up to [personal profile] phidari below. Is this how you're actually triggering the issue above? It seems a far-fetched workflow to me.

[personal profile] swaldman 2016-10-02 09:34 pm (UTC)(link)
That discussion was well over a year ago, in the context of a markdown editor, and didn't have any pressing reason not to follow current behaviour - except for Mark's preference for sticking to the Markdown standard, which he eventually put aside.

Maybe the idea of changing it is dead, but I think it's worthy of at least being raised again, since (at least in my view) it would now have significant advantages. (I *think* it would make the problem go away? or am I missing something?)
wohali: photograph of Joan (Default)

[personal profile] wohali 2016-10-03 12:14 am (UTC)(link)
Restricting br insertion to double newlines would likely fix the problem, yes.
wohali: photograph of Joan (Default)

[personal profile] wohali 2016-10-02 03:29 pm (UTC)(link)
Hi everyone!

[staff profile] denise thank you so much for posting this, I really appreciate the call to action.

Redesigning the Rich Text Editor (RTE) took a lot of hemming and hawing over some delicate, confusing design decisions, ones that are heavily constrained by history and expected behaviour.

If you're interested in a deeper dive into the work completed, I've written about these choices at length in my journal. I welcome follow-ups here or there.