Technical debt

[denise]

If you aren't familiar with the term technical debt, it basically refers to decisions that organizations and software projects make to get things done now in exchange for having to fix it later -- borrowing time from the future, essentially, in order to get things accomplished now. Repaying your technical debt can involve a whole host of activity from code refactoring to cleanup work to system documentation, yadda.

kareila has been doing yeoman's work in making massive interest payments on the 10 or so years' worth of technical debt that the DW codebase has accrued, and we have a bunch of bugs open (why-cleanup, why-dev, why-optimization, and about 1/3 to 1/2 of why-usability) to repay some more of it. I thought it might be time for a group evaluation of our outstanding technical debt, though, and brainstorm ideas on what we can do to make more payments.

So, what other forms of technical debt do we have "on the balance sheets", so to speak?

Things I can think of off the top of my head:

* finish converting the whole site to TT
* better install docs
* better "so you want to admin a DW clone site" docs
* finish moving cgi-bin/lj*.pl files into proper modules (in cgi-bin/LJ)
* better in-code commenting (ideally each method in User.pm would have a comment explaining what it does and how to call it, for instance)

That's just an example list, though, and I'm sure I'm missing stuff! What bits of 'technical debt' have you noticed?

Comments

[shadowspar]

I don't know specifics, but I've heard it mentioned that much of the code could use more/better unit tests.

[yvi]

Or any at all :D

[sophie]

The code has quite a lot of unit tests, actually! We've been getting them up to scratch.

[foxfirefey]

We need to standardize all of the app site pages. They need to have standard classes for different elements that behave the same way, so they are easier to style for other people, and so each new page will almost never require any special color styling if it conforms to conventions, only some layout styling. It will greatly reduce the size of our site scheme CSS.

[foxfirefey]

PS: First link to bugzilla broken, think this one works though for cleanup.

Also: we need to move the heck on over to jQuery from all the older JS.

[dreamatdrew]

I have yet to find where there is a list of "This url starts getting generated in this file" directions. I WANT THIS. It's easier to fix things if we know where they are.

[foxfirefey]

I'm not finding the bugs for these yet but I am pretty sure they exist and are important:

* Redoing the support system
* Redoing the translation system

[denise]

There are bugs for both of them, yes, but I don't count those as technical debt -- that's usability.

[azurelunatic]

I know there's at least half a gdoc with some support brainstorming; I should ask Kat when she's going to have a working phone again.

[kareila]

Oh, that's what I've been doing?

I'm sure we need more tests, and a few of the current tests still don't work.

[thorfinn]

Code Style Automation.

I assume DW has a current Perl coding style, but I'm betting that enforcing it costs you a huge amount of review time.

Perl's biggest problem when it comes to maintainability (which is a huge factor in technical debt) is TMTOWTDI. Adopting Damian Conway's Perl Best Practices as The Coding Style pretty much stops that problem, because it gives you One Way To Do It, for pretty much everything you might want to argue about.

Implementing it as an incremental process on an existing codebase is not hard.... Create a script which runs perlcritic and perltidy over a file and reports errors, and make use of that script as the first thing to be done when checking out code and the last thing to be done before checking in code.

I promise you, that one thing alone will save a huge immense amount of time in code review, code style arguments, and massively improve the readability of your codebase alongside that.

[pne]

*nods* Even if programmers disagree with this or that coding style guideline, having a uniform guideline amongst all programmers makes maintenance much easier (especially maintenance by another programmer than the one who wrote the code in the first place).

[thorfinn]

Yep. And the best thing about Conway's PBP is that it is *so* comprehensive - it really covers everything you might want in a perl style guide and more... and that a huge proportion of its ruleset is implemented in Perl::Critic, so it can be automatically detected. You don't even need to buy the book (although I really do recommend it for anyone doing a significant amount of Perl) - just install perlcritic and get his .perltidyrc (which is well published), and use them.

Automatic code style enforcement is a massive win on human time - human based code style review is horrendously time consuming, and sufficiently so that it is extremely marginal in terms of net benefit on programmer time.

[sophie]

I suspect you're right with this. Our current coding style is documented at http://wiki.dwscoalition.org/notes/Programming_Guidelines . There are parts of it I don't agree with, but I still always make sure my patches are in that format.

[thorfinn]

*nodnod* I just took a look. That is a very very tiny amount of style guide, compared to what exists in the Perl Best Practices book and the resultant Perl::Critic/Perl::Tidy. :-)

That's the difficulty with developing your own style guide - there's so many possible things to nail down, and so many good reasons to do any of the different possibilities.

Just adopting the PBP wholesale as is rather than trying to debate possibilities simplifies the entire process immensely, and buys you the massive advantage of being able to completely automate the style checking.

[sophie]

Not quite, from our perspective. While we're moving away from it, slowly but surely, a lot of our pages still use BML, which mostly serve as glorified Perl scripts nowadays. But it does mean that BML files aren't standard Perl files, so they probably wouldn't be automatically checkable.

When the conversion to Template Toolkit is complete, though, we'll be able to do that. :D

[edit: However, even when it's done, bear in mind that there's 10 years' worth of code to change. Perl::Tidy would probably help a lot, but I'm sure there would still be bits that would need checking.]

[thorfinn]

*nod* I don't actually recommend implementing this as any kind of "do the whole lot of code" thing.

Just do it on files that are being changed - and doing it is surprisingly quick and easy. Pipe code through perltidy, which cleanly reformats it for you. Run perlcritic on the code, fix errors until all errors gone. Commit as "cleaned up code". Then start doing the real work. :-)

You're right that BML might be an issue, but as you say, conversion is under way.

I definitely think it would be a good idea to do perltidy/perlcritic as the conversion is happening. :-)

[pauamma]

PODified documentation embedded in each module.

[kareila]

There are still some English-stripping bugs open, to get all user-facing text into the translation system.

[azurelunatic]

Does breadcrumbing count?

[kareila]

I just thought of another one: exporting all the changed site strings back into the Mercurial repositories.