Technical debt

[denise]

If you aren't familiar with the term technical debt, it basically refers to decisions that organizations and software projects make to get things done now in exchange for having to fix it later -- borrowing time from the future, essentially, in order to get things accomplished now. Repaying your technical debt can involve a whole host of activity from code refactoring to cleanup work to system documentation, yadda.

kareila has been doing yeoman's work in making massive interest payments on the 10 or so years' worth of technical debt that the DW codebase has accrued, and we have a bunch of bugs open (why-cleanup, why-dev, why-optimization, and about 1/3 to 1/2 of why-usability) to repay some more of it. I thought it might be time for a group evaluation of our outstanding technical debt, though, and brainstorm ideas on what we can do to make more payments.

So, what other forms of technical debt do we have "on the balance sheets", so to speak?

Things I can think of off the top of my head:

* finish converting the whole site to TT
* better install docs
* better "so you want to admin a DW clone site" docs
* finish moving cgi-bin/lj*.pl files into proper modules (in cgi-bin/LJ)
* better in-code commenting (ideally each method in User.pm would have a comment explaining what it does and how to call it, for instance)

That's just an example list, though, and I'm sure I'm missing stuff! What bits of 'technical debt' have you noticed?

Comments

[thorfinn]

Code Style Automation.

I assume DW has a current Perl coding style, but I'm betting that enforcing it costs you a huge amount of review time.

Perl's biggest problem when it comes to maintainability (which is a huge factor in technical debt) is TMTOWTDI. Adopting Damian Conway's Perl Best Practices as The Coding Style pretty much stops that problem, because it gives you One Way To Do It, for pretty much everything you might want to argue about.

Implementing it as an incremental process on an existing codebase is not hard.... Create a script which runs perlcritic and perltidy over a file and reports errors, and make use of that script as the first thing to be done when checking out code and the last thing to be done before checking in code.

I promise you, that one thing alone will save a huge immense amount of time in code review, code style arguments, and massively improve the readability of your codebase alongside that.

[pne]

*nods* Even if programmers disagree with this or that coding style guideline, having a uniform guideline amongst all programmers makes maintenance much easier (especially maintenance by another programmer than the one who wrote the code in the first place).

[thorfinn]

Yep. And the best thing about Conway's PBP is that it is *so* comprehensive - it really covers everything you might want in a perl style guide and more... and that a huge proportion of its ruleset is implemented in Perl::Critic, so it can be automatically detected. You don't even need to buy the book (although I really do recommend it for anyone doing a significant amount of Perl) - just install perlcritic and get his .perltidyrc (which is well published), and use them.

Automatic code style enforcement is a massive win on human time - human based code style review is horrendously time consuming, and sufficiently so that it is extremely marginal in terms of net benefit on programmer time.

[sophie]

I suspect you're right with this. Our current coding style is documented at http://wiki.dwscoalition.org/notes/Programming_Guidelines . There are parts of it I don't agree with, but I still always make sure my patches are in that format.

[thorfinn]

*nodnod* I just took a look. That is a very very tiny amount of style guide, compared to what exists in the Perl Best Practices book and the resultant Perl::Critic/Perl::Tidy. :-)

That's the difficulty with developing your own style guide - there's so many possible things to nail down, and so many good reasons to do any of the different possibilities.

Just adopting the PBP wholesale as is rather than trying to debate possibilities simplifies the entire process immensely, and buys you the massive advantage of being able to completely automate the style checking.

[sophie]

Not quite, from our perspective. While we're moving away from it, slowly but surely, a lot of our pages still use BML, which mostly serve as glorified Perl scripts nowadays. But it does mean that BML files aren't standard Perl files, so they probably wouldn't be automatically checkable.

When the conversion to Template Toolkit is complete, though, we'll be able to do that. :D

[edit: However, even when it's done, bear in mind that there's 10 years' worth of code to change. Perl::Tidy would probably help a lot, but I'm sure there would still be bits that would need checking.]

[thorfinn]

*nod* I don't actually recommend implementing this as any kind of "do the whole lot of code" thing.

Just do it on files that are being changed - and doing it is surprisingly quick and easy. Pipe code through perltidy, which cleanly reformats it for you. Run perlcritic on the code, fix errors until all errors gone. Commit as "cleaned up code". Then start doing the real work. :-)

You're right that BML might be an issue, but as you say, conversion is under way.

I definitely think it would be a good idea to do perltidy/perlcritic as the conversion is happening. :-)