Process documentation needed: automated test suite

[denise]

crschmidt pointed out in his bug walkthrough that the process of running the automated test suite is not very well documented! And I agree; I know I don't run the tests when I'm patching things since I don't really know much about it, and I don't contribute tests along with my code because I actually don't know how to write them or what they should do. I am probably not the only one! (The only thing I know about the test suite is that you shouldn't run it in production because it does weird and wonky things to the database.)

I do know that our test suite coverage is not extensive at all, and we've talked before about moving to a more test-driven devlopment mindset. If anybody wanted to write up some documentation on the Wiki about both how to run the tests (and what you should look for) and how to write new tests (and when you should), that would be awesome and I would love you forever. (Well, I mean, I love you all forever already. But I'd love you more forever.)

Comments

[exor674]

I think we really need to fix our tests to not corrupt the database before we start using them more. Because I will not be running the tests ( or only running select tests ) until they can be run safely -- especially after the next time I reset my database.

( forex: momji did NOT post this post on my dev env: http://momijizukamori.dwdev.andreanall.com/ )

[denise]

I maded a bug for that: http://bugs.dwscoalition.org/show_bug.cgi?id=4439

[sophie]

This is only vaguely related, but I think I might try to see if I can get automated daily testing for the Daily Snapshot 'hack. I've made a test run (hah!) at http://hack.dreamwidth.net/dev/test/index.html (warning: The page doesn't seem like it's very accessible for the blind).

I'd be curious to see if people think it's worth me putting this in as a permanent thing on the Daily Snapshot or not!

[yvi]

That looks awesome!

[deborah]

Excellent timing! Just on Friday, allen and I were talking about the automated tests, because he's actually writing some. Wacky, I know.

We talked about a couple of things, which I wanted to raise with the group anyway:

• Automated tests would be a lot easier if we had a test suite that set up the fake testing database ahead of time. That would make all of the tests run a lot slower, but make it much, much easier to write them and write them safely. Does this seem like a good goal? If so, I can open a bug for it.
• Since it's not like we have a ton of automated tests that would need to transfer, and a large number of our developers don't write or run them, would we think about moving to a more developer-friendly test framework? I see that some people have implemented Ruby's Cucumber in Perl, And I'm sure there would be other good candidates, but there's been a lot of progress in recent years on simple to use, logical test frameworks that don't look like code.
• And on that note, if we switch to something like that, would we consider having "writing tests for our old, untested stuff" be a good task for babydevs / technical people who don't want to actually write code? I think some other projects do that; I don't know about their success rates. We would have to write some good documentation if we went that route.

[denise]

I am not the technical person, but I am not against spending time migrating tests in the least!

[fu]

* Yes to setting up a different database for testing. This is how other testing suites are run. This is how we should do it as well.

* Hmm, yes to a more developer-friendly test framework. I don't have the time to commit my time fully to this, but at a glance, this basically means we'd be writing tests as more of... data input, rather than writing code right? I think that would be immensely helpful.

* Yes, I think that would be good practice. If nothing else, it could be a good way to train people to get as far as checking out the codebase and submitting a patch. Maybe good to check whether it works for other projects though before we dive in?

[mark]

It's great if people want to work on testing. I don't find it particularly useful though, in most cases it just takes time away from development when we already have a dearth of time available to develop things.

There are some situations in which writing tests makes a ton of sense -- I wrote some for the payment system when it was being built, and also for the big WTF changes. Those sorts of systems where getting it wrong makes for security risks and/or money loss is something really worth considering.

UI stuff, though, and other complicated systems? Eh. If people want, that's awesome, but I'm not going to do it or push it.

[exor674]

Yeah, I really really do not like most of our test suite. Even ignoring the DB corruption issues.

[fu]

UI stuff not so much (though a thing can be said for unit-testing JS), but even for relatively less security / money conscious things, testing can take a lot of the pain out of -- well things like making sure entries are parsed right. Because assuming we already had decent tests, it would be a lot easier to do something like add a test with an entry with <script> in the input and just run it, and you'd never have to double-check that case again assuming tests are run properly, than it would be to post a new entry and view it and make changes and refresh and etc.