szabgab: (Default)
Gabor Szabo ([personal profile] szabgab) wrote in [site community profile] dw_dev2009-09-02 02:51 pm
Entry tags:

testing DW

In the past few days I managed to fix some of the unit tests and set the rest to be skipped so once my latest patch of bug 1721 gets applied all tests in t/ will either pass or skip.

In order to make development easier in DW it would be important to keep the test suit intact and even adding more tests.

Currently the way to run the tests is by typing

cd $LJHOME
prove t/

In case you would like to run a single test file you can type

prove t/userpics.t


A better way to run the tests will be to run

prove -w t/

but currently it produces lots of warnings, something we should fix soon.


Every time to make a change you should at a minimum make sure that all the tests still pass after the change. If some of the tests stop working you should check if the test is incorrect or if the change you made to the code introduced a real bug and fix it.



TODO



In order to get the test system back on its feet quickly I set many of the test files to be skipped. So we have two main tasks in order to improve testing.

1) Go over the tests. For the ones that are passing, run them with prove -w and fix (either the test or the application) so there won't be warnings.
For the tests that are skipped remove the skip and find out why are they breaking. Sometimes the test are not relevant any more (I saw a few trying to tests modules that do not exists in the codebase), sometimes a little tweak to the configuration is needed (many tests were using modules that were relying on $LJ::HOME being set and it was not in the test script). Sometimes they might actually indicate some broken feature in the application itself.


2) Add more tests. Actually the best practice would be that every time you touch a piece of code you first make sure that it has tests. If it does not, or not enough then first add tests and only then change the application. If you are fixing a bug then first write a test that can reproduce the bug. The test will obviously fail. Then fix the code and see that the test passes.


For the actual techniques of writing tests, the best is to read the testing tutorial
http://search.cpan.org/dist/Test-Simple/lib/Test/Tutorial.pod
and then go on reading the docs of Test::More http://search.cpan.org/dist/Test-Simple/lib/Test/More.pm
and then those of Test::Most http://search.cpan.org/dist/Test-Most/lib/Test/Most.pm



Test Coverage



In order to see which part of the code has tests and which does not we can
use Devel::Cover to create a report. In order to use that we need to do the following:

cd $LJHOME
perl Build.PL
./Build
./Build test


# this will run the tests in regular mode and is not really needed here

DEVEL_COVER_OPTIONS=-ignore,/usr/local ./Build testcover


# this will run the tests using Devel::Cover and will generate a report.
# ignoring the files in /usr/local
# It will run the tests a lot slower than in normal mode


Files=85, Tests=1281, 2467 wallclock secs ( 0.72 usr  0.66 sys + 2238.73 cusr 50.97 csys = 2291.08 CPU)


opposed to

Files=85, Tests=1281, 69 wallclock secs ( 0.68 usr  0.54 sys + 36.69 cusr  7.41 csys = 45.32 CPU)


41 minutes opposed to 1 minute. (it is a bit unusual to me so we have to find out why is such a big different)

The result is here: http://hack.dreamwidth.net/coverage/20090902/coverage.html

We will run the test report once in a while so we can see progress.

Explanations about Devel::Cover can be found here: http://search.cpan.org/dist/Devel-Cover/lib/Devel/Cover/Tutorial.pod


Testing Strategy



I'd recommend the following testing strategy:

1) Every time you encounter a bug first write a test case that reproduces it. You can do this even if you are not going to fix it right away. In this case set the test as a TODO test. See the Test::More documentation on how to do this.

2) We should add page level tests (Using WWW::Mechanzie and/or Selenium) that will test certain functionality of the web site. These tests will remain with us as we refactor the code beneath.

3) Every new piece of code should be tested at the level of the function(s) and module(s) and also at the level of the application as well (see entry 2). This will both allow us to make sure the users get the correct functionality but will also make it easy to test small parts of the code.

4) In the longer term old code should also get function and module level tests in order to allow us more fine-tuned testing of the code.





cesy: "Cesy" - An old-fashioned quill and ink (Default)

[personal profile] cesy 2009-09-02 11:58 am (UTC)(link)
Two quick tips (I'll try the tests out properly later):
LJ-cuts are useful, so you don't overload people's reading lists. Put [cut text="Short summary"] and [/cut] with angle brackets around the bulk of the entry for long entries like this.
If you're cross-posting to comms where a lot of people will read both (which I think is true in this case), it's often better to post the full entry once place, then on the other, a link and short description/summary.
(Personal preference does vary on both of these issues, but I think my view is reasonably common.)
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)

[staff profile] mark 2009-09-02 04:33 pm (UTC)(link)
Oh, awesome! Thank you for putting this together. That's a lot of red on the code coverage report. I guess that's expected, but geeze...

On a practical level, what do you find is the best level to put the tests at? For example, we could test the page level functionality (can you subscribe to someone) but we could also test all the way down to the level of base methods (can you get a database handle, can you load a user's subscription list, can you add a subscription to the list).

The former implicitly tests the latter. But the latter will help identify where something has broken more quickly, although it involves more tests? Thoughts?
cesy: "Cesy" - An old-fashioned quill and ink (Default)

[personal profile] cesy 2009-09-02 07:01 pm (UTC)(link)
I've updated the wiki page at http://wiki.dwscoalition.org/notes/Dev_Testing a little, but it could probably do with more of this content eventually. Also, this needs a note that you need to be in $LJHOME to run prove t/ or whatever.