[Ns-developers] Waf and Testing: Request for Opinions

[Gustavo Carneiro]

Disclaimer: I have been too busy to follow these (probably interesting)
discussions, so here are some probably misinformed comments.

2009/5/18 <craigdo at ee.washington.edu>

> Dear All,
>
> I'm working on a way to rationally combine the unit tests (waf check), the
> regression tests (waf --regression) and the new validation and verification
> tests.  I haven't found a reason why all of our tests should not use the
> same framework.
>
> Right now, waf check is implemented as a waf command and runs a single
> C++-based program that acts as a runner that in turn runs all of the unit
> tests defined in our modules sequentially.
>
> Right now waf --regression is implemented as new task generators that spawn
> off a number of tasks that are run in parallel and don't have anything to
> do
> with the test framework.  This is the same as waf build --regression.
>
> It would be nice to have all of our testing use the same run model (and
> framework).  I mean change to one of either waf --check or waf regression.
> It would also be nice to be able to run all tests using one command,
> something like waf --test or waf test.
>
> As the system increases in size it seems like a good plan to parallelize as
> much as possible which would mean running waf --check and waf --regression
> (and running them both with waf --test).
>
> Does anyone have a strong reason to keep waf check around (versus waf
> --check)?  I'd like to change it.


You mean "... vs waf --regression", I think.

One reason I can think of is that unit tests are much easier to create than
regression tests, and usually run much faster.  I would not like to lose the
current unit test framework at all.

One way to converge is to keep the unit tests, but run one unit test group
at a time from the same program, i.e. we create N waf tasks, each task runs
"build/<variant>/utils/run-tests <group-name>", for all unit test group
name, and modify run-tests to run just those unit tests (somehow).

Converging to waf tasks is better than converging to a single program that
runs everything:

    1. Parallelization for free;

    2. Each test group in a separate process is better to isolate faults
(e.g. segfault or memory leaks in one test group does not affect the
others).



> I am also considering how to clean up the output of the tests which is
> currently bordering on noise.  I'm considering a way to build an html page
> as output, making something evenually somewhat like
> http://tungwaiyip.info/software/sample_test_report.html which would
> require
> a unified output model for all of our tests.
>
> Any comments on the kind of output you'd like to see?


Html is nice, but I have nothing against plain text output.  Perhaps it's
just that the output is too verbose, maybe it should only print when a test
fails, not when tests pass.  See for instance the Python unit tests, which
by default only print one small dot for each test that passes, and a summary
in the end with the count of executed tests.

Regards,
-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert