[Ns-developers] Request for Merge -- Validation

[Tom Henderson]

Craig, below are some inline comments and suggestions:

craigdo at ee.washington.edu wrote:
> Hi All,
> 
> Tom asked me to put together something that can be used to drive
> non-trace-based regression tests and get it merged as soon as possible.
> I've put together a change to regression.py that allows us to pass or fail
> regression tests based on the return value of the test program.  I've added
> versions of the random number generator validation tests that don't have any
> code to generate pretty plots as a demonstration of how this might work.
> 
> The attached patch has:
> 
> 1) Changes to regression.py to allow non-trace-based tests.  The files in
> regression/tests work just like they used to, except if there's an attribute
> called "trace_compare" in the test module.  In this case, it tells
> regression.py to just look at the return code from the test program.  See
> regression/tests/test-rng-exponential.py for an example.

I'd prefer that we do not make trace-based comparison the default 
choice, as a hint to not lean on this mode of testing too much.

> 
> 2) There are four new .py files in regression/tests corresponding to the
> tests for four of the rng distributions I socialized a while back.
> 
> 3) There is a new valver (validation and verification) directory to hold
> dedicated tests.

I would prefer tests/ to valver/.
> 
> 4) In the valver directory there is an rng directory in which you can find
> four .cc files corresponding to the chi-square tests for the four
> distributions.
> 
> N.B. The rng validation tests introduce a GSL dependency.
> 
> With this in place, we can begin to write validation and verification tests
> that do not use the trace comparison function.
> 

My main comment/question is that there doesn't seem to be any 
integration with the unit test framework, which could also be performing 
the rng checks you are doing.

- what type of test should be run by ./waf check, and what type of test 
by ./waf --regression?  I suppose that trace-based and python script 
tests are constrained to the python-based regression framework, but what 
is the guidance to give to people to put C++ test programs into the unit 
tests or into the python-based framework?

- can C++ tests in the new directory use the src/core/test.h framework 
including macros?  Should we have an example along those lines?

It seems like it would be useful to have these additional examples 
available for ns-3.5:
1) deterministic model verification example such as a TCP example
2) python-based example script that doesn't rely on trace comparison

but at this stage, it would be helpful to first resolve the ./waf check 
vs. ./waf regression question, and to get agreement that something like 
your regression/tests/test-rng-exponential.py is the way to insert 
regression tests into the testing framework when using ./waf --regression.

Tom