[Ns-developers] validation plans

[Tom Henderson]

> 
> 3) Validation. A primary goal of the project is to support not just new
> models but validated models. Presently, we have drifted into the state
> of equating regression tests with validation, and relying too heavily on
> trace-based regression tests. We haven't yet exploited the new ns-3
> tracing framework to write better validation tests and link them into
> our regression testing framework. Again, I will have more to say on this
> in a separate email, but I would like to harness the recent activity on
> 802.11 validation and define some best current practices and good
> examples of how to contribute validation results and programs to the
> ns-3 project.
> 

Craig has started to put together a framework for improved validation 
and model verification.  People interested in this topic can review and 
comment on what he has written so far at:
http://www.nsnam.org/wiki/index.php/VerificationValidationAndTesting

What I have in mind is that we try to separate better the example or 
sample scripts, regression tests, and validation or verification 
scripts.  Validation or verification tests should be designed to 
explicitly test for specific behavior.  A few possible examples to start 
with are our TCP model, 802.11 PHY models, and random variables.  But 
rather than just dump and archive trace files, it would be better to 
instrument the tests to explicitly test whether an expected event (e.g. 
a TCP timeout) occurred due to some stimulus.  Selected validation tests 
and all unit tests could be run through a regression testing framework 
(what we have now), where the objective would be to cover as much of our 
code as possible.  Example programs would be programs that show how 
users might want to build simulation scenarios.  It may be that some of 
our examples are also used in regression testing if other scripts to 
achieve the desired code coverage are not available, but we would try to 
be more rigorous in the future about separating these types of scripts.

- Tom