[Ns-developers] First Stab at Validation and Verification

[craigdo@ee.washington.edu]

Hi All,

Some of you may have noticed that I've been pondering what to do about validation and verification of ns-3 models.  I've played
around a bit and have an example put together that does some statistical validation of a subset of our random number generators.

Some folks around here have expressed an interest in getting this cleaned up and checked in fairly quickly.  So even though this is
not even close to fully baked I thought I'd socialize what I have.  Feel free to comment or tell me I'm on crack, as another one of
the developers is fond of saying.

The high-level idea is to sharpen our models for what we consider system/integration/regression tests.  Right now we have regression
tests that are basically example programs we have pressed into double duty.  We want to move toward purpose-build verification tests
and away from the trace-file-based example program hacks we have today.  We also want to add some form of validation framework where
people can demonstrate that their models are connected with some kind of reality.  As a byproduct of this validation and
verification test effort, we would like to see something that can be used as a regression suite and as a side-effect also generate
something that can be used on a validation and verification website (pretty pictures).

I've put together some words on the wiki.  They're kind of incomplete ramblings and thoughts right now, but you can probably get a
decent flavor of where I'm going (don't hold me to anything):

http://www.nsnam.org/wiki/index.php/VerificationValidationAndTesting

As it stands, the two areas of verification and validation are quite separate, but they'll most likely get folded together as things
develop.

I have random number generator tests built that do statistical validation, generate pretty plots and can be run as regression tests.
I'll eventually integrate them into waf, but they are stand-alone right now.  As an example of a validation wiki, I put together the
following:

http://www.nsnam.org/wiki/index.php/StochasticModelValidation

It's not intended to be complete and thorough (although the chi square tests are real) just a demonstration of where we are headed.

The code can be found at:

http://code.nsnam.org/craigdo/ns-3-valver

with the rng tests in validation/rng and the beginnings of a TCP verification test in verification/tcp (not much there yet).

If you have any plans or ideas please don't hesitate to let me know about them.  If you are doing validation tests on your own, I'd
love to figure out how to make it easy to integrate them.

Regards,

-- Craig