[Ns-developers] First Stab at Validation and Verification

[Mathieu Lacage]

hi craig,

On Fri, 2009-04-17 at 16:10 -0700, craigdo at ee.washington.edu wrote:

> http://www.nsnam.org/wiki/index.php/StochasticModelValidation

This looks really really cool !

> It's not intended to be complete and thorough (although the chi square
> tests are real) just a demonstration of where we are headed.
> 
> The code can be found at:
> 
> http://code.nsnam.org/craigdo/ns-3-valver
> 
> with the rng tests in validation/rng and the beginnings of a TCP
> verification test in verification/tcp (not much there yet).

Low-level comments:
-------------------

1) it would be nice to have a shared chi-square facility rather than
re-implementing your own all the time

2) it would be nice to reuse the gnuplot support in src/contrib/gnuplot*


Higher-level comments:
----------------------

1) I am very worried about the content of:
http://www.nsnam.org/wiki/index.php/VerificationValidationAndTesting

As a non-native english speaker, I had to lookup many words in a
dictionary and, even when I found the french translation, I had to
lookup the meaning of the french words. I am worried that the above is
overly formal through abstract and generic considerations about abstract
modelization/verification/validation theory. It would be much more
helpful if it could focus on the important things we (users) need to
know within the context of ns-3. Maybe starting (instead of ending) with
real use-cases network researchers are familiar with would help a lot
explain your goal and what you intend to do to achieve it.

2) I am a bit worried by the idea of trying to do too many things at the
same time: "as a side-effect also generate something that can be used on
a validation and verification website (pretty pictures)". It would be
nice to focus on a single tool to do a single thing really well rather
than build a validation system which does everything. I can see that
it's sort of cool to generate graphs for the web, and that it could help
debugging some failing testcases when they fail but, I feel that this
would steer away valuable resources from the crucial issue: that of
increasing the code coverage of our automated test code. 

For example, looking at the code in verification/rng/*.cc, I see that
quite a bit of code is invested in generating the graphs, and enabling
them from the command-line. If we did not try to do this, we could
easily re-use the existing unit test framework to write these rng tests
in src/core/random-variable.cc, and, then, the question becomes: "which
framework should I use ?" which is a problem related to item 3) below.

3) I am not sure I understand the point of knowing the difference
between verification and validation. Specifically, I don't really
understand why we need to split these two kinds of tests in two
different directories: can't we all bundle them in the 'tests'
directory, and, if we have really too many of them one day, split them ?
What I am worried about here is that, as a user, I would spend time
thinking about where I should put my test code while what really matters
is that I should be writing test code.

I suspect that a lot of the issues I pointed out above are things you
are aware of and are working on already, but, just in case you are not,
here they are.

Mathieu