[Ns-developers] First Stab at Validation and Verification

[Mathieu Lacage]

On Tue, 2009-04-21 at 10:38 -0700, craigdo at ee.washington.edu wrote:

> > 1) I am very worried about the content of:
> > http://www.nsnam.org/wiki/index.php/VerificationValidationAndTesting
> > 
> > As a non-native english speaker, I had to lookup many words in a
> > dictionary and, even when I found the french translation, I had to
> > lookup the meaning of the french words. I am worried that the above is
> > overly formal through abstract and generic considerations 
> > about abstract
> > modelization/verification/validation theory. 
> 
> One of the problems I've run into is that the terminology is confused.  I
> wanted to try and write down exactly what it was that I was trying to
> accomplish and define the terms I was using.  Unfortunately, for example,
> validation can mean different things to different people.  I had imagined
> that ambiguous terminology translated into French, German or Portuguese
> would be a disaster waiting to happen.  I've been known to be mistaken on
> this front, though.  Are there specifics that you found unnecessary or
> overly pedantic after your translation exercise?  I know there are terms
> there to help clarify what I'm *not* doing.

Sentences such as "Prescribed conditions for which the model has been
tested, compared against reality to the extent possible, and judged
suitable for use;" or "Substantiation that a model, within its domain of
applicability, possesses a satisfactory range of accuracy consistent
with the intended application of the model;" make my brain hurt in
french, and explode in english. I had to read/process them multiple
times.

[snip]

> > For example, looking at the code in verification/rng/*.cc, I see that
> > quite a bit of code is invested in generating the graphs, and enabling
> > them from the command-line. If we did not try to do this, we could
> > easily re-use the existing unit test framework to write these 
> > rng tests
> > in src/core/random-variable.cc, and, then, the question 
> > becomes: "which
> > framework should I use ?" which is a problem related to item 3) below.
> 
> The difference is related to the purpose of the exercise.  In the case of
> random number generators, the first goal is to do validation.  One result of
> the validation process should be the pretty web site that I showed.  If
> someone wonders about the  random number generators they can go to the web
> site and see that they have been validated.

I understand that what you are trying to do here is provide
documentation about regression tests to allow third parties to evaluate
the quality of these tests: the kind of website you showed is awesome in
that respect.

So, what I am really asking is whether it's a good idea to invest
resources in this rather than just write tests that you, as a test
writer, feel good about. i.e., I am worried that generating graphs
brings nothing to the writing of good tests which is what I feel really
concerned about.

Mathieu