[Ns-developers] First Stab at Validation and Verification

[craigdo@ee.washington.edu]

> hi craig,
> 
> On Fri, 2009-04-17 at 16:10 -0700, craigdo at ee.washington.edu wrote:
> 
> > http://www.nsnam.org/wiki/index.php/StochasticModelValidation
> 
> This looks really really cool !
> 
> > It's not intended to be complete and thorough (although the 
> chi square
> > tests are real) just a demonstration of where we are headed.
> > 
> > The code can be found at:
> > 
> > http://code.nsnam.org/craigdo/ns-3-valver
> > 
> > with the rng tests in validation/rng and the beginnings of a TCP
> > verification test in verification/tcp (not much there yet).
> 
> Low-level comments:
> -------------------
> 
> 1) it would be nice to have a shared chi-square facility rather than
> re-implementing your own all the time

Yes.  It's really just a very few lines of code, so I haven't worried about
it yet.

> 2) it would be nice to reuse the gnuplot support in 
> src/contrib/gnuplot*

I forgot about that completely.  Thanks ...

> Higher-level comments:
> ----------------------
> 
> 1) I am very worried about the content of:
> http://www.nsnam.org/wiki/index.php/VerificationValidationAndTesting
> 
> As a non-native english speaker, I had to lookup many words in a
> dictionary and, even when I found the french translation, I had to
> lookup the meaning of the french words. I am worried that the above is
> overly formal through abstract and generic considerations 
> about abstract
> modelization/verification/validation theory. 

One of the problems I've run into is that the terminology is confused.  I
wanted to try and write down exactly what it was that I was trying to
accomplish and define the terms I was using.  Unfortunately, for example,
validation can mean different things to different people.  I had imagined
that ambiguous terminology translated into French, German or Portuguese
would be a disaster waiting to happen.  I've been known to be mistaken on
this front, though.  Are there specifics that you found unnecessary or
overly pedantic after your translation exercise?  I know there are terms
there to help clarify what I'm *not* doing.

I don't expect this page to remain (perhaps a simplified version could end
up buried in a manual) but I did want to make sure that everyone could
figure out what I was talking about.  Tom commented on the term "Expectation
Value," which seems to be physics jargon.

> It would be much more
> helpful if it could focus on the important things we (users) need to
> know within the context of ns-3. Maybe starting (instead of 
> ending) with
> real use-cases network researchers are familiar with would help a lot
> explain your goal and what you intend to do to achieve it.

I haven't really written a requirements document yet.  What you see is
really an exploration preliminary to writing something like that.  I'll
definitely produce something like that as the ideas come together.  This is
all not even close to being fully baked.

> 2) I am a bit worried by the idea of trying to do too many 
> things at the
> same time: "as a side-effect also generate something that can 
> be used on
> a validation and verification website (pretty pictures)". It would be
> nice to focus on a single tool to do a single thing really well rather
> than build a validation system which does everything. I can see that
> it's sort of cool to generate graphs for the web, and that it 
> could help
> debugging some failing testcases when they fail but, I feel that this
> would steer away valuable resources from the crucial issue: that of
> increasing the code coverage of our automated test code. 

Well, "increasing the code coverage of our automated test code" is a goal.
How important this is depends on how you are looking at the bigger picture.

Another explicit goal I have been given is to produce the beginnings of an
ns-3 validation and verification web site.  We want to give model authors
examples of how to do validation and verification, and give them the tools
to add to the web suite (which is why I made the wiki pages).  The ability
to re-use the code used in validation and verification to provide an
automatic regression test suite is a very nice feature.

What I think you are talking about is reworking the regression test suite to
use a new framework that doesn't rely on pcap traces.  This is a goal.  This
transitions us from the kind of almost accidental testing we do to
purposeful testing that can then be used in an automated way.  So my goal is
really to create a framework that allows one to write code that can
demonstrate that, for example, our TCP is compliant to something (using a
web site), and then provide a way to package up those tests to perform a
regression to ensure that the TCP continues to be compliant over time.

> For example, looking at the code in verification/rng/*.cc, I see that
> quite a bit of code is invested in generating the graphs, and enabling
> them from the command-line. If we did not try to do this, we could
> easily re-use the existing unit test framework to write these 
> rng tests
> in src/core/random-variable.cc, and, then, the question 
> becomes: "which
> framework should I use ?" which is a problem related to item 3) below.

The difference is related to the purpose of the exercise.  In the case of
random number generators, the first goal is to do validation.  One result of
the validation process should be the pretty web site that I showed.  If
someone wonders about the  random number generators they can go to the web
site and see that they have been validated.

It is also nice to use the tests that someone wrote to automatically confirm
that the validated RNGs continue to be validated.  The question is then,
"how do you extract those tests and run them automatically."  This results
in the command-line stuff you see.  Exactly the same code is then used to
validate an RNG and to do regression testing (which isn't implemented yet).

I suppose it is just as reasonable to write validation tests as part of the
unit tests and reuse them "forward" into a validation suite as it is to
write purpose-built validation tests and reuse them "backward" into
regression tests.  The benefit of using the "backward" approach (heh) is
that you do have the environment to generate the pretty pictures for your
validation report on the web site.

I've talked about a "framework" to do this.  Right now the framework is
main(){} so it's not too impressive-looking.

> 3) I am not sure I understand the point of knowing the difference
> between verification and validation. Specifically, I don't really
> understand why we need to split these two kinds of tests in two
> different directories: can't we all bundle them in the 'tests'
> directory, and, if we have really too many of them one day, 
> split them ?

I suspect that eventually these two directories will be merged into one,
with one environment that allows for the different kinds of tests.  I
mentioned this somewhere I think ...

The difference between validation and verification is subtle; but there is a
difference.  Perhaps it is not important enough to worry about, but I'm not
sure yet.  Right now, I have a completely different idea of what it means to
verify that my TCP connection establishment works according to RFC; and that
my wifi model behaves like some given real radio with respect to
transmission range.

I think this is really a design and implementation process choice rather
than something cast in stone.  

> What I am worried about here is that, as a user, I would spend time
> thinking about where I should put my test code while what 
> really matters
> is that I should be writing test code.

Hopefully what I am encouraging is thinking clearly about what kind of
testing to do and what kinds ns-3 supports and encourages.

> I suspect that a lot of the issues I pointed out above are things you
> are aware of and are working on already, but, just in case 
> you are not,
> here they are.

Thanks.

-- Craig