[Ns-developers] First Stab at Validation and Verification

[Mathieu Lacage]

On Tue, 2009-04-21 at 23:03 -0700, Tom Henderson wrote:
> >>
> >> So, what I am really asking is whether it's a good idea to invest
> >> resources in this rather than just write tests that you, as a test
> >> writer, feel good about. i.e., I am worried that generating graphs
> >> brings nothing to the writing of good tests which is what I 
> >> feel really
> >> concerned about.
> > 
> > Okay, you have a meta-issue.  Our Fearless Leader is very concerned about
> > how to present to the rest of the world that all of that work writing good
> > tests has been done.  He wants to see the graphs and web sites.
> > 
> 
> Nice bullwinkle reference...  I don't think graphs or web sites are 
> strictly necessary.  I agree that the focus should be on the tests 
> themselves, but the fact that the tests have been done and are being 
> maintained (if so) should be documented somehow for users.  That can be 
> on the wiki, in the manual, in commented scripts or unit tests, in 
> separate technical reports, etc.   Mainly, I'd like to see a few good 
> examples of how this can be done, so that code contributors can follow 
> the lead.

Ok, I agree that we need to have a nice reference/howto on the process
of writing good tests which make sense and, having a good complete
detailed step-by-step example is a good idea. 

I think that I also see how a lot of per-test graphs could be
automatically generated with the right framework in place, mainly for
stochastic tests. i.e., I see how I could write a wifi test which
verifies that the wifi MAC reaches the right theoretical upper bound on
the application-level throughput for each physical layer transmission
mode when there are no transmission errors or static packet error rates
and I see how the data generated by these tests could be fed into an
automatic graph generator.

To summarize, if we can move the graph generation code outside of the
testing code into the testing framework, and if the only thing tests
have to be concerned about is generating data and a way to verify that
the data is valid, then, it sounds like the best way to go forward.
i.e., more than that, it's _real_ cool (tm) :)

Mathieu