[Ns-developers] open issues for ns-3 testing

[Tom Henderson]

Pavel, thanks for the feedback-- some responses below...

Pavel Boyko wrote:
>   Hi, 
> 
> On Thursday 09 July 2009 09:15:06 am Tom Henderson wrote:
>> I tried to collect and summarize below what I understand to be many of
>> the remaining issues, and maybe Craig and others can respond or add to
>> the below list, and propose their requirements or goals.
> 
>   Let me add few words. 
> 
>   First of all, I believe that the model validation is a research activity 
> which can't be automated or restricted to use some build-in "Validation 
> toolkit". IMHO the best thing we can do is to encourage a companion "The NS-3 
> Foo Model Validation" paper(s) for every foo module. The recent 802.11b PHY 
> model is an excellent example of this, see  
> http://www.nsnam.org/~pei/80211b.pdf 

I agree-- I think we need all of these things:
1) a place to list all models
2) some guidelines and examples on how to write validation tests and 
documentation about them
3) a place to store validation scripts and documents that show how they 
have been validated, if at all
4) some way to put them through regression testing, if they are suitable

M.b. we should create and maintain a wiki
> page listing all available models together with references to their validation 
> reports (see e.g. http://www.scalable-networks.com/pdf/QualNet_Library.pdf for 
> a nice table of all available models). 

That is a nice page; presently, we have been trying to build this list 
through the doxygen so that it is maintained inline with the code:
http://www.nsnam.org/doxygen/modules.html
but admittedly you need to click through some of those items to really 
see what is available, and it doesn't link to validation.

Maybe we could maintain a nicely formatted wiki page, as you suggest, 
that lists models and links to the relevant validation material for it.

When model is validated, usual
> regression tests (both unit and functional) can be used to assert that it is 
> not broken. To summarize -- I propose do not create any kind of validation 
> toolkit, rather than encourage and collect references to the validation 
> efforts.  

It may be that some validation programs are not easily suited to be run 
as regression tests, but ideally it would be nice to reuse them that way.

What I would like to produce, though, are some good examples of how to 
do various types of validation and regression testing so others can 
contribute similar things.

>> 10) Can we avoid baking in a GSL dependency in case we want to rewrite
>> histogram or chi-squared routines ourselves in the future?
> 
>   Why not interface with dedicated statistical systems like R 
> (http://www.rproject.org/) for data analysis? I don't believe that we really 
> need to create one more homegrown histogram implementation, to say nothing 
> about hypothesis testing.
> 

I personally don't want to rewrite those things, but we have to date 
avoided making most aspects of ns-3 core dependent on external 
libraries, so the only reason I see for writing a histogram routine 
would be to avoid having to install and link something like GSL.  But, 
your point about r-project.org seems to reinforce my point that maybe it 
will not be GSL that we use in the future for this statistical work, and 
that we could maybe decouple the tests from explicit GSL calls.

Tom