[Ns-developers] open issues for ns-3 testing

[Pavel Boyko]

  Hi, 

On Thursday 09 July 2009 09:15:06 am Tom Henderson wrote:
> I tried to collect and summarize below what I understand to be many of
> the remaining issues, and maybe Craig and others can respond or add to
> the below list, and propose their requirements or goals.

  Let me add few words. 

  First of all, I believe that the model validation is a research activity 
which can't be automated or restricted to use some build-in "Validation 
toolkit". IMHO the best thing we can do is to encourage a companion "The NS-3 
Foo Model Validation" paper(s) for every foo module. The recent 802.11b PHY 
model is an excellent example of this, see  
http://www.nsnam.org/~pei/80211b.pdf M.b. we should create and maintain a wiki 
page listing all available models together with references to their validation 
reports (see e.g. http://www.scalable-networks.com/pdf/QualNet_Library.pdf for 
a nice table of all available models). When model is validated, usual 
regression tests (both unit and functional) can be used to assert that it is 
not broken. To summarize -- I propose do not create any kind of validation 
toolkit, rather than encourage and collect references to the validation 
efforts.  

> we need a better way for generating, saving, and
> comparing input and output test vectors that are more specific than
> packet trace dumps.

  Agree.

> 2) Why not just borrow one of the existing GPLed testing frameworks and
> reuse it here?  Which ones have we looked at for comparison?

  Existing test.h + waf --check unit test facility just fits my needs and is 
not worse than boost::test + cmake/ctest . 

> 7) Where do more complicated test and validation programs (i.e. the
> actual tests) reside?  src/test?

  Why not?

> 9) What output do people want to see, typically (Gustavo had made a
> comment about minimal output for successful tests and verbose outputs
> for failed tests)?

  Personally I'd like to see (finally) usual green-red html table like 
http://www.cdash.org/CDash/index.php?project=Slicer3#Coverage or 
http://buildbot.net/trac/wiki/ScreenShots with optional cpu time / memory 
usage / stdout / stderr available by click.  

> 10) Can we avoid baking in a GSL dependency in case we want to rewrite
> histogram or chi-squared routines ourselves in the future?

  Why not interface with dedicated statistical systems like R 
(http://www.rproject.org/) for data analysis? I don't believe that we really 
need to create one more homegrown histogram implementation, to say nothing 
about hypothesis testing.

  Regards,
  Pavel Boyko, IITP