[Ns-developers] Request for Merge -- Validation

[Tom Henderson]

craigdo at ee.washington.edu wrote:

> 
> If you really want to discuss deeper things about test philosophy and implementation, then we really need to do that in another
> venue.  If there is no need to do this patch or it is unacceptable as is, then I'd prefer to simply forget it now.  
> 
> It was supposed to be a small thing to make at least some progress in this area ASAP.  I probably spent more time writing this email
> than I did writing the patch.
> 

Well, maybe I wasn't entirely clear, since I think this discussion is 
useful to have now.  I was hoping that we would indeed have the broader 
discussion about what we are doing with tests, after you prototyped a 
few things and socialized them on the list, just like what has happened.

I'm reading another mailing list this morning (quagga-dev) where the 
lead maintainer is complaining that only the maintainers write tests and 
no one is submitting tests with their patches.  I think it is a big 
problem for open source projects and we should really try to get the 
framework as best we can for ns-3.

To make progress, I'd like to see whether we can agree on the following:

- we have primarily two classes of tests; those that check whether the 
models are correct, and those that check whether something changed 
unexpectedly

- the tests that check whether something changed (regression) should 
encompass also the ones that check whether models are correct.  However, 
these should be in separate but related test frameworks.  In particular, 
we should be tracking code coverage results on validation alone 
(separate from regression) and trying to make that statistic go up over 
time.  We can also track coverage of the regression suite so that we 
could keep track of two statistics:
   validation coverage:  X%
   regression coverage:  greater than X%

- to improve our regression coverage, we can turn on coverage for things 
like example scripts, and these could use traces if needed, but we 
should avoid putting trace-based tests in validation framework unless we 
are really validating the behavior of tracing, or unless the convenience 
of such tracing outweighs other factors.

We already have a framework for validation, as Mathieu points out (./waf 
check), and now we have new validation tests in valver/ so I think it is 
a legitimate question to ask whether these should be so separate from 
./waf check, or whether transition to a new framework would help.

We also have a framework for regression and you showed how we can easily 
extend it by dropping in new python scripts.

So, one way to think about the above is that, presently, "./waf check" 
checks for validation, and "./waf --regression" checks for regression, 
and ./waf --regression should cover also ./waf check.  So, if we agree 
so far, the questions on the table now are:

1) how should your new validation tests work with ./waf check, so I can 
get an lcov statistic covering validation alone?  Related to this, do we 
need to have the separate python script for validation tests, to keep 
the testing metadata, or is this not needed for validation (it does seem 
to be needed for trace-based regression tests)?

2) Mathieu is asking whether src/core/test.h could be improved by the 
google test framework

Tom