[Ns-developers] test script and waf (was Re: release update)

[Tom Henderson]

Mathieu Lacage wrote:
> On Fri, 2009-06-05 at 16:49 -0700, craigdo at ee.washington.edu wrote:
> 
>> 2) I have decided that it is too painful to do all of the test running
>> inside waf, so I have packed up and moved into a new test.py
>> that drives the testing process.  I expect that people will clone
>> ns-3-allinone, then ./download.py, ./build.py and ./test.py.
>>
>> Initially test.py will call ./waf check and ./waf --regression for
>> thoroughness, but that should eventually go away.
>>
>> The code in test.py runs tests in parallel based on the number of
>> processors in the system.  It's really quite simple and much
>> easier to understand than, well, anything in waf.  
> 
> I personally don't care much how the test execution is implemented but I
> think that it makes sense to keep the current user interface to
> executing tests, that is, make "./waf check" run the tests. I am sure
> gustavo will be happy to make "./waf check" execute first your script in
> utils/test.py and then, execute the equivalent of --regression. If
> needed (that is, gustavo has no time to do this), I will take care of
> this myself before the release. In time, if gustavo wants to make this
> more integrated in waf by rewriting it using waf tasks, he will be free
> to do so.
> 
> To summarize, if you don't want to deal with waf, don't deal with it,
> but, please, move your script to ns-3-dev/utils/test.py and don't make
> it do anything but execute your test binary: this will make it easier
> for me to make the buildbot machinery run these tests.

Before Gustavo gets carried away with pushing the test suite into waf, I 
would like to support Craig's position here, and ask whether there is 
any good reason to reimplement test.py in waf.

I don't mind if we make "./waf check" a simple wrapper for "./test.py 
--unit," or "./waf test" a wrapper for running "./test.py" with default 
arguments, but I see downsides to trying to reimplement ./test.py in 
waf, and no upsides so far.

1) both people who have looked at waf in detail are cringing at the 
complexity.  Craig initially tried to do it this way.  Sure, you or 
Gustavo or Craig _could_ do it, but if we have a working test.py that is 
much simpler for future people to understand or maintain, what is the point?

2) one argument was that waf knows about the build context to run the 
right tests. However, whatever cached configure information 
(build/c4che/) that waf saves and uses to figure out what tests to run 
should be also readable by test.py.

3) Since test.py (or any test script) will be using arguments to control 
the fine-grained behavior, it will get confusing to detangle the 
command-line arguments if waf runs tests.  You can see a little bit of 
this when you try to run ./waf --run program, when "program" has command 
line arguments-- you have to put quotes around the program and arguments 
so waf associates the arguments with the ns-3 program.  How will waf 
know what arguments are for it and what for the test suite.  I am sure 
that Gustavo can figure out some kind of argument parsing logic for 
this, but again, what is the point, and won't it be more complicated 
syntactically?

Am I missing anything?

Tom