[Ns-developers] test script and waf (was Re: release update)

[Gustavo Carneiro]

2009/6/13 Tom Henderson <tomh at tomh.org>

> Mathieu Lacage wrote:
>
>> On Fri, 2009-06-05 at 16:49 -0700, craigdo at ee.washington.edu wrote:
>>
>>  2) I have decided that it is too painful to do all of the test running
>>> inside waf, so I have packed up and moved into a new test.py
>>> that drives the testing process.  I expect that people will clone
>>> ns-3-allinone, then ./download.py, ./build.py and ./test.py.
>>>
>>> Initially test.py will call ./waf check and ./waf --regression for
>>> thoroughness, but that should eventually go away.
>>>
>>> The code in test.py runs tests in parallel based on the number of
>>> processors in the system.  It's really quite simple and much
>>> easier to understand than, well, anything in waf.
>>>
>>
>> I personally don't care much how the test execution is implemented but I
>> think that it makes sense to keep the current user interface to
>> executing tests, that is, make "./waf check" run the tests. I am sure
>> gustavo will be happy to make "./waf check" execute first your script in
>> utils/test.py and then, execute the equivalent of --regression. If
>> needed (that is, gustavo has no time to do this), I will take care of
>> this myself before the release. In time, if gustavo wants to make this
>> more integrated in waf by rewriting it using waf tasks, he will be free
>> to do so.
>>
>> To summarize, if you don't want to deal with waf, don't deal with it,
>> but, please, move your script to ns-3-dev/utils/test.py and don't make
>> it do anything but execute your test binary: this will make it easier
>> for me to make the buildbot machinery run these tests.
>>
>
> Before Gustavo gets carried away with pushing the test suite into waf, I
> would like to support Craig's position here, and ask whether there is any
> good reason to reimplement test.py in waf.


Heh, sorry, I did get carried away.  However the last pushed code is just a
small incremental change on the way unit tests are run, easier to digest
than an entirely new testing framework from scratch.


> I don't mind if we make "./waf check" a simple wrapper for "./test.py
> --unit," or "./waf test" a wrapper for running "./test.py" with default
> arguments, but I see downsides to trying to reimplement ./test.py in waf,
> and no upsides so far.


I could comment on the relative complexity of test.py vs waf tasks, but I
have to admit to not have followed the discussions closely and I do not even
know where to find this `test.py'.  I'd be interested to take a look.

One thing I like about waf integration is that it looks better, we can see
tests are just waf tasks and it feels better integrated.  Maybe it's only a
pschological issue, but...    Another point is that running the unit tests
can be done with the python functions I defined in wutils.py and which
automatically take care of valgrind handling (valgrind handling is not
trivial to get right, as you may recall).


> 1) both people who have looked at waf in detail are cringing at the
> complexity.  Craig initially tried to do it this way.  Sure, you or Gustavo
> or Craig _could_ do it, but if we have a working test.py that is much
> simpler for future people to understand or maintain, what is the point?


It all depends on whether test.py is really much simpler than waf tasks.
Waf can be complicated to get into, but at least it is documented (see the
chapter about waf tasks, http://freehackers.org/~tnagy/wafbook/ch05.html ).


>
>
> 2) one argument was that waf knows about the build context to run the right
> tests. However, whatever cached configure information (build/c4che/) that
> waf saves and uses to figure out what tests to run should be also readable
> by test.py.
>
> 3) Since test.py (or any test script) will be using arguments to control
> the fine-grained behavior, it will get confusing to detangle the
> command-line arguments if waf runs tests.  You can see a little bit of this
> when you try to run ./waf --run program, when "program" has command line
> arguments-- you have to put quotes around the program and arguments so waf
> associates the arguments with the ns-3 program.  How will waf know what
> arguments are for it and what for the test suite.  I am sure that Gustavo
> can figure out some kind of argument parsing logic for this, but again, what
> is the point, and won't it be more complicated syntactically?


Now I may be missing somthing.  Why exactly does a test suite need to
receive arguments for?  I thought tests were supposed to be fully automated;
taking command-line arguments kind of defeats the purpose, no?

Regards,

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert