[Ns-developers] test script and waf (was Re: release update)

[Tom Henderson]

Gustavo Carneiro wrote:
> 
> 
> 2009/6/13 Tom Henderson <tomh at tomh.org <mailto:tomh at tomh.org>>
> 
>     Mathieu Lacage wrote:
> 
>         On Fri, 2009-06-05 at 16:49 -0700, craigdo at ee.washington.edu
>         <mailto:craigdo at ee.washington.edu> wrote:
> 
>             2) I have decided that it is too painful to do all of the
>             test running
>             inside waf, so I have packed up and moved into a new test.py
>             that drives the testing process.  I expect that people will
>             clone
>             ns-3-allinone, then ./download.py, ./build.py and ./test.py.
> 
>             Initially test.py will call ./waf check and ./waf
>             --regression for
>             thoroughness, but that should eventually go away.
> 
>             The code in test.py runs tests in parallel based on the
>             number of
>             processors in the system.  It's really quite simple and much
>             easier to understand than, well, anything in waf.  
> 
> 
>         I personally don't care much how the test execution is
>         implemented but I
>         think that it makes sense to keep the current user interface to
>         executing tests, that is, make "./waf check" run the tests. I am
>         sure
>         gustavo will be happy to make "./waf check" execute first your
>         script in
>         utils/test.py and then, execute the equivalent of --regression. If
>         needed (that is, gustavo has no time to do this), I will take
>         care of
>         this myself before the release. In time, if gustavo wants to
>         make this
>         more integrated in waf by rewriting it using waf tasks, he will
>         be free
>         to do so.
> 
>         To summarize, if you don't want to deal with waf, don't deal
>         with it,
>         but, please, move your script to ns-3-dev/utils/test.py and
>         don't make
>         it do anything but execute your test binary: this will make it
>         easier
>         for me to make the buildbot machinery run these tests.
> 
> 
>     Before Gustavo gets carried away with pushing the test suite into
>     waf, I would like to support Craig's position here, and ask whether
>     there is any good reason to reimplement test.py in waf.
> 
> 
> Heh, sorry, I did get carried away.  However the last pushed code is 
> just a small incremental change on the way unit tests are run, easier to 
> digest than an entirely new testing framework from scratch.
> 
> 
>     I don't mind if we make "./waf check" a simple wrapper for
>     "./test.py --unit," or "./waf test" a wrapper for running
>     "./test.py" with default arguments, but I see downsides to trying to
>     reimplement ./test.py in waf, and no upsides so far.
> 
> 
> I could comment on the relative complexity of test.py vs waf tasks, but 
> I have to admit to not have followed the discussions closely and I do 
> not even know where to find this `test.py'.  I'd be interested to take a 
> look.

Craig has been using mercurial queues; you can find the latest patch at:
http://code.nsnam.org/craigdo/ns-3-test-patches/summary

> 
> One thing I like about waf integration is that it looks better, we can 
> see tests are just waf tasks and it feels better integrated.  Maybe it's 
> only a pschological issue, but...    Another point is that running the 
> unit tests can be done with the python functions I defined in wutils.py 
> and which automatically take care of valgrind handling (valgrind 
> handling is not trivial to get right, as you may recall).

I think Craig recently changed this, but he was previously using waf to 
run the programs, in which case I think the valgrind handling might be 
reuseable.  It seems like in his latest patch he is just running the 
programs directly and not forking a separate waf process (and I don't 
see a --valgrind option).

> 
> 
>     1) both people who have looked at waf in detail are cringing at the
>     complexity.  Craig initially tried to do it this way.  Sure, you or
>     Gustavo or Craig _could_ do it, but if we have a working test.py
>     that is much simpler for future people to understand or maintain,
>     what is the point?
> 
> 
> It all depends on whether test.py is really much simpler than waf 
> tasks.  Waf can be complicated to get into, but at least it is 
> documented (see the chapter about waf tasks, 
> http://freehackers.org/~tnagy/wafbook/ch05.html ).
>  
> 

I am trusting that Craig spent enough time on it when he wrote:
"Ease of use is not a quality that the internals of waf
posess.  I wrote the multiprocessing piece of test.py *much* faster than 
it took me to even begin to understand the task system in waf."

> 
> 
>     2) one argument was that waf knows about the build context to run
>     the right tests. However, whatever cached configure information
>     (build/c4che/) that waf saves and uses to figure out what tests to
>     run should be also readable by test.py.

I think the above logic is now in test.py.

> 
>     3) Since test.py (or any test script) will be using arguments to
>     control the fine-grained behavior, it will get confusing to detangle
>     the command-line arguments if waf runs tests.  You can see a little
>     bit of this when you try to run ./waf --run program, when "program"
>     has command line arguments-- you have to put quotes around the
>     program and arguments so waf associates the arguments with the ns-3
>     program.  How will waf know what arguments are for it and what for
>     the test suite.  I am sure that Gustavo can figure out some kind of
>     argument parsing logic for this, but again, what is the point, and
>     won't it be more complicated syntactically?
> 
> 
> Now I may be missing somthing.  Why exactly does a test suite need to 
> receive arguments for?  I thought tests were supposed to be fully 
> automated; taking command-line arguments kind of defeats the purpose, no?

For instance, if you want to just run a particular test case instead of 
all of them, such as we now do in ./waf --regression.  Or perhaps to 
turn up verbosity levels.

I'm mainly questioning why you or anyone spends their time pushing this 
into waf given Craig's test.py, and even if you want to do it for fun or 
personal preference, is it really going to be better in the long run for 
future users and maintainers?   Anyway, I realize that most people 
haven't looked at test.py yet, so please have a look.

- Tom