[Ns-developers] test script and waf (was Re: release update)

[Gustavo Carneiro]

2009/6/13 Tom Henderson <tomh at tomh.org>

> Gustavo Carneiro wrote:
>
>>
>>
>> 2009/6/13 Tom Henderson <tomh at tomh.org <mailto:tomh at tomh.org>>
>>
>>    Mathieu Lacage wrote:
>>
>>        On Fri, 2009-06-05 at 16:49 -0700, craigdo at ee.washington.edu
>>        <mailto:craigdo at ee.washington.edu> wrote:
>>
>>            2) I have decided that it is too painful to do all of the
>>            test running
>>            inside waf, so I have packed up and moved into a new test.py
>>            that drives the testing process.  I expect that people will
>>            clone
>>            ns-3-allinone, then ./download.py, ./build.py and ./test.py.
>>
>>            Initially test.py will call ./waf check and ./waf
>>            --regression for
>>            thoroughness, but that should eventually go away.
>>
>>            The code in test.py runs tests in parallel based on the
>>            number of
>>            processors in the system.  It's really quite simple and much
>>            easier to understand than, well, anything in waf.
>>
>>        I personally don't care much how the test execution is
>>        implemented but I
>>        think that it makes sense to keep the current user interface to
>>        executing tests, that is, make "./waf check" run the tests. I am
>>        sure
>>        gustavo will be happy to make "./waf check" execute first your
>>        script in
>>        utils/test.py and then, execute the equivalent of --regression. If
>>        needed (that is, gustavo has no time to do this), I will take
>>        care of
>>        this myself before the release. In time, if gustavo wants to
>>        make this
>>        more integrated in waf by rewriting it using waf tasks, he will
>>        be free
>>        to do so.
>>
>>        To summarize, if you don't want to deal with waf, don't deal
>>        with it,
>>        but, please, move your script to ns-3-dev/utils/test.py and
>>        don't make
>>        it do anything but execute your test binary: this will make it
>>        easier
>>        for me to make the buildbot machinery run these tests.
>>
>>
>>    Before Gustavo gets carried away with pushing the test suite into
>>    waf, I would like to support Craig's position here, and ask whether
>>    there is any good reason to reimplement test.py in waf.
>>
>>
>> Heh, sorry, I did get carried away.  However the last pushed code is just
>> a small incremental change on the way unit tests are run, easier to digest
>> than an entirely new testing framework from scratch.
>>
>>
>>    I don't mind if we make "./waf check" a simple wrapper for
>>    "./test.py --unit," or "./waf test" a wrapper for running
>>    "./test.py" with default arguments, but I see downsides to trying to
>>    reimplement ./test.py in waf, and no upsides so far.
>>
>>
>> I could comment on the relative complexity of test.py vs waf tasks, but I
>> have to admit to not have followed the discussions closely and I do not even
>> know where to find this `test.py'.  I'd be interested to take a look.
>>
>
> Craig has been using mercurial queues; you can find the latest patch at:
> http://code.nsnam.org/craigdo/ns-3-test-patches/summary
>
>
>> One thing I like about waf integration is that it looks better, we can see
>> tests are just waf tasks and it feels better integrated.  Maybe it's only a
>> pschological issue, but...    Another point is that running the unit tests
>> can be done with the python functions I defined in wutils.py and which
>> automatically take care of valgrind handling (valgrind handling is not
>> trivial to get right, as you may recall).
>>
>
> I think Craig recently changed this, but he was previously using waf to run
> the programs, in which case I think the valgrind handling might be
> reuseable.  It seems like in his latest patch he is just running the
> programs directly and not forking a separate waf process (and I don't see a
> --valgrind option).
>
>
>>
>>    1) both people who have looked at waf in detail are cringing at the
>>    complexity.  Craig initially tried to do it this way.  Sure, you or
>>    Gustavo or Craig _could_ do it, but if we have a working test.py
>>    that is much simpler for future people to understand or maintain,
>>    what is the point?
>>
>>
>> It all depends on whether test.py is really much simpler than waf tasks.
>>  Waf can be complicated to get into, but at least it is documented (see the
>> chapter about waf tasks, http://freehackers.org/~tnagy/wafbook/ch05.html<http://freehackers.org/%7Etnagy/wafbook/ch05.html>).
>>
>>
>
> I am trusting that Craig spent enough time on it when he wrote:
> "Ease of use is not a quality that the internals of waf
> posess.  I wrote the multiprocessing piece of test.py *much* faster than it
> took me to even begin to understand the task system in waf."
>
>
>>
>>    2) one argument was that waf knows about the build context to run
>>    the right tests. However, whatever cached configure information
>>    (build/c4che/) that waf saves and uses to figure out what tests to
>>    run should be also readable by test.py.
>>
>
> I think the above logic is now in test.py.
>
>
>>    3) Since test.py (or any test script) will be using arguments to
>>    control the fine-grained behavior, it will get confusing to detangle
>>    the command-line arguments if waf runs tests.  You can see a little
>>    bit of this when you try to run ./waf --run program, when "program"
>>    has command line arguments-- you have to put quotes around the
>>    program and arguments so waf associates the arguments with the ns-3
>>    program.  How will waf know what arguments are for it and what for
>>    the test suite.  I am sure that Gustavo can figure out some kind of
>>    argument parsing logic for this, but again, what is the point, and
>>    won't it be more complicated syntactically?
>>
>>
>> Now I may be missing somthing.  Why exactly does a test suite need to
>> receive arguments for?  I thought tests were supposed to be fully automated;
>> taking command-line arguments kind of defeats the purpose, no?
>>
>
> For instance, if you want to just run a particular test case instead of all
> of them, such as we now do in ./waf --regression.  Or perhaps to turn up
> verbosity levels.


Ok, but those are options for the test runner, not for individual tests.  I
see no problem in command-line handling with or without waf.


>
>
> I'm mainly questioning why you or anyone spends their time pushing this
> into waf given Craig's test.py,


I am merely demonstrated that certain advanced things can be done in waf.
It took some bug fixing in waf itself, but it got done ;-)


> and even if you want to do it for fun or personal preference, is it really
> going to be better in the long run for future users and maintainers?


Well, the thing is, a lot of stuff is already happening using waf tasks and
task generators.  So, unless you want to replace waf with something else,
future maintainers already will have to learn about waf tasks anyway.  With
test.py, you have to learn not only about waf tasks but also about python
queues and threads.  It is not clear to me which situation is worse.

My personal preference has been indeed for waf, due to less incremental work
needed to get the job done.

That being said, if it is decided to use test.py, be my guest, and we can
just rip out my waf code, I won't mind.  But in that case I would rather not
become the maintainer of that script, which I did not write.  I remember
last time someone else wrote a regression.py script which I ended up sort of
maintaining... :-)


>   Anyway, I realize that most people haven't looked at test.py yet, so
> please have a look.


It was diffcult to look because I could only find it hidden inside a patch.
Why not a normal mercurial branch?  But here some comments:

  1. Why do we have a new "TestRunner" program; I thought utils/run-tests.cc
took care of this?

  2. The multiprocessing module is new in Python 2.6;

  3. I hate running subprocesses a) through a shell, b) with hardcoded '/'
as path separator.  These things make the system much less portable;

  4. The exec statment is considered evil by most Python programmers; if
there is a way around it, it should be taken;

  5. Of course reading the waf cache data directly is a bit hackish and
might break in the future.  Perhaps implementing a --print-env wscript
option might be better in the long run.

  6. Do I see all the parallel executing tests run with the same CWD?  This
sort of thing is usually a killer of multiprocessing... Hint:
subprocess.Popen() has a cwd parameter...

I hope this helps.

Best regards,
-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert