[Ns-developers] test script and waf (was Re: release update)

[craigdo@ee.washington.edu]

> >>    2) one argument was that waf knows about the build 
> context to run
> >>    the right tests. However, whatever cached configure information
> >>    (build/c4che/) that waf saves and uses to figure out 
> what tests to
> >>    run should be also readable by test.py.
> >>
> >
> > I think the above logic is now in test.py.

I'm not sure that this actually needs to be in there.  I'm just
experimenting with it right now.  Waf builds and links based on its
configuration.  If it doesn't build a file or module, the tests for that
file or module won't be built.  In the case of the test runner, this means
that the tests won't be listed by the runner and won't be run.  It all just
takes care of itself.

Examples is a little different.  We want to run all of the examples we can
as a "smoke test."  Not all of the examples are suitable to be automatically
run (emulation, for example), so I put in a list of examples that should be
run.  One way to deal with the case where a given example is not built due
to configuration is to simply check for existence.  If an example doesn't
exist, then test.py can just assume it's because of configuration.  A
counter-argument is that this would be a good place to do a last sanity
check and if, for example, ENABLE_NSC is true then the NSC examples aught to
be built and runnable on that system.  

[Of course it's easier to check for existence if you know where the
executable lives; which means it is nicer if you have internal waf state;
but if you have internal waf state you don't need to check for existence;
but you can easily add a "list" executables command to waf to find out what
you need -- so there many tradeoffs in this deal that need to be
considered.]

I originally just ran everything using "./waf --run" since it seemed to
cover the bases just fine.  One thing that bothered me was forking a process
to run waf that forked a process to run the test executable.  This is why I
prototyped a way to directly run the executable.  I'm about ready to go back
to using waf run, though, since it seems to be much cleaner despite the
extra forks.  I'm a big fan of simple and clean.

> >>    3) Since test.py (or any test script) will be using arguments to
> >>    control the fine-grained behavior, it will get 
> confusing to detangle
> >>    the command-line arguments if waf runs tests.  You can 
> see a little
> >>    bit of this when you try to run ./waf --run program, 
> when "program"
> >>    has command line arguments-- you have to put quotes around the
> >>    program and arguments so waf associates the arguments 
> with the ns-3
> >>    program.  How will waf know what arguments are for it 
> and what for
> >>    the test suite.  I am sure that Gustavo can figure out 
> some kind of
> >>    argument parsing logic for this, but again, what is the 
> point, and
> >>    won't it be more complicated syntactically?
> >>
> >>
> >> Now I may be missing somthing.  Why exactly does a test 
> suite need to
> >> receive arguments for?  I thought tests were supposed to 
> be fully automated;
> >> taking command-line arguments kind of defeats the purpose, no?
> >>
> >
> > For instance, if you want to just run a particular test 
> case instead of all
> > of them, such as we now do in ./waf --regression.  Or 
> perhaps to turn up
> > verbosity levels.
> 
> Ok, but those are options for the test runner, not for 
> individual tests.  I
> see no problem in command-line handling with or without waf.

Just curious.  How would you handle the following?

  ./waf --test --verbose --html=daily.html --unit --suite='Object Name
Service' --valgrind

I can imagine that you might do something like, 

  ./waf --test --testOpt="--verbose --html=daily.html --unit --suite='Object
Name Service'" --valgrind

but it seems much more reasonable to me to do

  ./test.py --verbose --html=daily.html --unit --suite='Object Name Service'
--valgrind

than to somehow disentangle/disambiguate test options from waf build-related
options.

> > I'm mainly questioning why you or anyone spends their time 
> pushing this
> > into waf given Craig's test.py,
> 
> I am merely demonstrated that certain advanced things can be 
> done in waf.
> It took some bug fixing in waf itself, but it got done ;-)

It can clearly all be done in waf.  I am looking at how much of the
implementation makes sense in waf and how much in simpler external programs;
and how much easier it is for normal people to deal with the combinations.

> > and even if you want to do it for fun or personal 
> preference, is it really
> > going to be better in the long run for future users and maintainers?
> 
> Well, the thing is, a lot of stuff is already happening using 
> waf tasks and
> task generators.  So, unless you want to replace waf with 
> something else,
> future maintainers already will have to learn about waf tasks 
> anyway.  With
> test.py, you have to learn not only about waf tasks but also 
> about python
> queues and threads.  It is not clear to me which situation is worse.

Well, it's not clear to me that you are the right person to answer that
question, since you are our waf guru :-)  It may seem relatively easy to
you, but I'm not sure the average person who wants to add a test feature is
going to be up for "some bug fixing in waf" in order to get it done.  I
don't know about anyone else, but I find doing something new in waf ...
difficult.

I think a little self-contained, well documented piece of vanilla Python,
without any trace of magic, is going to be much more accessible for the rest
of us mere mortals.

> My personal preference has been indeed for waf, due to less 
> incremental work
> needed to get the job done.
> 
> That being said, if it is decided to use test.py, be my 
> guest, and we can
> just rip out my waf code, I won't mind.  But in that case I 
> would rather not
> become the maintainer of that script, which I did not write.  
> I remember
> last time someone else wrote a regression.py script which I 
> ended up sort of
> maintaining... :-)

I don't think anyone is asking you to maintain test.py.  My main point is
that I want this to be easy so every time something goes wrong we *don't*
have to email you.  BTW, I'm not sure how you ended up "sort of maintaining"
the regression stuff, except that you parallelized it using waf tasks at
some point.

> >   Anyway, I realize that most people haven't looked at 
> test.py yet, so
> > please have a look.
> 
> It was diffcult to look because I could only find it hidden 
> inside a patch.
> Why not a normal mercurial branch?  

Because others have complained mightily about the noise from extraneous
commits in normal mercurial branches.  Mercurial queues let me treat
mercurial "like a fancy version enabled backup system" as Mathieu once put
it, but also keeps completely unnecessary extraneous noise out of the
history.

When it comes time for a proper review, I will create a mercurial branch in
my code.nsnam.org sandbox and apply the patch.  I agree that it is much
easier to deal with that way.

> But here some comments:
> 
>   1. Why do we have a new "TestRunner" program; I thought 
> utils/run-tests.cc
> took care of this?

The current plan is to add a new test framework which can be used for all
levels of testing.  In order to avoid disrupting the release and all of the
existing tests, a new third way has been developed.  It is expected that
"waf --check" and "waf --regression" will go unsupported and will eventually
disappear.

>   2. The multiprocessing module is new in Python 2.6;

Yes.  It has been back-ported to 2.4 and 2.5, but this dependency is going
away.  Thus, my comment,

    #
    # XXX Need to figure out number of CPUs without the multiprocessing
    # dependency since multiprocessing is not standard `till Python 2.6
    #

>   3. I hate running subprocesses a) through a shell, b) with 
> hardcoded '/'
> as path separator.  These things make the system much less portable;

(a) is a result of the prototyping the removal of waf dependencies -- it
made life easier.  It doesn't strictly need to be there.  I think (b) makes
the system *slightly* less portable.  All of our supported platforms use
file systems with slashes.  Are you concerned about MinGW and its use of
MSVCRT.DLL?  If so, I can change this for you, no problem.

>   4. The exec statment is considered evil by most Python 
> programmers; if
> there is a way around it, it should be taken;

It was an easy way to experiment with pulling in config info.  I think I
will pull out all of the waf state-related code and so this will go away.

>   5. Of course reading the waf cache data directly is a bit 
> hackish and
> might break in the future.  Perhaps implementing a --print-env wscript
> option might be better in the long run.

Sure.  But if something changes (ENABLE_NSC changes to
ENABLE_SOMETHING_ELSE) in the future and waf --print-env stops printing the
expected information *something* will need to be fixed in any case.  the
c4che is really just a python "database" holding declarations, though.  It
didn't seem too horrible to treat it that way.

But, then, it's not really important since I'm leaning toward not using any
waf state at all in the tests.
 
>   6. Do I see all the parallel executing tests run with the 
> same CWD?  This
> sort of thing is usually a killer of multiprocessing... Hint:
> subprocess.Popen() has a cwd parameter...

No, you see the example smoke tests running in the same CWD.  Currently they
all use different trace file names since they all write to the same CWD in
"normal life" so there will be no collisions.  It does make sense to create
a directory structure for unchecked trace files in /tmp in case someone
doesn't change file names in the future.  I just haven't gotten around to it
yet.

> I hope this helps.

Yes, thank you.

After quite a bit of prototyping, I've come to the conclusion that the best
way to approach this is to make something that Mr. or Ms. Average User can
easily understand and maintain.  It seems like a simple, vanilla Python
program that calls waf to run programs is all that we need.  I think that
the concepts required to understand this little program are taught by the
second quarter of beginning programming classes; and to paraphrase Tom, I
can see a lot of upside to doing it this way and very little downside.

I don't think anyone expects you to maintain this, Gustavo.  I think the
only way you will be sucked in is if I *do* write it in waf and it breaks
(or you can't stand the code).  In fact, I think the guy who gets to
maintain this has a name that sounds a lot like Craig.

Regards,

-- Craig (sound familiar?)

P.S.  If anyone else has any opinions on this, I'd like to hear them (even
privately if you prefer).