[Ns-developers] test script and waf (was Re: release update)

[Gustavo Carneiro]

2009/6/15 <craigdo at ee.washington.edu>

>
> > >>    2) one argument was that waf knows about the build
> > context to run
> > >>    the right tests. However, whatever cached configure information
> > >>    (build/c4che/) that waf saves and uses to figure out
> > what tests to
> > >>    run should be also readable by test.py.
> > >>
> > >
> > > I think the above logic is now in test.py.
>
> I'm not sure that this actually needs to be in there.  I'm just
> experimenting with it right now.  Waf builds and links based on its
> configuration.  If it doesn't build a file or module, the tests for that
> file or module won't be built.  In the case of the test runner, this means
> that the tests won't be listed by the runner and won't be run.  It all just
> takes care of itself.
>
> Examples is a little different.  We want to run all of the examples we can
> as a "smoke test."  Not all of the examples are suitable to be
> automatically
> run (emulation, for example), so I put in a list of examples that should be
> run.  One way to deal with the case where a given example is not built due
> to configuration is to simply check for existence.  If an example doesn't
> exist, then test.py can just assume it's because of configuration.  A
> counter-argument is that this would be a good place to do a last sanity
> check and if, for example, ENABLE_NSC is true then the NSC examples aught
> to
> be built and runnable on that system.
>
> [Of course it's easier to check for existence if you know where the
> executable lives; which means it is nicer if you have internal waf state;
> but if you have internal waf state you don't need to check for existence;
> but you can easily add a "list" executables command to waf to find out what
> you need -- so there many tradeoffs in this deal that need to be
> considered.]
>
> I originally just ran everything using "./waf --run" since it seemed to
> cover the bases just fine.  One thing that bothered me was forking a
> process
> to run waf that forked a process to run the test executable.  This is why I
> prototyped a way to directly run the executable.  I'm about ready to go
> back
> to using waf run, though, since it seems to be much cleaner despite the
> extra forks.  I'm a big fan of simple and clean.


There is an obvious problem with running everything with waf --run, which is
that it will _much_ slower.  Each time waf runs, it checks to see if
anything needs to be built, which means reading every source file, computing
md5sum, compare with last known signatures...  That it only takes 1 or 2
seconds each time is a small miracle of design and technology, actually.


>
>
> > >>    3) Since test.py (or any test script) will be using arguments to
> > >>    control the fine-grained behavior, it will get
> > confusing to detangle
> > >>    the command-line arguments if waf runs tests.  You can
> > see a little
> > >>    bit of this when you try to run ./waf --run program,
> > when "program"
> > >>    has command line arguments-- you have to put quotes around the
> > >>    program and arguments so waf associates the arguments
> > with the ns-3
> > >>    program.  How will waf know what arguments are for it
> > and what for
> > >>    the test suite.  I am sure that Gustavo can figure out
> > some kind of
> > >>    argument parsing logic for this, but again, what is the
> > point, and
> > >>    won't it be more complicated syntactically?
> > >>
> > >>
> > >> Now I may be missing somthing.  Why exactly does a test
> > suite need to
> > >> receive arguments for?  I thought tests were supposed to
> > be fully automated;
> > >> taking command-line arguments kind of defeats the purpose, no?
> > >>
> > >
> > > For instance, if you want to just run a particular test
> > case instead of all
> > > of them, such as we now do in ./waf --regression.  Or
> > perhaps to turn up
> > > verbosity levels.
> >
> > Ok, but those are options for the test runner, not for
> > individual tests.  I
> > see no problem in command-line handling with or without waf.
>
> Just curious.  How would you handle the following?
>
>  ./waf --test --verbose --html=daily.html --unit --suite='Object Name
> Service' --valgrind


If everything was integrated in waf, of course the above would be used
directly.  I see no problem with it.


>
>
> I can imagine that you might do something like,
>
>  ./waf --test --testOpt="--verbose --html=daily.html --unit --suite='Object
> Name Service'" --valgrind


No reason why I'd have to do it like this.  Unless the test runner was an
outside script, of course...


>
>
> but it seems much more reasonable to me to do
>
>  ./test.py --verbose --html=daily.html --unit --suite='Object Name Service'
> --valgrind
>
> than to somehow disentangle/disambiguate test options from waf
> build-related
> options.
>
> > > I'm mainly questioning why you or anyone spends their time
> > pushing this
> > > into waf given Craig's test.py,
> >
> > I am merely demonstrated that certain advanced things can be
> > done in waf.
> > It took some bug fixing in waf itself, but it got done ;-)
>
> It can clearly all be done in waf.  I am looking at how much of the
> implementation makes sense in waf and how much in simpler external
> programs;
> and how much easier it is for normal people to deal with the combinations.
>
> > > and even if you want to do it for fun or personal
> > preference, is it really
> > > going to be better in the long run for future users and maintainers?
> >
> > Well, the thing is, a lot of stuff is already happening using
> > waf tasks and
> > task generators.  So, unless you want to replace waf with
> > something else,
> > future maintainers already will have to learn about waf tasks
> > anyway.  With
> > test.py, you have to learn not only about waf tasks but also
> > about python
> > queues and threads.  It is not clear to me which situation is worse.
>
> Well, it's not clear to me that you are the right person to answer that
> question, since you are our waf guru :-)  It may seem relatively easy to
> you, but I'm not sure the average person who wants to add a test feature is
> going to be up for "some bug fixing in waf" in order to get it done.  I
> don't know about anyone else, but I find doing something new in waf ...
> difficult.
>
> I think a little self-contained, well documented piece of vanilla Python,
> without any trace of magic, is going to be much more accessible for the
> rest
> of us mere mortals.


Fair enough.

However, I reiterate that *someone* needs to keep understanding WAF because
the build system uses WAF, including WAF tasks.



>
>
> > My personal preference has been indeed for waf, due to less
> > incremental work
> > needed to get the job done.
> >
> > That being said, if it is decided to use test.py, be my
> > guest, and we can
> > just rip out my waf code, I won't mind.  But in that case I
> > would rather not
> > become the maintainer of that script, which I did not write.
> > I remember
> > last time someone else wrote a regression.py script which I
> > ended up sort of
> > maintaining... :-)
>
> I don't think anyone is asking you to maintain test.py.  My main point is
> that I want this to be easy so every time something goes wrong we *don't*
> have to email you.  BTW, I'm not sure how you ended up "sort of
> maintaining"
> the regression stuff, except that you parallelized it using waf tasks at
> some point.


Bugs were being reported and no one took care of fixing them.  Moving to waf
tasks seemed the best way to make regression tests run in parallel, for
instance.


>
>
> > >   Anyway, I realize that most people haven't looked at
> > test.py yet, so
> > > please have a look.
> >
> > It was diffcult to look because I could only find it hidden
> > inside a patch.
> > Why not a normal mercurial branch?
>
> Because others have complained mightily about the noise from extraneous
> commits in normal mercurial branches.  Mercurial queues let me treat
> mercurial "like a fancy version enabled backup system" as Mathieu once put
> it, but also keeps completely unnecessary extraneous noise out of the
> history.
>
> When it comes time for a proper review, I will create a mercurial branch in
> my code.nsnam.org sandbox and apply the patch.  I agree that it is much
> easier to deal with that way.


I don't get it.  In your "mercurial queue", at least the web interface, I
only see a patch file.  It seems a rather poor interface to view files or
changes.


>
>
> > But here some comments:
> >
> >   1. Why do we have a new "TestRunner" program; I thought
> > utils/run-tests.cc
> > took care of this?
>
> The current plan is to add a new test framework which can be used for all
> levels of testing.  In order to avoid disrupting the release and all of the
> existing tests, a new third way has been developed.  It is expected that
> "waf --check" and "waf --regression" will go unsupported and will
> eventually
> disappear.
>
> >   2. The multiprocessing module is new in Python 2.6;
>
> Yes.  It has been back-ported to 2.4 and 2.5, but this dependency is going
> away.  Thus, my comment,
>
>    #
>    # XXX Need to figure out number of CPUs without the multiprocessing
>    # dependency since multiprocessing is not standard `till Python 2.6
>     #
>

Sorry, in a hurry I did not read the comments.  I'm sure WAF has some
portable code to detect CPUs ;-)


>
>
> >   3. I hate running subprocesses a) through a shell, b) with
> > hardcoded '/'
> > as path separator.  These things make the system much less portable;
>
> (a) is a result of the prototyping the removal of waf dependencies -- it
> made life easier.  It doesn't strictly need to be there.  I think (b) makes
> the system *slightly* less portable.  All of our supported platforms use
> file systems with slashes.  Are you concerned about MinGW and its use of
> MSVCRT.DLL?  If so, I can change this for you, no problem.
>
> >   4. The exec statment is considered evil by most Python
> > programmers; if
> > there is a way around it, it should be taken;
>
> It was an easy way to experiment with pulling in config info.  I think I
> will pull out all of the waf state-related code and so this will go away.
>
> >   5. Of course reading the waf cache data directly is a bit
> > hackish and
> > might break in the future.  Perhaps implementing a --print-env wscript
> > option might be better in the long run.
>
> Sure.  But if something changes (ENABLE_NSC changes to
> ENABLE_SOMETHING_ELSE) in the future and waf --print-env stops printing the
> expected information *something* will need to be fixed in any case.  the
> c4che is really just a python "database" holding declarations, though.  It
> didn't seem too horrible to treat it that way.
>
> But, then, it's not really important since I'm leaning toward not using any
> waf state at all in the tests.


OK, but if you call waf --run 20 times it will be much slower...


>
>
> >   6. Do I see all the parallel executing tests run with the
> > same CWD?  This
> > sort of thing is usually a killer of multiprocessing... Hint:
> > subprocess.Popen() has a cwd parameter...
>
> No, you see the example smoke tests running in the same CWD.  Currently
> they
> all use different trace file names since they all write to the same CWD in
> "normal life" so there will be no collisions.  It does make sense to create
> a directory structure for unchecked trace files in /tmp in case someone
> doesn't change file names in the future.  I just haven't gotten around to
> it
> yet.


Yes, please pay great attention to this from the start.  When I parallelized
the regresion tests, which I think you wrote previously, one of the hardest
things to do was to fix this issue.  That and the constant chdir's which
leave the reader of the code utterly confused about what any relative path
means anymore.


>
>
> > I hope this helps.
>
> Yes, thank you.
>
> After quite a bit of prototyping, I've come to the conclusion that the best
> way to approach this is to make something that Mr. or Ms. Average User can
> easily understand and maintain.  It seems like a simple, vanilla Python
> program that calls waf to run programs is all that we need.  I think that
> the concepts required to understand this little program are taught by the
> second quarter of beginning programming classes; and to paraphrase Tom, I
> can see a lot of upside to doing it this way and very little downside.


Just one final question.  This test.py will live the main ns-3-dev tree,
right?  ns-3-allinone is great and all, but it should continue to be
optional, while unit tests at least should be well integrated in the main
tree, not be optional...


>
>
> I don't think anyone expects you to maintain this, Gustavo.  I think the
> only way you will be sucked in is if I *do* write it in waf and it breaks
> (or you can't stand the code).  In fact, I think the guy who gets to
> maintain this has a name that sounds a lot like Craig.


That's good to hear... ;-)

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert