[Ns-developers] test script and waf (was Re: release update)

[Gustavo Carneiro]

2009/6/16 <craigdo at ee.washington.edu>

> [ ... ]
>
> >
> > There is an obvious problem with running everything with waf
> > --run, which is
> > that it will _much_ slower.  Each time waf runs, it checks to see if
> > anything needs to be built, which means reading every source
> > file, computing
> > md5sum, compare with last known signatures...  That it only
> > takes 1 or 2
> > seconds each time is a small miracle of design and
> > technology, actually.
>
> It really is kind of amazing.  I really didn't notice the difference when
> running the current test suites.  It will probably become more obvious if
> we
> get hundreds of tests in place.  However, this is easily addressed by a
> command, say "exec" that doesn't call Scripting.build(bld) though; and this
> may be useful in general, I think.


Easier said than done.

Sure, we can add an exec command that doesn't run the build.  But at the
same time, the exec command will not read the build tree (task generators),
and so the programs will not be declared.  If the programs are not declared
then we cannot automatically determine where they are located.  Even loading
the Waf Environment from a non-build context does not appear to be easy, so
getting the currently configured variant name is not easy.


>
>
> [ ... ]
>
> > > Just curious.  How would you handle the following?
> > >
> > >  ./waf --test --verbose --html=daily.html --unit
> > --suite='Object Name
> > > Service' --valgrind
> >
> > If everything was integrated in waf, of course the above would be used
> > directly.  I see no problem with it.
>
> I was wondering about how you would differentiate between options for the
> test program and options for waf.  The option --verbose is an option in waf
> itself, and also in the test program.  How do you decide which program gets
> the --verbose option?
>
> > > I can imagine that you might do something like,
> > >
> > >  ./waf --test --testOpt="--verbose --html=daily.html --unit
> > --suite='Object
> > > Name Service'" --valgrind
> >
> > No reason why I'd have to do it like this.  Unless the test
> > runner was an
> > outside script, of course...
>
> The tests live in a C++ program and will take options.  Consider --verbose.
> The tests will have a --verbose option and so does waf.  How do you decide
> which --verbose you are given?


The same --verbose option can control both.  When launching the test
program, we add --verbose if waf itself received --verbose.


>
>
> > > but it seems much more reasonable to me to do
> > >
> > >  ./test.py --verbose --html=daily.html --unit
> > --suite='Object Name Service'
> > > --valgrind
> > >
> > > than to somehow disentangle/disambiguate test options from waf
> > > build-related
> > > options.
>
> [ ... ]
>
> > > I think a little self-contained, well documented piece of
> > vanilla Python,
> > > without any trace of magic, is going to be much more
> > accessible for the
> > > rest
> > > of us mere mortals.
> >
> > Fair enough.
> >
> > However, I reiterate that *someone* needs to keep
> > understanding WAF because
> > the build system uses WAF, including WAF tasks.
>
> Clearly. I agree completely.  Waf is a "hurdle" all ns-3 developers must
> cross to one degree or another.  It seems wrong to me to set up a situation
> where a deep understanding of waf internals is a prerequisite for modifying
> the test environment, though.


OK, fair enough.


>
>
> [ ... ]
>
> > > I don't think anyone is asking you to maintain test.py.  My
> > main point is
> > > that I want this to be easy so every time something goes
> > wrong we *don't*
> > > have to email you.  BTW, I'm not sure how you ended up "sort of
> > > maintaining"
> > > the regression stuff, except that you parallelized it using
> > waf tasks at
> > > some point.
> >
> > Bugs were being reported and no one took care of fixing them.
> >  Moving to waf
> > tasks seemed the best way to make regression tests run in
> > parallel, for
> > instance.
>
> Yes.  It was the easiest way particularly since it was all already in waf.
> I understand.
>
> [ ... ]
>
> > > When it comes time for a proper review, I will create a
> > mercurial branch in
> > > my code.nsnam.org sandbox and apply the patch.  I agree
> > that it is much
> > > easier to deal with that way.
> >
> > I don't get it.  In your "mercurial queue", at least the web
> > interface, I
> > only see a patch file.  It seems a rather poor interface to
> > view files or
> > changes.
>
> It's not supposed to be an interface to view files or changes.  It's
> supposed to allow me to work in an environment where I can do many *other*
> things, and at the same time coexist with several of the proposed review
> processes.  As I said, when it comes time for a proper review, I will
> create
> a nicer way to view files or changes.
>
> I don't expect anyone to review what I have now directly.  I think Tom just
> pointed you at my in-process patch for context.
>
> [ ... ]
>
> > Sorry, in a hurry I did not read the comments.  I'm sure WAF has some
> > portable code to detect CPUs ;-)
>
> A common way (which waf also uses) seems to be:
>
>  if os.sysconf_names.has_key('SC_NPROCESSORS_ONLN'):
>      default_jobs=os.sysconf('SC_NPROCESSORS_ONLN')
>  else:
>      default_jobs=int(Utils.cmd_output(['sysctl','-n','hw.ncpu']))
>
> This is another one of those things I just haven't done yet.  I started
> experimenting with the multiprocessing code and just left in that one
> version of the call to get the number of CPUs until I got around to fixing
> it.
>
> [ ... ]
>
> > Just one final question.  This test.py will live the main
> > ns-3-dev tree,
> > right?  ns-3-allinone is great and all, but it should continue to be
> > optional, while unit tests at least should be well integrated
> > in the main
> > tree, not be optional...
>
> Yes.
>
> > > I don't think anyone expects you to maintain this, Gustavo.
> >  I think the
> > > only way you will be sucked in is if I *do* write it in waf
> > and it breaks
> > > (or you can't stand the code).  In fact, I think the guy who gets to
> > > maintain this has a name that sounds a lot like Craig.
> >
> >
> > That's good to hear... ;-)
>
> I thought you might appreciate it ...
>


-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert