[Ns-developers] test script and waf (was Re: release update)

[craigdo@ee.washington.edu]

[ ... ]

> 
> There is an obvious problem with running everything with waf 
> --run, which is
> that it will _much_ slower.  Each time waf runs, it checks to see if
> anything needs to be built, which means reading every source 
> file, computing
> md5sum, compare with last known signatures...  That it only 
> takes 1 or 2
> seconds each time is a small miracle of design and 
> technology, actually.

It really is kind of amazing.  I really didn't notice the difference when
running the current test suites.  It will probably become more obvious if we
get hundreds of tests in place.  However, this is easily addressed by a
command, say "exec" that doesn't call Scripting.build(bld) though; and this
may be useful in general, I think.

[ ... ]

> > Just curious.  How would you handle the following?
> >
> >  ./waf --test --verbose --html=daily.html --unit 
> --suite='Object Name
> > Service' --valgrind
> 
> If everything was integrated in waf, of course the above would be used
> directly.  I see no problem with it.

I was wondering about how you would differentiate between options for the
test program and options for waf.  The option --verbose is an option in waf
itself, and also in the test program.  How do you decide which program gets
the --verbose option?

> > I can imagine that you might do something like,
> >
> >  ./waf --test --testOpt="--verbose --html=daily.html --unit 
> --suite='Object
> > Name Service'" --valgrind
> 
> No reason why I'd have to do it like this.  Unless the test 
> runner was an
> outside script, of course...

The tests live in a C++ program and will take options.  Consider --verbose.
The tests will have a --verbose option and so does waf.  How do you decide
which --verbose you are given?

> > but it seems much more reasonable to me to do
> >
> >  ./test.py --verbose --html=daily.html --unit 
> --suite='Object Name Service'
> > --valgrind
> >
> > than to somehow disentangle/disambiguate test options from waf
> > build-related
> > options.

[ ... ]

> > I think a little self-contained, well documented piece of 
> vanilla Python,
> > without any trace of magic, is going to be much more 
> accessible for the
> > rest
> > of us mere mortals.
> 
> Fair enough.
> 
> However, I reiterate that *someone* needs to keep 
> understanding WAF because
> the build system uses WAF, including WAF tasks.

Clearly. I agree completely.  Waf is a "hurdle" all ns-3 developers must
cross to one degree or another.  It seems wrong to me to set up a situation
where a deep understanding of waf internals is a prerequisite for modifying
the test environment, though.

[ ... ]

> > I don't think anyone is asking you to maintain test.py.  My 
> main point is
> > that I want this to be easy so every time something goes 
> wrong we *don't*
> > have to email you.  BTW, I'm not sure how you ended up "sort of
> > maintaining"
> > the regression stuff, except that you parallelized it using 
> waf tasks at
> > some point.
> 
> Bugs were being reported and no one took care of fixing them. 
>  Moving to waf
> tasks seemed the best way to make regression tests run in 
> parallel, for
> instance.

Yes.  It was the easiest way particularly since it was all already in waf.
I understand.

[ ... ]

> > When it comes time for a proper review, I will create a 
> mercurial branch in
> > my code.nsnam.org sandbox and apply the patch.  I agree 
> that it is much
> > easier to deal with that way.
> 
> I don't get it.  In your "mercurial queue", at least the web 
> interface, I
> only see a patch file.  It seems a rather poor interface to 
> view files or
> changes.

It's not supposed to be an interface to view files or changes.  It's
supposed to allow me to work in an environment where I can do many *other*
things, and at the same time coexist with several of the proposed review
processes.  As I said, when it comes time for a proper review, I will create
a nicer way to view files or changes.

I don't expect anyone to review what I have now directly.  I think Tom just
pointed you at my in-process patch for context.

[ ... ]

> Sorry, in a hurry I did not read the comments.  I'm sure WAF has some
> portable code to detect CPUs ;-)

A common way (which waf also uses) seems to be:

  if os.sysconf_names.has_key('SC_NPROCESSORS_ONLN'):
      default_jobs=os.sysconf('SC_NPROCESSORS_ONLN')
  else:
      default_jobs=int(Utils.cmd_output(['sysctl','-n','hw.ncpu']))

This is another one of those things I just haven't done yet.  I started
experimenting with the multiprocessing code and just left in that one
version of the call to get the number of CPUs until I got around to fixing
it.

[ ... ]

> Just one final question.  This test.py will live the main 
> ns-3-dev tree,
> right?  ns-3-allinone is great and all, but it should continue to be
> optional, while unit tests at least should be well integrated 
> in the main
> tree, not be optional...

Yes.

> > I don't think anyone expects you to maintain this, Gustavo. 
>  I think the
> > only way you will be sucked in is if I *do* write it in waf 
> and it breaks
> > (or you can't stand the code).  In fact, I think the guy who gets to
> > maintain this has a name that sounds a lot like Craig.
> 
> 
> That's good to hear... ;-)

I thought you might appreciate it ...