[Ns-developers] release update

[Craig Dowell]

> > 2) I have decided that it is too painful to do all of the 
> test running
> > inside waf, so I have packed up and moved into a new test.py
> > that drives the testing process.  I expect that people will clone
> > ns-3-allinone, then ./download.py, ./build.py and ./test.py.
> > 
> > Initially test.py will call ./waf check and ./waf --regression for
> > thoroughness, but that should eventually go away.
> > 
> > The code in test.py runs tests in parallel based on the number of
> > processors in the system.  It's really quite simple and much
> > easier to understand than, well, anything in waf.  
> 
> I personally don't care much how the test execution is 
> implemented but I
> think that it makes sense to keep the current user interface to
> executing tests, that is, make "./waf check" run the tests.  I am sure
> gustavo will be happy to make "./waf check" execute first 
> your script in
> utils/test.py and then, execute the equivalent of --regression. 

Gustavo may want to chime in, but but I spent a considerable amount of time
trying to understand this crap in waf -- it's quite inscrutable, BTW.  Sure,
it can be done, but not easily and not in an easily understandable and
maintainable way.  Ease of use is not a quality that the internals of waf
posess.  I wrote the multiprocessing piece of test.py *much* faster than it
took me to even begin to understand the task system in waf.  This is why
Gustavo gets the pleasure of dealing with all of the non-trivial waf issues.

It seems very worthwhile to me to have a test environment that is
understandable by mere mortals in polynomial time.  It seems to me to be
counterproductive to take a simple straightforward testing process and
obfuscate it by unnaturally forcing it to look like a build process so we
can run it inside a build system.  It seems, well, ... silly.

That said, as I implied above, I was not planning to change either "waf
check" or "waf --regression" for this go.  Pulling the main test code out of
waf actually makes calling a new test function out of the "check" command
reasonable (and trivial) to do, but I think it is smarter to do it the other
way around.

However, I thought we agreed a couple of weeks ago that the new testing
framework would be a "third way" that would eventually become the official
test mechanism and that check and regression would wither on the vine.  Are
you having second thoughts?

> If
> needed (that is, gustavo has no time to do this), I will take care of
> this myself before the release. In time, if gustavo wants to make this
> more integrated in waf by rewriting it using waf tasks, he 
> will be free
> to do so.

Rewrite it?  Why write it in the first place, then?  Why don't I just try
and be more proactive and efficient, stop what I'm doing, and just hand all
of this test BS to Gustavo now?  I'm sure he'd love it.  I know I would ;-)

But, really, what is the point of integrating this all into waf?  Why is
this important?  Doing this in waf seems *only* to make life more difficult
for pretty much *everybody*.  I like the idea of:

  ./download.py
  ./build.py
  ./test.py

much more than,

  ./download.py
  ./build.py
  cd ns-3-dev
  ./waf check
  ./waf --regression
  ./waf newtestframework

Why do it that way other than because of inertia?  If I want to debug a
program I don't use ./waf --gdb. Why should I be forced to use waf for
testing, even though it ends up being ugly from several different
perspectives?  Why force tests to look like builds?  I don't get it,
especially now that ns-3-allinone (that I like very much) exists.

> To summarize, if you don't want to deal with waf, don't deal with it,
> but, please, move your script to ns-3-dev/utils/test.py and don't make
> it do anything but execute your test binary: this will make it easier
> for me to make the buildbot machinery run these tests.

I can't imagine why executing a python program called "test" is much more
difficult than executing a python program called "waf"; and I can't imagine
why executing a program that in turn runs programs called csma-one-subnet
and csma-multicast is simpler than executing one that runs the programs
run-tests and then csma-one-subnet and csma-multicast.  Please enlighten me.

As for compatibility, I've said several times, I'm not planning on changing
"waf check" or "waf --regression."  I'm planning on adding something new.
I'm not planning on upsetting the release and changing piles of existing
tests, I'm planning on adding an initially small feature that is completely
orthogonal to the existing tests.

What I want to see for a user experience is,

  hg clone http://code.nsnam.org/ns-3-allinone
  ./download.py
  ./build.py
  ./test.py

>From this perspective, I think it makes perfect sense to run check and
regression out of test.py at least initially to get reasonable test
coverage.  I think it makes sense to do this until our testing environment
is filled out, at which time there will no longer be a need to call them out
of test.py and they can eventually disappear in favor of one testing
subsystem.  I don't think it makes *any* sense to build a testing engine of
any complexity into waf.

I would agree with putting most of test.py into utils (and also putting
regression.py and wutils.py as well); but also have a script called test.py
to drive it all in an easily found place.

Regards,

-- Craig