[Ns-developers] release update

[Gustavo Carneiro]

2009/6/9 Craig Dowell <craigdo at ee.washington.edu>

>
> > > 2) I have decided that it is too painful to do all of the
> > test running
> > > inside waf, so I have packed up and moved into a new test.py
> > > that drives the testing process.  I expect that people will clone
> > > ns-3-allinone, then ./download.py, ./build.py and ./test.py.
> > >
> > > Initially test.py will call ./waf check and ./waf --regression for
> > > thoroughness, but that should eventually go away.
> > >
> > > The code in test.py runs tests in parallel based on the number of
> > > processors in the system.  It's really quite simple and much
> > > easier to understand than, well, anything in waf.
> >
> > I personally don't care much how the test execution is
> > implemented but I
> > think that it makes sense to keep the current user interface to
> > executing tests, that is, make "./waf check" run the tests.  I am sure
> > gustavo will be happy to make "./waf check" execute first
> > your script in
> > utils/test.py and then, execute the equivalent of --regression.
>
> Gustavo may want to chime in, but but I spent a considerable amount of time
> trying to understand this crap in waf -- it's quite inscrutable, BTW.
>  Sure,
> it can be done, but not easily and not in an easily understandable and
> maintainable way.  Ease of use is not a quality that the internals of waf
> posess.  I wrote the multiprocessing piece of test.py *much* faster than it
> took me to even begin to understand the task system in waf.  This is why
> Gustavo gets the pleasure of dealing with all of the non-trivial waf
> issues.
>
> It seems very worthwhile to me to have a test environment that is
> understandable by mere mortals in polynomial time.  It seems to me to be
> counterproductive to take a simple straightforward testing process and
> obfuscate it by unnaturally forcing it to look like a build process so we
> can run it inside a build system.  It seems, well, ... silly.
>
> That said, as I implied above, I was not planning to change either "waf
> check" or "waf --regression" for this go.  Pulling the main test code out
> of
> waf actually makes calling a new test function out of the "check" command
> reasonable (and trivial) to do, but I think it is smarter to do it the
> other
> way around.
>
> However, I thought we agreed a couple of weeks ago that the new testing
> framework would be a "third way" that would eventually become the official
> test mechanism and that check and regression would wither on the vine.  Are
> you having second thoughts?
>
> > If
> > needed (that is, gustavo has no time to do this), I will take care of
> > this myself before the release. In time, if gustavo wants to make this
> > more integrated in waf by rewriting it using waf tasks, he
> > will be free
> > to do so.
>
> Rewrite it?  Why write it in the first place, then?  Why don't I just try
> and be more proactive and efficient, stop what I'm doing, and just hand all
> of this test BS to Gustavo now?  I'm sure he'd love it.  I know I would ;-)
>
> But, really, what is the point of integrating this all into waf?  Why is
> this important?  Doing this in waf seems *only* to make life more difficult
> for pretty much *everybody*.  I like the idea of:
>
>  ./download.py
>  ./build.py
>  ./test.py
>
> much more than,
>
>  ./download.py
>  ./build.py
>  cd ns-3-dev
>  ./waf check
>  ./waf --regression
>  ./waf newtestframework
>
> Why do it that way other than because of inertia?  If I want to debug a
> program I don't use ./waf --gdb. Why should I be forced to use waf for
> testing, even though it ends up being ugly from several different
> perspectives?  Why force tests to look like builds?  I don't get it,
> especially now that ns-3-allinone (that I like very much) exists.
>
> > To summarize, if you don't want to deal with waf, don't deal with it,
> > but, please, move your script to ns-3-dev/utils/test.py and don't make
> > it do anything but execute your test binary: this will make it easier
> > for me to make the buildbot machinery run these tests.
>
> I can't imagine why executing a python program called "test" is much more
> difficult than executing a python program called "waf"; and I can't imagine
> why executing a program that in turn runs programs called csma-one-subnet
> and csma-multicast is simpler than executing one that runs the programs
> run-tests and then csma-one-subnet and csma-multicast.  Please enlighten
> me.
>
> As for compatibility, I've said several times, I'm not planning on changing
> "waf check" or "waf --regression."  I'm planning on adding something new.
> I'm not planning on upsetting the release and changing piles of existing
> tests, I'm planning on adding an initially small feature that is completely
> orthogonal to the existing tests.
>
> What I want to see for a user experience is,
>
>  hg clone http://code.nsnam.org/ns-3-allinone
>  ./download.py
>  ./build.py
>  ./test.py
>
> >From this perspective, I think it makes perfect sense to run check and
> regression out of test.py at least initially to get reasonable test
> coverage.  I think it makes sense to do this until our testing environment
> is filled out, at which time there will no longer be a need to call them
> out
> of test.py and they can eventually disappear in favor of one testing
> subsystem.  I don't think it makes *any* sense to build a testing engine of
> any complexity into waf.
>
> I would agree with putting most of test.py into utils (and also putting
> regression.py and wutils.py as well); but also have a script called test.py
> to drive it all in an easily found place.


Hi Craig,

I share your sentiments regarding WAF internals being ugly.  This is a
reflection of the way WAF is being maintained, and I lost this battle a long
time ago :-(  I would have forked WAF a long time ago if only I had time to
maintain the fork...

I think it is plausible to have the testing outside of WAF, if you can
easily code a task scheduler in python that handles concurrency and task
dependencies; however, you have to agree it should at least be invoked from
WAF.  That is, "./waf check" should invoke the test script.  The test script
should accept parameters to indicate what build variant we are testing
(debug/release), and any other useful information that waf knows more
easily.

Regards,

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert