[Ns-developers] release update

[Mathieu Lacage]

On Tue, 2009-06-09 at 12:57 -0700, Craig Dowell wrote:

> > I personally don't care much how the test execution is 
> > implemented but I
> > think that it makes sense to keep the current user interface to
> > executing tests, that is, make "./waf check" run the tests.  I am sure
> > gustavo will be happy to make "./waf check" execute first 
> > your script in
> > utils/test.py and then, execute the equivalent of --regression. 
> 
> Gustavo may want to chime in, but but I spent a considerable amount of time
> trying to understand this crap in waf -- it's quite inscrutable, BTW.  Sure,

yes.

> it can be done, but not easily and not in an easily understandable and
> maintainable way.  Ease of use is not a quality that the internals of waf
> posess.  I wrote the multiprocessing piece of test.py *much* faster than it
> took me to even begin to understand the task system in waf.  This is why
> Gustavo gets the pleasure of dealing with all of the non-trivial waf issues.

yes.

> It seems very worthwhile to me to have a test environment that is
> understandable by mere mortals in polynomial time.  It seems to me to be
> counterproductive to take a simple straightforward testing process and
> obfuscate it by unnaturally forcing it to look like a build process so we
> can run it inside a build system.  It seems, well, ... silly.
> 
> That said, as I implied above, I was not planning to change either "waf
> check" or "waf --regression" for this go.  Pulling the main test code out of
> waf actually makes calling a new test function out of the "check" command
> reasonable (and trivial) to do, but I think it is smarter to do it the other
> way around.
> 
> However, I thought we agreed a couple of weeks ago that the new testing
> framework would be a "third way" that would eventually become the official
> test mechanism and that check and regression would wither on the vine.  Are
> you having second thoughts?
>
> > If
> > needed (that is, gustavo has no time to do this), I will take care of
> > this myself before the release. In time, if gustavo wants to make this
> > more integrated in waf by rewriting it using waf tasks, he 
> > will be free
> > to do so.
> 
> Rewrite it?  Why write it in the first place, then?  Why don't I just try
> and be more proactive and efficient, stop what I'm doing, and just hand all
> of this test BS to Gustavo now?  I'm sure he'd love it.  I know I would ;-)

I certainly think that gustavo would be willing to help come up with the
waf bits of the test BS. I also would be willing to do this. None of us
are asking (or asked) you to do it.

> But, really, what is the point of integrating this all into waf?  Why is
> this important?  Doing this in waf seems *only* to make life more difficult
> for pretty much *everybody*.  I like the idea of:
> 
>   ./download.py
>   ./build.py
>   ./test.py
> 
> much more than,
> 
>   ./download.py
>   ./build.py
>   cd ns-3-dev
>   ./waf check
>   ./waf --regression
>   ./waf newtestframework
> 
> Why do it that way other than because of inertia?  If I want to debug a
> program I don't use ./waf --gdb. Why should I be forced to use waf for
> testing, even though it ends up being ugly from several different
> perspectives?  Why force tests to look like builds?  I don't get it,

Because this is the way tests have traditionally been run in the free
software world. i.e., "make test" works for every autotooled package
and, whether we like it or not, we are trying to be good members of that
community which includes trying to adopt its conventions.

> especially now that ns-3-allinone (that I like very much) exists.
>
> > To summarize, if you don't want to deal with waf, don't deal with it,
> > but, please, move your script to ns-3-dev/utils/test.py and don't make
> > it do anything but execute your test binary: this will make it easier
> > for me to make the buildbot machinery run these tests.
> 
> I can't imagine why executing a python program called "test" is much more
> difficult than executing a python program called "waf"; and I can't imagine
> why executing a program that in turn runs programs called csma-one-subnet
> and csma-multicast is simpler than executing one that runs the programs
> run-tests and then csma-one-subnet and csma-multicast.  Please enlighten me.
> 
> As for compatibility, I've said several times, I'm not planning on changing
> "waf check" or "waf --regression."  I'm planning on adding something new.
> I'm not planning on upsetting the release and changing piles of existing
> tests, I'm planning on adding an initially small feature that is completely
> orthogonal to the existing tests.
> 
> What I want to see for a user experience is,
> 
>   hg clone http://code.nsnam.org/ns-3-allinone
>   ./download.py
>   ./build.py
>   ./test.py
> 
> From this perspective, I think it makes perfect sense to run check and
> regression out of test.py at least initially to get reasonable test
> coverage.  I think it makes sense to do this until our testing environment
> is filled out, at which time there will no longer be a need to call them out
> of test.py and they can eventually disappear in favor of one testing
> subsystem.  I don't think it makes *any* sense to build a testing engine of
> any complexity into waf.
> 
> I would agree with putting most of test.py into utils (and also putting
> regression.py and wutils.py as well); but also have a script called test.py

For me, the utils directory contains scripts which can be used by a
user. wutils.py can't be run by a user directly. The same goes for
regression.py. 

> to drive it all in an easily found place.

Adding a test.py in allinone to drive the one in ns-3-dev/utils would be
fine with me.

Again, I don't intend to force you or force anyone to use "./waf check"
but it's pretty clear to me that we would like to support this at some
point before the release. And no, as I said in my first email, I am not
asking you to do it: I am merely asking you to not make it painful to do
it and gustavo or myself will deal with it later.

Mathieu