[Ns-developers] Ns-3 Regression Testing

[craigdo@ee.washington.edu]

> On Tue, Apr 1, 2008 at 12:24 AM,  <craigdo at ee.washington.edu> wrote:
> >  [snip]
> >  The reference traces are kept in another private repo on 
> code.nsnam.org.  It
> >  will be cloned transparently during execution so you don't 
> have to worry
> >  about that.  Eventually we will have an ns-3-ref-traces 
> directory that
> >  parallels our release directory.  We have this separate 
> repository to avoid
> >  storing the possibly very large trace files in the main 
> code repository.
> 
> The clone referred to means "hg clone".  I am fine with this for
> developers, but it adds an extra dependency on mercurial for other
> regular users who will have gotten ns-3 from a tarball release (they
> have no need for mercurial).  

I have assumed that regression testing is done by developers who are making
changes and need to ensure they have not introduced a regression; and by
someone doing a release as a pass/fail smoke-type test to make sure that
things are consistent.

Passing a regression test would probably give warm and fuzzy feelings to
regular users, that might only have a partial or strange development
environment, though.

> One of the most frustrating things is
> unexpected dependencies when running a script you didn't write.  

Agreed.  There's a tradeoff, though, as you might expect.

> I'm
> also assuming here that the regression script will run automatically
> at the end of compilation eventually, much like ns-2's installer does
> validation at its end?

There have been conversations regarding a real validation test which this
explicitly is not.  I had not assumed that any test script will run
automatically at the end of a compilation.
 
> Could we tarball up the reference traces with everything else for
> releases?  Since the concern is unnecessarily bloating our repos /
> tarballs with trace files however, the other option I see to perform
> the clone is to use something that a regular non-developer type is
> more likely to have installed:  something like scp, rsync, wget, curl,
> etc.  I can't imagine there is a *nix machine out there that doesn't
> have scp, although I'm not sure if it's presence is part of the POSIX
> spec.

As I said earlier, there are tradeoffs.  First, I believe the reference
trace files should be under version control somehow.  These trace files need
to be associated somehow with a particular release and a particular
development snapshot.  So there should be a repository associated with
ns-3.x, ns-3-dev, etc.  Now, during the course of everyday events,
developers shouldn't have to worry about making new tarballs; they should
just be able generate new bits easily and commit and push them -- so there
really is a real need for a mercurial-based regression solution for
developers.

Your argument that people who download tarballs should be able to get a warm
and fuzzy feeling by running regression tests is a good one.  This means a
separate way to get at the reference traces for them, though.  I'll add a
check for mercurial and if it fails, I'll go out to a well-known place on
the web and look for a tarball.

This does mean more complexity in the release process, but it sounds like
it's worth it.

-- Craig