[Ns-developers] Ns-3 Regression Testing

[Gustavo Carneiro]

On 01/04/2008, craigdo at ee.washington.edu <craigdo at ee.washington.edu> wrote:
>
> Dear All,
>
> Tom has asked me to put together some kind of regression test for an
> upcoming ns-3 release.  I've come up with a quick and dirty version that
> uses some of our existing examples and associated trace files to provide
> some minimal regression testing.
>
> The code lives in a repo on code.nsnam.org as usual.  You can get to it
> by,
>
>   hg clone http://code.nsnam.org/craigdo/ns-3-regression
>
> if you like.
>
> The relevant code lives in the regression directory of the repo.  There
> you
> will find a regression.py script that drives the process.
>
> Among the first things this script does is to run the unit tests (Tom
> asked
> for this).  Then it clones another repository that includes the "reference
> traces."  These are copies of the trace files that were generated by the
> tests in a reference run.  It is these trace files that are considered
> golden and to which we compare trace files (pcap- and ascii-trace) during
> regression testing.
>
> The reference traces are kept in another private repo on code.nsnam.org
> .  It
> will be cloned transparently during execution so you don't have to worry
> about that.  Eventually we will have an ns-3-ref-traces directory that
> parallels our release directory.  We have this separate repository to
> avoid
> storing the possibly very large trace files in the main code repository.
>
> You run the regression tests by changing into the regression directory and
> typing
>
>   python regression.py
>
> You'll see the usual unit rests run and then some new regression tests
> executing.
>
>   > python regression.py
>   ========== Running Unit Tests ==========
>   Entering directory `/home/craigdo/repos/ns-3-regression/build'
>   Compilation finished successfully
>   PASS AddressHelper
>   PASS Wifi
>   ...
>   PASS DcfManager
>   PASS Object
>   PASS Ptr
>   PASS Callback
>   ========== Running Regression Tests ==========
>   Synchronizing reference traces.
>   Done.
>   PASS test-csma-broadcast
>   PASS test-csma-multicast
>   PASS test-csma-one-subnet
>   PASS test-csma-packet-socket
>   PASS test-simple-error-model
>   PASS test-simple-global-routing
>   PASS test-simple-point-to-point-olsr
>   PASS test-tcp-large-transfer
>   PASS test-udp-echo
>   >
>
> Each of the regression tests (test-csma-broadcast, for example) has a
> corresponding test-csma-broadcast.py files that lives in the
> regression/tests directory.  There is a separate python script for every
> test since we may want to pick executables to run, provide different
> command
> line arguments, and check for various return codes or different outputs
> for
> each of the test-cases.
>
> The regression.py script looks for files in regression/tests that begin
> with
> "test-" and end with ".py" and runs them.  If you don't provide any test
> cases to run, all of them will be run (as shown above).  You can also pick
> out individual tests by providing their names as arguments
>
>   python regression.py test-csma-broadcast test-csma-multicast
>
> Right now the only programs provided are those that check traces.  I added
> an option to the main wscript that allows for specifying a working
> directory
> to save the traces in.  This is now used in the scripts like,
>
>   ./waf --cwd regression/traces --run csma-broadcast
>
> This causes the trace files to be written to the directory
> regression/traces
> instead of in the top-level directory.
>
> The reference traces, as I said, are held in another
> repository.  Currently
> they are pulled out of a private repo into regression/ns-3-ref-traces;
> eventually from a public repo.  There are a number of directories there,
> one
> for each test
>
>   csma-broadcast.ref/      simple-global-routing.ref/
>   csma-multicast.ref/      simple-point-to-point-olsr.ref/
>   csma-one-subnet.ref/     tcp-large-transfer.ref/
>   csma-packet-socket.ref/  udp-echo.ref/
>   simple-error-model.ref/
>
> These directories contain the "golden" trace files.  All the regression
> test
> does is to run the example, point its trace output at the
> regression/traces
> file and then diff -q the two directories.
>
> If a test fails, you will get a FAIL test-xxx.py indication, but the
> regression script will continue running.  This currently overwrites the
> single traces directory.  Question:  is it worthwhile to write the test
> output to a separate test-xxx.trace directory to catch non-repeatable
> errors?
>
> In the case of a failure, in order to see what exactly happened, you can
> run
> a single regression test by passing the test name as a parameter
>
>   python regression.py test-udp-echo
>
> for example.  When this test is done, you can look at the differences
> between the regression/traces directory and the appropriate
> ns-3-ref-traces
> directory (in this case ns-3-ref-traces/udp-echo.ref)
>
> You have to be able to generate new golden trace files.  You do this by
> using the -g (generate) option, as in
>
>   python regression.py -g test-udp-echo
>
> This will point the current working directory at the correct place in the
> regression/ns-3-ref-traces directory.  It won't check anything in.  You
> are
> expected to verify that this output is somehow "correct" and then do the
> commit and push manually.  You can generate output from all of the tests
> by
> doing
>
>   python regression.py -g
>
> Or you can provide a list of tests to generate traces from like in the
> normal case
>
>   python regression.py -g test-csma-broadcast test-csma-multicast
>
> One more change that was required was to add a call to
> RandomVariable::SetGlobalSeed to the example programs that caused random
> variables to be instantiated.  This was required to get repeatable
> results.
> An example of where randomness might be found is in the CSMA net device
> where the backoff uses a uniform random number generator.
>
> Anyway, it's not beautiful, nor is it the end of the testing road, but it
> does give us a way to detect regressions.
>
> Comments?


The code does not work here because >& is a bash extension, while ubuntu
installs dash as /bin/sh which does not support >& (>& is not POSIX
compliant).  The solution is to replace ">&" with "> /dev/null 2>&1".

Moreover, I don't like the way the regression tests hide the diff output
(diff -q); when something fails, I would usually like to know why... diff
already outputs nothing when there are no differences, so why not remove the
-q?  Or maybe  diff xxxx | head, to avoid huge trace files completely
different drowning out output.

Finally, when we get Python bindings I think they would make writing test
cases much easier.  Right now writing a test means writing in C++ and also
writing a relatively complex python script.  I am sure you'll agree with me
when I say that all test scripts share a lot of common boilerplate code
which could be moved to a small python helper module...

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert