[Ns-developers] Ns-3 Regression Testing

[craigdo@ee.washington.edu]

Dear All,

Tom has asked me to put together some kind of regression test for an
upcoming ns-3 release.  I've come up with a quick and dirty version that
uses some of our existing examples and associated trace files to provide
some minimal regression testing.

The code lives in a repo on code.nsnam.org as usual.  You can get to it by,

  hg clone http://code.nsnam.org/craigdo/ns-3-regression

if you like.

The relevant code lives in the regression directory of the repo.  There you
will find a regression.py script that drives the process.

Among the first things this script does is to run the unit tests (Tom asked
for this).  Then it clones another repository that includes the "reference
traces."  These are copies of the trace files that were generated by the
tests in a reference run.  It is these trace files that are considered
golden and to which we compare trace files (pcap- and ascii-trace) during
regression testing.

The reference traces are kept in another private repo on code.nsnam.org.  It
will be cloned transparently during execution so you don't have to worry
about that.  Eventually we will have an ns-3-ref-traces directory that
parallels our release directory.  We have this separate repository to avoid
storing the possibly very large trace files in the main code repository.

You run the regression tests by changing into the regression directory and
typing

  python regression.py

You'll see the usual unit rests run and then some new regression tests
executing.

  > python regression.py
  ========== Running Unit Tests ==========
  Entering directory `/home/craigdo/repos/ns-3-regression/build'
  Compilation finished successfully
  PASS AddressHelper
  PASS Wifi
  ...
  PASS DcfManager
  PASS Object
  PASS Ptr
  PASS Callback
  ========== Running Regression Tests ==========
  Synchronizing reference traces.
  Done.
  PASS test-csma-broadcast
  PASS test-csma-multicast
  PASS test-csma-one-subnet
  PASS test-csma-packet-socket
  PASS test-simple-error-model
  PASS test-simple-global-routing
  PASS test-simple-point-to-point-olsr
  PASS test-tcp-large-transfer
  PASS test-udp-echo
  >

Each of the regression tests (test-csma-broadcast, for example) has a
corresponding test-csma-broadcast.py files that lives in the
regression/tests directory.  There is a separate python script for every
test since we may want to pick executables to run, provide different command
line arguments, and check for various return codes or different outputs for
each of the test-cases.

The regression.py script looks for files in regression/tests that begin with
"test-" and end with ".py" and runs them.  If you don't provide any test
cases to run, all of them will be run (as shown above).  You can also pick
out individual tests by providing their names as arguments

  python regression.py test-csma-broadcast test-csma-multicast

Right now the only programs provided are those that check traces.  I added
an option to the main wscript that allows for specifying a working directory
to save the traces in.  This is now used in the scripts like,

  ./waf --cwd regression/traces --run csma-broadcast

This causes the trace files to be written to the directory regression/traces
instead of in the top-level directory.

The reference traces, as I said, are held in another repository.  Currently
they are pulled out of a private repo into regression/ns-3-ref-traces;
eventually from a public repo.  There are a number of directories there, one
for each test 

  csma-broadcast.ref/      simple-global-routing.ref/
  csma-multicast.ref/      simple-point-to-point-olsr.ref/
  csma-one-subnet.ref/     tcp-large-transfer.ref/
  csma-packet-socket.ref/  udp-echo.ref/
  simple-error-model.ref/

These directories contain the "golden" trace files.  All the regression test
does is to run the example, point its trace output at the regression/traces
file and then diff -q the two directories.

If a test fails, you will get a FAIL test-xxx.py indication, but the
regression script will continue running.  This currently overwrites the
single traces directory.  Question:  is it worthwhile to write the test
output to a separate test-xxx.trace directory to catch non-repeatable
errors?

In the case of a failure, in order to see what exactly happened, you can run
a single regression test by passing the test name as a parameter

  python regression.py test-udp-echo

for example.  When this test is done, you can look at the differences
between the regression/traces directory and the appropriate ns-3-ref-traces
directory (in this case ns-3-ref-traces/udp-echo.ref)

You have to be able to generate new golden trace files.  You do this by
using the -g (generate) option, as in

  python regression.py -g test-udp-echo

This will point the current working directory at the correct place in the
regression/ns-3-ref-traces directory.  It won't check anything in.  You are
expected to verify that this output is somehow "correct" and then do the
commit and push manually.  You can generate output from all of the tests by
doing

  python regression.py -g

Or you can provide a list of tests to generate traces from like in the
normal case

  python regression.py -g test-csma-broadcast test-csma-multicast

One more change that was required was to add a call to
RandomVariable::SetGlobalSeed to the example programs that caused random
variables to be instantiated.  This was required to get repeatable results.
An example of where randomness might be found is in the CSMA net device
where the backoff uses a uniform random number generator.

Anyway, it's not beautiful, nor is it the end of the testing road, but it
does give us a way to detect regressions.

Comments?

-- Craig