[Ns-developers] release update

[craigdo@ee.washington.edu]

Dear All,

> > Current items still on the table for merging:
> 
> [ ... ]
> 
> >   - craig's testing: no idea about status
> 
> I'll have an update for you on Friday

Sorry for the delays, I have had to take some time off ...

---------- Current Testing Status ----------

The code is still in pieces.  I should have it all put back together next week, but Tom suggested I summarize where things are
today.  You should be able to get a flavor of what you will see then from the info below.

1) We've decided to leave ./waf check and ./waf --regression as is and invent a new framework which does it better.  We expect that
the regression and unit tests will be moved to the new framework going forward.  The existing code will be left to "wither on the
vine."  

So expect ./waf check and ./waf --regression will work as they do now for the foreseeable future; and something new appears soon.

2) I have decided that it is too painful to do all of the test running inside waf, so I have packed up and moved into a new test.py
that drives the testing process.  I expect that people will clone ns-3-allinone, then ./download.py, ./build.py and ./test.py.

Initially test.py will call ./waf check and ./waf --regression for thoroughness, but that should eventually go away.

The code in test.py runs tests in parallel based on the number of processors in the system.  It's really quite simple and much
easier to understand than, well, anything in waf.  

3) I expect there will be three main types of tests:

a.  Tests that live in a source file like the current unit tests;
b.  Tests that live in a module directory in which tests that are very large or span source files can live.
c.  Tests that span modules -- system tests or tests organized as entire scripts a la our examples.  These live in a tests directory
parallel to src.

4) The test runner will execute examples as "smoke tests" but will not do trace comparisons on them.  Trace output is just
redirected and not checked.  This is just to make sure that our examples are runnable and return a success code, not that they  are
carefully checked for anything.  Note that this means that ./waf --regression is still required at least temporarily, so regression
will still be run separately.  Which examples to run is encoded in a list.

5) Waf will build a large test file in the same way that it builds run-tests, but links the test-modules.  It will have a
command-line switch to dump all of the tests.  The test runner will dump the available tests and then execute them by spawning off
processes to run each test suite.

Like,

  ./waf --run tests --dump=true

  ./waf --run tests --testSuite=NamesTestSuite

[There can be an equivalent for python-based tests.]

6) Tests are split into TestSuite and TestCase.  Writing a new test means creating a test suite.  Something like,

  class NamesTestSuite : public TestSuite
  {
  public:
    NamesTestSuite ();
  };

  NamesTestSuite::NamesTestSuite ()
    : TestSuite ("Test suite to test the Object Name Service")
  {
    AddTestCase (new BasicAddTestCase);
    AddTestCase (new StringContextAddTestCase);
    ... 
  }

You have to write individual test cases by inheriting from a generic test case

  class BasicAddTestCase : public TestCase
  {
  public:
    BasicAddTestCase ();

  private:
    virtual bool DoRun (void);
    virtual void DoTeardown (void);
  };

  BasicAddTestCase::BasicAddTestCase ()
    : TestCase ("Check low level Names::Add and Names::FindName functionality")
  {
  }

  void
  BasicAddTestCase::DoTeardown (void)
  {
    Names::Delete ();
  }

  bool
  BasicAddTestCase::DoRun (void)
  {
    Ptr<TestObject> objectOne = CreateObject<TestObject> ();
    Names::Add (Ptr<Object> (0, false), "Name One", objectOne);

    ...
    
    found = Names::FindName (objectOne);
    TEST_ASSERT (found == "Name One", 
      "Could not Names::Add and Names::FindName an Object", testResult);
    
    return true;
  }

This is identical to most test frameworks in the real world.  The test framework will take care of the details of reporting back to
the test runner.  This way a formatter to build an HTML report can be easily added, but the first cut will just generate a text
output -- I think a text file report to start.

7) The test runner will provide a cwd for the test program when it is run.  So the test program can look for any data it needs in
the directory it is given (or in some subdirectory structure therein).  So if any test vectors are needed in the form of lists of
pieces of packets (such as headers) they can be stored in a file readable by the test.  I will provide support to read, write and
playback such files of headers.

8) I want to add support for making pretty pictures, such as the RNG chi-square results.  By this I mean gnuplot functions will be
brought into the test framework along with GSL (to get functions such as histos and CDFs)   This won't block the first release
though.

9) I will include examples of how to write a stochastic test suite

class RngTestSuite : public TestSuite
{
public:
  RngTestSuite ();
};

RngTestSuite::RngTestSuite ()
  : TestSuite ("Test suite to test the random number generators")
{
  AddTestCase (new RngUniformTestCase);
  AddTestCase (new RngNormalTestCase);
  AddTestCase (new RngExponentialTestCase);
  AddTestCase (new RngParetoTestCase);
}

I will also include a module-based test (see NamesTestSuite above).

I will also include a test suite using file-based test vectors that will check TCP connection establishment (not completely baked
but I have previewed pieces in the past).

As I said, this is still in pieces, but I am going to try and pull it together into something coherent in the next few days.

Regards,

-- Craig