[Ns-developers] [brain dump] testing ideas

[Mathieu Lacage]

hi,

Since I had to spend more time than usual tracking bugs and trying to
fix them recently, here is a brain dump of things which occured to me
while working on our code. I don't have time to work on any of this,
but, well, I thought that it would make sense to dump this here in case
someone has spare time.

Something I do a lot is get a 'clean' version of ns-3-dev from
code.nsnam.org, compile/test it, and then, apply a patch on it, compile/
test it. Most of the patches I test that way should not change the
behavior of ns-3. i.e., these patches are code reorganizations which I
often perform _before_ making behavioral-incompatible changes/fixing
bugs. 

In that context, ideally, what I would like to be able to do is to make
sure that as many example programs as possible behave _exactly_ as they
used to before. The easiest way to do this would be:

  - after downloading the code from code.nsnam.org, run every example
program we have, gather automatically all their output somewhere to use
them later as reference output.
  - after applying the patch, generate a new set of output for all
example programs
  - compare both sets of output
  - if there are differences, try to analyze them. Ideally, there would
be an easy-to-use tool which does not do a stupid diff. For example, it
should be possible to say, I want to diff the pcap files of this
example, using tcpdump (i.e., run tcpdump on the pcap files, and diff -u
the output). Another thing I want to do is diff the raw binary output of
the pcap files using hexdump -C as a filter when tcpdump -r does not
give me enough.

Basically, what I am trying to get to here is that, most likely, I don't
need to download reference traces: I can regenerate them because I have
a reference source tree. I also typically want reference traces for
_everything_. And, finally, I need to have some simple tools to do pcap
file diffs to analyze differences. And, yes, I do have enough cpu
resources to make all of this doable, and I would be willing to wait
20mins per test cycle (not compiling, just running examples and diffing
the outputs). I probably would be willing to wait longer for large,
complex, and risky changes to give me more confidence in the change I am
making.

Mathieu