[Ns-developers] Ns-3 Regression Testing

[Mathieu Lacage]

On Tue, 2008-04-01 at 11:49 +0100, Gustavo Carneiro wrote:

> The code does not work here because >& is a bash extension, while ubuntu
> installs dash as /bin/sh which does not support >& (>& is not POSIX
> compliant).  The solution is to replace ">&" with "> /dev/null 2>&1".
> 
> Moreover, I don't like the way the regression tests hide the diff output
> (diff -q); when something fails, I would usually like to know why... diff
> already outputs nothing when there are no differences, so why not remove the
> -q?  Or maybe  diff xxxx | head, to avoid huge trace files completely
> different drowning out output.

I think that the default output should be uncluttered if only because it
would be really nice if parsing that output was really easy but, I agree
that it would be nice to output something if there is a failure. I
suspect that the most useful thing to do would be to generate a small
message to the user which describes the command to run to reproduce the
failure. i.e., it would output something like: "if you want to see what
failed, run diff -u x y" where x and y would be full filename paths.

> Finally, when we get Python bindings I think they would make writing test
> cases much easier.  Right now writing a test means writing in C++ and also
> writing a relatively complex python script.  I am sure you'll agree with me
> when I say that all test scripts share a lot of common boilerplate code
> which could be moved to a small python helper module...

It seems to me that whether or not we can move boilerplate code to a
separate python module is a different issue from being able to write a
full test script in python. I agree that both would be desirable but the
former is most likely easier to get in quickly than the latter. I think
that the original focus of this quick and dirty test framework was not
really to make it easy to write good tests but to make it easy to run a
quick smoketest to make sure we did not break anything really big. So,
in that context, allowing us to reuse the c++ scripts is really nice and
provides an immediate improvement in our confidence that a specific
release is not too broken.

Mathieu