[Ns-developers] Request for Merge -- Validation

[craigdo@ee.washington.edu]

I was asked to provide a patch for ns-3.5 by last Monday that allows us to run test programs without relying on automatic trace file
comparisons using --regression.

The patch I sent provides a way to run a non-trace-comparing test program under the control of waf when using the --regression
option.  It adds a few test programs that do useful things, but are mostly there to give an illustration of how to use it.  It was
socialized on Monday.  That's about as far as I thought this would go.

Most of the comments I've seen go waaaay further than that.  Now replacing the unit test framework seems to be on the table.
Perhaps that is a good idea, but it doesn't have anything to do with the purpose of this patch.  I would like to limit comments in
this thread to:

1) Is it worthwhile to provide a way to run a non-trace-comparing test program under the control of waf using the --regression
option even though the whole testing environment is not worked out, present and documented?

2) If the answer to (1) is Yes, does this patch do what it was designed to do and are there any required changes.

Assuming the answer to (1) is Yes, the comments pertaining directly to the patch seem to be:

- I would prefer tests/ to valver

I considered using "tests," but what is really there are "some-of-the-tests."  We have unit tests scattered around the system in
their respective directories.  We have examples doing double-duty as tests but which are in the examples directory.  There are good
reasons to add integration tests in other directories (for example, wifi channels, and assorted objects tested together as a system
and scripted in src/devices/wifi).

So you can really find tests all over the system -- only some of them go in this directory.  The tests that go there are
integration/system/validation tests using components that span subdirectories.

So tests ended up seeming to fuzzy and generic to me.  This goes to the whole discussion about what is a unit test and where does it
live.  This also goes to where other kinds of tests live and what framework do they use to make test-writer lives easier.  I don't
think this is clear enough yet to make a statement.

What is in the directory in this patch are some programs that test something in the system without requiring traces.  This is just
an alternative directory to the examples directory used to build them.  I suspect it will change anyway, so I really don't care.  

Calling it "tests" is fine, but it will mislead people into believing we only have four tests when they look in that directory.
This is not true at all.

I really don't care what it is called.

- I'd prefer that we do not make trace-based comparison the default choice, as a hint to not lean on this mode of testing too much.

That's actually the way I started out, but I changed my mind.  Here's why:  We want some kind of test when we accept a contribution.
There are people who will write extensive tests of their code, and we don't really have to worry about them.  There are people who
think testing is unimportant and won't expend any effort to do it.  I decided that it was more important to get some kind of test
from the people who don't care by lowering the entry bar as much as possible for them.  The "hint" consisting of one line of Python
would probably just bounce off of these folks.

I figured that people who did care about tests were more likely to do more, and would then be willing to understand more and would
go figure out what to do in a python file.  They would be perfectly willing to add one line of Python since they are also willing to
do extensive testing.

Therefore I made the bar lowest for the people willing to do the least work and made the trace case the default.

I don't care from an implementation point of view one way or the other, though.  Changing the sense just means adding one line of
Python to each existing wrapper.  

- the current regression+valver python-based wrappers increase too much the size of the barrier to write tests.

In terms of the python-based wrappers, what is required is copying an empty file that implicitly causes a script to be run.  That
doesn't seem like a high barrier to me.  Anyway, the purpose of this patch is to extend the current mechanism to allow for
non-trace-comparing tests, not to invent a new mechanism.

----------

If you really want to discuss deeper things about test philosophy and implementation, then we really need to do that in another
venue.  If there is no need to do this patch or it is unacceptable as is, then I'd prefer to simply forget it now.  

It was supposed to be a small thing to make at least some progress in this area ASAP.  I probably spent more time writing this email
than I did writing the patch.

The big problems lie elsewhere ...

-- Craig