[Ns-developers] Request for Merge -- Validation

[Tom Henderson]

Craig, below are some comments on the rest of your message:

> Assuming the answer to (1) is Yes, the comments pertaining directly to the patch seem to be:
> 
> - I would prefer tests/ to valver
> 
> I considered using "tests," but what is really there are "some-of-the-tests."  We have unit tests scattered around the system in
> their respective directories.  We have examples doing double-duty as tests but which are in the examples directory.  There are good
> reasons to add integration tests in other directories (for example, wifi channels, and assorted objects tested together as a system
> and scripted in src/devices/wifi).
> 
> So you can really find tests all over the system -- only some of them go in this directory.  The tests that go there are
> integration/system/validation tests using components that span subdirectories.
> 
> So tests ended up seeming to fuzzy and generic to me.  This goes to the whole discussion about what is a unit test and where does it
> live.  This also goes to where other kinds of tests live and what framework do they use to make test-writer lives easier.  I don't
> think this is clear enough yet to make a statement.
> 
> What is in the directory in this patch are some programs that test something in the system without requiring traces.  This is just
> an alternative directory to the examples directory used to build them.  I suspect it will change anyway, so I really don't care.  
> 
> Calling it "tests" is fine, but it will mislead people into believing we only have four tests when they look in that directory.
> This is not true at all.

This is easily remedied by dropping a README in the directory saying 
"This directory is home for ns-3 validation, verification, and test 
programs that do not have a home elsewhere in the tree.  However, this 
directory does not contain all the ns-3 tests.  Read (insert document 
reference) for a description of the ns-3 testing framework."

> 
> I really don't care what it is called.
> 
> - I'd prefer that we do not make trace-based comparison the default choice, as a hint to not lean on this mode of testing too much.
> 
> That's actually the way I started out, but I changed my mind.  Here's why:  We want some kind of test when we accept a contribution.
> There are people who will write extensive tests of their code, and we don't really have to worry about them.  There are people who
> think testing is unimportant and won't expend any effort to do it.  I decided that it was more important to get some kind of test
> from the people who don't care by lowering the entry bar as much as possible for them.  The "hint" consisting of one line of Python
> would probably just bounce off of these folks.
> 
> I figured that people who did care about tests were more likely to do more, and would then be willing to understand more and would
> go figure out what to do in a python file.  They would be perfectly willing to add one line of Python since they are also willing to
> do extensive testing.
> 
> Therefore I made the bar lowest for the people willing to do the least work and made the trace case the default.
> 
> I don't care from an implementation point of view one way or the other, though.  Changing the sense just means adding one line of
> Python to each existing wrapper.  

I don't care so much about the default if we make it clear that these 
are not validation tests.  I suspect that it will be maintainers who are 
adding most of these .py files in the long run.

> 
> - the current regression+valver python-based wrappers increase too much the size of the barrier to write tests.
> 
> In terms of the python-based wrappers, what is required is copying an empty file that implicitly causes a script to be run.  That
> doesn't seem like a high barrier to me.  Anyway, the purpose of this patch is to extend the current mechanism to allow for
> non-trace-comparing tests, not to invent a new mechanism.

I haven't seen an alternative proposal that improves upon what you have 
for regression tests.  However, validation tests may not need this.  I 
don't think this is a big deal anyway.

Tom