[Ns-developers] First Stab at Validation and Verification

[George Riley]

Hi Craig,

Some random thoughts about this.

You refer to RFC793 as the TCP specification; while this is true, the  
ns-3 model
is based on RFC2581.

You discuss observing the real-world implementation of the artifact  
being modeled
as the "baseline" or "true" behavior.  There is no TCP implementation  
that I am
aware of that follows the RFC precisely.  For example Linux  
implementations use a
non-standard (no RFC) implementation called "BIC TCP", and later  
"CUBIC TCP".
So it's not clear how the discussion here can reasonably be applied to  
the ns3 TCP.
In our case, the validation is whether or not our model does what is  
specified in
RFC 2581.

Further, there is no randomness at all in TCP, either as implemented  
in ns3 or in
actual implementations. GIven the same set of stimuli, the TCP  
implementations
will always behave in exactly the same way.  The only non- 
deterministic behavior
in implementations of TCP is in the timeout values, which will only  
occur on scheduler
interrupts, and not exactly to the microsecond specified.  So the  
discussion about random
variables, while very nice and correct, is not precisely applicable  
here.  There is randomness
of course in things that happen outside of TCP (for example packet  
losses in a queue that
is overloaded by a large HTTP transfer).

Do you think it might make sense to use a different baseline for the  
discussion, perhaps
the path loss models or something else with true randomness?

George

--------------------------------------------------
George Riley
Associate Professor
Georgia Tech
Electrical and Computer Engineering
riley at ece.gatech.edu
404-894-4767
Klaus Advanced Computing Building
Room 3360
266 Ferst Drive
Atlanta, Georgia 30332-0765

ECE4110 Web page:
http://users.ece.gatech.edu/~riley/ece4110/

ECE4112 Web page:
http://users.ece.gatech.edu/~riley/ece4112/









On Apr 17, 2009, at 7:10 PM, <craigdo at ee.washington.edu> <craigdo at ee.washington.edu 
 > wrote:

> Hi All,
>
> Some of you may have noticed that I've been pondering what to do  
> about validation and verification of ns-3 models.  I've played
> around a bit and have an example put together that does some  
> statistical validation of a subset of our random number generators.
>
> Some folks around here have expressed an interest in getting this  
> cleaned up and checked in fairly quickly.  So even though this is
> not even close to fully baked I thought I'd socialize what I have.   
> Feel free to comment or tell me I'm on crack, as another one of
> the developers is fond of saying.
>
> The high-level idea is to sharpen our models for what we consider  
> system/integration/regression tests.  Right now we have regression
> tests that are basically example programs we have pressed into  
> double duty.  We want to move toward purpose-build verification tests
> and away from the trace-file-based example program hacks we have  
> today.  We also want to add some form of validation framework where
> people can demonstrate that their models are connected with some  
> kind of reality.  As a byproduct of this validation and
> verification test effort, we would like to see something that can be  
> used as a regression suite and as a side-effect also generate
> something that can be used on a validation and verification website  
> (pretty pictures).
>
> I've put together some words on the wiki.  They're kind of  
> incomplete ramblings and thoughts right now, but you can probably  
> get a
> decent flavor of where I'm going (don't hold me to anything):
>
> http://www.nsnam.org/wiki/index.php/VerificationValidationAndTesting
>
> As it stands, the two areas of verification and validation are quite  
> separate, but they'll most likely get folded together as things
> develop.
>
> I have random number generator tests built that do statistical  
> validation, generate pretty plots and can be run as regression tests.
> I'll eventually integrate them into waf, but they are stand-alone  
> right now.  As an example of a validation wiki, I put together the
> following:
>
> http://www.nsnam.org/wiki/index.php/StochasticModelValidation
>
> It's not intended to be complete and thorough (although the chi  
> square tests are real) just a demonstration of where we are headed.
>
> The code can be found at:
>
> http://code.nsnam.org/craigdo/ns-3-valver
>
> with the rng tests in validation/rng and the beginnings of a TCP  
> verification test in verification/tcp (not much there yet).
>
> If you have any plans or ideas please don't hesitate to let me know  
> about them.  If you are doing validation tests on your own, I'd
> love to figure out how to make it easy to integrate them.
>
> Regards,
>
> -- Craig
>