[Ns-developers] [ns3] Statistical framework (draft)

[Vincent Gauthier]

Hi everybody,

Joe, sorry to be so long to get back to you,

Le 2 mai 08 à 00:57, Joseph Kopena a écrit :

> Hi everybody,
>
> Some comments on the stat framework.
>
> Vincent, I think the approach you are describing is more tied to the
> simulation code and specific data types and events than what I've been
> thinking.  For example, I don't think the stat framework doesn't have
> to or should know if it's tracking TCP window size changes, dropped
> packets, nodes moving, or application events.
>
> I'm envisioning three main parts:
>
> - There's a stat framework interface.  Its job is to manage all the
> statistics produced in a run, and store it in some form such that it
> may be easily aggregated with data from other runs.
>
> + Two standard implementations of this would be to collect the data
>   into XML logs, and another to collect into a DB.
>
> - Stat objects are created to track and produce different statistics.
> For example, you might have one that simply tracks min and max,
> another for mean, one to count & average, another for confidence
> interval, some tracking timing, etc.
>
> + Stat objects have two basic methods: One through which the
>   framework may poll the object for its data to store, and another
>   through which the data is updated.  Both have to be fairly
>   generic, which will take some thought.
>
> - To use the framework, sim user code has to connect "events" to the
> stats.  You would instantiate your stat framework instance, then
> instantiate your stat objects.  Then you would connect them to code
> reporting events.  For example, in a custom application you might
> pass in a stat object and just have the app call the update method
> directly.  For an internal event, you would connect a relevant trace
> source to a stat object you've created, i.e. passing in the update
> method as the callback to use when the trace occurs.
>
>
> For example, *very* rough user code might look something like:
>
> ------------------------------------------------------------------------
> DBStatFramework foo("TrialSet1", "Run4");
> // Arg 1 is some label for this experiment
> // Arg 2 some label for this particular run
>
> foo.SetDescription("Data for the fourth run of my new protocol!");
> // Human-use metadata
>
> foo.AddMetaData("mobility", "random-walk");
> foo.AddMetaData("cds-algorithm", "ecds");
> // User specific < key, value > variables to store about the run,
> // augmenting the Config attributes as noted below.
>
> TotalCountStat counter("total-frame-tx");
> // Create a new statistic object.  This has been given an
> // appropriate label that analysis & presentation code can use
> // to find it.
>
> foo.Include(counter);
> // Inform foo of its existance, so it will poll for data.
>
> Config::Connect("/NodeList/*/DeviceList/0/$ns3::WifiNetDevice/Phy/Tx",
> MakeCallback(&TotalCountStat::UpdateByOne, &counter));
> // That config path is wrong, but something like it would connect the
> // stat object to the Tx trace source in the Wifi devices on all   
> nodes.
> // TotalCountStat::UpdateByOne will now be called whenever a frame
> // is generated on any node.  If we wanted to do something else,
> // e.g. count the total number of frame bytes, we would write a   
> little
> // function to take the packet passed by the trace, and update
> // counter by its size.
>
> Simulator::Run();
>
> foo.Store();
> // This uses the experiment & run labels to create new tables, etc,
> // then polls all the included stat objects for data to store.  It
> // also uses the Config system to store settings for all of the
> // attributes (variables), as well as storing the user
> // provided metadata.
>
> Simulator::Destroy();
> ------------------------------------------------------------------------
>
> The key thing here is that we don't need to instrument anything w/
> stat variables, write different objects for different variables, etc.
> Instead we connect a generic set of statistics components and data
> managers to trace sources, many of which already exist, and of which
> more could be added to plum other aspects of the internals.  We might
> have to provide a set of adapters that convert from common trace types
> or inputs (e.g. packets) to stat object updates, but the core of the
> framework shouldn't know about bytes, packets, etc.
>
> To do this the trace system might need to be all of better documented,
> less brittle, and more standardized, but that's not too much.
>
> Thoughts?

I understand now better what you are envisioning for the stats  
framework. I may try to begin a working example including your  
guidelines and get back to you after a broader set of extensions into  
the frameworks.

To recap what it has been said, we have to use the trace-callback to  
connect the stats framework to the variable we want to be monitored  
(the monitor should be able to be update by the stats framework or by  
the object itself). The Stats framework has to be in charge of  
converting the inputs variables types into the a more meaningful one  
for the statistical analysis.

My first concern was to limit the number of update (from a variable  
into the simulator to the stats DB) for a performance point of view.  
It may be judicious to define the way to update variables as function  
of the the variable type defined (defined when we link this variable  
to the stats DB). As example, If I get back to your code example, the  
way you defined the total- frame-tx  as a counter :
TotalCountStat counter("total-frame-tx");

With this information the stats DB will update this data only at the  
end of the sim (as example), in the contrary, if we had another type  
defined it will be updated differently. Do you agree ?

Regards,
Vincent Gauthier