[Ns-developers] Progessing Python bindings

[Gustavo Carneiro]

2008/5/17 <craigdo at ee.washington.edu>:

>
> Hi All,
>
> I spoke with Tom and Mathieu this afternoon about how to move the Python
> Bindings effort forward toward a release.  We came up with
> some basic requirements that we need to meet.
>
> A number of us spoke a couple of weeks ago about the need to have an
> intermediate step to allow independence from gccxml and
> pygccxml.  Gustavo has been working on implementing his pybindgen API as
> the intermediate step which will allow us to do both
> automatic and manual binding generation.  Automatic binding generation will
> generate code for this intermediate API rather than c++
> code.  I think everyone believes that this work will address the main
> concerns that had been raised about the previous bindings
> iteration.
>
> Given that we agree that this will be a mutually agreeable approach, we
> then talked about some specific functionality required to
> enable the basic development scenarios we want to support.
>
> Of course we want to be able to automatically generate the intermediate
> python code from the ns-3 tree to get everything we can get
> as easily as possible.  This does not mean that all bindings generation is
> done "for free" when ns-3 is built (i.e. bindings are not
> generated automagically by waf).  What it means is that a developer can, if
> she wants, download the required tools and build the
> intermediate code from the ns-3 headers in a separate step.
>
> We expect that the automatically generated binding intermediate code will
> be checked into the ns-3 tree.  That way someone with a
> reasonably modern Python will be able to run the intermediate code and
> generate the c++ and eventually the .so libraries without
> having to deal with the infamous subverted development version of gccxml.
>
> We expect this to work according to roughly the same model as the
> regression traces do now.  In the regression model, there is an
> ns-3 repo that must be in sync with known-good traces also stored in a
> repo.  There are regression tests that are run to drive the
> comparison between the generated code and the known-good traces.  If you
> change the code in the tree in such a way as the traces
> change, you must generate new known-good traces and check them in.  In the
> same vein, we will need to develop a set of Python-based
> tests that need to be run when a change is made.  Then, if you make a
> change to the ns-3 core APIs and the Python-based tests fail,
> you will need to update the bindings to make them work correctly.  If you
> extend the API, hopefully you will extend the Python tests
> to cover your new API; but we aren't all that motivated all the time.
>
> Another scenario we talked about is the following:  What if Developer-X
> makes an API extension but is too lazy to deal with the
> Python bindings at all.  This is certainly a possibility.  Say the next
> person (Developer-Y) comes along, makes another API change
> and, per the plan, regenerates the bindings automatically.  What happens if
> the bindings for Developer-X's changes cause an error?
> We don't want to force Developer-Y to fix Developer-X's problem, so we need
> the ability to exclude Developer-X's API from the
> automatic binding generation.
>
> Now, suppose that there is a third developer, call him Developer-M, who
> doesn't want to deal with the automagic binding generation
> at all.  This developer just wants to write his own intermediate code and
> have it included in the main ns-3 bindings module.  We
> need to be able to specify manually written bindings code to be included in
> the top level binding description.
>
> There is also the possible case where automatic binding generation fails in
> some unforeseen way.  In this case, we want Gustavo to
> eventually make his magic work "mo betta"; but in the short term it would
> be good to be able to use the exclusion mechanism to turn
> off the faulty binding generation and to manually write a binding that
> works and have it manually included at least temporarily.
>
> We also spoke about how one would write python bindings for a new model
> that lives outside the ns-3 tree -- contrib, for example.
> We then need a completely parallel situation in another "namespace."  For
> example, if Developer-C built a new CraigsPerfectDevice,
> he would want to be able to run the automatic binding generator over his
> headers and generate a new module.  He would also want to
> be able to use the exclusion, manual writing and inclusion mechanisms just
> as the other developers do in the main ns-3 tree.  The
> module would not become part of the ns-3 bindings but would exist
> separately (perhaps with dependencies).  In a script you would
> then do:
>
>  import ns3
>  import perfectdevice


Sorry, but this would need more work in PyBindGen I was not expecting.

In the above code you are importing two modules.  The second module will
most likely need to import types from ns3 to use as base classes for its own
types.  This is not done in PyBindGen yet, although I agree it would be nice
to implement some day.

But the same thing should work if instead the developer defines an
additional ns3.PerfectDevice class that becomes part of the main ns3 module.


> So we agreed that we need to do a few things before release:
>
> 1.  Define the process used for automatically generating intermediate code;
> 2.  Decide the structure of and where the intermediate code is checked in;
> 3.  Define the process used for making the libraries from the intermediate
> code;
> 4.  Decide the structure of and where manually written intermediate code is
> checked in;
> 5.  Define a mechanism for excluding bindings from the automatic generation
> process;
> 6.  Define a mechanism for including manually written bindings in the
> library generation process;
> 7.  Port regression tests/examples/samples to python to give us a start at
> being able to detect problems with the bindings.
> 8.  Begin doing Python-based regression testing.
>
> As soon as we get the automagically generated bindings to a state where we
> think they have reasonable coverage, we can have people
> other than Gustavo start the porting work to the existing tests.


>
> There have been some behind-the-scenes conversations on conceptual models
> and API, but I think we've all decided these are
> higher-order corrections at this point.  We want to get the bindings done
> and released and worry about the fine-grain details later.
>
> Sound reasonable?


As soon as NS-3 API stabilizes, I don't see anything wrong with the above,
at first glance.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert