[Ns-developers] Progessing Python bindings

[craigdo@ee.washington.edu]

Hi All,

I spoke with Tom and Mathieu this afternoon about how to move the Python Bindings effort forward toward a release.  We came up with
some basic requirements that we need to meet.

A number of us spoke a couple of weeks ago about the need to have an intermediate step to allow independence from gccxml and
pygccxml.  Gustavo has been working on implementing his pybindgen API as the intermediate step which will allow us to do both
automatic and manual binding generation.  Automatic binding generation will generate code for this intermediate API rather than c++
code.  I think everyone believes that this work will address the main concerns that had been raised about the previous bindings
iteration.

Given that we agree that this will be a mutually agreeable approach, we then talked about some specific functionality required to
enable the basic development scenarios we want to support.

Of course we want to be able to automatically generate the intermediate python code from the ns-3 tree to get everything we can get
as easily as possible.  This does not mean that all bindings generation is done "for free" when ns-3 is built (i.e. bindings are not
generated automagically by waf).  What it means is that a developer can, if she wants, download the required tools and build the
intermediate code from the ns-3 headers in a separate step.

We expect that the automatically generated binding intermediate code will be checked into the ns-3 tree.  That way someone with a
reasonably modern Python will be able to run the intermediate code and generate the c++ and eventually the .so libraries without
having to deal with the infamous subverted development version of gccxml.

We expect this to work according to roughly the same model as the regression traces do now.  In the regression model, there is an
ns-3 repo that must be in sync with known-good traces also stored in a repo.  There are regression tests that are run to drive the
comparison between the generated code and the known-good traces.  If you change the code in the tree in such a way as the traces
change, you must generate new known-good traces and check them in.  In the same vein, we will need to develop a set of Python-based
tests that need to be run when a change is made.  Then, if you make a change to the ns-3 core APIs and the Python-based tests fail,
you will need to update the bindings to make them work correctly.  If you extend the API, hopefully you will extend the Python tests
to cover your new API; but we aren't all that motivated all the time.

Another scenario we talked about is the following:  What if Developer-X makes an API extension but is too lazy to deal with the
Python bindings at all.  This is certainly a possibility.  Say the next person (Developer-Y) comes along, makes another API change
and, per the plan, regenerates the bindings automatically.  What happens if the bindings for Developer-X's changes cause an error?
We don't want to force Developer-Y to fix Developer-X's problem, so we need the ability to exclude Developer-X's API from the
automatic binding generation.

Now, suppose that there is a third developer, call him Developer-M, who doesn't want to deal with the automagic binding generation
at all.  This developer just wants to write his own intermediate code and have it included in the main ns-3 bindings module.  We
need to be able to specify manually written bindings code to be included in the top level binding description.

There is also the possible case where automatic binding generation fails in some unforeseen way.  In this case, we want Gustavo to
eventually make his magic work "mo betta"; but in the short term it would be good to be able to use the exclusion mechanism to turn
off the faulty binding generation and to manually write a binding that works and have it manually included at least temporarily.

We also spoke about how one would write python bindings for a new model that lives outside the ns-3 tree -- contrib, for example.
We then need a completely parallel situation in another "namespace."  For example, if Developer-C built a new CraigsPerfectDevice,
he would want to be able to run the automatic binding generator over his headers and generate a new module.  He would also want to
be able to use the exclusion, manual writing and inclusion mechanisms just as the other developers do in the main ns-3 tree.  The
module would not become part of the ns-3 bindings but would exist separately (perhaps with dependencies).  In a script you would
then do:

  import ns3
  import perfectdevice

So we agreed that we need to do a few things before release:

1.  Define the process used for automatically generating intermediate code;
2.  Decide the structure of and where the intermediate code is checked in;
3.  Define the process used for making the libraries from the intermediate code;
4.  Decide the structure of and where manually written intermediate code is checked in;
5.  Define a mechanism for excluding bindings from the automatic generation process;
6.  Define a mechanism for including manually written bindings in the library generation process;
7.  Port regression tests/examples/samples to python to give us a start at being able to detect problems with the bindings.
8.  Begin doing Python-based regression testing.

As soon as we get the automagically generated bindings to a state where we think they have reasonable coverage, we can have people
other than Gustavo start the porting work to the existing tests.

There have been some behind-the-scenes conversations on conceptual models and API, but I think we've all decided these are
higher-order corrections at this point.  We want to get the bindings done and released and worry about the fine-grain details later.

Sound reasonable?

Regards,

-- Craig