[Ns-developers] Status of Python bindings

[Tom Henderson]

Gustavo Carneiro wrote:
> Hi all.
> 
> The current status of Python bindings for NS-3 is as follows.
> 
> We had a phone conference a couple of weeks ago to discuss the
> pybindgen/ns-3 work.
> 
> Mathieu was concerned about:
> 
>      1) complexity of pybindgen with (py)gccxml, and that I am the only
> person that can understand it;
> 
>      2) requiring developers to install gccxml from CVS (since the gccxml
> maintainers have not made an official release in a very long time), is bad;
> 
>      3) pybindgen, even without gccxml involved, was poorly documented, at
> best.
> 
> We discussed the possibility of using boost::python instead, but only people
> skilled in C++ template programming, such as Mathieu, can grok the
> associated compiler error messages.  Another alternative discussed was SWIG,
> but I dismissed it as generating too ugly and inefficient code.
> 
> Finally we discussed ways to improve on the current pybindgen solution.
> Participants were under the impression that, if you take away the gccxml
> layer, the python scripts calling pybindgen API directly were simple enough
> and maintainable.  This was subject to confirmation.  Mathieu even
> maintained the position that it is better to completely forget gccxml and
> simply maintain the API definition script by hand.  I, on the other hand,
> believe this is not maintainable in the long run, and hand writing API
> definitions will lead to Python bindings always being one step behind and
> always feeling like a 2nd class citizen.
> 
> [ please correct anything above I recall wrong from the phone conference ]

Gustavo,
Thanks for summarizing; that's mainly how I remember it.  The use case 
that I raised, which seemed problematic, was supporting the user who 
extended or changed the C++ API somehow and wanted new bindings for the 
changes.  If that user did not (or could not) install the toolchain, he 
or she would be stuck.  Moreover, if the user did install the toolchain 
but there was some problem in the toolchain generating the binding, he 
or she would again be stuck, because debugging would be difficult.

> 
> As I result of these discussions, I started work on a new pybindgen mode
> that uses gccxml but which generates a python script which would generate
> the C++ module, instead of directly generating the module.  This work needs
> time and patience, and is being developed on a separate pybindgen branch,
> see
> 
>     https://code.launchpad.net/~gjc/pybindgen/gccxml-to-python
> 
> The work is not yet complete, but is well advanced (75% at least, I would
> say).  In attachment you can find a ns3 module generation script produced by
> this branch.  Developers should be able to judge the suitability of this API
> for maintaining bindings, although bearing in mind that some things can
> still be (slightly) improved.

 From what I saw of the test programs, it looked like these python 
scripts may be more manageable.

> 
> I am sure not every one agrees, but my "vision" for the bindings would be:
> 
> 1. Add the pybindgen to ns-3 tree via an automated branch checkout, like we
> do for regression traces (so that installation of PBG is not required);
> 
> 2. Once in a while, a developer does something like ./waf --scan-bindings,
> which calls pybindgen in gccxml mode, and outputs a ns3module_generated.py
> file (or several files, one per module, if the generated file ends up being
> large)
> 
> 3. The generated files are added to the ns-3 repository;
> 
> 4. The bindings generation script reads the autoscanned definitions plus a
> ns3module_local.py file which can contain user customizations and extra
> definitions for new API.
> 
> I think that's it.  Details left to be figured out, of course.
> 
> Comments are welcome.
> 

Craig and I had some discussion of the new repo on Friday.  Some of my 
notes from that:

- we have so far neglected to discuss how the bindings will be tested. 
Basically, they will become the second class citizens you mention if 
there is not a suitable regression testing framework for them, and ways 
for users to test their local customizations, regardless of whether we 
are autoscanning the bulk of them or not.

- it would probably be helpful if the python scripts were broken out 
into smaller files, perhaps on the granularity of each header file 
(maybe module, as you suggest, would be OK).  This would make it easier 
for people to extract bindings into a customization directory when they 
need to make a new or revised one.  For instance, someone who is editing 
an existing class/binding may want to just move that one into the local 
customizations directory.

- we would probably need to have conventions about where the bindings 
generation script reads the locally stored overrides and how to deal 
with collisions, although it seems from the above you are already 
thinking about this.

- Rather than just pybindgen in a separate repo, we could maybe store 
the bindings themselves in a separate repo, with a local directory in 
ns-3 for local customizations.  Again (testing comment above) we would 
need to make sure that they did not fall out of sync with the C++.

Tom