[Ns-developers] Status of Python bindings

[Gustavo Carneiro]

2008/5/5 Tom Henderson <tomh at tomh.org>:

> Gustavo Carneiro wrote:
>
> > Hi all.
> >
> > The current status of Python bindings for NS-3 is as follows.
> >
> > We had a phone conference a couple of weeks ago to discuss the
> > pybindgen/ns-3 work.
> >
> > Mathieu was concerned about:
> >
> >     1) complexity of pybindgen with (py)gccxml, and that I am the only
> > person that can understand it;
> >
> >     2) requiring developers to install gccxml from CVS (since the gccxml
> > maintainers have not made an official release in a very long time), is
> > bad;
> >
> >     3) pybindgen, even without gccxml involved, was poorly documented,
> > at
> > best.
> >
> > We discussed the possibility of using boost::python instead, but only
> > people
> > skilled in C++ template programming, such as Mathieu, can grok the
> > associated compiler error messages.  Another alternative discussed was
> > SWIG,
> > but I dismissed it as generating too ugly and inefficient code.
> >
> > Finally we discussed ways to improve on the current pybindgen solution.
> > Participants were under the impression that, if you take away the gccxml
> > layer, the python scripts calling pybindgen API directly were simple
> > enough
> > and maintainable.  This was subject to confirmation.  Mathieu even
> > maintained the position that it is better to completely forget gccxml
> > and
> > simply maintain the API definition script by hand.  I, on the other
> > hand,
> > believe this is not maintainable in the long run, and hand writing API
> > definitions will lead to Python bindings always being one step behind
> > and
> > always feeling like a 2nd class citizen.
> >
> > [ please correct anything above I recall wrong from the phone conference
> > ]
> >
>
I forgot to mention some work done on documenting PyBindGen:

  1. Mathieu provided me some simple C/C++ examples, which I imported into
PyBindGen and made the python bindings for them:


http://bazaar.launchpad.net/~gjc/pybindgen/trunk/files/gjc%40gnome.org-20080430113501-6wzu6ryxj8ld66gy?file_id=examples-20070531165635-sbpc6ceu9uq4cvaa-1

  2. I made some work to convert docstrings to epydoc format, write a short
introduction, and generated html API docs:

    http://telecom.inescporto.pt/~gjc/pybindgen/doc/pybindgen-apidocs/

More comments below.


> Gustavo,
> Thanks for summarizing; that's mainly how I remember it.  The use case
> that I raised, which seemed problematic, was supporting the user who
> extended or changed the C++ API somehow and wanted new bindings for the
> changes.  If that user did not (or could not) install the toolchain, he or
> she would be stuck.  Moreover, if the user did install the toolchain but
> there was some problem in the toolchain generating the binding, he or she
> would again be stuck, because debugging would be difficult.
>
>
> > As I result of these discussions, I started work on a new pybindgen mode
> > that uses gccxml but which generates a python script which would
> > generate
> > the C++ module, instead of directly generating the module.  This work
> > needs
> > time and patience, and is being developed on a separate pybindgen
> > branch,
> > see
> >
> >    https://code.launchpad.net/~gjc/pybindgen/gccxml-to-python<https://code.launchpad.net/%7Egjc/pybindgen/gccxml-to-python>
> >
> > The work is not yet complete, but is well advanced (75% at least, I
> > would
> > say).  In attachment you can find a ns3 module generation script
> > produced by
> > this branch.  Developers should be able to judge the suitability of this
> > API
> > for maintaining bindings, although bearing in mind that some things can
> > still be (slightly) improved.
> >
>
> From what I saw of the test programs, it looked like these python scripts
> may be more manageable.
>
>
> > I am sure not every one agrees, but my "vision" for the bindings would
> > be:
> >
> > 1. Add the pybindgen to ns-3 tree via an automated branch checkout, like
> > we
> > do for regression traces (so that installation of PBG is not required);
> >
> > 2. Once in a while, a developer does something like ./waf
> > --scan-bindings,
> > which calls pybindgen in gccxml mode, and outputs a
> > ns3module_generated.py
> > file (or several files, one per module, if the generated file ends up
> > being
> > large)
> >
> > 3. The generated files are added to the ns-3 repository;
> >
> > 4. The bindings generation script reads the autoscanned definitions plus
> > a
> > ns3module_local.py file which can contain user customizations and extra
> > definitions for new API.
> >
> > I think that's it.  Details left to be figured out, of course.
> >
> > Comments are welcome.
> >
> >
> Craig and I had some discussion of the new repo on Friday.  Some of my
> notes from that:
>
> - we have so far neglected to discuss how the bindings will be tested.
> Basically, they will become the second class citizens you mention if there
> is not a suitable regression testing framework for them, and ways for users
> to test their local customizations, regardless of whether we are
> autoscanning the bulk of them or not.


Python has a unit testing framework already, and I think it should be used
to add some python unit tests.  Additionally, the same regression testing
framework we have already could be enhanced to support python based
regression tests.  Anyway, due to the current API instability I think we can
plan for this task but would be wasted work to actually execute it, not to
mention the lack of time...


>
> - it would probably be helpful if the python scripts were broken out into
> smaller files, perhaps on the granularity of each header file (maybe module,
> as you suggest, would be OK).  This would make it easier for people to
> extract bindings into a customization directory when they need to make a new
> or revised one.  For instance, someone who is editing an existing
> class/binding may want to just move that one into the local customizations
> directory.


My feeling is that the pybindgen scripts are much smaller than the
corresponding C++ header files it wraps.  So IMHO split by ns3 module would
be more manageable, I think. But, it needs work, right not no splitting at
all is supported, and I need time to code it :P (after other tasks are
finished first)


> - we would probably need to have conventions about where the bindings
> generation script reads the locally stored overrides and how to deal with
> collisions, although it seems from the above you are already thinking about
> this.


I sort of thought about it, although not too deeply; does not seem like a
difficult task.  We just need to figure out how we want it.  We can either
have a single local customisations python source, or we could have one
customisation file per module.


> - Rather than just pybindgen in a separate repo, we could maybe store the
> bindings themselves in a separate repo, with a local directory in ns-3 for
> local customizations.  Again (testing comment above) we would need to make
> sure that they did not fall out of sync with the C++.


  Well, I don't see the point of the local customizations living in the main
ns-3 repository but not the bindings themselves.  And indeed they may easily
fall out of sync from C++.  Not 100% sure this is a good idea, but I
wouldn't oppose it if anyone else is more sure about it than I am :-)

  Thanks for the comments.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert