[Ns-developers] ns-3 python bindings (was: Re: NS-3 core API freeze?)

[Gustavo Carneiro]

2008/4/23 Tom Henderson <tomh at tomh.org>:

>
> >-----Original Message-----
> >From: Gustavo Carneiro [mailto:gjcarneiro at gmail.com]
> >Sent: Wednesday, April 23, 2008 09:37 AM
> >To: 'Mathieu Lacage'
> >Cc: 'ns-developers'
> >Subject: Re: [Ns-developers] NS-3 core API freeze?
> >
> >2008/4/23 Mathieu Lacage <mathieu.lacage at sophia.inria.fr>:
> >
> >>
> >> On Tue, 2008-04-22 at 20:06 +0100, Gustavo Carneiro wrote:
> >>
> >> > 2. Any python binding technology will require the user to install the
> >> > respective tools, so changing technology will not magically fix 5.1;
> >>
> >> Although I hate that perspective, I consider it much more reasonable to
> >> ask a user to install a released version of swig (or boost::python, or
> >> anything else already released) than a cvs version of gccxml. The later
> >> is _not_ ok for anyone but you.
> >>
> >> > 3. I have manpower issues and am not sure I could implement this for
> >> > 15th May even if API was already frozen.   I only work a few hours a
> >> > week on this, mostly just on weekends
> >>
> >> understood.
> >>
> >> >   3.1 One way to work around this is to have someone else help with
> >> > this task
> >> >
> >> >      3.1.1 The software engineering rule "adding more manpower to a
> >> > late project makes it later" may apply here, though I hope I'm
> >> > wrong :-(
> >>
> >> you are probably right though.
> >>
> >> >
> >> >   3.2 In addition, in your roadmap you are still proposing some
> >> > core/common API changes, which in my experience tend to disrupt or
> >> > destroy some work done on Python bindings.  Personally I am reluctant
> >> > to do more Python work on top of a unstable API, as it feels like
> >> > wasted work :|
> >>
> >> I was under the impression that you were not wrapping the Packet API
> so,
> >> the tag stuff should not impact you. The socket stuff might though.
> >>
> >> > 4. Also there are some other things left to do in the Python
> bindings;
> >> > do you think it is better to have a way to let users extend Python
> >> > bindings, or is it better to fix Python API issues sooner to avoid
> >> > breaking Python code?  This is why I keep saying fixing 5.1 is _not_
> >> > my priority.
> >>
> >> As I said in my previous email, an unfinished binding which is easy
> >> hacked on by users (and other developers like me) is vastly more useful
> >> than a finished binding with a single developer. i.e., I personally
> >> would be perfectly fine with using your python binding library by hand
> >> if it was decently documented: the load of writing the bindings can be
> >> distributed while the load of building the underlying infrastructure
> >> cannot.
> >>
> >> So, using gccxml to save on the load of writing the bindings is not
> >> solving the critical problem: it is solving a problem we can solve by
> >> other means through offloading work from you to others. On the other
> >> hand, I have a hard time seeing how we could offload work on your
> python
> >> binding library from you: that is why I encouraged you a couple of
> times
> >> to work on documenting its API and try to polish and improve it with
> >> your users in mind.
> >
> >
> >Lack of documentation is not the main problem here.  Usage of gccxml cvs
> >seems to be the main problem.
> >
> >
> >>
> >> > 5. Also, I think fixing 5.1 should be included in the "polishing of
> >> > python bindings" stage of the roadmap you propose.
> >>
> >> 5.1 is a must-have feature. So, we need to be sure that we can solve
> >> this problem in time for the release. If that means that the bindings
> >> themselves do not cover 100% of the API for the release, I think that
> it
> >> is ok because we can fix it later.
> >>
> >
> >If usage of gccxml cvs is such a burden (no one told me so when I started
> >using it), then I see one possible course of action.  One developer
> >regulardly uses gccxml and saves the output xml file in the ns-3
> >repository.  Then normal users and other developers don't need gccxml,
> only
> >pybindgen and pygccxml.  Adding API can be done using pybindgen API.
> >
> >Another course of development would be for pybindgen to serialize its own
> >data structures to an xml format.  This would make also pygccxml
> optional,
> >but implementing this would be a lot more work.
>
> Gustavo,
>
> I had a question about a statement on the wiki page:
>
> "Python bindings for NS-3 are being developed using a new tool called
> PyBindGen. This tool uses gccxml to scan header files and extract API
> definitions in a semi-automatic way."
>
> What do you mean by semi-automatic?  What portions are automatic and what
> not?  The example you posted on the wiki looks like it is completely
> automatic.


Like I say below, if you have APIs that pass parameters by pointer you need
to tell pybindgen the ownership transfer semantics (who owns the object
pointer).  Or if you pass by reference you need to tell the direction
(in/out/inout).  Probably more stuff like that.  You know, not _all_
relevant information is available by reading C++ function prototypes...


> The use case I am concerned about, for the moment, is the developer who
> writes a new ns-3 model in C++ (for instance, a new routing protocol) but
> wants to script in Python.  Let's suppose that the new model introduces
> several new classes and APIs using these classes in the C++ system.
>
> Assuming that the bindings toolchain can be installed, what is the
> workflow likely to look like for this developer?
> - at what point in the build are these new bindings generated?


If the needed tools are detected, python bindings are regenerated every time
a public header file changes.  Yes, this could slow down builds a lot, but
suggestions are welcome (e.g. only do it with a waf option, --pybindgen,
etc.).


>
> - how likely is this automated process to succeed, and if not, what does
> the developer have to do when it breaks down?


There are a bunch of things that can cause bindings to "fail", namely:

  1- Pass by reference works, but you need to tell pybindgen the direction
of the parameter (in/out/inout), if not PBG tries to guess it is an "in"
parameter which is not always right;

  2- Passing parameters by pointer (execpt C strings of course), or
returning by pointer.  I think this is obvious why...  Ptr smart pointers
are, of course, OK.

   (there are probably other case I don't remember right now)

In PyBindGen I have been trying hard to make sure that when something cannot
be wrapped automatically by PBG a warning is printed and the definition is
ignored, but the code still compiles and the rest is wrapped.  I consider it
a bug when PBG produces code that does not compile, but it can happen
occasionally.

So basically changing APIs rarely should break the build process,
increasingly less as PBG bugs get detected and fixed.  The notable exception
is NS-3 core APIs, like objects, and attributes, because I try to write
custom code for a lot of these things so that the Python API feels nicer,
not just a simple automated translation from C++ or when PBG can't handle
the very advanced C++ constructs being employed.  That's why I keep saying
ns-3 has to freeze the core APIs before Python bindings work can proceed.
By core APIs I mean just modules 'core', 'simulator' and, to some extent,
'common'.  The rest can keep changing, I don't mind from Python bindings
point of view (though I do mind from NS-3 user point of view).

So as long as the user does not touch the core module, and as longer as the
new APIs don't use raw pointers, it should be fine...

 How will the developer know what to do (is the error message helpful)?


The warning messages are partially useful.  I guess with more documentation
they can become more useful :P


>
>
> Another question is, if we were to work with the development versions of
> the various tools you are using, is there a reasonable possibility of
> bundling known-good versions of these tools and distributing with ns-3, such
> as what you do with waf?


Well, I guess pygccxml and pybindgen could be easily bundled.  gccxml is
more difficult because it needs to be compiled using cmake and make.  A
working gccxml snapshot could be provided in www.nsnam.org, I guess, but
bundling it with the ns-3 tarball itself is overkill IMHO as it is rather
large (gccxml 0.6 tarball is 9.6 MiB).


>  What are the dependencies of these tools (gccxml, pygccxml, pybindgen)?


I _think_ gccxml only depends on cmake, make, and gcc for building.  No
runtime libraries required.  pygccxml depends on python (I don't think it
uses decorators, so a fairly old python).  PyBindGen depends on Python >=
2.3.  So, dependencies are fairly light.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert