[Ns-developers] Status of Python bindings

[Gustavo Carneiro]

2008/5/5 Mathieu Lacage <mathieu.lacage at sophia.inria.fr>:

> hi gustavo,
>
> I think that your summary below captures the gist of the issues we
> discussed. A couple of comments below.
>
> On Sun, 2008-05-04 at 22:10 +0100, Gustavo Carneiro wrote:
>
> > 1. Add the pybindgen to ns-3 tree via an automated branch checkout, like
> we
> > do for regression traces (so that installation of PBG is not required);
> >
> > 2. Once in a while, a developer does something like ./waf
> --scan-bindings,
> > which calls pybindgen in gccxml mode, and outputs a
> ns3module_generated.py
> > file (or several files, one per module, if the generated file ends up
> being
> > large)
>
> +1 to the latter option (one file per ns3 module)


It is non-trivial work, but I can do it.


>
> >
> > 3. The generated files are added to the ns-3 repository;
> >
> > 4. The bindings generation script reads the autoscanned definitions plus
> a
> > ns3module_local.py file which can contain user customizations and extra
> > definitions for new API.
> >
> > I think that's it.  Details left to be figured out, of course.
>
> I would be fine with re-running --scan-bindings (2) regularly to attempt
> to generate the bindings for our codebase provided we can agree that if
> it fails and if you cannot fix it (because you are dead or just away for
> an extended period of time :), we can just use its output as a starting
> point and then, write the missing python code by hand. To summarize, if
> 2) works sufficiently well that the generated code is readable and
> editable and that we almost never have to hack its output, I am all for
> it because I know that, at worst, I can hack it without relying on you
> being alive.


But of course the code will be perfectly readable and hackable.

Only the customizations for supporting the core module, as well as simulador
schedule, will be hard to read, but that can't be helped.  If you want a
nice Python API you need custom code for certain low level stuff.  That
would be true for _any_ python binding technology you decided to use.

As for me disappearing, Python bindings for NS-3 would be in no worse shape
than NS-3 core module would be if you disappeared and took all that C++
template black magic with you... :-)


> Included below are a couple of comments based on ideas I already sent to
> you privately about the python API exported by pybindgen.
>
> If I try to go back a bit to look at the problem you are trying to solve
> (that is, describe a C/C++ API and its binding to python), I can see two
> ways to do that:
>
>  1) describe the python API you want to get and, for each element of
> the python API, describe its underlying C/C++ function/method/class
>
>  2) describe the underlying C/C++ API and then, use an automatic
> mapping algorithm to generate a python API except in a few cases where
> you want to override the default mapping behavior.
>
> If I look at the current pybindgen API, I believe that you are using a
> mix of 1) and 2). For example:
>
> mod = Module ('a')
> mod.add_function (CppFunction (return_val, name, args))
>
> 'a' represents the name of the output python module. 'name' represents
> the name of the C++ function. It is also the name of the python function
> contained in module 'a' which will call this C++ function.
>
> So, what I see here is that this API contains a lot of implicit
> information about how everything is mapped from C++ to python. To me, a
> clearer, more verbose, (and, maybe, more pedantic) way to do this would
> be this:
>
> mod = Module ('a')
> mod.add_function ('PyFunctionName',
>                  CppFunction (return_val, 'CppFunctionName', args))



PyBindGen is already supporting almost that:

mod = Module ('a')
mod.add_function (CppFunction (return_val, 'CppFunctionName', args),
                                 name='PyFunctionName')

The 'name' parameter is optional, though.  Since 'PyFunctionName' ==
'CppFunctionName' 99.99% of the time, I think making PyFunctioName a
mandatory parameter is just being pedantic, as you mention below.


> Which clearly outlines the underlying C++ function and its python
> equivalent.


If the python function name is different from the C++ one you are probably
doing something wrong.  Different <whatever> names in Python and C++ will
only cause confusion for NS-3 programmers, who often need to switch back and
forth between Python and C++ and do not want to learn two similar but subtly
different APIs.  So I really don't agree making the python name mandatory
will be helpful.


> Now, I do understand why you took the simpler path of avoiding the
> redundant information everywhere because the number of cases where
> CppFunctionName will be different from PyFunctionName will be very small
> but, that shortcut really does muddy the conceptual model of how things
> work.
>
> So, I can see a couple of ways to deal with this 'muddying':
>  a) become pedantic as suggested above at the cost of extra verbosity
> and pain
>  b) structure in the API in a different way which moves all the
> assumptions about the C++ -> python names to a single clear location.
>
> I think that a) is really too painful so, what could b) look like ?


Bah.. you are just using the old trick of offering a worse alternative as
option so that the one you prefer does not seem so bad by comparison... :P

But I am willing to give it a fair chance...


> Here
> is a proposal:
>
> # describe the C++ code we want to wrap
> header = CppHeader ('a.h')
> header.add_function ('CppFunctionName', return_type, args)
> A = header.add_class ('CppClassName, ...)
> A.add_method (...)
> ns3 = header.add_namespace ('ns3')
> ns3.add_function (...)
> ns3.add_class (...)


Since I really want PyBindGen to be applicable to pure C code as well as
C++, this would have to be just class Header, not CppHeader.


> # describe how to wrap the above C++ code.
> mod = PyModule ('a')
> mod.wrap_header (header)
> mod.generate_output (out_file)


Personally, I don't grok this model very well.  Types and functions belong
to namespaces, not headers.  When you are programming in C++ you don't think
in terms of which header file each type lives in, you think about
namespaces.  Or at least I do.

That being said, I think your model can be very easily supported on top of
the existing API.  I think it's weird, but if it makes you feel better we
can try that :-)


>
> The wrap_header function would be responsible for implementing a mapping
> policy from c++ function names to python function names (and c++
> namespaces to python modules and c++ classes to python classes) so, the
> only magic here is that the documentation of wrap_header clearly states
> the details of the default mapping policy and it points to the
> wrap_header_with_policy method which takes an extra argument, a policy:
>
> mod.wrap_header_with_policy (header, policy)
>
> where policy is a python class which defines a couple of methods to
> perform the mapping and can be provided by the user (please, excuse the
> C++-centric syntax below):
>
> class MyPyWrapPolicy : public PyWrapPolicy
> {
> public:
>  virtual std::string CppFunctionToPyFunction (std::string
> cppFunctionName);
>  virtual std::string CppNamespaceToPyModule (std::string
> cppFunctionName);
> };
>
> It would then be trivial for a user to provide his own implementation of
> these methods to provide an adhoc mapping when it is needed (very
> rarely).


Ugh!  This I really don't like.  And not just this particular syntax.

Initially I added something like this already to PyBindGen:

http://telecom.inescporto.pt/~gjc/pybindgen/doc/pybindgen-apidocs/pybindgen.module.Module-class.html#set_c_function_name_transformer

But guess what?  It has never been used before.  Sure, this kind of thing
can be done, but why bother?  Certainly NS-3 does not need it.


> I suspect that the above proposal could be implemented on top of
> pybindgen to avoid adding more work for you but it probably would make
> more sense to try to agree on the API pybindgen should export rather
> than add yet another intermediate layer to hide something under it.


Absolutely agreed more layers is not the solution.  But it turns out that
only a new Header class is required, and a add_header() method of Module.
Adding these is relatively straightforward and non-intrusive to the existing
API.

I still have work to do in the gccxml-to-python branch, Header will follow.

Thanks,

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert