[Ns-developers] Status of Python bindings

[Mathieu Lacage]

hi gustavo,

I think that your summary below captures the gist of the issues we
discussed. A couple of comments below.

On Sun, 2008-05-04 at 22:10 +0100, Gustavo Carneiro wrote:

> 1. Add the pybindgen to ns-3 tree via an automated branch checkout, like we
> do for regression traces (so that installation of PBG is not required);
> 
> 2. Once in a while, a developer does something like ./waf --scan-bindings,
> which calls pybindgen in gccxml mode, and outputs a ns3module_generated.py
> file (or several files, one per module, if the generated file ends up being
> large)

+1 to the latter option (one file per ns3 module)

> 
> 3. The generated files are added to the ns-3 repository;
> 
> 4. The bindings generation script reads the autoscanned definitions plus a
> ns3module_local.py file which can contain user customizations and extra
> definitions for new API.
> 
> I think that's it.  Details left to be figured out, of course.

I would be fine with re-running --scan-bindings (2) regularly to attempt
to generate the bindings for our codebase provided we can agree that if
it fails and if you cannot fix it (because you are dead or just away for
an extended period of time :), we can just use its output as a starting
point and then, write the missing python code by hand. To summarize, if
2) works sufficiently well that the generated code is readable and
editable and that we almost never have to hack its output, I am all for
it because I know that, at worst, I can hack it without relying on you
being alive.

Included below are a couple of comments based on ideas I already sent to
you privately about the python API exported by pybindgen.

If I try to go back a bit to look at the problem you are trying to solve
(that is, describe a C/C++ API and its binding to python), I can see two
ways to do that:

  1) describe the python API you want to get and, for each element of
the python API, describe its underlying C/C++ function/method/class

  2) describe the underlying C/C++ API and then, use an automatic
mapping algorithm to generate a python API except in a few cases where
you want to override the default mapping behavior.

If I look at the current pybindgen API, I believe that you are using a
mix of 1) and 2). For example:

mod = Module ('a')
mod.add_function (CppFunction (return_val, name, args))

'a' represents the name of the output python module. 'name' represents
the name of the C++ function. It is also the name of the python function
contained in module 'a' which will call this C++ function.

So, what I see here is that this API contains a lot of implicit
information about how everything is mapped from C++ to python. To me, a
clearer, more verbose, (and, maybe, more pedantic) way to do this would
be this:

mod = Module ('a')
mod.add_function ('PyFunctionName',
                  CppFunction (return_val, 'CppFunctionName', args))

Which clearly outlines the underlying C++ function and its python
equivalent.

Now, I do understand why you took the simpler path of avoiding the
redundant information everywhere because the number of cases where
CppFunctionName will be different from PyFunctionName will be very small
but, that shortcut really does muddy the conceptual model of how things
work. 

So, I can see a couple of ways to deal with this 'muddying':
  a) become pedantic as suggested above at the cost of extra verbosity
and pain
  b) structure in the API in a different way which moves all the
assumptions about the C++ -> python names to a single clear location.

I think that a) is really too painful so, what could b) look like ? Here
is a proposal:

# describe the C++ code we want to wrap
header = CppHeader ('a.h')
header.add_function ('CppFunctionName', return_type, args)
A = header.add_class ('CppClassName, ...)
A.add_method (...)
ns3 = header.add_namespace ('ns3')
ns3.add_function (...)
ns3.add_class (...)

# describe how to wrap the above C++ code.
mod = PyModule ('a')
mod.wrap_header (header)
mod.generate_output (out_file)

The wrap_header function would be responsible for implementing a mapping
policy from c++ function names to python function names (and c++
namespaces to python modules and c++ classes to python classes) so, the
only magic here is that the documentation of wrap_header clearly states
the details of the default mapping policy and it points to the
wrap_header_with_policy method which takes an extra argument, a policy:

mod.wrap_header_with_policy (header, policy)

where policy is a python class which defines a couple of methods to
perform the mapping and can be provided by the user (please, excuse the
C++-centric syntax below):

class MyPyWrapPolicy : public PyWrapPolicy
{
public:
  virtual std::string CppFunctionToPyFunction (std::string
cppFunctionName);
  virtual std::string CppNamespaceToPyModule (std::string
cppFunctionName);
};

It would then be trivial for a user to provide his own implementation of
these methods to provide an adhoc mapping when it is needed (very
rarely).

I suspect that the above proposal could be implemented on top of
pybindgen to avoid adding more work for you but it probably would make
more sense to try to agree on the API pybindgen should export rather
than add yet another intermediate layer to hide something under it.

best regards,
Mathieu