[Ns-developers] Language Bindings with SWIG or PyBindGen
craigdo@ee.washington.edu
craigdo at ee.washington.edu
Mon Aug 6 16:02:55 PDT 2007
Hi all,
I've been pondering how to proceed with language bindings for
the ns-3 system. I'm sending this email to summarize what I've
done and found, to pass on the results of some preliminary
Discussions (requested by Tom), and to start a wider discussion
leading to some "final" decisions. I'm asking for your thoughts
and to see if there's any kind of consensus on where to go from
here.
The first part of this email consists of a small note I recently
Wrote describing a comparison between bindings generated using
SWIG and bindings generated using Gustavo's PyBindGen (which he
originally wrote after a less than positive experience with Boost
library-based Python binding tools.
The second (smaller) part is a collection of some of the comments
made about Bindings -- some specific to one or the other approach
and some in general.
Some of the more important questions are:
1. Do we want only "scripting" or also implement new protocols
in a scripting language?
2. Do we want only Python, or do we want to support multiple
languages (especially Tcl)? What languages?
3. Who do we expect to maintain/provide bindings (thinking in
particular about contributed code)?
4. What do we want the Python API to look like? Tcl API?
------------------------- BEGIN NOTE -------------------------
I've spent the last couple of weeks doing some serious
prototyping of ns-3 language bindings for Python and TCL.
(There have recently been some comments made that having TCL
bindings would be good since we have a user-base that already
"speaks" TCL.)
OVERVIEW
I've used SWIG to create bindings for both Python and TCL. I
have a repository at http://code.nsnam.org/craigdo/ns-3-swig
that you can look at to see the result. The bindings are
built in (not too surprisingly) ./bindings/swig/{python,tcl}
and the interface file is ./bindings/swig/ns-3.i (used for both).
Neither is integrated with waf, they're just build with a shell
script, BTW.
The look and feel of the resulting language constructs comes
from the "SWIG way" of doing things, particularly handling
templates. I haven't gotten everything working, but there's
enough done to verify that SWIG will work, how the approach
works, and how the resulting language bindings will look.
Here are two excerpts of my test files that shows the look-and-feel:
Python:
import ns3 as ns
ns.Bind("Queue", "DropTailQueue")
in0 = ns.CreateInternetNode()
n0 = in0.QueryInterfaceNode(ns.Node.iid)
asciitrace= ns.AsciiTrace("simple-p2p.tr")
asciitrace.TraceAllQueues()
asciitrace.TraceAllNetDeviceRx()
TCL:
load ./ns3.so Ns3
Bind "Queue" "DropTailQueue"
set in0 [CreateInternetNode]
set n0 [$in0 QueryInterfaceNode Node_iid]
set asciitrace [AsciiTrace "simple-p2p.tr"]
$asciitrace TraceAllQueues
$asciitrace TraceAllNetDeviceRx
The non-obvious constructs are the references to
CreateInternetNode and QueryInterfaceNode. This is different
from what you might expect due to the presence of templates
in the underlying C++ code.
You may know that declaration of a template does not imply
the instantiation of any code. You have to actually use a
template in order to get any code generated. The target
languages don't have any concept of a template (as C++ does)
and so SWIG has to provide a way to cause the underlying C++
code to be instantiated. This mechanism is through the
"%template" directive.
In the interface (ns3.i) file, you will see lines that look like,
%template(CreateInternetNode) Create<InternetNode>;
This line causes the Create<InternetNode> code to be
instantiated in a wrapper called CreateInternetNode. You
have to provide one of these lines for every template that
you want to be instantiated. Another template directive is,
%template(QueryInterfaceNode) Object::QueryInterface<Node>;
This instantiates the code required to do a QI on an Object
that returns a Ptr<Node>. Note that there must also be a
template directive instantiating the Ptr<Node> code,
%template(PtrNode) Ptr<Node>;
The result of this is that the QueryInterface calls seem to
include redundant information,
n0 = in0.QueryInterfaceNode(ns.Node.iid)
This is because the QueryInterface<Node> C++ code is
instantiated via the QueryInterfaceNode template, and the
underlying C++ function takes an interface ID which SWIG happily uses.
The good news is that SWIG pretty much "just works" when
generating the different bindings from the interface file.
The one difference I needed to work out between Python and
TCL was due to CommandLine::Parse taking argc and argv as
parameters. In TCL there are two parameters $::argc and
$::argv that need to be mapped into this call. In Python,
there is one list, sys.argv that needs to be mapped. You can
see the two ifdefs in the ns3.i that take care of the
differences (see #ifdef SWIG<PYTHON> and #ifdef SWIG<TCL>).
I'm confident there is nothing in our system that will
confuse SWIG terribly. We did expose a bug in the SWIG code
generator due to our use of smart pointers and templated
methods that I reported and has been fixed. The current
codebase actually includes a sed script to go in and fix the
generated code. I need to download the new SWIG from svn to
get the fix and test it. The response from the SWIG folks
was spectacularly fast. They acknowledged the bug without
any BS and fixed it immediately. This gave me a nice warm,
fuzzy feeling about them.
COMPARISON
We are considering two ways to go with language bindings.
Gustavo has build a nice code generator for Python that he
wrote in Python. I thought I'd put an excerpt of a SWIG
interface next to a PYBINDGEN program excerpt for comparison.
First, a couple of simple classes,
SWIG:
---------- begin SWIG code ----------
%module ns3
%{
#include "ns3/ascii-trace.h"
#include "ns3/simulator.h"
%}
Namespace ns3 {
class Simulator
{
public:
static void Destroy (void);
static void Run (void);
};
class AsciiTrace
{
public:
AsciiTrace (std::string filename);
~AsciiTrace ();
void TraceAllQueues (void);
void TraceAllNetDeviceRx (void);
};
}; // namespace
---------- end SWIG code ----------
PYBINDGEN:
---------- begin PYBINDGEN code ----------
for header in ['ascii-trace', 'simulator']
print >> out_file, "#include \"ns3/%s.h\"" % header
print >> out_file, "using namespace ns3;"
mod = Module('ns3')
Simulator = CppClass('Simulator')
mod.add_class(Simulator)
Simulator.add_method(CppMethod(
ReturnValue.new('void'), 'Run', [], is_static=True))
Simulator.add_method(CppMethod(
ReturnValue.new('void'), 'Destroy', [], is_static=True))
AsciiTrace = CppClass('AsciiTrace')
mod.add_class(AsciiTrace)
AsciiTrace.add_constructor(CppConstructor(
[Parameter.new('std::string', 'filename')]))
AsciiTrace.add_method(CppMethod(
ReturnValue.new('void'), 'TraceAllQueues', []))
AsciiTrace.add_method(CppMethod(
ReturnValue.new('void'), 'TraceAllNetDeviceRx', []))
---------- end PYBINDGEN code ----------
Niether is too painful.
Now, smart pointers and reference counted objects are a bit
more serious, so I include a comparison of what's needed
there. I think I've got the two pieces extracted for a fair
comparison.
SWIG:
---------- begin SWIG code ----------
%module ns3
%{
#include "ns3/ptr.h"
#include "ns3/object.h"
%}
Namespace ns3 {
template <typename T>
class Ptr
{
private:
T *m_ptr;
void Acquire (void) const;
public:
Ptr ();
Ptr (T *ptr);
Ptr (Ptr const&o);
~Ptr () ;
T *operator -> () const;
T *operator -> ();
};
template <typename T> Ptr<T> Create (void);
class InterfaceId
{
public:
static InterfaceId LookupByName (std::string name);
static InterfaceId LookupParent (InterfaceId iid);
private:
InterfaceId (uint16_t iid);
};
%feature("ref") Object "$this->Ref();"
%feature("unref") Object "$this->Unref();"
class Object
{
public:
static const InterfaceId iid;
Object ();
virtual ~Object ();
void Ref (void) const;
void Unref (void) const;
template <typename T> Ptr<T> QueryInterface (InterfaceId iid) const;
void Dispose (void);
void AddInterface (Ptr<Object> other);
};
%template(PtrObject) Ptr<Object>;
%template(QueryInterfaceNode) Object::QueryInterface<Node>;
}; // namespace
---------- end SWIG code ----------
Note the support for AddRef and Release built into SWIG (to
my chagrin called ref and unref).
---------- begin PYBINDGEN code ----------
for header in ['ptr', 'object']
print >> out_file, "#include \"ns3/%s.h\"" % header
print >> out_file, "using namespace ns3;"
mod = Module('ns3')
class SmartPointerTransformation(typehandlers.TypeTransformation):
def __init__(self):
self.rx = re.compile(r'Ptr<(\w+)>')
def get_untransformed_name(self, name):
m = self.rx.match(name)
if m is None:
return None
else:
return m.group(1)+'*'
def create_type_handler(self, type_handler, *args, **kwargs):
if issubclass(type_handler, Parameter):
kwargs['transfer_ownership'] = False
elif issubclass(type_handler, ReturnValue):
kwargs['caller_owns_return'] = False
else:
raise AssertionError
handler = type_handler(*args, **kwargs)
handler.set_tranformation(self,
self.get_untransformed_name(args[0]))
return handler
def untransform(self, type_handler, declarations,
code_block, expression):
return 'PeekPointer (%s)' % (expression,)
def transform(self, type_handler, declarations,
code_block, expression):
assert type_handler.untransformed_ctype[-1] == '*'
return 'Ptr<%s> (%s)' %
(type_handler.untransformed_ctype[:-1], expression)
transf = SmartPointerTransformation()
typehandlers.return_type_matcher.register_transformation(transf)
typehandlers.param_type_matcher.register_transformation(transf)
del transf
Object = CppClass('Object', incref_method='Ref',
decref_method='Unref')
mod.add_class(Object)
Object.add_constructor(CppConstructor([]))
Object.add_method(CppMethod(ReturnValue.new('void'), 'Dispose', []))
InterfaceId = CppClass('InterfaceId')
mod.add_class(InterfaceId)
Interface = CppClass('Interface', incref_method='Ref',
decref_method='Unref')
mod.add_class(Interface)
Interface.add_method(CppMethod(ReturnValue.new('void'),
'Dispose', []))
class QueryInterfaceMethodWrapper(CppMethod):
def generate_call(self, class_):
"virtual method implementation; do not call"
assert isinstance(class_, CppClass)
pyclass = self.declarations.declare_variable(
'PyTypeObject*', 'pyclass')
pyiid = self.declarations.declare_variable(
'PyObject*', 'pyiid')
retval = self.declarations.declare_variable(
'Ptr<Interface>', 'retval')
pyretval = self.declarations.declare_variable(
'%s*' % Interface.pystruct, 'pyinterface')
self.parse_params.add_parameter(
'O!', ['&PyType_Type', '&'+pyclass], 'interfaceClass')
self.parse_params.add_parameter(
'O!', ['&PyType_Type', '&'+pyclass], 'interfaceClass')
self.before_call.write_error_check(
"!PyType_IsSubtype(%s, &%s)" % (
pyclass, Interface.pytypestruct),
failure_cleanup=
'PyErr_SetString(PyExc_TypeError, '
'"parameter must be a subclass of ns3.Interface");')
self.before_call.write_error_check(
'!(%s = PyObject_GetAttrString((PyObject *) %s, "iid"))'
% (pyiid, pyclass))
self.before_call.add_cleanup_code('Py_DECREF(%s);' % pyiid)
self.before_call.write_error_check(
'!PyObject_IsInstance(%s, (PyObject *) &%s)'
% (pyiid, InterfaceId.pytypestruct),
failure_cleanup=
'PyErr_SetString(PyExc_TypeError, "interface
class \'iid\' '
'member must be an InterfaceId object");')
self.call_params.append(
'*((%s *) %s)->obj' % (InterfaceId.pystruct, pyiid))
method = 'self->obj->%s<Interface>' % self.method_name
self.before_call.write_code(
'%s = %s(%s);' %
(retval, method, ", ".join(self.call_params)))
self.after_call.write_code(
'%s = PyObject_New(%s, %s);'
% (pyretval, Interface.pystruct, pyclass))
self.after_call.write_code(
'%s->obj = GetPointer(%s);' % (pyretval, retval))
self.build_params.add_parameter('N', [pyretval], prepend=True)
QueryInterface = QueryInterfaceMethodWrapper(
ReturnValue.new('void'),
'QueryInterface', [])
Interface.add_method(QueryInterface)
---------- end PYBINDGEN code ----------
GENERATED CODE
It is in the generated code that PYBINDGEN shines. SWIG
generates a 9039-line C++ file for a modest superset of the
ns-3 bindings. The code is pretty ugly. I don't have the
numbers for pybindgen handy, but I recall that it generated a
much smaller, much more readable output. The SWIG file
includes all kinds of goodies including exception handling,
memory leak detection, etc.
------------------------- END NOTE -------------------------
---------------------- BEGIN COMMENTS ---------------------
[Here is a collection of comments made during an initial
Discussion. They are slightly edited and organized for clarity.]
Before proceeding we should really define short term and long
term goals:
1. Do we want only "scripting" or also implement new protocols
in a scripting language?
2. Do we want only Python, or do we want to support multiple
languages?
3. Who do we expect to maintain/provide bindings (thinking in
particular about contributed code)?
4. What do we want the Python API to look like?
Everyone [Gustavo knows] that uses ns-2 says bad things about Tcl.
[Tom] used to think that and had that experience when talking to
people. However, [Tom has] been surprised by having a number of
people express disappointment to me that ns-3 would not support
Tcl, when I told them it was going to be Python. I also have
seen other networking projects adopt Tcl so that they could be
somewhat aligned with ns-2 (e.g., Emulab).
Both Kevin Fall and Steve McCanne have expressed to [Tom] that
they thought the ability in ns-2 to allow users to work almost
entirely in Tcl to rapidly prototype things was a big plus.
One could argue that allowing many languages will only fragment
the ns-3 development community, where each person writes in
his/her favorite language and no one else can read it or modify it.
One conclusion to draw is that nothing will make everyone happy
(ns-2 does have many thousands of users). I also that there are a
large number of users who are not programmers and who learned Tcl
so they could use ns-2 and now are hesitant to have to learn yet
another language.
One thing about the SWIG generated files that scares me is the
.py file. It looks like SWIG only generates function wrappers in
the C extension, and then it creates a layer with Python classes
on top of these functions. And using lambda functions too! And it
rewrites __getattr__ as a python function as well, for some reason.
I suspect that all this adds up to a significant runtime performance
penalty. I really get the feeling that SWIG only wants to get the
bindings working with the minimum of effort and do not pay attention
to performance issues.
---------------------- END COMMENTS ---------------------
More information about the Ns-developers
mailing list