[Ns-developers] ns-3 modular build
tomh at tomh.org
Tue Oct 26 08:54:13 PDT 2010
Gustavo, thanks for picking up this thread. Some responses inline below.
On Tue, 26 Oct 2010 13:58:45 +0100, Gustavo Carneiro
<gjcarneiro at gmail.com>
> Hi, some poorly organized thoughts (or issues) I have on modular build
> 1. We need to be able to select a subset of modules to build; for that,
> have revived the waf configure --enable-modules option. The option is
> working but:
> a) some example programs do not declare their dependencies
> b) some modules also don't have all dependencies declared (for
> instance, I see the build failing if I request only the 'common' module
> which needs the 'mobility' module but hasn't declared it as dep); this
> to be fixed progressively, as we find these bugs, but bottom line is
> _dependencies matter_;
> c) The unit test program, utils/test-runner.cc, seems to also
> some modules unconditionally;
Yes, we abandoned modules a while back to make it easier on people, but
kept the vestiges of them around so that we could plumb them back in.
this up seems straightforward but I have some questions about the user
The current capability ./waf --enable-modules is not documented and there
are some questions I have about how it should behave:
-- how to show all available modules
-- stickiness of the module enabling. Once I enable a module, will it
persist across reconfigurations? It seems to me that it may deserve
stickiness across reconfigurations, perhaps staying enabled until
./waf distclean or an explicit --disable-modules.
-- how to specify multiple modules in one argument (comma separated?)
-- what does "./waf build" build? All enabled modules? Or can user
specify "./waf build modulename"?
> 2. I think it would be a good idea to, in the long term, automatically
> detect available modules, instead of registering them in src/wscript
> a) Since modules are distributed in an arbitrarily nested hierarchy,
> finding them is not so easy
> i) we can add code to recursively list some directory contents,
> all subdirs of src recursively, but
> A) How do we know if a given subdir is a module or something
> We need to add some kind of marker => this marker could be actual
> metadata file that was planned...
I tend to favor the metadata approach.
> ii) As an alternative, we could make module directories flat, e.g.
> instead of src/devices/foo and, src/devices/bar, we'd have src/foo and
> A) this is admittedly more messy, but at least it's simpler;
> B) one could argue that a better way to classify modules is
> through metadata (e.g. labels/tags/keywords), and not through what is
> location of a module;
> C) classification through metadata allows richer
> ontology vs taxonomy
At least, I think we will eventually want to distinguish, by their
in the file system, between modules that are maintained in the ns-3 source
tree and (eventual) modules that are pulled from other places. For
this might be future use of the contrib/ directory.
I tend to prefer not flattening everything for the same reasons that I
prefer not having a flat or ontological filesystem.
> b) The automatic detection of modules allows scratch-like drop-in of
> modules, always a boon for developers;
> 3. Maybe it's time to address the issue of (shared) libraries. I'm
> partially in favor of making every ns-3 module an independent library,
> programs have to specify explicitly to which modules they want to link;
> this should bring linking time down considerably, once the monolithic
> 'helper' module disappears;
that is one of the goals; I believe. There is probably a core ns-3 module
(consisting of src/core, src/common, and src/simulator) that could be
together as the required core library, and everything else could be
Or keep src/common separate from src/core+src/simulator.
> 4. The unit tests (TestSuite) are not modular. They are built into the
> main ns-3 library, and that needs to stop. ns-3 is not slow, but is
> and these test suites that are built into ns-3 are consuming binary size
> data memory.
I recall that Craig attempted this modularity but concluded that the pain
involved in modifying waf in this regard was (to him) not worth it at
> 5. Additionally, test.py lists all tests to run, it should be
> autodetected, maybe?
The C++ unit tests are autodetected but the examples and python tests are
> 6. Python bindings are not modular. This is not at all simple to fix.
> it's best to do in stages of increasing complexity;
> a) First, make the waf --python-scan drop the apidefs files into
> module folder. I suggest a folder 'apidefs' inside each module;
> i) We cannot even start this step until all modules are
> the new layout, obviously;
Couldn't you start this in parallel by adding a new option --pyscan that
did this on a properly organized module?
pyscan could be renamed to python-scan at the appropriate time.
Again, --python-scan needs to know where to search for API.
> b) Then modify the code that reads the apidefs and generates the
> bindings to look for the apidefs in these new locations;
> c) The next step is to modify the api scanning tool to allow it to
> a single module, instead of all modules. This is very important since,
> you all know, scanning the whole API of ns-3 takes a very long time,
> d) The final and most difficult step is to develop pybindgen and
> allow modular python bindings. Each ns-3 module would map into a
> Python module binding it. The Python imports of scripts would reflect
> change, for instance "from ns3.flow_monitor import FlowMonitor";
agree, this seems to be a looming API change at the Python level, unless
kind of wrapper module for Python can be devised (that allows all ns-3
modules to be imported as ns3 and mostly preserve backward compatibility
with existing programs).
> i) I have some ideas how this needs to work but it's too early
> think of details. Suffice to say that pybindgen needs to implement a
> import types from other Python modules, like the pygtk code generator
> to to;
> Anyway, for now I see that we have two steps that anybody is encouraged
> help with:
> 1. Convert modules to the new directory layout structure (use
> src/devices/uan as model, a few more have been converted recently, like
> olsr, flow-monitor, bridge, virtual-net-device)
This can be done but we ought to define first what is the granularity of
before moving everything around.
A first cut would be that every subdirectory (applications, mobility,
internet-stack, node, mpi, etc.) is a module. Or we can get to a lower
but doing so seems to increase pain on users to enumerate all the needed
> 2. Try to build ns-3 with waf configure --enable-modules=some-module,
> it does not build due to a missing or circular dependency, report the
> fix it.
Agree that these should be fixed as they are found.
After having looked at this overall issue this month, and having Mitch
some of this through prototyping, I am thinking that there are pieces that
perhaps be pushed forward in parallel without central planning (such as
step 2 above) because they will be needed in any case. However, the
now seems to be to define how we want this system to eventually work, such
- granularity of modules
- impact on Python API
- user experience in enabling/disabling modules
- user experience in writing wscripts
- metadata aspects expressing dependencies and serving as build system
- how third party maintained packages can later be added
More information about the Ns-developers