[Ns-developers] ns-3 modular build

[Tom Henderson]

Gustavo, thanks for picking up this thread.  Some responses inline below.

On Tue, 26 Oct 2010 13:58:45 +0100, Gustavo Carneiro
<gjcarneiro at gmail.com>
wrote:
> Hi, some poorly organized thoughts (or issues) I have on modular build
> system.
> 
> 1. We need to be able to select a subset of modules to build; for that,
I
> have revived the waf configure --enable-modules option.  The option is
> working but:
>      a) some example programs do not declare their dependencies
correctly,
>      b) some modules also don't have all dependencies declared (for
> instance, I see the build failing if I request only the 'common' module
> which needs the 'mobility' module but hasn't declared it as dep); this
> needs
> to be fixed progressively, as we find these bugs, but bottom line is
that
> _dependencies matter_;
>      c) The unit test program, utils/test-runner.cc, seems to also
depend
>      on
> some modules unconditionally;

Yes, we abandoned modules a while back to make it easier on people, but
kept the vestiges of them around so that we could plumb them back in. 
Cleaning
this up seems straightforward but I have some questions about the user
API.

The current capability ./waf --enable-modules is not documented and there 
are some questions I have about how it should behave:

  -- how to show all available modules
  -- stickiness of the module enabling.  Once I enable a module, will it
     persist across reconfigurations?  It seems to me that it may deserve
     stickiness across reconfigurations, perhaps staying enabled until
     ./waf distclean or an explicit --disable-modules.
  -- how to specify multiple modules in one argument (comma separated?)
  -- what does "./waf build" build?  All enabled modules?  Or can user
     specify "./waf build modulename"?

> 
> 2. I think it would be a good idea to, in the long term, automatically
> detect available modules, instead of registering them in src/wscript

I agree

> 
>   a) Since modules are distributed in an arbitrarily nested hierarchy,
> finding them is not so easy
>      i) we can add code to recursively list some directory contents,
find
> all subdirs of src recursively, but
>          A) How do we know if a given subdir is a module or something
else?
>  We need to add some kind of marker  => this marker could be actual
module
> metadata file that was planned...

I tend to favor the metadata approach.

> 
>      ii) As an alternative, we could make module directories flat, e.g.
> instead of src/devices/foo and, src/devices/bar, we'd have src/foo and
> src/bar;
>            A) this is admittedly more messy, but at least it's simpler;
>            B) one could argue that a better way to classify modules is
> through metadata (e.g. labels/tags/keywords), and not through what is
the
> location of a module;
>            C) classification through metadata allows richer
classification:
> ontology vs taxonomy

At least, I think we will eventually want to distinguish, by their
location
in the file system, between modules that are maintained in the ns-3 source
tree and (eventual) modules that are pulled from other places.  For
example,
this might be future use of the contrib/ directory.

I tend to prefer not flattening everything for the same reasons that I
prefer not having a flat or ontological filesystem.

> 
>    b) The automatic detection of modules allows scratch-like drop-in of
> modules, always a  boon for developers;
> 
>  3. Maybe it's time to address the issue of (shared) libraries.  I'm
> partially in favor of making every ns-3 module an independent library,
and
> programs have to specify explicitly to which modules they want to link;
>  this should bring linking time down considerably, once the monolithic
> 'helper' module disappears;

that is one of the goals; I believe.  There is probably a core ns-3 module
(consisting of src/core, src/common, and src/simulator) that could be
bundled
together as the required core library, and everything else could be
optional.
Or keep src/common separate from src/core+src/simulator.

> 
>  4. The unit tests (TestSuite) are not modular.  They are built into the
> main ns-3 library, and that needs to stop. ns-3 is not slow, but is
large,
> and these test suites that are built into ns-3 are consuming binary size
> and
> data memory.

I recall that Craig attempted this modularity but concluded that the pain
involved in modifying waf in this regard was (to him) not worth it at
the time.

> 
>  5.  Additionally, test.py lists all tests to run, it should be
> autodetected, maybe?

The C++ unit tests are autodetected but the examples and python tests are
enumerated.

> 
>  6. Python bindings are not modular.  This is not at all simple to fix. 
>  And
> it's best to do in stages of increasing complexity;
> 
>     a) First, make the waf --python-scan drop the apidefs files into
each
> module folder.  I suggest a folder 'apidefs' inside each module;
>         i) We cannot even start this step until all modules are
converted
>         to
> the new layout, obviously;

Couldn't you start this in parallel by adding a new option --pyscan that
did this on a properly organized module?

./waf --pyscan=bridge
./waf --pyscan=uan

pyscan could be renamed to python-scan at the appropriate time.

Again, --python-scan needs to know where to search for API.

> 
>     b) Then modify the code that reads the apidefs and generates the
actual
> bindings to look for the apidefs in these new locations;
> 
>     c) The next step is to modify the api scanning tool to allow it to
scan
> a single module, instead of all modules.  This is very important since,
as
> you all know, scanning the whole API of ns-3 takes a very long time,
these
> days;
> 
>     d) The final and most difficult step is to develop pybindgen and
ns-3
>     to
> allow modular python bindings.  Each ns-3 module would map into a
separate
> Python module binding it.  The Python imports of scripts would reflect
this
> change, for instance "from ns3.flow_monitor import FlowMonitor";

agree, this seems to be a looming API change at the Python level, unless
some
kind of wrapper module for Python can be devised (that allows all ns-3
modules to be imported as ns3 and mostly preserve backward compatibility
with existing programs).

> 
>         i) I have some ideas how this needs to work but it's too early
to
> think of details.  Suffice to say that pybindgen needs to implement a
way
> to
> import types from other Python modules, like the pygtk code generator
used
> to to;
> 
> 
> Anyway, for now I see that we have two steps that anybody is encouraged
to
> help with:
> 
>    1. Convert modules to the new directory layout structure (use
> src/devices/uan as model, a few more have been converted recently, like
> olsr, flow-monitor, bridge, virtual-net-device)

This can be done but we ought to define first what is the granularity of
modules
before moving everything around.

A first cut would be that every subdirectory (applications, mobility,
devices,
internet-stack, node, mpi, etc.) is a module.  Or we can get to a lower
granularity
but doing so seems to increase pain on users to enumerate all the needed
modules.   

> 
>    2. Try to build ns-3 with waf configure --enable-modules=some-module,
if
> it does not build due to a missing or circular dependency, report the
bug
> or
> fix it.
> 

Agree that these should be fixed as they are found.

After having looked at this overall issue this month, and having Mitch
explore
some of this through prototyping, I am thinking that there are pieces that
can
perhaps be pushed forward in parallel without central planning (such as
your
step 2 above) because they will be needed in any case.  However, the
challenge
now seems to be to define how we want this system to eventually work, such
as
- granularity of modules
- impact on Python API
- user experience in enabling/disabling modules
- user experience in writing wscripts
- metadata aspects expressing dependencies and serving as build system
markers
- how third party maintained packages can later be added

Tom