[Ns-developers] Cooperative ns-3 simulation with third party software via JSON-RPC

Alexander Pelov apelov at gmail.com
Wed Jan 21 09:46:57 PST 2009


Hello again,

Thank you for the reply and for pointing out the existence of RPyC !


I think this is the right time to stop for the moment and decide on  
the specific uses of that functionality. In my view, there are several  
orthogonal criteria that may lead to the need of a cooperative  
simulation.

A* Here are the ones I could think of, but maybe I've missed some (The  
application = the third party application willing to use ns-3):
1) The application already exists
2) The application should be compatible with other network simulators
3) The application should be written in another programming language  
(other than C++ and Python)
4) The application should run on a unsupported by ns-3 platform
5) Resource sharing (e.g. a dedicated simulation server with clients  
connecting to execute their code) or performing distributed  
simulations (e.g. several servers running ns-3, governed by a single  
client)
6) For the fun of it

B* Then, there are the typical use cases I can think of (it is  
important to build a more detailed list of these):
1) Set up the scenario
2) Change the scenario during the course of the simulation (e.g. add a  
node, change the position of a node, drop a link)
3) Send/receive packets to/from the simulated network
4) Obtain the simulation results

I would say that points A.4), A.6), B.1) and B.4) are less interesting  
for the moment, as it would be easy to address them without adding the  
complexity of a cooperative simulation (for example generating the  
scenario and uploading it to a ftp server, or downloading the traces  
from such server). Of course, it would be perfect to have all these  
points satisfied by a single solution.

C* We may address these questions by either:
1) Providing a doorway to heaven (RPyC - amazing thing!)
2) Providing a doorway to hell (no matter what the question - CORBA is  
the answer:)
3) Fixing some predefined sets of available functions, which are  
independent of the current implementation
4) Use SWIG and link ns-3 to the calling application

(C.1)
I must say, that PRyC is almost the perfect solution, but  
unfortunately it is a Python only solution (please correct me if I'm  
wrong on this one). You can't even use it if you come from C++  
(bindings are not the answer here, either, because all calls are  
resolved dynamically from Python - even if there is a workaround for  
this, I think it would be something really messy). So C.1 fails to  
satisfy A.2), A.3) and in most cases A.1) - points I personally  
consider very important. I do agree, however that this is the easiest  
way provided that the third party application is written in Python and  
has no problems of being completely ns-3 dependent. So, I would say -  
go for it, distributed simulation guys (there again, you have to  
measure the performance hit).

(C.2)
I was only joking about CORBA. I actually meant Thrift (http://incubator.apache.org/thrift/ 
). In fact, we were considering it as a possible solution candidate.  
The specific RPC technology is not important. The point is that  
someone has to sit and write interface definitions of all/most of ns-3  
functions. This way everyone who can understand the protocol at hand  
can call all functions at will.

(C.3)
Fix a set of functions that are going to be useful to the enduser. She  
doesn't need to know what happens behind the scene, when she calls  
SetPosition(5,0,0,0). She doesn't need to do  
ns3.GetNode(5).SetPosition(new Vector(0,0,0)). And if she does need  
something that does not already exist - she will have to write 1-2  
functions on the ns-3 side of the project and that's it. For the  
implementation we have chosen this approach, with JSON-RPC as driving  
force. The code is not perfect, and there are many things that can be  
done to simplify the life of the users. Or, it could be as easily  
replaced with Thrift, or something else.

(C.4)
I think that this point is somewhat out of the discussion, so I'll  
come back if there is some interest (it rules out A.2), A.4) and A.5))


So, if I have to summarize:
- You want to plug in a simple solution and let the world figure out  
how to use ns-3 from the outside. I think RPyC is doing this for the  
Python part of the word and that's great! With the addition of a  
handful of simple, standard functions thats a solution after my own  
taste. In my own case - I am using Python for my application, so I  
could profit from this solution today - with the exception that I want  
to be able to interface with other simulators, so I would have to  
write the RPC functions anyway.
- Regarding the other rest of the world - there is no magic, plug-o- 
caller (unfortunately). And here, I would rather say that it would be  
nice to have a set of core, standard functions that are callable from  
anyone smart enough (e.g. via JSON-RPC or Thrift) and show them what  
to do in case they want to call something more specific (making sure  
the specific things are not more than 10% of the cases). And by  
extending the core functions, the number of special cases will go down.

Don't get me wrong - I also want to be able to call any function I  
wish, and have the complete power of NS-3 under my (remote) fingers.  
But doing so for everyone would require the interface of each function  
to be defined (which does not seem as an easy task to me). Maybe there  
are some tools that can aid this? I hope so.


Don't hesitate to point out if I missed or misunderstood something.

Regards,
Alex




Le 21 janv. 09 à 16:02, Mathieu Lacage a écrit :

> hi,
>
> Comments inline below: I hope that I answered your comments and that I
> did not misrepresent your email.
>
> On Tue, 2009-01-20 at 20:04 +0100, Alexander Pelov wrote:
>
>> The question about BuildSimulation was worrying us a lot - we wanted
>> to make it as flexible as possible, yet Keep It Simple.
>>
>> The general idea of the whole solution from the beginning was, that
>> the user would have to manually create the topology in ns-3, and the
>> external software would be able to modify the behavior of the
>> different nodes afterwards. Thus, the BuildSimulation is provided  
>> only
>> as a convenience. I do agree that it should be more flexible, and I
>> think that we may have a lot of useful discussions on this subject
>> (for example - how to manage topologies in a non-IP environment,  
>> etc).
>> Could you please sketch how can we make it as general as possible,
>> while KISSing it?
>
> The easiest/simplest way to make the simulation topology setup and
> control flexible is to export all of the ns-3 API and not try to  
> define
> a subset of the ns-3 API.
>
>>> Now, I understand that you might be reluctant to use RPC-XML for
>>> various
>>> reasons but:
>>> - it is totally free because we have a complete python binding so,  
>>> you
>>> can trivially call RPC-XML services _today_ and remotely construct
>>> arbitrary ns3 topologies
>>> - it is orders of magnitude more extendable and flexible
>>> - it will solve your problem (among others): your mobility simulator
>>> is written in python so, you don't even have the excuse that you  
>>> don't
>>> have an XML-RPC layer in your target language
>>>
>>> I believe that you mentioned performance as a potential issue with
>>> XML-RPC (other than sheer dislike of XML): do you have any hard
>>> numbers
>>> which show that this would be problematic for your use-case ? I  
>>> mean,
>>> sure, json is theoretically faster than an xml-based serialization,
>>> but,
>>> does it actually matter for your use-case ?
>>
>>
>> The following part is discussing JSON-RPC vs XML-RPC - a subject I
>> consider not worth getting much into, so if you're in a hurry you can
>> skip it.
>
> [snip]
>
> I have to confess that I am not too interested in the details of the
> on-the-wire protocol so, I did skip it :)
>
>> Why I see the question on XML-RPC vs JSON-RPC as a non-issue - once
>> the functionality is there, it should be (mostly) trivial to change
>> the specific information encoding. XML, JSON, binary - as you wish.
>
> ok. I agree.
>
>> Additionally, I would like to point out, that this is just the
>> beginning of a more complete solution,  and adding Python bindings is
>> the next logical step. This way both the C++ and the Python parts of
>> the ns-3 "world" would be able to use the same functionality with  
>> ease.
>
> ok.
>
>>> To summarize, I see very little added benefit in merging this code
>>> and a
>>> lot of downsides. I would much rather point users who wish to
>>> inter-operate with ns-3 to XML-RPC than this very partial, complex,
>>> and
>>> hard to maintain solution.
>>>
>>
>> If I have understood correctly (correct me if I'm wrong), the major
>> issue which is bothering you is - why bother writing marshalling/
>> unmarshalling code at all? Why not "just" allow the third party to
>> execute whatever function she likes and forget about all that crap?
>
> Yes, that is exactly my question/point, and, indeed, is the main  
> reason
> why I reacted so negatively to the code I reviewed: it brings me 20
> years back in time when there was no other way to build distributed
> applications. (no, I was not writing code at that time :)
>
>> I should say that Python is a great language, so it would probably be
>> feasible to write a single XML-RPC entry point that would go through
>> all existing object in the Python namespace and execute that method  
>> on
>> the fly. That's great - it would save us the need to write code for
>> each RPC function we expose. However, I think that this is going to
>> create a very convoluted way of calling the functions.
>
> Ok, well you forced me to investigate seriously these issues so, here
> are a couple of important points:
>
> 1) I thought that the native python XML-RPC code could use the python
> introspection facilities to automatically export and import functions
> _and_ objects to make remote function/method calls transparent without
> having to write _any_ marshalling/demarshalling code. It turns out  
> that
> I was wrong: the native python XML-RPC code does this for functions,  
> but
> not for objects. This sucks :/
>
> 2) I spent a while investigating other technologies: JSON-RPC as  
> well as
> SOAP and none of these provide the fully-transparent solution I know  
> is
> technically possible but which none of these guys seem very interested
> in. This also sucks :/
>
> 3) However, there is a solution: it is called RPyC:
> http://rpyc.wikidot.com/. It is probably the coolest piece of code I
> have stumbled upon recently. Here is a sample remote session I just  
> run
> on my system:
>
> in one terminal:
> [mlacage at diese ns-3-dev]$ ../rpyc-3.02/rpyc/servers/classic_server.py
>
> in another terminal:
> [mlacage at diese code]$ python
> Python 2.5.2 (r252:60911, Sep 30 2008, 15:41:38)
> [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import rpyc
>>>> conn = rpyc.classic.connect("localhost")
>>>> n0 = conn.modules.ns3.Node ()
>>>> print n0.GetId ()
> 0
>>>> n1 = conn.modules.ns3.Node ()
>>>> print n1.GetId ()
> 1
>>>>
>
> I did not show it but this thing actually supports callbacks so you  
> can
> trivially schedule events _in the server_ from the client and be
> notified _in the client_.
>
>> But this will not solve the problems, as Python bindings are only  
>> what
>> they are - bindings - the return value of a method invocation is a
>> proxy to the underlying C++ object. You cannot automatically marshal/
>> unmarshal that (unless you force ALL methods to provide the necessary
>> methods). This is going to create a true nightmare for all third  
>> party
>
> No: a c++-implemented object is just like a python object. What is
> problematic in all the technologies I looked at is that they don't
> really support the handling of server-side object references. Whether
> these server-side objects are implemented in python or c++ does not
> matter.
>
> [snip other related comments]
>
>> Lets look at the way mobility is implemented in the moment. There are
>> two methods acting on node movement direction - GetDirection and
>> SetDirection, which return/receive a Vector. The  Vector does support
>> an internal serialization format, but it does not have anything to do
>> with XML. So, in the best case, the caller will obtain something she
>> has to parse (after the whole XML-RPC over HTTP processing), change,
>> then generate the fixed string format and send it over the net. And
>> remember - all this is that easy just because there is a killer-all- 
>> in-
>> one-state-of-the-art-reflection-using-python-based-single-entry-XML
>> function.
>>
>> So, we are going to have to write several lines of code for each
>> function exposed via RPC (which is what everyone does) - exactly the
>> approach we have taken. I do agree that the code would be more
>> succinct in Python than in C++, but I think that with the right  
>> helper
>> functions the difference will be minimal, and the user will be able  
>> to
>> harvest the full potential of the C++ roots of ns-3.
>
> I agree that this would not really be an improvement over what you and
> sebastien did but I did not suggest doing this: I believe that a O(1)
> solution is possible (as opposed to a O(n) solution where n is the
> number of objects/methods to export).
>
>> I think that a better approach is to define interfaces - specify a  
>> set
>> of universally accepted methods, along with their parameters and
>> return values in a language-independent way. Doing so would have
>> several benefits:
>>
>> 1) Implementing it in several network simulators would allow third
>> party software to communicate interchangeably with all of them. It
>> would become trivial to write an application in Java and see how it
>> would behave in ns-3 and omnet++ for example.
>>
>> 2) Be implementation independent. Changing an implementation detail
>> would be hidden from the external user.
>>
>> 3) Be easier to use.
>
> I agree that the above would be cool but I see this as an orthogonal
> problem which you don't necessarily have to solve to be able to  
> remotely
> setup and control an ns-3 simulation. The original email sebastien  
> sent
> stated that this was the goal so, if you define new goals, then, yes,
> there might be new things to consider :)
>
>> I'm sure this list could go on, but I think that these reasons are
>> sufficient.
>>
>>
>>> I am really very sorry to be so negative after the review :/
>>>
>>> Mathieu
>>>
>>
>>
>> In conclusion I have to say, that maybe we didn't prepare the
>> announcement to the full extent, and we have taken note from your
>> remarks. We are going to prepare an additional document explaining  
>> the
>> proposed RPC functions, so that we could have a fruitful discussion  
>> on
>> concrete use-cases.
>
> I agree that this will be useful: if you can come up with a use-case
> which cannot be fulfilled with the rpyc-based solution I outlined  
> above,
> that would be a strong incentive to consider another option.
>
> thanks for taking time to answer my initial email,
> Mathieu
>






More information about the Ns-developers mailing list