[Ns-developers] [ns3] socket API

[Tom Henderson]

>-----Original Message-----
>From: Mathieu Lacage [mailto:mathieu.lacage at sophia.inria.fr]
>Sent: Tuesday, April 15, 2008 09:48 AM
>To: 'Tom Henderson'
>Cc: 'ns-developers'
>Subject: Re: [Ns-developers] [ns3] socket API
>
>
>On Tue, 2008-04-15 at 17:10 +0000, Tom Henderson wrote:
>
>> If I understand correctly, the efficiency goal here is to try to avoid
>> buffer copies at the various APIs.
>
>The goal is to avoid buffer copies _and_ memory allocations for "fake"
>payload. The latter is actually _the_ critical constraint for the
>use-case sally described in our first meeting in berkeley because memory
>allocations for unused payload will quickly dominate and bound fast-tcp
>simulations.
>
>> 
>> In the send direction, whether Send (Ptr<Packet>) or Send (const
>> uint8_t* buf) is used does not matter; one copy into the packet buffer
>> is performed in either case, and a fake buffer is supported similarly
>> in either case.
>
>No, the two cases are different:
>
>// no memory allocated for payload.
>Ptr<Packet> p = Create<Packet> (100000);
>socket->Send (p);
>
>Socket::Send (Ptr<const Packet> p)
>{
>  // copies just the packet data structure: does not allocate 
>  // memory for the payload.
>  m_list.push_back (p->Copy ());
>}
>
>and the second case:
>
>uint8_t buffer[100000];
>memset (buffer, 100000, 0);
>socket->Send (buffer);
>Socket::Send (uint8_t *buffer)
>{
>  // allocate a packet buffer of size 100000 and copy zeros in it.
>  Ptr<Packet> p = Create<Packet> (buffer);
>  m_list.push_back (p);
>}
>
>So, in the second case, you:
>
>  - always allocate a buffer for the payload
>  - always memcpy it
>  - have to memset it.
>
>while in the first case, none of the above happens.

I was assuming that you just pass a null pointer, but set the size parameter to the size of fake data, and the socket converts it to the null packet for you.  This is already supported:

int Socket::SendTo (const Address &address, const uint8_t* buf, uint32_t size)
{
  NS_LOG_FUNCTION;
  Ptr<Packet> p;
  if(buf)
    {
      p = Create<Packet> (buf, size);
    }
  else
    {
      p = Create<Packet> (size);
    }
  return SendTo (address,p);
}

>
>
>> In the receive direction, if you want to support recv/recvfrom
>> semantics, the choices are Recv (..., Ptr<Packet>, ...) or Recv (...,
>> uint8_t* buf, ...).
>> 
>> In the case of fake data, there is no efficiency difference because
>> zero bytes are copied.  In the case of real data, the use of the buf
>
>There is a difference, even in the case of fake data: If you have fake
>data, Recv (uint8_t *buf) will require that you actually convert that
>fake data to a real buffer and copy it. The Ptr<Packet> version, on the
>other hand, will not require that last conversion and will track of the
>fake data without allocating it.

Similarly, I was assuming that you could implement a variant of recv(uint8_t* buf) that worked with fake data.  for instance, Socket::Recv (0, len); and you implement a check in the socket to avoid copying in any data to a null pointer.

>
>To be clear, an example:
>
>// don't allocate anything.
>Ptr<Packet> p = 0;
>
>p = socket->Recv (10);
>
>Ptr<Packet> Socket::Recv (uint32_t maxSize)
>{
>  Ptr<Packet> p = m_list.front ();
>  if (p->GetSize () > maxSize)
>  {
>    // must split next packet.
>    Ptr<Packet> left = p->CreateFragment (maxSize, p->GetSize () -
>maxSize);
>    m_list.pop_front ();
>    m_list.push_front (left);
>    // return the requested size.
>    return p->CreateFragment (maxSize);
>  } 
>  else
>  {
>    // no need to split.
>    m_list.pop_front ();
>    return p->Copy ();
>  } 
>}
>
>None of the above allocates _any_ buffer for the payload if it was fake
>because CreateFragment is smart enough to not to it. That, however,
>clearly assumes that you have used Packet::Packet (uint32_t size) to
>create the original payload packet, that is, that you have used
>Socket::Send (Ptr<Packet>) and not the Send (uint8_t *buffer) version.
>
>On the other hand, the following:
>
>uint8_t buffer[1024];
>
>int recved = socket->Recv (buffer, 1024);
>
>int
>Socket::Recv (uint8_t *buffer, uint32_t size)
>{
>  Ptr<Packet> p = Recv (size);
>  uint8_t *buf = p->PeekData ();
>  memcpy (buffer, buf, p->GetSize ());
>  return p->GetSize ();
>}
>
>will require that you:
>  - perform an extra memcpy
>  - if the data in the packet was fake, PeekData will make that fake
>data become real so, even if the sending side was super-careful to
>allocate payload with Packet::Packet (uint32_t size), then, you lose
>here because you end up actually allocating a buffer for the data
>anyway.
>
>>  version requires a copy from Packet to buf.  If you use the Recv
>> (Ptr<Packet>) version, you can avoid this copy if you use
>> Packet::PeekData().  Is this what you mean by _really_ efficient
>
>PeekData will trigger a real data buffer allocation. As soon as you
>start using a real byte buffer, the fake data becomes real. Example:
>
>// no buffer allocated for payload.
>Ptr<Packet> p = Create<Packet> (100000);
>
>// buffer magically allocated: fake data has become real zeros.
>uint8_t *buf = p->PeekData ();
>
>>  synchronous applications?   Otherwise, it doesn't seem like the two
>> variants differ in efficiency, for the common case.
>> 
>> There are also cases in which the Recv does not align with the Packet
>> boundaries as Packets were delivered to the receive socket.  For
>> instance, in TCP, you may have a read that spans multiple packet (and
>> packet Buffers).  The application may also read less than what is
>> available.  In these cases, do you end up losing the _really_
>> efficient operation of the Ptr<Packet> version, when your read spans
>> underlying Buffers?  
>
>No: In the case of fake data stored in a Ptr<Packet>, Ptr<Packet> is
>still smart enough to keep track of slices of the fake data if it is
>sliced.

but in the case of real data?

>
>> 
>> In the case of a large read, wouldn't you have to create a new Packet
>> that concatenated the underlying Buffers (and hence required a copy
>> into this new Buffer)?  If not, how would that work?
>
>If you kept track of the received data with a list of pointers to
>packets, you can aggregate them with Packet::AddAtEnd and that will also
>be smart enough to avoid buffer allocations in case of fake data.

but in the case of real data?

>
>> 
>> In the case of the small read, you may have multiple Packet objects
>> pointing at the same Buffer, with differing offsets (m_start, m_end),
>> so unless the read spans Buffers, then it seems like you can also
>> avoid copies here. 
>
>I am sorry but I do not understand what you mean by this.

(for real data)  You received two packets of size 1000, but your application does two reads of size 700 followed by a read of 600.  What happens to the Packet buffer on this second read, which spans multiple received Buffers?

The basic question is, when you have real data, if you implement the receive buffer in the socket using the received Packets (to avoid copies of the underlying Buffers), what happens when your reads do not align with these Buffer boundaries-- do copies happen?

>> 
>> So, in summary, it seems to me that the real issue is that in the
>> receive direction, you may be able to get by in many (but not all)
>> cases by effectively passing a pointer (to const) to the underlying
>> Buffer up to the application, since the smart pointer semantics of
>> Packet make this a safe operation.  But if you use the "const uint8_t*
>> buf" variant of recv, you always must copy into buf, from a memory
>> management perspective.  Is this an accurate summary?
>
>I don't think so. As soon as you start manipulating a raw buffer, you
>have to populate it with data, whether it is just a bunch of fake zeros
>or not. So, you have allocate the buffer to be big enough to contain
>that data, hence, you lose from a memory-management perspective. If,
>instead, you use a Ptr<Packet> with fake data, you never allocate the
>buffer:
>
>// 100 KB of fake data.
>Ptr<Packet> p = Create<Packet> (100000);
>
>// still 10KB of fake data.
>Ptr<Packet> a = p.CreateFrament (10000);
>
>// still 90KB of fake data.
>Ptr<Packet> b = p.CreateFrament (90000);
>
>// 100KB of fake data is back.
>a->AddAtEnd (b);
>
>All of that fancy super-efficient lack of memory allocation was
>implemented to support the one case sally cared about, that, is being
>able to deal with as many fast-tcp streams as humanly possible without
>overflowing the system with memory allocations for the payload which,
>for her use-cases, was considered useless.
>
>So, as far as I can tell, a better summary of the issues is that, if you
>use the API I proposed consistently in applications and in the socket
>code, you will be able to avoid:
>  - allocating buffers for fake payload
>  - copying the fake payload around
>
>but, if you put anywhere in the source or destination code paths a real
>raw buffer, you will have to convert your data to a raw buffer and that
>will trigger extra memcpys as well as extra memory allocations because
>the fake data will become real.

It still seems to me that, for fake data, whether you use the byte buffer version or Packet version of these socket calls does not matter (or does not have to matter), since in either case you can avoid allocating buffers for fake payload.

Tom