[Ns-developers] [ns3] socket API

[Mathieu Lacage]

On Tue, 2008-04-15 at 17:10 +0000, Tom Henderson wrote:

> If I understand correctly, the efficiency goal here is to try to avoid
> buffer copies at the various APIs.

The goal is to avoid buffer copies _and_ memory allocations for "fake"
payload. The latter is actually _the_ critical constraint for the
use-case sally described in our first meeting in berkeley because memory
allocations for unused payload will quickly dominate and bound fast-tcp
simulations.

> 
> In the send direction, whether Send (Ptr<Packet>) or Send (const
> uint8_t* buf) is used does not matter; one copy into the packet buffer
> is performed in either case, and a fake buffer is supported similarly
> in either case.

No, the two cases are different:

// no memory allocated for payload.
Ptr<Packet> p = Create<Packet> (100000);
socket->Send (p);

Socket::Send (Ptr<const Packet> p)
{
  // copies just the packet data structure: does not allocate 
  // memory for the payload.
  m_list.push_back (p->Copy ());
}

and the second case:

uint8_t buffer[100000];
memset (buffer, 100000, 0);
socket->Send (buffer);
Socket::Send (uint8_t *buffer)
{
  // allocate a packet buffer of size 100000 and copy zeros in it.
  Ptr<Packet> p = Create<Packet> (buffer);
  m_list.push_back (p);
}

So, in the second case, you:

  - always allocate a buffer for the payload
  - always memcpy it
  - have to memset it.

while in the first case, none of the above happens.


> In the receive direction, if you want to support recv/recvfrom
> semantics, the choices are Recv (..., Ptr<Packet>, ...) or Recv (...,
> uint8_t* buf, ...).
> 
> In the case of fake data, there is no efficiency difference because
> zero bytes are copied.  In the case of real data, the use of the buf

There is a difference, even in the case of fake data: If you have fake
data, Recv (uint8_t *buf) will require that you actually convert that
fake data to a real buffer and copy it. The Ptr<Packet> version, on the
other hand, will not require that last conversion and will track of the
fake data without allocating it.

To be clear, an example:

// don't allocate anything.
Ptr<Packet> p = 0;

p = socket->Recv (10);

Ptr<Packet> Socket::Recv (uint32_t maxSize)
{
  Ptr<Packet> p = m_list.front ();
  if (p->GetSize () > maxSize)
  {
    // must split next packet.
    Ptr<Packet> left = p->CreateFragment (maxSize, p->GetSize () -
maxSize);
    m_list.pop_front ();
    m_list.push_front (left);
    // return the requested size.
    return p->CreateFragment (maxSize);
  } 
  else
  {
    // no need to split.
    m_list.pop_front ();
    return p->Copy ();
  } 
}

None of the above allocates _any_ buffer for the payload if it was fake
because CreateFragment is smart enough to not to it. That, however,
clearly assumes that you have used Packet::Packet (uint32_t size) to
create the original payload packet, that is, that you have used
Socket::Send (Ptr<Packet>) and not the Send (uint8_t *buffer) version.

On the other hand, the following:

uint8_t buffer[1024];

int recved = socket->Recv (buffer, 1024);

int
Socket::Recv (uint8_t *buffer, uint32_t size)
{
  Ptr<Packet> p = Recv (size);
  uint8_t *buf = p->PeekData ();
  memcpy (buffer, buf, p->GetSize ());
  return p->GetSize ();
}

will require that you:
  - perform an extra memcpy
  - if the data in the packet was fake, PeekData will make that fake
data become real so, even if the sending side was super-careful to
allocate payload with Packet::Packet (uint32_t size), then, you lose
here because you end up actually allocating a buffer for the data
anyway.

>  version requires a copy from Packet to buf.  If you use the Recv
> (Ptr<Packet>) version, you can avoid this copy if you use
> Packet::PeekData().  Is this what you mean by _really_ efficient

PeekData will trigger a real data buffer allocation. As soon as you
start using a real byte buffer, the fake data becomes real. Example:

// no buffer allocated for payload.
Ptr<Packet> p = Create<Packet> (100000);

// buffer magically allocated: fake data has become real zeros.
uint8_t *buf = p->PeekData ();

>  synchronous applications?   Otherwise, it doesn't seem like the two
> variants differ in efficiency, for the common case.
> 
> There are also cases in which the Recv does not align with the Packet
> boundaries as Packets were delivered to the receive socket.  For
> instance, in TCP, you may have a read that spans multiple packet (and
> packet Buffers).  The application may also read less than what is
> available.  In these cases, do you end up losing the _really_
> efficient operation of the Ptr<Packet> version, when your read spans
> underlying Buffers?  

No: In the case of fake data stored in a Ptr<Packet>, Ptr<Packet> is
still smart enough to keep track of slices of the fake data if it is
sliced.

> 
> In the case of a large read, wouldn't you have to create a new Packet
> that concatenated the underlying Buffers (and hence required a copy
> into this new Buffer)?  If not, how would that work?

If you kept track of the received data with a list of pointers to
packets, you can aggregate them with Packet::AddAtEnd and that will also
be smart enough to avoid buffer allocations in case of fake data.

> 
> In the case of the small read, you may have multiple Packet objects
> pointing at the same Buffer, with differing offsets (m_start, m_end),
> so unless the read spans Buffers, then it seems like you can also
> avoid copies here. 

I am sorry but I do not understand what you mean by this.

> 
> So, in summary, it seems to me that the real issue is that in the
> receive direction, you may be able to get by in many (but not all)
> cases by effectively passing a pointer (to const) to the underlying
> Buffer up to the application, since the smart pointer semantics of
> Packet make this a safe operation.  But if you use the "const uint8_t*
> buf" variant of recv, you always must copy into buf, from a memory
> management perspective.  Is this an accurate summary?

I don't think so. As soon as you start manipulating a raw buffer, you
have to populate it with data, whether it is just a bunch of fake zeros
or not. So, you have allocate the buffer to be big enough to contain
that data, hence, you lose from a memory-management perspective. If,
instead, you use a Ptr<Packet> with fake data, you never allocate the
buffer:

// 100 KB of fake data.
Ptr<Packet> p = Create<Packet> (100000);

// still 10KB of fake data.
Ptr<Packet> a = p.CreateFrament (10000);

// still 90KB of fake data.
Ptr<Packet> b = p.CreateFrament (90000);

// 100KB of fake data is back.
a->AddAtEnd (b);

All of that fancy super-efficient lack of memory allocation was
implemented to support the one case sally cared about, that, is being
able to deal with as many fast-tcp streams as humanly possible without
overflowing the system with memory allocations for the payload which,
for her use-cases, was considered useless.

So, as far as I can tell, a better summary of the issues is that, if you
use the API I proposed consistently in applications and in the socket
code, you will be able to avoid:
  - allocating buffers for fake payload
  - copying the fake payload around

but, if you put anywhere in the source or destination code paths a real
raw buffer, you will have to convert your data to a raw buffer and that
will trigger extra memcpys as well as extra memory allocations because
the fake data will become real.

Mathieu