NDK Performance Issue, raw socket drops packets

Mitch Nelson

Problem: Raw socket drops incoming packets.

Platform: Evm6748, current NDK, NSP

The problem is demonstrated in an application that opens a raw socket and then echoes raw packets back to the remote host. The program receives buffers and then has to release them. When packets are sent to the DSP too quickly, they are dropped. It appears that the drop rate is sensitive to how soon the buffer is freed..

In the code snippet copied below, the function sendmessage() can be a simple send(), or getsendncbuff() followed by mmCopy() andsend(). The problem behavior is the same regardless of what is in the send function.

Here is the behavior

If the remote sends ethernet packets one at a time, everything works very well.
If the remote sends two packets with no delay in between, one packet is received and one packet is dropped.
If the remote sends five packets, then two out of five pacekts are dropped.
If the call to sendmessage() is commented out, then recvnc() receives all of the packets that are sent.
If a mmCopy() and recvncfree( ) are done before sendmessage() instead of the recvncfree() afterwards, then one in five packets is dropped.

Behavior 5 is interesting. The loop is longer because of the extra mmCopy(), yet the drop rate is lower (compare to 3.). The difference is that the buffer is freed sooner. In other words, it looks like it does not have enough buffers, or perhaps the raw socket only has one buffer.

Mitch Nelson

Code Snipper for the problem report:

fdOpenSession( TaskSelf() );

hRawSocket = socket(AF_RAWETH, SOCK_RAWETH, rawether_type );

val = 1;

retval = setsockopt( hRawSocket, SOL_SOCKET, SO_IFDEVICE, &val,

sizeof(val) );

val = rawchannel_num;

retval = setsockopt(hRawSocket, SOL_SOCKET, SO_PRIORITY, &val,

sizeof(val));

val = 8192;

retval = setsockopt(hRawSocket, SOL_SOCKET, SO_RCVBUF, &val,

sizeof(val));

to.tv_sec = 0;

to.tv_usec = 0;

setsockopt( hRawSocket, SOL_SOCKET, SO_SNDTIMEO, &to,

sizeof( to ) );

setsockopt( hRawSocket, SOL_SOCKET, SO_RCVTIMEO, &to,

sizeof( to ) );

for( ; ; )

{

LOG_printf(&trace,

"linkstatus %d, calling recvnc\n", preprocessor_linkstatus );

retval = (

int) recvnc( hRawSocket, (void **) &prcvbuf, 0, &hrcvbuf );

if ( retval < 0 ) {

);

else if ( retval > 14 ) {

LOG_printf(&trace,

"recvnc returned %d\n", retval );

sendmessage( (unsigned char *)rawbuffer, nsend ) == ENOBUFS );

recvncfree()

}

over 14 years ago

Steven Connell over 14 years ago

TI__Mastermind 45025 points

Hi Mitch,

Just to see if the problem really is due to the buffer size, have you tried setting the receive buffer size to be larger than 8K? E.g.:

val = 16384;
retval = setsockopt(hRawSocket, SOL_SOCKET, SO_RCVBUF, &val, sizeof(val));

I also noticed that you have set the receive timeouts to be 0. Have you tried increasing these timeout values?

to.tv_sec = 0;
to.tv_usec = 0;
setsockopt( hRawSocket, SOL_SOCKET, SO_SNDTIMEO, &to, sizeof( to ) );
setsockopt( hRawSocket, SOL_SOCKET, SO_RCVTIMEO, &to, sizeof( to ) );

If the above doesn't help, you may need to increase the buffer sizes of the Packet Buffer Manager (PBM) and/or the NDK's memory manager (in mem.c) and rebuild NDK libraries to get those changes into your application.

I see that a packet is allocated in one of two ways. Basically, the flow is RawEthSockCreatePacket() calls -> NIMUCreatePacket() calls -> PBM_alloc().

PBM_alloc() then calls either (in pbm.c):

A. PBM_deq()

B. mmAlloc()

To allocate the packet. If there isn't enough memory to allocate, these functions fail and that results in the ENOBUFS error that you are seeing. So, you'd need to increase the available memory that's available for allocation in either the PBM or in mem.c, depending on whether case A. or B. is hit above. You could determine if A. or B. is happening by setting breakpoints in the code, or just increase the available memory for both cases, as I'll describe below.

The PBM uses the following (in ti/ndk/src/os/pbm.c):

// Number of buffers in PBM packet buffer free pool
//
// The number of buffers in the free pool can have a significant effect
// on performance, especially in UDP packet loss. Increasing this number
// will increase the size of the static packet pool use for both sending
// and receiving packets.
//
// DM642 Users Note: The DM642 Ethernet driver assumes that its local
// buffer allocation (EMAC_MAX_RX in dm642.c) plus PKT_NUM_FRAMEBUF
// defined below is less than or equal to 256. The default value for
// EMAC_MAX_RX in the DM642 Ethernet driver is 16.
//
#define PKT_NUM_FRAMEBUF 192

// Max size buffer
//
// On L2 cached CPUs, this size must be cache-line multiple
// The LogicIO Etherent (Macronix MAC) requires a larger buffer because
// it transfers data in 16 byte chunks, and with its pre-pad and data
// alignment, it will overflow a 1536 byte buffer.
//
// If the LogicIO/Maxcronix support is not required, this value can
// be reduced to 1536.
//
#define PKT_SIZE_FRAMEBUF 1664

The memory manager creates buffer sizes using these (in ti/ndk/src/os/mem.c):

#define RAW_PAGE_SIZE 3072
#define RAW_PAGE_COUNT 16

...

// P.I.T.
#pragma DATA_SECTION(pit, ".far:NDK_MMBUFFER");
PITENTRY pit[RAW_PAGE_COUNT];
#pragma DATA_SECTION(pitBuffer, ".far:NDK_MMBUFFER");
UINT8 pitBuffer[RAW_PAGE_SIZE*RAW_PAGE_COUNT];

A quick way to get these new values into your app is to add these stack sources into your project and rebuild. This will cause the linker to pick up the object files from the stack sources in your project ahead of the library objects (that contain the old sizes). You may need to add some include search paths to your project to find headers included by these stack sources, though.

Steve

Mitch Nelson over 14 years ago in reply to Steven Connell

Expert 1440 points

Steve

i increased the recvbuf setting to 16K and it still drops incoming packets.

Note that we are talking about two packets, 1000 bytes each, and unless there is a pause in between, it drops the second packet.

Also recall that we have a similar problem on send. If we do a getsendncbuff() immediately after calling sendnc(), we get a n error ENOBUFS.

Your post seems to suggest that it has 192 packet bufffers available. Yet it invariably fails the second packet for both sendnc() and recvnc().

To me that looks a lot like somebody is not chaining buffers, or at least not correctly.

thank you

Mitch

Steven Connell over 14 years ago in reply to Mitch Nelson

TI__Mastermind 45025 points

Mitch,

Did you try increasing the #define values (used for buffers) in pbm.c and mem.c?

#define PKT_NUM_FRAMEBUF 192

#define PKT_SIZE_FRAMEBUF 1664

#define RAW_PAGE_SIZE 3072
#define RAW_PAGE_COUNT 16

If that doesn't work, the next thing I can think of is if you can try to put some break points at the following lines in pbm.c to see which of the two places (A. or B. mentioned in the previous post) are being hit?

PBM_Handle PBM_alloc( uint MaxSize )
{
    PBM_Pkt     *pPkt = 0;

    // Verify we're open
    if( !IsOpen )
        return(0);

    // Allocate Buffer off the Free Pool is size is OK
    if( MaxSize <= PKT_SIZE_FRAMEBUF )
        pPkt = (PBM_Pkt *)PBMQ_deq( &PBMQ_free ); // <- put a break point here
    else
    {
        // Allocate header from memory
        pPkt = (PBM_Pkt *)mmAlloc( sizeof(PBM_Pkt) ); // <- put a break point here
        if( pPkt )
        {
            pPkt->Type        = PBM_MAGIC_ALLOC;
            pPkt->BufferLen   = MaxSize;

            // Allocate Buffer from Memory
            // mmAlloc() is safe at interrupt time
            if( MaxSize <= MMALLOC_MAXSIZE )
                pPkt->pDataBuffer = mmAlloc( MaxSize );
            else
            {
                // Note: If the system needs to support packet buffers

...

The idea is to find out which of the above calls is the one that's failing. Once you know which one is failing, then hopefully this will give us a clue as to why.

Also, can you remind me of your configuration again? I believe you have the DSP connected directly to a Linux box via crossover cable on a closed network, correct?

Lastly, since you are using your own protocol over raw Ethernet, does this protocol have any re-transmit logic coded into it? Or is it just a one shot like UDP?

Steve

Mitch Nelson over 14 years ago in reply to Steven Connell

Expert 1440 points

I did not try that. and i am willing to give it a try, i am a bit overloaded at the moment. it will take about half hour to set up the test.

Meanwhile, i have to admit that i am a little confused as to why this would help. We are talking about two packets here. It fails on the second packet for recvnc() or getsendncbuff(). Why would 192 not be enough?

Also, i might understand how the getsendncbuff() would fail in PBM_alloc(). But how would that come into the failure in recvnc()?

Processors

Processors forum

NDK Performance Issue, raw socket drops packets