• Locked Locked
  • Replies 4 replies
  • Subscribers 103 subscribers
  • Views 3704 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

NDK transmit packet-dropping

Ralf Goebel
Genius 3715 points

Hi,

the NDK and TI's packet drivers are designed to drop transmit packets if the device's transmit buffers are full. This can cause undesired behaviour of TCP and UDP sockets, if the CPU is putting packets to the TX queue faster than the link can handle.

For UDP sockets, if the application sends multiple packets at once, packets will be randomly dropped without any notification to the application. I know that UDP packed delivery is unreliable by definition, but a blocking behaviour of the sendto() function or returning EWOULDBLOCK for non-blocking sockets would be much better in my opinion.

This problem can also cause a huge performance impact for TCP if the transmit buffers are depleted. For example, the NIMU packet driver in the C66-MCSDK uses only 16 transmit buffers. If the TCP transmit buffer is configured to hold 32kB of data, packets will be continuously be dropped. The TCP retransmission mechanism will keep the transmission reliable, but it also gets very slow. This problem can also occur if multiple TCP sockets are sending data simultaneously.

I think a cleaner solution than dropping the packets without notification would be to notify the UDP and TCP layers about the packet drop. Then these layers could decide to block until space gets available or to return an error to the application. But I also understand that this would be a major change to the NDK.

We worked around this problem by changing the packet driver: If the transmit buffer is full, we will spin in a loop until space gets available. It's not a very nice solution, but it seems to work.

Ralf

  • TI__Mastermind 45025 points

    Hi Ralf,

    Thanks for pointing this out, feedback is definitely appreciated.

    Ralf Goebel said:
    This problem can also cause a huge performance impact for TCP if the transmit buffers are depleted. For example, the NIMU packet driver in the C66-MCSDK uses only 16 transmit buffers. If the TCP transmit buffer is configured to hold 32kB of data, packets will be continuously be dropped. The TCP retransmission mechanism will keep the transmission reliable, but it also gets very slow. This problem can also occur if multiple TCP sockets are sending data simultaneously.

    Which exact C66 device are you using?  I'm not too familiar with the C66 EMAC driver, but I'm curious about the transmit buffer size of 16 you mention.  Could you point me to that?  Which file, line of code, and in which version of the MCSDK?

    I'm wondering if you could modify the driver to increase the number of frame buffers to coincide with the TCP send (or receive) buffer size(s).

    Ralf Goebel said:
    We worked around this problem by changing the packet driver: If the transmit buffer is full, we will spin in a loop until space gets available. It's not a very nice solution, but it seems to work.

    So after you did this you saw a significant decrease in the number of TCP retransmits?  It would be great if you could share this as well, as your solution may be helpful to others in the community.

    Steve

  • in reply to Steven Connell
    Genius 3715 points

    Hi Steve,

    I'm using the C6678 device, but I think other C66 devices are using the same MCSDK code.

    The source file is resource_mgr.h, line 155 (C6678 PDK 1.1.2.6 / MCSDK 2.1.2.6):

    #define NIMU_NUM_TX_DESC                16u /**< Maximum number of TX descriptors used by NIMU */

    If I increase the number of TX descriptors to hold the complete TCP transmit buffer, the packet dropping goes away. But it doesn't solve the problem when multiple sockets or UDP is used.

    After my change to the packet driver, I didn't see any lost packets.
    In nimu_eth.c / line 517, I changed the following code within EmacSend():

        if ((QMSS_QPOP (gTxFreeQHnd, QHANDLER_QPOP_FDQ_NO_ATTACHEDBUF, (Cppi_HostDesc **)&pCppiDesc)) != NULL)
        {
    //      gTxDropCounter++;
    //      goto error_cond;

            // Wait for transmitted packet to return
            while ((QMSS_QPOP (gTxReturnQHnd, QHANDLER_QPOP_FDQ_ATTACHEDBUF, (Cppi_HostDesc **)&pCppiDesc)) != 0);

            pHostDesc = (Cppi_HostDesc *)pCppiDesc;

            // free the PBM packet
            PBM_free((PBM_Handle)pHostDesc->softwareInfo0);
        }

    Ralf

  • in reply to Ralf Goebel
    Intellectual 300 points

    Hi Ralf

    After we increase NIMU_NUM_TX_DESC to 32, an error occured in qmss:


    [C66XX_0] Error: Inserting memory region 0, Error code : -129
    [C66XX_0] Failed to initialize the QMSS subsystem

    Can you please tell us how you increased NIMU_NUM_TX_DESC? Thanks a lot.

  • in reply to yu fei
    Genius 3715 points

    Hi,

    I only redefined the ratio between RX and TX buffers inside nimu_eth.c, the total amount of buffers remains the same:

    ...
    #include "resource_mgr.h"
    #undef NIMU_NUM_TX_DESC
    #undef NIMU_NUM_RX_DESC
    #define NIMU_NUM_TX_DESC 48u
    #define NIMU_NUM_RX_DESC 78u
    ...

    Ralf