This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMS320F28388S: LwIP memory allocation issues

Part Number: TMS320F28388S
Other Parts Discussed in Thread: C2000WARE

Tool/software:

I initially posted this information here, but never got a reply, so I'm posting it again as a new thread.

I am experiencing the problems with double freeing of LwIP buffers. I've been trying to power through it, but I think there is some fundamental error with the LwIP examples even with the 5.4.0.0 C2000ware. I am using the TMS320F28388S processor.

Turning on the LWIP_STATS and MEM_STATS defines show the illegal count being incremented for double free operations. Is there a fix available that properly handles the buffer allocations?

I have traced part of the problem to the loop in f2838xif_process_transmit from f2838xif.c that is freeing a chain of packet descriptors.

    do
    {
        pktDescPtrShadow = pktDescPtr->nextPacketDesc;
        mem_free(pktDescPtr);
        pktDescPtr = pktDescPtrShadow;
    }
    while(pktDescPtr != 0);

This code can be traced back to being called from Ethernet_removePacketsFromTxQueue [driverlib_cm/ethernet.c] which is popping packets one at a time off of the descQueue using Ethernet_performPopOnPacketQueue and calling the pfcbFreePacket function on them.

static Ethernet_Pkt_Desc *Ethernet_performPopOnPacketQueue(
            Ethernet_PKT_Queue_T *pktQueuePtr)
{
    Ethernet_Pkt_Desc *pktDescHdrPtr;

    pktDescHdrPtr = pktQueuePtr->head;

    if(0U != pktDescHdrPtr)
    {
        pktQueuePtr->head = pktDescHdrPtr->nextPacketDesc;
        pktQueuePtr->count--;
    }

    return(pktDescHdrPtr);
}

pktQueuePtr->head keeps a pointer to pktDescHdrPtr->nextPacketDesc which is getting freed in f2838xif_process_transmit. The problem can get particularly tough if there are more packets getting processed that reuse these buffers and cause circular references in the descriptor nextPacketDesc links.

One of my attempts to work through this was to only free one item at a time in f2838xif_process_transmit, but that led to memory leaks and ultimately running out of available memory.

My latest attempt to fix the issue is to add a loop to Ethernet_removePacketsFromTxQueue that calls Ethernet_performPopOnPacketQueue until a NULL is returned - making the logic match what the pfcbFreePacket call would free. This has worked better, but still appears to have issues when there are several packets being sent at a time.

Please let me know if there is a fix available or any further information that you may need.

  • Hey C2000 Team,

    I pinged you over email but also wanted to give a friendly nudge on this thread as well. Can you please help to provide an update to Scott?

    -Matt

  • Hey Scott,

    Thank you so much for your patience on this! I am going to follow up with you directly over email with additional info but in parallel I also wanted to share an update on this E2E post for reference.

    I received feedback from the C2000 SW Apps team that their LwIP and Ethernet experts are heavily loaded at the moment but have added your request for support to their backlog of requests. Early next week they will begin testing to verify some workarounds for your scenario so we can expect initial feedback from their team around mid-next week. 

    Additionally, they expect to be able to provide a more in-depth solution in February since more resources will be available to work on this request later this month.

    I will let them comment on this thread as well with any additional comments or clarifications needed.

    Best regards,

    Matt

  • Hi Scott,
      I have tried to have detailed look at the implementation and tried to reproduce the issue. Over multiple runs, I am not seeing any issues with this using UDP traffic.
    1. Can you pls tell me steps to reproduce this on my end?
    2. Can you also share the value of variables mem_alloc_count,, mem_free_count, process_transmit_count when the issue is seen.

    One issue I found is related to packet buffer getting freed without awaiting for packet Tx completion. This may result in packet data corruption, but may not show the same problem as that of multiple free from mem pool.

    With regards,

    Pradeep

  • Pradeep,

    The packet buffer freed without waiting for Tx completion on its own is a large issue. I believe that is the root cause of occasionally seeing infinite loops when trying to iterate through the linked list of buffers. Focusing on fixing that aspect of the bug will likely help make the system much more stable.

    If you turn on LWIP_STATS and MEM_STATS defines, you should be able to run the enet_lwip example and see the issue.

    • Connect to the webpage that the enet_lwip example provides.
    • Add a watch expression for lwip_stats in the debugger. 
    • When clicking the webpage button that toggles the LED I was seeing the lwip_stats.mem.illegal count increment
    • To see that double free operations were triggering the illegal counter I changed the LWIP_ASSERT macro to trigger a breakpoint

    I am not currently set up to rerun this test, but I remember the alloc/free/transmit counts were low (<20) when the issue was seen.

    Thanks,

    Scott

  • Hey ,

    Just wanted to give a friendly nudge on this thread since it's been about a week since Scott's last message.

    Can you please help to provide feedback to Scott? If you're still looking into these issues or working on reproducing things on your side any update or timeline when we can expect a reply would be much appreciated!

    Thanks!

    -Matt

  • Hi Scott,

    you may look at this thread:
    https://e2e.ti.com/support/microcontrollers/c2000-microcontrollers-group/c2000/f/c2000-microcontrollers-forum/1074652/tms320f28388d-lwip-webserver-example-memory-errors-due-to-ethernet-descriptors-queue-management/4027536
    This was my collegue post. We encounter similar problems with TI provided driver. It looks similar.
    Now we are using modified driver and ethernet with F28388D runs without any issues in our product.

    The attached source file in Jacek post can be somewhat outdated, I could attach here our recent file if you are interested.

    And unfortunately it is disappointing that TI support for the ethernet driver is virtually non-existent. The bugs are there for years.

    Best regards,
    Andy

  • Hi Scott,
     Have identified few issues as below-
    1. Tx Packet buffer was released even before waiting for transmit completion

    2.More than one mem_free is called for ethernet_pkt_desc in case of pbuff_chain. This could be a main reason for illegal free.

    Both issues are fixed and the patch is available in the attachment. Please let me know if it improved.

    diff --git a/driverlib/f2838x/driverlib_cm/ethernet.h b/driverlib/f2838x/driverlib_cm/ethernet.h
    index 727da4ab..44f46707 100644
    --- a/driverlib/f2838x/driverlib_cm/ethernet.h
    +++ b/driverlib/f2838x/driverlib_cm/ethernet.h
    @@ -940,8 +940,7 @@ typedef struct Ethernet_Pkt_Desc_T
         uint16_t innerVlanTag;
         uint32_t mssTso;
         uint16_t vlanTag;
    -
    -
    +    void* pAppData;
     }Ethernet_Pkt_Desc;
     
     //*****************************************************************************
    diff --git a/libraries/communications/Ethernet/third_party/lwip/lwip-2.1.2/ports/C2000/netif/f2838xif.c b/libraries/communications/Ethernet/third_party/lwip/lwip-2.1.2/ports/C2000/netif/f2838xif.c
    index df215887..0026bfdd 100644
    --- a/libraries/communications/Ethernet/third_party/lwip/lwip-2.1.2/ports/C2000/netif/f2838xif.c
    +++ b/libraries/communications/Ethernet/third_party/lwip/lwip-2.1.2/ports/C2000/netif/f2838xif.c
    @@ -339,7 +339,8 @@ f2838xif_transmit(struct netif *netif, struct pbuf *p)
             pktDescPtr->pktLength = q->tot_len;
             pktDescPtr->bufferLength = q->len;
             pktDescPtr->validLength = q->len;
    -
    +        pbuf_ref(q);
    +        pktDescPtr->pAppData = (void*)q;
     
               if(i==0)
               {
    @@ -418,28 +419,19 @@ int descriptorFreeErr=0;
     static void
     f2838xif_process_transmit(struct netif *netif, Ethernet_Pkt_Desc *pPacket)
     {
    -    Ethernet_Pkt_Desc *pktDescPtr, *pktDescPtrShadow;
    -
         /*
          * Free the packet descriptor memory.
          */
         if (pPacket == 0)
             return;
     
    -    pktDescPtr = pPacket;
    -
         /* ENTER CRITICAL SECTION */
         Interrupt_disable(INT_EMAC_TX0);
         //Interrupt_disable(INT_EMAC_RX0);
     
    -    do
    -    {
    -        pktDescPtrShadow = pktDescPtr->nextPacketDesc;
    -        mem_free(pktDescPtr);
    -        pktDescPtr = pktDescPtrShadow;
    -    }
    -    while(pktDescPtr != 0);
    -
    +    pbuf_free((struct pbuf*)pPacket->pAppData);
    +    pPacket->pAppData = NULL;
    +    mem_free(pPacket);
         /* EXIT CRITICAL SECTION */
         Interrupt_enable(INT_EMAC_TX0);
         //Interrupt_enable(INT_EMAC_RX0);
    


    With regards,

    Pradeep