• State
  • Locked Locked
  • Replies 6 replies
  • Subscribers 83 subscribers
  • Views 1453 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMS320F28388D: lwIP stack crash

Markus A
Intellectual 435 points

Part Number: TMS320F28388D
Other Parts Discussed in Thread: C2000WARE

Dear support team,

I am currently implementing a TCP server which shall provide connection to 2 different TCP ports.
The lwIP stack works fine with 2 connections as long as I am transferring data only at 1 of the 2 connected clients.
When I start to transfer data on both connections at the same time, the lwIP stack crashes. (sometimes after a few seconds, sometimes after 10+ seconds)
It also does not react to ICMP echo requests anymore.

Debugging (after the lwIP stack does not respond anymore) shows that the 'mem_free' function, being called from 'f2838xif_process_transmit' does always result in a 'memory double free' error condition.
The packet descriptor pointer 'pktDescPtr' does not become zero and therefore the function stays in the while loop forever.

Problematic 'double free' procedure in mem.c: => see image: https://ibb.co/Y7hjR52
Infinite while loop in 'f2838xif_process_transmit' function including stack trace: => see image: https://ibb.co/CQSxTNW

I also enabled the lwIP stats showing multiple 'illegal' entries in the mem section.
=> see image: https://ibb.co/71zLJwg


So I decided to check these 'illegal' memory entries in your default 'enet_lwip' demo application.
I enabled the lwIP stats in your demo application and downloaded it.

Result:
These 'illegal' memory events also occur in the TI demo application.

After a device reset, the counter is zero.
When sending 2-4 ICMP echo packets to the TI, the counter raises to 8 and stays at this value.
Sending additional ICMP echo packets do not raise the value anymore.

But when browsing the HTTP page included in the demo, the counter increases by 1 for every HTTP connection (web page reload or pushing the button).
Connecting with a debugger shows that this 'illegal' entries are caused by the (same) 'double free memory' procedure.
The only difference is that the demo application seems to recover from this error and leave the while loop in the function 'f2838xif_process_transmit' which I've mentioned above.



My application currently doesn't use the TI RTOS but uses the functions, provided in the TI 'enet_demo' application.
Is it possible that this might be an concurrency issue? (but it also occurs in the TI demo application)

Thank you,
Markus

  • 0
    TI__Expert 4145 points

    Hi,

    What version of C2000Ware are you using?

    Can you try with the updates in the attached project.

    1121.enet_lwip.zip

  • 0 in reply to Yashwant Temburu1
    Intellectual 435 points

    Hello,

    thank you for the quick response and the quick-fix.
    I use C2000Ware 3.04.00.00.

    I had to do some additional adjustements in the CCS project to make your quick-fix work:
    * Remove the link to the ethernet.c file
    * Copy the ethernet.c file from C2000Ware to my CCS project
    * Change the static function 'Ethernet_initRxChannel' to a global function

    Can you please verify that this change is the only one you did in your ethernet.c file?


    I tried your quick-fix.
    It seems to last a little bit longer than before but unfortunately the very same issue still occurs.
    => see image: https://ibb.co/fX5jp5N

  • 0 in reply to Markus A
    TI__Expert 4145 points

    If you are using multiple interrupts, make sure to use priorities.

    Give highest priority to tx_isr so that the mem_free for buffers is called on time.

    Then to the rx isr and then to generic isr etc.

  • 0 in reply to Yashwant Temburu1
    Intellectual 435 points

    My application does not use any additional interrupts but only those being used by the lwIP implementation.
    I tried to set the interrupt priorities as you have suggested (EMAC TX0, EMAC RX0, INT_EMAC) but it doesn't make a difference.
    The issue still occurs.

  • 0 in reply to Markus A
    TI__Expert 4145 points

    The double free memory issue could be because of pbuf_free being called immediately after 1 packet transmission.

    In this e2e a user points out similar issue https://e2e.ti.com/support/microcontrollers/c2000-microcontrollers-group/c2000/f/c2000-microcontrollers-forum/896100/tms320f28388d-is-this-a-bug-in-c2000ware-lwip-example-code .

    Can you check if the solution provided by the user of using pbuf_ref() to increment reference counter of the pbuf so there is no double free memory is useful.

  • 0 in reply to Yashwant Temburu1
    Intellectual 435 points

    I implemented the suggestions, mentioned by the other thread, but unfortunately it doesn't make any difference.
    The type ('double free mem') and the number of illegal mem accesses are exactly the same than before.

    I did the following steps:

    1. Reverted to a clean C2000Ware state (v3.04.00.00)

    2.Added the interrupt priorites in function "Ethernet_init" as you have suggested in a previous post:

        Interrupt_setPriority(INT_EMAC_TX0, 0);
        Interrupt_setPriority(INT_EMAC_RX0, 2);
        Interrupt_setPriority(INT_EMAC, 4);

    3. Implemented the pbuf_ref / pbuf_free procedure as suggested in the other thread

    3.1 Added a pbuf pointer at the end of the "Ethernet_Pkt_Desc" data type to be able to store the pbuf pointer

    3.2 Store the pbuf poiner and apply the 'pbuf_ref' in function "f2838xif_transmit"

            pktDescPtr->dataOffset = 0;
            pktDescPtr->dataBuffer = q->payload;
            pktDescPtr->pktChannel = ETHERNET_DMA_CHANNEL_NUM_0;
            pktDescPtr->pktLength = q->tot_len;
            pktDescPtr->bufferLength = q->len;
            pktDescPtr->validLength = q->len;
            pktDescPtr->pb = q;
            pbuf_ref(pktDescPtr->pb);

    3.3  Free the pbuf in function "f2838xif_process_transmit"

        do
        {
            pktDescPtrShadow = pktDescPtr->nextPacketDesc;
            if ((pktDescPtr != NULL) && (pktDescPtr->pb != NULL))
            {
                if (pktDescPtr->pb->ref > 0)
                {
                    pbufRefNonZero++;
                    pbuf_free(pktDescPtr->pb);
                    pktDescPtr->pb = NULL;
                }
                else
                {
                    pbufRefZero++;
                }
            }
            mem_free(pktDescPtr);
            pktDescPtr = pktDescPtrShadow;
        }
        while(pktDescPtr != 0);

    Note: pbufRefZero and pbufRefNonZero are simple uint32 variables so I can see, how often each of the events occur.
    pbufRefZero always stays at zero.