This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

RTOS/TM4C129ENCZAD: When ethernet cable is pulled, closing sockets does not clean up all dynamic memory causing memory leak

Part Number: TM4C129ENCZAD

Tool/software: TI-RTOS

I am investigating a memory leak issue with the TIVA microcontroller.

The microcontroller is a TCP server.  A client establishes a TCP connection to the microcontroller.  While the TCP socket is still engaged, yank out the ethernet cable from the launchpad.  At this point, the microcontroller detects the loss of ethernet and attempts to clean up by closing the sockets.  However, not all memory is reclaimed and 4112 bytes are lost.  The TCP send buffer is 2048 bytes and TCP receive buffer is 2048 bytes as well.  So the amount of memory lost is 16 bytes more than the TCP send and receive buffer.  Another issue is that when the ethernet cable is plugged in again, I can no longer establish a TCP server on the same port number, I have to use another port number.

I have verified that if the client disconnects from the socket prior to pulling the ethernet cable, there is no memory loss.  I have also made sure that I am calling fdclose when the loss of ethernet link is detected as well.  

I have recreated the same bug in a small app that runs on the TI Launchpad.  Please help.  NDKTest.zip

  • Hi Yan,

    Thanks for supplying an example! I'll take a look at it and get back to you.

    Todd
  • Hi Yan,

    I ran your example. I'm not seeing a memory leak. I'm checking ROV->HeapMem->Detailed before and after unplugging and then again after plugging in. How are you determining a leak? I've attached a file that contains the CCS Console output along with the HeapMem free size added in.

    EPHY_BMCR = 0x3100
    BMSR = 0x786d
    EPHY_ANA = 0x1e1
    EPHY_ANLPA = 0xcde1
    EPHY_STS = 0x915
    Host Comm Task Started. Starting Message Exchange
    Ethernet Status report 0x5, 0x4, 0x0
    Service Status: DHCPC    : Enabled  :          : 000
    Ethernet DHCP Client Started
    Service Status: DHCPC    : Enabled  : Running  : 000
    Network Added: If-1:146.252.162.135
    Ethernet DHCP IP address added
    Service Status: DHCPC    : Enabled  : Running  : 017
    Ethernet IP address added: 146.252.162.135
    Ethernet UDP Socket Created. Port Num = 10065.
    Ethernet TCP Socket Created. Port Num = 10070.
    Ethernet Sockets Created. Port Num = 10070.
    -------->ROV->HeapMem->Detailed->totalFreeSize = 0xee60<--------
    Ethernet TCP ports closed
    Ethernet cable unplugged
    Network Removed: If-1:146.252.162.135
    Ethernet Status report 0x5, 0x0, 0x0
    Service Status: DHCPC    : Disabled :          : 000
    DisableNDKDHCP success
    -------->ROV->HeapMem->Detailed->totalFreeSize = 0xfdf0<--------
    Ethernet Status report 0x5, 0x4, 0x0
    Service Status: DHCPC    : Enabled  :          : 000
    Starting DHCP
    Ethernet DHCP Client Started
    Service Status: DHCPC    : Enabled  : Running  : 000
    Network Added: If-1:146.252.162.135
    Ethernet DHCP IP address added
    Service Status: DHCPC    : Enabled  : Running  : 017
    Ethernet IP address added: 146.252.162.135
    Ethernet UDP Socket Created. Port Num = 10065.
    Ethernet TCP Socket Created. Port Num = 10070.
    Ethernet Sockets Created. Port Num = 10070.
    -------->ROV->HeapMem->Detailed->totalFreeSize = 0xee60<--------
    

    Todd

  • I am using ROV to see the memory leak.  From your log, I see that you are not connecting with a client.  When the cable is pulled with a client still connected, memory is lost.  I use Hercules to connect to the launchpad.  You can use any TCP client and the behavior will be the same.  I have attached a log from my run.

    EPHY_BMCR = 0x0
    BMSR = 0x7849
    EPHY_ANA = 0x1e1
    EPHY_ANLPA = 0x0
    EPHY_STS = 0x2
    Ethernet Status report 0x5, 0x4, 0x0
    Service Status: DHCPC    : Enabled  :          : 000
    Ethernet DHCP Client Started
    Service Status: DHCPC    : Enabled  : Running  : 000
    Host Comm Task Started. Starting Message Exchange
    Network Added: If-1:10.40.1.7
    Ethernet DHCP IP address added
    Service Status: DHCPC    : Enabled  : Running  : 017
    Ethernet IP address added: 10.40.1.7
    Ethernet cable detected
    Ethernet TCP Socket Created. Port Num = 10070.
    Ethernet Sockets Created. Port Num = 10070.
    -------->ROV->HeapMem->Detailed->totalFreeSize = 0xee60<--------
    Ethernet TCP - Client connected. Addr = 10.40.1.200
    Socket Connected
    -------->ROV->HeapMem->Detailed->totalFreeSize = 0xde50<--------
    -------->Disconnected ethernet cable<--------
    Ethernet TCP ports closed
    Ethernet cable unplugged
    Network Removed: If-1:10.40.1.7
    Ethernet Status report 0x5, 0x0, 0x0
    Service Status: DHCPC    : Disabled :          : 000
    DisableNDKDHCP success
    -------->ROV->HeapMem->Detailed->totalFreeSize = 0xf5e8<--------
    -------->Connect Ethernet cable back in<--------
    Ethernet cable detected
    Ethernet DHCP Client Started
    00035.700 TcpTimeoutRexmt: Retransmit Timeout
    00036.300 TcpTimeoutRexmt: Retransmit Timeout
    00037.500 TcpTimeoutRexmt: Retransmit Timeout
    00039.900 TcpTimeoutRexmt: Retransmit Timeout
    00044.700 TcpTimeoutRexmt: Retransmit Timeout
    00054.300 TcpTimeoutRexmt: Retransmit Timeout
    00073.500 TcpTimeoutRexmt: Retransmit Timeout
    Service Status: DHCPC    : Enabled  : Running  : 000
    Ethernet DHCP Client Started
    Service Status: DHCPC    : Enabled  : Running  : 000
    Network Added: If-1:10.40.1.7
    Ethernet DHCP IP address added
    Service Status: DHCPC    : Enabled  : Running  : 017
    Starting DHCP
    Ethernet IP address added: 10.40.1.7
    Error - cannot open Ethernet TCP Socket. Cannot bind to socket. Port Num = 10070. err = 0x30
    Ethernet TCP Socket Created. Port Num = 10071.
    Ethernet Sockets Created. Port Num = 10071.
    Etherent TCP Client Disconnected. Addr = 0x0
    -------->ROV->HeapMem->Detailed->totalFreeSize = 0xde50  (should be 0xee60)<--------
    -------->Client connects to scoket<--------
    Ethernet TCP - Client connected. Addr = 10.40.1.200
    Socket Connected
    -------->ROV->HeapMem->Detailed->totalFreeSize = 0xce40  (should be 0xde50)<--------
    

    Yan

  • I'll try it, but can you give me exact instructions on how to connect a client. I don't know what Hercules is.
  •  Attached is an image of the app.  It is just a TCP test tool:

    1. Open the program

    2. Select the TCP Client tab.  I have circled it it red in the attached photo

    3. In the Module IP box, enter the IP address that the launchpad is assigned.  From your log, it should be 146.252.162.135

    4. In the Port Box, enter 10700.

    5. Now hit the connect button right below the Port box.

    6. You should see: Ethernet TCP - Client connected. Addr =... in the printf output

  • Can you extract the executable from the attached file?HerculesSetup.zip

  • Yan,

    I'm getting different results. Sometimes I'm seeing a memory leak, but I cannot get consistent runs. I've enabled HeapTrack to see when and who allocated the leaked blocks. To enable, just add the following into the .cfg:

    BIOS.heapTrackEnabled = true;

    Then in Tools->ROV->HeapTrack->HeapAllocList, you can see the currently allocated blocks, the thread that it and the Clock tick when the allocation occurred. I also changed the TCP/UDP Rx/Tx sizes to be unique values (e.g. 2048+64 for one, 2048-64 for another, etc.) to help determine which allocation is which. Can try this also to see if it helps you debug it. You can use ROV->Tasks->Basic to figure out which task is which.

  • Todd,

    BIOS.heapTrackEnabled = true; is a nice trick.  I will use that next time I run into memory leaks.

    From your image, I see that you have changed the buffer sizes to 1920 and 1984.  This matches the allocation of 0x7c8 and 0x788.  The memory leak will only occur if there is a socket connected.  I always see a memory leak on the first unplug.  And usually see it on the second unplug as well.  When you unplug and replug with a socket connected, you should see

    00069.600 TcpTimeoutRexmt: Retransmit Timeout

    "Ethernet TCP Socket Created. Port Num = 10071."

    At this time, the old port number 10070 can no longer be used and in the Hercules app, you need to connect to the new port number which is 10071.   

    I followed your advice and changed the TCP send buffer size to 1984 in the configuration file and left the receive at 2048.  Now I am seeing memory leaks of size of (1984 + 8) and (2048 + 8).  The attached image is captured after unplugging the cable 3 times and 8096 bytes have already been leaked.  From the image, I can see that both send and receive buffers are not getting freed.  The memory should be freed by the NDK when I call fdClose, so I believe this is an issue with the NDK itself. 

    Can you please resolve this issue.

    Thank you.

    -Yan

  • Can you give the console output you are seeing. I'm never able to connect the second time (with 10071 or 10070). Here what I see after I plug the cable back in:

    Ethernet UDP Socket Created. Port Num = 10065.

    Error - cannot open Ethernet TCP Socket. Cannot bind to socket. Port Num = 10070. err = 0x30

    Ethernet TCP Socket Created. Port Num = 10071.

    Ethernet Sockets Created. Port Num = 10071.

    Etherent TCP Client Disconnected. Addr = 0x0

    Note: I don't manually disconnect the client before or after I plug the cable back in.

    We have a bug that has been fixed in the NDK. Can you try this fix and rebuild the NDK (and relink the application) to see if this helps. 

    /*
     * Copyright (c) 2012-2015, Texas Instruments Incorporated
     * All rights reserved.
     *
     * Redistribution and use in source and binary forms, with or without
     * modification, are permitted provided that the following conditions
     * are met:
     *
     * *  Redistributions of source code must retain the above copyright
     *    notice, this list of conditions and the following disclaimer.
     *
     * *  Redistributions in binary form must reproduce the above copyright
     *    notice, this list of conditions and the following disclaimer in the
     *    documentation and/or other materials provided with the distribution.
     *
     * *  Neither the name of Texas Instruments Incorporated nor the names of
     *    its contributors may be used to endorse or promote products derived
     *    from this software without specific prior written permission.
     *
     * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
     * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
     * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
     * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
     * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
     * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
     * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
     * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
     * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
     * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
     * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
     * */
    /*
     * ======== sockpcb.c ========
     *
     * Object member functions for the Sock device object. These
     * functions include protocol control type access to the SOCKET
     * object.
     *
     */
    
    #include <stkmain.h>
    #include "sock.h"
    
    static SOCK *pSockList[] = { 0, 0, 0, 0, 0 };
    
    /* Next User Port for Ephemeral Ports */
    static uint wUserPort = SOCK_USERPORT_FIRST;
    
    static SOCK *SockPcbFind( SOCK *psFirst, IPN LIP, uint LPort,
                              IPN FIP, uint FPort, int *pwc, int FindAll );
    
    static SOCK *SockPcbFindRaw( SOCK *psFirst, uint Prot, IPN LIP,
                                 IPN FIP, int *pwc, int FindAll );
    
    /*-------------------------------------------------------------------- */
    /* SockPcbAttach() */
    /* Adds a socket to the supplied socket list */
    /*-------------------------------------------------------------------- */
    int SockPcbAttach( HANDLE hSock )
    {
        SOCK *ps = (SOCK *)hSock;
    
    #ifdef _STRONG_CHECKING
        if( ps->fd.Type != HTYPE_SOCK )
        {
            DbgPrintf(DBG_ERROR,"SockPcbAttach: HTYPE %04x",ps->fd.Type);
            return( 0 );
        }
    
        if( ps->SockProt > 4 )
        {
            DbgPrintf(DBG_ERROR,"SockPcbAttach: Invalid SockProt");
            return( 0 );
        }
    #endif
    
        /* We'll insert at head of list since its quicker */
        ps->pProtPrev = 0;
        ps->pProtNext = pSockList[ps->SockProt];
        pSockList[ps->SockProt] = (HANDLE)ps;
    
        /* Patch entry which follows us (if any) */
        if( ps->pProtNext )
            ps->pProtNext->pProtPrev = ps;
    
        return(0);
    }
    
    /*-------------------------------------------------------------------- */
    /* SockPcbDetach() */
    /* Removes a socket to the supplied socket list */
    /*-------------------------------------------------------------------- */
    int SockPcbDetach( HANDLE hSock )
    {
        SOCK *ps   = (SOCK *)hSock;
    
    #ifdef _STRONG_CHECKING
        if( ps->fd.Type != HTYPE_SOCK )
        {
            DbgPrintf(DBG_ERROR,"SockPcbDetach: HTYPE %04x",ps->fd.Type);
            return( 0 );
        }
    
        if( ps->SockProt > 4 )
        {
            DbgPrintf(DBG_ERROR,"SockPcbDetach: Invalid SockProt");
            return( 0 );
        }
    #endif
    
        /* Patch preceeding entry */
        if( !ps->pProtPrev )
            pSockList[ps->SockProt] = (HANDLE)ps->pProtNext;
        else
            ps->pProtPrev->pProtNext = ps->pProtNext;
    
        /* Patch following entry */
        if( ps->pProtNext )
            ps->pProtNext->pProtPrev = ps->pProtPrev;
    
        /* Just to be tidy... */
        ps->pProtPrev = 0;
        ps->pProtNext = 0;
    
        return(0);
    }
    
    /*-------------------------------------------------------------------- */
    /* SockPcbFind() */
    /* Find the "best" match for the supplied parameters. */
    /*-------------------------------------------------------------------- */
    static SOCK *SockPcbFind( SOCK *ps, IPN LIP, uint LPort, IPN FIP, uint FPort,
                              int *pwc, int FindAll )
    {
        SOCK *psBest = 0;
        int  wcBest = 9;
        int  wc;
    
        /* Find the "best" match for the supplied parameters. */
    
        for( ; ps; ps = ps->pProtNext )
        {
            /* Clear wild cards */
            wc = 0;
    
            /* Local port NULL means socket is not bound */
            if( !ps->LPort )
                continue;
    
            /* If local ports don't match, this entry can't match */
            if( ps->LPort != LPort )
                continue;
    
            /* Check if the local IP is a multicast packet. In that case we need to search
             * the multicast socket chain for a match */
            if (IN_MULTICAST(LIP))
            {
                MCAST_SOCK_REC* ptr_mcast_rec;
    
                /* Cycle through all the multicast records on the socket. */
                ptr_mcast_rec = (MCAST_SOCK_REC *)list_get_head ((LIST_NODE**)&ps->pMcastList);
                while (ptr_mcast_rec != NULL)
                {
                    /* Check if we get a hit? */
                    if (ptr_mcast_rec->mreq.imr_multiaddr.s_addr == LIP)
                        return ps;
    
                    /* Get the next multicast record. */
                    ptr_mcast_rec = (MCAST_SOCK_REC *)list_get_next((LIST_NODE*)ptr_mcast_rec);
                }
    
                /* Control comes here on no match. */
                continue;
           }
    
            /* Local IP must match, or be wildcard, or be a Broadcast */
            if( ps->LIP != LIP )
            {
                if (!LIP || !ps->LIP || LIP == INADDR_BROADCAST)
                    wc++;
                else
                    continue;
            }
    
            /* Foreign Port+IP must both match or both be wildcard */
            if( ps->FPort != FPort || ps->FIP != FIP )
            {
                if( (!FIP && !FPort) || (!ps->FIP && !ps->FPort) )
                    wc++;
                else
                    continue;
            }
    
            if( wc < wcBest )
            {
                wcBest = wc;
                psBest = ps;
                if( !wcBest )
                    break;
            }
    
            if( FindAll )
                break;
        }
    
        if( pwc )
            *pwc = wcBest;
        return(psBest);
    }
    
    /*-------------------------------------------------------------------- */
    /* SockPcbFindRaw() */
    /* Find the "best" match for the supplied parameters. */
    /*-------------------------------------------------------------------- */
    static SOCK *SockPcbFindRaw( SOCK *ps, uint Prot, IPN LIP, IPN FIP,
                                 int *pwc, int FindAll )
    {
        SOCK *psBest = 0;
        int  wcBest = 9;
        int  wc;
    
        /* Find the "best" match for the supplied parameters. */
    
        for( ; ps; ps = ps->pProtNext )
        {
            /* Clear wild cards */
            wc = 0;
    
            /* Protocol must match, or be wildcard */
            if( ps->Protocol && ps->Protocol != Prot )
                continue;
    
            /* Local IP must match, or be wildcard */
            if( ps->LIP != LIP )
            {
                if( !LIP || !ps->LIP )
                    wc++;
                else
                    continue;
            }
    
            /* Foreign IP must match, or be wildcard */
            if( ps->FIP != FIP )
            {
                if( !FIP || !ps->FIP )
                    wc++;
                else
                    continue;
            }
    
            if( wc < wcBest )
            {
                wcBest = wc;
                psBest = ps;
                if( !wcBest )
                    break;
            }
    
            if( FindAll )
                break;
        }
    
        if( pwc )
            *pwc = wcBest;
        return(psBest);
    }
    
    /*-------------------------------------------------------------------- */
    /* SockPcbBind() */
    /* Binds a socket to specified local LIP and LPort */
    /* Note: LIP can be NULL (wildcard), but in this implementaion, LPort */
    /*       is never a wildcard. When not specified, LPort is assigned to */
    /*       an ephemeral port. */
    /*-------------------------------------------------------------------- */
    int SockPcbBind( HANDLE hSock, IPN LIP, uint LPort )
    {
        SOCK   *ps    = (SOCK *)hSock;
        SOCK   *psTmp;
        int    fEphem = 0;
        uint   SockOpts;
    
    #ifdef _STRONG_CHECKING
        if( ps->fd.Type != HTYPE_SOCK )
        {
            DbgPrintf(DBG_ERROR,"SockPcbBind: HTYPE %04x",ps->fd.Type);
            return( 0 );
        }
    
        if( ps->SockProt > 4 )
        {
            DbgPrintf(DBG_ERROR,"SockPcbBind: Invalid SockProt");
            return( 0 );
        }
    #endif
    
        /* Anything goes with Raw sockets */
        if( ps->SockProt == SOCKPROT_RAW )
        {
            SockOpts = 0;
            goto SockPcbBindOk;
        }
    
        /* On multicast addrs, REUSEADDR is the same as REUSEPORT */
        SockOpts = ps->OptionFlags & (SO_REUSEPORT|SO_REUSEADDR);
        if( SockOpts == SO_REUSEADDR )
        {
            if( IN_MULTICAST( LIP ) )
                SockOpts = (SO_REUSEADDR|SO_REUSEPORT);
        }
    
        /* If LPort is NULL, and we are already bound to an LPort, */
        /* then we'll use the previously bound value. */
        if( !LPort && ps->LPort )
            LPort = ps->LPort;
    
        /* OPTION: We *may* want to restrict IP addresses to those actually */
        /* available on our system. */
        /* OPTION: We *may* want to restrict reserved ports to some "system" */
        /* class user. */
    
        /* Check for ephemeral port */
    SockPcbBindEphem:
        if( !LPort )
        {
            if( ++wUserPort > SOCK_USERPORT_LAST )
                  wUserPort = SOCK_USERPORT_FIRST;
            LPort = HNC16(wUserPort);
            fEphem = 1;
        }
    
        /* Find the best match */
        psTmp = SockPcbFind( pSockList[ps->SockProt], LIP, LPort, 0, 0, 0, 0 );
    
        /* Check for conflict (and that we didn't match ourselves) */
        if( psTmp && psTmp != ps && LPort == psTmp->LPort )
        {
            /* If the IP's are different, we're OK */
            if( LIP && psTmp->LIP && LIP != psTmp->LIP )
                goto SockPcbBindOk;
    
            /* We don't allow conflicts on ephemeral ports */
            if( fEphem )
            {
                LPort = 0;
                goto SockPcbBindEphem;
            }
    
            /* We have a port conflict. See if we keep going anyway */
    
            /* Address Conflict */
    
            /* Exception 1: Check for "REUSEPORT" on both sockets */
            if( psTmp->OptionFlags & SockOpts & SO_REUSEPORT )
                goto SockPcbBindOk;
    
            /* Exception 2: One or the other Local IP is "wildcard", but not both, */
            /*              and both have REUSEADDR set. */
            if( ((!LIP && psTmp->LIP) || (LIP && !psTmp->LIP)) &&
                          (psTmp->OptionFlags & SockOpts & SO_REUSEADDR) )
                goto SockPcbBindOk;
    
            return( EADDRINUSE );
        }
    
    SockPcbBindOk:
        /* If we got here, the addr is OK */
        ps->LIP   = LIP;
        ps->LPort = LPort;
        ps->OptionFlags |= SockOpts;
    
        return(0);
    }
    
    /*-------------------------------------------------------------------- */
    /* SockPcbConnect() */
    /* Set the foreign address to that specified in FIP and FPort */
    /*-------------------------------------------------------------------- */
    int SockPcbConnect( HANDLE hSock, IPN FIP, uint FPort )
    {
        SOCK   *ps    = (SOCK *)hSock;
        SOCK   *psTmp;
        int    wc;
    
    #ifdef _STRONG_CHECKING
        if( ps->fd.Type != HTYPE_SOCK )
        {
            DbgPrintf(DBG_ERROR,"SockPcbConnect: HTYPE %04x",ps->fd.Type);
            return( 0 );
        }
    
        if( ps->SockProt > 4 )
        {
            DbgPrintf(DBG_ERROR,"SockPcbConnect: Invalid SockProt");
            return( 0 );
        }
    #endif
    
        /* Anything goes with Raw sockets */
        if( ps->SockProt == SOCKPROT_RAW )
            goto SockPcbConnectOk;
    
        /* Verify that we have a local binding */
        /* Note that LIP can be NULL if one really odd case... */
        if( !ps->LIP || !ps->LPort )
        {
            /* We can bind to [0.0.0.0] if we've specified an IFTx device */
            if( !ps->LPort || !ps->hIFTx )
                return( EINVAL );
        }
    
        /* Verify we have a desired IP, and port */
        if( !FIP || !FPort )
            return( EINVAL );
    
        /* Find the best match */
        psTmp = SockPcbFind( pSockList[ps->SockProt],
                             ps->LIP, ps->LPort, FIP, FPort, &wc, 0 );
    
        /* Check for conflict */
        if( psTmp && psTmp != ps && wc == 0 )
            return( EADDRINUSE );
    
    SockPcbConnectOk:
        /* If we got here, the addr is OK */
        ps->FIP   = FIP;
        ps->FPort = FPort;
    
        return(0);
    }
    
    /*-------------------------------------------------------------------- */
    /* SockPcbResolve() */
    /* Find the "best" match given the supplied parameters, and */
    /* potentially spawn a new socket from an "accepting" (listening) */
    /* socket. */
    /*-------------------------------------------------------------------- */
    HANDLE SockPcbResolve( uint SockProt, IPN LIP, uint LPort,
                           IPN FIP, uint FPort, uint Match, uint * MaxFlag )
    {
        SOCK *ps;
        SOCK *psSpawn;
        int  wc;
    
        /* Assume max connections are not exceeded */
        *MaxFlag = 0;
    
        /* The first thing to do is the find the best match */
        ps = SockPcbFind( pSockList[SockProt],
                          LIP, LPort, FIP, FPort, &wc, 0 );
    
        /* Return NULL for no match, or for wildcard match when EXACT required */
        if( !ps || (wc && Match == SOCK_RESOLVE_EXACT) )
            return(0);
    
        /* If match type is not SPAWN, then we return what we got */
        if( Match != SOCK_RESOLVE_SPAWN )
            return( ps );
    
        /* We have a match type of SPAWN with wildcards. Here, we only return */
        /* a socket if we can spawn it. i.e.: exact match is a failure */
        /* Also, it must be accepting connections */
        if( !wc || !(ps->OptionFlags & SO_ACCEPTCONN) )
            return(0);
    
        /* If we're out of connect space, abort */
        if( ps->ConnTotal >= ps->ConnMax ) {
            *MaxFlag = 1;
            return(0);
        }
    
        /* Create a new socket - return NULL if we can't create it */
        if( SockNew(ps->Family, ps->SockType, ps->Protocol,
                    ps->RxBufSize, ps->TxBufSize, (HANDLE *)&psSpawn) )
            return(0);
    
        /* Inherit SOCKET Options */
        psSpawn->OptionFlags |= ps->OptionFlags & ~(SO_ACCEPTCONN);
        psSpawn->dwLingerTime = ps->dwLingerTime;
        psSpawn->RxTimeout    = ps->RxTimeout;
        psSpawn->TxTimeout    = ps->TxTimeout;
    
        /* Inherit IP Options */
        psSpawn->IpFlags   = ps->IpFlags;
        psSpawn->IpTtl     = ps->IpTtl;
        psSpawn->IpTos     = ps->IpTos;
        if( ps->IpOptSize )
        {
            psSpawn->IpOptSize = ps->IpOptSize;
            mmCopy( psSpawn->IpOptions, ps->IpOptions, ps->IpOptSize );
        }
    
        /* Inherit Some Protocol Options */
        SockPrInherit( ps, psSpawn );
    
        /* We're going to cheat here. We know the specified parameters do */
        /* not conflict (as there was no exact match). */
        psSpawn->LIP    = LIP;
        psSpawn->LPort  = LPort;
        psSpawn->FIP    = FIP;
        psSpawn->FPort  = FPort;
    
        /* Enqueue the spawned packet onto the parent's Pending queue */
        psSpawn->pParent     = ps;
        psSpawn->pPrevQ      = ps;
        psSpawn->pPending    = ps->pPending;
        if( psSpawn->pPending )
            psSpawn->pPending->pPrevQ = psSpawn;
        ps->pPending         = psSpawn;
        psSpawn->StateFlags |= SS_PENDINGQ;
        ps->ConnTotal++;
    
        return( psSpawn );
    }
    
    /*-------------------------------------------------------------------- */
    /* SockPcbResolveChain() */
    /* This is a chain version of PCB resolve, finding all sockets which */
    /* match the selection criteria. */
    /*-------------------------------------------------------------------- */
    HANDLE SockPcbResolveChain( HANDLE hSock, uint SockProt, uint Prot,
                                IPN LIP, uint LPort, IPN FIP, uint FPort )
    {
        SOCK *ps = (SOCK *)hSock;
    
        /* We chain off the supplied SOCK, or start at the head of the list */
        /* on a NULL */
        if( !ps )
            ps = pSockList[SockProt];
        else
            ps = ps->pProtNext;
    
        /* Find a match */
        if( SockProt != SOCKPROT_RAW )
            ps = SockPcbFind( ps, LIP, LPort, FIP, FPort, 0, 1 );
        else
            ps = SockPcbFindRaw( ps, Prot, LIP, FIP, 0, 1 );
    
        return(ps);
    }
    
    /*-------------------------------------------------------------------- */
    /* SockPcbCleanup() */
    /* Abort all closing sockets */
    /*-------------------------------------------------------------------- */
    void SockPcbCleanup()
    {
        SOCK *ps;
        int  GotOne;
    
        /* The only type of socket that can be "closing" is TCP */
        /* We'll search and we'll search until no Closing socket lives */
        do
        {
            GotOne = 0;
            ps = pSockList[SOCKPROT_TCP];
            while( ps )
            {
                if( ps->StateFlags & SS_CLOSING )
                {
                    GotOne = 1;
                    SockIntAbort( ps );
                    break;
                }
                ps = ps->pProtNext;
            }
        } while( GotOne );
    }
    
    /*-------------------------------------------------------------------- */
    /* SockCtlError() */
    /* Called to notify the socket layer that an error has occurred */
    /*-------------------------------------------------------------------- */
    void SockPcbCtlError( uint Code, IPHDR *pIpHdr )
    {
        IPN    LIP,FIP;
        uint   offset,SockProt;
        int    wc;
        UDPHDR *pUdpHdr;
        SOCK   *ps;
    
        FIP = RdNet32( &pIpHdr->IPDst );
        LIP = RdNet32( &pIpHdr->IPSrc );
    
        /* Get the offset to the transport hdr (assume UDP) */
        offset = (pIpHdr->VerLen & 0xF) * 4;
        pUdpHdr = (UDPHDR *)(((UINT8 *)pIpHdr)+offset);
    
        /* Get the correct SockProt to notify */
        if( pIpHdr->Protocol == 6 )
            SockProt = SOCKPROT_TCP;
        else if( pIpHdr->Protocol == 17 )
            SockProt = SOCKPROT_UDP;
        else
            return;
    
        /* Notify all exact matches */
        ps = pSockList[SockProt];
    
        while( ps )
        {
            /* Find a match */
            ps = SockPcbFind( ps, LIP, pUdpHdr->SrcPort,
                                         FIP, pUdpHdr->DstPort, &wc, 1 );
            /* Notify the protocol */
            if( ps && !wc )
            {
                /* For redirect errors, we just update the route. */
                if( Code >= PRC_REDIRECT_NET && Code <= PRC_REDIRECT_TOSHOST )
                    SockValidateRoute( ps );
                else
                    SockPrCtlError( ps, Code );
            }
    
            /* Get the next matching socket */
            if( ps )
                ps = ps->pProtNext;
        }
    }
    
    /*-------------------------------------------------------------------- */
    /* SockPcbRtChange() */
    /* Called to notify the socket layer that a route has changed */
    /*-------------------------------------------------------------------- */
    void SockPcbRtChange( HANDLE hRt )
    {
        SOCK   *ps;
        int    i;
    
        for( i=SOCKPROT_TCP; i<=SOCKPROT_RAW; i++ )
        {
            ps = pSockList[i];
    
            while(ps)
            {
                if( ps->hRoute == hRt )
                {
                    ps->hRoute = 0;
                    RtDeRef( hRt );
                }
                SockValidateRoute( ps );
                ps = ps->pProtNext;
            }
        }
    }
    
    /*-------------------------------------------------------------------- */
    /* SockGetPcb() */
    /* Return socket PCB list */
    /*-------------------------------------------------------------------- */
    int SockGetPcb( uint SockProt, uint BufSize, UINT8 *pBuf )
    {
        SOCK    *ps;
        SOCKPCB *ppcb;
        int     i;
    
        if( SockProt<SOCKPROT_TCP || SockProt>SOCKPROT_RAW )
            return(0);
    
        ps = pSockList[ SockProt ];
    
        i=0;
        while( ps && BufSize >= sizeof( SOCKPCB ))
        {
            ppcb = (SOCKPCB *)pBuf;
    
            ppcb->IPAddrLocal   = ps->LIP;
            ppcb->PortLocal     = ps->LPort;
            ppcb->IPAddrForeign = ps->FIP;
            ppcb->PortForeign   = ps->FPort;
            if( ps->hTP )
                ppcb->State = SockPrGetState( ps, ps->hTP );
            else
                ppcb->State = 0;
            if( !ppcb->State && (ps->OptionFlags & SO_ACCEPTCONN) )
                ppcb->State = 1;
    
            i++;
            BufSize -= sizeof(SOCKPCB);
            pBuf    += sizeof(SOCKPCB);
    
            ps = ps->pProtNext;
        }
    
        return(i);
    }
    
    /**
     *  @b Description
     *  @n
     *      The function cycles through all the socket entries for the
     *      specified protocol family and closes any socket which matches the
     *      IP Address specified. The function was added to ensure that when
     *      an IP address is modified then any sockets which are using the OLD
     *      IP Address should be closed.
     *
     *      If a socket is waiting to accept incoming connections, an error is set
     *      with a wake up signal to allow a blocked accept() call to return. This
     *      allows an clean reboot/shutdown with no dangling sockets, as well as
     *      giving control back to an application Task so it can handle stack
     *      reboot or shutdown appropriately.
     *
     *      This is for *internal* NDK Stack Usage.
     *
     *  @param[in]  SockProt
     *      Socket Family
     *  @param[in]  IPAddress
     *      OLD IP Address to which a socket might be bound to and which needs
     *      to be cleaned.
     *
     *  @retval
     *      Not Applicable
     */
    void SockCleanPcb (uint SockProt, IPN IPAddress)
    {
        SOCK    *ps;
        SOCK    *psNext;
        int error;
    
        /* Validate the socket protocol family. */
        if( SockProt<SOCKPROT_TCP || SockProt>SOCKPROT_RAW )
            return;
    
        /* Cycle through all the entries in the socket table. */
        ps = pSockList [SockProt];
        while (ps != NULL)
        {
            /* Save next entry before SockClose() sets it to 0 (fixes mem leak) */
            psNext = ps->pProtNext;
    
            /* Clean the entry only if we have a match */
            if (ps->LIP == IPAddress) {
                /*
                 *  Look for any sockets that are blocked on accept() call
                 *  (fix for SDOCM00115513)
                 */
                if (ps->OptionFlags & SO_ACCEPTCONN) {
                    /* Set an error for this socket to force accept() to return */
                    error = ECONNABORTED;
                    SockSet((HANDLE)ps, SOL_SOCKET, SO_ERROR, &error,
                            sizeof(error));
                    /*
                     *  Wake the socket blocked on accept(), allowing accept() to
                     *  handle the error and return
                     */
                    FdSignalEvent(ps, FD_EVENT_READ);
                }
    
                SockClose (ps);
            }
    
            /* Get the next entry. */
            ps = psNext;
        }
        return;
    }
    
    

    There are just a few lines different. The file is in C:\ti\tirtos_tivac_2_12_01_33\products\ndk_2_24_02_31\packages\ti\ndk\stack\sock\sockpcb.c

  • After cable is pulled, you can click the disconnect button on Hercules either before or after the cable gets plugged in again.  It doesn't matter since the link is severed:

    Ethernet TCP ports closed

    Ethernet cable unplugged

    Network Removed: If-1:10.40.1.82

    Ethernet Status report 0x5, 0x0, 0x0

    Service Status: DHCPC    : Disabled :          : 000

    DisableNDKDHCP success

    After plugging cable back in:

    Error - cannot open Ethernet TCP Socket. Cannot bind to socket. Port Num = 10070. err = 0x30

    Ethernet TCP Socket Created. Port Num = 10071.

    Ethernet Sockets Created. Port Num = 10071.

    Etherent TCP Client Disconnected. Addr = 0x0

    Now change port number in Hercules to 10071 and click connect:  

    "client request detectedEthernet TCP - Client connected. Addr = 10.40.1.200

    Socket Connected"

  • Todd,

    Regarding the ndk file, I located the file here: C:\ti\tirtos_tivac_2_12_01_33\products\new_ndk_2_24_02_31\packages\ti\ndk\stack\sock\sockpcb.c

    However, I don't know how to recompile. I only added the NDK include path to my project and didn't include the NDK library. Is there even a library file for the ndk? How do I compile the ndk and link the project to the new library?

    -Yan
  • Take a look in the TI-RTOS User Guide. There is a section on rebuilding.
  • I followed the instructions and rebuilt the RTOS - took several minutes.  Still having the same issue.  Do you see this issue go away when you applied the new sockpcb.c?

  • I'm pulling in one of the NDK engineers to look at this. It might take a day or two to get a response.
  • Hi Yan,

    I just wanted to update you that I am in the process of reviewing and reproducing the issue. I'll post back here once I have more.

    Steve

  • Hi Steve,

    Any updates on this?

    Thanks.

    -Yan

  • Hi Yan,

    I don't have much of an update yet, but want to at least let you know that I have not forgotten about this problem.

    We have been able to run your application here locally, but could not see the issue yet. On Monday, we will track down Todd to have him review and make sure we're doing things correctly.

    Just to confirm with you, we are setting break points in the code and checking the heap in ROV, using your output log as a guideline (basically, we're putting break points at the parts where you added "------>" type comments into the log file you attached). It could help if you confirmed that this sounds correct to you.

    Steve

  • Hi Steve,

    It doesn't matter where you put the breakpoint as long as you are breaking at the same spot each time you are checking heap in the ROV.  The steps to recreate are posted in this thread earlier.  A summary is as follows:

    1. Establish TCP connection to the Launchpad. Sample the heap in ROV.

    2. Yank out ethernet cable.

    3. Put ethernet cable back in

    4. Establish TCP connection to the Launchpad. Sample the heap in ROV again and notice that heap has decreased.

    Make sure a TCP connection is still established when pulling out the ethernet cable.

    -Yan

  • Hi Yan,

    OK, I think we are seeing an issue here, although what we're seeing here is closer to 2K that appears to be leaking.

    I'm digging into this and will let you know once I find out more.

    Steve

  • Steve,

    That's great you can reproduce the issue.  Please keep me posted. I do need a fix for this at some point.

    -Yan

  • Yan,

    I was finally able to come back to this issue. I added some debug code to print out the socket table to see what's going on.

    What I'm seeing is that the original TCP socket (the one that you must connect to in order to reproduce the issue) is still hanging around after the IP address has been removed and re-added due to the cable being unplugged and re-plugged.

    When you call close on that socket, it enters the close stage(s) of TCP. In this case, since you are issuing the close on it, this results in an active close. The socket then enters the TIMEWAIT state, in which it will wait for 2x the MSL, which should be 60 seconds.

    So, I do not think we are seeing a memory leak here, rather the socket and its resources haven't actually been freed yet, because the socket is held up in the TIME_WAIT state.

    The following excerpt of the output I got shows the issue. Note that the output will show that the socket is in the FINWAIT1 state, not the TIME_WAIT state, but rest assured that I did see the socket enter TIME_WAIT. (The reason it's showing FINWAIT1 is because I was actively halting the target, and most likely missed the ACK + FIN/ACK from the client side that would cause the actual transition to TIME_WAIT. Also, see end of this message for full output):

    Local IP         LPort  Foreign IP       FPort  State
    ---------------  -----  ---------------  -----  -----------
    0.0.0.0          0      0.0.0.0          0      CLOSED
    146.252.162.135  10070  146.252.162.253  62381  FINWAIT1
    

    Regarding the port error message, this is again due to the same socket still being in existence. When you try to create a new socket with the same port, the original one still exists (in the TIME_WAIT state), and so you get the error that the port is already in use.

    FYI - I recently helped another customer to resolve some issues when changing IP addresses. What came out of that is that all sockets for the IP address being removed are now forcibly closed, as they should no longer be valid when something drastic like an IP address is removed or the stack is shut down.

    If you are interested in that, see this thread which contains a patch and steps for how to rebuild the NDK with that patch update.

    Steve

    EPHY_BMCR = 0x0
    BMSR = 0x7849
    EPHY_ANA = 0x1e1
    EPHY_ANLPA = 0x0
    EPHY_STS = 0x2
    Host Comm Task Started. Starting Message Exchange
    Ethernet Status report 0x5, 0x4, 0x0
    Service Status: DHCPC    : Enabled  :          : 000
    Ethernet DHCP Client Started
    Service Status: DHCPC    : Enabled  : Running  : 000
    StartNDKDHCP: DHCP entry already exists. attempting to remove
    Ethernet Status report 0x5, 0x0, 0x0
    Service Status: DHCPC    : Disabled :          : 000
    Ethernet Status report 0x5, 0x4, 0x0
    Service Status: DHCPC    : Enabled  :          : 000
    Starting DHCP
    Ethernet DHCP Client Started
    Service Status: DHCPC    : Enabled  : Running  : 000
    Network Added: If-1:146.252.162.135
    Ethernet DHCP IP address added
    Service Status: DHCPC    : Enabled  : Running  : 017
    Ethernet IP address added: 146.252.162.135
    Ethernet UDP Socket Created. Port Num = 10065.
    Ethernet::OpenTCPSocket(), printing socket table
    
    
    
    Ethernet TCP Socket Created. Port Num = 10070.
    
    
    Ethernet Sockets Created. Port Num = 10070.
    Ethernet TCP - Client connected. Addr = 146.252.162.253
    Ethernet::TCPAccept(), printing socket table
    
    Socket Connected
    Ethernet TCP ports closed
    Ethernet cable unplugged
        CLOSED
    
    
    Local IP         LPort  Foreign IP       FPort  State
    ---------------  -----  ---------------  -----  -----------
    146.252.162.135  10070  146.252.162.253  62381  ESTABLISHED
    146.252.162.135  10070  0.0.0.0          0      LISTEN
    
    
    Local IP         LPort  Foreign IP       FPort  State
    ---------------  -----  ---------------  -----  -----------
    146.252.162.135  10070  146.252.162.253  62381  ESTABLISHED
    146.252.162.135  10070  0.0.0.0          0      LISTEN
    
    Network Removed: If-1:146.252.162.135
    Ethernet Status report 0x5, 0x0, 0x0
    Service Status: DHCPC    : Disabled :          : 000
    DisableNDKDHCP success
    
    
    Ethernet Status report 0x5, 0x4, 0x0
    
    Local IP         LPort  Foreign IP       FPort  State
    ---------------  -----  ---------------  -----  -----------
    146.252.162.135  10070  146.252.162.253  62381  FINWAIT1
    
    Service Status: DHCPC    : Enabled  :          : 000
    Starting DHCP
    Ethernet DHCP Client Started
    Service Status: DHCPC    : Enabled  : Running  : 000
    00029.000 TcpTimeoutRexmt: Retransmit Timeout
    00029.600 TcpTimeoutRexmt: Retransmit Timeout
    00031.600 TcpTimeoutRexmt: Retransmit Timeout
    Network Added: If-1:146.252.162.135
    Ethernet DHCP IP address added
    Service Status: DHCPC    : Enabled  : Running  : 017
    Ethernet IP address added: 146.252.162.135
    Ethernet UDP Socket Created. Port Num = 10065.
    Ethernet::OpenTCPSocket(), printing socket table
    
    
    
    Error - cannot open Ethernet TCP Socket. Cannot bind to socket. Port Num = 10070. err = 0x30
    Ethernet::OpenTCPSocket(), printing socket table
    
    
    
    Ethernet TCP Socket Created. Port Num = 10071.
    
    
    Ethernet Sockets Created. Port Num = 10071.
    Etherent TCP Client Disconnected. Addr = 0x0
    Ethernet TCP ports closed
    Ethernet cable unplugged
       0      0.0.0.0          0      CLOSED
    146.252.162.135  10070  146.252.162.253  62381  FINWAIT1
    
    
    Local IP         LPort  Foreign IP       FPort  State
    ---------------  -----  ---------------  -----  -----------
    146.252.162.135  10071  0.0.0.0          0      CLOSED
    0.0.0.0          0      0.0.0.0          0      CLOSED
    146.252.162.135  10070  146.252.162.253  62381  FINWAIT1
    
    00038.800 TcpTimeoutRexmt: Retransmit Timeout
    00048.400 TcpTimeoutRexmt: Retransmit Timeout
    Network Removed: If-1:146.252.162.135
    Ethernet Status report 0x5, 0x0, 0x0
    Service Status: DHCPC    : Disabled :          : 000
    DisableNDKDHCP success
    
    Local IP         LPort  Foreign IP       FPort  State
    ---------------  -----  ---------------  -----  -----------
    0.0.0.0          0      0.0.0.0          0      CLOSED
    146.252.162.135  10070  146.252.162.253  62381  FINWAIT1
    
    
    

  • Here's the updated project that will show that socket table print out.

    See SockUtils.c/h in the project for the code.

    Steve

    NDKTest_SockUtils.zip

  • What is the MSL and is there a way to reduce it?  The code to forcibly close all sockets sounds interesting.  I will verify this some time next week.  Thank you Steve.  

    -Yan

  • MSL is the max segment lifetime.

    You can change it but I wouldn't recommend it. Doing so could result in undesirable effects for TCP.

    If you still want to change it, you have the code and I can't stop you :) ... but note that you would be doing this at your own risk:

    // ti/ndk/stack/tcp/tcp.h
    #define TCPTV_MSL 300

    (this would require a stack rebuild)

    Steve
  • Yan,

    Did you get a chance to verify this? Can I mark this as resolved?

    Todd
  • Hi Todd,

    I was finally able to get to this on Friday, but got interrupted before I can get logs and screenshots for you.  I rebuilt the NDK tried out the new library on Friday, but did not see any difference in behavior.  I will get logs for you sometime later today or tomorrow.

    -Yan

  • Are all devices running NDK vulnerable to this memory leak?
  • Todd,

    I am running tirtos_tivac_2_12_01_33 with NDK v2.24.02.31 and have verified that all the .aem4f files in the packages sub-directories all have the Friday 4/13 date-stamp so I believe the NDK build was successful.

    I had to change a few lines of code so that the client would connect a second time. NDKTest_v2.zip

    1. Heap memory is always sampled at host_comm.cpp line 26 (NetworkInterface::MaintainLink()). When the Program first runs and Ethernet sockets are first created, heap is at 0xee40:

    2. TCP client connected and heap falls to 0xde50:

    3. Ethernet Cable pulled, heap regained to 0xF5e8

    4. Network cable re-plugged.  Heap drops to 0xd690 which is less than the previous value of 0xde50:

    5. After 3 iterations, heap has dropped to 0xae20. Therefore the memory leak is permanent and not temporary.