This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMS320C6678: NDK stops responding after some time

Part Number: TMS320C6678

Hi,

I have an application using NDK package that creates .cgi files and treats them as REST requests.

When the client polls on the DSP IP address requesting for the specific .cgi link, the callback will return the appropriate response in json format (char arrays obviously).

In one of my long time tests I have seen that after some time the DSP does not respond to requests anymore, but it still responds to ping.

After attaching with debugger I see the core running idle task mainly and it does not break to the breakpoints added in the callback functions.

It seems like those callback are not called at all, either because the request event is not caught or not correctly routed to the callback.

I am using:

SYS/BIOS 6.73.0.12

NDK 3.40.1.01

PDK 2.0.11

  • Hi,


    Any update on this?

    I have noticed that the actual reason why the requests fail after "crash" is because of a CONNECTION_REFUSED error.

    Still DSP is available for ping requests.


    Thank you,

    Fabrizio

  • Hi 

    A few follow up questions 

    1. Can you elaborate your usage case, do you run the C6678 as a HTTP server? And you added some CGI functions into the embedded file system?
    2. Have you looked at the NDK API Guide, section E: https://software-dl.ti.com/simplelink/esd/simplelink_msp432e4_sdk/2.30.00.14/docs/ndk/NDK_API_Reference.html#e-web-programming-with-the-http-serveror the PDF version https://www.ti.com/lit/ug/spru524k/spru524k.pdf?
    3. What is the heap size? Will increase it help? 
    4. In the ROV view, there should be several tasks. Is the http server task called "daemon", from the stackSize and stackPeak, do you see any issue? E.g, stack overflow? Besides the daemon and idle, do you have any other NDK threads running? Or have any other tasks with priority higher than the daemon?
    5. You mentioned "CONNECTION_REFUSED" error, which NDK file listed this error code? I don't see this in the latest NDK. 

    Additionally , it appears that you would've had a stable product/software for a while, so what changed in the hardware, software or system testing that you are seeing these issues?

  • There were some wireshark captures shared offline, follow up questions on those 

    1. There are two TCP tear-down sequences (success and failure) in the capture, both came with FIN/ACK--->ACK followed by FIN/ACK----->ACK. It looks OK.
    2. After the first tear-down, there is a TCP connection sequence: SYN--->SYN/ACK--->ACK, it looks OK.
    3. in the failure case, there is a packet #1685912 right before the HTTP packet. Is there a similar one in the success case, before #1685899?
    4. In the success case, what DSP is ACKing before sending out FIN/ACK?
    5. In the failure case, do you expect similar ACKing before sending out FIN/ACK immediately after request.

     

    A longer Wireshark capture with a few more successful cases would be helpful to compare the trace difference. 

  • Hi,

    Thank you for your reply.

    I will start responding to your first 5 questions first:

    1. The C6678 uses a dedicated core as a HTTP server. I added some CGI functions that do not send HTML but send data in JSON format instead.
    2. The code is developed following the programming guide and based on the example given with the NDK.
    3. Heap size is default used. I do not see any heap problems when error happens.
    4. ROV does not show any stack overflow as visible in the next picture
      /resized-image/__size/640x480/__key/communityserver-discussions-components-files/791/pastedimage1617804919866v1.png
    5. Connection refused is what we see from client. No CGI file is sending that. I think it might be because the DSP actively refuses the failing request with the FIN ACK packet immediately after client send the request.

    We have facing this issue now because we are testing our system in greater anger (more hours, higher request polling rate to DSP).

    Regarding your next 5 questions, these will also make clearer the response given at previous point 5:

    1. YES
    2. YES
    3. Yes, packet #1685912 contains the request headers, packet #1685913 contains the request content and it completes the request.
    4. In the success case the DSP ACKs for request accepted, then it sends the response and finally sends the FIN/ACK to notice the client that the request is completed.
    5. Yes, we would expect similar ACKing (meaning request accepted) but instead FIN/ACK (refusing connection) is sent.

    I will attach a wireshark capture with some successful requests and the unsuccessful one to the same offline topic

    It seems to me that at some point the DSP actively refuses any other request.

    Can this be for some buffer saturation or TCP stack saturation?

  • Hi,

    also, I looked at the programmer guide and did not find any possible way or method to reset the stack without restarting the entire thread.

    Could it be possible to do this in some way?

  • Thanks for the additional information provided , it gives us better understanding of your usage case and confirmation of packet trace analysis.

    We looked at the new Wireshark capture with a few more successful cases, this also confirmed that DSP immediately refused the connection after receiving the request, in the failure case.

    You mentioned that you followed the client.c under the PDK/NIMU example stackTest function, did you change the TCP Rx and Tx buffer to a bigger value other than the 8192 bytes? You can use 65536 for all three cfg.

        // TCP Transmit buffer size

        rc = 8192;

        CfgAddEntry( hCfg, CFGTAG_IP, CFGITEM_IP_SOCKTCPTXBUF,

                     CFG_ADDMODE_UNIQUE, sizeof(uint32_t), (uint8_t *)&rc, 0 );

     

        // TCP Receive buffer size (copy mode)

        rc = 8192;

        CfgAddEntry( hCfg, CFGTAG_IP, CFGITEM_IP_SOCKTCPRXBUF,

                     CFG_ADDMODE_UNIQUE, sizeof(uint32_t), (uint8_t *)&rc, 0 );

     

        // TCP Receive limit (non-copy mode)

        rc = 8192;

        CfgAddEntry( hCfg, CFGTAG_IP, CFGITEM_IP_SOCKTCPRXLIMIT,

                     CFG_ADDMODE_UNIQUE, sizeof(uint32_t), (uint8_t *)&rc, 0 );

    As the ICMP ping is responding when the HTTP failed (over TCP), so the DSP is still alive. This is most likely a network stack issue. The NC_NetStop(1) followed by NC_NetStart() should be able to reboot the stack. Do you mean that ICMP can’t recover after reboot? In the NC_NetStart() there are 3 callback functions, do you see the first callback (NetStartCb) is executed? In the ROV viewer, daemon is the Http server, there is no other servers like a TCP server, correct?

     

  • HI,

    Thank you for your answer.

    I will immediately start a test with the increased buffers (I was using default 8192).

    I just ran NC_NetStop(1) as suggested by mail, I did not explicitly call NC_NetStart again. I have seen that it would be called in that while loop:

     do
        {
            rc = NC_NetStart( hCfg, NIMU_testNetworkOpen, NIMU_testNetworkClose, NIMU_testNetworkIpAddr );
        } while( rc > 0 );

    Are you saying this would not be called after I use NC_NetStop(1) and I shall call NC_NetStart after using the stop function?

    also, as you can see, I do not have the same callbacks you are mentioning. The three callbacks are:

     

    static void NIMU_testNetworkOpen()
    {
        PRINT("StackTest: entering in NIMU_testNetworkOpen...\n");
    
        // Create our local servers
        hEcho = DaemonNew( SOCK_STREAMNC, 0, 7, dtask_tcp_echo,
                           OS_TASKPRINORM, OS_TASKSTKNORM, 0, 3 );
        hEchoUdp = DaemonNew( SOCK_DGRAM, 0, 7, dtask_udp_echo,
                              OS_TASKPRINORM, OS_TASKSTKNORM, 0, 1 );
        hData = DaemonNew( SOCK_STREAM, 0, 1000, dtask_tcp_datasrv,
                           OS_TASKPRINORM, OS_TASKSTKNORM, 0, 3 );
        hNull = DaemonNew( SOCK_STREAMNC, 0, 1001, dtask_tcp_nullsrv,
                           OS_TASKPRINORM, OS_TASKSTKNORM, 0, 3 );
        hOob  = DaemonNew( SOCK_STREAMNC, 0, 999, dtask_tcp_oobsrv,
                           OS_TASKPRINORM, OS_TASKSTKNORM, 0, 3 );
        // Create the IPv6 Local Servers.
    #ifdef _INCLUDE_IPv6_CODE
        hEcho6    = Daemon6New (SOCK_STREAM, IPV6_UNSPECIFIED_ADDRESS, 7, dtask_tcp_echo6,
                                OS_TASKPRINORM, OS_TASKSTKNORM, 0, 3 );
        hEchoUdp6 = Daemon6New (SOCK_DGRAM, IPV6_UNSPECIFIED_ADDRESS, 7, dtask_udp_echo6,
                                OS_TASKPRINORM, OS_TASKSTKNORM, 0, 1 );
        hTelnet6  = Daemon6New (SOCK_STREAM, IPV6_UNSPECIFIED_ADDRESS, 23,
                                (int(*)(SOCKET,uint32_t))telnetClientProcess, OS_TASKPRINORM, OS_TASKSTKLOW,
                                (uint32_t)ConsoleOpen, 2 );
        hOob6     = Daemon6New (SOCK_STREAM, IPV6_UNSPECIFIED_ADDRESS, 999, dtask_tcp_oobsrv,
                                OS_TASKPRINORM, OS_TASKSTKNORM, 0, 3 );
        hWeb6     = Daemon6New (SOCK_STREAM, IPV6_UNSPECIFIED_ADDRESS, HTTPPORT, httpClientProcess,
                                OS_TASKPRINORM, OS_TASKSTKHIGH, 0, 4);
    #endif
    }

    static void NIMU_testNetworkClose()
    {
        DaemonFree( hOob );
        DaemonFree( hNull );
        DaemonFree( hData );
        DaemonFree( hEchoUdp );
        DaemonFree( hEcho );
    
    #ifdef _INCLUDE_IPv6_CODE
        Daemon6Free (hEcho6);
        Daemon6Free (hEchoUdp6);
        Daemon6Free (hTelnet6);
        Daemon6Free (hOob6);
        Daemon6Free (hWeb6);
    #endif
    
    #ifdef  TEST_RAW_SEND
        TaskDestroy (hSendRaw);
    #endif
    #ifdef  TEST_RAW_RECV
        TaskDestroy (hRecvRaw);
    #endif
    
        // Kill any active console
        ConsoleClose();
    }

    static void NIMU_testNetworkIpAddr( uint32_t IPAddr, uint32_t IfIdx, uint32_t fAdd )
    {
        RTOSAL_LOG_CONSOLE_PRINT("StackTest: entering in NIMU_testNetworkIpAddr...\n");
    
        static uint32_t fAddGroups = 0;
        uint32_t IPTmp;
    
        if( fAdd )
            NIMU_testLog("Network Added: ");
        else
            NIMU_testLog("Network Removed: ");
    
        // Print a message
        IPTmp = NDK_ntohl( IPAddr );
        NIMU_testLog("If-%d:%d.%d.%d.%d\n", IfIdx,
                (uint8_t)(IPTmp>>24)&0xFF, (uint8_t)(IPTmp>>16)&0xFF,
                (uint8_t)(IPTmp>>8)&0xFF, (uint8_t)IPTmp&0xFF );
    
        // This is a good time to join any multicast group we require
        if( fAdd && !fAddGroups )
        {
            fAddGroups = 1;
    //              IGMPJoinHostGroup( inet_addr("224.1.2.3"), IfIdx );
        }
    
        /* Create a Task to send/receive Raw ethernet traffic */
    #ifdef  TEST_RAW_SEND
        hSendRaw = TaskCreate( Nimu_testSendRawEth, "TxRawEthTsk", OS_TASKPRINORM, 0x1400, 0, 0, 0 );
    #endif
    #ifdef  TEST_RAW_RECV
        hRecvRaw = TaskCreate( Nimu_testRecvRawEth, "PerformRawRX", OS_TASKPRIHIGH, 0x1400, 0, 0, 0 );
    #endif
    }

    All code under #ifdef condition is not built.

    ROV status after the restart tentative is exactly how shown in this screenshot:

  • Hi,

    As this is very critical for us, can we arrange a call to have a look at the code and live debug?

    Thank you

    BR

  • Hi,

    The test just ran with the buffer size changed as suggested (65536) failed exactly like previous iterations with default size to 8192.

    Any other idea?

  • Hello Fabrizio

    Thanks for the information. I will discuss your latest updates with the software team. 

    The missing callbacks could be because you have an older NDK release? 

    One additional question that came up was 

    Are you  using any fragmentation on the packets resulting reassembly? Does the issue (out of packet) happen with only reassembly?

  • No other ideas at the moment. 

    There is one thread, on a different product that sound similar, but the folks that supported that debug are no longer with TI

    https://e2e.ti.com/support/microcontrollers/other/f/other-microcontrollers-forum/656848/rtos-tm4c1294ncpdt-ndk-http-server-stops-responding-until-tcp-accept

    Given the problem shows up after several hours of testing - it is likely not a straightforward issue to isolate

  • Hi Fabrizio,

    In the NDK source code ti\ndk\inc\netctrl\netctrl.h header file,

    extern int NC_NetStart(void *hCfg, void (*NetStartCb)(),
    void (*NetStopCb)(), void (*NetIPCb)(uint32_t, uint32_t, uint32_t));

    I thought what Mukul mentioned earlier "In the NC_NetStart() there are 3 callback functions, do you see the first callback (NetStartCb)", the NetStartCb() he meant the function argument prototype. In your code, you have the  rc = NC_NetStart( hCfg, NIMU_testNetworkOpen, NIMU_testNetworkClose, NIMU_testNetworkIpAddr ); where NIMU_testNetworkOpen() is the real function following the client.c example, this is totally fine.

    To determine if you need to explicitly call NC_NetStart again after NC_NetStop(1) is called, you can try:

    1. in your NIMU_testNetworkOpen() function, adding a debug printf() or a counter to track each time this function is entered.

    2. Without calling NC_NetStop(), run your application code normally, it should call NC_NetStart(). Then you should see this function NIMU_testNetworkOpen() entered once (I don't think it will be called multiple times), then the NC_NetStart return 0, so the do .. while(rc >0) should exit. 

    3. Then you use some way to trigger/simulate the error condition and calls NC_NetStop(1), from the source code: 

    /*
    * NC_NetStop()
    * Initiate system shutdown
    */
    void NC_NetStop( int rc )
    {
    NetReturnCode = rc;
    NetHaltFlag = 1;
    }

    This merely gave the rc=1 to NetReturnCode. 

    4. Do you see your NIMU_testNetworkOpen() is entered again? I don't expect that, because the do... while(rc>0) existed already.

    5. If #4 is true, You need to call this again after #3:

     do
    {
    rc = NC_NetStart( hCfg, NIMU_testNetworkOpen, NIMU_testNetworkClose, NIMU_testNetworkIpAddr );
    } while( rc > 0 ); 

    Then you should see the NIMU_testNetworkOpen() is entered again, this is network stack reboot. 

    Let me know if this makes sense and what you observed.

    Regards, Eric

  • I am able to replicate the issue in about 1h and 20 mins when sending requests every 30/40 ms. You can maybe try that?

    This does not seem to be an issue related to mine, I have actually the feeling that these might be more similar:

    https://e2e.ti.com/support/legacy_forums/embedded/tirtos/f/ti-rtos-forum-read-only-archived/490959/tm4c1294-ndk-emac-hangs/1776221#1776221

    https://e2e.ti.com/support/microcontrollers/other/f/other-microcontrollers-forum/710149/rtos-tm4c1290ncpdt-ndk-in-ti-rtos-works-intermittently-after-being-shutdown-and-restarted

    https://e2e.ti.com/support/legacy_forums/embedded/tirtos/f/ti-rtos-forum-read-only-archived/146989/omapl138-c6748-ndk-emac-potential-driver-bug

    https://e2e.ti.com/support/legacy_forums/embedded/tirtos/f/ti-rtos-forum-read-only-archived/514100/ndk-stops-responding-on-ethernet

    https://e2e.ti.com/support/legacy_forums/embedded/tirtos/f/ti-rtos-forum-read-only-archived/223029/ndk-stops-responding#pifragment-322104=2

    Does this give you any more suggestions?

    Another thing we have noticed is:

    We ran another test using a different client. Originally we used requests (https://docs.python-requests.org/en/master/) library in python, this will have the behavior as described:

    • Sending requests every 40ms, DSP hangs in 1h 30m
    • Sending requests every 100ms, DSP hangs in 3h 45m
    • Sending requests every 1s, DSP hangs in 15h

    When using pycurl client (http://pycurl.io/docs/latest/quickstart.html) we have noticed that the requests are sent differently:

    • the client will send the headers at time t0
    • the client will send the request body at time t1 = t0 + 1s

    This "lack" of performance with pycurl resulted in having the test running successful for 3h, without any restart the test was run successfully again for 20h and without any restart the test was launched again for 48h. Unfortunately in the 48h launch the DSP hangs after 11 and 30 mins.

    Basically pycurl client worked for 34h and 30 mins but considering some interruptions in the middle. We are launching a test for straight 48h to see if the behavior changes.

    As pycurl takes more time to send the request (first initiates the request then send the payload after 1s) makes me think that there might be some race conditions happening in the internal dispatching buffers (NDK uses Multicore Navigator).

    Might these race condition create memory leaks in the Multicore navigator buffers?

    May the leaks happen less frequently with slower requests and thus make the test last longer?

  • Hi,

    I do not use much care in reassembling the packets, I simply use the combination of 

    httpSendStatusLine(Sock, StatusCode, ContentType);
    httpSendClientStr(Sock, CRLF);
     to send the headers and 
    httpSendClientStr(sock, str);
     to send the data.

    I let the server do the fragmentation on its own if the string is bigger than the transmit buffer.

  • Hi Eric,

    Thank you for your reply!

    I have a different header for that function:

    extern int NC_NetStart( void *hCfg, void (*NetStart)(),
                            void (*NetStop)(), void (*NetIP)(uint32_t,uint32_t,uint32_t) );
    

    This is why I thought it was a different function he was mentioning about!

    Responding to your other points:

    1. I am working to do that!
    2. NC_NetStart internally calls the NetScheduler function. The scheduler has a while loop that in is interrupted only when NC_NetStop is called as NC_NetStop sets the NetHaltFlag = 1. By setting this the NetScheduler exits its while loop and finally allows NC_NetStart  to return a value. This value should be NetReturnCode (previously set to 1 by NC_NetStop). Thus, NC_NetStart never returns 0 before  NC_NetStop is called so NC_NetStart should return 1 in that do-while loop and thus I think it should be called again. Unless I am missing something, apologies for that Slight smile
    3. I have implemented a trigger, I see the net go down but do not see the net go up again unfortunately! If the code works as stated in point 2, NC_NetStop has no other job to do other than setting those 2 flags
    4. I will confirm this soon
    5. Will also confirm this soon

    Thank you again,

    Regards, Fabrizio

  • Hi,

    I have ndk_3_61_01_01 installed so I looked at that, I am not aware of the API is different from your NDK 3.40.1.01, sorry for this! But you can follow my suggestion and find out how the NIMU_testNetworkOpen is called and if it is called after you run NC_NetStop(1).

    Regards, Eric

  • Hi,

    No problem at all and thank you for your suggestion!

    I have verified this and I can confirm that the NIMU_testNetworkOpen is called a second time after net NC_NetStop(1)

    Have a look at the flag printed here (I have censored non relevant data):

    callback count:         0x00000001(1)

    callback count:         0x00000002(2)

    I believe that there is something going wrong in the middle because if I try to restart the net again this variable is not incremented again.

    The callback completes two times fully. Maybe one of the other callbacks fail. I will investigate further

  • So what happens is the following:

    After I run the restart I can see the function called correctly and also I am seeing these prints:

    Service Status: Telnet   : Disabled :          : 000
    Service Status: HTTP     : Disabled :          : 000
    StackTest: entering in NIMU_testNetworkIpAddr...
    Network Removed: If-1:192.168.1.4
    Service Status: Telnet   : Enabled  :          : 000
    Service Status: HTTP     : Enabled  :          : 000
    StackTest: entering in NIMU_testNetworkOpen...

    but still DSP is not available for ping

  • I also confirmed by using some degug flags that the NIMU_testNetworkIpAddr function is called to remove the network after the net is stopped but it is not called again to add the IP again.

    I wonder why this is not happening...

    In the teardown sequence of the Net, after exiting the scheduler, I see a lot of shut downs and resetting of configurations (highlighted). May these be a reason?

    int NC_NetStart( void *hCfg, void (*NetStart)(),
                     void (*NetStop)(), void (*NetIP)(uint32_t,uint32_t,uint32_t) )
    {
        uint32_t   EtherDeviceCount = 0;
        uint32_t   SerialDeviceCount = 0;
        CFGMAIN *pc = NULL;
        CFGENTRY *pe = NULL;
        int elemCnt = 0;
        int bootStkSz = OS_TASKSTKBOOT_DEF;

        /* Make sure we're ready to run */
        if( !SystemOpen )
            return(0);

        /* Initialize our state to be "not halted" */
        NetHaltFlag = 0;

        /* Initialize the timer */
        /* If we wanted, we could initialize the timer value with the */
        /* seconds elapsed from some standard time (1-1-70, 1-1-80, etc.). */
        _llTimerInit( &stkEvent, 0 );

        /* Initialize the User LED driver */
        _llUserLedInit();

        /* Initialize the serial port */
        SerialDeviceCount = _llSerialInit( &stkEvent );
        if( SerialDeviceCount > 1 )
            SerialDeviceCount = 1;

        /* Initialize the Network Interface Management Unit */
        NIMUInit (&stkEvent);

        /* Initialize the VLAN Module. */
        VLANInit();

    #ifdef _INCLUDE_IPv6_CODE
        /* Initialize the IPv6 Stack. */
        IPv6Init ();
    #endif

        /* Set the deault config handle */
        CfgSetDefault( hCfg );

        /* Record the function pointers */
        NetStartFun = NetStart;
        NetStopFun  = NetStop;
        NetIPFun    = NetIP;

        /* Initialize the stack executive */
        ExecOpen();

        /*
         *  Get boot task stack size from the database as configuration hasn't been
         *  applied yet (fix for NDK-110)
         */
        pc = (CFGMAIN *)hCfg;

        /* Get linked list of OS config entries */
        pe = (CFGENTRY *)pc->pCfgEntry[CFGTAG_OS - 1];

        /* Traverse the list until the entry for the boot task is found */
        while (pe && elemCnt < CFGITEM_OS_MAX) {
            if (pe->Item == CFGITEM_OS_TASKSTKBOOT) {
                bootStkSz = *((int *)pe->UserData);
                break;
            }
            elemCnt++;
            pe = pe->pNext;
        }

        /* Boot the configuration */
        if (!(TaskCreate(NS_BootTask, "ConfigBoot", OS_TASKPRINORM, bootStkSz,
                (uintptr_t)hCfg, 0, 0))) {
            /* Couldn't create boot task, don't start scheduler, close down stack */
            NetHaltFlag = 1;
            NetReturnCode = -1;
        }

        /* Start running the stack */
        NetScheduler(SerialDeviceCount,EtherDeviceCount);

        /*
         *  Fix for SDOCM00115318 - must tear down serial/PPP config before
         *  disabling it! Otherwise config tear down code in PPP deinit fails
         */
        _llSerialShutdown();

        /* Disable the configuration */
        CfgExecute( hCfg, 0 );

        /* Call the Net stop function */
        if( !flagSysTasks ) {
            DbgPrintf(DBG_INFO,
                    "\n\nNC_NetStart: WARNING: Boot thread has not completed!\n\n");
        }
        else
        {
            flagSysTasks = 0;
            if( NetStopFun )
                (*NetStopFun)();
        }

        /*
         * In order for the system to settle out properly, we must
         * be "OS_SCHEDULER_LOWPRI" at this point. Don't bother checking the
         * return value here, if there's a problem, we're already exiting the
         * stack ...
         */
        TaskSetPri( TaskSelf(), OS_SCHEDULER_LOWPRI );

        /* Shutdown the stack */
        ExecClose();

        /* Clear function callbacks */
        NetStartFun = 0;
        NetStopFun  = 0;
        NetIPFun    = 0;

        /* Clear the default config handle */
        CfgSetDefault( 0 );

        /* Close the VLAN Module. */
        VLANDeinit();

        /* Shutdown the low-level environment */
        NIMUShutdown();

        _llUserLedShutdown();
        _llTimerShutdown();

        return( NetReturnCode );

    }

  • Hi Fabrizio,

    Thanks for providing the feedback. So the experiment confirmed that if you called the NC_NetStop(1), the network removed the IP address, close it then open again, however the NIMU_testNetworkIpAddr is not called.

    I'm not familiar with this so you may need to debug why, a simple way is you add the netctrl.c source code (from NDK) into your application and build. The functions inside the source code will override the same functions in the NDK library. Then you can modify the netctrl.c for debug purpose in your application without rebuilding NDK library every time. You may need to add header include path (packages\ti\ndk\inc) to application.

    As you confirmed that when HTTP is dead, the ping is still working. This means the hardware is still alive. I am not sure if at that time UDP and TCP still work. Under ndk_3_xx_xx_xx\packages\ti\ndk\winapps, there are several executables you can test from your computer with C6678, like send.exe (this is TCP), testudp.exe (this is UDP). Always, compare your system in good condition with failure condition to understand what was broken. The HW has queues and interrupt (it is a L2 switch), it has no differentiation of traffic type (L3/L4: udp, icmp, tcp, etc).

    https://software-dl.ti.com/processor-sdk-rtos/esd/docs/latest/rtos/index_Foundational_Components.html#ndk has some diagram showing how NDK and NIMU worked together. Under CCS (if you use JTAG/CCS to debug) expression window, you can watch some global variables (NDK_tcps, NDK_ICMPIn, NDK_ICMPOut) for statistics. Under CCS (if you use JTAG/CCS to debug) expression window, you can watch some global variables (NDK_tcps, NDK_ICMPIn, NDK_ICMPOut) for statistics.   

    There is no clean way to reset the Ethernet hardware (PA, QMSS, CPPI, EMAC), power cycle those domains may work, but not sure if anybody tried. I am trying to say if you can prove the Ethernet hardware is still alive, please try not to do any reset.

    You may need to look at NIMU_testNetworkClose, NIMU_testNetworkIpAddr (remove IP address) if they do anything to HW. If not, when the network reopen, the NC_Netstart code calls NIMUInit() again. It most likely you need to bypass this NIMUInit() call. 

    Regards, Eric

  • Hi Eric,

    Thank you for your suggestions!

    I will check the global variables you suggested and compare any difference with working target vs non-working target.

    Regarding the NIMUInit() function: I think it is correct to call it again during NC_NetStart as in the same function, during the teardown sequence, it is also called NIMUShutdown. So when NC_NetStart runs again after reboot correctly starts the NIMU again. If there is something wrong in this only TI guys can suggest what to change Slight smile

    based on the feedback and thoughts shared, any suggestion on why TCP stops and why Net does not restart?

    Regards,

    Fabrizio

  • Hi Fabrizio, 

    Curious if TCP test or UDP test still works when the HTTP failed.

    NDK module processes the L3 and L4, the NIMU (or we call it as hardware) handles L2. For C6678, the NIMU driver is under <C6678pdk installer>\packages\ti\transport\ndk\nimu\src\v1\nimu_eth.c. The NDK interface to NIMU layer is ti\ndk_3_61_01_01\packages\ti\ndk\stack\nimu\nimu.c.

    In the NDK NC_NetStart(), you can see a call nimuInitRC = NIMUInit(&stkEvent); This function calls into NIMUDeviceTable[], which is defined in the NIMU driver: 

    NIMU_DEVICE_TABLE_ENTRY NIMUDeviceTable[] =
    {
    /**
    * @brief EmacInit for the EVM
    */
    EmacInit,
    NULL
    };

    This is how the hardware initialized. EmacInit() -----> EMACInit_Core(). Also, you can see:

    /* Populate the Driver Interface Functions. */
    ptr_device->start = EmacStart;
    ptr_device->stop = EmacStop;
    ptr_device->poll = EmacPoll;
    ptr_device->send = EmacSend;
    ptr_device->pkt_service = EmacPktService;
    ptr_device->ioctl = Emacioctl;
    ptr_device->add_header = NIMUAddEthernetHeader;
    ptr_device->flags = NIMU_DEVICE_NO_FREE;

    Then when the NIMUShutdown() is called, it calls into NIMUUnregister----> ptr_netif_device->stop (ptr_netif_device);---->EmacStop(). I am not sure if EmacStop() close everything in a clean way.

    My hope is, if UDP and TCP still can work, then you may restart the network but try to avoid calling into those NIMU drivers again (including stop function). You can also add nimu_eth.c into your application to build, and set some counter to track a function is called once then use a flag to excluding being called again.

    If UDP or TCP test failed, then restarting the NDK stack and HW are unavoidable, you may need debug further why some API is not called and how to properly shutdown and restart (because you have ping unreachable after restart).

    Regards, Eric 

  • Hi,

    Sorry for the delay in the response, I was busy with something else.

    I am checking those global variables (NDK_tcps, NDK_ICMPIn, NDK_ICMPOut) but I cannot see them changing. I am checking on expression tab and also in memory browser but the values are all 0s. This is happening when the system is in good health and the test is running.

    I have also tested the send.exe as suggested and I can see it working (at least it say passed) when the system is in good health.

    I have done another change: I have increased the stack size of the dchild thask. This is a task that is spawned by NDK to serve the HTTP requests.

    With this change, the test that was only 1h and 30 minutes long is now running for 2h and 30 minutes and still going.

    I have also started another test that usually was averagely 4h long and will update on the result.

    If any of these tests stops I will run again the send.exe tests to check the result I get from it.

    Still, how can this not being visible in ROV?

    How could we check how this stack is influencing my test execution?

    Thank you

    BR, Fabrizio

  • Hi Fabrizio,

    No issue! "NDK_tcps, NDK_ICMPIn, NDK_ICMPOut" all zero. It is possible that they use different names (maybe ips, tcps, ICMPIn, ICMPOut, etc....) in early NDK versions. In my ndk\inc\stack\inc\tcpif.h, I have:

    ...
    } TCPSTATS;

    extern TCPSTATS NDK_tcps;

    Please check those files under your NDK what the global variable name, sorry for this. At least in working case, those counter should increase.

    From an e-mail thread Mukul mentioned, https://e2e.ti.com/support/microcontrollers/other/f/other-microcontrollers-forum/656848/rtos-tm4c1294ncpdt-ndk-http-server-stops-responding-until-tcp-accept

    It seems when the HTTP stops, the user does a TCP connection and somehow the HTTP recovered. Not sure if this will also happen in your case, then you do NOT need to do a network shutdown.

    Regards, Eric

  • Hi Eric,

    The previously mentioned test ran for 3h and 30 minutes (usually took 1h and 20/30 mins). Just to be clear, the change was in the stack size of the dchild function (from 5120 to 8192 bytes).

    After the test failed I ran the send.exe tcp test and it actually gave passed messages. After that I checked the status of the tasks in ROV and this is what I have:

    This status is after I stop the send.exe test (with ctrl+c from terminal). This same operation was done before starting my test with system in good health.

    After this, I cannot ping DSP anymore (not sure if this is because I stopped the core to check the status in ROV)

    Anyways, the stack increase definitely improved my test execution time. How do you explain this?

    Regarding the flags you are asking: I have checked the names and it is correctly NDK_tcps, yet the values are all 0s. the only tcp-like global variables I have seen that are not 0s are the following (taken after system crash):

    just for reference, here also the NDK_tcps variable:

    This also should respond to the link you have sent: HTTP crashed (we do not know why yet) TCP was accepted but then DSP not pingable anymore.

    I will try to reproduce without stopping the DSP to check ROV.

    I also tried the restart again but apparently DSP is in un unrecoverable state now.

    Any idea?

    BR, Fabrizio

  • Hi,

    as mentioning before, another test expected to run for averagely 4 hours was repeated with the increased stack size of the dchild function.

    here unfortunately the test terminated as expected at averagely 4 hours as before change. So in this case the stack size apparently did not take effect.

    The test specs are:

    Test spec Stack size (bytes) Expected Duration Actual Duration
    HTTP requests every 40ms 5120 1h 30 mins 1h 30 mins
    HTTP requests every 40ms 8192 1h 30 mins 3h 30 mins
    HTTP requests every 100ms 5120 3h 45 mins 3h 45 mins
    HTTP requests every 100ms 8192 3h 45 mins 3h 45 mins

    Maybe I will repeat again the test with stack size increased but I do not understand why in the 40ms case the duration increased but in the 100ms it did not.

    I am still suspecting some memory issues/leaks in the TCP or HTTP code or in the Data link layer (EMAC-QMSS-CPPI).

    This has gone for over 45 days now, it is very critical for us.

    Thank you,

    BR

    Fabrizio

  • Hi,

    I have started a test that start a request and DSP will keep the session open sending data continuously every 0.1s. That test ran for 25h correctly, then the host PC stopped.

    I have started a new one to go over the weekend.

    Anyway, it seems that DSP Stack crashes when receiving a certain number of requests (estimated 120k requests).

    Any update?

    Thank you,

    BR

    Fabrizio

  • Hi,

    I was able to get the NDK stats we were talking about, but I do not see any difference between healthy and unhealthy system:

    tcps healthy

    tcp unhealthy

    ips healthy

    ips unhealthy

    tcp send winapp works after the crash happens and I can also see the NDK_tcps stats increase in that case.

    It seems that the HTTP layer is the one that crashes after receiving a certain number of connections and it does not come up after using TCP connection successfully.

    When sending HTTP requests I see no change in the NDK_tcps.

    I have also tried to break into the httpClientProcess function but apparently it does not break there, so I believe there is no binding to the HTTP layer after the crash.

    Any  suggestions?

  • Hi Fabrizio,

    Sorry for the late response for your last several updates! Now you confirmed that after the HTTP crash, the TCP is still working, the counter is kept going up. The previously mentioned ICMP ping didn't response is most likely you halt the CPU for ROV view. From NDK_ips and NDK_tcps you know the crash only happened in HTTP level.  

    Some suggestions:

    1. You may tried to update the NDK to the latest to see if that helps, in case the newest version has some update that resolves this issue. The API and libraries should be the same for you integration. The latest C6678 RTOS package is here: https://software-dl.ti.com/processor-sdk-rtos/esd/C667x/latest/index_FDS.html. It has the NDK inside. The trial may be quicker than debug, although the chance is slim. 

    2. I mentioned one thread "https://e2e.ti.com/support/microcontrollers/other/f/other-microcontrollers-forum/656848/rtos-tm4c1294ncpdt-ndk-http-server-stops-responding-until-tcp-accept" It looks to be relevant to your case, after the http failure, the re-connection to TCP (like you did for "send" test) somehow restore the HTTP server. You may have looked at it already, but did not help?

    3. I don't know if anyone familiar with NDK can help, either TI engineers or community members. From my earlier suggestion, given the problem is only on the HTTP stack, you may try to close and open the network without touching the hardware (NIMU level) to avoid HW level complication.

    You can run a http or TCP or ICMP ping test, then from the CCS expression window manually poke below to simulate NC_Netstop() 

    NetReturnCode = rc;
    NetHaltFlag = 1;

    You should see some IP address removed, net close, then net open/restart etc. In the current situation, you see the network can't recover after restart. E.g. ping failed. My thought is to find how to stop/start NDK only without touching the NIMU level.

    When you stop network, set break point in the NIMU level to see how the NIMU functions are called, then you need to bypass those NIMU calls. And when NC_NetStart() is called, bypass the same NIMU calls. If you can successfully restore the network, then you can move to the real HTTP issue to see the sequence can restore the HTTP as well. You found out that reducing the dchild stack size (what exact you changed, where?) or fast http connections can recreate the issue sooner.

    I knew it is easier for me to say, but you may have to practice this given no people familiar with the NDK can help. Good luck! Please still post findings if I can help or discuss. 

    Regards, Eric 

  • Hi,

    1. I will try to update the NDK with the latest version, but this will take some time to happen.
    2. This did not help. HTTP went down, used the tcp send winapp but the http did not recover. Anyways it is not an acceptable solution for our application.
    3. I will try to debug further in NIMU level depending on our priorities in the project.

    In the meantime, is it possible to ask a NDK expert to look at this in more detail?

    Regards, Fabrizio

  • Hi,

    I tried bypassing the NIMU calls nut nothing has changed.

    It is going on for almost 2 months now and there was no progress so far...

    Can any NDK expert have a look?

    There are two big issues in the NDK:

    1. HTTP server crashes after a certain number of requests
    2. NDK does not restart properly when calling NC_NetStop(1)

    Regards,

    Fabrizio

  • Hi,

    looking at the forum I have found this topic that apparently solves the net restart:

    https://e2e.ti.com/support/processors/f/processors-forum/247150/pdk6678-nimu

    I would like to try this but I do not know how to integrate it in my code:

    1. I have no res_mgr_init_qmss function like the one in the example code (it is an 8 years old code and seems to be based on the obsolete mcsdk package)
    2. I do not find any resourcemgr.c file that I can change with the one in the example code
    3. the added queueReset function added that should fix the issue is not acceptable (has a lot of magic numbers, not sure will comply any application and might have some memory leask as said by the author in the topic).

    I have tried to find similar ones in the libraries, maybe the closes would be transportQmssInstCleanup in the transportQmss.c file but I am not sure.

    Is it possible to have a patch for a similar tweak to test and insert in my NDK-PDK version?

    Regards,

    Fabrizio

  • Hi Fabrizio,

    We have validated NDK restart as part of network device driver shutdown testing on a different platform.

    The platform is Jacinto 6 SoC and Ethernet device on this platform is GMACSW. NDK is the same.

    I think you can base on the attached source to implement the restart sequence on your platform.

    You will need to replace GMACSW_open()/GMACSW_close() with equivalent network device open/close APIs on your platform.

    /*
     * Copyright (C) 2013 Texas Instruments Incorporated - http://www.ti.com/
     *
     *
     *  Redistribution and use in source and binary forms, with or without
     *  modification, are permitted provided that the following conditions
     *  are met:
     *
     *    Redistributions of source code must retain the above copyright
     *    notice, this list of conditions and the following disclaimer.
     *
     *    Redistributions in binary form must reproduce the above copyright
     *    notice, this list of conditions and the following disclaimer in the
     *    documentation and/or other materials provided with the
     *    distribution.
     *
     *    Neither the name of Texas Instruments Incorporated nor the names of
     *    its contributors may be used to endorse or promote products derived
     *    from this software without specific prior written permission.
     *
     *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
     *  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
     *  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
     *  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
     *  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
     *  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
     *  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
     *  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
     *  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
     *  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
     *  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
     *
    */
    
    /*!
     *  \file nsp_test_server.c
     *
     *  \brief NSP test server
     *
     */
    
    
    /*---------------------------------------------------------------------------*\
    |                                Header Files                                 |
    \*---------------------------------------------------------------------------*/
    
    /* Standard language headers */
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <assert.h>
    #include <stdbool.h>
    
    /* OS/Posix headers */
    #include <ti/sysbios/BIOS.h>
    #include <ti/sysbios/knl/Task.h>
    #include <ti/sysbios/knl/Semaphore.h>
    #include <ti/sysbios/utils/Load.h>
    #include <ti/sysbios/hal/Hwi.h>
    
    /* NDK Dependencies */
    #include <ti/ndk/inc/netmain.h>
    #include <ti/ndk/inc/tools/servers.h>
    #include <ti/ndk/inc/stkmain.h>
    #include <ti/ndk/inc/stack/stack.h>
    
    /* NSP Dependencies */
    #include <ti/nsp/drv/inc/gmacsw.h>
    
    /* NULL stack header file */
    #include "null_stack.h"
    
    /* Client/Server Common Header */
    #include "../../../common/test_common.h"
    
    
    /*---------------------------------------------------------------------------*\
    |                             Extern Declarations                             |
    \*---------------------------------------------------------------------------*/
    
    extern xdc_Void ti_ndk_config_Global_stackThread(xdc_UArg,xdc_UArg);
    
    
    /*---------------------------------------------------------------------------*\
    |                            Local Macros/Defines                             |
    \*---------------------------------------------------------------------------*/
    
    #define EXPERIMENTAL_ETHERTYPE      (0x88B5U)
    #define CPTS_ETHERTYPE              (0x88F7U)
    
    
    /*---------------------------------------------------------------------------*\
    |                            Local Typedefs/Enums                             |
    \*---------------------------------------------------------------------------*/
    
    typedef enum status_e
    {
        SUCCESS = 0,
        RESTART = 1,
        FAILURE = -1,
        SEND_SIZE_FAILURE = -2,
        UNABLE_TO_OBTAIN_LOAD = -3,
        UNEXPECTED_CMD_SIZE = -4,
        NULL_POINTER = -5,
        ACK_FAILURE = -6
    }
    Status_e;
    
    
    /*---------------------------------------------------------------------------*\
    |                         Local Function Declarations                         |
    \*---------------------------------------------------------------------------*/
    
    static int LOCAL_sendLoadInfo(SOCKET s);
    
    static int LOCAL_sendHWErrors(SOCKET s);
    static int LOCAL_sendHWStats(SOCKET s);
    static int LOCAL_clearHWStats(void);
    
    static int LOCAL_sendSWStats(SOCKET s);
    static int LOCAL_clearSWStats(void);
    
    static int LOCAL_sendNDKIpStats(SOCKET s);
    static int LOCAL_clearNDKIpStats(void);
    
    static bool LOCAL_needsAck(uint32_t cmdId);
    static int LOCAL_handleCommand(SOCKET s, uint8_t *buf, uint32_t size);
    static int LOCAL_commandServer(SOCKET s, UINT32 unused);
    
    static void LOCAL_restartTask(UArg arg0, UArg arg1);
    
    static void LOCAL_cptsEventNotify(void *hCallbackArg);
    static int LOCAL_sendCPTSFreqs(SOCKET s, uint32_t measured, uint32_t reported);
    static int LOCAL_sendCPTSEvent(SOCKET s);
    static int LOCAL_sendTxCPTSFrames(void);
    static int LOCAL_checkCPTSFreq(SOCKET s);
    
    
    /*---------------------------------------------------------------------------*\
    |                         Global Variable Declarations                        |
    \*---------------------------------------------------------------------------*/
    
    static HANDLE hTCPEcho = 0;
    static HANDLE hUDPEcho = 0;
    static HANDLE hTestServer = 0;
    static Task_Handle hRawEchoTask = NULL;
    static GMACSW_DeviceHandle hGMACSW = NULL;
    static NULL_Handle hNullStack = NULL;
    
    static bool     LOCAL_TCPEchoOn = true;
    static bool     LOCAL_UDPEchoOn = true;
    static bool     LOCAL_RawEchoOn = false;
    
    static bool     LOCAL_CPTSHandlerRegistered = false;
    
    /* Semaphore to sync the NSP/NDK restart activity */
    static Semaphore_Handle restartSem = NULL;
    
    static volatile uint32_t LOCAL_tcpPacketCount = 0;
    static volatile uint32_t LOCAL_udpPacketCount = 0;
    static volatile uint32_t LOCAL_rawPacketCount = 0;
    static volatile uint32_t LOCAL_cptsEventCount = 0;
    
    /* Memory to be used for the CPTS event SW queue */
    static CPTS_Event gCPTSEvents[16];
    
    
    /*---------------------------------------------------------------------------*\
    |                          Global Function Definitions                        |
    \*---------------------------------------------------------------------------*/
    
    /* my_tcp_echo() - TCP Echo Server Daemon Function (SOCK_STREAMNC) */
    /* (SOCK_STREAMNC, port 7) */
    /* Returns "1" if socket 's' is still open, and "0" if its been closed */
    int my_tcp_echo( SOCKET s, UINT32 unused )
    {
        struct timeval to;
        int            i;
        char           *pBuf;
        HANDLE         hBuffer;
    
        (void)unused;
    
        /* Configure our socket timeout to be 5 seconds */
        to.tv_sec  = 5;
        to.tv_usec = 0;
        setsockopt( s, SOL_SOCKET, SO_SNDTIMEO, &to, sizeof( to ) );
        setsockopt( s, SOL_SOCKET, SO_RCVTIMEO, &to, sizeof( to ) );
    
        i = 1;
        setsockopt( s, IPPROTO_TCP, TCP_NOPUSH, &i, 4 );
    
        for(;;)
        {
            i = (int)recvnc( s, (void **)&pBuf, 0, &hBuffer );
    
            /* If we read data, echo it back if echo is enabled */
            if( i > 0 )
            {
                if (LOCAL_TCPEchoOn)
                {
                    if( send( s, pBuf, i, 0 ) < 0 )
                        break;
                }
                LOCAL_tcpPacketCount++;
                recvncfree( hBuffer );
            }
            /* If the connection got an error or disconnect, close */
            else
                break;
        }
    
        fdClose( s );
    
        /* Return "0" since we closed the socket */
        return(0);
    }
    
    /* my_udp_echo() - UDP Echo Server Daemon Function */
    /* (SOCK_DGRAM, port 7) */
    /* Returns "1" if socket 's' is still open, and "0" if its been closed */
    int my_udp_echo( SOCKET s, UINT32 unused )
    {
        struct sockaddr_in sin1;
        int                i,tmp;
        char               *pBuf;
        HANDLE             hBuffer;
    
        (void)unused;
    
        /* Configure our socket timeout to be 3 seconds */
        {
            struct timeval     to;
    
            to.tv_sec  = 3;
            to.tv_usec = 0;
            setsockopt( s, SOL_SOCKET, SO_SNDTIMEO, &to, sizeof( to ) );
            setsockopt( s, SOL_SOCKET, SO_RCVTIMEO, &to, sizeof( to ) );
        }
    
    #if (0)
        /* Increase the socket buffer size */
        {
            int32_t size = 32*1024;
            setsockopt( s, SOL_SOCKET, SO_RCVBUF, &size, sizeof(size)));
        }
    #endif
    
        for(;;)
        {
            tmp = sizeof( sin1 );
            i = (int)recvncfrom( s, (void **)&pBuf, 0,(struct sockaddr *)&sin1, &tmp, &hBuffer );
    
            /* Spit any data back out */
            if( i >= 0 )
            {
                if (LOCAL_UDPEchoOn)
                {
                    sendto( s, pBuf, i, 0,(struct sockaddr *)&sin1, sizeof(sin1) );
                }
                LOCAL_udpPacketCount++;
                recvncfree( hBuffer );
            }
            else
                break;
        }
    
        /* Since the socket is still open, return "1" */
        /* (we need to leave UDP sockets open) */
        return(1);
    }
    
    /* Rx task for RAW socket packets */
    void my_raw_echo_task(UArg arg0, UArg arg1)
    {
        SOCKET s = INVALID_SOCKET;
        uint8_t macAddress[6];
    
        HANDLE pBufHandle = NULL;
        uint8_t *pRecvPacket = NULL;
        int32_t i;
    
        /* Allocate the file environment for this task */
        fdOpenSession( TaskSelf() );
    
        /* Open the raw socket which gets used for both Tx and Rx */
        s = socket(AF_RAWETH, SOCK_RAWETH, EXPERIMENTAL_ETHERTYPE);
    
        if( s == INVALID_SOCKET )
        {
            printf("Fail socket, %d\n", fdError());
            fdCloseSession (Task_self());
            return;
        }
    
        /* Set some socket options */
        {
            int ifVal = 1;
            int retVal = 0;
    
            /* Set socket option to set IF device */
            retVal = setsockopt(s, SOL_SOCKET, SO_IFDEVICE, &ifVal, sizeof(ifVal));
            if (retVal)
            {
                printf("Error in setsockopt() API.\n");
            }
    
            /*
             * Set max receive size to handle all possible packets that could be received at
             * once from the hardware.
             */
            ifVal = 64*1536;
            retVal = setsockopt(s, SOL_SOCKET, SO_RCVBUF, &ifVal, sizeof(ifVal));
            if (retVal)
            {
                printf("Error in setsockopt() API.\n");
            }
        }
    
        /* Get the interface MAC address using the NIMU ioctl()*/
        {
            int ret_code;
            uint16_t        dev_index;
            NIMU_IF_REQ     if_req;
    
            if_req.index = 1;
            if (NIMUIoctl (NIMU_GET_DEVICE_INDEX, &if_req, &dev_index, sizeof(dev_index)) < 0)
            {
                printf ("Error: Incorrect device index specified\n");
                exit(1);
            }
    
            if_req.name[0] = 0;
            if_req.index   = dev_index;
    
            /* Get the device MAC Address */
            ret_code = NIMUIoctl (NIMU_GET_DEVICE_MAC, &if_req, &macAddress, sizeof(macAddress));
            if (ret_code < 0)
            {
                printf ("NIMUIOCTL Failed with error code: %d\n",ret_code);
                return;
            }
        }
    
        /* Rx/Loopback loop */
        for(;;)
        {
            /* Attempt to recv a packet */
            i = recvnc(s, (void **)&pRecvPacket, 0x0, &pBufHandle);
    
            if( i >= 0 )
            {
                if (LOCAL_RawEchoOn)
                {
                    /* Swap source and destination MAC address */
                    memcpy((void *)&pRecvPacket[0],(void *)&pRecvPacket[6],sizeof(macAddress));
    
                    memcpy((void *)&pRecvPacket[6], (void *)macAddress, sizeof(macAddress));
    
                    send(s, (void *)pRecvPacket, i, 0x0);
                }
                LOCAL_rawPacketCount++;
                recvncfree(pBufHandle);
            }
            else
            {
                break;
            }
        }
    }
    
    void netOpenHook(void)
    {
        printf("[Test server]: netOpenHook!\n");
    
        /* Get GMACSW hardware driver handle */
        hGMACSW = GMACSW_open(NULL);
    
        /* Clear HW Stats */
        LOCAL_clearHWStats();
    
        /* Clear SW Stats */
        LOCAL_clearSWStats();
    
        /* Clear NDK IP stack stats */
        LOCAL_clearNDKIpStats();
    
        /* Set the control port daemon to be normal priority */
        hTestServer = DaemonNew( SOCK_STREAMNC, 0, TEST_CMD_PORT, LOCAL_commandServer,
                               OS_TASKPRINORM, OS_TASKSTKNORM, 0, 1 );
    
        /* Set the traffic test daemons to be higher priority */
        hTCPEcho = DaemonNew( SOCK_STREAMNC, 0, TEST_TCP_PORT, my_tcp_echo,
                           OS_TASKPRIHIGH, OS_TASKSTKNORM, 0, 3 );
    
        hUDPEcho = DaemonNew( SOCK_DGRAM, 0, TEST_UDP_PORT, my_udp_echo,
                           OS_TASKPRIHIGH, OS_TASKSTKNORM, 0, 1 );
    
        /* NDK Raw socket Rx task */
        {
            Task_Params taskParams;
            Task_Params_init(&taskParams);
            taskParams.instance->name = "Raw_Receiver";
            taskParams.stackSize = 0x1000;
            taskParams.priority = 1;
            hRawEchoTask = Task_create((Task_FuncPtr)my_raw_echo_task, &taskParams,NULL);
        }
    }
    
    void netCloseHook(void)
    {
        printf("[Test server]: netCloseHook!\n");
    
        Task_delete(&hRawEchoTask);
    
        DaemonFree(hUDPEcho);
        DaemonFree(hTCPEcho);
        DaemonFree(hTestServer);
    
        if (hGMACSW != NULL)
        {
            GMACSW_close(hGMACSW);
        }
    
        Semaphore_post(restartSem);
    }
    
    int main()
    {
        Semaphore_Params semParams;
        Task_Params taskParams;
    
        Semaphore_Params_init(&semParams);
        semParams.mode = Semaphore_Mode_COUNTING;
        restartSem = Semaphore_create(0, &semParams, NULL);
    
        Task_Params_init(&taskParams);
        taskParams.stackSize = 2048;
        taskParams.priority = 1;
        taskParams.instance->name = "NSP_Test_Server_RestartTask";
        Task_create(&LOCAL_restartTask, &taskParams, NULL);
    
        BIOS_start();
    
        return 0;
    }
    
    
    /*---------------------------------------------------------------------------*\
    |                           Local Function Definitions                        |
    \*---------------------------------------------------------------------------*/
    
    static bool LOCAL_needsAck(uint32_t cmdId)
    {
        bool retVal = false;
    
        switch(cmdId)
        {
            case TEST_CMD_RESTART_NSP:
            case TEST_CMD_START_NULL_STACK:
            case TEST_CMD_STOP_NULL_STACK:
            case TEST_CMD_GET_LOAD_INFO:
            case TEST_CMD_GET_HW_ERRORS:
            case TEST_CMD_GET_HW_STATS:
            case TEST_CMD_CLEAR_HW_STATS:
            case TEST_CMD_GET_SW_STATS:
            case TEST_CMD_CLEAR_SW_STATS:
            case TEST_CMD_GET_NDK_STATS:
            case TEST_CMD_CLEAR_NDK_STATS:
            case TEST_CMD_DISABLE_TCP_ECHO:
            case TEST_CMD_ENABLE_TCP_ECHO:
            case TEST_CMD_DISABLE_UDP_ECHO:
            case TEST_CMD_ENABLE_UDP_ECHO:
            case TEST_CMD_CPTS_REGISTER:
            case TEST_CMD_CPTS_UNREGISTER:
            case TEST_CMD_CPTS_FREQ_CHECK:
            case TEST_CMD_CPTS_SEND_ETH_TX:
            case TEST_CMD_GET_CPTS_EVENT:
                retVal = true;
                break;
            default:
                break;
        }
        return retVal;
    }
    
    static int LOCAL_handleCommand(SOCKET s, uint8_t *buf, uint32_t size)
    {
        uint32_t *pBuf = (uint32_t *)buf;
        uint32_t cmdId;
        int ret = SUCCESS;
    
        if (size < 4) return UNEXPECTED_CMD_SIZE;
    
        cmdId = ntohl(pBuf[0]);
    
        /*
         * Protocol is 32-bit command ID,  0 or more arguments depending on
         * command id. ACK is done by echoing the command ID back to the host.
         */
        if (LOCAL_needsAck(cmdId) == true)
        {
            int result = send(s, buf, size, 0);
            if (result < 0) return ACK_FAILURE;
            if (result != size) return SEND_SIZE_FAILURE;
        }
    
        switch(cmdId)
        {
            case TEST_CMD_RESTART_NSP:
                printf("[Test server]: TEST_CMD_RESTART_NSP!\n");
                /* Stop the network stack. */
                NC_NetStop(0);
                ret = RESTART;
                break;
            case TEST_CMD_START_NULL_STACK:
                hNullStack = NULL_open();
                if (NULL == hNullStack)
                {
                    ret = FAILURE;
                }
                else
                {
                    ret = SUCCESS;
                }
                break;
            case TEST_CMD_STOP_NULL_STACK:
                NULL_close(hNullStack);
                hNullStack = NULL;
                ret = SUCCESS;
                break;
            case TEST_CMD_GET_LOAD_INFO:
                ret = LOCAL_sendLoadInfo(s);
                break;
            case TEST_CMD_GET_HW_ERRORS:
                ret = LOCAL_sendHWErrors(s);
                break;
            case TEST_CMD_GET_HW_STATS:
                ret = LOCAL_sendHWStats(s);
                break;
            case TEST_CMD_CLEAR_HW_STATS:
                ret = LOCAL_clearHWStats();
                break;
            case TEST_CMD_GET_SW_STATS:
                ret = LOCAL_sendSWStats(s);
                break;
            case TEST_CMD_CLEAR_SW_STATS:
                ret = LOCAL_clearSWStats();
                break;
            case TEST_CMD_GET_NDK_STATS:
                ret = LOCAL_sendNDKIpStats(s);
                break;
            case TEST_CMD_CLEAR_NDK_STATS:
                ret = LOCAL_clearNDKIpStats();
                break;
            case TEST_CMD_DISABLE_TCP_ECHO:
                LOCAL_TCPEchoOn = false;
                ret = SUCCESS;
                break;
            case TEST_CMD_ENABLE_TCP_ECHO:
                LOCAL_TCPEchoOn = true;
                ret = SUCCESS;
                break;
            case TEST_CMD_DISABLE_UDP_ECHO:
                LOCAL_UDPEchoOn = false;
                ret = SUCCESS;
                break;
            case TEST_CMD_ENABLE_UDP_ECHO:
                LOCAL_UDPEchoOn = true;
                ret = SUCCESS;
                break;
            case TEST_CMD_CPTS_REGISTER:
            {
                CPTS_StackConfig  cptsStackConfig;
                uint32_t status;
    
                cptsStackConfig.eventMem                = gCPTSEvents;
                cptsStackConfig.eventCount              = 16;
                cptsStackConfig.vlanType                = CPTS_VLAN_TYPE_NONE;
                cptsStackConfig.portSelection           = CPTS_PORT_SELECTION_MAC_BOTH;
                cptsStackConfig.pEventNotifyCallback    = &LOCAL_cptsEventNotify;
                cptsStackConfig.hEventNotifyCallbackArg = (void *) &LOCAL_cptsEventCount;
    
                /* ToDo: Check return value to make sure it worked */
                status = GMACSW_ioctl(hGMACSW,
                            GMACSW_IOCTL_CPTS_REGISTER_STACK,
                            (void *)&cptsStackConfig,
                            sizeof(CPTS_StackConfig));
    
                ret = (status == CPTS_SUCCESS) ? SUCCESS : FAILURE;
                LOCAL_CPTSHandlerRegistered = (status == CPTS_SUCCESS) ?  true : false;
    
    
                break;
            }
            case TEST_CMD_CPTS_UNREGISTER:
            {
                uint32_t status;
    
                /* Unregister the CPTS notify callback */
                status = GMACSW_ioctl(hGMACSW,
                            GMACSW_IOCTL_CPTS_UNREGISTER_STACK,
                            NULL,
                            0U);
    
                LOCAL_CPTSHandlerRegistered = false;
    
                ret = (status == CPTS_SUCCESS) ? SUCCESS : FAILURE;
                break;
            }
            case TEST_CMD_CPTS_FREQ_CHECK:
            {
                ret = LOCAL_checkCPTSFreq(s);
                break;
            }
            case TEST_CMD_GET_CPTS_EVENT:
            {
                ret = LOCAL_sendCPTSEvent(s);
                break;
            }
            case TEST_CMD_CPTS_SEND_ETH_TX:
            {
                ret = LOCAL_sendTxCPTSFrames();
                break;
            }
            default:
                printf("[Test server]: Received unknown command = 0x%8X!\n", (unsigned int)cmdId);
                ret = FAILURE;
                break;
        }
        return ret;
    }
    
    static int LOCAL_commandServer(SOCKET s, UINT32 unused)
    {
        struct timeval to;
        int ret, recv_size;
        uint8_t *pBuf;
        HANDLE hBuffer;
        const int socketOptVal = 1;
    
        /* Configure our socket timeout to be 5 seconds */
        to.tv_sec = 5;
        to.tv_usec = 0;
        ret = setsockopt(s, SOL_SOCKET, SO_SNDTIMEO, &to, sizeof(to));
        if (ret < 0) {
            printf("[Test server]: Unable to configure send timeout!\n");
            goto teardown;
        }
    
        ret = setsockopt(s, IPPROTO_TCP, TCP_NOPUSH, (void *)&socketOptVal, sizeof(socketOptVal));
        if (ret < 0) {
            printf("[Test server]: Unable to configure NOPUSH option for TCP!\n");
            goto teardown;
        }
    
        while (true)
        {
            pBuf = NULL;
            hBuffer = NULL;
            recv_size = recvnc(s, (void **)&pBuf, 0, &hBuffer);
            if (recv_size < 0) {
                printf("[Test server]: receive failed!\n");
                ret = recv_size;
                break;
            }
            if (recv_size == 0) {
                printf("[Test server]: server connection closed!\n");
                ret = FAILURE;
                break;
            }
    
            if (pBuf == NULL || hBuffer == NULL) {
                printf("[Test server]: no valid buffer pointer or handle!\n");
                ret = FAILURE;
                break;
            }
    
            ret = LOCAL_handleCommand(s, pBuf, recv_size);
            recvncfree(hBuffer);
            if (ret == RESTART)
            {
                goto teardown;
            }
            else if (ret != SUCCESS)
            {
                if (ret == ACK_FAILURE)
                {
                    printf("[Test server]: command acknowledgement failed!\n");
                }
                else
                {
                    printf("[Test server]: error handling command (ret: %d)!\n", ret);
                }
            }
        }
    
    teardown:
        fdClose(s);
        printf("[Test server]: exiting test server daemon!\n");
        return ret;
    }
    
    static void LOCAL_restartTask(UArg arg0, UArg arg1)
    {
        while (1)
        {
            Semaphore_pend(restartSem, BIOS_WAIT_FOREVER);
    
            printf("[Test server]: restarting the NDK/NSP!\n");
    
            /* NDK Stack will be started. */
            {
                Task_Handle hNetStackTask = NULL;
                Task_Params taskParams;
    
                Task_Params_init(&taskParams);
                taskParams.instance->name = "ti_ndk_config_Global_stackThread";
                taskParams.priority = 0x5;
                taskParams.arg0 = 0u;
                taskParams.arg1 = 0u;
                taskParams.stackSize = (SizeT) 0x2000;
    
                hNetStackTask = Task_create(&ti_ndk_config_Global_stackThread, &taskParams, NULL);
                if (hNetStackTask == NULL)
                {
                    printf("[Test server]: restarting ti_ndk_config_Global_stackThread failed\n");
                    break;
                }
                else
                {
                    /* Register with NDK. Only then socket open will succeed.*/
                    fdOpenSession(hNetStackTask);
                }
            }
        }
    }
    
    static int LOCAL_sendLoadInfo(SOCKET s)
    {
        Load_Stat stat;
        uint32_t buf[6];
        int ret;
    
        ret = Load_getGlobalHwiLoad(&stat);
        if (ret == 0) return UNABLE_TO_OBTAIN_LOAD;
    
        buf[0] = htonl(Load_getCPULoad());
        buf[1] = htonl(Load_calculateLoad(&stat));
        /* TODO: Get actual ndk2nsp task loads here */
        //buf[2] = htonl(NDK2NSP_ioctl(hTfdtp->hNDK2NSP, NDK2NSP_IOCTL_GET_RXTASK_LOAD, 0, 0));
        //buf[3] = htonl(NDK2NSP_ioctl(hTfdtp->hNDK2NSP, NDK2NSP_IOCTL_GET_TXTASK_LOAD, 0, 0));
        buf[2] = htonl(0);
        buf[3] = htonl(0);
        if (NULL != hNullStack)
        {
            buf[4] = htonl(NULL_getRxLoad(hNullStack));
            buf[5] = htonl(NULL_getTxLoad(hNullStack));
        }
        else
        {
            buf[4] = htonl(0);
            buf[5] = htonl(0);
        }
    
        ret = send(s, &buf, sizeof(buf), 0);
    
        if (ret < 0) return ret;
        if (ret != sizeof(buf)) return SEND_SIZE_FAILURE;
        return SUCCESS;
    }
    
    static int LOCAL_sendHWErrors(SOCKET s)
    {
        int ret;
        STATS_Statistics stats;
        uint32_t buf[9];
    
        if (hGMACSW == NULL) return NULL_POINTER;
    
        GMACSW_ioctl(hGMACSW,
                     GMACSW_IOCTL_STATS_GET_STATS,
                     (void *)&stats,
                     sizeof(STATS_Statistics));
    
        buf[0] = htonl((uint32_t)stats.RxSOFOverruns);
        buf[1] = htonl((uint32_t)stats.RxMOFOverruns);
        buf[2] = htonl((uint32_t)stats.RxDMAOverruns);
        buf[3] = htonl((uint32_t)stats.RxCRCErrors);
        buf[4] = htonl((uint32_t)stats.RxAlignCodeErrors);
        buf[5] = htonl((uint32_t)stats.RxOversized);
        buf[6] = htonl((uint32_t)stats.RxJabber);
        buf[7] = htonl((uint32_t)stats.RxUndersized);
        buf[8] = htonl((uint32_t)stats.RxFragments);
    
        ret = send(s, &buf, sizeof(buf), 0);
        if (ret < 0) return ret;
        if (ret != sizeof(buf)) return SEND_SIZE_FAILURE;
    
        return SUCCESS;
    }
    
    static int LOCAL_sendHWStats(SOCKET s)
    {
        int ret;
        STATS_Statistics stats;
        uint32_t buf[10];
    
        if (hGMACSW == NULL) return NULL_POINTER;
    
        GMACSW_ioctl(hGMACSW,
                     GMACSW_IOCTL_STATS_GET_STATS,
                     (void *)&stats,
                     sizeof(STATS_Statistics));
    
        buf[0] = htonl((uint32_t)stats.RxGoodFrames);
        buf[1] = htonl((uint32_t)stats.RxBCastFrames);
        buf[2] = htonl((uint32_t)stats.RxMCastFrames);
        buf[3] = htonl((uint32_t)stats.RxPauseFrames);
        buf[4] = htonl((uint32_t)stats.RxOctets);
        buf[5] = htonl((uint32_t)stats.TxGoodFrames);
        buf[6] = htonl((uint32_t)stats.TxBCastFrames);
        buf[7] = htonl((uint32_t)stats.TxMCastFrames);
        buf[8] = htonl((uint32_t)stats.TxPauseFrames);
        buf[9] = htonl((uint32_t)stats.TxOctets);
    
        ret = send(s, &buf, sizeof(buf), 0);
        if (ret < 0) return ret;
        if (ret != sizeof(buf)) return SEND_SIZE_FAILURE;
    
        return SUCCESS;
    }
    
    static int LOCAL_clearHWStats(void)
    {
        GMACSW_ioctl(
            hGMACSW,
            GMACSW_IOCTL_STATS_CLEAR_STATS,
            NULL,
            0U);
    
        return SUCCESS;
    }
    
    static int LOCAL_sendSWStats(SOCKET s)
    {
        int ret;
        uint32_t buf[4];
    
        buf[0] = htonl((uint32_t)LOCAL_tcpPacketCount);
        buf[1] = htonl((uint32_t)LOCAL_udpPacketCount);
    
        if (NULL != hNullStack)
        {
            NULL_Stats stats;
            NULL_getStatistics(hNullStack, &stats);
            buf[2] = htonl(stats.packetCnt);
        }
        else
        {
            buf[2] = htonl((uint32_t)LOCAL_rawPacketCount);
        }
    
        buf[3] = htonl(LOCAL_cptsEventCount);
    
        ret = send(s, &buf, sizeof(buf), 0);
        if (ret < 0) return ret;
        if (ret != sizeof(buf)) return SEND_SIZE_FAILURE;
    
        return SUCCESS;
    }
    
    static int LOCAL_clearSWStats(void)
    {
        LOCAL_tcpPacketCount = 0;
        LOCAL_udpPacketCount = 0;
    
        if (NULL != hNullStack)
        {
            NULL_clearStatistics(hNullStack);
        }
        else
        {
            LOCAL_rawPacketCount = 0;
        }
    
        if (LOCAL_CPTSHandlerRegistered)
        {
            GMACSW_ioctl(
                hGMACSW,
                GMACSW_IOCTL_CPTS_CLEAR_ALL_EVENTS,
                NULL,
                0U);
        }
        LOCAL_cptsEventCount = 0;
    
        return SUCCESS;
    }
    
    static int LOCAL_sendNDKIpStats(SOCKET s)
    {
        int ret;
        uint32_t buf[20];
    
        /* UDP Tx Stats */
        buf[0]  = htonl((uint32_t)udps.RcvTotal);
        buf[1]  = htonl((uint32_t)udps.RcvShort);
        buf[2]  = htonl((uint32_t)udps.RcvBadLen);
        buf[3]  = htonl((uint32_t)udps.RcvBadSum);
        buf[4]  = htonl((uint32_t)udps.RcvFull);
        buf[5]  = htonl((uint32_t)udps.RcvNoPort);
        buf[6]  = htonl((uint32_t)udps.RcvNoPortB);
    
        /* TCP Tx Stats */
        buf[7]  = htonl((uint32_t)tcps.RcvTotal);
        buf[8]  = htonl((uint32_t)tcps.RcvShort);
        buf[9]  = htonl((uint32_t)tcps.RcvHdrSize);
        buf[10] = htonl((uint32_t)tcps.RcvBadSum);
    
        /* IP stats */
        buf[11] = htonl((uint32_t)ips.Total);
        buf[12] = htonl((uint32_t)ips.Odropped);
        buf[13] = htonl((uint32_t)ips.Badsum);
        buf[14] = htonl((uint32_t)ips.Badhlen);
        buf[15] = htonl((uint32_t)ips.Badlen);
        buf[16] = htonl((uint32_t)ips.Badoptions);
        buf[17] = htonl((uint32_t)ips.Badvers);
        buf[18] = htonl((uint32_t)ips.Noproto);
        buf[19] = htonl((uint32_t)ips.Delivered);
    
        ret = send(s, &buf, sizeof(buf), 0);
        if (ret < 0) return ret;
        if (ret != sizeof(buf)) return SEND_SIZE_FAILURE;
    
        return SUCCESS;
    }
    
    static int LOCAL_clearNDKIpStats(void)
    {
        mmZeroInit( &udps, sizeof( UDPSTATS ) );
        mmZeroInit( &tcps, sizeof( TCPSTATS ) );
        mmZeroInit( &ips, sizeof( IPSTATS ) );
    
        return SUCCESS;
    }
    
    static void LOCAL_cptsEventNotify(void *hCallbackArg)
    {
        uint32_t *pCounter = (uint32_t *) hCallbackArg;
    
        uint32_t key = Hwi_disable();
    
        *pCounter = *pCounter + 1;
    
        Hwi_restore(key);
    }
    
    static int LOCAL_sendCPTSFreqs(SOCKET s, uint32_t measured, uint32_t reported)
    {
        int ret;
        uint32_t buf[2];
    
        buf[0] = htonl(measured);
        buf[1] = htonl(reported);
    
        ret = send(s, &buf, sizeof(buf), 0);
        return (ret < 0 ) ? ret : 0;
    }
    
    static int LOCAL_sendCPTSEvent(SOCKET s)
    {
        int ret = FAILURE;
        uint32_t status;
        CPTS_Event *pEvent;
        uint32_t buf[5];
    
         /* Get CPTS events from GMACSW driver */
        status = GMACSW_ioctl(
            hGMACSW,
            GMACSW_IOCTL_CPTS_GET_EVENT,
            (void *) &pEvent,
            sizeof(CPTS_Event *));
    
        if (CPTS_SUCCESS == status)
        {
            uint32_t tsVal;
    
            {
                uint32_t key = Hwi_disable();
                LOCAL_cptsEventCount--;
                Hwi_restore(key);
            }
            buf[0] = htonl(pEvent->eventType);
            buf[1] = htonl(pEvent->messageType);
            buf[2] = htonl(pEvent->sequenceId);
            tsVal = (uint32_t)((pEvent->timeStamp >> 32) & 0xFFFFFFFFU);
            buf[3] = htonl(tsVal);
            tsVal = (uint32_t)(pEvent->timeStamp & 0xFFFFFFFFU);
            buf[4] = htonl(tsVal);
    
            /* Release the event back to the driver */
            GMACSW_ioctl(
                hGMACSW,
                GMACSW_IOCTL_CPTS_RELEASE_EVENT,
                pEvent,
                sizeof(CPTS_Event *));
    
            ret = send(s, &buf, sizeof(buf), 0);
            return (ret < 0 ) ? ret : SUCCESS;
        }
    
        return ret;
    }
    
    static int LOCAL_sendTxCPTSFrames(void)
    {
        SOCKET s = INVALID_SOCKET;
        uint8_t srcMacAddress[6];
        uint8_t destMacAddress[6];
    
        HANDLE pBufHandle = NULL;
        uint8_t *pRecvPacket = NULL;
        int32_t i;
    
        int16_t sequenceID = 0;
    
        /* Allocate the file environment for this task */
        fdOpenSession( TaskSelf() );
    
        /* Open the raw socket which gets used for both Tx and Rx */
        s = socket(AF_RAWETH, SOCK_RAWETH, CPTS_ETHERTYPE);
    
        if( s == INVALID_SOCKET )
        {
            printf("Fail socket, %d\n", fdError());
            fdCloseSession (Task_self());
            return FAILURE;
        }
    
        /* Set some socket options */
        {
            int ifVal = 1;
            int retVal = 0;
    
            /* Set socket option to set IF device */
            retVal = setsockopt(s, SOL_SOCKET, SO_IFDEVICE, &ifVal, sizeof(ifVal));
            if (retVal)
            {
                printf("Error in setsockopt() API.\n");
            }
        }
    
        /* Get the interface MAC address using the NIMU ioctl()*/
        {
            int ret_code;
            uint16_t        dev_index;
            NIMU_IF_REQ     if_req;
    
            if_req.index = 1;
            if (NIMUIoctl (NIMU_GET_DEVICE_INDEX, &if_req, &dev_index, sizeof(dev_index)) < 0)
            {
                printf ("Error: Incorrect device index specified\n");
                exit(1);
            }
    
            if_req.name[0] = 0;
            if_req.index   = dev_index;
    
            /* Get the device MAC Address */
            ret_code = NIMUIoctl (NIMU_GET_DEVICE_MAC, &if_req, &srcMacAddress, sizeof(srcMacAddress));
            if (ret_code < 0)
            {
                printf ("NIMUIOCTL Failed with error code: %d\n",ret_code);
                return FAILURE;
            }
        }
    
        /* Attempt to recv a packet to get destination MAC and indicate readiness */
        i = recvnc(s, (void **)&pRecvPacket, 0x0, &pBufHandle);
        if( i >= 0 )
        {
            /* Swap source and destination MAC address */
            memcpy((void *)destMacAddress,(void *)&pRecvPacket[6],sizeof(destMacAddress));
    
            /* Setup the source and destination MAC addresses */
            memcpy((void *)&pRecvPacket[0], (void *)destMacAddress, sizeof(destMacAddress));
            memcpy((void *)&pRecvPacket[6], (void *)srcMacAddress, sizeof(srcMacAddress));
    
            /* Don't need to set Ethertype, it should be there already from what we received */
    
            /* Set message type and the sequence number*/
            pRecvPacket[14] = (0x10U | CPTS_MESSAGE_SYNC);
            *((int16_t *)(pRecvPacket + 44)) = htons(sequenceID);
            sequenceID++;
    
            /* Send the packet */
            send(s, (void *)pRecvPacket, 64, 0x0);
    
            Task_sleep(10);
    
            /* Set message type and the sequence number */
            pRecvPacket[14] = (0x10U | CPTS_MESSAGE_DELAY_REQ);
            *((int16_t *)(pRecvPacket + 44)) = htons(sequenceID);
            sequenceID++;
    
            /* Send the packet */
            send(s, (void *)pRecvPacket, 64, 0x0);
    
            Task_sleep(10);
    
            /* Set message type and the sequence number */
            pRecvPacket[14] = (0x10U | CPTS_MESSAGE_PDELAY_REQ);
            *((int16_t *)(pRecvPacket + 44)) = htons(sequenceID);
            sequenceID++;
    
            /* Send the packet */
            send(s, (void *)pRecvPacket, 64, 0x0);
    
            Task_sleep(10);
    
            /* Set message type and the sequence number */
            pRecvPacket[14] = (0x10U | CPTS_MESSAGE_PDELAY_RESP);
            *((int16_t *)(pRecvPacket + 44)) = htons(sequenceID);
            sequenceID++;
    
            /* Send the packet */
            send(s, (void *)pRecvPacket, 64, 0x0);
    
            Task_sleep(10);
    
            /* Set message type and the sequence number */
            pRecvPacket[14] = (0x10U | CPTS_MESSAGE_FOLLOW_UP);
            *((int16_t *)(pRecvPacket + 44)) = htons(sequenceID);
            sequenceID++;
    
            /* Send the packet */
            send(s, (void *)pRecvPacket, 64, 0x0);
    
            Task_sleep(10);
    
            /* Set message type and the sequence number */
            pRecvPacket[14] = (0x10U | CPTS_MESSAGE_DELAY_RESP);
            *((int16_t *)(pRecvPacket + 44)) = htons(sequenceID);
            sequenceID++;
    
            /* Send the packet */
            send(s, (void *)pRecvPacket, 64, 0x0);
    
            Task_sleep(10);
    
            /* Set message type and the sequence number */
            pRecvPacket[14] = (0x10U | CPTS_MESSAGE_PDELAY_RESP_FOLLOW_UP);
            *((int16_t *)(pRecvPacket + 44)) = htons(sequenceID);
            sequenceID++;
    
            /* Send the packet */
            send(s, (void *)pRecvPacket, 64, 0x0);
    
            Task_sleep(10);
    
            /* Set message type and the sequence number */
            pRecvPacket[14] = (0x10U | CPTS_MESSAGE_ANNOUNCE);
            *((int16_t *)(pRecvPacket + 44)) = htons(sequenceID);
            sequenceID++;
    
            /* Send the packet */
            send(s, (void *)pRecvPacket, 64, 0x0);
    
            Task_sleep(10);
    
            /* Set message type and the sequence number */
            pRecvPacket[14] = (0x10U | CPTS_MESSAGE_SIGNALING);
            *((int16_t *)(pRecvPacket + 44)) = htons(sequenceID);
            sequenceID++;
    
            /* Send the packet */
            send(s, (void *)pRecvPacket, 64, 0x0);
    
            Task_sleep(10);
    
            /* Set message type and the sequence number */
            pRecvPacket[14] = (0x10U | CPTS_MESSAGE_MANAGEMENT);
            *((int16_t *)(pRecvPacket + 44)) = htons(sequenceID);
            sequenceID++;
    
            /* Send the packet */
            send(s, (void *)pRecvPacket, 64, 0x0);
    
            Task_sleep(10);
    
            recvncfree(pBufHandle);
        }
        else
        {
            return FAILURE;
        }
    
        fdClose(s);
    
        return SUCCESS;
    }
    
    static int LOCAL_checkCPTSFreq(SOCKET s)
    {
        uint64_t ts1, ts2;
        uint32_t measuredFreq;
        uint32_t reportedFreq;
        uint32_t status;
        uint32_t currEventCount;
    
        CPTS_Event eventTemplate;
        CPTS_Event *pEvent = NULL;
        CPTS_LookupEventIoctlCmd cmd;
    
        eventTemplate.eventType = CPTS_EVENT_TIME_STAMP_PUSH;
    
        cmd.pEventTemplate = &eventTemplate;
        cmd.pReturnEvent = &pEvent;
    
        /* First issue a TS push and wait for it to get noticed */
        currEventCount = LOCAL_cptsEventCount;
        status = GMACSW_ioctl(hGMACSW,
                     GMACSW_IOCTL_CPTS_TIMESTAMP_PUSH,
                     NULL,
                     0U);
        if (CPTS_SUCCESS == status)
        {
            while (LOCAL_cptsEventCount == currEventCount);
        }
        else
        {
            return FAILURE;
        }
    
    
    
        /* Get CPTS events from GMACSW driver */
        status = GMACSW_ioctl(hGMACSW,
                     GMACSW_IOCTL_CPTS_LOOKUP_EVENT,
                     (void *) &cmd,
                     sizeof(CPTS_LookupEventIoctlCmd));
        if (CPTS_SUCCESS == status)
        {
            {
                uint32_t key = Hwi_disable();
                LOCAL_cptsEventCount--;
                Hwi_restore(key);
            }
            ts1 = pEvent->timeStamp;
    
            /* Release the CPTS event back to the GMACSW driver */
            GMACSW_ioctl(hGMACSW,
                         GMACSW_IOCTL_CPTS_RELEASE_EVENT,
                         (void *) pEvent,
                         sizeof(CPTS_Event *));
    
            /* Sleep 1 s */
            Task_sleep(1000);
    
            /* Issue another TS push */
            currEventCount = LOCAL_cptsEventCount;
            status = GMACSW_ioctl(hGMACSW,
                         GMACSW_IOCTL_CPTS_TIMESTAMP_PUSH,
                         NULL,
                         0U);
            if (CPTS_SUCCESS == status)
            {
                /* Wait for push event to get noticed */
                while (LOCAL_cptsEventCount == currEventCount);
            }
            else
            {
                return FAILURE;
            }
    
            /* Get CPTS events from GMACSW driver */
            status = GMACSW_ioctl(hGMACSW,
                        GMACSW_IOCTL_CPTS_LOOKUP_EVENT,
                        (void *) &cmd,
                        sizeof(CPTS_LookupEventIoctlCmd));
    
            if (CPTS_SUCCESS == status)
            {
                LOCAL_cptsEventCount--;
                ts2 = pEvent->timeStamp;
    
                /* Release the CPTS event back to the GMACSW driver */
                GMACSW_ioctl(hGMACSW,
                             GMACSW_IOCTL_CPTS_RELEASE_EVENT,
                             (void *) pEvent,
                             sizeof(CPTS_Event *));
    
                /* Calculate frequency based on timestamp measurements */
                measuredFreq = (uint32_t) (ts2-ts1);
    
                GMACSW_ioctl(hGMACSW,
                             GMACSW_IOCTL_CPTS_GET_FREQUENCY,
                             (void *) &reportedFreq,
                             sizeof(reportedFreq));
                return LOCAL_sendCPTSFreqs(s, measuredFreq, reportedFreq);
            }
            else
            {
                return FAILURE;
            }
        }
        else
        {
            return FAILURE;
        }
    
        return SUCCESS;
    }
    
    /*---------------------------------------------------------------------------*\
    |                                 End of File                                 |
    \*---------------------------------------------------------------------------*/
    

    Regards,
    Stanley