This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMS320F28388D: TCP communication work slow

Other Parts Discussed in Thread: TMS320F28388D

I write application that run on the M4 CPU, the code is based on the project example tcpEchoF2838X.

I send and receive data from some PC application in a loop in a rate of ~40Kbyte/sec( transmit and received),

However the problem I face is that  each few seconds communication stop for about ~300msec ( device not response to last transmit message with delay of ~300msec) and after that it continue ok.

If I slow the communication rate to about ~15Kbye/sec then this phenomenon happen less frequently .

This phenomenon limit our communication average speed which we want to be high.

 

Is someone have idea way this happen and how can we overcome this ?

 

Avraham

  • Hi,

    avraham levkovizh said:
    the code is based on the project example tcpEchoF2838X.

    Is it simple echo back or you are doing something else in your application?

    avraham levkovizh said:
    Is someone have idea way this happen and how can we overcome this ?

    Is it possible to share a wireshark trace showing the issue captured? 

    Regards,

    Sudharsanan

  • Hi Sudharsanan.

    Here is  wireshark capture.

    IP = 10.50.1.59:50296     PC application

    IP = 10.50.1.229:7500    Application run on TMS320F28388D

     

    Packet 14670 send at time 4.624433 second  (17 bytes data)

    Packet 14671 send at time 4.928869 second (~300msec after), this packet retransmit the previous 17 bytes  that probably  was not acknowledged by the DSP.

    It seem that DSP not received/lost packet 1470 

    Avraham

  • Hi Avraham,

    Which NDK version are you using?

    Depending on your NDK version there is a variable called tcps or NDK_tcps (<ndk>/source/ti/ndk/inc/tcpif.h). There is also a structure called IPSTATS (defined in <ndk>/source/ti/ndk/inc/ipif.h).


    Could you place these structures in your CCS Expressions View and take a picture of their contents after the retransmission occurs? This will help isolate which layer the problem is occurring in and why the retransmission is happening.

    Thanks,

    Brandon

  • The NDK version cording to directory name is ndk_f2838x_3_61_01_01

    I copy the NDK_tcps to common memory with C1 and monitor it by our tools .

     

    The right table is when communicate directly from PC to DSP (breaks in communications is not exist)

    The left table is when communicate via our network.

    When working via network then every break we see that this counters  are incrementing:

    RcvDupByte, RcvDupPack , RcvOOByte, RcvOOPack .

     What does this mean ?

  • Hello,

    The RcvDupByte, RcvDupPack, RcvOOByte, RcvOOPack fields increment when there are duplicate transmissions such as the TCP Retransmission at No. 14671 in your Wireshark capture.

    The TCPSTATS you shared from the direct connection between the PC and the DSP looks like it was taken when much less data was transmitted. If you ran that connection longer and examined the TCPSTATS again, do you see any changes with RcvDupByte, RcvDupPack , RcvOOByte, RcvOOPack?

    Do you experience a similar delay in communication when you run the tcpEchoF2838X example? If not, is it possible for you to share your application code?

    Best,
    Brandon

  • The direct connection was done in more short time but it was same data rate and we didn't face communication breaks, anyway I will repeat this test in Sunday.

    I put some source code in dropbox link. 

    https://www.dropbox.com/s/g610ulkcgfvimep/tcpEchoF2838X.zip?dl=0

    The main code is in tcpecho.c .in tcpHandler It create one UDP socket for PC monitoring , create TCP server socket , call listen(server,1) and wait on accept for one only connection.

    The communication loop is in function udpWorkerRC_TR. It copy received data to  Nsrc_buf , and transmit data from  Nstr_buf ( buffers that are common with cpu1, which response to the recieved data in Nsrc_buf and fill Nstr_buf ).

    This function also work for UDP connection if pform pointer is  not NULL. for TCP assume pform=NULL.

  • I want to add that our application also support UDP connection, in this case udpWorkerRC_TR is invoked from udpWorkerRC_TR_start which run in separate thread triggered from udpWorkerBC which call openUdpSocket. (udpWorkerBC is run always in a thread that listen to UDP socket that received broadcast messages on port 30718, this use for PC monitoring which devices exist on the network ).

    The UDP communication loop also run in udpWorkerRC_TR (in this case pform is not NULL and point to PC destination address ).

    So when we communicate in UDP mode not directly then we face a lot of messages lost , when work directly there is almost no message lost.

    I think it come from the same root cause as we face in TCP, there are packets lost ( in TCP protocol packet lost make retransmit, in UDP there is no retransmit ).

     

    Avraham

     

  • Today I test the direct connection for more long time and there was almost not have breaks.

    However form time to time also in direct mode there were breaks but much less then connect via network, and when communicate via UDP then it seems to be even more less.

    Is it possible that communication breaks happen because the NDK sw running on M4 is disturb by extra messages ( such as ARP) which happen more frequently when connection via network?

     

    Avraham

  • Hi Avraham,

    It sounds like a possibility, or even the UDP communication that's happening in tandem. Thank you for running the direct connection test again. I'm curious if the breaks in communication during the direct connection test were the same length as the breaks you experience over the network.

    I'm trying to form an idea of where the driver or CPU may be running into trouble.
    Is the pause in communication a fairly consistent ~300ms? Can you share the full wireshark capture file from running your application over the network and/or from running your application via direct connection?

    What is the performance like when running the unmodified tcpEchoF2838X example?

    Thanks,
    Brandon

  • After connecting sniffer we found that one of the PC on the network run some application that was sending many ARP broadcast messages each 10-100msec.

    Then we stop this application and the communication breaks in TCP on network was decreased dramatically ( every about 10-30 second instead of 1-3 second before ), and in UDP communication there was no any message lost for long time ( so no breaks in UDP ).

     I still wonder way there are still breaks in TCP , in the statistic I see that RcvPartDupByte, RcvPartDupPack and sometime also RexmtTimeout is counting when break happen.

     In the wireshark capture we see that packet number 181436 (send form PC ) was miss to be acknowledge, so there is retransmit after ~300sec, packet 181437.

     Attach link to full wireshark capture file (also picture below on the relevant location):

    https://www.dropbox.com/s/o4fvgg4534morj4/tcp_fast.pcapng?dl=0

     

  • The excessive ARP packages may be overflowing the RX queue in the driver as you mentioned. To get a better idea can you do the following:

    There is an instance of the EMACF2838X_Object struct in the driver: 

    static EMACF2838X_Object emacData

    Can you inspect this variable while running your application and report back? In CCS you can put it in the Expressions View by entering 

    'EMACF2838X.c'::emacData

    The struct has some fields that aren't included in the TCPSTATS struct.

    The values of the following variables would also be helpful. You can copy and paste these in your CCS Expressions view as well, or the tool you have been using to inspect variables. To view the statistics provided by these variables the EMAC driver code needs to have been built with "ETHERNET_DEBUG" defined.

    //                                                                                                                                                                             
    //These variables are for debug purpose                                                                                                                                        
    //Number of Time the Tx Packet Free Callback was called                                                                                                                        
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_numTxPktFreeCallback = 0U;                                                                                                                              
    //                                                                                                                                                                             
    //Number of Times Rx Callback while handling RX packets                                                                                                                        
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_numRxCallback = 0U;                                                                                                                                     
    //                                                                                                                                                                             
    //Number of time Get packet buffer callback was called                                                                                                                         
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_numGetPacketBufferCallback = 0U;                                                                                                                        
    //                                                                                                                                                                             
    //Number of times Tx Completion ISR was called                                                                                                                                 
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_txInterruptCount = 0U;                                                                                                                                  
    //                                                                                                                                                                             
    //Number of times Rx Completion ISR was called                                                                                                                                 
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxInterruptCount = 0U;                                                                                                                                  
    //                                                                                                                                                                             
    //Number of Receive Context Descriptors received                                                                                                                               
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxContextTimeStamp = 0U;                                                                                                                                
    //                                                                                                                                                                             
    //Number of times Misc Interrupt ISR was called                                                                                                                                
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_genericInterruptCount = 0U;                                                                                                                             
    //                                                                                                                                                                             
    //Number of times Early Rx interrupt is received                                                                                                                               
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_earlyRxInterruptCount = 0U ;                                                                                                                            
    //                                                                                                                                                                             
    //Number of times Early Tx interrupt is received                                                                                                                               
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_earlyTxInterruptCount = 0U;                                                                                                                             
    //                                                                                                                                                                             
    //Number of times RX Generic interrupt is received for completion                                                                                                              
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_genericRxCompletionInterruptCount = 0U ; 
    //                                                                                                                                                                             
    //Number of PHY Status change interrupt from RevMII block                                                                                                                      
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_phyLinkStatusChangeInterruptCount = 0U;                                                                                                                 
    //                                                                                                                                                                             
    //Number of times application did not give replacement Rx buffer                                                                                                               
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxReplacementFailedCount = 0;                                                                                                                           
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // Number of times RX Queue capacity was replenished                                                                                                                           
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxReplishments = 0;                                                                                                                                     
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // Count of Receive Buffer Unavailable (RBU) for channel zero                                                                                                                  
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxChan0RxBufUnavail = 0;                                                                                                                                
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // Count of Transmit Buffer Unavailable (TBU) for channel zero                                                                                                                 
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxChan0TxBufUnavail = 0;                                                                                                                                
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // Number of times RX ISR ran for channel zero                                                                                                                                 
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxChan0Cnt = 0;                                                                                                                                         
    //                                                                                                                                                                             
    // Number of times RX ISR ran for channel one                                                                                                                                  
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxChan1Cnt = 0;                                                                                                                                         
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // Number of times TX ISR ran for channel zero                                                                                                                                 
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_txChan0Cnt = 0;                                                                                                                                         
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // Number of times TX ISR ran for channel one                                                                                                                                  
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_txChan1Cnt = 0;
    //                                                                                                                                                                             
    // Track number of times DMA still owned descriptor during RX dequeue                                                                                                          
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_dmaStillOwnsDesc = 0;                                                                                                                                   
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // Count of RX dequeue failures                                                                                                                                                
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxPktQueuePopFail = 0;                                                                                                                                  
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // Count of early RX frames                                                                                                                                                    
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxEarlyComplete = 0;                                                                                                                                    
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // Count of normal RX frames                                                                                                                                                   
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxNormComplete = 0;                                                                                                                                     
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // IP Header errors reported by RX Checksum Offload Engine                                                                                                                     
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxIpHdrChksmErrors = 0;                                                                                                                                 
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // IP payload errors reported by RX Checksum Offload Engine                                                                                                                    
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_rxPayloadChksmErrors = 0;                                                                                                                               
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // IP Header errors reported by TX Checksum Offload Engine                                                                                                                     
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_txIpHdrChksmErrors = 0;                                                                                                                                 
                                                                                                                                                                                   
    //                                                                                                                                                                             
    // IP payload errors reported by TX Checksum Offload Engine                                                                                                                    
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_txPayloadChksmErrors = 0;                                                                                                                               
                                                                                                                                                                                   
    //                                                                                                                                                                             
    //Number of EEE Tx LPI count
    uint32_t EMACF2838XLLD_eeeTxLPIEntryCount = 0;                                                                                                                                 
    //                                                                                                                                                                             
    //Number of EEE Tx LPI count                                                                                                                                                   
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_eeeTxLPIExitCount = 0;                                                                                                                                  
    //                                                                                                                                                                             
    //Number of EEE Rx LPI Count                                                                                                                                                   
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_eeeRxLPIEntryCount = 0;                                                                                                                                 
    //                                                                                                                                                                             
    //Number of EEE Tx LPI count                                                                                                                                                   
    //                                                                                                                                                                             
    uint32_t EMACF2838XLLD_eeeRxLPIExitCount = 0;                                                                                                                                  
    uint32_t EMACF2838XLLD_pmtInterruptCount = 0;                                                                                                                                  
    uint32_t EMACF2838XLLD_magicPktReceivedCount = 0;                                                                                                                              
    uint32_t EMACF2838XLLD_rwakeupPktReceivedCount = 0;  
    

    Thanks,
    Brandon

  • I watch the variable you suggested ( the EMACF2838XLLD_xxx was grouped to one structure for easy copy to it in common memory with CPU1 ).

    The left picture below is when exist an application on network that send intensively ARP broadcast messages ( I mark some of the counters that count a lot with red point)

    The right picture below is when this application stop running.

    Each test is about 10min

    Avraham

  • Hi Avraham,

    We are looking into possible fixes, working under the assumption that network congestion due to ARP broadcasts is the issue.

    The hardware can be configured to reply to ARP requests. The ARP requests will still be forwarded to the driver, but if you are okay with disabling broadcast packets, you can filter out ARP packets sent to the NDK. Again, note, this may affect other protocols that utilize broadcasts. 

    All of the below will require a rebuild of the driver.

    1. Disable broadcasts
      1. Edit the code of ti/ndk/drivers/f2838x/ethernet.c
      2. Go to EMACF2838XLLD_getInitConfig()
      3. Find the line that configures the broadcast filter (around line 975) and change it to ETHERNET_ENABLE
        configPtr->macFilterConfig.disableBroadCastPackets = ETHERNET_ENABLE;

      4. The above should result in the following register to now have bit 5 set (the 3in the line below)
        MAC_Packet_Filter    0x00000030    // previously 0x20
    2. Enable ARP offloading
      1. Update the call to EMACF2838XLLD_setMACConfiguration() in ethernet.c (around line 1970) with the highlighted code to enable ARP offload (do not forget the additional "|"on the previous line (line 8 in the below code)
           EMACF2838XLLD_setMACConfiguration(
                EMACF2838XLLD_device_struct.baseAddresses.eqos_base,
                (((uint32_t)(ethernetInitConfigPtr->linkMode <<
                 (uint32_t) ETHERNET_MAC_CONFIGURATION_DM_S)) |
                 ((uint32_t)(ethernetInitConfigPtr->loopbackMode <<
                  (uint32_t)ETHERNET_MAC_CONFIGURATION_LM_S)) |
                 ((uint32_t)(ethernetInitConfigPtr->ipRxCsOffloadMode <<
                  (uint32_t)ETHERNET_MAC_CONFIGURATION_IPCS_S)) |
                 ((uint32_t)(ETHERNET_MAC_CONFIGURATION_ARPEN << 0)))
                );
      2. The above should result in the following updated register
        MAC_Configuration    0x8800A003    // previously 0x0800A003
    3. Set the ARP offload IP address
      1. You need to set the value of the below register to the IP address of the device. A suitable place is the IP address hook function (found in tcpechohooks.c) that fires when an IP address is obtained from the DHCP server (or when an IP address is set statically).
      2. So, within the IP address hook or wherever you decide to place this (there may be other suitable call sites), call the HWREG()function to write the IP address to the MAC ARP address register. The IP address should be stored in network byte order: 
        HWREG(EMACF2838XLLD_device_struct.baseAddresses.eqos_base + ETHERNET_O_MAC_ARP_ADDRESS) = 0x2000A8C0 // 192.168.1.32 IP address in hex format, watch out for byte order issues
      3. For example, the above should result in the following updated register
        MAC_ARP_Address  0xC0A80020    // previously 0x0

    Please comment if you have further questions.

    Best,
    Brandon

  • Thanks Brandon

    I am not sure we want to disable broadcast packets, as in our implementation there is one UDP socket that listen to broadcast messages. also we want to have the DHCP option ( I think it support by ARP messages ).

     

    BTW the application that send many ARP messages seems to be abnormal behavior ( maybe virus ? ), so it is not a normal situation . I think this many ARP messages send also make the M4 crash sometime as I report in other Post:

    https://e2e.ti.com/support/microcontrollers/c2000/f/171/t/858217

     

    Avraham