This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMS320C6657: Retransmissions and CRC Errors when streaming small TCP Packets

Part Number: TMS320C6657

Hi,

We are using C6657 DSP with NDK Version :ndk_2_21_02_43, and encountered a problem, which needs clarification urgently,

since our customers ran into problems here! .if any help is appreciated.

 

Brief Description of the problem:

In our application we are continuously sending very small TCP Packets with 32 Byte to the DSP in 1ms intervals.

We encountered TCP retransmissions in about 10% of the packets, caused by a missing ACK from the NDK.

We used Wireshark for the analysis.

Since we implemented a HIL application, these retransmissions are a huge problem, because we cannot wait for the retransmission (~300ms retransmission timer).

 

Test Setup:

The test was conducted with direct connection between sender (PC) and receiver (DSP). No switches in between that could cause packet loss by collisions.

 

Observations:

When sending 32 Byte Packets, we see retransmissions.

We dug deeper into the NDK and saw CRC errors on the IP layer, but surprisingly no CRC errors on the physical layer (PHY and EMAC) => See attached statistics

We continuously increased the packet size to 320 Byte and the problem with the retransmissions suddenly disappeared.

Values between 32 Byte and 320 Byte had showed a continuous improvement, but CRC errors on the IP layer occurred nonetheless. No CRC errors on the physical layer (PHY and EMAC) like before

With a 320 Byte packet size there were more CRC errors on the IP layer and No CRC errors on the physical layer (PHY and EMAC) like before.

 

Just to clarify: We are running BER Tests on the electrical link on the MAC layer. The electrical link is fine.

Just to clarify: We ran the two tests for several hours with millions of packets. 320 Byte packets never caused a retransmission or CRC errors, while 32 Byte packets do in 10% of times.

 

Questions:

Is this a known issue?

Any ideas if we could fix that by a different configuration of the NDK?

Since we found a temporary workaround, is our approach to simply increase the packet size to 320 Byte a failsafe approach, that will work under all conditions?

 

Attachments:

Attached you find the statistics from the NDK info structures for both cases, 32 Byte (Bad case) and 320 Byte (Good Case)

Suspicious stats are marked.

 

Attachment #1: Bad case: 1000 Packets with 32 Bytes Payload:

TCP Statistics from TI NDK

    RcvTotal = 996            

    RcvShort = 0           

    RcvHdrSize = 0          

    RcvBadSum = 6           => CRC Errors

    RcvDupPack = 0         

    RcvDupByte = 0          

    RcvPartDupPack = 0      

    RcvPartDupByte = 0      

    RcvAfterClose = 0       

    RcvAfterWinPack = 0     

    RcvAfterWinByte = 0      

    RcvWinProbe = 0         

    RcvDupAck = 0           

    RcvAckTooMuch = 0       

    RcvAckPack = 1          

    RcvAckByte = 1          

    RcvWinUpd = 0           

    RcvPack = 986            

    RcvByte = 32000             

    RcvOOPack = 0           

    RcvOOByte = 0

 

IP Statistics from TI NDK : 

   Total = 1589            

    Odropped = 0            

    Badsum = 0          

    Badhlen = 0           

    Badlen = 8                            =Why? Because of CRC Errors?

    Badoptions = 0          

    Badvers = 0      

    Forward = 0      

    Noproto = 0       

    Delivered = 1520     

    Cantforward = 0      

    CantforwardBA = 61          

    Expired = 0           

    Redirectsent = 0       

    Localout = 990          

    Localnoroute = 0          

    Reassembled = 0           

    Fragments = 0             

    Fragdropped = 0             

    Fragtimeout = 0           

    Fragmented = 0

    Ofragments = 0       

    Cantfrag = 0          

    CacheHit = 0          

    CacheMiss = 1           

    Filtered = 0      

 

Attachment #2: Good case: 1000 Packets with 320 Bytes Payload:

TCP Statistics from TI NDK

    RcvTotal = 1003            

    RcvShort = 0           

    RcvHdrSize = 0          

    RcvBadSum = 0            => No CRC Errors

    RcvDupPack = 0         

    RcvDupByte = 0          

    RcvPartDupPack = 0      

    RcvPartDupByte = 0      

    RcvAfterClose = 0       

    RcvAfterWinPack = 0     

    RcvAfterWinByte = 0      

    RcvWinProbe = 0         

    RcvDupAck = 0           

    RcvAckTooMuch = 0       

    RcvAckPack = 1          

    RcvAckByte = 1          

    RcvWinUpd = 0           

    RcvPack = 1000             

    RcvByte = 320000             

    RcvOOPack = 0           

    RcvOOByte = 0

 

IP Statistics from TI NDK : 

   Total = 1436            

    Odropped = 0           

    Badsum = 0          

    Badhlen = 0           

    Badlen = 0                            => Fine now

    Badoptions = 0          

    Badvers = 0      

    Forward = 0      

    Noproto = 0       

    Delivered = 1410     

    Cantforward = 0      

    CantforwardBA = 26         

    Expired = 0           

    Redirectsent = 0       

    Localout = 1004          

    Localnoroute = 0          

    Reassembled = 0           

    Fragments = 0             

    Fragdropped = 0             

    Fragtimeout = 0           

    Fragmented = 0

    Ofragments = 0       

    Cantfrag = 0          

    CacheHit = 0          

    CacheMiss = 1           

    Filtered = 0     

 

  • Small correction:

    Hi,

    We are using C6657 DSP with NDK Version :ndk_2_21_02_43, and encountered a problem, which needs clarification urgently,

    since our customers ran into problems here! .if any help is appreciated.

     

    Brief Description of the problem:

    In our application we are continuously sending very small TCP Packets with 32 Byte to the DSP in 1ms intervals.

    We encountered TCP retransmissions in about 10% of the packets, caused by a missing ACK from the NDK.

    We used Wireshark for the analysis.

    Since we implemented a HIL application, these retransmissions are a huge problem, because we cannot wait for the retransmission (~300ms retransmission timer).

     

    Test Setup:

    The test was conducted with direct connection between sender (PC) and receiver (DSP). No switches in between that could cause packet loss by collisions.

     

    Observations:

    When sending 32 Byte Packets, we see retransmissions.

    We dug deeper into the NDK and saw CRC errors on the IP layer, but surprisingly no CRC errors on the physical layer (PHY and EMAC) => See attached statistics

    We continuously increased the packet size to 320 Byte and the problem with the retransmissions suddenly disappeared.

    Values between 32 Byte and 320 Byte had showed a continuous improvement, but CRC errors on the IP layer occurred nonetheless. No CRC errors on the physical layer (PHY and EMAC) like before

    With a 320 Byte packet size there were NO MORE CRC errors on the IP layer and No CRC errors on the physical layer (PHY and EMAC) like before.

     

    Just to clarify: We are running BER Tests on the electrical link on the MAC layer. The electrical link is fine.

    Just to clarify: We ran the two tests for several hours with millions of packets. 320 Byte packets never caused a retransmission or CRC errors, while 32 Byte packets do in 10% of times.

     

    Questions:

    Is this a known issue?

    Any ideas if we could fix that by a different configuration of the NDK?

    Since we found a temporary workaround, is our approach to simply increase the packet size to 320 Byte a failsafe approach, that will work under all conditions?

     

    Attachments:

    Attached you find the statistics from the NDK info structures for both cases, 32 Byte (Bad case) and 320 Byte (Good Case)

    Suspicious stats are marked.

     

    Attachment #1: Bad case: 1000 Packets with 32 Bytes Payload:

    TCP Statistics from TI NDK

        RcvTotal = 996            

        RcvShort = 0           

        RcvHdrSize = 0          

        RcvBadSum = 6           => CRC Errors

        RcvDupPack = 0         

        RcvDupByte = 0          

        RcvPartDupPack = 0      

        RcvPartDupByte = 0      

        RcvAfterClose = 0       

        RcvAfterWinPack = 0     

        RcvAfterWinByte = 0      

        RcvWinProbe = 0         

        RcvDupAck = 0           

        RcvAckTooMuch = 0       

        RcvAckPack = 1          

        RcvAckByte = 1          

        RcvWinUpd = 0           

        RcvPack = 986            

        RcvByte = 32000             

        RcvOOPack = 0           

        RcvOOByte = 0

     

    IP Statistics from TI NDK : 

       Total = 1589            

        Odropped = 0            

        Badsum = 0          

        Badhlen = 0           

        Badlen = 8                            =Why? Because of CRC Errors?

        Badoptions = 0          

        Badvers = 0      

        Forward = 0      

        Noproto = 0       

        Delivered = 1520     

        Cantforward = 0      

        CantforwardBA = 61          

        Expired = 0           

        Redirectsent = 0       

        Localout = 990          

        Localnoroute = 0          

        Reassembled = 0           

        Fragments = 0             

        Fragdropped = 0             

        Fragtimeout = 0           

        Fragmented = 0

        Ofragments = 0       

        Cantfrag = 0          

        CacheHit = 0          

        CacheMiss = 1           

        Filtered = 0      

     

    Attachment #2: Good case: 1000 Packets with 320 Bytes Payload:

    TCP Statistics from TI NDK

        RcvTotal = 1003            

        RcvShort = 0           

        RcvHdrSize = 0          

        RcvBadSum = 0            => No CRC Errors

        RcvDupPack = 0         

        RcvDupByte = 0          

        RcvPartDupPack = 0      

        RcvPartDupByte = 0      

        RcvAfterClose = 0       

        RcvAfterWinPack = 0     

        RcvAfterWinByte = 0      

        RcvWinProbe = 0         

        RcvDupAck = 0           

        RcvAckTooMuch = 0       

        RcvAckPack = 1          

        RcvAckByte = 1          

        RcvWinUpd = 0           

        RcvPack = 1000             

        RcvByte = 320000             

        RcvOOPack = 0           

        RcvOOByte = 0

     

    IP Statistics from TI NDK : 

       Total = 1436            

        Odropped = 0           

        Badsum = 0          

        Badhlen = 0           

        Badlen = 0                            => Fine now

        Badoptions = 0          

        Badvers = 0      

        Forward = 0      

        Noproto = 0       

        Delivered = 1410     

        Cantforward = 0      

        CantforwardBA = 26         

        Expired = 0           

        Redirectsent = 0       

        Localout = 1004          

        Localnoroute = 0          

        Reassembled = 0           

        Fragments = 0             

        Fragdropped = 0             

        Fragtimeout = 0           

        Fragmented = 0

        Ofragments = 0       

        Cantfrag = 0          

        CacheHit = 0          

        CacheMiss = 1           

        Filtered = 0     

     

    Thanks,

    Ram.

  • Hello Ram

    Thank you for the detailed message. 

    I am not aware of any known issues per your problem description. 

    However you seem to be using an older version of NDK, not sure if it is easy for you to try with the latest version of the PSDK and the NDK that comes with it etc. 

    At a high level It would be good to check if you are running into cache coherency issues. Short packets are fully received more quickly than large packet,  which  would mean that they are written out to memory more quickly from the time buffers are available.

    As an experiment you could either try to disable all data cache  to see if the checksum errors go away. 

    Regards

    Mukul 

  • Hi Mukul,

    thanks for the  fast reply. we  just digged out and see that we are using NDK_2_25_01_11. Do we have any other new versions compared to this? and also for PSDK.

    Can you explain more, how we can disable all data cache and also where we have to do this? with some coding example also helps us.

    Thanks,

    Best Regards,

    Ramana.

  • Hello

    Looks like you are on a pretty old version of the NDK. Per my understanding the latest NDK version is NDK 3.61.01.01, you can find that as part of the latest Processor SDK RTOS package

    https://www.ti.com/tool/PROCESSOR-SDK-C665X

    I am not sure how straightforward or easy it is for you to evaluate or incorporate this in your existing code base/ application – given you have been using this product and software for a few years now. However there is no way to confirm what all issues have been fixed over the years and in subsequent revisions, that could help your issues.

     

    In terms of disabling cache, it will depend on how this is being done in your application code. Typically in NDK, the cache needs to be disabled in BIOS. Here is an example to disable L1D data cache. Same should also apply to L2

     

    https://e2e.ti.com/support/legacy_forums/embedded/tirtos/f/355/t/234893?BIOS-C6670-L1D-Cache-disable

     

     

    You can also look at the cache user guide

    https://www.ti.com/lit/pdf/sprugy8

     

    The DSP cache can be disabled by programming the L2MODE in the L2CFG and L1Mode in the L1CFG registers.

    There are CSL APIs.

    CACHE_L1dSetSize();CACHE_L2SetSize();

    You will also need to disable relevant MAR bits for external memory caching

     

    Details from above document chapter 2.

     

    You will need to do this as the first instructions after reaching to main(). (Before accessing any caches) Also, to make sure nothing remains in Cache Line, it is good to WbInv the Cache before changing the cache to RAM (0K Cache size).

    Other helper resources

    https://www.ti.com/lit/ug/spru523k/spru523k.pdf

    https://e2e.ti.com/support/legacy_forums/embedded/tirtos/f/355/t/154576

    https://www.ti.com/lit/an/sprac57/sprac57.pdf