This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

  • Resolved

RTOS/AM4376: PRU-ICSS and PRU-EMAC sometimes drops packets

Intellectual 510 points

Replies: 49

Views: 1692

Part Number: AM4376

Tool/software: TI-RTOS

Hello,

We expirience rare problems with both PRP mode via PRU-ICSS and PRU_EMAC when PRU stops transferring packets for short period of time. I managed to identify that it caused by PRU by taking PRP statistics. Currently it happens on PRU-ICSS-HSR-PRP-DAN_01.00.03.02 package but I do not hope it is fixed in PRU-ICSS-HSR-PRP-DAN_01.00.04.02, though I will try to migrate my code.

The time is around 32.5ms.

We send packets with constant rate (sampled valued data) and this problem sometimes happens once per 10 seconds, sometimes once per half hour. But the the pause in RX (32.5ms) is more or less constant.

For me it looks like internal watchdog reboot of PRU or kind of DDoS protection, but I'm not sure and have no idea of how to proof that theory.

I'm now trying to migrate to 01.00.04.02 to check if fixed already, but I'd like to hear if it is known issue and if this time means something to TI developers.

Thanks, Alexander.

  • In reply to Alexander Aganichev:

    Alex,

    I did miss those patches as you mentioned that you removed most of unnecessary code. Will continue to try to reproduce the issue.
    Also I have forwarded your responses to those questions from team...

    Regards,
    Garrett

  • In reply to Garrett Ding:

    Alex,

    After applying the patches, I am just able to build and run your application with no more crash. I could assign a IP address to the device, but somehow it's not ping-able. Am I still missing anything? I haven't closely looked into your application. Does the application still need to be run with the test setup C as you described for reproducing the error stats?

    Regards,
    Garrett
  • In reply to Garrett Ding:

    Garrett,

    For sampled values we don't use IP, so I removed IP stack initialization.

    Here is C# code on top of PCAP which I used for testing. It works on my notebook Win7Pro x64 with Intel card, but caused BSOD on my desktop Win10Pro x64 with Realtek. That seems to be incompatibility with PCAP/Realtek drivers pair, though I'm not sure. The app is looking for interface with address 10.0.1.1 to send packets, you can change it in the code.

    The PC app is sending sampled values packets in group of 3 by default, each group of packet is numbered from 0 to 3999 within a roughly second. The small_main.c in the task taskLedBlink2 checks order of received packets and prints received (Curr) and expected (Exp) counter for the packet.

    Today I run it in few configurations:

    1) PC<-1G->Switch 1G<-100M->IDK, the problem reproduced

    2) PC<-100M->Switch 1G<-100M->IDK, the problem not reproduced but the Windows drops frame rate on unknown reason; there also was noted two big gaps in frames but the PRU statistics were clean, so I'm not sure what caused that

    3) PC<-1G->Switch 1G<-100M->Switch 100M<-100M->IDK, the problem reproduced

    I added printing of interval time in microseconds for which packets not received to small_main.c.

    ==

    *
    Curr=3802,Exp=3668,dT=33535
    *
    Curr=2715,Exp=2581,dT=33510
    *
    Curr=3566,Exp=3431,dT=33780
    *
    Curr=3221,Exp=3086,dT=33774
    *
    Curr=3629,Exp=3494,dT=33763
    *
    Curr=3737,Exp=3603,dT=33476

    ==

    small_main.c

    RawSocket.7z

  • In reply to Alexander Aganichev:

    I run same test with 4 SV simulation. The down time increased to 75ms (about 1200 errors block per failure).
    ==
    *
    Curr=846,Exp=545,dT=75217
    *
    Curr=2421,Exp=2119,dT=75474
    *
    Curr=2119,Exp=1818,dT=75192
    *
    Curr=680,Exp=380,dT=74942
    *
    Curr=1654,Exp=1352,dT=75477
    *
    Curr=1653,Exp=1351,dT=75478
    ==
  • In reply to Alexander Aganichev:

    And one more test result: 4SV via 100Mbit/s switch. Errors still happens (13 for the 2 cases) but very rare and for the shorter time:
    ==
    *
    Curr=2161,Exp=2158,dT=1207
    *
    Curr=1582,Exp=1580,dT=810
    ==
  • In reply to Alexander Aganichev:

    Alex,

    Can you please send me a binary file of your C# project? I don't use Microsoft visual studio for many years : -)

    Regards,
    Garrett
  • In reply to Garrett Ding:

    Garrett,

    Here it is. I noticed that with increasing number of streams the errors appearance increasing. The RawSocket.exe app took two arguments: first is the repeat timer (250us by default) and the second is number of streams. By calling "RawSocket.exe 250 8" you can simulate 8 SV streams and the problem will appear very quick. If I calcualted correctly it is about 33 Mbit/s load and I may expect random packet drop with 100 Mbit/s interface, but I don't understand why PRU stops receiving for the period of time.

    Though with 8 SV streams I don't see direct correlation between the number of streams and "pause time", but for 3 SV the time is always around 33ms and for 4 SV - 75 ms.

    Release.7z

  • In reply to Alexander Aganichev:

    Garrett,

    I ported this test to idk572x using PDK 1.0.14 and I was able to reproduce same issue.

  • In reply to Alexander Aganichev:

    Alex,

    sorry I was sidetracked, will try to work on this next Monday.

    Regards,
    Garrett
  • In reply to Garrett Ding:

    I was able to recover my setup, but the RawSocket.exe crashes.

    I will try to find a win7 PC. With my win10 workstation, I got 'unhandled Exception' as below which could be related to Beckhoff driver installed in my PC too, will continue to look into...Alternatively, can you tell me your SV packet format (header and payload)? I can try to use a PC packet generator...

    hsr_alex\Release>RawSocket.exe 250 8
    ...
    Unhandled Exception: System.InvalidOperationException: Unable to open the adapter. Adapter name: rpcap://\Device\NPF_{}. Error: failed to set hardware filter to promiscuous mode
    at PcapDotNet.Core.LivePacketCommunicator.PcapOpen(SByte* source, Int32 snapshotLength, PacketDeviceOpenAttributes attributes, Int32 ...

    Regards,
    Garrett

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.