PROCESSOR-SDK-AM64X: Duplicated packets on ICSSG_PRU HSR offloaded solution

Tianyi Liu

Intellectual 650 points

Part Number: PROCESSOR-SDK-AM64X

Tool/software:

Hello Ti,

I'm using the latest TI SDK release Image + firmware:

https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-RT-AM64X

=> 09.02.01.10 (30 May 2024)

I observe some duplicated packets on hsr0 interface on 100 Mbps for "high throughput" on 100 Mbps.

Here is our set up :

We have 2 EVMs

eth1 <-> eth2

eth2 <-> eth2

First we set the boards in u_boot, as DUAL EMAC mode

setenv name_overlays 'k3-am642-evm-icssg1-dualemac.dtbo'
run bootcmd

and once it's boot, we disable network on EVM1 and EVM2:

systemctl disable dhcpcd.service
systemctl disable NetworkManager.service
systemctl disable systemd-resolved.service
systemctl disable systemd-networkd.service
sync
reboot

Also, we set up the board EVM1 to 100Mbps link speed

ethtool -s eth1 speed 100 duplex full 
ethtool -s eth2 speed 100 duplex full

We up interfaces for EVM1 and EVM2 :

ip link set dev eth1 up
ip link set dev eth2 up

And we enable HSR offload solution on both boards.

TEST1 :Send packets with ping

We observe some duplicated packets on HSR interface with our applications.

The application basically is sending message every 1ms et wait the receive of a message before re-sending it again. Since we cannot share this application, we tried to use an open source application : ping

ping 192.168.100.5 -s 400 -i 0.001

root@am64xx-evm:~# ping 192.168.100.5 -s 400 -i 0.005

PING 192.168.100.5 (192.168.100.5): 400 data bytes

408 bytes from 192.168.100.5: seq=0 ttl=64 time=0.565 ms

408 bytes from 192.168.100.5: seq=1 ttl=64 time=0.397 ms

408 bytes from 192.168.100.5: seq=2 ttl=64 time=0.301 ms

408 bytes from 192.168.100.5: seq=3 ttl=64 time=0.301 ms

408 bytes from 192.168.100.5: seq=4 ttl=64 time=0.287 ms
(...)
408 bytes from 192.168.100.5: seq=170 ttl=64 time=72.846 ms

408 bytes from 192.168.100.5: seq=175 ttl=64 time=72.879 ms

408 bytes from 192.168.100.5: seq=180 ttl=64 time=72.891 ms

408 bytes from 192.168.100.5: seq=185 ttl=64 time=72.893 ms

408 bytes from 192.168.100.5: seq=190 ttl=64 time=72.881 ms

408 bytes from 192.168.100.5: seq=195 ttl=64 time=72.824 ms

408 bytes from 192.168.100.5: seq=201 ttl=64 time=66.246 ms

408 bytes from 192.168.100.5: seq=206 ttl=64 time=66.207 ms

408 bytes from 192.168.100.5: seq=211 ttl=64 time=66.270 ms

408 bytes from 192.168.100.5: seq=216 ttl=64 time=66.271 ms

408 bytes from 192.168.100.5: seq=221 ttl=64 time=66.330 ms

408 bytes from 192.168.100.5: seq=226 ttl=64 time=67.387 ms

408 bytes from 192.168.100.5: seq=231 ttl=64 time=67.386 ms

408 bytes from 192.168.100.5: seq=236 ttl=64 time=67.394 ms

408 bytes from 192.168.100.5: seq=241 ttl=64 time=67.454 ms

408 bytes from 192.168.100.5: seq=246 ttl=64 time=67.389 ms

408 bytes from 192.168.100.5: seq=251 ttl=64 time=67.412 ms

408 bytes from 192.168.100.5: seq=256 ttl=64 time=67.376 ms

408 bytes from 192.168.100.5: seq=261 ttl=64 time=67.377 ms

408 bytes from 192.168.100.5: seq=262 ttl=64 time=71.744 ms

408 bytes from 192.168.100.5: seq=266 ttl=64 time=72.865 ms

408 bytes from 192.168.100.5: seq=266 ttl=64 time=78.379 ms (DUP!)

408 bytes from 192.168.100.5: seq=276 ttl=64 time=72.784 ms

408 bytes from 192.168.100.5: seq=281 ttl=64 time=72.775 ms

408 bytes from 192.168.100.5: seq=287 ttl=64 time=71.716 ms

408 bytes from 192.168.100.5: seq=292 ttl=64 time=71.861 ms

408 bytes from 192.168.100.5: seq=297 ttl=64 time=71.867 ms

408 bytes from 192.168.100.5: seq=302 ttl=64 time=71.893 ms

408 bytes from 192.168.100.5: seq=307 ttl=64 time=71.911 ms

408 bytes from 192.168.100.5: seq=312 ttl=64 time=71.921 ms

408 bytes from 192.168.100.5: seq=317 ttl=64 time=66.460 ms

408 bytes from 192.168.100.5: seq=322 ttl=64 time=66.487 ms

408 bytes from 192.168.100.5: seq=327 ttl=64 time=66.595 ms

408 bytes from 192.168.100.5: seq=332 ttl=64 time=66.597 ms

408 bytes from 192.168.100.5: seq=337 ttl=64 time=66.618 ms

408 bytes from 192.168.100.5: seq=342 ttl=64 time=67.827 ms

408 bytes from 192.168.100.5: seq=342 ttl=64 time=73.238 ms (DUP!)

408 bytes from 192.168.100.5: seq=352 ttl=64 time=67.672 ms

408 bytes from 192.168.100.5: seq=352 ttl=64 time=73.202 ms (DUP!)

408 bytes from 192.168.100.5: seq=362 ttl=64 time=67.557 ms

408 bytes from 192.168.100.5: seq=362 ttl=64 time=73.031 ms (DUP!)

408 bytes from 192.168.100.5: seq=372 ttl=64 time=67.519 ms

408 bytes from 192.168.100.5: seq=377 ttl=64 time=67.478 ms

408 bytes from 192.168.100.5: seq=378 ttl=64 time=71.889 ms

408 bytes from 192.168.100.5: seq=378 ttl=64 time=77.416 ms (DUP!)

408 bytes from 192.168.100.5: seq=387 ttl=64 time=72.881 ms

408 bytes from 192.168.100.5: seq=387 ttl=64 time=78.399 ms (DUP!)

408 bytes from 192.168.100.5: seq=397 ttl=64 time=72.858 ms

408 bytes from 192.168.100.5: seq=403 ttl=64 time=71.761 ms

408 bytes from 192.168.100.5: seq=403 ttl=64 time=77.266 ms (DUP!)

408 bytes from 192.168.100.5: seq=413 ttl=64 time=71.761 ms

408 bytes from 192.168.100.5: seq=418 ttl=64 time=71.806 ms

408 bytes from 192.168.100.5: seq=423 ttl=64 time=71.895 ms

408 bytes from 192.168.100.5: seq=428 ttl=64 time=71.917 ms

408 bytes from 192.168.100.5: seq=433 ttl=64 time=73.019 ms

408 bytes from 192.168.100.5: seq=438 ttl=64 time=73.015 ms

408 bytes from 192.168.100.5: seq=443 ttl=64 time=73.056 ms

408 bytes from 192.168.100.5: seq=448 ttl=64 time=72.988 ms

408 bytes from 192.168.100.5: seq=453 ttl=64 time=73.097 ms

408 bytes from 192.168.100.5: seq=453 ttl=64 time=78.526 ms (DUP!)

408 bytes from 192.168.100.5: seq=463 ttl=64 time=73.046 ms

408 bytes from 192.168.100.5: seq=468 ttl=64 time=73.018 ms

408 bytes from 192.168.100.5: seq=473 ttl=64 time=73.032 ms

408 bytes from 192.168.100.5: seq=478 ttl=64 time=72.986 ms

408 bytes from 192.168.100.5: seq=483 ttl=64 time=72.960 ms

408 bytes from 192.168.100.5: seq=488 ttl=64 time=72.877 ms

408 bytes from 192.168.100.5: seq=493 ttl=64 time=72.837 ms

408 bytes from 192.168.100.5: seq=499 ttl=64 time=71.699 ms

408 bytes from 192.168.100.5: seq=504 ttl=64 time=66.191 ms

408 bytes from 192.168.100.5: seq=509 ttl=64 time=66.153 ms

408 bytes from 192.168.100.5: seq=509 ttl=64 time=71.691 ms (DUP!)

408 bytes from 192.168.100.5: seq=519 ttl=64 time=66.164 ms

408 bytes from 192.168.100.5: seq=524 ttl=64 time=67.229 ms

408 bytes from 192.168.100.5: seq=529 ttl=64 time=67.182 ms

408 bytes from 192.168.100.5: seq=534 ttl=64 time=67.163 ms

408 bytes from 192.168.100.5: seq=539 ttl=64 time=67.169 ms

408 bytes from 192.168.100.5: seq=544 ttl=64 time=67.198 ms

408 bytes from 192.168.100.5: seq=549 ttl=64 time=67.250 ms

408 bytes from 192.168.100.5: seq=554 ttl=64 time=67.256 ms

408 bytes from 192.168.100.5: seq=559 ttl=64 time=67.251 ms

408 bytes from 192.168.100.5: seq=564 ttl=64 time=67.241 ms

408 bytes from 192.168.100.5: seq=565 ttl=64 time=71.658 ms

408 bytes from 192.168.100.5: seq=569 ttl=64 time=72.754 ms

408 bytes from 192.168.100.5: seq=574 ttl=64 time=72.745 ms

408 bytes from 192.168.100.5: seq=579 ttl=64 time=72.834 ms

408 bytes from 192.168.100.5: seq=585 ttl=64 time=71.735 ms

408 bytes from 192.168.100.5: seq=590 ttl=64 time=71.712 ms

408 bytes from 192.168.100.5: seq=595 ttl=64 time=71.731 ms

408 bytes from 192.168.100.5: seq=600 ttl=64 time=71.834 ms

408 bytes from 192.168.100.5: seq=605 ttl=64 time=71.886 ms

408 bytes from 192.168.100.5: seq=610 ttl=64 time=72.933 ms

408 bytes from 192.168.100.5: seq=615 ttl=64 time=67.448 ms

408 bytes from 192.168.100.5: seq=620 ttl=64 time=67.457 ms

408 bytes from 192.168.100.5: seq=625 ttl=64 time=67.526 ms

408 bytes from 192.168.100.5: seq=630 ttl=64 time=67.444 ms

408 bytes from 192.168.100.5: seq=635 ttl=64 time=67.449 ms

408 bytes from 192.168.100.5: seq=640 ttl=64 time=67.495 ms

408 bytes from 192.168.100.5: seq=645 ttl=64 time=67.403 ms

408 bytes from 192.168.100.5: seq=650 ttl=64 time=67.412 ms

408 bytes from 192.168.100.5: seq=655 ttl=64 time=67.396 ms

408 bytes from 192.168.100.5: seq=660 ttl=64 time=67.416 ms

408 bytes from 192.168.100.5: seq=665 ttl=64 time=67.427 ms

408 bytes from 192.168.100.5: seq=670 ttl=64 time=67.302 ms

408 bytes from 192.168.100.5: seq=676 ttl=64 time=66.176 ms

408 bytes from 192.168.100.5: seq=677 ttl=64 time=70.632 ms

408 bytes from 192.168.100.5: seq=681 ttl=64 time=71.692 ms

408 bytes from 192.168.100.5: seq=686 ttl=64 time=71.697 ms

As you can see, according to ping, we have some duplicated packets. We was wondering if this is coming from the PRU firmware.

Here the speed is 408B every 1ms

=> throughput = 408kB/s

By assuming the device needs to handle Rx and Tx, so it can handle ~ 816kB/s ~ 1Mbit/s << 100 Mbps

is it possible to reproduce on your side this issue ?

Tianyi

over 2 years ago

+1 Daolin Qiu over 2 years ago

TI__Mastermind 20380 points

Hello Tianyi,

Thanks for sharing your steps to see this issue. I will give it a try on my setup and see if I can reproduce the same issue.

I noticed you set the size of ping packetsize to 400B each ping, is it possible for you to set a larger size that would result in closer to 100Mbps?

-Daolin

0 Tianyi Liu over 2 years ago in reply to Daolin Qiu

Intellectual 650 points

Hello Daolin,

Yes for sure ! is it possible to do on your side first 400 B (just to reproduce this issue ), then larger packets size ?

Tianyi

0 Daolin Qiu over 2 years ago in reply to Tianyi Liu

TI__Mastermind 20380 points

Hi Tianyi,

I have tried 400B per 1ms and have noticed occasional DUP! ping packets. For larger packet size, I tried 6250B per 1ms, which calculating according to your analysis should be around 100mbps. At 6250B per 1ms, I also notice occasional DUP! ping packets.

My software versioning is as below:

SDK: 09.02.01.10
HSR PRU Firmware: Recent firmware from https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1382911/processor-sdk-am64x-hsr-tag-and-vlan-tag-order-question
*Linux PRU_ICSSG Driver: Patch for 09.02.01.10 in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1389269/processor-sdk-am64x-multicast-filtering-on-raw-interfaces-not-working

Testing from firmware development team today shows no DUP! ping packets with this latest firmware, so I would give this latest firmware (see above bolded text) a try and see if you still see these issues.

-Daolin

0 Tianyi Liu over 2 years ago in reply to Daolin Qiu

Intellectual 650 points

Hi Daolin,

Thank you for the sharing, I will give a try and come back to you.

Tianyi

0 Daolin Qiu over 2 years ago in reply to Tianyi Liu

TI__Mastermind 20380 points

Hi Tianyi,

Update:

Firmware development team testing when Profishark in the network (between EVM1 and EVM2 on one port) did not show DUP! ping packets, however once they removed the Profishark, the DUP! ping packets do show up. The developer currently has some ideas on how to fix this issue but will need some time to implement and test.

Regardless, the latest HSR firmware mentioned in my last post should fix the VLAN and HSR tag ordering mentioned in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1382911/processor-sdk-am64x-hsr-tag-and-vlan-tag-order-question. Please give it try anyways to check if that issue is fixed.

-Daolin

0 Tianyi Liu over 2 years ago in reply to Daolin Qiu

Intellectual 650 points

Hi Daolin,

Thank you for the update.

Indeed, I have similar issue. When I put Profishark or another device between EVM1 and EVM2, I didn't have DUP! .

I just test with the latest firmware shared on the forum and still got the DUP!

Tianyi LIU

0 Daolin Qiu over 2 years ago in reply to Tianyi Liu

TI__Mastermind 20380 points

Hi Tianyi,

Thanks for confirming. The firmware development team is still working on this DUP! issue, I will respond with an update once they are able to implement a fix.

-Daolin

0 Daolin Qiu over 2 years ago in reply to Daolin Qiu

TI__Mastermind 20380 points

Update:

The firmware developer currently has made changes that fixes duplicate packets to host (to some extent) by adding spinlock during host duplicate table access. That is, it reduces the number of DUP! but doesn't guarantee no DUP! will show up. Currently, this is an internal active PR so I have not tested and verified on my setup.

Is this fix acceptable for your release?

-Daolin

0 Tianyi Liu over 2 years ago in reply to Daolin Qiu

Intellectual 650 points

Hello Daolin,

I will share this workaround with the customer but I doubt they will accept this ... The perfect solution would to have 0 duplication

-Tianyi

0 Daolin Qiu over 2 years ago in reply to Tianyi Liu

TI__Mastermind 20380 points

Hi Tianyi,

Thanks for the clarification. As explained in email,

the feedback from PRU development team is that due the fact that there is a limit to the size of the duplicate discard table for HSR duplicate packets, for larger packet transmission speeds, it would be difficult/not possible to account for the ideal case of guaranteeing 0 duplicates. The HSR spec also allows occasional duplicate frames to be sent to the host core. See below for the details:

“In the firmware, we have 256 entries table which gets populated with (SA + Sequence number) signature on receiving HSR frames.
When we receive new frame we compare against these entries and see if the received packet is duplicate.
If this duplicate table is full and we receive more entries then we will overwrite an existing entry (based on some rules).
So if a frame with this overwritten signature is now received, we will forward it to host.

This limitation arises from the fact that we have a limited size duplicate discard table. Ideally to address all possible signature values of 8B (= 6B SA + 2B SeqNr), we will need a (2^64) entries table - which is kind of impossible to realize.
So, even the spec allows occasional duplicate frames to be sent to the host core.”

After your discussion with your team, please let us know if this is still acceptable

-Daolin

0 Daolin Qiu over 2 years ago in reply to Daolin Qiu

TI__Mastermind 20380 points

Update:

Testing from development team shows that when sending 100000 packets at 1ms ping intervals results in 0 duplicated packets, 0% packet loss. At smaller ping intervals such as 0.1ms or 0.5ms, there will still be some duplicate packets (12-500 duplicates out of 1000000 packets transmitted).

Your initial post describes an application sending every 1ms and we've been using ping at 1ms intervals as a test. Do you expect smaller transmission intervals than 1ms for end use case?

Tianyi Liu said:
application basically is sending message every 1ms

-Daolin

0 Tianyi Liu over 2 years ago in reply to Daolin Qiu

Intellectual 650 points

Hi Daolin,

Thank you for the update.

I need to discuss with the internal team about this.

-Tianyi

0 Tianyi Liu over 2 years ago in reply to Tianyi Liu

Intellectual 650 points

Hi Daolin,

I come back to you about this Dupplicate packets observed on the PRU_ICSSG.

Just to confirm, the entry table you mentioned ealier, is the Entry table that holds the (SA + Sequence number) signature on receiving HSR frames of

- eth1 AND eth2,

- or ONLY eth1,

- or ONLY eth2 ?

-Tianyi

0 Daolin Qiu over 2 years ago in reply to Tianyi Liu

TI__Mastermind 20380 points

Hi Tianyi,

Let me ask the firmware team about this. My guess is both eth1 and eth2 just because it could be either port that receive the first packet and the other port receiving the duplicated packet (i.e. no guarantee that eth1 would receive the initial packet as opposed to eth2 so need to capture on both ports).

-Daolin

0 Daolin Qiu over 2 years ago in reply to Daolin Qiu

TI__Mastermind 20380 points

Update:

There is a separate dedicated duplicate table for each port in firmware

So I'm understanding this as the duplicate table size for eth1 and the duplicate table size for eth2 is not sufficient to guaranteeing absolutely 0 DUP! message shows up.

-Daolin

Processors

Processors forum

PROCESSOR-SDK-AM64X: Duplicated packets on ICSSG_PRU HSR offloaded solution