PROCESSOR-SDK-AM64X: Enable Cut-Through (forward) in HSR offload solution

Hi Tianyi,

To clarify, you would like to have the following features? 

• HSR offload for TX tag insertion and RX tag removal, which is coupled with Tx/Rx duplication offload?
• Port forwarding offload to be achieved with cut through mode?

Hsr offload can be achieved with cut through mode simply by enabling it through the devlink command you indicated along side the hsr setup script. More information on how enabling cut through can be found here: https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/latest/exports/docs/linux/Foundational_Components/PRU-ICSS/Linux_Drivers/PRU_ICSSG_Ethernet_Switch.html?highlight=cut%20through#cut-through-forwarding. My understanding is that you can add a line in the HSR script after devlink dev param set "$device" name hsr_offload_mode value true cmode runtime with devlink dev param set "$device" name cut_thru value 1 cmode runtime (value 1 if using only tx0 queue).

It looks like HSR Tx tag insertion and Rx tag removal offload is now available on SDK 9.2: https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/latest/exports/docs/devices/AM64X/linux/Release_Specific_Release_Notes.html#what-s-new 

-Daolin

Hi Daolin,

Thank you for your fast response !

Yes you are right we would like to use all the features at the same time :

We didn't test yet but my concern is that if we enable first HSR offload for TAG, Rx/TX duplication offload and then we enable cut-through mode using the following command (with devlink),

devlink dev param set platform/icssg1-eth name cut_thru value 65535 cmode runtime

will the system keep the HSR offload for TAG, Rx/TX duplication offload and add the new feature cut-through for forwarding ?

Tianyi

Hi Tianyi,

I believe as long as the following configs from the setup script stay "on" or "true" after running the devlink command to setup cut-through, the system should keep the HSR offload.

ethtool -K "$ifb" hsr-fwd-offload on

ethtool -K "$ifb" hsr-dup-offload on

ethtool -K "$ifb" hsr-tag-ins-offload on

ethtool -K "$ifb" hsr-tag-rm-offload on

devlink dev param set "$device" name hsr_offload_mode value true cmode runtime

After setting up cut through, you can try checking the current config of the fwd, dup, tag-ins, tag-rm offloads with "ethtool -k ethX" where X are the eth interfaces setup for HSR. Make sure to use lowercase k (NOT uppercase k)

Additionally, you can check current config of devlink "hsr_offload_mode" with "devlink dev param show platform/icssg1-eth name hsr_offload_mode"

This would be a good first test to see if setting up cut through does keep the original configurations. Of course, afterwards it would be best to test for actual expected functionality.

I will try to test on my setup as well and get back to you.

-Daolin

Hi Tianyi,

As Milan requested, attached is the test sequence we used to test the following

1. Test sequence/script for Tx duplicate offload
2. Test sequence/script for Tx Tag insertion offload 
3. Test sequence/script for Rx Tag removal and Rx duplicate discard offload

HSR Offload Test Procedure-v4-20240419_120627.pdf

As I mentioned before HSR Tx tag insertion and Rx tag removal offload is now available on SDK 9.2: https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/latest/exports/docs/devices/AM64X/linux/Release_Specific_Release_Notes.html#what-s-new for you to test with.

Please let me know if these test sequences are suitable for you.

As you and Milan mentioned in the call this week, cut-through with HSR offload is working for you?

-Daolin

Hi Daolin,

Thank you for your fast response and your sharing about the test!

We will try on your side for the tests and let you know about :

1. Test sequence/script for Tx duplicate offload
2. Test sequence/script for Tx Tag insertion offload 
3. Test sequence/script for Rx Tag removal and Rx duplicate discard offload

We didn't test yet cut-thourgh with HSR offload features (the features mentioned above). We will try and let you know !

Tianyi

Hi Daolin,

I come back to you for the Testplan. I have a similar setup than you, except instead of having 3 AM64x GPEVM boards I have 2. 

I found similar results than the Testplan you shared for the following test:

- Test sequence/script for Tx duplicate offload

- Test sequence/script for Rx Tag removal and Rx duplicate discard offload

However, I didn't find same result for the

- Test sequence/script for Tx Tag insertion offload : HSR Tx Tag Insertion

Whether Tx Tag insertion functionality is offloaded on the PRU or not,

with tcpdump -i eth1 -xx, I still have HSR Ethertype ...

Tianyi

Hi Tianyi,

Can you share the output of  tcpdump -i eth1 -xx and  tcpdump -i eth2 -xx for the Tx Tag insertion functionality?

Just to clarify, are you still running these tests with the network disable commands below as you mentioned using on other HSR tests?

systemctl $1 dhcpcd.service
systemctl $1 NetworkManager.service
systemctl $1 systemd-resolved.service
systemctl $1 systemd-networkd.service
sync
reboot

I'll test on both out of box and with network disable and see if I get the same results.

-Daolin

Does your tcpdump -i eth1 -xx look similar to mine (3-board setup, out-of-box, no network disabling) when testing with TX tag offloaded?

root@am64xx-evm:~# tcpdump -i eth1 -xx
[  957.422878] device eth1 entered promiscuous mode
[  957.422997] audit: type=1700 audit(1652797889.516:19): dev=eth1 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[  957.426933] audit: type=1300 audit(1652797889.516:19): arch=c00000b7 syscall=208 success=yes exit=0 a0=4 a1=107 a2=1 a3=ffffe5c7e9d0 items=0 ppid=975 pid=1361 auid=4294967295 uid=0 gid=0 euid=0 suid=0)
[  957.426968] audit: type=1327 audit(1652797889.516:19): proctitle=74637064756D70002D690065746831002D7878
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:31:29.755427 IP am64xx-evm > 192.168.2.22: ICMP echo request, id 1360, seq 4, length 64
  0x0000:  70ff 761f 3c45 70ff 761e 9c45 0800 4500
  0x0010:  0054 8a53 4000 4001 2adb c0a8 0214 c0a8
  0x0020:  0216 0800 93bb 0550 0004 4ab7 1439 0000
  0x0030:  0000 0000 0000 0000 0000 0000 0000 0000
  0x0040:  0000 0000 0000 0000 0000 0000 0000 0000
  0x0050:  0000 0000 0000 0000 0000 0000 0000 0000
  0x0060:  0000
14:31:29.888403 70:ff:76:1e:9c:45 (oui Unknown) > 01:15:4e:00:01:2d (oui Unknown), ethertype Unknown (0x88fb), length 60: 
  0x0000:  0115 4e00 012d 70ff 761e 9c45 88fb 0001
  0x0010:  7e03 1706 70ff 761e 9c45 0000 0000 0000
  0x0020:  0000 0000 0000 0000 0000 0000 0000 0000
  0x0030:  0000 0000 0000 0000 0000 0000
14:31:30.297766 70:ff:76:1f:3c:45 (oui Unknown) > 01:15:4e:00:01:2d (oui Unknown), ethertype Unknown (0x88fb), length 60: 
  0x0000:  0115 4e00 012d 70ff 761f 3c45 88fb 0001
  0x0010:  7f31 1706 70ff 761f 3c45 0000 0000 0000
  0x0020:  0000 0000 0000 0000 0000 0000 0000 0000
  0x0030:  0000 0000 0000 0000 0000 0000
14:31:30.496199 70:ff:76:1f:3d:c5 (oui Unknown) > 01:15:4e:00:01:2d (oui Unknown), ethertype Unknown (0x88fb), length 60: 
  0x0000:  0115 4e00 012d 70ff 761f 3dc5 88fb 0001
  0x0010:  7f3c 1706 70ff 761f 3dc5 0000 0000 0000
  0x0020:  0000 0000 0000 0000 0000 0000 0000 0000
  0x0030:  0000 0000 0000 0000 0000 0000
14:31:30.755750 IP am64xx-evm > 192.168.2.22: ICMP echo request, id 1360, seq 5, length 64
  0x0000:  70ff 761f 3c45 70ff 761e 9c45 0800 4500
  0x0010:  0054 8d7c 4000 4001 27b2 c0a8 0214 c0a8
  0x0020:  0216 0800 e576 0550 0005 e9fa 2339 0000
  0x0030:  0000 0000 0000 0000 0000 0000 0000 0000
  0x0040:  0000 0000 0000 0000 0000 0000 0000 0000
  0x0050:  0000 0000 0000 0000 0000 0000 0000 0000
  0x0060:  0000
14:31:31.193775 ARP, Reply am64xx-evm is-at 70:ff:76:1e:9c:45 (oui Unknown), length 28
  0x0000:  70ff 761f 3c45 70ff 761e 9c45 0806 0001
  0x0010:  0800 0604 0002 70ff 761e 9c45 c0a8 0214
  0x0020:  70ff 761f 3c45 c0a8 0216
14:31:31.755966 IP am64xx-evm > 192.168.2.22: ICMP echo request, id 1360, seq 6, length 64
  0x0000:  70ff 761f 3c45 70ff 761e 9c45 0800 4500
  0x0010:  0054 8ffe 4000 4001 2530 c0a8 0214 c0a8
  0x0020:  0216 0800 c032 0550 0006 ff3d 3339 0000
  0x0030:  0000 0000 0000 0000 0000 0000 0000 0000
  0x0040:  0000 0000 0000 0000 0000 0000 0000 0000
  0x0050:  0000 0000 0000 0000 0000 0000 0000 0000
  0x0060:  0000

What I noticed is when TX tag is not offloaded there is the below message detected with ethertype 0x892f tied to the destination MAC address.

70:ff:76:1e:9c:45 (oui Unknown) > 70:ff:76:1f:3c:45 (oui Unknown), ethertype Unknown (0x892f), length 104

When TX tag is offloaded, there is only the following message without ethertype with no reference to the destination MAC address.

IP 192.168.2.22 > am64xx-evm: ICMP echo reply, id 1360, seq 372, length 64

The "70:ff:76:1f:3d:c5 (oui Unknown) > 01:15:4e:00:01:2d (oui Unknown), ethertype Unknown (0x88fb), length 60" I noticed on both TX tag offloaded vs nonoffloaded is not using the same ethertype.

Are you sure you're not mistaking this message as an HSR ethertype in the TX Tag offloaded case?

-Daolin

Hi Daolin,

Yes I'm testing these tests with the network disable commands that you mentionned.

I will double check on my side and share with you the output of the tcpdump -i eth1 -xx and  tcpdump -i eth2 -xx for the Tx Tag insertion.

Tianyi

Hi Tianyi,

As an update, my results with network disable (3-board setup) are similar to what I shared in my previous response.

With no network disable (2-board setup) I also see similar results to my previous response. 

Also, looking from an overall perspective, Tx Tag insertion offload would not save much CPU load as compared to the other offloading. The bigger/main issue is if you still see the ping packet loss from the 3-board setup: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1350947/processor-sdk-am64x-hsr-offload-pru-firmware-packet-lost

-Daolin

Hi Daolin,

Indeed you are right ! By redoing the test for Tx Tag insertion, I mix icmp packets (generated by ping) and hsr supervision frames ...

Thank you very much !

Tianyi

Hi Tianyi, 

Thanks for confirming the Tx Tag insertion test case. For now let's focus on the issue https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1350947/processor-sdk-am64x-hsr-offload-pru-firmware-packet-lost before trying to address any cut-through testing.

-Daolin