PROCESSOR-SDK-AM64X: HSR PRU firmware load issue

Hi Milan,

Can you give me information on how your hardware setup looks like? 

Regards,

NItika

Hi,

We have two AM64x-EVM boards with default image from PROCESSOR-SDK-LINUX-RT-AM64X — Processor SDK RT-Linux for AM64x  Version: 09.01.00.08

They are connected with two cables eth1 to eth1 and eth2 to eth2... These ports are PRU ICSSG as DTB overlay is used in U-boot.

On both boards we executed HSR script from your documentation with hsr_hw parameter. we can see and we can confirm that offload is configured  properly

One one board we have tcpdump on hsr0 to wait for packets

tcpdump -XX -i hsr0 -e -p

Second board is board where we are loading firmware two times.. This board has own app to send raw MAC multicast packets 

./myapp -i hsr0 -m 01:0c:cd:01:00:05 

Hi Milan,

Just to clarify, is mac address 01:0c:cd:01:00:05 for the eth1 and 01:0c:cd:01:00:10 for eth2, both on the first EVM (not the second board with the myapp)?

Without setting up HSR, could you run your multicast test and see if the issue still occurs? Let's try to isolate if the problem is due to multicast or HSR first. 

Additionally, from my understanding, to fully test an HSR network, at least three EVMs should be connected in a ring topology rather than just two EVMs. Is there a particular reason for testing the setup with 2 EVMs?

-Daolin

Hi 

Hi Milan,

Thanks for explaining your set up.

I will try to reproduce your setup and see if I observe the same issue. I will let you know with an update in a day or two.

-Daolin

Hi Milan,

Have you tried running the following to make sure your mac multicast addresses are as expected?

$ip maddr show dev hsr0

-Daolin

Hi,

yes this is fine too.. we did add/del without issue

Hi Milan,

Are you able to share the results of $ip maddr show dev hsr0? 

-Daolin

Hi Milan,

Without adding the multicast MAC address, have you tried testing the HSR offload setup with a ping between the two EVMs with no packet loss after running the hsr script one time when disconnecting one of the ethernet cables?

I actually noticed that there was packet loss with a ping when only running the hsr script once. After I run the hsr script one more time, I don't see packet loss when disconnecting an ethernet cable. I'm trying to verify if HSR with a ping test works as intended, isolated from the multicast filtering first.

-Daolin

We have test in our test banch for this but we did not started yet... Let me work on this and share results with you

But, there was issue in previous releses with load firmware two times

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1255842/processor-sdk-am64x-hsr-load-firmware-issue/4752433#4752433

we started with multicast as this was not supported in previous releases but during next two-three weeks we will do full test execution

Hi Milan,

The bug from before might not have been fully fixed, I'm trying to confirm with the team. 

If you do see a similar issue that I saw with the basic HSR setup with a ping, that will indicate to me that there is at least another data point that the basic HSR setup has an issue.

Additionally, can you help me understand how the EVM is receiving multicast packets via tcpdump? Is the hsr0 interface added to a multicast group? I'm not too sure how packets sent to the multicast address 01:0c:cd:01:00:05 are being detected by the EVM. Since I don't have access to the "myapp" you are using in your test, I'm attempting to ping the IPv4 address equivalent of the MAC multicast address you added. However, upon some research into testing Multicast Traffic in Linux, according to this site, there are additional steps to allow icmp messages to go through.

-Daolin

we are using raw packets with MAC multicats, it is not IP... You just need to have app that is using raw socket

Probbaly it is possible to use https://packeth.sourceforge.net/packeth/Home.html to send raw packets or to write simple app like

https://gist.github.com/austinmarton/1922600

socket( AF_PACKET, SOCK_RAW, htons(etherType)

https://linux.die.net/man/2/socket

This open question for me - how you are testing your multicast code? Link that you send is explanation how to test multicast on IP level

sysctl net.ipv4.icmp_echo_ignore_broadcasts=0

route add -net 224.0.0.0 netmask 240.0.0.0 dev  eth0

IPERF 2 support myulticast and IPERF 3 does not support... next command is with iperf2

iperf -c 224.0.0.5 -u -b 500M -t 300

Packets should be dropped... then add multicast address with next command and do perf again... Packets should be received

ip addr add 224.0.0.5/24 dev eth0 autojoin

However, we are testing goose packets and it is MAC layer, not IP... This is why we did not test on IP level

Hi Milan,

I will need more time to investigate how to use the open source to send raw packets, to replicate the MAC layer multicast packet sending - I will need to discuss with internal software team as well. 

It is possible to share the "myapp" directly? Please note that anything shared here will be public information.

-Daolin

Hi Daolin,

Not possible to share myapp code... but why you cannot use packETH as TI  recommneded this tool

https://software-dl.ti.com/mcu-plus-sdk/esd/AM64X/09_01_00_41/exports/docs/api_guide_am64x/EXAMPLES_ENET_LAYER2_CPSW.html

 let me know and we can orginize call and show you how to do this with packETH

This issue is very urgent for us to be fixed in April TI SDK release...

Hi Milan,

Let me confirm with you the following first

1. Have you been able to test HSR without the multicast with your current two-board setup yet? I know you mentioned you would work on it and share the results.

2. With the first run of the HSR script with the multicast testing, the behavior is as you expect and require?

3. Is the reason you are reloading/rerunning the HSR script (with your multicast testing) a second time due to the workaround conclusion from https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1255842/processor-sdk-am64x-hsr-load-firmware-issue/4752433#4752433 

-Daolin

Hi,

we did not test HSR yet on our side... Plan is to do next week or week after... I can give you results as soon as I have

Hi Milan,

Thank you for confirming #1

Can you clarify 

2. With the first run of the HSR script with the multicast testing, the behavior is as you expect and require?

3. Is the reason you are reloading/rerunning the HSR script (with your multicast testing) a second time due to the workaround conclusion from https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1255842/processor-sdk-am64x-hsr-load-firmware-issue/4752433#4752433 

Double checking on these as I need to understand the clarity on these and relationship with the previous hsr fw load twice issue. This is based on the discussion with the internal team.

-Daolin

Point 2 : Correct

Point 3: yes

Hi Milan,

Just to keep you updated:

I will be meeting with the internal team to discuss the first issue of HSR offload without multicast tomorrow. 

I hope to make progress on narrowing down and fixing this first issue before tackling the multicast issue. 

I will update you again next week

-Daolin

Hi Daolin,

I'm a colleague of Milan.
By checking your latest Image release note :

https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/09_02_00_08/exports/docs/devices/AM64X/linux/Release_Specific_Release_Notes.html

It seems :

HSR switching offload firmware needs to be loaded twice on plateform am64xx-hsevm is closed for this release !

       Is that mean we can now ping between 2 EVMs with offload hsr without your issue described at the beginning of this ticket ?

Could you confirm that ?

Also did you reproduce on your side loading twice hsr feature creates issues on multicast?

Tianyi

Hi Tianyi,

You are correct that the "HSR switching offload firmware needs to be loaded twice" issue is noted as fixed for the latest release as well as SDK v09.01.00.08 (https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/09_01_00_08/exports/docs/devices/AM64X/linux/Release_Specific_Release_Notes.html); however, the reason why I am still concerned is because on my setup with 2 EVMs, I still see a similar issue of packet loss after a first run of the HSR offload setup script. This is one of the two issues I'm trying to investigate (the other being the multicast issue). 

I was able to recreate the multicast issue Milan was facing (on a MAC level) and looked further into the icssg_prueth.c driver (https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/drivers/net/ethernet/ti/icssg_prueth.c?h=ti-rt-linux-6.1.y). I noticed that every time the HSR setup script is ran twice, eth1 enters promiscuous mode which triggers the "promisc" flag and causes a return from the "emac_ndo_set_rx_mode_work" function before the multicast section of the function can be run. I also checked with the internal software development team and was informed that promiscuous should not be entered in the HW offload case. 

The following is the boot log of what I see on the EVM that receives the multicast packets. Where I have not added the MAC multicast address under test to the list of MAC multicast addresses to filter and the test application sends 500 packets to the MAC multicast address under test. You can see that upon the second run of the hsr setup script, 500 packets were received based on the ifconfig statistics but that is not what is expected. Additionally, you can see a message that eth1 entered promiscious mode.

|  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_ 
|     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
|__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|  
              |___|                    |___|            

Arago Project am64xx-evm -

Arago 2023.10 am64xx-evm -

am64xx-evm login: [   22.039492] TI DP83869 300b2400.mdio:0f: attached PHY driver (mii_bus:phy_addr=300b2400.mdio:0f, irq=POLL)
[   22.054284] TI DP83869 mdio_mux-0.0:03: attached PHY driver (mii_bus:phy_addr=mdio_mux-0.0:03, irq=POLL)
[   22.054327] icssg-prueth icssg1-eth: TI PRU ethernet driver initialized: dual EMAC mode
[   22.190742] remoteproc remoteproc13: powering up 300b4000.pru
[   22.195518] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, size 40936
[   22.195559] remoteproc remoteproc13: unsupported resource 5
[   22.195588] remoteproc remoteproc13: remote processor 300b4000.pru is now up
[   22.195631] remoteproc remoteproc14: powering up 30084000.rtu
[   22.198212] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu0-prueth-fw.elf, size 30884
[   22.198262] remoteproc remoteproc14: remote processor 30084000.rtu is now up
[   22.198303] remoteproc remoteproc7: powering up 3008a000.txpru
[   22.203334] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, size 36672
[   22.203392] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[   22.203674] remoteproc remoteproc15: powering up 300b8000.pru
[   22.205110] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru1-prueth-fw.elf, size 41208
[   22.205149] remoteproc remoteproc15: unsupported resource 5
[   22.205177] remoteproc remoteproc15: remote processor 300b8000.pru is now up
[   22.205224] remoteproc remoteproc16: powering up 30086000.rtu
[   22.206419] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu1-prueth-fw.elf, size 30124
[   22.206471] remoteproc remoteproc16: remote processor 30086000.rtu is now up
[   22.206515] remoteproc remoteproc8: powering up 3008c000.txpru
[   22.207684] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-prueth-fw.elf, size 35184
[   22.207740] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[   22.209205] pps pps1: new PPS source ptp2
[   25.317696] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   25.321406] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[   26.405653] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[   26.405732] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[   32.485490] icssg-prueth icssg1-eth eth1: Link is Down
[   32.549492] icssg-prueth icssg1-eth eth2: Link is Down
[   35.557698] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[   35.621689] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[   68.325677] icssg-prueth icssg1-eth eth1: Link is Down
[   68.389492] icssg-prueth icssg1-eth eth2: Link is Down

am64xx-evm login: [   72.421679] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
r[   73.509674] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
oot
[   74.588669] audit: type=1006 audit(1651489418.499:2): pid=976 uid=0 old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=1 res=1
[   74.588700] audit: type=1300 audit(1651489418.499:2): arch=c00000b7 syscall=64 success=yes exit=1 a0=8 a1=ffffc631f418 a2=1 a3=ffffbd68e020 items=0 ppid=1 pid=976 auid=0 uid=0 gid=0 euid=0 suid=0 fsui)
[   74.588716] audit: type=1327 audit(1651489418.499:2): proctitle="(systemd)"
root@am64xx-evm:~# 
root@am64xx-evm:~# sh hsr_setup.sh hsr_hw eth1 eth2 192.168.2.20                                                                                                                                            
ip=192.168.2.20
if=hsr0
mac=70:ff:76:1f:3d:c5
slave-a=eth1
slave-b=eth2
device=platform/icssg1-eth
[   78.305129] icssg-prueth icssg1-eth eth1: Link is Down
[   78.358155] icssg-prueth icssg1-eth eth2: Link is Down
[   78.362465] remoteproc remoteproc7: stopped remote processor 3008a000.txpru
[   78.362490] remoteproc remoteproc14: stopped remote processor 30084000.rtu
[   78.362499] remoteproc remoteproc13: stopped remote processor 300b4000.pru
[   78.362508] remoteproc remoteproc8: stopped remote processor 3008c000.txpru
[   78.362517] remoteproc remoteproc16: stopped remote processor 30086000.rtu
[   78.362525] remoteproc remoteproc15: stopped remote processor 300b8000.pru
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off
hsr-dup-offload: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off
hsr-dup-offload: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
[   79.506656] icssg-prueth icssg1-eth: Enabling HSR offload mode
[   79.544917] remoteproc remoteproc13: powering up 300b4000.pru
[   79.549598] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 40436
[   79.549637] remoteproc remoteproc13: unsupported resource 5
[   79.549667] remoteproc remoteproc13: remote processor 300b4000.pru is now up
[   79.549710] remoteproc remoteproc14: powering up 30084000.rtu
[   79.553864] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 32904
[   79.553923] remoteproc remoteproc14: remote processor 30084000.rtu is now up
[   79.553967] remoteproc remoteproc7: powering up 3008a000.txpru
[   79.559898] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 35772
[   79.559958] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[   79.561357] remoteproc remoteproc15: powering up 300b8000.pru
[   79.562589] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 40652
[   79.562627] remoteproc remoteproc15: unsupported resource 5
[   79.562656] remoteproc remoteproc15: remote processor 300b8000.pru is now up
[   79.562698] remoteproc remoteproc16: powering up 30086000.rtu
[   79.563943] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 32120
[   79.563998] remoteproc remoteproc16: remote processor 30086000.rtu is now up
[   79.564109] remoteproc remoteproc8: powering up 3008c000.txpru
[   79.565282] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 34280
[   79.565336] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[   79.567656] pps pps1: new PPS source ptp2
[   82.662175] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[   82.662264] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   83.685904] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[   83.685983] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[   84.690308] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
root@am64xx-evm:~# 
root@am64xx-evm:~# 
root@am64xx-evm:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 1c:63:49:1a:d8:aa brd ff:ff:ff:ff:ff:ff
3: can0: <NOARP,ECHO> mtu 16 qdisc noop state DOWN group default qlen 10
    link/can 
4: can1: <NOARP,ECHO> mtu 16 qdisc noop state DOWN group default qlen 10
    link/can 
5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 70:ff:76:1f:3d:c5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::72ff:76ff:fe1f:3dc5/64 scope link 
       valid_lft forever preferred_lft forever
6: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 70:ff:76:1f:3d:c5 brd ff:ff:ff:ff:ff:ff permaddr 70:ff:76:1f:3d:c7
    inet6 fe80::72ff:76ff:fe1f:3dc5/64 scope link 
       valid_lft forever preferred_lft forever
7: hsr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1494 qdisc noqueue state UP group default qlen 1000
    link/ether 70:ff:76:1f:3d:c5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.20/24 scope global hsr0
       valid_lft forever preferred_lft forever
    inet6 fe80::72ff:76ff:fe1f:3dc5/64 scope link 
       valid_lft forever preferred_lft forever
root@am64xx-evm:~# [  102.117677] icssg-prueth icssg1-eth eth1: Link is Down
[  102.119682] icssg-prueth icssg1-eth eth2: Link is Down
[  106.215962] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  107.237901] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

root@am64xx-evm:~# 
root@am64xx-evm:~# ifconfig hsr0
hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494
        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::72ff:76ff:fe1f:3dc5  prefixlen 64  scopeid 0x20<link>
        ether 70:ff:76:1f:3d:c5  txqueuelen 1000  (Ethernet)
        RX packets 25  bytes 3452 (3.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 67  bytes 6894 (6.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@am64xx-evm:~# ip maddr show dev hsr0                                                                                                                                            
7:      hsr0
        link  33:33:00:00:00:01 users 3
        link  01:00:5e:00:00:01 users 3
        link  33:33:ff:1f:3d:c5 users 3
        link  01:00:5e:00:00:fb users 3
        link  33:33:00:00:00:fb users 3
        link  01:00:5e:00:00:fc users 3
        link  33:33:00:01:00:03 users 3
        inet  224.0.0.252
        inet  224.0.0.251
        inet  224.0.0.1
        inet6 ff02::1:3
        inet6 ff02::fb
        inet6 ff02::1:ff1f:3dc5
        inet6 ff02::1
        inet6 ff01::1
root@am64xx-evm:~# ifconfig hsr0
hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494
        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::72ff:76ff:fe1f:3dc5  prefixlen 64  scopeid 0x20<link>
        ether 70:ff:76:1f:3d:c5  txqueuelen 1000  (Ethernet)
        RX packets 25  bytes 3452 (3.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 74  bytes 7324 (7.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@am64xx-evm:~# 
root@am64xx-evm:~# sh hsr_setup.sh hsr_hw eth1 eth2 192.168.2.20
ip=192.168.2.20
if=hsr0
mac=70:ff:76:1f:3d:c5
slave-a=eth1
slave-b=eth2
device=platform/icssg1-eth
[  131.947093] device eth1 entered promiscuous mode
[  131.947219] audit: type=1700 audit(1651489475.852:3): dev=eth1 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[  131.971118] device eth2 entered promiscuous mode
[  131.971512] audit: type=1700 audit(1651489475.852:3): dev=eth2 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[  132.034954] audit: type=1300 audit(1651489475.852:3): arch=c00000b7 syscall=211 success=yes exit=32 a0=3 a1=ffffd5286bf8 a2=0 a3=ffffae6b7840 items=0 ppid=1082 pid=1093 auid=4294967295 uid=0 gid=0 eui)
[  132.034985] audit: type=1327 audit(1651489475.852:3): proctitle=6970006C696E6B0064656C6574650068737230
[  132.079362] icssg-prueth icssg1-eth eth1: Link is Down
[  132.129837] icssg-prueth icssg1-eth eth2: Link is Down
[  132.145404] remoteproc remoteproc7: stopped remote processor 3008a000.txpru
[  132.145431] remoteproc remoteproc14: stopped remote processor 30084000.rtu
[  132.145440] remoteproc remoteproc13: stopped remote processor 300b4000.pru
[  132.145449] remoteproc remoteproc8: stopped remote processor 3008c000.txpru
[  132.145458] remoteproc remoteproc16: stopped remote processor 30086000.rtu
[  132.145467] remoteproc remoteproc15: stopped remote processor 300b8000.pru
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
[  133.302805] remoteproc remoteproc13: powering up 300b4000.pru
[  133.303021] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 40436
[  133.305329] remoteproc remoteproc13: unsupported resource 5
[  133.305369] remoteproc remoteproc13: remote processor 300b4000.pru is now up
[  133.305414] remoteproc remoteproc14: powering up 30084000.rtu
[  133.305589] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 32904
[  133.305623] remoteproc remoteproc14: remote processor 30084000.rtu is now up
[  133.305645] remoteproc remoteproc7: powering up 3008a000.txpru
[  133.305757] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 35772
[  133.305786] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[  133.311137] remoteproc remoteproc15: powering up 300b8000.pru
[  133.311329] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 40652
[  133.311360] remoteproc remoteproc15: unsupported resource 5
[  133.311389] remoteproc remoteproc15: remote processor 300b8000.pru is now up
[  133.311426] remoteproc remoteproc16: powering up 30086000.rtu
[  133.311537] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 32120
[  133.311564] remoteproc remoteproc16: remote processor 30086000.rtu is now up
[  133.311584] remoteproc remoteproc8: powering up 3008c000.txpru
[  133.311685] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 34280
[  133.311712] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[  133.313178] pps pps1: new PPS source ptp2
[  135.375155] icssg-prueth icssg1-eth: Both interfaces must be linked to same upper device
[  135.386210] icssg-prueth icssg1-eth: Both interfaces must be linked to same upper device
[  136.422188] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  136.422259] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[  137.445702] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[  137.446686] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  138.423419] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
root@am64xx-evm:~# 
root@am64xx-evm:~# 
root@am64xx-evm:~# ip maddr show dev hsr0                                                                                                                                                                   
8:      hsr0
        link  33:33:00:00:00:01 users 2
        link  01:00:5e:00:00:01 users 2
        link  33:33:ff:1f:3d:c5 users 2
        link  01:00:5e:00:00:fb users 2
        link  01:00:5e:00:00:fc users 2
        link  33:33:00:00:00:fb users 2
        link  33:33:00:01:00:03 users 2
        inet  224.0.0.252
        inet  224.0.0.251
        inet  224.0.0.1
        inet6 ff02::1:3
        inet6 ff02::fb
        inet6 ff02::1:ff1f:3dc5
        inet6 ff02::1
        inet6 ff01::1
root@am64xx-evm:~# ifconfig hsr0                                                                                                                                                                   
hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494
        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::72ff:76ff:fe1f:3dc5  prefixlen 64  scopeid 0x20<link>
        ether 70:ff:76:1f:3d:c5  txqueuelen 1000  (Ethernet)
        RX packets 1  bytes 56 (56.0 B)
        RX errors 0  dropped 20  overruns 0  frame 0
        TX packets 38  bytes 4560 (4.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@am64xx-evm:~# ifconfig hsr0
hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494
        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::72ff:76ff:fe1f:3dc5  prefixlen 64  scopeid 0x20<link>
        ether 70:ff:76:1f:3d:c5  txqueuelen 1000  (Ethernet)
        RX packets 501  bytes 25056 (24.4 KiB)
        RX errors 0  dropped 20  overruns 0  frame 0
        TX packets 44  bytes 5198 (5.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@am64xx-evm:~# 

Could you or Milan share the entire boot log from running the hsr feature twice to testing the multicast packet send (for the EVM that is receiving the multicast packets)? I'm wondering there is a eth1 entering promiscuous mode message as well. You could insert the log using the "Insert-->Code" feature. 

-Daolin

root@am64xx-evm:~# ifconfig

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10<host>

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 82  bytes 6460 (6.3 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 82  bytes 6460 (6.3 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



root@am64xx-evm:~# sh ./hsr_setUp.sh hsr_hw eth1 eth2 192.168.2.20  

ip=192.168.2.20

if=hsr0

mac=70:ff:76:1e:9f:30

slave-a=eth1

slave-b=eth2

device=platform/

hsr-tag-ins-offload: off [fixed]

hsr-tag-rm-offload: off [fixed]

hsr-fwd-offload: off [fixed]

hsr-dup-offload: off [fixed]

Actual changes:

hsr-fwd-offload: off [requested on]

Could not change any device features

hsr-tag-ins-offload: off [fixed]

hsr-tag-rm-offload: off [fixed]

hsr-fwd-offload: off [fixed]

hsr-dup-offload: off [fixed]

hsr-tag-ins-offload: off [fixed]

hsr-tag-rm-offload: off [fixed]

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off [fixed]

hsr-tag-rm-offload: off [fixed]

hsr-fwd-offload: on

hsr-dup-offload: off

kernel answers: No such device

[   45.900835] remoteproc remoteproc15: unsupported resource 5

[   50.004992] hsr0: Slave B (eth2) is not up; please bring it up to get a fully working HSR network

root@am64xx-evm:~# dmesg | tail -n 20

[   45.907146] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu0-prueth-fw.elf, size4

[   45.907205] remoteproc remoteproc16: remote processor 30084000.rtu is now up

[   45.907249] remoteproc remoteproc7: powering up 3008a000.txpru

[   45.910547] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, siz2

[   45.910606] remoteproc remoteproc7: remote processor 3008a000.txpru is now up

[   45.915932] pps pps1: new PPS source ptp2

[   46.952692] device eth1 entered promiscuous mode

[   46.952793] audit: type=1700 audit(1651357756.932:3): dev=eth1 prom=256 old_prom=0 auid=4294967295

[   46.958558] device eth2 entered promiscuous mode

[   46.958641] audit: type=1700 audit(1651357756.932:3): dev=eth2 prom=256 old_prom=0 auid=4294967295

[   46.971980] audit: type=1300 audit(1651357756.932:3): arch=c00000b7 syscall=211 success=yes exit=)

[   46.973957] audit: type=1307 audit(1651357756.932:3): cwd="/home/root"

[   46.974545] audit: type=1302 audit(1651357756.932:3): item=0 name=(null) inode=9666 dev=00:07 mod0

[   46.976437] audit: type=1302 audit(1651357756.932:3): item=1 name=(null) inode=11005 dev=00:07 mo0

[   46.976867] audit: type=1302 audit(1651357756.932:3): item=2 name=(null) inode=11005 dev=00:07 mo0

[   46.977145] audit: type=1302 audit(1651357756.932:3): item=3 name=(null) inode=11006 dev=00:07 mo0

[   46.979101] audit: type=1327 audit(1651357756.932:3): proctitle=6970006C696E6B00616464006E616D6501

[   47.911083] am65-cpsw-nuss 8000000.ethernet eth1: Link is Up - 1Gbps/Full - flow control rx/tx

[   47.911219] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

[   50.004992] hsr0: Slave B (eth2) is not up; please bring it up to get a fully working HSR network

root@am64xx-evm:~# ifconfig

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet6 fe80::72ff:76ff:fe1e:9f30  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:9f:30  txqueuelen 1000  (Ethernet)

        RX packets 14  bytes 1088 (1.0 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 16  bytes 1360 (1.3 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



eth2: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500

        ether 70:ff:76:1e:9f:30  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 0  bytes 0 (0.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494

        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0

        inet6 fe80::72ff:76ff:fe1e:9f30  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:9f:30  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 4  overruns 0  frame 0

        TX packets 9  bytes 698 (698.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10<host>

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 162  bytes 12780 (12.4 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 162  bytes 12780 (12.4 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



root@am64xx-evm:~# ifconfig hsr0

hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494

        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0

        inet6 fe80::72ff:76ff:fe1e:9f30  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:9f:30  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 4  overruns 0  frame 0

        TX packets 9  bytes 698 (698.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



root@am64xx-evm:~# ip maddr show dev hsr0

7:      hsr0

        link  33:33:00:00:00:01 users 3

        link  01:00:5e:00:00:01 users 3

        link  33:33:ff:1e:9f:30 users 3

        inet  224.0.0.1

        inet6 ff02::1:ff1e:9f30

        inet6 ff02::1

        inet6 ff01::1

root@am64xx-evm:~# sh ./hsr_setUp.sh hsr_hw eth1 eth2 192.168.2.20  

ip=192.168.2.20

if=hsr0

mac=70:ff:76:1e:9f:30

slave-a=eth1

slave-b=eth2

device=platform/

hsr-tag-ins-offload: off [fixed]

hsr-tag-rm-offload: off [fixed]

hsr-fwd-offload: off [fixed]

hsr-dup-offload: off [fixed]

Actual changes:

hsr-fwd-offload: off [requested on]

Could not change any device features

hsr-tag-ins-offload: off [fixed]

hsr-tag-rm-offload: off [fixed]

hsr-fwd-offload: off [fixed]

hsr-dup-offload: off [fixed]

hsr-tag-ins-offload: off [fixed]

hsr-tag-rm-offload: off [fixed]

hsr-fwd-offload: on

hsr-dup-offload: off

hsr-tag-ins-offload: off [fixed]

hsr-tag-rm-offload: off [fixed]

hsr-fwd-offload: on

hsr-dup-offload: off

kernel answers: No such device

[   93.467428] remoteproc remoteproc15: unsupported resource 5

[   97.563829] hsr0: Slave B (eth2) is not up; please bring it up to get a fully working HSR network

root@am64xx-evm:~# dmesg | tail -n 30

[   91.241461] audit: type=1300 audit(1651357801.173:4): arch=c00000b7 syscall=211 success=yes exit=)

[   91.241498] audit: type=1327 audit(1651357801.173:4): proctitle=6970006C696E6B0064656C657465006870

[   91.270627] am65-cpsw-nuss 8000000.ethernet eth1: Link is Down

[   91.297653] remoteproc remoteproc7: stopped remote processor 3008a000.txpru

[   91.297676] remoteproc remoteproc16: stopped remote processor 30084000.rtu

[   91.297685] remoteproc remoteproc15: stopped remote processor 300b4000.pru

[   92.424605] am65-cpsw-nuss 8000000.ethernet eth1: PHY [mdio_mux-0.1:03] driver [TI DP83869] (irq=)

[   92.424638] am65-cpsw-nuss 8000000.ethernet eth1: configuring for phy/rgmii-rxid link mode

[   93.465091] remoteproc remoteproc15: powering up 300b4000.pru

[   93.467387] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, size6

[   93.467428] remoteproc remoteproc15: unsupported resource 5

[   93.467457] remoteproc remoteproc15: remote processor 300b4000.pru is now up

[   93.467503] remoteproc remoteproc16: powering up 30084000.rtu

[   93.467685] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu0-prueth-fw.elf, size4

[   93.467717] remoteproc remoteproc16: remote processor 30084000.rtu is now up

[   93.467744] remoteproc remoteproc7: powering up 3008a000.txpru

[   93.467842] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, siz2

[   93.467873] remoteproc remoteproc7: remote processor 3008a000.txpru is now up

[   93.477500] pps pps1: new PPS source ptp2

[   94.518464] device eth1 entered promiscuous mode

[   94.518560] audit: type=1700 audit(1651357804.497:5): dev=eth1 prom=256 old_prom=0 auid=4294967295

[   94.524439] device eth2 entered promiscuous mode

[   94.524522] audit: type=1700 audit(1651357804.497:5): dev=eth2 prom=256 old_prom=0 auid=4294967295

[   94.535111] audit: type=1300 audit(1651357804.497:5): arch=c00000b7 syscall=211 success=yes exit=)

[   94.537014] audit: type=1307 audit(1651357804.497:5): cwd="/home/root"

[   94.537530] audit: type=1302 audit(1651357804.497:5): item=0 name=(null) inode=9666 dev=00:07 mod0

[   94.539151] audit: type=1302 audit(1651357804.497:5): item=1 name=(null) inode=11339 dev=00:07 mo0

[   95.527568] am65-cpsw-nuss 8000000.ethernet eth1: Link is Up - 1Gbps/Full - flow control rx/tx

[   95.527637] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

[   97.563829] hsr0: Slave B (eth2) is not up; please bring it up to get a fully working HSR network

root@am64xx-evm:~# ip maddr show dev hsr0 

8:      hsr0

        link  33:33:00:00:00:01 users 2

        link  01:00:5e:00:00:01 users 2

        link  33:33:ff:1e:9f:30 users 2

        inet  224.0.0.1

        inet6 ff02::1:ff1e:9f30

        inet6 ff02::1

        inet6 ff01::1

root@am64xx-evm:~# ifconfig hsr0

hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494

        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0

        inet6 fe80::72ff:76ff:fe1e:9f30  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:9f:30  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 4  overruns 0  frame 0

        TX packets 9  bytes 698 (698.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



root@am64xx-evm:~# 

Hello, we indeed have the same issue.
eth1 is getting in promiscuous mode

Indeed eth1 is getting in promiscuous mode when we load hsr offload twice !

Hi Tianyi,

I noticed the boot log (under first ifconfig) eth1 and eth2 don't seem to be brought up before the first run of the hsr offload setup script. Subsequently, I see some messages such as "Could not change any device features" and "kernel answers: No such device" and "Slave B (eth2) is not up; please bring it up to get a fully working HSR network". 

I also see "am65-cpsw-nuss 8000000.ethernet eth1: Link is Up - 1Gbps/Full - flow control rx/tx" which indicates that you have not set up the eth1 interface as ICSSG_PRU ethernet. By default eth1 is set up as CPSW ethernet. HSR hardware offload is only supported from ICSSG_PRU ethernet. 

Based on you also seeing eth1 entering promiscuous mode, I'm assuming you may have attached the wrong boot log?

Additionally, are you able to test out if the HSR double load issue has been fixed in your setup? If you verify it has been fixed, you might just need to run the HSR offload setup script once for your multicast testing as it was mentioned for the first run of the script, the multicast filtering works as expected. The double run of the HSR offload script seemed to be solely for the workaround for the previous double firmware load issue.

I have already filed two bugs with the internal software team on the issues I see based on my replication of the original issue in this thread:

1. HSR still requires to runs of the hardware offload setup script for 2 board setup

2. Multicast filtering doesn't work on second run of HSR hardware offload setup script

The team is still working on debugging these issues so no fix release date has been confirmed.

-Daolin

Hi Daolin,

Yes i share with you the wrong logs ...

Thank you for reporting these two issues to the internal team.

Yes, I did the double run load hsr offload fw with Milan for multicast and we created this issue :

- 1 time load hsr offload and it does work for multicast

- 2 times load hsr_offload and it has the issue that milan described earlier.

As mentioned Milan earlier, the ticket's issue is very urgent for us. Is there any chance that could be fixed for the April TI SDK release ?

Hi Tianyi,

Here are the ticket/record IDs for the two bugs I reported so that you might also be able to track them.

LCPD-37501

LCPD-37500 

I was informed by a colleague that it is possible to share these here but this is the first time I'm trying, please let me know if you have difficulty viewing them.

I was wondering if you were able to test the double fw load HSR offload without the multicast. What I meant in my previous response was that if you find that the original issue with the base HSR offload (without multicast) does not appear in your setup with the latest SDK release, you wouldn't need to use the workaround of double run of the HSR offload setup script, which led to the multicast filtering issue.

I will try to prompt the software team if it's possible to be fixed for the April TI SDK release.

-Daolin

Hi Daolin,

Thank you for the sharing. I don't know it's normal but I cannot access to the Jiras link.

Just to be clear, the main goal of this ticket is

  - Double loads firmware of hsr offload (with multicast filtering ) does not work.

Loading twice the hsr offload fw may be one of the need for our customers. And loading twice the hsr offload fw is not a workaround of the ping issue between 2 evm with hsr offload without multicast.

We 've tried to reproduce on our side the ping issue without multicast between 2 boards with hsr offloads and we observe there were no packets lost.

Could you describe more your test with the ping issue without multicast between 2 boards with hsr offloads ?

Tianyi LIU

Hi Tianyu,

I apologize for the ticket links, after talking to a member of the team, it looks like the links are only accessible for the internal team. The ticket numbers could be useful for checking in future release notes to see if the bug has been fixed or not. Getting the multicast issue fixed for the next TI SDK release is difficult but I am working on trying to convince the team.

I asked and clarified earlier with Milan that the whole point of double load of hsr offload was to workaround the issue from https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1255842/processor-sdk-am64x-hsr-load-firmware-issue/4752433#4752433. However, I can see your point that loading HSR offload twice may be needed by your customers.

Here are the details of what I tested in terms of ping issue without multicast between 2 boards with HSR offload:

Problem: 

Testing HSR hardware offload setup on two AM64x EVMs connected eth1 to eth1 and eth2 to eth2 results in packet loss during disconnect of one of the two ethernet connections (either eth1 or eth2) while a ping is conducted from one EVM to the other EVM. 

Upon running the HSR offload setup script in https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/09_01_00_08/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/Network/HSR_Offload.html a second time, the packet loss is no longer seen and HSR appears to work as intended.

This packet loss is only seen in HSR hardware offload mode and not in non-offload mode. 

Steps to reproduce bug:

1.Connect two AM64x EVMs eth1 to eth1 and eth2 to eth2

2. Run the following on both EVMs in the U-boot command line to set up eth1 and eth2 to use ICSSG_PRU ethernet

setenv bootcmd 'run findfdt; run envboot;run init_${boot}; run get_kern_${boot}; run get_fdt_${boot};setenv name_overlays k3-am642-evm-icssg1-dualemac.dtbo; run get_overlay_${boot}; run run_kern'

EVM 1:

1. Run the hsr hardware offload script from https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/09_01_00_08/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/Network/HSR_Offload.html
2. After EVM 2 also ran the hsr hardware offload script once, run a ping command to EVM 2
3. While the ping command is running, disconnect one of the ethernet cables (either eth1 or eth2 connection) and notice that the ping messages pause
4. Reconnect the ethernet cable and notice that the ping messages resume
5. Cancel the ping command the statistics will show packet loss
6. Repeat steps 1-5 and notice that this time, no pausing of ping messages appears and no packet loss is observed

EVM 1 Console Log:

am64xx-evm login: root
[   20.557144] audit: type=1006 audit(1651488805.102:2): pid=950 uid=0 old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=1 res=1
[   20.557182] audit: type=1300 audit(1651488805.102:2): arch=c00000b7 syscall=64 success=yes exit=1 a0=8 a1=ffffd2b12cb8 a2=1 a3=ffff8132e020 items=0 ppid=1 pid=950 auid=0 uid=0 gid=0 euid=0 suid=0 fsui)
[   20.557198] audit: type=1327 audit(1651488805.102:2): proctitle="(systemd)"
root@am64xx-evm:~# [   21.528343] TI DP83869 300b2400.mdio:0f: attached PHY driver (mii_bus:phy_addr=300b2400.mdio:0f, irq=POLL)
[   21.553535] TI DP83869 mdio_mux-0.0:03: attached PHY driver (mii_bus:phy_addr=mdio_mux-0.0:03, irq=POLL)
[   21.553581] icssg-prueth icssg1-eth: TI PRU ethernet driver initialized: dual EMAC mode
[   21.665118] remoteproc remoteproc12: powering up 300b4000.pru
[   21.670630] remoteproc remoteproc12: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, size 40936
[   21.670672] remoteproc remoteproc12: unsupported resource 5
[   21.670703] remoteproc remoteproc12: remote processor 300b4000.pru is now up
[   21.670744] remoteproc remoteproc13: powering up 30084000.rtu
[   21.678361] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-rtu0-prueth-fw.elf, size 30884
[   21.678419] remoteproc remoteproc13: remote processor 30084000.rtu is now up
[   21.678464] remoteproc remoteproc7: powering up 3008a000.txpru
[   21.679691] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, size 36672
[   21.679742] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[   21.683073] remoteproc remoteproc14: powering up 300b8000.pru
[   21.684459] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-pru1-prueth-fw.elf, size 41208
[   21.684501] remoteproc remoteproc14: unsupported resource 5
[   21.684529] remoteproc remoteproc14: remote processor 300b8000.pru is now up
[   21.684571] remoteproc remoteproc15: powering up 30086000.rtu
[   21.687522] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-rtu1-prueth-fw.elf, size 30124
[   21.687578] remoteproc remoteproc15: remote processor 30086000.rtu is now up
[   21.687619] remoteproc remoteproc8: powering up 3008c000.txpru
[   21.688980] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-prueth-fw.elf, size 35184
[   21.691562] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[   21.696624] pps pps1: new PPS source ptp2
[   24.805667] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[   24.805757] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

root@am64xx-evm:~# reboot[   25.893673] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   25.894331] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
^C
root@am64xx-evm:~# sh hsr_setup.sh hsr_hw eth1 eth2 192.168.2.20                                                                                                                                            
ip=192.168.2.20
if=hsr0
mac=70:ff:76:1f:3d:c5
slave-a=eth1
slave-b=eth2
device=platform/icssg1-eth
[   41.744122] icssg-prueth icssg1-eth eth1: Link is Down
[   41.783407] icssg-prueth icssg1-eth eth2: Link is Down
[   41.790348] remoteproc remoteproc7: stopped remote processor 3008a000.txpru
[   41.790371] remoteproc remoteproc13: stopped remote processor 30084000.rtu
[   41.790381] remoteproc remoteproc12: stopped remote processor 300b4000.pru
[   41.790390] remoteproc remoteproc8: stopped remote processor 3008c000.txpru
[   41.790399] remoteproc remoteproc15: stopped remote processor 30086000.rtu
[   41.790409] remoteproc remoteproc14: stopped remote processor 300b8000.pru
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off
hsr-dup-offload: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off
hsr-dup-offload: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
[   42.928743] icssg-prueth icssg1-eth: Enabling HSR offload mode
[   42.969818] remoteproc remoteproc12: powering up 300b4000.pru
[   42.973356] remoteproc remoteproc12: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 40436
[   42.973391] remoteproc remoteproc12: unsupported resource 5
[   42.973421] remoteproc remoteproc12: remote processor 300b4000.pru is now up
[   42.973460] remoteproc remoteproc13: powering up 30084000.rtu
[   42.977872] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 32904
[   42.977928] remoteproc remoteproc13: remote processor 30084000.rtu is now up
[   42.977973] remoteproc remoteproc7: powering up 3008a000.txpru
[   42.983882] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 35772
[   42.983939] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[   42.986327] remoteproc remoteproc14: powering up 300b8000.pru
[   42.987537] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 40652
[   42.987575] remoteproc remoteproc14: unsupported resource 5
[   42.987604] remoteproc remoteproc14: remote processor 300b8000.pru is now up
[   42.987646] remoteproc remoteproc15: powering up 30086000.rtu
[   42.988871] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 32120
[   42.988924] remoteproc remoteproc15: remote processor 30086000.rtu is now up
[   42.988965] remoteproc remoteproc8: powering up 3008c000.txpru
[   42.990287] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 34280
[   42.990340] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[   42.991781] pps pps1: new PPS source ptp2
[   46.118172] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[   46.118246] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   47.141855] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[   47.141931] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[   48.133601] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
root@am64xx-evm:~# ping 192.168.2.21                                                                                                                                                                        
PING 192.168.2.21 (192.168.2.21): 56 data bytes
64 bytes from 192.168.2.21: seq=0 ttl=64 time=0.672 ms
64 bytes from 192.168.2.21: seq=1 ttl=64 time=0.510 ms
64 bytes from 192.168.2.21: seq=2 ttl=64 time=0.497 ms
64 bytes from 192.168.2.21: seq=3 ttl=64 time=0.540 ms
[   66.597674] icssg-prueth icssg1-eth eth2: Link is Down
[   71.718266] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
64 bytes from 192.168.2.21: seq=10 ttl=64 time=0.628 ms
64 bytes from 192.168.2.21: seq=11 ttl=64 time=0.494 ms
64 bytes from 192.168.2.21: seq=12 ttl=64 time=0.470 ms
64 bytes from 192.168.2.21: seq=13 ttl=64 time=0.440 ms
[   75.813861] icssg-prueth icssg1-eth eth1: Link is Down
[   80.935268] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
64 bytes from 192.168.2.21: seq=19 ttl=64 time=0.409 ms
64 bytes from 192.168.2.21: seq=20 ttl=64 time=0.478 ms
64 bytes from 192.168.2.21: seq=21 ttl=64 time=0.491 ms
^C
--- 192.168.2.21 ping statistics ---
22 packets transmitted, 11 packets received, 50% packet loss
round-trip min/avg/max = 0.409/0.511/0.672 ms
root@am64xx-evm:~# sh hsr_setup.sh hsr_hw eth1 eth2 192.168.2.20
ip=192.168.2.20
if=hsr0
mac=70:ff:76:1f:3d:c5
slave-a=eth1
slave-b=eth2
device=platform/icssg1-eth
[   93.763061] device eth1 entered promiscuous mode
[   93.763162] audit: type=1700 audit(1651488878.302:3): dev=eth1 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[   93.789041] device eth2 entered promiscuous mode
[   93.789142] audit: type=1700 audit(1651488878.302:3): dev=eth2 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[   93.856291] audit: type=1300 audit(1651488878.302:3): arch=c00000b7 syscall=211 success=yes exit=32 a0=3 a1=fffffc742328 a2=0 a3=ffff8f937840 items=0 ppid=1049 pid=1060 auid=4294967295 uid=0 gid=0 eui)
[   93.856330] audit: type=1327 audit(1651488878.302:3): proctitle=6970006C696E6B0064656C6574650068737230
[   93.904477] icssg-prueth icssg1-eth eth1: Link is Down
[   93.938172] icssg-prueth icssg1-eth eth2: Link is Down
[   93.942127] remoteproc remoteproc7: stopped remote processor 3008a000.txpru
[   93.942151] remoteproc remoteproc13: stopped remote processor 30084000.rtu
[   93.942160] remoteproc remoteproc12: stopped remote processor 300b4000.pru
[   93.942169] remoteproc remoteproc8: stopped remote processor 3008c000.txpru
[   93.942178] remoteproc remoteproc15: stopped remote processor 30086000.rtu
[   93.942186] remoteproc remoteproc14: stopped remote processor 300b8000.pru
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
[   95.105876] remoteproc remoteproc12: powering up 300b4000.pru
[   95.112017] remoteproc remoteproc12: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 40436
[   95.112086] remoteproc remoteproc12: unsupported resource 5
[   95.112116] remoteproc remoteproc12: remote processor 300b4000.pru is now up
[   95.112155] remoteproc remoteproc13: powering up 30084000.rtu
[   95.112327] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 32904
[   95.112360] remoteproc remoteproc13: remote processor 30084000.rtu is now up
[   95.112381] remoteproc remoteproc7: powering up 3008a000.txpru
[   95.112480] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 35772
[   95.112509] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[   95.112769] remoteproc remoteproc14: powering up 300b8000.pru
[   95.112875] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 40652
[   95.112891] remoteproc remoteproc14: unsupported resource 5
[   95.112914] remoteproc remoteproc14: remote processor 300b8000.pru is now up
[   95.112932] remoteproc remoteproc15: powering up 30086000.rtu
[   95.113171] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 32120
[   95.113205] remoteproc remoteproc15: remote processor 30086000.rtu is now up
[   95.113230] remoteproc remoteproc8: powering up 3008c000.txpru
[   95.113358] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 34280
[   95.113390] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[   95.130232] pps pps1: new PPS source ptp2
[   97.199297] icssg-prueth icssg1-eth: Both interfaces must be linked to same upper device
[   97.213356] icssg-prueth icssg1-eth: Both interfaces must be linked to same upper device
[   98.214160] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[   98.214233] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   99.237858] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[   99.237941] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[  100.253174] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
root@am64xx-evm:~# ping 192.168.2.21
PING 192.168.2.21 (192.168.2.21): 56 data bytes
64 bytes from 192.168.2.21: seq=0 ttl=64 time=0.728 ms
64 bytes from 192.168.2.21: seq=1 ttl=64 time=0.472 ms
64 bytes from 192.168.2.21: seq=2 ttl=64 time=0.427 ms
64 bytes from 192.168.2.21: seq=3 ttl=64 time=0.480 ms
[  108.453853] icssg-prueth icssg1-eth eth2: Link is Down
[  114.599926] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
64 bytes from 192.168.2.21: seq=10 ttl=64 time=0.570 ms
64 bytes from 192.168.2.21: seq=11 ttl=64 time=0.435 ms
[  116.645870] icssg-prueth icssg1-eth eth1: Link is Down
[  121.766439] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
64 bytes from 192.168.2.21: seq=17 ttl=64 time=0.445 ms
64 bytes from 192.168.2.21: seq=18 ttl=64 time=0.471 ms
^C
--- 192.168.2.21 ping statistics ---
19 packets transmitted, 8 packets received, 57% packet loss
round-trip min/avg/max = 0.427/0.503/0.728 ms
root@am64xx-evm:~# [  133.029556] icssg-prueth icssg1-eth eth1: Link is Down
[  133.035568] icssg-prueth icssg1-eth eth2: Link is Down
[  136.106454] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  138.149850] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

root@am64xx-evm:~# 
root@am64xx-evm:~# ping 192.168.2.21
PING 192.168.2.21 (192.168.2.21): 56 data bytes
64 bytes from 192.168.2.21: seq=0 ttl=64 time=0.869 ms
64 bytes from 192.168.2.21: seq=1 ttl=64 time=0.413 ms
64 bytes from 192.168.2.21: seq=2 ttl=64 time=0.500 ms
64 bytes from 192.168.2.21: seq=3 ttl=64 time=0.405 ms
64 bytes from 192.168.2.21: seq=4 ttl=64 time=0.498 ms
[  160.677657] icssg-prueth icssg1-eth eth2: Link is Down
64 bytes from 192.168.2.21: seq=5 ttl=64 time=0.423 ms
64 bytes from 192.168.2.21: seq=6 ttl=64 time=0.464 ms
64 bytes from 192.168.2.21: seq=7 ttl=64 time=0.378 ms
64 bytes from 192.168.2.21: seq=8 ttl=64 time=0.464 ms
64 bytes from 192.168.2.21: seq=9 ttl=64 time=0.431 ms
64 bytes from 192.168.2.21: seq=10 ttl=64 time=0.488 ms
[  166.821888] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
64 bytes from 192.168.2.21: seq=11 ttl=64 time=0.523 ms
64 bytes from 192.168.2.21: seq=12 ttl=64 time=0.430 ms
[  168.869660] icssg-prueth icssg1-eth eth1: Link is Down
64 bytes from 192.168.2.21: seq=13 ttl=64 time=0.461 ms
64 bytes from 192.168.2.21: seq=14 ttl=64 time=0.524 ms
64 bytes from 192.168.2.21: seq=15 ttl=64 time=0.432 ms
64 bytes from 192.168.2.21: seq=16 ttl=64 time=0.428 ms
64 bytes from 192.168.2.21: seq=17 ttl=64 time=0.443 ms
64 bytes from 192.168.2.21: seq=18 ttl=64 time=0.442 ms
64 bytes from 192.168.2.21: seq=19 ttl=64 time=0.434 ms
[  176.037663] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
64 bytes from 192.168.2.21: seq=20 ttl=64 time=0.423 ms
64 bytes from 192.168.2.21: seq=21 ttl=64 time=0.436 ms
64 bytes from 192.168.2.21: seq=22 ttl=64 time=0.552 ms
64 bytes from 192.168.2.21: seq=23 ttl=64 time=0.470 ms
^C
--- 192.168.2.21 ping statistics ---
24 packets transmitted, 24 packets received, 0% packet loss
round-trip min/avg/max = 0.378/0.472/0.869 ms
root@am64xx-evm:~# 

EVM 2 Console Log:

am64xx-evm login: [   19.445687] TI DP83869 300b2400.mdio:0f: attached PHY driver (mii_bus:phy_addr=300b2400.mdio:0f, irq=POLL)
[   19.473047] TI DP83869 mdio_mux-0.0:03: attached PHY driver (mii_bus:phy_addr=mdio_mux-0.0:03, irq=POLL)
[   19.473093] icssg-prueth icssg1-eth: TI PRU ethernet driver initialized: dual EMAC mode
[   19.626223] remoteproc remoteproc15: powering up 300b4000.pru
[   19.634146] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, size 40936
[   19.634195] remoteproc remoteproc15: unsupported resource 5
[   19.634224] remoteproc remoteproc15: remote processor 300b4000.pru is now up
[   19.634267] remoteproc remoteproc16: powering up 30084000.rtu
[   19.637735] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu0-prueth-fw.elf, size 30884
[   19.637795] remoteproc remoteproc16: remote processor 30084000.rtu is now up
[   19.637839] remoteproc remoteproc7: powering up 3008a000.txpru
[   19.639187] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, size 36672
[   19.639257] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[   19.639681] remoteproc remoteproc13: powering up 300b8000.pru
[   19.641092] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru1-prueth-fw.elf, size 41208
[   19.641136] remoteproc remoteproc13: unsupported resource 5
[   19.641166] remoteproc remoteproc13: remote processor 300b8000.pru is now up
[   19.641215] remoteproc remoteproc14: powering up 30086000.rtu
[   19.642727] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu1-prueth-fw.elf, size 30124
[   19.642790] remoteproc remoteproc14: remote processor 30086000.rtu is now up
[   19.642836] remoteproc remoteproc8: powering up 3008c000.txpru
[   19.644227] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-prueth-fw.elf, size 35184
[   19.644292] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[   19.646833] pps pps1: new PPS source ptp2

am64xx-evm login: root
[   22.856544] audit: type=1006 audit(1651974322.140:2): pid=1013 uid=0 old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=1 res=1
[   22.856633] audit: type=1300 audit(1651974322.140:2): arch=c00000b7 syscall=64 success=yes exit=1 a0=8 a1=ffffe60814c8 a2=1 a3=ffff94d99020 items=0 ppid=1 pid=1013 auid=0 uid=0 gid=0 euid=0 suid=0 fsu)
[   22.856682] audit: type=1327 audit(1651974322.140:2): proctitle="(systemd)"
root@am64xx-evm:~# [   23.779097] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[   23.779180] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   23.843077] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[   23.843157] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[   34.019063] icssg-prueth icssg1-eth eth1: Link is Down
[   34.083085] icssg-prueth icssg1-eth eth2: Link is Down

root@am64xx-evm:~# 
root@am64xx-evm:~# [   38.115088] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
reboot[   39.204638] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
^C
root@am64xx-evm:~# 
root@am64xx-evm:~# 
root@am64xx-evm:~# sh hsr_setup.sh hsr_hw eth1 eth2 192.168.2.21                                                                                                                                            
ip=192.168.2.21
if=hsr0
mac=70:ff:76:1f:3e:10
slave-a=eth1
slave-b=eth2
device=platform/icssg1-eth
[   58.475525] icssg-prueth icssg1-eth eth1: Link is Down
[   58.509966] icssg-prueth icssg1-eth eth2: Link is Down
[   58.516797] remoteproc remoteproc7: stopped remote processor 3008a000.txpru
[   58.516823] remoteproc remoteproc16: stopped remote processor 30084000.rtu
[   58.516833] remoteproc remoteproc15: stopped remote processor 300b4000.pru
[   58.516843] remoteproc remoteproc8: stopped remote processor 3008c000.txpru
[   58.516852] remoteproc remoteproc14: stopped remote processor 30086000.rtu
[   58.516862] remoteproc remoteproc13: stopped remote processor 300b8000.pru
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off
hsr-dup-offload: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off
hsr-dup-offload: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
[   59.656942] icssg-prueth icssg1-eth: Enabling HSR offload mode
[   59.692553] remoteproc remoteproc15: powering up 300b4000.pru
[   59.695908] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 40436
[   59.695953] remoteproc remoteproc15: unsupported resource 5
[   59.695982] remoteproc remoteproc15: remote processor 300b4000.pru is now up
[   59.696026] remoteproc remoteproc16: powering up 30084000.rtu
[   59.700411] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 32904
[   59.701616] remoteproc remoteproc16: remote processor 30084000.rtu is now up
[   59.701671] remoteproc remoteproc7: powering up 3008a000.txpru
[   59.704659] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 35772
[   59.704720] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[   59.705006] remoteproc remoteproc13: powering up 300b8000.pru
[   59.706318] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 40652
[   59.706359] remoteproc remoteproc13: unsupported resource 5
[   59.706388] remoteproc remoteproc13: remote processor 300b8000.pru is now up
[   59.706429] remoteproc remoteproc14: powering up 30086000.rtu
[   59.710742] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 32120
[   59.710802] remoteproc remoteproc14: remote processor 30086000.rtu is now up
[   59.710844] remoteproc remoteproc8: powering up 3008c000.txpru
[   59.712032] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 34280
[   59.712089] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[   59.715304] pps pps1: new PPS source ptp2
[   62.819576] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[   62.819655] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   63.843279] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[   63.843369] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[   64.823068] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
root@am64xx-evm:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 1c:63:49:1a:de:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.50.173/24 metric 1024 brd 192.168.50.255 scope global dynamic eth0
       valid_lft 515sec preferred_lft 515sec
    inet6 fe80::1e63:49ff:fe1a:de12/64 scope link 
       valid_lft forever preferred_lft forever
3: can0: <NOARP,ECHO> mtu 16 qdisc noop state DOWN group default qlen 10
    link/can 
4: can1: <NOARP,ECHO> mtu 16 qdisc noop state DOWN group default qlen 10
    link/can 
5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 70:ff:76:1f:3e:10 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::72ff:76ff:fe1f:3e10/64 scope link 
       valid_lft forever preferred_lft forever
6: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 70:ff:76:1f:3e:10 brd ff:ff:ff:ff:ff:ff permaddr 70:ff:76:1f:3e:12
    inet6 fe80::72ff:76ff:fe1f:3e10/64 scope link 
       valid_lft forever preferred_lft forever
7: hsr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1494 qdisc noqueue state UP group default qlen 1000
    link/ether 70:ff:76:1f:3e:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.21/24 scope global hsr0
       valid_lft forever preferred_lft forever
    inet6 fe80::72ff:76ff:fe1f:3e10/64 scope link 
       valid_lft forever preferred_lft forever
root@am64xx-evm:~# [  153.955578] icssg-prueth icssg1-eth eth1: Link is Down
[  153.955662] icssg-prueth icssg1-eth eth2: Link is Down
[  157.027590] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  158.051271] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  174.435575] icssg-prueth icssg1-eth eth1: Link is Down
[  174.435646] icssg-prueth icssg1-eth eth2: Link is Down
[  178.531061] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  179.555275] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

root@am64xx-evm:~# [  199.011272] icssg-prueth icssg1-eth eth2: Link is Down
[  204.131277] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  208.227093] icssg-prueth icssg1-eth eth1: Link is Down
[  213.347268] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  226.659184] icssg-prueth icssg1-eth eth2: Link is Down
[  226.659259] icssg-prueth icssg1-eth eth1: Link is Down
[  230.756171] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  231.779293] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  240.995266] icssg-prueth icssg1-eth eth2: Link is Down
[  247.141069] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  249.187074] icssg-prueth icssg1-eth eth1: Link is Down
[  253.283334] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

root@am64xx-evm:~# sh hsr_setup.sh hsr_hw eth1 eth2 192.168.2.21
ip=192.168.2.21
if=hsr0
mac=70:ff:76:1f:3e:10
slave-a=eth1
slave-b=eth2
device=platform/icssg1-eth
[  264.375483] device eth1 entered promiscuous mode
[  264.375592] audit: type=1700 audit(1651974563.656:3): dev=eth1 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[  264.403463] device eth2 entered promiscuous mode
[  264.403566] audit: type=1700 audit(1651974563.656:3): dev=eth2 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[  264.490342] audit: type=1300 audit(1651974563.656:3): arch=c00000b7 syscall=211 success=yes exit=32 a0=3 a1=ffffe4e74d58 a2=0 a3=ffff8c4e2840 items=0 ppid=1096 pid=1107 auid=4294967295 uid=0 gid=0 eui)
[  264.490381] audit: type=1327 audit(1651974563.656:3): proctitle=6970006C696E6B0064656C6574650068737230
[  264.552911] icssg-prueth icssg1-eth eth1: Link is Down
[  264.588584] icssg-prueth icssg1-eth eth2: Link is Down
[  264.595613] remoteproc remoteproc7: stopped remote processor 3008a000.txpru
[  264.595640] remoteproc remoteproc16: stopped remote processor 30084000.rtu
[  264.595650] remoteproc remoteproc15: stopped remote processor 300b4000.pru
[  264.595660] remoteproc remoteproc8: stopped remote processor 3008c000.txpru
[  264.595669] remoteproc remoteproc14: stopped remote processor 30086000.rtu
[  264.595678] remoteproc remoteproc13: stopped remote processor 300b8000.pru
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: on
hsr-dup-offload: on
[  265.740164] remoteproc remoteproc15: powering up 300b4000.pru
[  265.740392] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 40436
[  265.740418] remoteproc remoteproc15: unsupported resource 5
[  265.743466] remoteproc remoteproc15: remote processor 300b4000.pru is now up
[  265.743533] remoteproc remoteproc16: powering up 30084000.rtu
[  265.743711] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 32904
[  265.743750] remoteproc remoteproc16: remote processor 30084000.rtu is now up
[  265.743775] remoteproc remoteproc7: powering up 3008a000.txpru
[  265.743876] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 35772
[  265.743906] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[  265.744167] remoteproc remoteproc13: powering up 300b8000.pru
[  265.744271] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 40652
[  265.744288] remoteproc remoteproc13: unsupported resource 5
[  265.744311] remoteproc remoteproc13: remote processor 300b8000.pru is now up
[  265.744330] remoteproc remoteproc14: powering up 30086000.rtu
[  265.744427] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 32120
[  265.750594] remoteproc remoteproc14: remote processor 30086000.rtu is now up
[  265.750647] remoteproc remoteproc8: powering up 3008c000.txpru
[  265.750803] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 34280
[  265.750837] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[  265.753150] pps pps1: new PPS source ptp2
[  267.819630] icssg-prueth icssg1-eth: Both interfaces must be linked to same upper device
[  267.832398] icssg-prueth icssg1-eth: Both interfaces must be linked to same upper device
[  268.835070] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  268.835515] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[  269.859264] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  269.859342] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[  270.865413] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
root@am64xx-evm:~# [  292.387072] icssg-prueth icssg1-eth eth2: Link is Down
[  298.531269] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  300.579291] icssg-prueth icssg1-eth eth1: Link is Down
[  308.771361] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

"We 've tried to reproduce on our side the ping issue without multicast between 2 boards with hsr offloads and we observe there were no packets lost."

Based on your response, it appears that the double load of HSR offload issue described in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1255842/processor-sdk-am64x-hsr-load-firmware-issue/4752433#4752433 which is further indicated by https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/09_01_00_08/exports/docs/devices/AM64X/linux/Release_Specific_Release_Notes.html as you mentioned previously. If you could replicate the steps I have listed and still don't see any issue then it indicates to me that there may be something wrong with my setup.

-Daolin

Hi Daolin,

Thank you very much for the deatailed steps that you described earlier.

Finally, we've reproduce on our side the same behavior by following your steps to reproduce the bug. (packet lost with ping without multicast)

We also find that :

  When disable the network interfaces :

systemctl disable dhcpcd.service
systemctl disable NetworkManager.service
systemctl disable systemd-resolved.service
systemctl disable systemd-networkd.service
systemctl disable dnsmasq.service


reboot the board and, then redo the steps that you mentionned earlier, we didn't have any packet lost for pinging without hsr. 
Therefore, by putting the interfaces down  (not loading some fw in PRU before loading hsr offload fw), we do not have 
the hsr offload ping issue without multicast. 

 - Tianyi 

Hi Tianyi,

Thanks for clarifying that there is no packet loss only for the case of disabling the network interfaces/not loading PRU fw before HSR offload fw. This sounds to me like HSR offload doesn't work out-of-the-box as making sure the PRU fw is not loaded first is required for proper functionality. I've communicated this detail with the software development team.

This also clarifies that most likely the double load of HSR fw issue is not related to the multicast issue under discussion here. As mentioned before, the main cause for the multicast issue is that eth1 enters promiscuous mode when it should not be. These details have already been passed to the software development team and we are working to understand why promiscuous mode is entered and to get a fix in for the next SDK release. 

I will update you as more details come up.

-Daolin

Hello Tianyi, Milan,

Based on feedback from the software team, these bugs are a kernel framework related issue and requires 2-3 weeks of effort to fix which would likely not make it into the code freeze for the SDK 9.2 release. They are recommending that they can make a release with the fixes post SDK 9.2 as patches which you can pick up from our public git repo. 

I will ask them approximately how long post 9.2 release will the patches be available.

Is this acceptable for you?

-Daolin

Hi,

We will need to have this patches end of april 

Best regards

Milan

Hi Milan,

Thanks for accepting the patches solution. The software team has the issues specified in this thread as one of the top priorities to fix and will get the fixes out as soon as possible. They are aware of the end of April deadline.

-Daolin

Milan,

The fixes for this Multicast Filtering with HSR HW offload issue and the issue described in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1255842/processor-sdk-am64x-hsr-load-firmware-issue/4752433#4752433 (so no disabling of the network manager before loading the HSR firmware is required now) are now merged in ti-linux-6.1.y-cicd branch on git.ti.com. 

I tested the fix for Multicast Filtering issue at the commit: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/?h=ti-linux-6.1.y-cicd&id=902a51e8a5c8bf2f9efbd76a750665878dae063b with a new hsr script (not on SDK documentation yet) and verified the multicast issue is fixed. Below is my test sequence, the new hsr script, and resulting console log for the EVM (EVM1) that receiving multicast frames.

1. Two AM64x-EVM setup, connected eth1-eth1 and eth2-eth2 with both ports ICSSG_PRU ethernet

2. Run new hsr script for hw offloading twice on both EVMs

3. With no adding multicast MAC address, check no multicast frames received from EVM2 to EVM1 by running send application on EVM2. Do this with ifconfig hsr0 before and after running send application

3. Add the multicast MAC address that the send application sends to (it was 01:80:c4:00:00:0e on my setup) to EVM1

4. Run send application on EVM2 and observe increased RX packets in ifconfig hsr0 on EVM1

5. Delete the multicast MAC address that was added in step 3 from EVM1 and run send application again

6. Observe that there is no increase in RX packets on EVM1

// New HSR setup script

#!/bin/sh

  #For non offload - sh hsr_setup.sh hsr_sw <INTF_A> <INTF_B> <HSR_INTF_IP_ADDR>
  #For offload - sh hsr_setup.sh hsr_hw <INTF_A> <INTF_B> <HSR_INTF_IP_ADDR>

  if [ "$#" != "4" ]
  then
          echo "$0 <hsr_sw/hsr_hw> <intf1> <intf2> <ip addr>"
          exit
  fi

  if [ "$1" != "hsr_sw" ] && [ "$1" != "hsr_hw" ]
  then
          echo "$0 <hsr_sw|hsr_hw>"
          exit
  fi

  if=hsr0

  ifa=$2
  ifb=$3

  ip=$4
  mac=`ifconfig $ifa | grep ether | cut -d " " -f 10`

  device="platform/"
  device+=`dmesg | grep $ifa | grep icssg-prueth | grep -m 1 "Link is Up" | awk '{print $4}'`

  echo "ip=$ip"
  echo "if=$if"
  echo "mac=$mac"
  echo "slave-a=$ifa"
  echo "slave-b=$ifb"
  echo "device=$device"

  ip link delete hsr0  2> /dev/null

  ip link set $ifa down
  ip link set $ifb down
  sleep 1

  if [ "$1" == "hsr_hw" ]
  then
          ethtool -k $ifa | grep hsr
          ethtool -K $ifa hsr-fwd-offload on
          ethtool -K $ifa hsr-dup-offload on
          ethtool -k $ifa | grep hsr

          ethtool -k $ifb | grep hsr
          ethtool -K $ifb hsr-fwd-offload on
          ethtool -K $ifb hsr-dup-offload on
          ethtool -k $ifb | grep hsr

          devlink dev param set $device name hsr_offload_mode value true cmode runtime




  fi

  ip link set dev $ifa address $mac
  ip link set dev $ifb address $mac

  ip link set $ifa up
  sleep 1

  ip link set $ifb up
  sleep 1

  ip link add name $if type hsr slave1 $ifa slave2 $ifb supervision 45 version 1

  sleep 3

  ip addr add $ip/24 dev $if
  ip link set $if up

//EVM 1 Console Log

root@am64xx-evm:~# sh hsr_setup_new.sh hsr_hw eth1 eth2 192.168.2.20                                                                                                                                  
ip=192.168.2.20
if=hsr0
mac=70:ff:76:1f:3e:10
slave-a=eth1
slave-b=eth2
device=platform/icssg1-eth
[   47.527700] icssg-prueth icssg1-eth eth1: Link is Down
[   47.552209] icssg-prueth icssg1-eth eth2: Link is Down
[   47.563809] remoteproc remoteproc7: stopped remote processor 3008a000.txpru
[   47.570843] remoteproc remoteproc14: stopped remote processor 30084000.rtu
[   47.577758] remoteproc remoteproc13: stopped remote processor 300b4000.pru
[   47.584658] remoteproc remoteproc8: stopped remote processor 3008c000.txpru
[   47.591659] remoteproc remoteproc16: stopped remote processor 30086000.rtu
[   47.598555] remoteproc remoteproc15: stopped remote processor 300b8000.pru
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: off
hsr-dup-offload: off
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: off
hsr-dup-offload: off
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: on
hsr-dup-offload: on
[   48.695698] icssg-prueth icssg1-eth: Enabling HSR offload mode
[   48.729983] remoteproc remoteproc13: powering up 300b4000.pru
[   48.739118] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 40436
[   48.748561] remoteproc remoteproc13: unsupported resource 5
[   48.754168] remoteproc remoteproc13: remote processor 300b4000.pru is now up
[   48.761269] remoteproc remoteproc14: powering up 30084000.rtu
[   48.770221] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 32904
[   48.779684] remoteproc remoteproc14: remote processor 30084000.rtu is now up
[   48.786771] remoteproc remoteproc7: powering up 3008a000.txpru
[   48.795774] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 35772
[   48.805306] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[   48.812722] remoteproc remoteproc15: powering up 300b8000.pru
[   48.821705] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 40652
[   48.831139] remoteproc remoteproc15: unsupported resource 5
[   48.836744] remoteproc remoteproc15: remote processor 300b8000.pru is now up
[   48.843840] remoteproc remoteproc16: powering up 30086000.rtu
[   48.852747] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 32120
[   48.862202] remoteproc remoteproc16: remote processor 30086000.rtu is now up
[   48.869288] remoteproc remoteproc8: powering up 3008c000.txpru
[   48.878212] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 34280
[   48.887747] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[   48.897026] pps pps1: new PPS source ptp2
[   51.998774] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[   52.006627] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   53.022436] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[   53.030271] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[   53.988839] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
root@am64xx-evm:~# [   59.166628] icssg-prueth icssg1-eth eth1: Link is Down
[   59.166796] icssg-prueth icssg1-eth eth2: Link is Down
[   63.262712] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[   64.286414] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

root@am64xx-evm:~# ifconfig hsr0
hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494
        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::72ff:76ff:fe1f:3e10  prefixlen 64  scopeid 0x20<link>
        ether 70:ff:76:1f:3e:10  txqueuelen 1000  (Ethernet)
        RX packets 25  bytes 3440 (3.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 69  bytes 7100 (6.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@am64xx-evm:~# ip maddr show dev hsr0                                                                                                                                        
7: hsr0
   link  33:33:00:00:00:01 users 3
   link  01:00:5e:00:00:01 users 3
   link  33:33:ff:1f:3e:10 users 3
   link  01:00:5e:00:00:fb users 3
   link  33:33:00:00:00:fb users 3
   link  01:00:5e:00:00:fc users 3
   link  33:33:00:01:00:03 users 3
   inet  224.0.0.252
   inet  224.0.0.251
   inet  224.0.0.1
   inet6 ff02::1:3
   inet6 ff02::fb
   inet6 ff02::1:ff1f:3e10
   inet6 ff02::1
   inet6 ff01::1
root@am64xx-evm:~# 
root@am64xx-evm:~# ifconfig hsr0
hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494
        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::72ff:76ff:fe1f:3e10  prefixlen 64  scopeid 0x20<link>
        ether 70:ff:76:1f:3e:10  txqueuelen 1000  (Ethernet)
        RX packets 25  bytes 3440 (3.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 77  bytes 7590 (7.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@am64xx-evm:~# ip maddr show dev hsr0                                                                                                                                                                   
7: hsr0
   link  33:33:00:00:00:01 users 3
   link  01:00:5e:00:00:01 users 3
   link  33:33:ff:1f:3e:10 users 3
   link  01:00:5e:00:00:fb users 3
   link  33:33:00:00:00:fb users 3
   link  01:00:5e:00:00:fc users 3
   link  33:33:00:01:00:03 users 3
   inet  224.0.0.252
   inet  224.0.0.251
   inet  224.0.0.1
   inet6 ff02::1:3
   inet6 ff02::fb
   inet6 ff02::1:ff1f:3e10
   inet6 ff02::1
   inet6 ff01::1
root@am64xx-evm:~# [  107.294308] icssg-prueth icssg1-eth eth1: Link is Down
[  107.294384] icssg-prueth icssg1-eth eth2: Link is Down

root@am64xx-evm:~# 
(reverse-i-search)`sh ': sh hsr_setup_new.[  111.390718] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off                                                                           
[  112.414506] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
^C
root@am64xx-evm:~# sh hsr_setup_new.sh hsr_hw eth1 eth2 192.168.2.20                                                                                                                                  
ip=192.168.2.20
if=hsr0
mac=70:ff:76:1f:3e:10
slave-a=eth1
slave-b=eth2
device=platform/icssg1-eth
[  118.777325] icssg-prueth icssg1-eth eth1: Link is Down
[  118.811170] icssg-prueth icssg1-eth eth2: Link is Down
[  118.820500] remoteproc remoteproc7: stopped remote processor 3008a000.txpru
[  118.827547] remoteproc remoteproc14: stopped remote processor 30084000.rtu
[  118.834447] remoteproc remoteproc13: stopped remote processor 300b4000.pru
[  118.841339] remoteproc remoteproc8: stopped remote processor 3008c000.txpru
[  118.848309] remoteproc remoteproc16: stopped remote processor 30086000.rtu
[  118.855201] remoteproc remoteproc15: stopped remote processor 300b8000.pru
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: on
hsr-dup-offload: on
[  119.954768] remoteproc remoteproc13: powering up 300b4000.pru
[  119.960791] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 40436
[  119.970249] remoteproc remoteproc13: unsupported resource 5
[  119.975866] remoteproc remoteproc13: remote processor 300b4000.pru is now up
[  119.983004] remoteproc remoteproc14: powering up 30084000.rtu
[  119.988965] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 32904
[  119.998464] remoteproc remoteproc14: remote processor 30084000.rtu is now up
[  120.005557] remoteproc remoteproc7: powering up 3008a000.txpru
[  120.011552] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 35772
[  120.021083] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[  120.028507] remoteproc remoteproc15: powering up 300b8000.pru
[  120.034450] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 40652
[  120.043894] remoteproc remoteproc15: unsupported resource 5
[  120.049508] remoteproc remoteproc15: remote processor 300b8000.pru is now up
[  120.056615] remoteproc remoteproc16: powering up 30086000.rtu
[  120.062520] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 32120
[  120.071946] remoteproc remoteproc16: remote processor 30086000.rtu is now up
[  120.079125] remoteproc remoteproc8: powering up 3008c000.txpru
[  120.085181] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 34280
[  120.094742] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[  120.104010] pps pps1: new PPS source ptp2
[  123.166723] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  123.174562] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[  124.190429] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  124.198256] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[  125.193928] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
root@am64xx-evm:~# 
root@am64xx-evm:~# 
root@am64xx-evm:~# ip maddr show dev hsr0
8: hsr0
   link  33:33:00:00:00:01 users 2
   link  01:00:5e:00:00:01 users 2
   link  01:00:5e:00:00:fc users 2
   link  33:33:ff:1f:3e:10 users 2
   link  01:00:5e:00:00:fb users 2
   link  33:33:00:01:00:03 users 2
   link  33:33:00:00:00:fb users 2
   inet  224.0.0.251
   inet  224.0.0.252
   inet  224.0.0.1
   inet6 ff02::fb
   inet6 ff02::1:3
   inet6 ff02::1:ff1f:3e10
   inet6 ff02::1
   inet6 ff01::1
root@am64xx-evm:~# ifconfig hsr0                                                                                                                                        
hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494
        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::72ff:76ff:fe1f:3e10  prefixlen 64  scopeid 0x20<link>
        ether 70:ff:76:1f:3e:10  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 12  overruns 0  frame 0
        TX packets 42  bytes 5098 (4.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@am64xx-evm:~# ifconfig hsr0                                                                                                                                                                   
hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494
        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::72ff:76ff:fe1f:3e10  prefixlen 64  scopeid 0x20<link>
        ether 70:ff:76:1f:3e:10  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 12  overruns 0  frame 0
        TX packets 49  bytes 5528 (5.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@am64xx-evm:~# ip maddr add 01:80:c4:00:00:0e dev hsr0                                                                                                                                   
root@am64xx-evm:~# ifconfig hsr0
hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494
        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::72ff:76ff:fe1f:3e10  prefixlen 64  scopeid 0x20<link>
        ether 70:ff:76:1f:3e:10  txqueuelen 1000  (Ethernet)
        RX packets 500  bytes 25000 (24.4 KiB)
        RX errors 0  dropped 12  overruns 0  frame 0
        TX packets 59  bytes 6138 (5.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@am64xx-evm:~# ip maddr del^C1:80:c4:00:00:0e dev hsr0                                                                                                                                                  
root@am64xx-evm:~# ip maddr show dev hsr0                                                                                                                                                                   
8: hsr0
   link  33:33:00:00:00:01 users 2
   link  01:00:5e:00:00:01 users 2
   link  01:00:5e:00:00:fc users 2
   link  33:33:ff:1f:3e:10 users 2
   link  01:00:5e:00:00:fb users 2
   link  33:33:00:01:00:03 users 2
   link  33:33:00:00:00:fb users 2
   link  01:80:c4:00:00:0e users 3 static
   inet  224.0.0.251
   inet  224.0.0.252
   inet  224.0.0.1
   inet6 ff02::fb
   inet6 ff02::1:3
   inet6 ff02::1:ff1f:3e10
   inet6 ff02::1
   inet6 ff01::1
root@am64xx-evm:~# ip maddr show dev eth1                                                                                                                                                                   
5: eth1
   link  01:00:5e:00:00:01 users 3
   link  33:33:00:00:00:01 users 3
   link  33:33:ff:1f:3e:10 users 3
   link  33:33:00:00:00:fb users 3
   link  01:00:5e:00:00:fb users 2
   link  01:00:5e:00:00:fc users 2
   link  33:33:00:01:00:03 users 3
   link  01:80:c2:00:00:00
   link  01:80:c2:00:00:03
   link  01:80:c2:00:00:0e
   link  01:80:c4:00:00:0e
   inet  224.0.0.1
   inet6 ff02::1:3
   inet6 ff02::fb
   inet6 ff02::1:ff1f:3e10
   inet6 ff02::1 users 2
   inet6 ff01::1
root@am64xx-evm:~# ip maddr show dev eth2
6: eth2
   link  01:00:5e:00:00:01 users 2
   link  33:33:00:00:00:01 users 2
   link  33:33:ff:1f:3e:10 users 2
   link  01:00:5e:00:00:fb
   link  33:33:00:00:00:fb users 2
   link  01:00:5e:00:00:fc
   link  33:33:00:01:00:03 users 2
   link  01:80:c2:00:00:00
   link  01:80:c2:00:00:03
   link  01:80:c2:00:00:0e
   link  01:80:c4:00:00:0e
   inet  224.0.0.1
   inet6 ff02::1:3
   inet6 ff02::fb
   inet6 ff02::1:ff1f:3e10
   inet6 ff02::1 users 2
   inet6 ff01::1
root@am64xx-evm:~# ip maddr del 01:80:c4:00:00:0e dev hsr0                                                                                                                                                  
root@am64xx-evm:~# ip maddr show dev hsr0
8: hsr0
   link  33:33:00:00:00:01 users 2
   link  01:00:5e:00:00:01 users 2
   link  01:00:5e:00:00:fc users 2
   link  33:33:ff:1f:3e:10 users 2
   link  01:00:5e:00:00:fb users 2
   link  33:33:00:01:00:03 users 2
   link  33:33:00:00:00:fb users 2
   inet  224.0.0.251
   inet  224.0.0.252
   inet  224.0.0.1
   inet6 ff02::fb
   inet6 ff02::1:3
   inet6 ff02::1:ff1f:3e10
   inet6 ff02::1
   inet6 ff01::1
root@am64xx-evm:~# ip maddr show dev eth1                                                                                                                                                  
5: eth1
   link  01:00:5e:00:00:01 users 3
   link  33:33:00:00:00:01 users 3
   link  33:33:ff:1f:3e:10 users 3
   link  33:33:00:00:00:fb users 3
   link  01:00:5e:00:00:fb users 2
   link  01:00:5e:00:00:fc users 2
   link  33:33:00:01:00:03 users 3
   link  01:80:c2:00:00:00
   link  01:80:c2:00:00:03
   link  01:80:c2:00:00:0e
   inet  224.0.0.1
   inet6 ff02::1:3
   inet6 ff02::fb
   inet6 ff02::1:ff1f:3e10
   inet6 ff02::1 users 2
   inet6 ff01::1
root@am64xx-evm:~# ip maddr show dev eth2                                                                                                                                                  
6: eth2
   link  01:00:5e:00:00:01 users 2
   link  33:33:00:00:00:01 users 2
   link  33:33:ff:1f:3e:10 users 2
   link  01:00:5e:00:00:fb
   link  33:33:00:00:00:fb users 2
   link  01:00:5e:00:00:fc
   link  33:33:00:01:00:03 users 2
   link  01:80:c2:00:00:00
   link  01:80:c2:00:00:03
   link  01:80:c2:00:00:0e
   inet  224.0.0.1
   inet6 ff02::1:3
   inet6 ff02::fb
   inet6 ff02::1:ff1f:3e10
   inet6 ff02::1 users 2
   inet6 ff01::1
root@am64xx-evm:~# ifconfig hsr0                                                                                                                                                  
hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494
        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::72ff:76ff:fe1f:3e10  prefixlen 64  scopeid 0x20<link>
        ether 70:ff:76:1f:3e:10  txqueuelen 1000  (Ethernet)
        RX packets 500  bytes 25000 (24.4 KiB)
        RX errors 0  dropped 12  overruns 0  frame 0
        TX packets 104  bytes 8858 (8.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@am64xx-evm:~# ifconfig hsr0
hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494
        inet 192.168.2.20  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::72ff:76ff:fe1f:3e10  prefixlen 64  scopeid 0x20<link>
        ether 70:ff:76:1f:3e:10  txqueuelen 1000  (Ethernet)
        RX packets 500  bytes 25000 (24.4 KiB)
        RX errors 0  dropped 12  overruns 0  frame 0
        TX packets 107  bytes 9038 (8.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@am64xx-evm:~# 

Please give this a try on your setup and see if the issue has been fixed.

-Daolin

Hi Daolin,

Thanks for info.. We will try this and provide feedback... I assume we will have something during next week

Hello Daolin,

Thank you for the sharing !

We tested on our side with your patch and your new script. It does resolve the loading twice hsr offload firmware. However with the same setUp, when we load once the hsr offload firmware, it doesn't anymore.

Here is the setUp :

We are using the rt-kernel with sha 3c08dbfd7b : https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/log/?h=ti-rt-linux-6.1.y-cicd

Before starting the test, we disable the network part by doing (this part is important for us)

systemctl disable dhcpcd.service
systemctl disable NetworkManager.service
systemctl disable systemd-resolved.service
systemctl disable systemd-networkd.service
sync
reboot

When you reboot the board, and do ifconfig, you should have this output :

am64xx-evm login: root

root@am64xx-evm:~# ifconfig

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10<host>

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 80  bytes 6320 (6.1 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 80  bytes 6320 (6.1 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

We need to disable network for both EVM1 and EVM2.

Then we did :

1. Run new hsr script for hw offloading ONCE on both EVMs

2. Add the multicast MAC address that the send application sends to (it was 01:00:5e:00:00:05 on my setup) to EVM1

3. Check RX packets in ifconfig hsr0 on EVM1

4. Run send application on EVM2

5. Recheck RX packets in ifconfig hsr0 on EVM1

RX packets in ifconfig hsr0 on EVM1 is not increasing.

It seems, there is issue when loading ONCE the hsr_offload firmware. Could you reproduce this case on your side ?

Tianyi

Hi Tianyi,

Thanks for sharing your findings!

I was under the impression that the steps to disable the network was to work around the original twice load hsr offload firmware bug. This twice load of hsr offload firmware bug should now be fixed and you should not need to disable the network for hsr offload to work. May I know why disabling the network is still important to do?

Just to clarify, do you still see the issue you are describing when the network was not disabled on both EVMs?

I can try to reproduce this on Monday or Tuesday next week and will get back to you on my findings.

-Daolin

Hi Daolin,

Thank you for the fast answer !

The step to disable the network is not a workaround of the original twice load hsr offload. It's one of the requirement of our customer ! Therefore, for each hsr offload features, we test the HSR firmware by

- disabling the network (using the script given above)

- not disabling the network (up the interfaces eth1 and eth2 before launching the script to offload hsr feature)

As you requested when no disabling the network, multicast does work for for loading once or twice the fw ! However, I just discover that when not disable the network (up the interfaces eth1 and eth2 before launching the script to offload hsr feature), I have some packets lost when pinging between two boards :

Here is the commands on EVM1 :

root@am64xx-evm:~# ifconfig eth1 up 

[   84.208379] remoteproc remoteproc10: unsupported resource 5

[   84.218100] remoteproc remoteproc12: unsupported resource 5

root@am64xx-evm:~# ifconfig eth2 up                                                                  root@am64xx-evm:~# ./hsr_setup.sh hsr_hw eth1 eth2 192.168.200.1

ip=192.168.200.1

if=hsr0

mac=70:ff:76:1e:2e:2d

slave-a=eth1

slave-b=eth2

device=platform/icssg1-eth

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: on

hsr-dup-offload: on

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: on

hsr-dup-offload: on

[   92.952371] remoteproc remoteproc10: unsupported resource 5

[   92.964526] remoteproc remoteproc12: unsupported resource 5

root@am64xx-evm:~# ping 192.168.200.2                              

PING 192.168.200.2 (192.168.200.2): 56 data bytes

64 bytes from 192.168.200.2: seq=0 ttl=64 time=2032.508 ms

64 bytes from 192.168.200.2: seq=3 ttl=64 time=0.448 ms

64 bytes from 192.168.200.2: seq=6 ttl=64 time=0.440 ms

64 bytes from 192.168.200.2: seq=9 ttl=64 time=0.359 ms

^C

--- 192.168.200.2 ping statistics ---

11 packets transmitted, 4 packets received, 63% packet loss

round-trip min/avg/max = 0.359/508.438/2032.508 ms

Here is the output on EVM2 :

am64xx-evm login: root

root@am64xx-evm:~# ifconfig eth1 up

[   23.115217] remoteproc remoteproc9: unsupported resource 5

[   23.121493] remoteproc remoteproc11: unsupported resource 5

root@am64xx-evm:~# ifconfig eth  up                                                                                                2

root@am64xx-evm:~# ifconfig

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet6 fe80::72ff:76ff:fe1e:2e00  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:2e:00  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 6  bytes 516 (516.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet6 fe80::72ff:76ff:fe1e:2e02  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:2e:02  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 6  bytes 516 (516.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10<host>

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 80  bytes 6320 (6.1 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 80  bytes 6320 (6.1 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



(reverse-i-search)`. ': ./hsr_setup.sh hsr_hw eth1 eth2 192.168.200.2                               root@am64xx-evm:~# 

ip=192.168.200.2

if=hsr0

mac=70:ff:76:1e:2e:00

slave-a=eth1

slave-b=eth2

device=platform/icssg1-eth

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: on

hsr-dup-offload: on

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: on

hsr-dup-offload: on

[   57.990828] remoteproc remoteproc9: unsupported resource 5

[   58.001714] remoteproc remoteproc11: unsupported resource 5

root@am64xx-evm:~# [  254.829333] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[  255.837089] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

To resume :

when no disable the network :

- multicast is working (loading once or twice the hsr fw)!

- ping between two boards lead to some packets lost

when disable the network :

- multicast is working when loading twice

- multicast is not working when loading once

- ping between two boards is working

Tianyi

Hi Tianyi,

Thanks for sharing additional information. Tomorrow I will try to reproduce the two issues you specified and come back with an update.

-Daolin

Hi Daolin,

Thank you for your fast response !

We also find an issue when loading once hsr offload by disable the network.

I create a new thread to separate from this one (Here is the link with more details : https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1341782/processor-sdk-am64x-hsr-offload-pru-firmware-functionalities-issue )

Tianyi

Hi Tianyi,

Tianyi Liu said:

when no disable the network :

- multicast is working (loading once or twice the hsr fw)!

- ping between two boards lead to some packets lost

I noticed that when you run "ifconfig eth1 up" there is a kernel message of the following and it doesn't look like there is a kernel message for the "ip link set eth1 down" part of the hsr setup script. I'm expecting to see something along the lines of "icssg-prueth icssg1-eth eth1: Link is Down". Along with these differences, I tried to replicate the ping loss with no disable of the network and I do not see any packet losses (see below console log). Please note that when I say I tested with no disable of the network, I mean using the default environment where the PRU firmware is loaded during boot-up. Can you verify that the PRU firmware is loaded on your test of no disabling the network?

[ 84.208379] remoteproc remoteproc10: unsupported resource 5

[ 84.218100] remoteproc remoteproc12: unsupported resource 5

//EVM 1
Arago 2023.10 am64xx-evm -

am64xx-evm login: [   30.437663] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[   30.442801] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[   45.925650] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   45.927558] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

am64xx-evm login: 
am64xx-evm login: 
am64xx-evm login: root
[   60.481727] audit: type=1006 audit(1653256172.851:2): pid=969 uid=0 old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=1 res=1
[   60.481906] audit: type=1300 audit(1653256172.851:2): arch=c00000b7 syscall=64 success=yes exit=1 a0=8 a1=ffffe40832c8 a2=1 a3=ffff99b3e020 items=0 ppid=1 pid=969 auid=0 uid=0 gid=0 euid=0 suid=0 fsui)
[   60.485423] audit: type=1327 audit(1653256172.851:2): proctitle="(systemd)"
root@am64xx-evm:~# vi configure_network.sh 
root@am64xx-evm:~# [   95.973637] icssg-prueth icssg1-eth eth2: Link is Down
[   97.125631] icssg-prueth icssg1-eth eth1: Link is Down

root@am64xx-evm:~# ifconfig -s
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0             1500        0      0      0 0             0      0      0      0 BMU
eth1             1500        0      0      0 0            26      0      0      0 BMU
eth2             1500        0      0      0 0            27      0      0      0 BMU
lo              65536       93      0      0 0            93      0      0      0 LRU
root@am64xx-evm:~# uname -a 
Linux am64xx-evm 6.1.80-rt26-00021-g8dfe902f9207 #2 SMP PREEMPT_RT Tue Mar 26 09:55:56 CDT 2024 aarch64 aarch64 aarch64 GNU/Linux
root@am64xx-evm:~# ifconfig eth1 up
root@am64xx-evm:~# ifconfig eth2 up
root@am64xx-evm:~# [  469.733632] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  472.934192] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

root@am64xx-evm:~# 
root@am64xx-evm:~# sh hsr_setup_new.sh hsr_hw eth1 eth2 192.168.2.20                                                                                                                               
ip=192.168.2.20
if=hsr0
mac=70:ff:76:1f:3d:c5
slave-a=eth1
slave-b=eth2
device=platform/icssg1-eth
[  498.888085] icssg-prueth icssg1-eth eth1: Link is Down
[  498.925164] icssg-prueth icssg1-eth eth2: Link is Down
[  498.929292] remoteproc remoteproc7: stopped remote processor 3008a000.txpru
[  498.929314] remoteproc remoteproc16: stopped remote processor 30084000.rtu
[  498.929322] remoteproc remoteproc15: stopped remote processor 300b4000.pru
[  498.929331] remoteproc remoteproc8: stopped remote processor 3008c000.txpru
[  498.929340] remoteproc remoteproc13: stopped remote processor 30086000.rtu
[  498.929348] remoteproc remoteproc12: stopped remote processor 300b8000.pru
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: off
hsr-dup-offload: off
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: off
hsr-dup-offload: off
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: on
hsr-dup-offload: on
[  500.064349] icssg-prueth icssg1-eth: Enabling HSR offload mode
[  500.099837] remoteproc remoteproc15: powering up 300b4000.pru
[  500.104405] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 40436
[  500.104445] remoteproc remoteproc15: unsupported resource 5
[  500.104474] remoteproc remoteproc15: remote processor 300b4000.pru is now up
[  500.104514] remoteproc remoteproc16: powering up 30084000.rtu
[  500.108798] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 32904
[  500.108857] remoteproc remoteproc16: remote processor 30084000.rtu is now up
[  500.108903] remoteproc remoteproc7: powering up 3008a000.txpru
[  500.115089] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 35772
[  500.115147] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[  500.115427] remoteproc remoteproc12: powering up 300b8000.pru
[  500.117152] remoteproc remoteproc12: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 40652
[  500.117191] remoteproc remoteproc12: unsupported resource 5
[  500.117220] remoteproc remoteproc12: remote processor 300b8000.pru is now up
[  500.117261] remoteproc remoteproc13: powering up 30086000.rtu
[  500.121212] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 32120
[  500.121265] remoteproc remoteproc13: remote processor 30086000.rtu is now up
[  500.121305] remoteproc remoteproc8: powering up 3008c000.txpru
[  500.122472] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 34280
[  500.122528] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[  500.123988] pps pps1: new PPS source ptp2
[  503.206146] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  503.206226] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[  504.229837] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  504.229917] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[  505.249740] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
root@am64xx-evm:~# 
(failed reverse-i-search)`': ^C
root@am64xx-evm:~# [  540.069752] icssg-prueth icssg1-eth eth2: Link is Down
[  540.069829] icssg-prueth icssg1-eth eth1: Link is Down
[  543.142153] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  545.189842] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

root@am64xx-evm:~# 
root@am64xx-evm:~# ping 192.168.2.21                                                                                                                                                                 
PING 192.168.2.21 (192.168.2.21): 56 data bytes
64 bytes from 192.168.2.21: seq=0 ttl=64 time=0.775 ms
64 bytes from 192.168.2.21: seq=1 ttl=64 time=0.483 ms
64 bytes from 192.168.2.21: seq=2 ttl=64 time=0.499 ms
64 bytes from 192.168.2.21: seq=3 ttl=64 time=0.464 ms
64 bytes from 192.168.2.21: seq=4 ttl=64 time=0.505 ms
64 bytes from 192.168.2.21: seq=5 ttl=64 time=0.448 ms
64 bytes from 192.168.2.21: seq=6 ttl=64 time=0.471 ms
[  599.461849] icssg-prueth icssg1-eth eth2: Link is Down
64 bytes from 192.168.2.21: seq=7 ttl=64 time=0.620 ms
64 bytes from 192.168.2.21: seq=8 ttl=64 time=0.480 ms
64 bytes from 192.168.2.21: seq=9 ttl=64 time=0.480 ms
64 bytes from 192.168.2.21: seq=10 ttl=64 time=0.457 ms
64 bytes from 192.168.2.21: seq=11 ttl=64 time=0.458 ms
64 bytes from 192.168.2.21: seq=12 ttl=64 time=0.458 ms
64 bytes from 192.168.2.21: seq=13 ttl=64 time=0.476 ms
[  606.629842] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
64 bytes from 192.168.2.21: seq=14 ttl=64 time=0.470 ms
64 bytes from 192.168.2.21: seq=15 ttl=64 time=0.466 ms
64 bytes from 192.168.2.21: seq=16 ttl=64 time=0.441 ms
64 bytes from 192.168.2.21: seq=17 ttl=64 time=0.461 ms
64 bytes from 192.168.2.21: seq=18 ttl=64 time=0.415 ms
64 bytes from 192.168.2.21: seq=19 ttl=64 time=0.460 ms
[  612.773829] icssg-prueth icssg1-eth eth1: Link is Down
64 bytes from 192.168.2.21: seq=20 ttl=64 time=0.495 ms
64 bytes from 192.168.2.21: seq=21 ttl=64 time=0.486 ms
64 bytes from 192.168.2.21: seq=22 ttl=64 time=0.474 ms
64 bytes from 192.168.2.21: seq=23 ttl=64 time=0.483 ms
64 bytes from 192.168.2.21: seq=24 ttl=64 time=0.394 ms
64 bytes from 192.168.2.21: seq=25 ttl=64 time=0.485 ms
64 bytes from 192.168.2.21: seq=26 ttl=64 time=0.439 ms
[  619.941844] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
64 bytes from 192.168.2.21: seq=27 ttl=64 time=0.453 ms
64 bytes from 192.168.2.21: seq=28 ttl=64 time=0.467 ms
64 bytes from 192.168.2.21: seq=29 ttl=64 time=0.466 ms
64 bytes from 192.168.2.21: seq=30 ttl=64 time=0.469 ms
^C
--- 192.168.2.21 ping statistics ---
31 packets transmitted, 31 packets received, 0% packet loss
round-trip min/avg/max = 0.394/0.480/0.775 ms
root@am64xx-evm:~# 
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.8 | VT102 | Offlin

Technically without disabling the network services, the PRU firmware should load on boot up and the eth1 and eth2 interfaces should already be up with the PRU firmware load, so there is no need to bring up eth1 and eth2 with "ifconfig ethX up". Additionally, while ifconfig ethX up does work, please note that ifconfig is being depreciated and the more up to date command to bring up (and down) interfaces is "ip link set ethX up". You can also give "ip link set ethX up" a try for eth1 and eth2 to see if you still see ping packet loss.

>>>

when disable the network :

- multicast is working when loading twice

- multicast is not working when loading once

- ping between two boards is working

>>>>

I write updates on this in another response here

>>>

when disable the network :

- multicast is working when loading twice

- multicast is not working when loading once

- ping between two boards is working

>>>>

I can see multicast is not working with disable of the network with the systemctl commands you shared when HSR offload is set up once (no packets received in hsr0 interface). I think this might be related to the issue reported in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1341782/processor-sdk-am64x-hsr-offload-pru-firmware-functionalities-issue that you created, where the basic functionality of HSR is not working with disable of the network. I think the HSR offload is not being setup properly with the network services being disabled and perhaps running the setup twice allows the HSR offload to be setup properly the second time. I need to check with the team regarding any potential network services that HSR setup requires to be running in order to be configured properly.

-Daolin

Hello Daolin,

Thank you for the feedback !

We also notice on https://git.ti.com/cgit/processor-firmware/ti-linux-firmware/log/ti-pruss?h=ti-linux-firmware-next some updates for icssg hsr firmware. As you told us to use ti kernel ti-rt-linux-6.1.y-cicd , should we also update the icssg hsr firmware ?

-Tianyi

Hi Tianyi,

Thanks for noticing the ICSSG HSR firmwares updates!

In fact, I just receiving feedback from the software developer that the latest firmwares for ICSSG HSR are committed in https://git.ti.com/cgit/processor-firmware/ti-linux-firmware/commit/?h=ti-linux-firmware&id=3bf8a3177810aec64a41d3a20af507ab0becfc7c. Apologies as I did not know about these firmware updates until feedback from the software team. 

Additionally, assuming your requirement of disabling the network services is mainly to make sure no ethernet interface is up before HSR firmwares are loaded, the issues being observed may be due to missing the device name (usually "platform/icssg1-eth"), which the hsr setup script requires to set up HSR properly. I'm assuming that the console log output from running the hsr setup script for you contains something along the lines of

slave-b=eth2
device=platform/
...
[   54.207836] remoteproc remoteproc9: powering up 300b4000.pru
[   54.213723] remoteproc remoteproc9: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, size 40936
[   54.213762] remoteproc remoteproc9: unsupported resource 5
...

These messages are what showed up for me while replicating your steps to start HSR without network services. As you can see "device=platform/" which is not enough for "devlink dev param set $device name hsr_offload_mode value true cmode runtime" in the hsr setup script to setup HSR properly. The hsr setup script uses the following to grab the device name and since there is no kernel message of "icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off", the script isn't able to find the correct device name.

device="platform/"
device+=`dmesg | grep $ifa | grep icssg-prueth | grep -m 1 "Link is Up" | awk ' {print $4}'`

Additionally, "Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf" indicates that HSR firmware was not loaded at all and the MAC mode firmware was loaded.

To achieve proper functionality with your requirement, you can try hardcoding the device name in the hsr setup script. Something like modifying the hsr setup script with device="platform/icssg1-eth" assuming icssg1-eth is the "device name".

Please give it a try and let us know if you encounter issues.

-Daolin

Hi Daolin,

Thank you for the update !

I just hardcoded with device="platform/icssg1-eth"

The good new is the hsr offload is enable and the packets are duplicated in both eth1 and eth2 (checked with command ethtool -S eth2 | grep -i tx_good_frames and ethtool -S eth1 | grep -i tx_good_frame ). However, we have some packet lost with ping!

am64xx-evm login: root

root@am64xx-evm:~# ./hsr_setup.sh hsr_hw eth1 eth2 192.168.200.2                             

ip=192.168.200.2

if=hsr0

mac=70:ff:76:1e:2e:2d

slave-a=eth1

slave-b=eth2

device=platform/icssg1-eth

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: on

hsr-dup-offload: on

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: on

hsr-dup-offload: on

[   39.911249] remoteproc remoteproc10: unsupported resource 5

[   39.916544] remoteproc remoteproc12: unsupported resource 5

root@am64xx-evm:~# [   45.087256] hsr0: Slave B (eth2) is not up; please bring it up to get a fullyk



root@am64xx-evm:~# dmesg | grep -i pru

[    5.711978] remoteproc remoteproc2: 3000a000.txpru is available

[    5.723652] remoteproc remoteproc3: 3000c000.txpru is available

[    5.737444] remoteproc remoteproc4: 3008a000.txpru is available

[    5.743333] remoteproc remoteproc5: 3008c000.txpru is available

[    6.253597] remoteproc remoteproc6: 30034000.pru is available

[    6.259700] remoteproc remoteproc8: 30038000.pru is available

[    6.267917] remoteproc remoteproc10: 300b4000.pru is available

[    6.286883] remoteproc remoteproc12: 300b8000.pru is available

[    6.330352] icssg-prueth icssg1-eth: TI PRU ethernet driver initialized: dual EMAC mode

[   39.837296] icssg-prueth icssg1-eth: Enabling HSR offload mode

[   39.908704] remoteproc remoteproc10: powering up 300b4000.pru

[   39.911210] remoteproc remoteproc10: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, siz6

[   39.911280] remoteproc remoteproc10: remote processor 300b4000.pru is now up

[   39.912741] remoteproc remoteproc11: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, siz4

[   39.912846] remoteproc remoteproc4: powering up 3008a000.txpru

[   39.914393] remoteproc remoteproc4: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, si2

[   39.914451] remoteproc remoteproc4: remote processor 3008a000.txpru is now up

[   39.914748] remoteproc remoteproc12: powering up 300b8000.pru

[   39.916505] remoteproc remoteproc12: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, siz2

[   39.916575] remoteproc remoteproc12: remote processor 300b8000.pru is now up

[   39.920881] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, siz0

[   39.920987] remoteproc remoteproc5: powering up 3008c000.txpru

[   39.922467] remoteproc remoteproc5: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, si0

[   39.922520] remoteproc remoteproc5: remote processor 3008c000.txpru is now up

[   44.035608] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

[   45.091679] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

root@am64xx-evm:~# ping 192.168.200.1 -c 20 -i 0.1                                           

PING 192.168.200.1 (192.168.200.1): 56 data bytes

64 bytes from 192.168.200.1: seq=2 ttl=64 time=808.432 ms

64 bytes from 192.168.200.1: seq=3 ttl=64 time=708.460 ms

64 bytes from 192.168.200.1: seq=5 ttl=64 time=508.376 ms

64 bytes from 192.168.200.1: seq=6 ttl=64 time=408.345 ms

64 bytes from 192.168.200.1: seq=7 ttl=64 time=308.317 ms

64 bytes from 192.168.200.1: seq=11 ttl=64 time=0.268 ms

64 bytes from 192.168.200.1: seq=15 ttl=64 time=0.266 ms

64 bytes from 192.168.200.1: seq=16 ttl=64 time=0.234 ms

64 bytes from 192.168.200.1: seq=19 ttl=64 time=0.256 ms



--- 192.168.200.1 ping statistics ---

20 packets transmitted, 9 packets received, 55% packet loss

round-trip min/avg/max = 0.234/304.772/808.432 ms

(reverse-i-search)`e': dmesg | gr^C -i pru

root@am64xx-evm:~# ethtool -S eth2 | grep -i tx_good_frames                                 

     tx_good_frames: 59

root@am64xx-evm:~# ethtool -S eth2 | grep -i rx_good_frames                                         

     rx_good_frames: 38

root@am64xx-evm:~# ethtool -S eth1 | grep -i rx_good_frames                                         

     rx_good_frames: 38

root@am64xx-evm:~# ethtool -S eth2 | grep -i rx_good_frames                                         

     rx_good_frames: 38

root@am64xx-evm:~# ethtool -S eth2 | grep -i rx_good_frames

     rx_good_frames: 42

root@am64xx-evm:~# ethtool -S eth1 | grep -i rx_good_frames

     rx_good_frames: 42

root@am64xx-evm:~# ethtool -S eth1 | grep -i tx_good_frames                                         

     tx_good_frames: 63

- Tianyi

Hi Tianyi,

Thanks for quick update, did you also incorporate the latest firmware updates for ICSSG HSR? https://git.ti.com/cgit/processor-firmware/ti-linux-firmware/commit/?h=ti-linux-firmware&id=3bf8a3177810aec64a41d3a20af507ab0becfc7c

Are those packet loss with ping occurring when you disconnect the cable for either eth1 or eth2 or with no disconnection and there is packet loss?

Additionally, I see "[   45.087256] hsr0: Slave B (eth2) is not up; please bring it up to get a fullyk" in your log so I'm wondering if the "ip link set $ifb up" part of the script ran sucessfully since it should have brought the eth2 interface up.

I will try to replicate this issue (with the hardcoded change to hsr setup script and no network services).

-Daolin

Update:

So I ran a ping test with the hardcoded change to hsr setup script and disabling the network services and I was not able to reproduce the ping packet loss that you are reporting. I don't see the "hsr0: Slave B (eth2) is not up; please bring it up to get a fully" that showed up in your log. What exactly is the hardware setup of your test? Do you have the two EVMs connected via eth1 and eth2 before you ran the hsr setup script or after you ran the hsr setup script?

root@am64xx-evm:~# ifconfig -s
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
lo              65536      255      0      0 0           255      0      0      0 LRU
root@am64xx-evm:~# uname -a
Linux am64xx-evm 6.1.80-rt26-00021-g8dfe902f9207 #2 SMP PREEMPT_RT Tue Mar 26 09:55:56 CDT 2024 aarch64 aarch64 aarch64 GNU/Linux
root@am64xx-evm:~# sh hsr_setup_new.sh hsr_hw eth1 eth2 192.168.2.21                                                                                                                                
ip=192.168.2.21
if=hsr0
mac=70:ff:76:1f:3c:45
slave-a=eth1
slave-b=eth2
device=platform/icssg1-eth
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: off
hsr-dup-offload: off
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: on
hsr-dup-offload: on
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: off
hsr-dup-offload: off
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: on
hsr-dup-offload: on
[  525.236370] icssg-prueth icssg1-eth: Enabling HSR offload mode
[  525.277893] remoteproc remoteproc11: powering up 300b4000.pru
[  525.283481] remoteproc remoteproc11: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 40436
[  525.283520] remoteproc remoteproc11: unsupported resource 5
[  525.283550] remoteproc remoteproc11: remote processor 300b4000.pru is now up
[  525.283589] remoteproc remoteproc12: powering up 30084000.rtu
[  525.285289] remoteproc remoteproc12: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 32904
[  525.285343] remoteproc remoteproc12: remote processor 30084000.rtu is now up
[  525.285384] remoteproc remoteproc7: powering up 3008a000.txpru
[  525.287225] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 35772
[  525.287283] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[  525.287567] remoteproc remoteproc13: powering up 300b8000.pru
[  525.294279] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 40652
[  525.294319] remoteproc remoteproc13: unsupported resource 5
[  525.294348] remoteproc remoteproc13: remote processor 300b8000.pru is now up
[  525.294391] remoteproc remoteproc14: powering up 30086000.rtu
[  525.297602] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 32120
[  525.297655] remoteproc remoteproc14: remote processor 30086000.rtu is now up
[  525.297697] remoteproc remoteproc8: powering up 3008c000.txpru
[  525.298976] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 34280
[  525.299035] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[  525.300594] pps pps1: new PPS source ptp2
[  529.445720] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  529.446156] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[  529.446708] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  529.446770] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[  530.408689] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
root@am64xx-evm:~# ifconfig -s
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth1             1500        1      0      0 0            45      0      0      0 BMRU
eth2             1500        1      0      0 0            15      0      0      0 BMRU
hsr0             1494        1      0      0 0            31      0      0      0 BMRU
lo              65536     1462      0      0 0          1462      0      0      0 LRU
root@am64xx-evm:~# [  547.877913] icssg-prueth icssg1-eth eth2: Link is Down
[  563.238352] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  572.453725] icssg-prueth icssg1-eth eth1: Link is Down
[  584.741923] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

root@am64xx-evm:~# 
root@am64xx-evm:~# ping 192.168.2.20                                                                                                                                                                 
PING 192.168.2.20 (192.168.2.20): 56 data bytes
64 bytes from 192.168.2.20: seq=0 ttl=64 time=0.496 ms
64 bytes from 192.168.2.20: seq=1 ttl=64 time=0.479 ms
64 bytes from 192.168.2.20: seq=2 ttl=64 time=0.470 ms
64 bytes from 192.168.2.20: seq=3 ttl=64 time=0.461 ms
64 bytes from 192.168.2.20: seq=4 ttl=64 time=0.480 ms
64 bytes from 192.168.2.20: seq=5 ttl=64 time=0.465 ms
64 bytes from 192.168.2.20: seq=6 ttl=64 time=0.446 ms
[  603.173932] icssg-prueth icssg1-eth eth2: Link is Down
64 bytes from 192.168.2.20: seq=7 ttl=64 time=0.488 ms
64 bytes from 192.168.2.20: seq=8 ttl=64 time=0.481 ms
64 bytes from 192.168.2.20: seq=9 ttl=64 time=0.465 ms
64 bytes from 192.168.2.20: seq=10 ttl=64 time=0.455 ms
64 bytes from 192.168.2.20: seq=11 ttl=64 time=0.420 ms
64 bytes from 192.168.2.20: seq=12 ttl=64 time=0.515 ms
64 bytes from 192.168.2.20: seq=13 ttl=64 time=0.399 ms
64 bytes from 192.168.2.20: seq=14 ttl=64 time=0.480 ms
64 bytes from 192.168.2.20: seq=15 ttl=64 time=0.403 ms
64 bytes from 192.168.2.20: seq=16 ttl=64 time=0.449 ms
[  613.414462] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
64 bytes from 192.168.2.20: seq=17 ttl=64 time=0.454 ms
64 bytes from 192.168.2.20: seq=18 ttl=64 time=0.458 ms
64 bytes from 192.168.2.20: seq=19 ttl=64 time=0.491 ms
64 bytes from 192.168.2.20: seq=20 ttl=64 time=0.468 ms
64 bytes from 192.168.2.20: seq=21 ttl=64 time=0.455 ms
64 bytes from 192.168.2.20: seq=22 ttl=64 time=0.497 ms
64 bytes from 192.168.2.20: seq=23 ttl=64 time=0.471 ms
64 bytes from 192.168.2.20: seq=24 ttl=64 time=0.474 ms
64 bytes from 192.168.2.20: seq=25 ttl=64 time=0.462 ms
64 bytes from 192.168.2.20: seq=26 ttl=64 time=0.516 ms
[  623.653951] icssg-prueth icssg1-eth eth1: Link is Down
64 bytes from 192.168.2.20: seq=27 ttl=64 time=0.503 ms
64 bytes from 192.168.2.20: seq=28 ttl=64 time=0.446 ms
64 bytes from 192.168.2.20: seq=29 ttl=64 time=0.478 ms
64 bytes from 192.168.2.20: seq=30 ttl=64 time=0.458 ms
64 bytes from 192.168.2.20: seq=31 ttl=64 time=0.489 ms
64 bytes from 192.168.2.20: seq=32 ttl=64 time=0.436 ms
64 bytes from 192.168.2.20: seq=33 ttl=64 time=0.446 ms
64 bytes from 192.168.2.20: seq=34 ttl=64 time=0.522 ms
64 bytes from 192.168.2.20: seq=35 ttl=64 time=0.539 ms
64 bytes from 192.168.2.20: seq=36 ttl=64 time=0.481 ms
[  632.869932] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
64 bytes from 192.168.2.20: seq=37 ttl=64 time=0.519 ms
64 bytes from 192.168.2.20: seq=38 ttl=64 time=0.519 ms
64 bytes from 192.168.2.20: seq=39 ttl=64 time=0.473 ms
64 bytes from 192.168.2.20: seq=40 ttl=64 time=0.482 ms
64 bytes from 192.168.2.20: seq=41 ttl=64 time=0.494 ms
64 bytes from 192.168.2.20: seq=42 ttl=64 time=0.489 ms
^C
--- 192.168.2.20 ping statistics ---
43 packets transmitted, 43 packets received, 0% packet loss
round-trip min/avg/max = 0.399/0.473/0.539 ms
root@am64xx-evm:~# ifconfig -s
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth1             1500       75      0      0 0           175      0      0      0 BMRU
eth2             1500       73      0      0 0            56      0      0      0 BMRU
hsr0             1494       99      0      0 0           188      0      0      0 BMRU
lo              65536     1782      0      0 0          1782      0      0      0 LRU
root@am64xx-evm:~# 

This is with the older firmware for ICSSG HSR as I don't think the issue is directly related to the firmware updates.

I also expected to see more kernel messages than just the following from your run of the hsr setup (below is a copy from your log). See my console log above for what you should be seeing.

root@am64xx-evm:~# ./hsr_setup.sh hsr_hw eth1 eth2 192.168.200.2                             

ip=192.168.200.2

if=hsr0

mac=70:ff:76:1e:2e:2d

slave-a=eth1

slave-b=eth2

device=platform/icssg1-eth

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: on

hsr-dup-offload: on

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: on

hsr-dup-offload: on

[   39.911249] remoteproc remoteproc10: unsupported resource 5

[   39.916544] remoteproc remoteproc12: unsupported resource 5

-Daolin

Hello Daolin,

Thank you for your fast response !

The hardware setup is two EVMs with two cables that are already connected (eth1<->eth1)and (eth2<->eth2) before launching the script. (no disconnection of cables).

I launch the script to load hsr firmware on pru . And startto ping. The result above is done with the latest patch you provided.

on EVM1 :

am64xx-evm login: root

root@am64xx-evm:~# ./hsr_setup.sh hsr_hw eth1 eth2 192.168.200.1

hsr_hw eth1 eth2 192.168.200.1

ip=192.168.200.1

if=hsr0

mac=70:ff:76:1e:2e:2d

slave-a=eth1

slave-b=eth2

device=platform/

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: on

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: on

hsr-dup-offload: off

[  143.103318] remoteproc remoteproc7: unsupported resource 5

[  143.114495] remoteproc remoteproc11: unsupported resource 5

[  148.281029] hsr0: Slave B (eth2) is not up; please bring it up to get a fully working HSR network

root@am64xx-evm:~# dmesg | grep -i pru

[    5.824148] remoteproc remoteproc2: 3000a000.txpru is available

[    5.824752] remoteproc remoteproc3: 3000c000.txpru is available

[    5.825312] remoteproc remoteproc4: 3008a000.txpru is available

[    5.825766] remoteproc remoteproc5: 3008c000.txpru is available

[    6.436653] remoteproc remoteproc6: 30034000.pru is available

[    6.437577] remoteproc remoteproc7: 300b4000.pru is available

[    6.444695] remoteproc remoteproc10: 30038000.pru is available

[    6.445597] remoteproc remoteproc11: 300b8000.pru is available

[    6.553080] icssg-prueth icssg1-eth: TI PRU ethernet driver initialized: dual EMAC mode

[  143.025979] icssg-prueth icssg1-eth: Enabling HSR offload mode

[  143.101108] remoteproc remoteproc7: powering up 300b4000.pru

[  143.103279] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 404                                                                                                  6

[  143.103349] remoteproc remoteproc7: remote processor 300b4000.pru is now up

[  143.110106] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 329                                                                                                  4

[  143.110212] remoteproc remoteproc4: powering up 3008a000.txpru

[  143.112547] remoteproc remoteproc4: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 3                                                                                                  2

[  143.112612] remoteproc remoteproc4: remote processor 3008a000.txpru is now up

[  143.112911] remoteproc remoteproc11: powering up 300b8000.pru

[  143.114456] remoteproc remoteproc11: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 40                                                                                                  2

[  143.114526] remoteproc remoteproc11: remote processor 300b8000.pru is now up

[  143.117074] remoteproc remoteproc12: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 32                                                                                                  0

[  143.117181] remoteproc remoteproc5: powering up 3008c000.txpru

[  143.120539] remoteproc remoteproc5: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 3                                                                                                  0

[  143.120599] remoteproc remoteproc5: remote processor 3008c000.txpru is now up

[  146.208048] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

[  148.288155] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

[  151.327851] icssg-prueth icssg1-eth eth1: Link is Down

[  152.387837] icssg-prueth icssg1-eth eth2: Link is Down

[  154.400047] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

[  155.456035] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

root@am64xx-evm:~# ping 192.168.200.2

PING 192.168.200.2 (192.168.200.2): 56 data bytes

64 bytes from 192.168.200.2: seq=6 ttl=64 time=0.516 ms

[  181.222018] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[  181.222071] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node

[  182.061072] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[  182.061117] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node

[  182.246071] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[  182.246125] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node

[  183.061397] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[  183.061476] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node

64 bytes from 192.168.200.2: seq=14 ttl=64 time=0.472 ms

[  191.063572] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[  191.063649] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node

[  192.063903] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[  192.063948] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node

[  193.064222] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[  193.064304] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node

[  194.064582] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[  194.064627] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node

[  195.064870] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[  195.064956] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node

^C

--- 192.168.200.2 ping statistics ---

22 packets transmitted, 2 packets received, 90% packet loss

round-trip min/avg/max = 0.472/0.494/0.516 ms

On EVM2 :

am64xx-evm login: root

root@am64xx-evm:~# ./hsr_setup.sh hsr_hw eth1 eth2 192.168.200.2

hsr_hw eth1 eth2 192.168.200.2

ip=192.168.200.2

if=hsr0

mac=70:ff:76:1e:2e:00

slave-a=eth1

slave-b=eth2

device=platform/

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: on

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: on

hsr-dup-offload: off

[   28.817122] remoteproc remoteproc10: unsupported resource 5

[   28.825353] remoteproc remoteproc12: unsupported resource 5

[   34.003371] hsr0: Slave B (eth2) is not up; please bring it up to get a fully working HSR network

root@am64xx-evm:~# dmesg | grep -i pru

[    5.701931] remoteproc remoteproc2: 3000a000.txpru is available

[    5.707070] remoteproc remoteproc3: 3000c000.txpru is available

[    5.707721] remoteproc remoteproc4: 3008a000.txpru is available

[    5.708250] remoteproc remoteproc5: 3008c000.txpru is available

[    6.379106] remoteproc remoteproc6: 30038000.pru is available

[    6.381881] remoteproc remoteproc8: 30034000.pru is available

[    6.401678] remoteproc remoteproc10: 300b4000.pru is available

[    6.408313] remoteproc remoteproc12: 300b8000.pru is available

[    6.529890] icssg-prueth icssg1-eth: TI PRU ethernet driver initialized: dual EMAC mode

[   28.744914] icssg-prueth icssg1-eth: Enabling HSR offload mode

[   28.812741] remoteproc remoteproc10: powering up 300b4000.pru

[   28.817085] remoteproc remoteproc10: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, siz6

[   28.817152] remoteproc remoteproc10: remote processor 300b4000.pru is now up

[   28.818340] remoteproc remoteproc11: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, siz4

[   28.818428] remoteproc remoteproc4: powering up 3008a000.txpru

[   28.823234] remoteproc remoteproc4: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, si2

[   28.823294] remoteproc remoteproc4: remote processor 3008a000.txpru is now up

[   28.823582] remoteproc remoteproc12: powering up 300b8000.pru

[   28.825314] remoteproc remoteproc12: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, siz2

[   28.825384] remoteproc remoteproc12: remote processor 300b8000.pru is now up

[   28.826710] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, siz0

[   28.826807] remoteproc remoteproc5: powering up 3008c000.txpru

[   28.829417] remoteproc remoteproc5: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, si0

[   28.829475] remoteproc remoteproc5: remote processor 3008c000.txpru is now up

[   32.931626] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

[   34.016233] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

root@am64xx-evm:~# [   53.452627] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[   53.452685] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node

[   67.544792] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[   67.544830] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node

[   68.456668] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

[   68.456705] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node

We also did the same test with the release 09.02.00.08 (06 Feb 2024, without the latest patch you provided). And there is no packet lost !

-Tianyi

Moreover, has the most recent test you shared been conducted using the latest patch that you provided us with?