TMDS64EVM: HSR with PTP problems

The query has been assigned to the expert. Please expect a response within a day or two.

Thanks!

Hello Mustafa, 

First I have a couple initial questions

Mustafa Gül said:

I have 2 developments boards. Ive updated their linux system which ive downloaded from here. I didnt touch any settings, just flashed wic file to sd card and ran it.

To clarify, you are using 2 TMDS64EVM (AM64x EVM) boards? I see from the link you are using latest PROCESSOR-SDK-LINUX-AM64X. I would suggest changing to the latest PROCESSOR-SDK-LINUX-RT-AM64X if this is meant for a real-time application.

Mustafa Gül said:

HSR worked well, but we couldnt activate PTP over HSR (offload/non-offload). It gaves configuration file error, some parameters not defined.

Can you describe specifically what commands, scripts, configuration files are you using to setup PTP and HSR? Specifically, you should be setting up HSR first, then initiating PTP. Assuming you are using the HSR nonoffload script provided in link you shared (https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/10_01_10_04/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/Network/HSR_PRP_Non_Offload.html), what is the PTP configuration and command you are using? 

Can you share the exact log output after running PTP over HSR nonoffload? 

Mustafa Gül said:

While researching we saw this TI-git commit : port: add PTP support over HSR/PRP network

Tried, not working either. Its not giving parameter error but gives hsr0 interface does not have clock.

How should we proceed in order to use HSR offload/non-offload with PTP ?

Thanks for bringing this commit to my attention. From my understanding and my own test results, PTP over HSR nonoffloaded should be working on latest RT-Linux SDK (version 10.1), but PTP over HSR offloaded still requires additional development. I suspect that this commit to linuxptp that you pointed out was meant to enable PTP over HSR nonoffloaded. Offload side requires implementation changes from the ti-pruss firmware side. 

Mustafa Gül said:

For use it we had to update our ti-pruss firmware from TI-git.

Which ti-pruss firmware are you using? Can you point out the specific firmware commit?

-Daolin

Hello,

• Yes, were using 2 TMDS64EVM boards. We didnt try RT-Linux.
• Were using provided scripts and setting files which is in the link.
• This error when we enable hsr non-offload mode. Note: uEnv.txt edited same as first post.

  [  346.738703] page_pool_release_retry() stalled pool shutdown 19 inflight 120 sec
  [  346.834709] page_pool_release_retry() stalled pool shutdown 47 inflight 120 sec
  [  407.154707] page_pool_release_retry() stalled pool shutdown 19 inflight 181 sec
  [  407.250702] page_pool_release_retry() stalled pool shutdown 47 inflight 181 sec

• Trying to run PTP with while HSR non-offload active "ptp4l -i hsr0"

  failed to create a clock

• Trying to run PTP with config while HSR non-offload active "ptp4l -f oc_hsr0.cfg -m -s"

  option slaveOnly is deprecated, please use clientOnly instead
  unknown option doubly_attached_clock at line 13 in global section
  failed to parse configuration file oc_hsr0.cfg
  

• Trying to run PTP with "ti-linuxptp"
  • copied files to device.
  • make & make install
  • ptp4l -f oc_hsr0.cfg -m -s

    ptp4l[1204.255]: interface 'hsr0' does not support requested timestamping mode
    failed to create a clock

• Weve downloaded ti-pruss firmware from this link. My latest local "git log -1" is: commit 1eaf07dc4ec5cdeb98078f17a9d4635e88f43f75 Author: Soumya Tripathy <s-tripathy@ti.com> Date: Mon Nov 18 13:38:03 2024 +0530

I will try same things with RT-Linux asap. Thanks for helping.

Hello Mustafa, 

Mustafa Gül said:

Trying to run PTP with "ti-linuxptp

I found out this commit from ti-linuxptp was a rejected from upstream commit and was originally meant for some older devices like AM57x and not for AM64x. For this reason, you should not apply any changes from this commit.

Mustafa Gül said:

Weve downloaded ti-pruss firmware from this link. My latest local "git log -1" is: commit 1eaf07dc4ec5cdeb98078f17a9d4635e88f43f75 Author: Soumya Tripathy <s-tripathy@ti.com> Date: Mon Nov 18 13:38:03 2024 +0530

Thanks for sharing. From my understanding, the ti-pruss firmware should already be included in the latest wic image you downloaded. So you should technically not have download the ti-pruss firmware as a separate step.

Mustafa Gül said:

This error when we enable hsr non-offload mode. Note: uEnv.txt edited same as first post

Can you share the full log result of running the hsr nonoffload script? We need to ensure hsr nonoffload (without ptp) is working properly first before testing out ptp.

Mustafa Gül said:

Trying to run PTP with while HSR non-offload active "ptp4l -i hsr0"

Can you try using the "ptp4l -i eth1 -i eth2"? Assuming eth1 and eth2 are the interfaces used to configure hsr0. 

-Daolin

Hello,

Daolin Qiu said:

I found out this commit from ti-linuxptp was a rejected from upstream commit and was originally meant for some older devices like AM57x and not for AM64x. For this reason, you should not apply any changes from this commit.

We didnt know its rejected. Thanks for information.

Daolin Qiu said:

Thanks for sharing. From my understanding, the ti-pruss firmware should already be included in the latest wic image you downloaded. So you should technically not have download the ti-pruss firmware as a separate step.

Maybe ive mixed it up with yocto builds. We also tried that one. But it seems fine right now, at least it has capability to do hsr-offload.

Daolin Qiu said:

Can you share the full log result of running the hsr nonoffload script? We need to ensure hsr nonoffload (without ptp) is working properly first before testing out ptp.

Sure. MAC addresses filtered due security.

root@am64xx-evm:~# ./hsr-prp.sh hsr eth1 eth2 10.10.10.2
ip=10.10.10.2
if=hsr0
mac=70:ff:76:xx:xx:xx
slave-a=eth1
slave-b=eth2
[  136.395160] icssg-prueth icssg1-eth eth1: Link is Down
[  136.432234] icssg-prueth icssg1-eth eth2: Link is Down
[  136.439932] remoteproc remoteproc7: stopped remote processor 3008a000.txpru
[  136.440115] remoteproc remoteproc14: stopped remote processor 30084000.rtu
[  136.440128] remoteproc remoteproc13: stopped remote processor 300b4000.pru
[  136.440139] remoteproc remoteproc8: stopped remote processor 3008c000.txpru
[  136.440148] remoteproc remoteproc16: stopped remote processor 30086000.rtu
[  136.440158] remoteproc remoteproc15: stopped remote processor 300b8000.pru
[  137.541224] remoteproc remoteproc13: powering up 300b4000.pru
[  137.541521] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, size 39636
[  137.541551] remoteproc remoteproc13: unsupported resource 5
[  137.541580] remoteproc remoteproc13: remote processor 300b4000.pru is now up
[  137.541613] remoteproc remoteproc14: powering up 30084000.rtu
[  137.541754] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu0-prueth-fw.elf, size 30444
[  137.541780] remoteproc remoteproc14: remote processor 30084000.rtu is now up
[  137.541800] remoteproc remoteproc7: powering up 3008a000.txpru
[  137.541922] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, size 39080
[  137.541954] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[  137.544289] remoteproc remoteproc15: powering up 300b8000.pru
[  137.544516] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru1-prueth-fw.elf, size 39796
[  137.544544] remoteproc remoteproc15: unsupported resource 5
[  137.544571] remoteproc remoteproc15: remote processor 300b8000.pru is now up
[  137.544605] remoteproc remoteproc16: powering up 30086000.rtu
[  137.544739] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu1-prueth-fw.elf, size 29680
[  137.544765] remoteproc remoteproc16: remote processor 30086000.rtu is now up
[  137.544784] remoteproc remoteproc8: powering up 3008c000.txpru
[  137.544903] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-prueth-fw.elf, size 37568
[  137.544933] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[  137.546499] pps pps1: new PPS source ptp2
[  139.630963] icssg-prueth icssg1-eth eth1: entered promiscuous mode
[  139.633004] audit: type=1700 audit(1728493751.670:3): dev=eth1 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[  139.650539] icssg-prueth icssg1-eth eth2: entered promiscuous mode
[  139.651436] audit: type=1700 audit(1728493751.670:3): dev=eth2 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[  139.673173] audit: type=1300 audit(1728493751.670:3): arch=c00000b7 syscall=211 success=yes exit=92 a0=3 a1=ffffea7a9e28 a2=0 a3=1 items=4 ppid=1024 pid=1048 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=ttyS2 ses=4294967295 comm="ip" exe="/usr/sbin/ip.iproute2" subj=kernel key=(null)
[  139.673266] audit: type=1307 audit(1728493751.670:3): cwd="/root"
[  139.673280] audit: type=1302 audit(1728493751.670:3): item=0 name=(null) inode=6987 dev=00:07 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
[  139.673295] audit: type=1302 audit(1728493751.670:3): item=1 name=(null) inode=11278 dev=00:07 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
[  139.673311] audit: type=1302 audit(1728493751.670:3): item=2 name=(null) inode=11278 dev=00:07 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
[  139.673326] audit: type=1302 audit(1728493751.670:3): item=3 name=(null) inode=11279 dev=00:07 mode=0100444 ouid=0 ogid=0 rdev=00:00 obj=unlabeled nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
[  139.673341] audit: type=1327 audit(1728493751.670:3): proctitle=6970006C696E6B00616464006E616D65006873723000747970650068737200736C61766531006574683100736C617665320065746832007375706572766973696F6E0034350076657273696F6E0031
[  141.670243] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[  141.673073] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
root@am64xx-evm:~# 
[  196.837057] page_pool_release_retry() stalled pool shutdown 12 inflight 60 sec
[  196.901062] page_pool_release_retry() stalled pool shutdown 37 inflight 60 sec
[  257.253163] page_pool_release_retry() stalled pool shutdown 12 inflight 120 sec
[  257.317063] page_pool_release_retry() stalled pool shutdown 37 inflight 120 sec
[  317.669077] page_pool_release_retry() stalled pool shutdown 12 inflight 181 sec
[  317.733076] page_pool_release_retry() stalled pool shutdown 37 inflight 181 sec

Daolin Qiu said:

Can you try using the "ptp4l -i eth1 -i eth2"? Assuming eth1 and eth2 are the interfaces used to configure hsr0. 

Then, cards chooses local clocks as best master. MAC addresses filtered due security.

--- CARD 1 ---

root@am64xx-evm:~# ptp4l -i eth1 -i eth2 -m -s
ptp4l[588.942]: selected /dev/ptp2 as PTP clock
ptp4l[588.948]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[588.954]: port 2 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[588.954]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[588.955]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[595.979]: selected local clock 70ff76.fffe.xxxxxx as best master

--- CARD 2 ---

root@am64xx-evm:~# ptp4l -i eth1 -i eth2 -m -s
ptp4l[591.909]: selected /dev/ptp2 as PTP clock
ptp4l[591.916]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[591.921]: port 2 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[591.921]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[591.922]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[598.408]: selected local clock 70ff76.fffe.yyyyyy as best master

Ive seen a page explains it as "use it on L2 network instead of L3". Then we tried to use "ptp4l -i eth1 -i eth2 -2 -m -s", now cards can see eachother while HSR connected. But with this, ptp4l works on normal frames, not in HSR one. Im not quite sure about how its supposed to work, will research more on different devices.

Thanks.

Hi Mustafa, 

Mustafa Gül said:

Daolin Qiu said:

Can you share the full log result of running the hsr nonoffload script? We need to ensure hsr nonoffload (without ptp) is working properly first before testing out ptp.

Sure. MAC addresses filtered due security.

Can you additionally show the results of "ethtool -k eth1 | grep hsr" and the same for eth2? 

Mustafa Gül said:

But with this, ptp4l works on normal frames, not in HSR one. Im not quite sure about how its supposed to work, will research more on different devices.

Can you explain what you mean by "ptp4l works on normal frames, not in HSR one"? What did you mean by "HSR one"?

Can you also clarify the topology? I.e. eth1 on EVM1 connected to eth1 on EVM2, eth2 on EVM1 connected to eth2 on EVM2?

Mustafa Gül said:

Ive seen a page explains it as "use it on L2 network instead of L3".

Which page are you referring to?

The "-s" option used on both EVMs ensure both are in clientMode (in other words PTP follower). Can you try removing the "-s" from one of the two and see what happens?

-Daolin

Hello,

Daolin Qiu said:

Can you additionally show the results of "ethtool -k eth1 | grep hsr" and the same for eth2? 

root@am64xx-evm:~# ethtool -k eth1 | grep hsr
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: off
hsr-dup-offload: off
root@am64xx-evm:~# ethtool -k eth2 | grep hsr
hsr-tag-ins-offload: off
hsr-tag-rm-offload: off
hsr-fwd-offload: off
hsr-dup-offload: off

Daolin Qiu said:

Can you explain what you mean by "ptp4l works on normal frames, not in HSR one"? What did you mean by "HSR one"?

Daolin Qiu said:

Which page are you referring to?

Was one of the pages i came across while continuing research, couldnt find the page again rn. But it was suggesting to use OSI L2 instead of OSI L3. While using HSR, eth1 and eth2 dont have any IP addresses so they cant communicate on OSI L3.

With that solution all of those PTP packages communicates over MAC addresses instead of IP addresses which im pointing to. 

Mustafa Gül said:

Im not quite sure about how its supposed to work, will research more on different devices.

I will continue to research this mechanic with different devices which is supporting HSR+PTP currently.

Daolin Qiu said:

Can you also clarify the topology? I.e. eth1 on EVM1 connected to eth1 on EVM2, eth2 on EVM1 connected to eth2 on EVM2?

EVM1 and EVM2's eth1 and eth2 set as hsr0 network. PC's enp3s0 and enx00 is set to bridge mode br0. My PC adds 1ms to this network when packets going over bridge. Also helps me to watch it over Wireshark.

EVM1 and EVM2's eth1 connected to PC and eth2 connected between them.

Daolin Qiu said:

The "-s" option used on both EVMs ensure both are in clientMode (in other words PTP follower). Can you try removing the "-s" from one of the two and see what happens?

Sorry, was wondering why they didnt matched up, but done it before.

--- EVM 1 L3 ---

root@am64xx-evm:~# ptp4l -i eth1 -i eth2 -m   
ptp4l[1927.168]: selected /dev/ptp2 as PTP clock
ptp4l[1927.174]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[1927.179]: port 2 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[1927.180]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[1927.180]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[1933.707]: port 1 (eth1): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[1933.707]: selected local clock 70ff76.fffe.xxxxxx as best master
ptp4l[1933.707]: port 1 (eth1): assuming the grand master role
ptp4l[1933.830]: port 2 (eth2): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[1933.830]: port 1 (eth1): assuming the grand master role
ptp4l[1933.830]: port 2 (eth2): assuming the grand master role

--- EVM 2 L3 ---

oot@am64xx-evm:~# ptp4l -i eth1 -i eth2 -m   
ptp4l[1931.503]: selected /dev/ptp2 as PTP clock
ptp4l[1931.509]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[1931.514]: port 2 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[1931.515]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[1931.515]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[1937.570]: port 1 (eth1): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[1937.570]: selected local clock 70ff76.fffe.yyyyyy as best master
ptp4l[1937.570]: port 1 (eth1): assuming the grand master role
ptp4l[1938.352]: port 2 (eth2): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[1938.352]: port 1 (eth1): assuming the grand master role
ptp4l[1938.352]: port 2 (eth2): assuming the grand master role


--- EVM 1 L2 ---

ptp4l -i eth1 -i eth2 -m -2
ptp4l[2025.565]: selected /dev/ptp2 as PTP clock
ptp4l[2025.579]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[2025.594]: port 2 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[2025.595]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[2025.595]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[2033.185]: port 1 (eth1): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[2033.185]: selected local clock 70ff76.fffe.xxxxxx as best master
ptp4l[2033.185]: port 1 (eth1): assuming the grand master role
ptp4l[2033.427]: port 2 (eth2): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[2033.427]: port 1 (eth1): assuming the grand master role
ptp4l[2033.427]: port 2 (eth2): assuming the grand master role
ptp4l[2036.352]: port 2 (eth2): new foreign master 70ff76.fffe.yyyyyy-2
ptp4l[2037.068]: port 1 (eth1): new foreign master 70ff76.fffe.yyyyyy-1
ptp4l[2040.352]: selected best master clock 70ff76.fffe.yyyyyy
ptp4l[2040.352]: port 2 (eth2): MASTER to UNCALIBRATED on RS_SLAVE
ptp4l[2041.068]: selected best master clock 70ff76.fffe.yyyyyy
ptp4l[2041.068]: port 1 (eth1): MASTER to UNCALIBRATED on RS_SLAVE
ptp4l[2041.068]: port 2 (eth2): UNCALIBRATED to PASSIVE on RS_PASSIVE
ptp4l[2044.068]: master offset   -2882436 s0 freq      -0 path delay       456
ptp4l[2045.069]: master offset   -2882464 s1 freq     -28 path delay       460
ptp4l[2046.068]: master offset      -2848 s2 freq   -2876 path delay       460
ptp4l[2046.068]: port 1 (eth1): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[2047.068]: master offset          5 s2 freq    -877 path delay       460
ptp4l[2048.068]: master offset        873 s2 freq      -8 path delay       456
ptp4l[2049.068]: master offset        843 s2 freq    +224 path delay       465

--- EVM 2 L2 ---

root@am64xx-evm:~# ptp4l -i eth1 -i eth2 -m -2
ptp4l[2031.459]: selected /dev/ptp2 as PTP clock
ptp4l[2031.473]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[2031.488]: port 2 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[2031.489]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[2031.489]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[2035.572]: port 1 (eth1): new foreign master 70ff76.fffe.xxxxxx-1
ptp4l[2035.814]: port 2 (eth2): new foreign master 70ff76.fffe.xxxxxx-2
ptp4l[2038.735]: port 2 (eth2): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[2038.736]: selected local clock 70ff76.fffe.yyyyyy as best master
ptp4l[2038.736]: port 2 (eth2): assuming the grand master role
ptp4l[2039.452]: port 1 (eth1): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[2039.452]: port 1 (eth1): assuming the grand master role
ptp4l[2039.452]: port 2 (eth2): assuming the grand master role
ptp4l[2039.572]: selected best master clock 70ff76.fffe.yyyyyy
ptp4l[2039.572]: port 1 (eth1): assuming the grand master role
ptp4l[2039.572]: port 2 (eth2): assuming the grand master role
ptp4l[2039.814]: port 1 (eth1): assuming the grand master role
ptp4l[2039.814]: port 2 (eth2): assuming the grand master role

Thanks.

Hi Mustafa, 

Mustafa Gül said:

Was one of the pages i came across while continuing research, couldnt find the page again rn. But it was suggesting to use OSI L2 instead of OSI L3. While using HSR, eth1 and eth2 dont have any IP addresses so they cant communicate on OSI L3.

With that solution all of those PTP packages communicates over MAC addresses instead of IP addresses which im pointing to. 

Mustafa Gül said:

Im not quite sure about how its supposed to work, will research more on different devices.

I will continue to research this mechanic with different devices which is supporting HSR+PTP currently.

Thanks for clarifying this, if you are able to find more information on this, please do let us know.

Based on the ptp4l manpage (https://man.cx/ptp4l), it looks like the "-2" option is to specify IEEE 802.3 network transport (as opposed to UDP IPv4 network transport which is the default). Is there a particular requirement to using the default option for you?

-Daolin

Daolin Qiu said:

Is there a particular requirement to using the default option for you?

Yes, I cant reach all documents about "PTP Power Profile" which is used in "real time industrial applications" side.

Mustafa Gül said:

I will continue to research this mechanic with different devices which is supporting HSR+PTP currently.

Weve found devices from another companies which supports HSR+PTP, we will research and will inform you how its working. 

Any update for kernel warning for HSR network on icssg-dualemac ? 

page_pool_release_retry() stalled pool shutdown 12 inflight 60 sec

Hi Mustafa, 

Mustafa Gül said:

Any update for kernel warning for HSR network on icssg-dualemac ? 

I need to check with some team members internally about this page_pool_release_retry message. I am also able to see these messages when HSR is enabled (definitely see with HSR offload, I would have to double check with HSR nonoffload).

However, I currently don't see any issues with the basic functionality of HSR (no packet loss under cable disconnection) even with these messages showing up.

Mustafa Gül said:

Weve found devices from another companies which supports HSR+PTP, we will research and will inform you how its working. 

May I ask, which companies are you finding HSR+PTP support?

-Daolin

Daolin Qiu said:

May I ask, which companies are you finding HSR+PTP support?

Mostly protection relay companies and redbox-industrial switch manufacturers. This is the list i found their devices in online, they claiming as "supporting HSR+PTP" on some models but dont have those devices to test them how they work. We only have old model of ABB device rn which i will test. 

• ABB
• Schneider
• Siemens
• Cisco
• Moxa
• AMD

Daolin Qiu said:

I need to check with some team members internally about this page_pool_release_retry message. I am also able to see these messages when HSR is enabled (definitely see with HSR offload, I would have to double check with HSR nonoffload).

However, I currently don't see any issues with the basic functionality of HSR (no packet loss under cable disconnection) even with these messages showing up.

I know its not generating any issues right now but we want to be sure for future applications. We will run lots of communication protocols simultaneously. It can lead to page_pool/memory corruption. 

Hi Mustafa, 

Mustafa Gül said:

I know its not generating any issues right now but we want to be sure for future applications. We will run lots of communication protocols simultaneously. It can lead to page_pool/memory corruption. 

I checked with our developer and he informed me that "XDP support changed from skb to page_pool" on this Linux kernel (SDK 10.1, Kernel 6.6) and "the driver might not be cleaning up the pages properly". He suspects that this is why these warnings are showing up but shared that the warnings are fixed in the latest Linux upstream so likely these warnings won't show in future SDKs (Kernel 6.12). 

If you would like to check before our next SDK releases, I would suggest building the Linux kernel image on 6.12 (https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/log/?h=ti-linux-6.12.y-cicd) and see if these warnings still show up. Please note, since our next SDK release with 6.12 is not released yet, the branch I shared is not yet fully stable and should only be used for development purposes.

-Daolin

Daolin Qiu said:

I need to check with some team members internally about this page_pool_release_retry message. I am also able to see these messages when HSR is enabled (definitely see with HSR offload, I would have to double check with HSR nonoffload).

While testing it in offload mode you should see this while trying to run PTP in L2 mode. Whenever we see "NETDEV WATCHDOG" error, it completely crashes interfaces and not be able to communicate. If you try to "ifconfig eth1 down" its waiting in "[ 464.463621] icssg-prueth icssg1-eth eth2: timeout waiting for command done" or kernel crashes and not giving any response.

root@am64xx-evm:~# ptp4l -i eth1 -i eth2 -m -2
ptp4l[82.840]: selected /dev/ptp2 as PTP clock
ptp4l[82.854]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[82.867]: port 2 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[82.868]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[82.868]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[88.707]: port 2 (eth2): new foreign master 70ff76.fffe.1f3d63-2
ptp4l[89.576]: port 2 (eth2): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[89.577]: selected local clock 70ff76.fffe.1f3ef5 as best master
ptp4l[89.577]: port 2 (eth2): assuming the grand master role
ptp4l[89.672]: port 1 (eth1): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[89.672]: port 1 (eth1): assuming the grand master role
ptp4l[89.672]: port 2 (eth2): assuming the grand master role
ptp4l[89.796]: port 1 (eth1): new foreign master 70ff76.fffe.1f3d63-1
[   90.582786] icssg-prueth icssg1-eth eth1: Invalid TX TS cookie 0x0
[   90.677627] icssg-prueth icssg1-eth eth2: Invalid TX TS cookie 0x0
[   91.582927] icssg-prueth icssg1-eth eth1: Invalid TX TS cookie 0x0
[   91.677798] icssg-prueth icssg1-eth eth2: Invalid TX TS cookie 0x0
[   92.583062] icssg-prueth icssg1-eth eth1: Invalid TX TS cookie 0x0
[   92.677875] icssg-prueth icssg1-eth eth2: Invalid TX TS cookie 0x0
ptp4l[92.707]: selected best master clock 70ff76.fffe.1f3d63
ptp4l[92.707]: port 2 (eth2): MASTER to UNCALIBRATED on RS_SLAVE
[   93.678048] icssg-prueth icssg1-eth eth2: Invalid TX TS cookie 0x0
ptp4l[93.796]: selected best master clock 70ff76.fffe.1f3d63
ptp4l[93.796]: port 1 (eth1): MASTER to UNCALIBRATED on RS_SLAVE
ptp4l[93.797]: port 2 (eth2): UNCALIBRATED to PASSIVE on RS_PASSIVE
[   95.148672] icssg-prueth icssg1-eth eth2: Invalid TX TS cookie 0x0
[   96.357135] icssg-prueth icssg1-eth eth2: Invalid TX TS cookie 0x0
[   98.222380] icssg-prueth icssg1-eth eth2: Invalid TX TS cookie 0x0
[   98.360646] icssg-prueth icssg1-eth eth2: Invalid TX TS cookie 0x0
[  102.165812] icssg-prueth icssg1-eth eth2: Invalid TX TS cookie 0x0
[  103.441425] icssg-prueth icssg1-eth eth2: Invalid TX TS cookie 0x0
[  104.677143] page_pool_release_retry() stalled pool shutdown 10 inflight 60 sec
[  104.741149] page_pool_release_retry() stalled pool shutdown 11 inflight 60 sec
[  105.302400] icssg-prueth icssg1-eth eth2: Invalid TX TS cookie 0x0
ptp4l[105.307]: timed out while polling for tx timestamp
ptp4l[105.307]: increasing tx_timestamp_timeout or increasing kworker priority may correct this issue, but a driver bug likely causes it
ptp4l[105.307]: port 1 (eth1): send delay request failed
ptp4l[105.307]: port 1 (eth1): UNCALIBRATED to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[111.301]: port 2 (eth2): PASSIVE to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[111.301]: selected local clock 70ff76.fffe.1f3ef5 as best master
ptp4l[111.301]: port 1 (eth1): assuming the grand master role
ptp4l[111.301]: port 2 (eth2): assuming the grand master role
ptp4l[112.311]: timed out while polling for tx timestamp
ptp4l[112.311]: increasing tx_timestamp_timeout or increasing kworker priority may correct this issue, but a driver bug likely causes it
ptp4l[112.312]: port 2 (eth2): send sync failed
ptp4l[112.312]: port 2 (eth2): MASTER to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
^Croot@am64xx-evm:~# [  120.805136] icssg-prueth icssg1-eth eth2: NETDEV WATCHDOG: CPU: 1: transmit queue 0 timed out 9964 ms
[  125.925140] icssg-prueth icssg1-eth eth2: NETDEV WATCHDOG: CPU: 1: transmit queue 0 timed out 15084 ms
[  131.045138] icssg-prueth icssg1-eth eth2: NETDEV WATCHDOG: CPU: 1: transmit queue 0 timed out 20204 ms
[  137.189149] icssg-prueth icssg1-eth eth2: NETDEV WATCHDOG: CPU: 1: transmit queue 0 timed out 26348 ms
[  142.821137] icssg-prueth icssg1-eth eth2: NETDEV WATCHDOG: CPU: 1: transmit queue 0 timed out 31980 ms
[  147.941140] icssg-prueth icssg1-eth eth2: NETDEV WATCHDOG: CPU: 1: transmit queue 0 timed out 37100 ms
[  153.061137] icssg-prueth icssg1-eth eth2: NETDEV WATCHDOG: CPU: 1: transmit queue 0 timed out 42220 ms

[  464.442730] icssg-prueth icssg1-eth eth2: timeout waiting for command done
[  464.453037] icssg-prueth icssg1-eth eth2: timeout waiting for command done
[  464.463621] icssg-prueth icssg1-eth eth2: timeout waiting for command done
[  464.473705] icssg-prueth icssg1-eth eth2: timeout waiting for command done
[  465.905627] icssg-prueth icssg1-eth eth2: timeout waiting for command done
[  465.916352] icssg-prueth icssg1-eth eth2: timeout waiting for command done

[ 1021.156897] icssg-prueth icssg1-eth eth1: tx teardown timeout
[ 1021.157317] ------------[ cut here ]------------
[ 1021.157324] kernel BUG at /lib/genalloc.c:508!
[ 1021.157333] Internal error: Oops - BUG: 00000000f2000800 [#1] PREEMPT_RT SMP
[ 1021.157341] Modules linked in: rpmsg_ctrl rpmsg_char cdns3 cdns_usb_common overlay irq_pruss_intc cfg80211 crct10dif_ce icssg_prueth sch_taprio sch_mqprio_lib pru_rproc mux_gpio hsr icss_iep bluetooth phy_can_transceiver ecdh_generic ecc cdns3_ti pci_endpoint_test rfkill dp83869 rti_wdt ti_k3_r5_remoteproc ti_k3_m4_remoteproc k3_j72xx_bandgap sa2ul pruss omap_hwspinlock at24 omap_mailbox m_can_platform m_can can_dev pwm_tiecap spi_omap2_mcspi cryptodev(O) drm fuse drm_panel_orientation_quirks backlight ipv6
[ 1021.157473] CPU: 1 PID: 1283 Comm: irq/145-icssg1- Tainted: G           O       6.6.58-rt45-ti-rt-01780-gc79d7ef3a56f-dirty #1
[ 1021.157482] Hardware name: Texas Instruments AM642 EVM (DT)
[ 1021.157487] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 1021.157497] pc : 0xffff80008041711c
[ 1021.157500] lr : 0xffff8000804170d4
[ 1021.157504] sp : ffff00000c753af0
[ 1021.157507] x29: ffff00000c753af0 x28: ffff00007bff0000 x27: ffff8000792cb008
[ 1021.157519] x26: 0000000000000040 x25: 0000000000000000 x24: 0000000000000007
[ 1021.157530] x23: 0000000000000080 x22: 0000000000000001 x21: 0000000000000000
[ 1021.157541] x20: ffff000008ca0000 x19: ffff00007bff0000 x18: fffffffffffeabd0
[ 1021.157552] x17: ffff7fffff23c000 x16: ffff00007ff87840 x15: ffff00007ff8c740
[ 1021.157564] x14: 0000000000000001 x13: 0a74756f656d6974 x12: ffff000001ad251c
[ 1021.157575] x11: ffff8000812f8000 x10: ffff800080b92810 x9 : 1fffe00000311cc1
[ 1021.157586] x8 : ffff00000c753ab0 x7 : 0000000000000040 x6 : fffffffffffffffe
[ 1021.157598] x5 : 0000000000000000 x4 : ffffffffffffffff x3 : ffff8000812f8038
[ 1021.157609] x2 : 0000000000000002 x1 : 0000000000000001 x0 : 0000000000000001
[ 1021.157620] Call trace:
[ 1021.157625]  0xffff80008041711c
[ 1021.157629]  0xffff8000806da498
[ 1021.157632]  0xffff8000792be704
[ 1021.157635]  0xffff8000792be9b0
[ 1021.157639]  0xffff8000792beb54
[ 1021.157642]  0xffff8000808b23d8
[ 1021.157645]  0xffff8000808b2620
[ 1021.157649]  0xffff80008003b850
[ 1021.157652]  0xffff80008003baf0
[ 1021.157655]  0xffff800080089d6c
[ 1021.157659]  0xffff80008008a218
[ 1021.157662]  0xffff800080059c40
[ 1021.157665]  0xffff800080014870
[ 1021.157674] Code: a94363f7 a8c47bfd d65f03c0 d4210000 (d4210000) 
[ 1021.157682] ---[ end trace 0000000000000000 ]---
[ 1021.157689] Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt
[ 1021.391042] SMP: stopping secondary CPUs
[ 1021.394967] Kernel Offset: disabled
[ 1021.398443] CPU features: 0x0,00000008,00000000,1000400b
[ 1021.403743] Memory Limit: none
[ 1021.406791] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt ]---

Daolin Qiu said:

If you would like to check before our next SDK releases, I would suggest building the Linux kernel image on 6.12

Ive updated it to "6.12.13-g1a2d611ee02d", page_pool problem is solved.

Mustafa Gül said:

While testing it in offload mode you should see this while trying to run PTP in L2 mode. Whenever we see "NETDEV WATCHDOG" error, it completely crashes interfaces and not be able to communicate.

I see same issue from 7 months ago here.

Hello,

I would like to confirm, is it not supporting P2P L2 with one step ?

configfile.cfg
[global]
network_transport	L2
delay_mechanism		P2P
time_stamping		hardware
twoStepFlag		    0
clientOnly		    1
clock_type		    BC
boundary_clock_jbod	1
tc_spanning_tree	1

$ ptp4l -f configfile.cfg -i eth1 -i eth2 -p /dev/ptp2

ptp4l[5418.779]: selected /dev/ptp2 as PTP clock
ptp4l[5418.820]: driver rejected most general HWTSTAMP filter
ptp4l[5418.820]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range
ptp4l[5418.860]: port 1 (eth1): INITIALIZING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[5418.892]: driver rejected most general HWTSTAMP filter
ptp4l[5418.892]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range
ptp4l[5418.924]: port 2 (eth2): INITIALIZING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[5418.924]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[5418.925]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE

Hello Mustafa,

Daolin is out of office until the end of March. Please expect a delayed response.

Hello Mustafa,

Mustafa Gül said:

I would like to confirm, is it not supporting P2P L2 with one step ?

In general, it appears that PTP over HSR offloaded is not supported. I have only specifically tested P2P L2 with two step configurations, but I suspect that with one-step HSR offloaded with PTP will still not work.

-Daolin

Hello,

Ah, Ive asked for HSR non-offload mode. Forgot to write it down.

Hi Mustafa, 

I have not specifically tested it out with the one-step configuration. Just to double check, does your PTP one-step configuration work without any HSR configuration (nonoffload)?

-Daolin

It gaves me this error in both configuration no-HSR network and HSR non-offload.

Ive shared full configs and output in previous post.

driver rejected most general HWTSTAMP filter
ioctl SIOCSHWTSTAMP failed: Numerical result out of range

Hi Mustafa, 

I will double check on the no-HSR network with PTP using one-step configuration to see if I can reproduce the same issue. I will try to get back to you by Friday. Please kindly ping this thread if you haven't received a response by Friday.

-Daolin

Hello,

Thank you for clarification. 

Will one-step mode be added to PRU_ICSSG driver or not in plan ?

I'm asking because of PTP Power Profile standart requires it.