AM6442: PTP path delay huge value

Hello Mateusz, 

Mateusz Szalkowski said:

It uses PRU interfaces to sync using PTP, but the way it synchronizes is inconsistent. Using 2 boards, one working as a master and second one as a slave. The issue comes up if the board booted after connecting the power and starting Linux without Eth cable connected. I that case PTP path delay has huge value ~300000 and master delay jumping around

Just to clarify, you are finding this large PTP path delay when the following sequence of events occurs?

1. Keep the PRU_ICSSG interfaces under test between the two boards disconnected

2. Boot the board into Linux

3. Connect the cable between the PRU_ICSSG interfaces

4. Run linuxptp on the PRU_ICSSG interfaces

Can you share the exact ptp4l command you used to produce the logs shared in this thread?

Mateusz Szalkowski said:

was this issue also known and was fixed in that release or 6.6 kernel

From my understanding, we have not specifically tested the case of what path delay looks like when the Ethernet cable is kept disconnected before booting into Linux. Are you able to produce the same behavior when using the TMDS64EVM rather than your custom designed boards?

-Daolin

Hello Daolin,

Thanks for your response.

Daolin Qiu said:

Just to clarify, you are finding this large PTP path delay when the following sequence of events occurs?

1. Keep the PRU_ICSSG interfaces under test between the two boards disconnected

2. Boot the board into Linux

3. Connect the cable between the PRU_ICSSG interfaces

4. Run linuxptp on the PRU_ICSSG interfaces

Points 3 and 4 should be swtiched so the sequence is:

1. Keep PRU_ICSSG disconnected

2. Boot board into Linux

3. Run linux ptp on PRU_ICSSG with following command
ptp4l -i eth3 -i eth2 -p /dev/ptp2 -s -2 -P -f ptp4l.0.cfg -q -m -l 6
(additional -s flag for slave device)

4. Connect the cable between PRU_ICSSG interfaces

Config file is as follows

[global]
#
# Default Data Set
#
ptp_minor_version 0
#twoStepFlag            1
clientOnly              1
priority1 128
priority2 128
domainNumber 0
#clockClass             248
#clockAccuracy          0xFE
#offsetScaledLogVariance        0xFFFF
#free_running           0
#freq_est_interval      1
#
# Port Data Set
#
#logAnnounceInterval    1
#logSyncInterval                -3
#logMinPdelayReqInterval        0
#announceReceiptTimeout 3
#delayAsymmetry         0
#fault_reset_interval   4
#neighborPropDelayThresh        800
#
# Run time options
#
#assume_two_step                1
#logging_level          6
#path_trace_enabled     1
#follow_up_info         1
#tx_timestamp_timeout   10
#use_syslog             1
#verbose                        0
#summary_interval       0
#kernel_leap            1
#
# Servo options
#
#pi_proportional_const  0.0
#pi_integral_const      0.0
#pi_proportional_scale  0.0
#pi_proportional_exponent       -0.3
#pi_proportional_norm_max       0.7
#pi_integral_scale      0.0
#pi_integral_exponent   0.4
#pi_integral_norm_max   0.3
step_threshold          0.1
#pi_f_offset_const      0.0000001
#pi_max_frequency       900000000
#clock_servo            pi
#
# Transport options
#
#transportSpecific      0x1
#ptp_dst_mac            01:80:C2:00:00:0E
#p2p_dst_mac            01:80:C2:00:00:0E
#
# Default interface options
#
#network_transport      L2
#delay_mechanism                P2P
#time_stamping          hardware

uds_address /var/run/ptp4l.0.socket

raw_send_vlan_enable 1

I don't have access to TMDS64EVM right now but I will try to reproduce it once I get access to it.

Regards,
Mateusz

Hello Mateusz,

Daolin is out of the office for the rest of March. Feel free to ping the thread in early April if you have not received a response.

Regards,

Nick

Hi Mateusz,

Mateusz Szalkowski said:

Using 2 boards, one working as a master and second one as a slave.

Mateusz Szalkowski said:

ptp4l -i eth3 -i eth2 -p /dev/ptp2 -s -2 -P -f ptp4l.0.cfg -q -m -l 6

You mention using only two boards in your test but your ptp4l options indicate using both ports of each board. What does your Ethernet connections look like/what is your test topology? You have both interfaces of each board connected with each other? (e.g. eth3 on board1 to eth3 on board2 and eth2 on board1 to eth2 on board2?)

I'm trying to see if I can replicate the issue you are seeing. In order to do so, I need more information about your test topology.

Mateusz Szalkowski said:

I don't have access to TMDS64EVM right now but I will try to reproduce it once I get access to it.

Additionally, were you able to observe the same issue on TMDS64EVM?

-Daolin

Hello Daolin,

Daolin Qiu said:

You have both interfaces of each board connected with each other? (e.g. eth3 on board1 to eth3 on board2 and eth2 on board1 to eth2 on board2?)

The test connection is eth3 to eth3. Both ports are passed already to ptp4l for the future use of redundant connection.

Daolin Qiu said:

Additionally, were you able to observe the same issue on TMDS64EVM?

One coleague from our team is working on reproducing this issue on EVM board this week. Once we observe that, I'll confirm.

Regards,
Mateusz

Hi Mateusz, 

Mateusz Szalkowski said:

We use 6.1 Linux RT kernel, I know that there have been some fixes for PRU and ICSSG in 10.01 SDK that includes 6.6 kernel, was this issue also known and was fixed in that release or 6.6 kernel?

Mateusz Szalkowski said:

ptp4l -i eth3 -i eth2 -p /dev/ptp2 -s -2 -P -f ptp4l.0.cfg -q -m -l 6

I attempted to use the same ptp4l options as you shared but there were some issues with reading the contents of ptp4l.0.cfg. Instead, I tested with the following ptp4l commands on Kernel 6.6 (SDK 10.1) with two TMDS64EVMs connected eth1 to eth1. I do see large path delay and master offset; however, after bringing down and back up the eth1 interface, the same large path delay and master offset is observed.

In general, we typically test with the hardware timestamping option (-H), which enables much better time synchronization than using software timestamping. Software timestamping appears to be what is used with your ptp4l options (no -H is specified). 

The steps in this link to try out hardware timestamped ptp4l on PRU_ICSSG interfaces might be of interest to you: https://software-dl.ti.com/processor-sdk-linux-rt/esd/AM64X/10_01_10_04/exports/docs/linux/Foundational_Components/PRU-ICSS/Linux_Drivers/PRU_ICSSG_Ethernet_Switch.html#ptp 

EVM1 (grandmaster) log:

root@am64xx-evm:~# ptp4l -i eth1 -i eth2 -S -2 -m 
ptp4l[117.702]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[117.710]: port 2 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[117.711]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[117.711]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[117.712]: port 2 (eth2): link down
ptp4l[117.712]: port 2 (eth2): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[117.722]: selected local clock 70ff76.fffe.1e9c46 as best master
ptp4l[117.722]: port 2 (eth2): assuming the grand master role
ptp4l[124.446]: port 1 (eth1): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[124.446]: port 1 (eth1): assuming the grand master role
ptp4l[124.446]: port 2 (eth2): assuming the grand master role
[ 168.675910] icssg-prueth icssg1-eth eth1: Link is Down
ptp4l[168.678]: port 1 (eth1): link down
ptp4l[168.680]: port 1 (eth1): MASTER to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[168.689]: port 1 (eth1): assuming the grand master role
ptp4l[168.689]: port 2 (eth2): assuming the grand master role
[ 180.964102] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
ptp4l[180.960]: port 1 (eth1): link up
ptp4l[180.980]: port 1 (eth1): FAULTY to LISTENING on INIT_COMPLETE
ptp4l[188.411]: port 1 (eth1): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[188.411]: port 1 (eth1): assuming the grand master role
ptp4l[188.411]: port 2 (eth2): assuming the grand master role
^Croot@am64xx-evm:~# uname -a
Linux am64xx-evm 6.6.58-rt45-ti-rt-01780-gc79d7ef3a56f-dirty #1 SMP PREEMPT_RT Wed Nov 27 14:15:26 x
root@am64xx-evm:~# 

EVM2 (follower) log:

root@am64xx-evm:~# ptp4l -i eth1 -i eth2 -S -2 -m -s
ptp4l[130.638]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[130.646]: port 2 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[130.647]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[130.647]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[130.648]: port 2 (eth2): link down
ptp4l[130.648]: port 2 (eth2): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[130.659]: selected local clock 70ff76.fffe.1f3dc6 as best master
ptp4l[130.659]: port 2 (eth2): assuming the grand master role
ptp4l[130.659]: port 2 (eth2): master state recommended in slave only mode
ptp4l[130.659]: port 2 (eth2): defaultDS.priority1 probably misconfigured
ptp4l[131.062]: port 1 (eth1): new foreign master 70ff76.fffe.1e9c46-1
ptp4l[135.062]: selected best master clock 70ff76.fffe.1e9c46
ptp4l[135.062]: foreign master not using PTP timescale
ptp4l[135.062]: port 1 (eth1): LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[138.062]: master offset 150474680 s0 freq     -0 path delay    88200
ptp4l[139.062]: master offset 150482582 s0 freq     -0 path delay    73753
ptp4l[140.062]: master offset 150482120 s0 freq     -0 path delay    62960
ptp4l[141.062]: master offset 150493344 s0 freq     -0 path delay    61626
ptp4l[142.062]: master offset 150486569 s0 freq     -0 path delay    61626
ptp4l[143.063]: master offset 150478900 s0 freq     -0 path delay    62960
ptp4l[144.063]: master offset 150470172 s0 freq     -0 path delay    68168
ptp4l[145.063]: master offset 150470613 s0 freq     -0 path delay    63727
ptp4l[146.063]: master offset 150458185 s0 freq     -0 path delay    65395
ptp4l[147.063]: master offset 150427695 s0 freq     -0 path delay    65395
ptp4l[148.063]: master offset 150435657 s0 freq     -0 path delay    63343
ptp4l[149.063]: master offset 150421196 s0 freq     -0 path delay    62009
ptp4l[150.064]: master offset 150456756 s0 freq     -0 path delay    56729
ptp4l[151.063]: master offset 150427289 s0 freq     -0 path delay    50901
ptp4l[152.064]: master offset 150459229 s0 freq     -0 path delay    49101
ptp4l[153.064]: master offset 150421944 s0 freq     -0 path delay    49101
ptp4l[154.064]: master offset 150451563 s1 freq  -1445 path delay    47487
ptp4l[155.064]: master offset    -29740 s2 freq  -4448 path delay    47487
ptp4l[155.064]: port 1 (eth1): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[156.064]: master offset     -6960 s2 freq  -2177 path delay    48420
ptp4l[157.064]: master offset    -31584 s2 freq  -4671 path delay    48786
ptp4l[158.064]: master offset    -22164 s2 freq  -3751 path delay    49312
ptp4l[159.064]: master offset    -35560 s2 freq  -5127 path delay    49861
ptp4l[160.064]: master offset      6193 s2 freq   -945 path delay    49861
ptp4l[161.064]: master offset    -32891 s2 freq  -4886 path delay    50795
ptp4l[162.065]: master offset     14514 s2 freq   -131 path delay    50442
ptp4l[163.064]: master offset    -34646 s2 freq  -5082 path delay    50936
ptp4l[164.065]: master offset     -6095 s2 freq  -2233 path delay    50936
ptp4l[165.065]: master offset      9943 s2 freq   -619 path delay    52272
ptp4l[166.065]: master offset     -5847 s2 freq  -2204 path delay    52272
ptp4l[167.065]: master offset    -41476 s2 freq  -5809 path delay    54054
ptp4l[168.065]: master offset     -5955 s2 freq  -2262 path delay    54054
ptp4l[169.065]: master offset    -42608 s2 freq  -5970 path delay    54054
ptp4l[170.065]: master offset     -8283 s2 freq  -2546 path delay    51920
ptp4l[171.065]: master offset    -40257 s2 freq  -5784 path delay    51920
root@am64xx-evm:~# ifconfig eth1 down
[ 174.274259] icssg-prueth icssg1-eth eth1: Link is Down
root@am64xx-evm:~# ifconfig eth1 up 
root@am64xx-evm:~# [ 187.299698] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow contrf

root@am64xx-evm:~# ptp4l -i eth1 -i eth2 -S -2 -m -s
ptp4l[190.613]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[190.628]: port 2 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[190.629]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[190.629]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[190.630]: port 2 (eth2): link down
ptp4l[190.630]: port 2 (eth2): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[190.640]: selected local clock 70ff76.fffe.1f3dc6 as best master
ptp4l[190.640]: port 2 (eth2): assuming the grand master role
ptp4l[190.640]: port 2 (eth2): master state recommended in slave only mode
ptp4l[190.640]: port 2 (eth2): defaultDS.priority1 probably misconfigured
ptp4l[195.027]: port 1 (eth1): new foreign master 70ff76.fffe.1e9c46-1
ptp4l[197.219]: port 2 (eth2): assuming the grand master role
ptp4l[197.219]: port 2 (eth2): master state recommended in slave only mode
ptp4l[197.219]: port 2 (eth2): defaultDS.priority1 probably misconfigured
ptp4l[199.027]: selected best master clock 70ff76.fffe.1e9c46
ptp4l[199.027]: foreign master not using PTP timescale
ptp4l[199.027]: port 1 (eth1): LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[201.027]: master offset     40416 s0 freq  -5784 path delay    65409
ptp4l[202.027]: master offset     22572 s0 freq  -5784 path delay    65102
ptp4l[203.027]: master offset     50452 s0 freq  -5784 path delay    56112
ptp4l[204.027]: master offset     49091 s0 freq  -5784 path delay    56112
ptp4l[205.028]: master offset     46493 s0 freq  -5784 path delay    63210
ptp4l[206.028]: master offset     52357 s0 freq  -5784 path delay    63210
ptp4l[207.028]: master offset     61200 s0 freq  -5784 path delay    62181
ptp4l[208.028]: master offset     36928 s0 freq  -5784 path delay    60493
ptp4l[209.028]: master offset     65843 s0 freq  -5784 path delay    60493
^Croot@am64xx-evm:~# uname -a
Linux am64xx-evm 6.6.58-rt45-ti-rt-01780-gc79d7ef3a56f-dirty #1 SMP PREEMPT_RT Wed Nov 27 14:15:26 Ux
root@am64xx-evm:~#

Error reading contents of ptp4l.0.cfg:

root@am64xx-evm:~# ptp4l -P -2 -i eth1 -i eth2 -f ptp4l.cfg -m -q -p /dev/ptp2                     
unknown option ptp_minor_version at line 5 in global section
failed to parse configuration file ptp4l.cfg

Do you have a requirement to use software timestamping or hardware timestamping?

-Daolin

Hello Daolin,

Daolin Qiu said:

Do you have a requirement to use software timestamping or hardware timestamping?

We require hardware timestamping. I checked also with -H flag, the behavior is the same. It seems like ptp4l manual mentions that hardware timestamping is the default option if -S or -L flag is not added explicitly.

Daolin Qiu said:

however, after bringing down and back up the eth1 interface, the same large path delay and master offset is observed.

Did you make sure that when you bring eth1 down also eth2 was down? In my case path delay shrinked only when both eth interfaces binded to PRU were brought down and at least one of them back up again (maybe even on both master and slave devices). If I understood correctly bringing both interfaces down causes PRU to unload the firmware and bringing at least one back up causes it to load the firmware again.

Regards,
Mateusz

Hello again,

I tried to use config file from the link that you attached and similar commands:

https://software-dl.ti.com/processor-sdk-linux-rt/esd/AM64X/10_01_10_04/exports/docs/linux/Foundational_Components/PRU-ICSS/Linux_Drivers/

In that case the devices don't even synchronize until I bring both interfaces down and then up again. I guess it's because of neighborPropDelayThresh value in the config file, it's quite small compared to the values that I got earlier before restarting the interfaces (800 in config vs ~200000 before restart). After removing that line from config file it synchronized with huge delay until interfaces restart, here are the logs from slave device:

root@proot@linux:/~# ptp4l -i eth2 -i eth3 -p /dev/ptp2 -2 -P -f ptp4l.0.cfg -q -m --step_threshold=1 -H -s
ptp4l[Fri Apr 04 10:05:23 2025]: selected /dev/ptp2 as PTP clock
ptp4l[Fri Apr 04 10:05:23 2025]: port 1 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 04 10:05:23 2025]: port 2 (eth3): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 04 10:05:23 2025]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 04 10:05:23 2025]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 04 10:05:23 2025]: port 1 (eth2): link down
ptp4l[Fri Apr 04 10:05:23 2025]: port 1 (eth2): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[Fri Apr 04 10:05:23 2025]: port 2 (eth3): link down
ptp4l[Fri Apr 04 10:05:23 2025]: port 2 (eth3): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[Fri Apr 04 10:05:23 2025]: selected local clock 8e3b08.fffe.27ab7c as best master
ptp4l[Fri Apr 04 10:05:23 2025]: port 1 (eth2): assuming the grand master role
ptp4l[Fri Apr 04 10:05:23 2025]: port 1 (eth2): master state recommended in slave only mode
ptp4l[Fri Apr 04 10:05:23 2025]: port 1 (eth2): defaultDS.priority1 probably misconfigured
ptp4l[Fri Apr 04 10:05:23 2025]: port 2 (eth3): assuming the grand master role
ptp4l[Fri Apr 04 10:05:23 2025]: port 2 (eth3): master state recommended in slave only mode
ptp4l[Fri Apr 04 10:05:23 2025]: port 2 (eth3): defaultDS.priority1 probably misconfigured
ptp4l[Fri Apr 04 10:05:32 2025]: port 2 (eth3): link up
ptp4l[Fri Apr 04 10:05:32 2025]: port 2 (eth3): FAULTY to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 04 10:05:36 2025]: port 1 (eth2): assuming the grand master role
ptp4l[Fri Apr 04 10:05:36 2025]: port 1 (eth2): master state recommended in slave only mode
ptp4l[Fri Apr 04 10:05:36 2025]: port 1 (eth2): defaultDS.priority1 probably misconfigured
ptp4l[Fri Apr 04 10:05:36 2025]: clock has been adjusted to frequency:     +0
ptp4l[Fri Apr 04 10:05:36 2025]: port 2 (eth3): new foreign master 06f8c1.fffe.518234-2
ptp4l[Fri Apr 04 10:05:38 2025]: selected best master clock 06f8c1.fffe.518234
ptp4l[Fri Apr 04 10:05:38 2025]: port 2 (eth3): LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[Fri Apr 04 10:05:38 2025]: port 2 (eth3): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[Fri Apr 04 10:05:39 2025]: rms 625243849 max 1250487724 freq  -2132 +/- 1455 delay 416455 +/-   0
ptp4l[Fri Apr 04 10:05:40 2025]: rms 1758 max 2622 freq  +2325 +/- 319 delay 413333 +/-   0
ptp4l[Fri Apr 04 10:05:41 2025]: rms 3670 max 6548 freq  +7106 +/- 1931 delay 407412 +/-   0
ptp4l[Fri Apr 04 10:05:42 2025]: rms 2849 max 5705 freq  +7772 +/- 2671 delay 401491 +/-   0
ptp4l[Fri Apr 04 10:05:43 2025]: rms 2552 max 5115 freq  +8027 +/- 3043 delay 394625 +/-   0
ptp4l[Fri Apr 04 10:05:44 2025]: rms 55197 max 98532 freq +102159 +/- 29245 delay 293504 +/-   0
ptp4l[Fri Apr 04 10:05:45 2025]: rms 66953 max 129927 freq +156519 +/- 52492 delay 160187 +/-   0
ptp4l[Fri Apr 04 10:05:46 2025]: rms 31047 max 47356 freq +65618 +/- 35746 delay 117750 +/-   0
ptp4l[Fri Apr 04 10:05:47 2025]: rms 41853 max 43792 freq   -176 +/- 13501 delay 106518 +/-   0
ptp4l[Fri Apr 04 10:05:48 2025]: rms 19450 max 20340 freq  -1698 +/- 5507 delay 87536 +/-   0
ptp4l[Fri Apr 04 10:05:49 2025]: rms 12740 max 16757 freq  -9453 +/- 435 delay 87536 +/-   0
ptp4l[Fri Apr 04 10:05:50 2025]: rms 4278 max 6984 freq  -5891 +/- 1439 delay 87536 +/-   0
ptp4l[Fri Apr 04 10:05:51 2025]: rms  705 max 1077 freq  -1593 +/- 949 delay 87536 +/-   0
ptp4l[Fri Apr 04 10:05:52 2025]: rms 7628 max 13912 freq -14623 +/- 4481 delay 102609 +/-   0
ptp4l[Fri Apr 04 10:05:53 2025]: rms 5046 max 10217 freq -13964 +/- 5384 delay 113840 +/-   0
ptp4l[Fri Apr 04 10:05:54 2025]: rms 3655 max 6299 freq -11273 +/- 5091 delay 124249 +/-   0
ptp4l[Fri Apr 04 10:05:55 2025]: rms 25405 max 59146 freq -43998 +/- 25362 delay 188558 +/- 31108
ptp4l[Fri Apr 04 10:05:56 2025]: port 2 (eth3): SLAVE to LISTENING on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[Fri Apr 04 10:05:56 2025]: selected local clock 8e3b08.fffe.27ab7c as best master
ptp4l[Fri Apr 04 10:05:56 2025]: port 1 (eth2): assuming the grand master role
ptp4l[Fri Apr 04 10:05:56 2025]: port 1 (eth2): master state recommended in slave only mode
ptp4l[Fri Apr 04 10:05:56 2025]: port 1 (eth2): defaultDS.priority1 probably misconfigured
ptp4l[Fri Apr 04 10:05:56 2025]: clock has been adjusted to frequency: -16961
^Cptp4l[Fri Apr 04 10:05:56 2025]: clock has been adjusted to frequency: -16961
root@linux:~# ifconfig eth2 down
root@linux:~# ifconfig eth3 down
root@linux:~# ifconfig eth3 up
root@linux:~# ifconfig eth2 up
root@linux:~# ptp4l -i eth2 -i eth3 -p /dev/ptp2 -2 -P -f ptp4l.0.cfg -q -m --step_threshold=1 -H -s
ptp4l[Fri Apr 04 10:06:15 2025]: selected /dev/ptp2 as PTP clock
ptp4l[Fri Apr 04 10:06:15 2025]: port 1 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 04 10:06:15 2025]: port 2 (eth3): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 04 10:06:15 2025]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 04 10:06:15 2025]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 04 10:06:15 2025]: port 1 (eth2): link down
ptp4l[Fri Apr 04 10:06:15 2025]: port 1 (eth2): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[Fri Apr 04 10:06:15 2025]: selected local clock 8e3b08.fffe.27ab7c as best master
ptp4l[Fri Apr 04 10:06:15 2025]: port 1 (eth2): assuming the grand master role
ptp4l[Fri Apr 04 10:06:15 2025]: port 1 (eth2): master state recommended in slave only mode
ptp4l[Fri Apr 04 10:06:15 2025]: port 1 (eth2): defaultDS.priority1 probably misconfigured
ptp4l[Fri Apr 04 10:06:19 2025]: port 2 (eth3): new foreign master 06f8c1.fffe.518234-2
ptp4l[Fri Apr 04 10:06:19 2025]: port 1 (eth2): assuming the grand master role
ptp4l[Fri Apr 04 10:06:19 2025]: port 1 (eth2): master state recommended in slave only mode
ptp4l[Fri Apr 04 10:06:19 2025]: port 1 (eth2): defaultDS.priority1 probably misconfigured
ptp4l[Fri Apr 04 10:06:19 2025]: clock has been adjusted to frequency:     +0
ptp4l[Fri Apr 04 10:06:21 2025]: selected best master clock 06f8c1.fffe.518234
ptp4l[Fri Apr 04 10:06:21 2025]: port 2 (eth3): LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[Fri Apr 04 10:06:22 2025]: port 2 (eth3): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[Fri Apr 04 10:06:22 2025]: rms 624974221 max 1249948458 freq  -2932 +/- 1906 delay   369 +/-   0
ptp4l[Fri Apr 04 10:06:23 2025]: rms  533 max  823 freq  -1151 +/- 717 delay   367 +/-   0
ptp4l[Fri Apr 04 10:06:24 2025]: rms  900 max  954 freq   +471 +/- 242 delay   359 +/-   0
ptp4l[Fri Apr 04 10:06:25 2025]: rms  565 max  760 freq   +777 +/-  37 delay   367 +/-   0
ptp4l[Fri Apr 04 10:06:26 2025]: rms  191 max  303 freq   +616 +/-  74 delay   362 +/-   0
ptp4l[Fri Apr 04 10:06:27 2025]: rms   28 max   42 freq   +420 +/-  34 delay   359 +/-   0
ptp4l[Fri Apr 04 10:06:28 2025]: rms   59 max   89 freq   +323 +/-  31 delay   361 +/-   0
ptp4l[Fri Apr 04 10:06:29 2025]: rms   41 max   65 freq   +300 +/-  25 delay   361 +/-   0
ptp4l[Fri Apr 04 10:06:30 2025]: rms    9 max   20 freq   +327 +/-  12 delay   361 +/-   0

Regards,
Mateusz

Hi Mateusz,

Mateusz Szalkowski said:

Did you make sure that when you bring eth1 down also eth2 was down? In my case path delay shrinked only when both eth interfaces binded to PRU were brought down and at least one of them back up again (maybe even on both master and slave devices). If I understood correctly bringing both interfaces down causes PRU to unload the firmware and bringing at least one back up causes it to load the firmware again.

Yes, I also tried by bringing both eth1 and eth2 down and both back up again. Your observation that bringing both down will bring down the PRU firmware and bringing at least one back up will load the PRU firmware again is the same as what I observe. Unfortunately, even after bringing both eth1 and eth2 down and back up again, the master offset and path delay remains large.

Mateusz Szalkowski said:

software-dl.ti.com/.../quote]

Thanks for trying out the options I pointed out in the SDK documentation. However, on my side, when I use the configurations in gPTP.cfg, right from the start, the delay is small and max offset decreases over time (expected). 

root@am64xx-evm:~# ptp4l -i eth1 -i eth2 -p /dev/ptp2 -2 -P -f gPTP.cfg -q -m --step_threshold=1 [ c
sH - 
ptp4l[459.280]: selected /dev/ptp2 as PTP clock
ptp4l[459.296]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[459.320]: port 2 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[459.321]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[459.321]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[459.321]: port 1 (eth1): link down
ptp4l[459.321]: port 1 (eth1): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[459.336]: selected local clock 70ff76.fffe.1f3dc6 as best master
ptp4l[459.336]: port 1 (eth1): assuming the grand master role
ptp4l[459.336]: port 1 (eth1): master state recommended in slave only mode
ptp4l[459.336]: port 1 (eth1): defaultDS.priority1 probably misconfigured
ptp4l[462.783]: port 2 (eth2): new foreign master 70ff76.fffe.1e9c46-2
ptp4l[463.209]: port 1 (eth1): assuming the grand master role
ptp4l[463.209]: port 1 (eth1): master state recommended in slave only mode
ptp4l[463.209]: port 1 (eth1): defaultDS.priority1 probably misconfigured
ptp4l[464.783]: selected best master clock 70ff76.fffe.1e9c46
ptp4l[464.783]: port 2 (eth2): LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[465.682]: port 2 (eth2): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[466.307]: rms 597555 max 1003340 freq -846809 +/- 521995 delay   70 +/-  0
ptp4l[467.308]: rms 139283 max 213313 freq -391583 +/- 189106 delay   71 +/-  0
ptp4l[468.308]: rms 231908 max 243217 freq +30368 +/- 61274 delay   71 +/-  0
ptp4l[469.309]: rms 147141 max 195253 freq +112675 +/- 5755 delay   71 +/-  0
ptp4l[470.309]: rms 47700 max 78752 freq +67841 +/- 17160 delay   70 +/-  0
[ 471.203331] page_pool_release_retry() stalled pool shutdown 62 inflight 181 sec
ptp4l[471.310]: rms 8537 max 13244 freq +17356 +/- 10845 delay   71 +/-  0
ptp4l[472.311]: rms 13816 max 14589 freq -6485 +/- 3363 delay   72 +/-  0
ptp4l[473.311]: rms 8445 max 11334 freq -10706 +/- 414 delay   72 +/-  0
ptp4l[474.312]: rms 2620 max 4392 freq -7873 +/- 1025 delay   72 +/-  0
ptp4l[475.312]: rms 534 max 821 freq -4921 +/- 623 delay   73 +/-  0
ptp4l[476.313]: rms 820 max 875 freq -3578 +/- 183 delay   73 +/-  0
ptp4l[477.314]: rms 483 max 650 freq -3365 +/- 29 delay   73 +/-  0
ptp4l[478.314]: rms 142 max 239 freq -3544 +/- 60 delay   73 +/-  0
ptp4l[479.315]: rms  33 max  49 freq -3714 +/- 35 delay   73 +/-  0
ptp4l[480.315]: rms  48 max  55 freq -3788 +/-  9 delay   73 +/-  0
ptp4l[481.316]: rms  28 max  40 freq -3799 +/-  4 delay   73 +/-  0
ptp4l[482.317]: rms   9 max  18 freq -3790 +/-  5 delay   72 +/-  0
ptp4l[483.318]: rms   4 max   9 freq -3789 +/-  3 delay   72 +/-  0
ptp4l[484.318]: rms   2 max   4 freq -3785 +/-  3 delay   71 +/-  0
ptp4l[485.319]: rms   5 max   7 freq -3779 +/-  5 delay   71 +/-  0
ptp4l[486.320]: rms   3 max   5 freq -3777 +/-  3 delay   70 +/-  0
ptp4l[487.320]: rms   2 max   2 freq -3778 +/-  2 delay   70 +/-  0
ptp4l[488.321]: rms   4 max   7 freq -3785 +/-  4 delay   70 +/-  0
ptp4l[489.322]: rms   3 max   5 freq -3786 +/-  3 delay   71 +/-  0
ptp4l[490.322]: rms   5 max   7 freq -3792 +/-  3 delay   70 +/-  0
ptp4l[491.323]: rms   2 max   4 freq -3787 +/-  3 delay   70 +/-  0
ptp4l[492.324]: rms   3 max   7 freq -3791 +/-  3 delay   70 +/-  0
ptp4l[493.324]: rms   4 max   5 freq -3791 +/-  5 delay   71 +/-  0
ptp4l[494.325]: rms   4 max   6 freq -3788 +/-  6 delay   72 +/-  0
ptp4l[495.325]: rms   3 max   6 freq -3784 +/-  3 delay   72 +/-  0
ptp4l[496.326]: rms   4 max   6 freq -3780 +/-  3 delay   72 +/-  0
ptp4l[497.327]: rms   6 max   9 freq -3791 +/-  6 delay   72 +/-  0
ptp4l[498.327]: rms   5 max   9 freq -3791 +/-  6 delay   72 +/-  0
ptp4l[499.328]: rms   3 max   6 freq -3784 +/-  3 delay   72 +/-  0
ptp4l[500.329]: rms   3 max   6 freq -3789 +/-  3 delay   72 +/-  0
ptp4l[501.329]: rms   3 max   3 freq -3790 +/-  3 delay   72 +/-  0
ptp4l[502.330]: rms   3 max   6 freq -3789 +/-  4 delay   72 +/-  0
ptp4l[503.330]: rms   3 max   6 freq -3789 +/-  4 delay   72 +/-  0
ptp4l[504.331]: rms   5 max   6 freq -3795 +/-  5 delay   72 +/-  0
ptp4l[505.332]: rms   4 max   6 freq -3797 +/-  5 delay   71 +/-  0
ptp4l[506.332]: rms   3 max   7 freq -3791 +/-  4 delay   72 +/-  0
ptp4l[507.333]: rms   3 max   4 freq -3791 +/-  4 delay   71 +/-  0
ptp4l[508.333]: rms   4 max   8 freq -3795 +/-  5 delay   71 +/-  0
ptp4l[509.334]: rms   3 max   4 freq -3790 +/-  3 delay   72 +/-  0
ptp4l[510.335]: rms   4 max   6 freq -3796 +/-  4 delay   71 +/-  0
ptp4l[511.335]: rms   3 max   5 freq -3793 +/-  4 delay   71 +/-  0
ptp4l[512.336]: rms   5 max   8 freq -3796 +/-  6 delay   71 +/-  0
ptp4l[513.337]: rms   3 max   5 freq -3796 +/-  4 delay   71 +/-  0

Mateusz Szalkowski said:

We use 6.1 Linux RT kernel, I know that there have been some fixes for PRU and ICSSG in 10.01 SDK that includes 6.6 kernel, was this issue also known and was fixed in that release or 6.6 kernel?

Are you able to try testing out with the 6.6 Kernel? My results were from using the 6.6 Kernel. To my knowledge, I was not aware of the issue you observed on 6.1 Kernel but testing with the 6.6. Kernel might help us understand whether or not the issue you observed ended up fixed on 6.6.

-Daolin

[/quote]

Hello Daolin,

Daolin Qiu said:

Are you able to try testing out with the 6.6 Kernel?

I just flashed the newer kernel version on the device and the same problem, logs below

root@linux:/~# ptp4l -P -2 -H -i eth2 -i eth3 -f ptp4l.0.cfg --step_threshold=1 -m -q -p /dev/ptp2 -s
ptp4l[Tue Apr 08 08:52:40 2025]: selected /dev/ptp2 as PTP clock
ptp4l[Tue Apr 08 08:52:40 2025]: port 1 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Tue Apr 08 08:52:40 2025]: port 2 (eth3): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Tue Apr 08 08:52:40 2025]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Tue Apr 08 08:52:40 2025]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Tue Apr 08 08:52:40 2025]: port 1 (eth2): error on fda[0]: Network is down
ptp4l[Tue Apr 08 08:52:40 2025]: port 1 (eth2): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[Tue Apr 08 08:52:40 2025]: port 2 (eth3): link down
ptp4l[Tue Apr 08 08:52:40 2025]: port 2 (eth3): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[Tue Apr 08 08:52:40 2025]: selected local clock 728da8.fffe.d5bb10 as best master
ptp4l[Tue Apr 08 08:52:40 2025]: port 1 (eth2): assuming the grand master role
ptp4l[Tue Apr 08 08:52:40 2025]: port 1 (eth2): master state recommended in slave only mode
ptp4l[Tue Apr 08 08:52:40 2025]: port 1 (eth2): defaultDS.priority1 probably misconfigured
ptp4l[Tue Apr 08 08:52:40 2025]: port 2 (eth3): assuming the grand master role
ptp4l[Tue Apr 08 08:52:40 2025]: port 2 (eth3): master state recommended in slave only mode
ptp4l[Tue Apr 08 08:52:40 2025]: port 2 (eth3): defaultDS.priority1 probably misconfigured
ptp4l[Tue Apr 08 08:52:40 2025]: port 1 (eth2): link down
ptp4l[Tue Apr 08 08:52:40 2025]: port 1 (eth2): assuming the grand master role
ptp4l[Tue Apr 08 08:52:40 2025]: port 1 (eth2): master state recommended in slave only mode
ptp4l[Tue Apr 08 08:52:40 2025]: port 1 (eth2): defaultDS.priority1 probably misconfigured
ptp4l[Tue Apr 08 08:52:40 2025]: port 2 (eth3): assuming the grand master role
ptp4l[Tue Apr 08 08:52:40 2025]: port 2 (eth3): master state recommended in slave only mode
ptp4l[Tue Apr 08 08:52:40 2025]: port 2 (eth3): defaultDS.priority1 probably misconfigured
ptp4l[Tue Apr 08 08:52:46 2025]: port 2 (eth3): link up
ptp4l[Tue Apr 08 08:52:46 2025]: port 2 (eth3): FAULTY to LISTENING on INIT_COMPLETE
ptp4l[Tue Apr 08 08:52:49 2025]: port 1 (eth2): assuming the grand master role
ptp4l[Tue Apr 08 08:52:49 2025]: port 1 (eth2): master state recommended in slave only mode
ptp4l[Tue Apr 08 08:52:49 2025]: port 1 (eth2): defaultDS.priority1 probably misconfigured
ptp4l[Tue Apr 08 08:52:49 2025]: clock has been adjusted to frequency:     +0
ptp4l[Tue Apr 08 08:52:51 2025]: port 2 (eth3): new foreign master 1e96f1.fffe.e2bdf1-2
ptp4l[Tue Apr 08 08:52:53 2025]: selected best master clock 1e96f1.fffe.e2bdf1
ptp4l[Tue Apr 08 08:52:53 2025]: port 2 (eth3): LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[Tue Apr 08 08:52:54 2025]: port 2 (eth3): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[Tue Apr 08 08:52:54 2025]: rms 18876979891 max 37753987238 freq -1153780 +/- 628424 delay 500186 +/-   0
ptp4l[Tue Apr 08 08:52:55 2025]: rms 324416 max 634574 freq -535880 +/- 383207 delay 546142 +/-   0
ptp4l[Tue Apr 08 08:52:56 2025]: rms 613887 max 840310 freq +297763 +/- 476278 delay 500186 +/-   0
ptp4l[Tue Apr 08 08:52:57 2025]: rms 275704 max 451005 freq +173217 +/- 310775 delay 498614 +/-   0
ptp4l[Tue Apr 08 08:52:58 2025]: rms 317205 max 532164 freq +127219 +/- 432450 delay 468606 +/-   0
ptp4l[Tue Apr 08 08:52:59 2025]: rms 319771 max 597081 freq -36012 +/- 434988 delay 396646 +/-   0
ptp4l[Tue Apr 08 08:53:00 2025]: rms 304925 max 607541 freq -71442 +/- 418428 delay 418527 +/-   0
ptp4l[Tue Apr 08 08:53:01 2025]: rms 327852 max 519659 freq +44572 +/- 446393 delay 348444 +/-   0
ptp4l[Tue Apr 08 08:53:02 2025]: rms 346485 max 510702 freq +68100 +/- 473336 delay 300136 +/-   0
ptp4l[Tue Apr 08 08:53:03 2025]: rms 368990 max 605712 freq +108156 +/- 507085 delay 300136 +/-   0
ptp4l[Tue Apr 08 08:53:04 2025]: rms 328911 max 600804 freq -13998 +/- 454513 delay 300136 +/-   0
ptp4l[Tue Apr 08 08:53:05 2025]: rms 335629 max 530007 freq -67110 +/- 461769 delay 285656 +/-   0
ptp4l[Tue Apr 08 08:53:06 2025]: rms 233233 max 382734 freq +151059 +/- 297104 delay 348576 +/-   0
ptp4l[Tue Apr 08 08:53:07 2025]: rms 319244 max 602300 freq -100371 +/- 416322 delay 423438 +/-   0
ptp4l[Tue Apr 08 08:53:08 2025]: rms 293373 max 687245 freq +53664 +/- 401268 delay 488567 +/-   0
ptp4l[Tue Apr 08 08:53:09 2025]: rms 354212 max 737131 freq -217726 +/- 455484 delay 515859 +/-   0
ptp4l[Tue Apr 08 08:53:10 2025]: rms 305376 max 551147 freq -34585 +/- 418045 delay 550402 +/-   0
ptp4l[Tue Apr 08 08:53:11 2025]: rms 300590 max 618537 freq -184625 +/- 408210 delay 585305 +/-   0
ptp4l[Tue Apr 08 08:53:12 2025]: rms 303808 max 489216 freq -27295 +/- 408525 delay 585305 +/-   0
ptp4l[Tue Apr 08 08:53:13 2025]: rms 341311 max 557190 freq +121888 +/- 442643 delay 585305 +/-   0
ptp4l[Tue Apr 08 08:53:14 2025]: rms 298426 max 598265 freq +10610 +/- 410881 delay 591258 +/-   0
ptp4l[Tue Apr 08 08:53:15 2025]: rms 277468 max 401039 freq +67721 +/- 390008 delay 591258 +/-   0
ptp4l[Tue Apr 08 08:53:16 2025]: rms 300430 max 524260 freq -48806 +/- 400626 delay 581610 +/-   0
ptp4l[Tue Apr 08 08:53:17 2025]: rms 320326 max 571756 freq +14332 +/- 442869 delay 581610 +/-   0
ptp4l[Tue Apr 08 08:53:18 2025]: rms 304295 max 566143 freq +62241 +/- 416590 delay 541115 +/-   0
ptp4l[Tue Apr 08 08:53:19 2025]: rms 316634 max 606079 freq -36699 +/- 434952 delay 525684 +/-   0
ptp4l[Tue Apr 08 08:53:20 2025]: rms 333188 max 543782 freq +129001 +/- 447943 delay 472374 +/-   0
ptp4l[Tue Apr 08 08:53:21 2025]: rms 322946 max 646793 freq  -9395 +/- 448614 delay 472374 +/-   0
ptp4l[Tue Apr 08 08:53:22 2025]: rms 335849 max 676928 freq -59275 +/- 465440 delay 481842 +/-   0
ptp4l[Tue Apr 08 08:53:23 2025]: rms 327323 max 577408 freq  +1136 +/- 449190 delay 523430 +/-   0
ptp4l[Tue Apr 08 08:53:24 2025]: rms 324972 max 489742 freq -72014 +/- 446105 delay 431057 +/-   0
ptp4l[Tue Apr 08 08:53:25 2025]: rms 150389 max 246885 freq +57924 +/- 192556 delay 519804 +/-   0
ptp4l[Tue Apr 08 08:53:26 2025]: rms 370323 max 618455 freq +122760 +/- 501851 delay 519804 +/-   0
ptp4l[Tue Apr 08 08:53:27 2025]: rms 373415 max 605270 freq -87962 +/- 504239 delay 519804 +/-   0
ptp4l[Tue Apr 08 08:53:28 2025]: rms 288852 max 431584 freq  +6707 +/- 403765 delay 542156 +/-   0
ptp4l[Tue Apr 08 08:53:29 2025]: rms 274937 max 513645 freq -76607 +/- 364133 delay 542156 +/-   0
ptp4l[Tue Apr 08 08:53:30 2025]: rms 332241 max 651583 freq +36113 +/- 461201 delay 542156 +/-   0
ptp4l[Tue Apr 08 08:53:31 2025]: rms 314792 max 629167 freq -30764 +/- 437327 delay 550583 +/-   0
ptp4l[Tue Apr 08 08:53:32 2025]: rms 297103 max 588634 freq -13689 +/- 409451 delay 558477 +/-   0
ptp4l[Tue Apr 08 08:53:34 2025]: rms 297564 max 601589 freq -72524 +/- 410504 delay 542156 +/-   0
ptp4l[Tue Apr 08 08:53:35 2025]: rms 294563 max 491644 freq -20324 +/- 403268 delay 542156 +/-   0
ptp4l[Tue Apr 08 08:53:36 2025]: rms 188455 max 353931 freq +43533 +/- 248101 delay 553676 +/-   0
ptp4l[Tue Apr 08 08:53:36 2025]: clock has been adjusted to frequency: +23409

And to clarify, the steps to reproduce it were:

1. Flash the firmware with newer kernel
2. Disconnect eth cables
3. Unplug power source, plug it back and wait for system to boot (if the devices synchronized correctly once, reboot from commandline doesn't cause this issue, but disconnecting power source does)
4. After boot, enable eth3 (PRU) ports on both devices
5. Run ptp4l commands on 2 devices (command for slave device in the logs above, for master without -s flag). Config file same for both:
   

   [global]
   gmCapable 1
   priority1 248
   priority2 248
   logAnnounceInterval 0
   logSyncInterval -3
   syncReceiptTimeout 3
   # not synchronizing at all withaur re-initializing eth interfaces with that low threshold below
   # neighborPropDelayThresh 800
   min_neighbor_prop_delay -20000000
   assume_two_step 1
   path_trace_enabled 1
   follow_up_info 1
   transportSpecific 0x1
   ptp_dst_mac 01:80:C2:00:00:0E
   network_transport L2
   delay_mechanism P2P
   ingressLatency 88
   egressLatency 288

6. Connect eth cable between eth3 interfaces on both devices - ptp4l synchronizes with high delay value
7. Disable and re-enable PRU ethernet ports on both devices - slave device synchronizes with low delay value

Regards,
Mateusz

Hi Mateusz, 

One other thing to try: don't add the -s flag at all, simply change the "priority1" configuration in the ptp configuration file to a larger value than the configuration on the grandmaster to configure that device as a follower. Do you see the same issue?

Mateusz Szalkowski said:

In that case the devices don't even synchronize until I bring both interfaces down and then up again. I guess it's because of neighborPropDelayThresh value in the config file, it's quite small compared to the values that I got earlier before restarting the interfaces (800 in config vs ~200000 before restart). After removing that line from config file it synchronized with huge delay until interfaces restart

Unfortunately, using the gPTP configuration as I specified before, I was unable to see the large offset values that you see. The only thing I can think of at the moment is perhaps there is a ptp configuration change that is necessary on your hardware. Which configuration that needs to be changed is still unclear to me. Can you try perhaps adjusting the neighborPropDelayThresh instead of removing it? Maybe change it to some other value than 200000.

-Daolin

Hello Daolin,

It seems like one of the differences between our device and evaluation board is that our device has 2 PRU ports configured in device tree while for EVM board only 1 PRU port is active (fragment of EVM device tree below).

        ethernet-ports {
			#address-cells = <1>;
			#size-cells = <0>;
			icssg1_emac0: port@0 {
				reg = <0>;
				phy-handle = <&icssg1_phy1>;
				phy-mode = "rgmii-id";
				/* Filled in by bootloader */
				local-mac-address = [00 00 00 00 00 00];
			};
			icssg1_emac1: port@1 {
				reg = <1>;
				/* Filled in by bootloader */
				local-mac-address = [00 00 00 00 00 00];
				status = "disabled";
			};
		};

I changed our device tree to use only 1 PRU port and second port has status "disabled" now and the issue is gone, now it synchronizes with small path delay every time. On the other hand, when I switched ports status in dts file so that emac0 is disabled and emac1 is enabled - I cannot get the sync at all, logs below

root@linux:~# ptp4l -P -2 -H -i eth2 -f ptp4l.0.cfg --step_threshold=1 -m -q -p /dev/ptp2 -s
ptp4l[Fri Apr 11 09:26:46 2025]: selected /dev/ptp2 as PTP clock
ptp4l[Fri Apr 11 09:26:46 2025]: port 1 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 11 09:26:46 2025]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 11 09:26:46 2025]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 11 09:26:47 2025]: timed out while polling for tx timestamp
ptp4l[Fri Apr 11 09:26:47 2025]: increasing tx_timestamp_timeout or increasing kworker priority may correct this issue, but a driver bug likely causes it
ptp4l[Fri Apr 11 09:26:47 2025]: port 1 (eth2): send peer delay request failed
ptp4l[Fri Apr 11 09:26:47 2025]: port 1 (eth2): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[Fri Apr 11 09:27:04 2025]: port 1 (eth2): FAULTY to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 11 09:27:05 2025]: timed out while polling for tx timestamp
ptp4l[Fri Apr 11 09:27:05 2025]: increasing tx_timestamp_timeout or increasing kworker priority may correct this issue, but a driver bug likely causes it
ptp4l[Fri Apr 11 09:27:05 2025]: port 1 (eth2): send peer delay request failed
ptp4l[Fri Apr 11 09:27:05 2025]: port 1 (eth2): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[Fri Apr 11 09:27:21 2025]: port 1 (eth2): FAULTY to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 11 09:27:22 2025]: timed out while polling for tx timestamp
ptp4l[Fri Apr 11 09:27:22 2025]: increasing tx_timestamp_timeout or increasing kworker priority may correct this issue, but a driver bug likely causes it
ptp4l[Fri Apr 11 09:27:22 2025]: port 1 (eth2): send peer delay request failed
ptp4l[Fri Apr 11 09:27:22 2025]: port 1 (eth2): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[Fri Apr 11 09:27:38 2025]: port 1 (eth2): FAULTY to LISTENING on INIT_COMPLETE
ptp4l[Fri Apr 11 09:27:39 2025]: timed out while polling for tx timestamp
ptp4l[Fri Apr 11 09:27:39 2025]: increasing tx_timestamp_timeout or increasing kworker priority may correct this issue, but a driver bug likely causes it
ptp4l[Fri Apr 11 09:27:39 2025]: port 1 (eth2): send peer delay request failed
ptp4l[Fri Apr 11 09:27:39 2025]: port 1 (eth2): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)

Are you able to check the scenario with both ports enabled in the dts file? I'm aware that there's only one physical port on the evaluation board but I would assume that just enabling both in device tree might be enough to recreate the issue.

Regards,
Mateusz

Hi Mateusz,

Mateusz Szalkowski said:

Are you able to check the scenario with both ports enabled in the dts file?

Actually, although if you check the EVM dts (k3-am642-evm.dts) itself you will see only one PRU port enabled as you have shared, the second PRU port is later enabled if the k3-am642-evm-icssg1-dualemac.dtbo overlay file is applied (https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/latest/exports/docs/linux/Foundational_Components/PRU-ICSS/Linux_Drivers/PRU_ICSSG_Ethernet.html#cpsw-pru-ethernet-selection). This essentially switches one of the CPSW ports to be a PRU_ICSSG port. Is it this way that I have been testing on my EVM to enable both PRU ICSSG1 ports and test PTP synchronization on these two PRU_ICSSG ports.

Mateusz Szalkowski said:

I changed our device tree to use only 1 PRU port and second port has status "disabled" now and the issue is gone, now it synchronizes with small path delay every time.

When you do this, it seems that in the ptp4l command you are only adding one eth interface instead of the two you were previously using?

Mateusz Szalkowski said:

On the other hand, when I switched ports status in dts file so that emac0 is disabled and emac1 is enabled - I cannot get the sync at all, logs below

Have you verified that when emac0 is disabled and emac1 is enabled, you are able to have basic Ethernet traffic pass through the second PRU port without any issue? In other words, have you tested with a ping (no PTP) that packets actually can get sent and received on the second PRU port?

One thing I am suspecting but am trying to get validation from a colleague on is that the PRU firmware that gets loaded in Linux when a PRU_ICSSG port is enabled is only tied to the first PRU ICSSG port or not. If is it only tied to the first port, that may be a reason for the second PRU port not working.

Specifically I'm talking about this firmware you can view in the boot logs:

root@am64xx-evm:~# dmesg | grep pru 
[  11.341751] remoteproc remoteproc5: 3000a000.txpru is available
[  11.380535] remoteproc remoteproc6: 3000c000.txpru is available
[  11.389457] remoteproc remoteproc7: 3008a000.txpru is available
[  11.415953] remoteproc remoteproc8: 3008c000.txpru is available
[  12.433020] remoteproc remoteproc9: 30034000.pru is available
[  12.452531] remoteproc remoteproc11: 300b4000.pru is available
[  12.466158] remoteproc remoteproc13: 30038000.pru is available
[  12.511052] remoteproc remoteproc15: 300b8000.pru is available
[  12.663534] icssg-prueth icssg1-eth: TI PRU ethernet driver initialized: dual EMAC mode
[  14.131611] remoteproc remoteproc11: powering up 300b4000.pru
[  14.142527] remoteproc remoteproc11: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, siz6
[  14.142647] remoteproc remoteproc11: remote processor 300b4000.pru is now up
[  14.156388] remoteproc remoteproc12: Booting fw image ti-pruss/am65x-sr2-rtu0-prueth-fw.elf, siz4
[  14.156489] remoteproc remoteproc7: powering up 3008a000.txpru
[  14.164150] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, si0
[  14.164215] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[  14.165612] remoteproc remoteproc15: powering up 300b8000.pru
[  14.173261] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru1-prueth-fw.elf, siz6
[  14.173389] remoteproc remoteproc15: remote processor 300b8000.pru is now up
[  14.177557] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu1-prueth-fw.elf, siz0
[  14.177659] remoteproc remoteproc8: powering up 3008c000.txpru
[  14.187267] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1

-Daolin

Hello Daolin,

Daolin Qiu said:

This essentially switches one of the CPSW ports to be a PRU_ICSSG port

Ok, I missed that part, sorry about that.

Daolin Qiu said:

When you do this, it seems that in the ptp4l command you are only adding one eth interface instead of the two you were previously using?

That's correct. After further tests though I managed to recreate the issue also with configuration with only one PRU eth port...

Daolin Qiu said:

Have you verified that when emac0 is disabled and emac1 is enabled, you are able to have basic Ethernet traffic pass through the second PRU port without any issue?

I checked that today, there is no traffic on emac1 and no packets in ifconfig

eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.11  netmask 255.255.255.0  broadcast 192.168.2.255
        ether 9e:87:84:fa:6b:6c  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Daolin Qiu said:

the PRU firmware that gets loaded in Linux when a PRU_ICSSG port is enabled is only tied to the first PRU ICSSG port

It seems like the firmware gets loaded also when that single emac1 is enabled/disabled, at least that's what I can see in dmesg logs

root@linux:~# dmesg | grep -i pru
[    0.869954] remoteproc remoteproc0: 30034000.pru is available
[    0.872360] remoteproc remoteproc2: 3000a000.txpru is available
[    0.873554] remoteproc remoteproc3: 30038000.pru is available
[    0.875727] remoteproc remoteproc5: 3000c000.txpru is available
[    0.936247] remoteproc remoteproc6: 300b4000.pru is available
[    0.938575] remoteproc remoteproc8: 3008a000.txpru is available
[    0.939714] remoteproc remoteproc9: 300b8000.pru is available
[    0.942267] remoteproc remoteproc11: 3008c000.txpru is available
[   12.307165] icssg-prueth icssg0-eth: port 2: using random MAC addr: ae:d0:46:a2:9e:1c
[   12.398898] icssg-prueth icssg0-eth: TI PRU ethernet driver initialized: single EMAC mode
[  620.830211] remoteproc remoteproc3: powering up 30038000.pru
[  620.834056] remoteproc remoteproc3: Booting fw image ti-pruss/am65x-sr2-pru1-prueth-fw.elf, size 41208
[  620.834169] remoteproc remoteproc3: remote processor 30038000.pru is now up
[  620.835882] remoteproc remoteproc4: Booting fw image ti-pruss/am65x-sr2-rtu1-prueth-fw.elf, size 30124
[  620.840227] remoteproc remoteproc5: powering up 3000c000.txpru
[  620.841851] remoteproc remoteproc5: Booting fw image ti-pruss/am65x-sr2-txpru1-prueth-fw.elf, size 35184
[  620.841945] remoteproc remoteproc5: remote processor 3000c000.txpru is now up
[  620.847506] icssg-prueth icssg0-eth: settime timeout
[  623.908490] icssg-prueth icssg0-eth eth2: Link is Up - 100Mbps/Full - flow control off
[  652.580379] icssg-prueth icssg0-eth eth2: Link is Down
[  656.676498] icssg-prueth icssg0-eth eth2: Link is Up - 100Mbps/Full - flow control off
[ 3646.510484] icssg-prueth icssg0-eth eth2: Link is Down
[ 3646.515115] remoteproc remoteproc5: stopped remote processor 3000c000.txpru
[ 3646.515207] remoteproc remoteproc3: stopped remote processor 30038000.pru
[ 3648.404681] remoteproc remoteproc3: powering up 30038000.pru
[ 3648.405172] remoteproc remoteproc3: Booting fw image ti-pruss/am65x-sr2-pru1-prueth-fw.elf, size 41208
[ 3648.405285] remoteproc remoteproc3: remote processor 30038000.pru is now up
[ 3648.405570] remoteproc remoteproc4: Booting fw image ti-pruss/am65x-sr2-rtu1-prueth-fw.elf, size 30124
[ 3648.405636] remoteproc remoteproc5: powering up 3000c000.txpru
[ 3648.405825] remoteproc remoteproc5: Booting fw image ti-pruss/am65x-sr2-txpru1-prueth-fw.elf, size 35184
[ 3648.405861] remoteproc remoteproc5: remote processor 3000c000.txpru is now up
[ 3648.413182] icssg-prueth icssg0-eth: settime timeout

Regards,
Mateusz

Hi Mateusz, 

Mateusz Szalkowski said:

I checked that today, there is no traffic on emac1 and no packets in ifconfig

Mateusz Szalkowski said:

It seems like the firmware gets loaded also when that single emac1 is enabled/disabled, at least that's what I can see in dmesg logs

After checking with my colleague, it appears that disabling the icssg ports (either one) should not impact if the firmware gets loaded or not. The firmware gets loaded as long as at least one port is enabled so the firmware doesn't appear to be the reason for no traffic from appearing for emac1, when emac1 is the only port enabled.

Mateusz Szalkowski said:

After further tests though I managed to recreate the issue also with configuration with only one PRU eth port...

It sounds like the single port enablement method also doesn't resolve the issue. What changed in your further tests to enable you to recreate the issue?

Daolin Qiu said:

Unfortunately, using the gPTP configuration as I specified before, I was unable to see the large offset values that you see. The only thing I can think of at the moment is perhaps there is a ptp configuration change that is necessary on your hardware. Which configuration that needs to be changed is still unclear to me. Can you try perhaps adjusting the neighborPropDelayThresh instead of removing it? Maybe change it to some other value than 200000.

Did you get a chance to try this?

-Daolin

Hello Daolin,

Daolin Qiu said:

It sounds like the single port enablement method also doesn't resolve the issue. What changed in your further tests to enable you to recreate the issue?

The only thing I changed during the test was disconnecting power cable for a little longer time, initially when it was unplugged for ~1 second it seemed to work but then during later tests I unplugged it for 5-10 seconds and in that case the issue was back.

Daolin Qiu said:

Unfortunately, using the gPTP configuration as I specified before, I was unable to see the large offset values that you see. The only thing I can think of at the moment is perhaps there is a ptp configuration change that is necessary on your hardware. Which configuration that needs to be changed is still unclear to me. Can you try perhaps adjusting the neighborPropDelayThresh instead of removing it? Maybe change it to some other value than 200000.

Did you get a chance to try this?

Yes, I tried playing with those settings, but it seemed like if delay value was larger than that neighborPropDelayThresh, external master clock was ignored and there was no synchronization at all.

Regards,
Mateusz

Hi Mateusz,

Mateusz Szalkowski said:

The only thing I changed during the test was disconnecting power cable for a little longer time, initially when it was unplugged for ~1 second it seemed to work but then during later tests I unplugged it for 5-10 seconds and in that case the issue was back.

Mateusz Szalkowski said:

The test connection is eth3 to eth3. Both ports are passed already to ptp4l for the future use of redundant connection.

You mentioned before that your topology was two custom boards. When you say disconnecting the power cable, are you powering off one of the two custom boards and keeping the other on and continuously running PTP?

-Daolin

Hi Daolin,

Daolin Qiu said:

are you powering off one of the two custom boards and keeping the other on and continuously running PTP

Exactly that. The master device is still powered on and running PTP.

Regards,
Mateusz

Hi Mateusz,

Mateusz Szalkowski said:

Exactly that. The master device is still powered on and running PTP.

Thanks for clarifying this, I was not aware of this setup before when I last tested. I hope to try testing again with this setup tomorrow or early next week.

In the meantime, do you have access to TMDS64EVMs (AM64x TI EVMs)? If so, are you able to replicate this issue and how the large offset can be reduced when the interfaces are brought down and back up again on TMDS64EVMs?

-Daolin

Hi Mateusz,

Daolin Qiu said:

Thanks for clarifying this, I was not aware of this setup before when I last tested. I hope to try testing again with this setup tomorrow or early next week.

Here is the breakdown of what I tried tested following the info about keeping the grandmaster board running PTP while the follower board is powered down.

Physical topology:

TMDS64EVM (EVM1 - Grandmaster clock) <----eth1---> TMDS64EVM (EVM2 - Follower clock)

Steps:

1. Use SDK 10.1 (Kernel 6.6)

2. Keep all eth ports disconnected on both EVMs

3. Boot both EVMs up and synchronize to small PTP offset with both eth1 and eth2 configured as part of ptp4l, ensuring ptp4l on EVM1 is running in background

ptp4l -P -2 -H -i eth1 -i eth2 -f gPTP.cfg --step_threshold=1 -m -q -p /dev/ptp2 &

4. Power off EVM2 (toggle power-down switch & unplug power supply for 5-10 sec) and keep EVM1 running ptp4l in background

5. Disconnect eth1 cable, ensuring all other eth ports also kept disconnected

6. After turning back on EVM2, enable eth1 PRU port on both EVMs (by disabling both eth1 and eth2 ports and only enabling eth1 port)

7. Run ptp4l on EVM2

8. Connect eth1 cable and check for large delay value 

The results show that with ptp4l, EVM2 was still able to synchronize to single digit offset values (see log below)

Log:

EVM2 (follower clock) after powering back on after power off:

root@am64xx-evm:~# ip link set dev eth1 down
root@am64xx-evm:~# ip link set dev eth2 down                                                        
[ 875.546880] remoteproc remoteproc7: stopped remote processor 3008a000.txpru
[ 875.546970] remoteproc remoteproc14: stopped remote processor 30084000.rtu
[ 875.546981] remoteproc remoteproc13: stopped remote processor 300b4000.pru
[ 875.546992] remoteproc remoteproc8: stopped remote processor 3008c000.txpru
[ 875.547001] remoteproc remoteproc16: stopped remote processor 30086000.rtu
[ 875.547010] remoteproc remoteproc15: stopped remote processor 300b8000.pru                                                       
root@am64xx-evm:~# ip link set dev eth1 up 
[ 888.737379] remoteproc remoteproc13: powering up 300b4000.pru
[ 888.737816] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, size6
[ 888.737901] remoteproc remoteproc13: unsupported resource 5
[ 888.737933] remoteproc remoteproc13: remote processor 300b4000.pru is now up
[ 888.737972] remoteproc remoteproc14: powering up 30084000.rtu
[ 888.738169] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu0-prueth-fw.elf, size4
[ 888.738207] remoteproc remoteproc14: remote processor 30084000.rtu is now up
[ 888.738238] remoteproc remoteproc7: powering up 3008a000.txpru
[ 888.738400] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, siz0
[ 888.738434] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[ 888.738840] remoteproc remoteproc15: powering up 300b8000.pru
[ 888.739010] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru1-prueth-fw.elf, size6
[ 888.739030] remoteproc remoteproc15: unsupported resource 5
[ 888.739054] remoteproc remoteproc15: remote processor 300b8000.pru is now up
[ 888.739077] remoteproc remoteproc16: powering up 30086000.rtu
[ 888.739197] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu1-prueth-fw.elf, size0
[ 888.739222] remoteproc remoteproc16: remote processor 30086000.rtu is now up
[ 888.739241] remoteproc remoteproc8: powering up 3008c000.txpru
[ 888.739351] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-prueth-fw.elf, siz8
[ 888.739382] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
root@am64xx-evm:~# [ 888.813478] pps pps1: new PPS source ptp2

root@am64xx-evm:~#
root@am64xx-evm:~# ptp4l -P -2 -H -i eth1 -i eth2 -f gPTP.cfg --step_threshold=1 -m -q -p /dev/ptp2 
ptp4l[153.128]: selected /dev/ptp2 as PTP clock
ptp4l[153.145]: port 1 (eth1): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[153.157]: port 2 (eth2): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[153.158]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[153.158]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[153.158]: port 1 (eth1): link down
ptp4l[153.158]: port 1 (eth1): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[153.174]: port 2 (eth2): link down
ptp4l[153.174]: port 2 (eth2): LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[153.190]: selected local clock 70ff76.fffe.1f3dc6 as best master
ptp4l[153.190]: port 1 (eth1): assuming the grand master role
ptp4l[153.190]: port 2 (eth2): assuming the grand master role
[ 169.636144] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
ptp4l[169.634]: port 1 (eth1): link up
ptp4l[169.649]: port 1 (eth1): FAULTY to LISTENING on INIT_COMPLETE
ptp4l[173.023]: port 1 (eth1): new foreign master 70ff76.fffe.1e9c46-1
ptp4l[173.105]: port 1 (eth1): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[173.105]: port 1 (eth1): assuming the grand master role
ptp4l[173.105]: port 2 (eth2): assuming the grand master role
ptp4l[175.023]: selected best master clock 70ff76.fffe.1e9c46
ptp4l[175.023]: port 1 (eth1): MASTER to UNCALIBRATED on RS_SLAVE
ptp4l[176.024]: port 1 (eth1): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[176.650]: rms 127538930347451 max 255077860695135 freq -846124 +/- 521603 delay   68 +/-  0
ptp4l[177.650]: rms 139151 max 213139 freq -391250 +/- 188926 delay   68 +/-  0
ptp4l[178.651]: rms 231749 max 243046 freq +30416 +/- 61238 delay   68 +/-  0
ptp4l[179.651]: rms 147048 max 195121 freq +112683 +/- 5756 delay   67 +/-  0
ptp4l[180.652]: rms 47673 max 78710 freq +67878 +/- 17156 delay   67 +/-  0
ptp4l[181.652]: rms 8534 max 13237 freq +17423 +/- 10843 delay   67 +/-  0
ptp4l[182.653]: rms 13807 max 14587 freq -6403 +/- 3355 delay   67 +/-  0
ptp4l[183.654]: rms 8438 max 11317 freq -10618 +/- 414 delay   67 +/-  0
ptp4l[184.654]: rms 2621 max 4393 freq -7795 +/- 1024 delay   67 +/-  0
ptp4l[185.655]: rms 531 max 809 freq -4851 +/- 622 delay   67 +/-  0
ptp4l[186.655]: rms 816 max 869 freq -3511 +/- 184 delay   67 +/-  0
ptp4l[187.656]: rms 482 max 653 freq -3297 +/- 31 delay   69 +/-  0
ptp4l[188.656]: rms 139 max 237 freq -3482 +/- 63 delay   70 +/-  0
ptp4l[189.657]: rms  34 max  49 freq -3653 +/- 34 delay   70 +/-  0
ptp4l[190.657]: rms  49 max  55 freq -3728 +/- 10 delay   69 +/-  0
ptp4l[191.658]: rms  29 max  33 freq -3740 +/-  4 delay   69 +/-  0
ptp4l[192.658]: rms  13 max  18 freq -3738 +/-  5 delay   69 +/-  0
ptp4l[193.659]: rms   3 max   3 freq -3730 +/-  3 delay   69 +/-  0
ptp4l[194.660]: rms   4 max   6 freq -3721 +/-  4 delay   69 +/-  0
ptp4l[195.660]: rms   3 max   6 freq -3724 +/-  4 delay   69 +/-  0
ptp4l[196.661]: rms   4 max   6 freq -3719 +/-  4 delay   69 +/-  0
ptp4l[197.661]: rms   4 max   6 freq -3723 +/-  5 delay   69 +/-  0
ptp4l[198.662]: rms   3 max   6 freq -3722 +/-  4 delay   69 +/-  0
ptp4l[199.663]: rms  10 max  12 freq -3740 +/-  4 delay   69 +/-  0
ptp4l[200.663]: rms   4 max   6 freq -3737 +/-  5 delay   69 +/-  0
ptp4l[201.664]: rms   4 max   6 freq -3734 +/-  6 delay   69 +/-  0
ptp4l[202.664]: rms   4 max   9 freq -3736 +/-  5 delay   69 +/-  0
ptp4l[203.665]: rms   4 max   6 freq -3736 +/-  5 delay   69 +/-  0
ptp4l[204.665]: rms   3 max   6 freq -3737 +/-  5 delay   70 +/-  0
ptp4l[205.666]: rms   4 max   7 freq -3740 +/-  5 delay   70 +/-  0
ptp4l[206.667]: rms   5 max   7 freq -3742 +/-  6 delay   70 +/-  0
ptp4l[207.667]: rms   3 max   7 freq -3740 +/-  5 delay   70 +/-  0
ptp4l[208.668]: rms   4 max   7 freq -3741 +/-  5 delay   70 +/-  0
ptp4l[209.668]: rms   5 max  10 freq -3746 +/-  5 delay   70 +/-  0
ptp4l[210.669]: rms   6 max  10 freq -3752 +/-  6 delay   70 +/-  0
ptp4l[211.669]: rms   4 max   7 freq -3754 +/-  4 delay   70 +/-  0
ptp4l[212.670]: rms   5 max   7 freq -3749 +/-  6 delay   70 +/-  0
ptp4l[213.670]: rms   4 max   7 freq -3751 +/-  5 delay   70 +/-  0
ptp4l[214.671]: rms   4 max   7 freq -3754 +/-  6 delay   70 +/-  0
^Croot@am64xx-evm:~# uname -a
Linux am64xx-evm 6.6.58-rt45-ti-rt-01780-gc79d7ef3a56f-dirty #1 SMP PREEMPT_RT Wed Nov 27 14:15:26 Ux
root@am64xx-evm:~#

My suspicion is that you are powering off your custom board (the follower clock device) by directly unplugging power supply?

Are you able replicate the large path delay you are seeing with TMDS64EVMs? For the steps I described here, I was still unable to see the large delay path that you were seeing on your custom board.

-Daolin