PROCESSOR-SDK-AM64X: PRU PPS PTP Sychronization Issues

Hello Daolin,

Thank you for your reply.  

Daolin Qiu said:

Please note that the PPS signal from testptp can only be synchronized to ptp4l by ensuring that testptp runs after ptp4l is running. If ptp4l were ever to restart, you should also ensure that testptp restarts so that the PPS signal can be resynchronized to ptp4l.

The problem I described also occurs in another situation. According to the note, even if I first start the ptp4l (wait for synchronization) and then turn on the PPS output and in the next step disconnect the master clock for a long time (for example, 30 minutes), then the AM64x clock loses synchronization and thus the PPS output moves a lot away from the reference PPS pulse from the Grand Master Clock then after reconnecting the master clock to the AM64x the PPS output never synchronizes with the reference PPS as it did before disconnecting the master clock. It will be offset by a few milliseconds until the PPS output is turned off and on again using testptp.

Maybe there is a solution (fix / improvement in the kernel drivers) to avoid this behavior and make ptp4l always able to synchronize the PPS output ?
It's hard to detect from linux userspace if PPS is properly synchronized or not. Ptp4l shows that the master offset is a few nano seconds while the PPS output is offset by a few milliseconds.

Hi Michal, 

Michal Bajkacz said:

According to the note, even if I first start the ptp4l (wait for synchronization) and then turn on the PPS output and in the next step disconnect the master clock for a long time (for example, 30 minutes), then the AM64x clock loses synchronization and thus the PPS output moves a lot away from the reference PPS pulse from the Grand Master Clock then after reconnecting the master clock to the AM64x the PPS output never synchronizes with the reference PPS as it did before disconnecting the master clock. It will be offset by a few milliseconds until the PPS output is turned off and on again using testptp.

Thanks for explaining this particular case. Just as an additional question, when the master clock was disconnected, did ptp4l ever stop running? According to the note, when ptp4l ever were to stop running or if synchronization stops, the PPS signal would have to be stopped and started again after PTP synchronization resumes. 

Michal Bajkacz said:

I found information in the ti-linux-kernel repository that the fix for PPS synchronization was implemented in icssg-prueth around 11.11.2024 https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/?id=dc065076ee7768377d7c16af7d1b0767782d8c98 and is only executed when the PPS signal is turned on. Shouldn't a similar fix be implemented in icss-iep, so that it properly corrects the PPS signal offset during the corrections performed by ptp4l ?

I'll also check with the developer who made this initial patch to see if PPS can be synchronized while PTP is synchronizing without having to restart PPS.

Michal Bajkacz said:

I'm testing PROCESSOR-SDK-LINUX-RT-AM64X Version: 11.00.09.04 (debian-trixie) on custom board using TQMa64xxL.

Michal Bajkacz said:

According to the note, even if I first start the ptp4l (wait for synchronization) and then turn on the PPS output and in the next step disconnect the master clock for a long time (for example, 30 minutes), then the AM64x clock loses synchronization and thus the PPS output moves a lot away from the reference PPS pulse from the Grand Master Clock then after reconnecting the master clock to the AM64x the PPS output never synchronizes with the reference PPS as it did before disconnecting the master clock. It will be offset by a few milliseconds until the PPS output is turned off and on again using testptp.

Do you have access to an AM64x EVM? If so, are you able to replicate the case described on an AM64x EVM?

-Daolin

Hi Daolin,

Daolin Qiu said:

Thanks for explaining this particular case. Just as an additional question, when the master clock was disconnected, did ptp4l ever stop running? According to the note, when ptp4l ever were to stop running or if synchronization stops, the PPS signal would have to be stopped and started again after PTP synchronization resumes. 

When the master clock is disconnected the ptp4l still works and looks for the master clock so according to the note stopping and starting again the PPS signal should not be required.

Daolin Qiu said:

I'll also check with the developer who made this initial patch to see if PPS can be synchronized while PTP is synchronizing without having to restart PPS.

Thanks. That would be the best solution ;-).

Daolin Qiu said:

Do you have access to an AM64x EVM? If so, are you able to replicate the case described on an AM64x EVM?

Unfortunately, I do not have access to AM64x EVM, but on my custom board the PPS output wiring is the same as on EVM.

Hi Michal, 

Michal Bajkacz said:

When the master clock is disconnected the ptp4l still works and looks for the master clock so according to the note stopping and starting again the PPS signal should not be required.

Thanks for confirming, let me check internally on this particular case as well. However, based on the feedback from I got from the developer ("PTP synchronization should happen and reach a stable state before triggering PPS:) it appears it's not just the case of PTP stopping but if PTP synchronization is not stable and needs to be resynchronized, PPS should be restarted after PTP is resynchronized. Which appears to apply to this particular case where after disconnecting and reconnecting the cable, PTP synchronization needs to be corrected at which time the PPS does not update dynamically to reflect this.

Michal Bajkacz said:

I'll also check with the developer who made this initial patch to see if PPS can be synchronized while PTP is synchronizing without having to restart PPS.

Thanks. That would be the best solution ;-).

I got feedback and it appears that for PRU_ICSSG interfaces, typically PPS must be manually restarted after PTP finishes synchronizing. "When you do the PTP synchronization the underlying PTP clock gets corrected but the PPS signal which already has been generated won't take any effect. So you won't see the PPS signals getting synchronized despite the clocks being in sync"

However, I've asked if there is a chance to get PPS signal dynamically adjusted to the PTP clock correction without having to restart PPS. As a disclaimer, it might take some time get some feedback as the developer is working on some other high priority activities this week.

Michal Bajkacz said:

Do you have access to an AM64x EVM? If so, are you able to replicate the case described on an AM64x EVM?

Unfortunately, I do not have access to AM64x EVM, but on my custom board the PPS output wiring is the same as on EVM.

I plan to try and test it out on an AM64x EVM on my side but I likely won't get to it until Thursday. Please kindly ping this thread if you don't hear a response by Thursday.

-Daolin

Hi Michal, 

Daolin Qiu said:

Michal Bajkacz said:

Do you have access to an AM64x EVM? If so, are you able to replicate the case described on an AM64x EVM?

Unfortunately, I do not have access to AM64x EVM, but on my custom board the PPS output wiring is the same as on EVM.

I plan to try and test it out on an AM64x EVM on my side but I likely won't get to it until Thursday. Please kindly ping this thread if you don't hear a response by Thursday.

I tried to replicate the particular case you described by setting up the following:

Test topology:

• AM64x EVM (Grandmaster clock) <---> AM64x EVM (Follower clock)
• Both EVMs running Yocto built RT-Linux SDK 10.1 (this was what I had handy, it will take me some time to setup with the same Debian RT-Linux SDK 11.0) 
• Connected on eth2 (PRU-ICSSG) interfaces on both EVMs
• Measure the PPS signal on the J18 header on both EVMs with a Logic Analyzer (did not have an oscilloscope handy with me)

Test sequence:

1. Start synchronization with ptp4l -P -2 -H -i eth2 -f gPTP.cfg --step_threshold=1 -m -q -p /dev/ptp2 where the contents of the gPTP.cfg file is specified in the below code snippet.
2. Wait for synchronization to complete
3. Turn on PPS output with /usr/kernel-selftest/ptp/testptp -d /dev/ptp2 -P 1
4. Disconnect the cable and wait for 30 minutes
5. Reconnect the cable and observe the PPS offset

root@am64xx-evm:~# cat gPTP.cfg 
# 802.1AS example configuration containing those attributes which
# differ from the defaults. See the file, default.cfg, for the
# complete list of available options.
#
[global]
gmCapable 1
priority1 100 <---100 for Grandmaster, 240 for follower
priority2 248
logAnnounceInterval 0
logSyncInterval -3
syncReceiptTimeout 3
neighborPropDelayThresh 800
min_neighbor_prop_delay -20000000
assume_two_step 1
path_trace_enabled 1
follow_up_info 1
transportSpecific 0x1
ptp_dst_mac 01:80:C2:00:00:0E
network_transport L2
delay_mechanism P2P
ingressLatency 96
egressLatency 288 

However I was unable to see the milisecond level PPS offset you mentioned. I captured this similarly with a wait time of about 10 minutes after cable disconnection see my ptp4l logs and PPS capture from my logic analyzer.

Grandmaster EVM log:

root@am64xx-evm:~# /usr/kernel-selftest/ptp/testptp -d /dev/ptp2 -P 1
pps for system time request okay
root@am64xx-evm:~# [ 531.236334] icssg-prueth icssg1-eth eth2: Link is Down

root@am64xx-evm:~# cat gPTP.cfg 
# 802.1AS example configuration containing those attributes which
# differ from the defaults. See the file, default.cfg, for the
# complete list of available options.
#
[global]
gmCapable 1
priority1 100
priority2 248
logAnnounceInterval 0
logSyncInterval -3
syncReceiptTimeout 3
neighborPropDelayThresh 800
min_neighbor_prop_delay -20000000
assume_two_step 1
path_trace_enabled 1
follow_up_info 1
transportSpecific 0x1
ptp_dst_mac 01:80:C2:00:00:0E
network_transport L2
delay_mechanism P2P
ingressLatency 96
egressLatency 288
root@am64xx-evm:~# ls
access_dp83869.sh        eth0-no-rate-limit.pcap iperf3_s_output.txt
automate_linkup_linkdown.sh   gPTP.cfg         mpstat-idle.sh
configure_network_schneider.sh hsr_latest.sh      num_linkup_down_cycle.txt
cpu-load-awk.sh         hsr_nonoffload_setup.sh ptp4l-gm-output.txt
cpu-load-mpstat.sh       hsr_setup.sh       top_store.sh
dut3_eth1_tcpdump.pcap     hsr_setup_cut_thru.sh
dut3_eth2_tcpdump.pcap     hsr_setup_v2.sh
root@am64xx-evm:~# cat ptp4l-gm-output.txt | tail -10
ptp4l[419.899]: port 1 (eth2): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[419.899]: selected local clock 70ff76.fffe.1f3c45 as best master
ptp4l[419.899]: port 1 (eth2): assuming the grand master role
ptp4l[468.771]: port 1 (eth2): new foreign master 70ff76.fffe.1f3dc5-1
ptp4l[470.771]: selected best master clock 70ff76.fffe.1f3dc5
ptp4l[470.771]: port 1 (eth2): assuming the grand master role
ptp4l[531.239]: port 1 (eth2): link down
ptp4l[531.239]: port 1 (eth2): MASTER to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[531.248]: selected local clock 70ff76.fffe.1f3c45 as best master
ptp4l[531.248]: port 1 (eth2): assuming the grand master role
root@am64xx-evm:~# [ 872.228314] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow contf

root@am64xx-evm:~# cat ptp4l-gm-output.txt | tail -10
ptp4l[470.771]: port 1 (eth2): assuming the grand master role
ptp4l[531.239]: port 1 (eth2): link down
ptp4l[531.239]: port 1 (eth2): MASTER to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[531.248]: selected local clock 70ff76.fffe.1f3c45 as best master
ptp4l[531.248]: port 1 (eth2): assuming the grand master role
ptp4l[872.224]: port 1 (eth2): link up
ptp4l[872.248]: port 1 (eth2): FAULTY to LISTENING on INIT_COMPLETE
ptp4l[875.934]: port 1 (eth2): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[875.934]: port 1 (eth2): assuming the grand master role
ptp4l[875.993]: port 1 (eth2): new foreign master 70ff76.fffe.1f3dc5-1
root@am64xx-evm:~#

Follower EVM log:

root@am64xx-evm:~# /usr/kernel-selftest/ptp/testptp -d /dev/ptp2 -P 1
pps for system time request okay
root@am64xx-evm:~# [ 529.572816] icssg-prueth icssg1-eth eth2: Link is Down

root@am64xx-evm:~# cat ptp4l-follower.txt | tail -10
ptp4l[525.597]: rms  3 max  4 freq -3521 +/-  4 delay  67 +/-  0
ptp4l[526.598]: rms  5 max  8 freq -3523 +/-  7 delay  67 +/-  0
ptp4l[527.598]: rms  5 max  8 freq -3514 +/-  4 delay  67 +/-  0
ptp4l[528.599]: rms  8 max  11 freq -3514 +/- 11 delay  67 +/-  0
ptp4l[528.974]: port 1 (eth2): SLAVE to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[528.974]: selected local clock 70ff76.fffe.1f3dc5 as best master
ptp4l[528.974]: port 1 (eth2): assuming the grand master role
ptp4l[529.576]: port 1 (eth2): link down
ptp4l[529.576]: port 1 (eth2): MASTER to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[529.585]: port 1 (eth2): assuming the grand master role
root@am64xx-evm:~# [ 870.564949] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow contrf

root@am64xx-evm:~# cat ptp4l-follower.txt | tail -10
ptp4l[529.576]: port 1 (eth2): link down
ptp4l[529.576]: port 1 (eth2): MASTER to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[529.585]: port 1 (eth2): assuming the grand master role
ptp4l[870.565]: port 1 (eth2): link up
ptp4l[870.579]: port 1 (eth2): FAULTY to LISTENING on INIT_COMPLETE
ptp4l[873.694]: port 1 (eth2): new foreign master 70ff76.fffe.1f3c45-1
ptp4l[873.750]: port 1 (eth2): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[873.750]: port 1 (eth2): assuming the grand master role
ptp4l[875.694]: selected best master clock 70ff76.fffe.1f3c45
ptp4l[875.695]: port 1 (eth2): MASTER to UNCALIBRATED on RS_SLAVE
root@am64xx-evm:~# cat ptp4l-follower.txt | tail -10
ptp4l[873.750]: port 1 (eth2): LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[873.750]: port 1 (eth2): assuming the grand master role
ptp4l[875.694]: selected best master clock 70ff76.fffe.1f3c45
ptp4l[875.695]: port 1 (eth2): MASTER to UNCALIBRATED on RS_SLAVE
ptp4l[876.695]: port 1 (eth2): UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[877.570]: rms 13873 max 24294 freq +20890 +/- 6716 delay  68 +/-  1
ptp4l[878.571]: rms 4124 max 5773 freq +2355 +/- 3711 delay  70 +/-  0
ptp4l[879.571]: rms 5356 max 5880 freq -5337 +/- 950 delay  72 +/-  0
ptp4l[880.572]: rms 2888 max 4020 freq -6154 +/- 269 delay  72 +/-  0
ptp4l[881.572]: rms 753 max 1338 freq -4865 +/- 395 delay  72 +/-  0
root@am64xx-evm:~#  
root@am64xx-evm:~# cat ptp4l-follower.txt | tail -10                      
ptp4l[1748.517]: rms  8 max  14 freq -3423 +/- 10 delay  554 +/-  0
ptp4l[1749.517]: rms  15 max  22 freq -3436 +/- 20 delay  556 +/-  0
ptp4l[1750.518]: rms  14 max  19 freq -3429 +/- 19 delay  556 +/-  0
ptp4l[1751.518]: rms  11 max  20 freq -3423 +/- 14 delay  556 +/-  0
ptp4l[1752.519]: rms  11 max  25 freq -3422 +/- 15 delay  554 +/-  0
ptp4l[1753.520]: rms  13 max  28 freq -3414 +/- 17 delay  556 +/-  0
ptp4l[1754.520]: rms  12 max  17 freq -3418 +/- 16 delay  556 +/-  0
ptp4l[1755.521]: rms  10 max  14 freq -3422 +/- 14 delay  556 +/-  0
ptp4l[1756.521]: rms  13 max  23 freq -3420 +/- 18 delay  557 +/-  0
ptp4l[1757.522]: rms  15 max  23 freq -3419 +/- 21 delay  557 +/-  0

As part of our debug process at TI, we typically need to be able to replicate the issue observed on a TI EVM. Please let me know if there is something I missed in terms of trying to replicate your issue.

Also, just as a heads up, my response might be delayed next week as I will be attending some training sessions. 

-Daolin

Hi Daolin,

To accurately reproduce the case I described, you would have to use a “real” grand master clock to have a reference PPS ouput signal and reference source of time synchronization.

My tests shows that if I disconnect the masterclock (I used a Meinberg Lantime M600 or Hyperion400 for testing) from the PRU-ICSSG eth interface for a long time (it's hard to say exactly how long), then the PPS output will shift relative to the reference PPS output from the master clock. There is an offset threshold (I can't determine the exact value) and if it is exceeded then the ptp4l is not able to synchronize the pps output with the AM64x

Daolin Qiu said:

However, I've asked if there is a chance to get PPS signal dynamically adjusted to the PTP clock correction without having to restart PPS. As a disclaimer, it might take some time get some feedback as the developer is working on some other high priority activities this week.

That would be the best solution. Then the information provided by ptp4l regarding synchronization accuracy would be correlated with the PPS output.

The PPS output from my board using AM64x was to synchronize multiple slave modules, which must be precisely synchronized with the reference PPS signal. Under AM64x linux user space I am not able to confirm 100% that the PPS signal coming out of the board is synchronized with the PPS output from the master clock, which disqualifies its use in an industrial solution, it can only be used for testing purposes.

Hi Michal,

Michal Bajkacz said:

To accurately reproduce the case I described, you would have to use a “real” grand master clock to have a reference PPS ouput signal and reference source of time synchronization.

Can you explain more about what you mean by a "real" grand master clock? Does my setup using one AM64x EVM configured to be the grandmaster not count as a "real" grand master clock? 

Michal Bajkacz said:

That would be the best solution. Then the information provided by ptp4l regarding synchronization accuracy would be correlated with the PPS output.

Let me follow up with the developer about this. 

Michal Bajkacz said:

Under AM64x linux user space I am not able to confirm 100% that the PPS signal coming out of the board is synchronized with the PPS output from the master clock, which disqualifies its use in an industrial solution, it can only be used for testing purposes.

From my understanding, the most accurate way to measure PPS synchronization accuracy is to measure the actual PPS signal with an oscilloscope and compare the PPS signals of each device/modules in your network. I'm not aware of a specific way from linux user space to measure PPS synchronization accuracy across all connected devices. The ptp4l synchronization logs should in theory be a reflection of the PPS synchronization but the most accurate way to check would still be using an oscilloscope.

-Daolin