TMDX654IDKEVM: PRU-ICSSG PPS output signal

Lukas Libal1 said:

Is the PPS signal in based on the correct PTP device?

Based on your printouts eth1 is /dev/ptp3 and eth3 is /dev/ptp4.

Lukas Libal1 said:

Is the PPS signal starting each 1s absolute time of PTP clock?

I recommend using testptp, but looks like directly your file write works as well. testptp is already in the filesystem or you can compile yourself. Some examples below.

#turn 1 PPS on
/usr/bin/kselftests/ptp/testptp -d /dev/ptp4 -P 1
#turn 1 PPS off
/usr/bin/kselftests/ptp/testptp -d /dev/ptp4 -P 0
#turn 2 PPS on
/usr/bin/kselftests/ptp/testptp -d /dev/ptp4 -p 500000000
#turn 2 PPS off
/usr/bin/kselftests/ptp/testptp -d /dev/ptp4 -p 0

Lukas Libal1 said:

Check the PPS signal for synchronized clock of PTP device between two IDK board.

I'm working on getting access to a two board AM65x IDK setup. I have not run the exact sequence you are running, especially your step 6 catches my eye. Can you try 

/usr/bin/kselftests/ptp/testptp -d /dev/ptp3 -P 1

instead. It might be something else but at least worth checking.

  Pekka

Hello,

I have tested PPS control with proposed tool /usr/bin/kselftests/ptp/testptp on one board.

The "-P 1" option enables the output, but the start of the period is still shifted, even if the clock is synchronized by phc2sys.

The "-p 500000000" option enables output sometimes immediately, sometimes it take few seconds sometimes not after 1 minute and with not synchronized signal. So this is not working.


I have not yet tested PPS control with proposed tool on two synchronized boards like in original post.

Regards,
Lukas Libal

Lukas Libal1 said:

The "-p 500000000" option enables output sometimes immediately, sometimes it take few seconds sometimes not after 1 minute and with not synchronized signal. So this is not working.

testptp is waiting for the pps signal error to converge before the LED toggles. It is intended as an open source example you can base your application on.

Lukas Libal1 said:

start of the period is still shifted, even if the clock is synchronized by phc2sys

The rate (1 PPS) seems accurate / synchronized. I'm checking on the options and parameters on this why there is an offset, to me it is not yet clear is this just a parameter or is there just a random offset. 

I'm also setting up the 2x IDK setup today.

  Pekka

Hello Pekka,

I observed visible offset between PPS signals for two synchronized boards with controlling PPS via testptp also. There should not be a difference between general enabling PPS via filesystem or via syscall (testptp).

Pekka Varis said:

The rate (1 PPS) seems accurate / synchronized. I'm checking on the options and parameters on this why there is an offset, to me it is not yet clear is this just a parameter or is there just a random offset. 

Do you have some update regarding observed offset between signals. The correct PPS signal is necessary for synchronization accuracy measurement and related ingress and egress latency calibration. Therefore, additional offset is not acceptable for its purpose.

Regards,
Lukas

I now have two AM6548 IDK set up, with both running the release candidate for 8.6 SDK that should be released this week. I also have a further AM64x board in the LAN using Ethernet based on CPSW3G. So trying a few permutations across the three boards (also just the 2x IDK). Generally looks like ptp4l works but the pps synchronization on the IDKs to this does not. So all of the above is repeatable and after running a little longer runs the pps blinking of the LEDs (LD2 or LD5) is not synchronized in offset or rate to the clock from ptp. Initially it can look it is just an offset problem but with two board setup the blinking does drift so it is rate as well.

I believe the issue is in the clock synchronization part, so the phc2sys (or ts2phc) use, and what pin is connected to the LED. I suspect the LED blinking is coming from the IEP timer in each ICSSG which is used for the timestamping and is just monotonically incrementing, not synchronized or tuned.

  Pekka

Can you check if you are seeing this error as well

root@am65xx-evm:~# systemctl status sync-clocks.service
* sync-clocks.service - Synchronize System and HW clocks
     Loaded: loaded (/etc/systemd/system/sync-clocks.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Sat 2023-03-25 14:50:13 UTC; 1 weeks 0 days ago
    Process: 841 ExecStart=/sbin/hwclock --systohc (code=exited, status=1/FAILURE)
   Main PID: 841 (code=exited, status=1/FAILURE)

Mar 25 14:50:13 am65xx-evm systemd[1]: Starting Synchronize System and HW clocks...
Mar 25 14:50:13 am65xx-evm hwclock[841]: hwclock: Cannot access the Hardware Clock via any known method.
Mar 25 14:50:13 am65xx-evm hwclock[841]: hwclock: Use the --verbose option to see the details of our sea>
Mar 25 14:50:13 am65xx-evm systemd[1]: sync-clocks.service: Main process exited, code=exited, status=1/F>
Mar 25 14:50:13 am65xx-evm systemd[1]: sync-clocks.service: Failed with result 'exit-code'.
Mar 25 14:50:13 am65xx-evm systemd[1]: Failed to start Synchronize System and HW clocks.
root@am65xx-evm:~# 

  Pekka

I don't think it is related after all, but anyway something that caught my eye compared to some other AM6x SDKs.

  Pekka

Hello,

I have the same error for sync-clock.service for LINUX SDKs 08.02.00.01 and 08.06.00.47. I did not test other versions.

#08.06.00.47
root@am65xx-evm:~# systemctl status sync-clocks.service
* sync-clocks.service - Synchronize System and HW clocks
     Loaded: loaded (/etc/systemd/system/sync-clocks.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Sat 2023-03-25 14:50:13 UTC; 5min ago
    Process: 831 ExecStart=/sbin/hwclock --systohc (code=exited, status=1/FAILURE)
   Main PID: 831 (code=exited, status=1/FAILURE)

Mar 25 14:50:13 am65xx-evm systemd[1]: Starting Synchronize System and HW clocks...
Mar 25 14:50:13 am65xx-evm hwclock[831]: hwclock: Cannot access the Hardware Clock via any known method.
Mar 25 14:50:13 am65xx-evm hwclock[831]: hwclock: Use the --verbose option to see the details of our search for an access method.
Mar 25 14:50:13 am65xx-evm systemd[1]: sync-clocks.service: Main process exited, code=exited, status=1/FAILURE
Mar 25 14:50:13 am65xx-evm systemd[1]: sync-clocks.service: Failed with result 'exit-code'.
Mar 25 14:50:13 am65xx-evm systemd[1]: Failed to start Synchronize System and HW clocks.
root@am65xx-evm:~# /sbin/hwclock --verbose
hwclock from util-linux 2.35.1
System Time: 1679756133.436164
Trying to open: /dev/rtc0
Trying to open: /dev/rtc
Trying to open: /dev/misc/rtc
No usable clock interface found.
hwclock: Cannot access the Hardware Clock via any known method.


#08.02.00.01
root@am65xx-evm:~# systemctl status sync-clocks.service
* sync-clocks.service - Synchronize System and HW clocks
     Loaded: loaded (/etc/systemd/system/sync-clocks.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Thu 2022-03-24 05:18:37 UTC; 35s ago
    Process: 1038 ExecStart=/sbin/hwclock --systohc (code=exited, status=1/FAILURE)
   Main PID: 1038 (code=exited, status=1/FAILURE)

Mar 24 05:18:37 am65xx-evm systemd[1]: Starting Synchronize System and HW clocks...
Mar 24 05:18:37 am65xx-evm hwclock[1038]: hwclock: Cannot access the Hardware Clock via any known method.
Mar 24 05:18:37 am65xx-evm hwclock[1038]: hwclock: Use the --verbose option to see the details of our search for an access method.
Mar 24 05:18:37 am65xx-evm systemd[1]: sync-clocks.service: Main process exited, code=exited, status=1/FAILURE
Mar 24 05:18:37 am65xx-evm systemd[1]: sync-clocks.service: Failed with result 'exit-code'.
Mar 24 05:18:37 am65xx-evm systemd[1]: Failed to start Synchronize System and HW clocks.
root@am65xx-evm:~# /sbin/hwclock --verbose
hwclock from util-linux 2.35.1
System Time: 1648099177.508796
Trying to open: /dev/rtc0
Trying to open: /dev/rtc
Trying to open: /dev/misc/rtc
No usable clock interface found.
hwclock: Cannot access the Hardware Clock via any known method.

Regards,
Lukas

(I am posting here so that this thread is not auto closed. We are still interested in the fix.)

A SW bug has been filed, I don't have a fix schedule yet.

  Pekka

The bug id is LCPD-34790 (shows up in the release notes), and current schedule is for 9.1. Once the fix proceeds and assuming it is simple enough I will post the change here.

  Pekka

I don't see  LCPD-34790 in release notes:

software-dl.ti.com/.../Release_Specific_Release_Notes.html

LCPD-34790 is the issue in 8.6 that has been filed and is planned to be fixed in 9.1. The release notes list issues fixed, they are not updated with new issues found.

  Pekka

This was pushed out to 9.2 release, the fix is not in 9.1.

  Pekka

Hi 

Here is the update from the developer post SDK9.1 testing 

We tried reproducing this bug.but unable to do on the latest ti-rt-linux-kernel
Once both interfaces (eth1 and eth3) are synchronized using phc2sys. And PPS signal is generated. We see the LEDs blink in sync and  don't see any variation. Test was run for 30 mins and didn't see any deviation.

How long do you run the test to see the failure. 

Note: There was earlier sync related issue / time jumps in phc2sys and ptp4l. But they were fixed as part of SDK 9.1. Post SDK 9.1 there is no issue in syncing two ICSSG PTP clocks. So I think this issue will also not be seen post SDK 9.1.
From the e2e seems like the customer has tested this on SDK 8.2 https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1198136/tmdx654idkevm-pru-icssg-pps-output-signal/#:~:text=HW%3A%20TMDX654IDKEVM%20PROC062A-,SW%3A%20PROCESSOR%2DSDK%2DLINUX%2DRT%2DAM65X%2008.02.00.01,-Check%20the%20PPS

Can you please try to reproduce the issue on latest SDK / cicd image for AM65x.

Logs::

Connection:
==========

Connected eth1 and eth3 of am654x-idk via ethernet cable.

am65xx-evm:~# difconfig m
%%eth0%%46000000.ethernet down using ptp clock 1 NoIP,40:2e:71:3f:3e:44 RxPkt=0 RxByte=0 TxPkt=0 TxByte=0
%%eth1%%icssg0-eth up using ptp clock 2 NoIP,70:ff:76:1d:63:28 RxPkt=18 RxByte=3060 TxPkt=63 TxByte=7412
%%eth2%%icssg0-eth down using ptp clock 2 NoIP,70:ff:76:1d:63:29 RxPkt=0 RxByte=0 TxPkt=0 TxByte=0
%%eth3%%icssg1-eth up using ptp clock 3 NoIP,70:ff:76:1d:63:2a RxPkt=25 RxByte=3294 TxPkt=41 TxByte=5102
%%eth4%%icssg1-eth down using ptp clock 3 NoIP,70:ff:76:1d:63:2b RxPkt=0 RxByte=0 TxPkt=0 TxByte=0
%%eth5%%icssg2-eth up using ptp clock 4 192.168.7.143,70:ff:76:1d:ea:f8 RxPkt=394 RxByte=475015 TxPkt=323 TxByte=31768
%%eth6%%icssg2-eth up using ptp clock 4 192.168.7.146,70:ff:76:1d:ea:f9 RxPkt=21 RxByte=4676 TxPkt=69 TxByte=10162
am65xx-evm:~# [  270.555686] icssg-prueth icssg0-eth eth1: Link is Down
[  270.747941] icssg-prueth icssg1-eth eth3: Link is Down
[  275.676436] icssg-prueth icssg0-eth eth1: Link is Up - 1Gbps/Full - flow control off
[  275.868471] icssg-prueth icssg1-eth eth3: Link is Up - 1Gbps/Full - flow control off

am65xx-evm:~# 
am65xx-evm:~# 

Kernel and Firmware Used:
========================

am65xx-evm:~# uname -a
Linux am65xx-evm 6.1.82-01672-ge44f83d2aa43 #28 SMP PREEMPT Tue Mar 26 16:56:02 IST 2024 aarch64 aarch64 aarch64 GNU/Linux
am65xx-evm:~# fwVersion

String dump of section '.version_string':
  [     0]  Version REL.PRU-ICSS-ETHERNET-SWITCH_02.02.13.00

Capablities of eth1 and eth3:
============================

am65xx-evm:~# ethtool -T eth1
Time stamping parameters for eth1:
Capabilities:
        hardware-transmit
        software-transmit
        hardware-receive
        software-receive
        software-system-clock
        hardware-raw-clock
PTP Hardware Clock: 2
Hardware Transmit Timestamp Modes:
        off
        on
Hardware Receive Filter Modes:
        none
        all
am65xx-evm:~# ethtool -T eth3
Time stamping parameters for eth3:
Capabilities:
        hardware-transmit
        software-transmit
        hardware-receive
        software-receive
        software-system-clock
        hardware-raw-clock
PTP Hardware Clock: 3
Hardware Transmit Timestamp Modes:
        off
        on
Hardware Receive Filter Modes:
        none
        all

Syncing eth1 with eth3 using phc2sys:
====================================

am65xx-evm:~# phc2sys -c eth1 -s eth3 -O 0 -m
phc2sys[309.821]: eth1 phc offset       436 s0 freq      +0 delay   7136
phc2sys[310.822]: eth1 phc offset       410 s2 freq     -26 delay   7132
phc2sys[311.822]: eth1 phc offset       428 s2 freq    +402 delay   7128
phc2sys[312.823]: eth1 phc offset        38 s2 freq    +140 delay   7140
phc2sys[313.823]: eth1 phc offset       -96 s2 freq     +18 delay   7160
phc2sys[314.824]: eth1 phc offset      -128 s2 freq     -43 delay   7120
phc2sys[315.824]: eth1 phc offset       -64 s2 freq     -17 delay   7136
phc2sys[316.824]: eth1 phc offset       -68 s2 freq     -41 delay   7192
phc2sys[317.825]: eth1 phc offset       -10 s2 freq      -3 delay   7140
phc2sys[318.825]: eth1 phc offset        -4 s2 freq      +0 delay   7144
phc2sys[319.825]: eth1 phc offset        -8 s2 freq      -5 delay   7144
phc2sys[320.826]: eth1 phc offset       -22 s2 freq     -22 delay   7132
phc2sys[321.826]: eth1 phc offset        22 s2 freq     +16 delay   7124
phc2sys[322.827]: eth1 phc offset       -22 s2 freq     -22 delay   7148
phc2sys[323.827]: eth1 phc offset         0 s2 freq      -6 delay   7160
phc2sys[324.827]: eth1 phc offset        14 s2 freq      +8 delay   7140
phc2sys[325.828]: eth1 phc offset        -6 s2 freq      -8 delay   7204
phc2sys[326.828]: eth1 phc offset         0 s2 freq      -4 delay   7168
phc2sys[327.828]: eth1 phc offset        14 s2 freq     +10 delay   7140
phc2sys[328.829]: eth1 phc offset         0 s2 freq      +0 delay   7152
phc2sys[329.829]: eth1 phc offset         4 s2 freq      +4 delay   7208
phc2sys[330.829]: eth1 phc offset        -4 s2 freq      -2 delay   7168
phc2sys[331.830]: eth1 phc offset         6 s2 freq      +6 delay   7132
phc2sys[332.830]: eth1 phc offset        -8 s2 freq      -6 delay   7112
phc2sys[333.830]: eth1 phc offset        10 s2 freq     +10 delay   7156
phc2sys[334.831]: eth1 phc offset       -20 s2 freq     -17 delay   7120
phc2sys[335.831]: eth1 phc offset        22 s2 freq     +19 delay   7132
phc2sys[336.832]: eth1 phc offset        -4 s2 freq      -1 delay   7152
phc2sys[337.832]: eth1 phc offset       -10 s2 freq      -8 delay   7124
phc2sys[338.832]: eth1 phc offset         2 s2 freq      +1 delay   7140
phc2sys[339.833]: eth1 phc offset        -8 s2 freq      -8 delay   7184
^Cphc2sys[340.426]: eth1 phc offset       -10 s2 freq     -13 delay   7164

Enabling PPS on both eth1 and eth3:
==================================

echo 1 > /sys/class/ptp/ptp2/pps_enable; echo 1 > /sys/class/ptp/ptp3/pps_enable;

Observation / Result:
====================

The LEDs LD3 and LD5 on the extension board start blinking together (No time difference visible in the LEDs by eyes)

Kept the PPS signal running for 30 mins, didn't notice any significant / visible variation between the LEDs.

Conclusion:
========== 

Not able to reproduce this issue on AM654x-IDK

Hello Mukul,

we have done test with PROCESSOR-SDK-LINUX-RT-AM65X 09.01.00.01 - tisdk-default-image-am65xx-evm.wic.xz

The PPS signal does not work as expected. The PPS signal should depends only on clock, but not at a time when PPS is enabled. Since you enable both PPS signals "at a one time" the issue is covered up. And the difference is not visible by eyes.

Please consider that check by eyes and on one IDK is only rough fast initial check. Final verification shall be done with osciloskope and between two IDK boards.

The issue is reproducible for SDK 09.01.00.01

Best regards,
Lukas

Lukas Libal1 said:

Since you enable both PPS signals "at a one time" the issue is covered up.

Can you give an example of how to you want it done so it would not be covered up? Do you mean test needs to be done using 2 boards, not the single board and 2 Ethernet devices above?

Lukas Libal1 said:

Final verification shall be done with osciloskope and between two IDK boards.

I agree PTP PPS verification in a LAN should be done with a oscilloscope or similar tester. But we don't currently verify time synchronization PPS accross a LAN in a software release test. 

  Pekka

Hello Pekka,

Pekka Varis said:

Can you give an example of how to you want it done so it would not be covered up? Do you mean test needs to be done using 2 boards, not the single board and 2 Ethernet devices above?

The start time of the PPS currently depends on the time when is enabled. When you enable the signal by two commands immediately one after one. The start of PPS is in similar time and looks correct. The dependency on time of enable is covered up. Enable PPS for each PTP device with some delay. One board should be enough.

Pekka Varis said:

I agree PTP PPS verification in a LAN should be done with a oscilloscope or similar tester. But we don't currently verify time synchronization PPS accross a LAN in a software release test. 

Make sense is not necessary to do PPS verification with two boards. My intention behind is verify it according to real use case and verify it with whole synchronization chain on independent hardware.

Regards,
Lukas