Other Parts Discussed in Thread: TMDS64EVM
Tool/software:
Dear TI team,
while experimenting with the QoS features of the icssg-prueth driver in the Linux SDK 11.00.09.04 we encountered some incomplete features or problems that haven't been considered regarding IEP PPS time synchronization.
Setup:
- SoC: AM64X SR2.0 HS-FS
- Model: Texas Instruments AM642 EVM
- Board: AM64-EVM rev C
- Image: PROCESSOR-SDK-LINUX-RT-AM64X Debian Trixie Version: 11.00.09.04
1st issue: IEP frequency offset at 333MHz
# install tools apt install ethtool linuxptp memtool # disable NTP timedatectl set-ntp false # synchronize CLOCK_REALTIME to the IEP phc2sys -s /dev/ptp2 -c CLOCK_REALTIME -O 0 -l 7 -m
Since both clocks are derived from the same quartz one would expect them to be able to synchronize to +-0ppb frequency offset (minor jitter because of the software cross-timestamping). With the IEP running at 333MHz we observe a frequency offset of +2,000ppb:
phc2sys[466.360]: CLOCK_REALTIME phc offset 1893344 s0 freq -0 delay 270 phc2sys[467.360]: CLOCK_REALTIME phc offset 1895344 s1 freq +1998 delay 270 phc2sys[468.361]: CLOCK_REALTIME phc offset 4 s2 freq +2002 delay 265 phc2sys[469.361]: CLOCK_REALTIME phc offset 4 s2 freq +2003 delay 270 phc2sys[470.362]: CLOCK_REALTIME phc offset 2 s2 freq +2002 delay 270 phc2sys[471.362]: CLOCK_REALTIME phc offset -6 s2 freq +1995 delay 275 phc2sys[472.362]: CLOCK_REALTIME phc offset 4 s2 freq +2003 delay 270 phc2sys[473.363]: CLOCK_REALTIME phc offset -2 s2 freq +1998 delay 270 phc2sys[474.364]: CLOCK_REALTIME phc offset 1 s2 freq +2000 delay 270
This can be explained by looking at the way the IEP is generating the internal 1ms cycle from the 333MHz clock with these register settings:
0x00 IEP_GLOBAL_CFG_REG: 0x00000331 == 3ns increment per 333MHz clock cycle (no compensation per default)
0x7c IEP_CMP0_REG1: 0x000f423d == 999,997ns
Clock cycle 0: 0 cycles, 0ns
Clock cycle 1: 0 cycles, 3ns
...
Clock cycle 333,332: 0 cycles, 999,996ns
Clock cycle 333,333: 0 cycles, 999,999ns <- since 999,999ns >= 999,997ns (CMP0) the counter is reset at the next clock edge
Clock cycle 333,334: 1 cycles, 0ns <- the IEP has "added" 1ns instead of 3ns -> we are losing 2ns per 1ms cycle -> 2,000ppb
Q: Is this the intended behavior for the IEP? 1ms in the IEP clock domain does not really equal 1ms in "real time", it's only 999,998 ns long.
Q: Shouldn't the firmware take care of compensating this by setting a 2,000ppb frequency offset in the IEP compensation registers by default?
Q: Must the IEP run at the same clock speed as the prueth-firmware (333MHz)? If the IEP could run at 200MHz or 250MHz this would not be a problem.
2nd issue: PPS output with taprio schedule
For a customer application we would like to use the PPS output of the IEP to synchronize one or more other system clocks (e.g. the MAIN_CPTS0) to the IEP time while the prueth port is running an arbitrary taprio schedule.
# route AM64X_DEV_PRU_ICSSG1-pr1_edc0_sync0_out to AM64X_DEV_CPTS0-cpts_hw1_push through the TIMESYNC_EVENT_INTROUTER0 memtool mw -l 0x00A40044 0x1001d # e.g. setup PRU for 4 TX queues ip link set down end2 ethtool -L end2 rx 1 tx 4 ip link set up end2 # setup a 1ms schedule for 4 traffic classes tc qdisc replace dev end2 parent root taprio \ num_tc 4 \ map 0 1 2 3 \ queues 1@0 1@1 1@2 1@3 \ base-time 0 \ sched-entry S 0x01 100000 \ sched-entry S 0xfe 900000 \ flags 0x2 # synchronize the MAIN_CPTS0 to the IEP clock ts2phc -c /dev/ptp1 -s /dev/ptp2 -m -l7 ... ts2phc[1263.944]: adding tstamp 1746708674.000000000 to clock /dev/ptp1 ts2phc[1263.944]: adding tstamp 1746708674.000000000 to clock /dev/ptp2 ts2phc[1263.944]: /dev/ptp1 offset 0 s2 freq +2000 ts2phc[1264.944]: adding tstamp 1746708675.000000000 to clock /dev/ptp1 ts2phc[1264.944]: adding tstamp 1746708675.000000000 to clock /dev/ptp2 ts2phc[1264.944]: /dev/ptp1 offset 0 s2 freq +2000 ts2phc[1265.944]: adding tstamp 1746708676.000000000 to clock /dev/ptp1 ts2phc[1265.944]: adding tstamp 1746708676.000000000 to clock /dev/ptp2 ts2phc[1265.944]: /dev/ptp1 offset 0 s2 freq +2000
So far so good (sans the 2,000ppb frequency offset from 1.).
Unfortunately, changing to a schedule that is != 1ms long breaks the time synchronization:
# setup a 1.5ms schedule for 4 traffic classes tc qdisc replace dev end2 parent root taprio \ num_tc 4 \ map 0 1 2 3 \ queues 1@0 1@1 1@2 1@3 \ base-time 0 \ sched-entry S 0x01 100000 \ sched-entry S 0xfe 900000 \ sched-entry S 0xf8 500000 \ flags 0x2 # Running synchronization in another shell: ts2phc[1470.944]: adding tstamp 1746708881.000000000 to clock /dev/ptp1 ts2phc[1470.945]: adding tstamp 1746708881.000000000 to clock /dev/ptp2 ts2phc[1470.945]: /dev/ptp1 offset 0 s2 freq +1999 ts2phc[1472.165]: adding tstamp 1746708882.220498677 to clock /dev/ptp1 ts2phc[1472.167]: adding tstamp 1746708882.000000000 to clock /dev/ptp2 ts2phc[1472.167]: /dev/ptp1 offset 220498677 s2 freq +488281 ts2phc[1473.665]: adding tstamp 1746708883.719767564 to clock /dev/ptp1 ts2phc[1473.665]: adding tstamp 1746708883.000000000 to clock /dev/ptp2 ts2phc[1473.665]: /dev/ptp1 offset 719767564 s2 freq +488281 ts2phc[1475.165]: adding tstamp 1746708885.219035142 to clock /dev/ptp1 ts2phc[1475.165]: adding tstamp 1746708884.000000000 to clock /dev/ptp2 ts2phc[1475.165]: /dev/ptp1 offset 1219035142 s2 freq +488281 ts2phc[1476.665]: adding tstamp 1746708886.718302720 to clock /dev/ptp1 ts2phc[1476.665]: adding tstamp 1746708885.000000000 to clock /dev/ptp2 ts2phc[1476.665]: /dev/ptp1 offset 1718302720 s2 freq +488281
Even restarting the ts2phc application won't fix it. Looking at the implementation in the icss_iep.c/icssg_pruet.c kernel driver this is no surprise:
static int prueth_perout_enable(void *clockops_data,
struct ptp_perout_request *req, int on,
u64 *cmp)
{
struct prueth_emac *emac = clockops_data;
u32 reduction_factor = 0, offset = 0;
struct timespec64 ts;
u64 current_cycle;
u64 start_offset;
u64 ns_period;
if (!on)
return 0;
/* Any firmware specific stuff for PPS/PEROUT handling */
ts.tv_sec = req->period.sec;
ts.tv_nsec = req->period.nsec;
ns_period = timespec64_to_ns(&ts);
/* f/w doesn't support period less than cycle time */
if (ns_period < IEP_DEFAULT_CYCLE_TIME_NS)
return -ENXIO;
reduction_factor = ns_period / IEP_DEFAULT_CYCLE_TIME_NS;
offset = ns_period % IEP_DEFAULT_CYCLE_TIME_NS;
/* f/w requires at least 1uS within a cycle so CMP
* can trigger after SYNC is enabled
*/
if (offset < 5 * NSEC_PER_USEC)
offset = 5 * NSEC_PER_USEC;
/* if offset is close to cycle time then we will miss
* the CMP event for last tick when IEP rolls over.
* In normal mode, IEP tick is 4ns.
* In slow compensation it could be 0ns or 8ns at
* every slow compensation cycle.
*/
if (offset > IEP_DEFAULT_CYCLE_TIME_NS - 8)
offset = IEP_DEFAULT_CYCLE_TIME_NS - 8;
/* we're in shadow mode so need to set upper 32-bits */
*cmp = (u64)offset << 32;
writel(reduction_factor, emac->prueth->shram.va +
TIMESYNC_FW_WC_SYNCOUT_REDUCTION_FACTOR_OFFSET);
current_cycle = icssg_read_time(emac->prueth->shram.va +
TIMESYNC_FW_WC_CYCLECOUNT_OFFSET);
/* Rounding of current_cycle count to next second */
start_offset = roundup(current_cycle, MSEC_PER_SEC);
hi_lo_writeq(start_offset, emac->prueth->shram.va +
TIMESYNC_FW_WC_SYNCOUT_START_TIME_CYCLECOUNT_OFFSET);
return 0;
}
The PPS output configuration is calculated for the IEP_DEFAULT_CYCLE_TIME_NS (1,000,000ns), not the actual cycle time (1.5ms) after changing the taprio schedule.
Q: Will this be fixed in an upcoming Kernel version?
A simple fix, where one would have to restart the time synchronization after the schedule is changed would be sufficient for us for now.
Q: Are there any plans to fix this in the PRU firmware? In the cycle where the schedule is changed the firmware would need to recalculate and set the reduction factor and offset etc. if the PPS signal is enabled.
3rd issue: PPS output with offset (CMP1)
If we recalculate the register values for e.g. a 1.5ms cycle and set them accordingly in the IEP/Firmware registers we hit another issue. If 1s is not divisible by the cycle time the reduction factor (number of full cycles per second) and/or the offset (time in the cycle to get to 1s) would need to change every PPS cycle.
First the working 1ms cycle time:
reduction_factor: 1,000
offset: 5,000ns (because of firmware restrictions)
At 1ms cycle time the first PPS be at "start cycle" time:
-> 0 * 1ms + 5,000ns = 5,000ns
The second PPS signal would be at (activating the sync output in the 1000th cycle):
-> 995,000ns (remaining from last sync cylce) + 999 * 1ms + 5,000ns = 1,000,005,000ns
The third PPS signal would be at:
-> 995,000ns (remaining from last sync cylce) + 999 * 1ms + 5,000ns = 2,000,005,000ns
Now for the 1.5ms cycle time without any register update:
reduction_factor: 666
offset: 1,000,000ns
The first PPS would be at "start cycle" time:
-> 0 * 1.5ms + 1,000,000ns = 1,000,000ns
The second PPS would be at:
-> 500,000ns (remaining from the last cycle) + 665 * 1.5ms + 1,000,000ns = 1,000,000,000ns
The third PPS would be at:
-> 500,000ns (remaining from the last cycle) + 665 * 1.5ms + 1,000,000ns = 1.999.000.000ns
The fourth PPS would be at:
-> 500,000ns (remaining from the last cycle) + 665 * 1.5ms + 1,000,000ns = 2.998.000.000ns
After the second PPS it will be short 1ms every time.
This would have to be fixed by updating the offset and reduction_factor after every PPS output:
reduction_factor: 666
offset: 5,000ns
The first PPS would be at "start cycle" time:
-> 0 * 1.5ms + 1,000,000ns = 5,000ns
reduction_factor: 666
offset: 1,005,000ns
The second PPS would be at:
-> 1,495,000ns (remaining from the last cycle) + 665 * 1.5ms + 1,005,000ns = 1,000,005,000ns
reduction_factor: 667
offset: 5,000ns
The third PPS would be at:
-> 495,000ns (remaining from the last cycle) + 666 * 1.5ms + 5,000ns = 2,000,005,000ns
Implementing this outside of the PRU firmware would require an interrupt on the PPS in the Kernel driver and then racing against the reduction factor up-counter to set the reduction factor and offset before the next PPS output, which is why we believe that this needs to be fixed within the firmware.
Q: Shouldn't the firmware take care of these calculations if the PPS interval is not divisible by the cycle time? Are there any fixes planned?
Best Regards,
Dominic
