TMDX654IDKEVM: HW timestamps - PortRx, PortTx delay values

Hello Lukas,

Reassigning your thread to another teammate as discussed at https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1123101/tmdx654idkevm-wrong-hw-timestamps---bad-precision-with-10mbit-and-100mbit

Regards,

Nick

Lukas,

The latency values are Ethernet PHY datasheet values. The IDK uses DP83867 PHY (see IDK user guide) and by default in RGMII mode (the PHY and EVM also supports MII but you'll need to configure). Normally the datasheet https://www.ti.com/lit/ds/symlink/dp83867ir.pdf would be the best but in this case the app report https://www.ti.com/lit/an/snla240a/snla240a.pdf has a better table for this (datasheet https://www.ti.com/lit/ds/symlink/dp83867ir.pdf only has 1G number). 

Do note there is some variation in this (as in many analog components), see https://e2e.ti.com/support/interface-group/interface/f/interface-forum/1105605/dp83867ir-phy-latency-and-jitter-in-rgmii-mode-required for a discussion on that.

This might also explain the jitter you are observing for HW timestamps in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1123101/tmdx654idkevm-wrong-hw-timestamps---bad-precision-with-10mbit-and-100mbit .

  Pekka

Hello Pekka,

I did delay measurement on IDK PG2.0 with image from PROCESSOR-SDK-LINUX-RT-AM65X 08.02.00.01. In setup is one board with loopback between PRU interfaces with 10m cable. PHY is by default in RGMII and links peed is 1Gbit. HW timestamps are configured for used interfaces for Rx and Tx.

Delay measurement is based on sending frame both directions and obtaining all timestamps.

After applying the portDelays to timestamps I calculated delay ~130ns. This value is 4 times bigger than ~33ns, which is expected value for 10m cable (for ideal case).

1. Obtained timestamps corrected by provided portDelays are not satisfying for delay measurement. Are in PRU done some additional corrections? Are the portDelay values valid for PHYs on development board?

2. I still missing portDelay values for 10Mbit.

Lukas

Lukas Libal1 said:

1. Obtained timestamps corrected by provided portDelays are not satisfying for delay measurement. Are in PRU done some additional corrections? Are the portDelay values valid for PHYs on development board?

Which values did you use for the DP83867IR PHY (guessing 88ns TX and 288ns RX from https://www.ti.com/lit/an/snla240a/snla240a.pdf ) that resulted in the ~77ns error?  Can you try other cable lengths to isolate is the ~77ns constant? Do you observe the +-8ns jitter the PHY team mentions in https://e2e.ti.com/support/interface-group/interface/f/interface-forum/1105605/dp83867ir-phy-latency-and-jitter-in-rgmii-mode-required . I believe this is both rx and tx, so up to double that would be observed in a loopback.

The path, "MAC latency", from the RGMII interface to ICSS timestamping with IEP clock won't be infinitely fast, I suspect there is a fixed delay in the 10-20ns range at 1G loopback in addition to the PHY datasheet values. I observe ~40ns additional delay in ptp4l tests (Sitara AM6x EVM on both ends).

Lukas Libal1 said:

2. I still missing portDelay values for 10Mbit.

Unfortunately the DP83867IR does not document the 10M delay values, as stated in https://e2e.ti.com/support/interface-group/interface/f/interface-forum/1105605/dp83867ir-phy-latency-and-jitter-in-rgmii-mode-required . The assumption is that if you are building a 10M system that cares about latency un the nanoseconds you should use MII, the evaluation board is designed primarily for 1G standard Ethernet (RGMII) and 100M EtherCAT (MII). If you are using the TMDX654IDKEVM as a 10M tester I'd suggest you measure the delay you observe and use that.

  Pekka

Hello,

Pekka Varis said:

Which values did you use for the DP83867IR PHY (guessing 88ns TX and 288ns RX from https://www.ti.com/lit/an/snla240a/snla240a.pdf ) that resulted in the ~77ns error?  Can you try other cable lengths to isolate is the ~77ns constant? Do you observe the +-8ns jitter the PHY team mentions in https://e2e.ti.com/support/interface-group/interface/f/interface-forum/1105605/dp83867ir-phy-latency-and-jitter-in-rgmii-mode-required . I believe this is both rx and tx, so up to double that would be observed in a loopback.

The path, "MAC latency", from the RGMII interface to ICSS timestamping with IEP clock won't be infinitely fast, I suspect there is a fixed delay in the 10-20ns range at 1G loopback in addition to the PHY datasheet values. I observe ~40ns additional delay in ptp4l tests (Sitara AM6x EVM on both ends).

Yes, for calculation I use 88ns TX and 288ns RX. Based on measure for different cables I isolate additional delay 90ns. Few measures shows that the jitter is in mentioned range.

T1 tx timestamp obtained on interface1
T2 rx timestamp obtained on interface2
T3 tx timestamp obtained on interface2
T4 rx timestamp obtained on interface1
Txd portDelay compensated timestamp

delay = ((T4d - T1d) - (T3d - T2d)) / 2

According to your post and Figure 2. in www.ti.com/.../snla240a.pdf, the additional delay is MAC Latency. So my question is to these MAC Transmit and Receive Latency, are known exact values for PRUs? 

Lukas

So it seems like each 10m of cable adds just over 44ns (datapoints are 42ns, 46ns, 44ns), which I guess comes out to about 0.7 times the speed of light in vacuum. Which I think is normal for an Ethernet cable (I see 0.6c to 0.9c when searching). 

For the MAC latency I agree we should document it, but in practice we'd just measure it like you just did. 90ns for the sum of RX and TX latencies, which is in the ballpark of my 40ns observation (might have been only RX).

There will also be at least the +-8ns jitter (RGMII 1G).

  Pekka

Hello Pekka,

Pekka Varis said:

So it seems like each 10m of cable adds just over 44ns (datapoints are 42ns, 46ns, 44ns), which I guess comes out to about 0.7 times the speed of light in vacuum. Which I think is normal for an Ethernet cable (I see 0.6c to 0.9c when searching). 

I agree the calculated values are fine with respect to the measurement conditions and the jitter.

Pekka Varis said:

For the MAC latency I agree we should document it, but in practice we'd just measure it like you just did. 90ns for the sum of RX and TX latencies, which is in the ballpark of my 40ns observation (might have been only RX).

I have question relate to MAC latency. We isolated the sum of RX and TX MAC delay. How is the delay 90ns distributed between RX and RX?

Thank you,
Lukas

We don't have the RX and TX MAC latency documented or part of the explicit tests for AM65x. First step would be to measure the sum (you measured at 90ns on the AM65x IDK) and include it in the SDK documentation. I would think the RX MAC latency is higher than TX. The linuxptp discussion at https://sourceforge.net/p/linuxptp/mailman/linuxptp-users/thread/20190702031043.d32sm6kjufchnoz5@localhost/ points to about 1:3 relationship which would mean something like 23ns TX and 70ns RX.

If you truly need to know the value I'd say you need to empirically measure it for the board and configuration you are using. 

  Pekka

unlocking the thread...

Hello,

There were done some measurements and identified the following PortDelay Values for the Maxwell AM654x PG1/PG2/PG2.1 chips:

• PortDelay values for 100 MBit and RGMII Interface:
  • PortTxDelay (ns): 428
  • PortRxDelay (ns): 510

• PortDelay values for 1 GBit and RGMII Interface:
  • PortTxDelay (ns): 112
  • PortRxDelay (ns): 352

We have some questions that require clarification from TI:

• What is the setting for the DP83867-PHY (DP83867IRRGZ) related to latency?
• Are timestamps done in PRU HW only? Can the timestamping be influenced by the Firmware?

Basically we need to know if those values should also be valid for PRU FW which comes from TI RT Linux SDK.

Paul

Paul Bacila said:

What is the setting for the DP83867-PHY (DP83867IRRGZ) related to latency?

Can you clarify this?

Paul Bacila said:

Are timestamps done in PRU HW only? Can the timestamping be influenced by the Firmware?

Timestamp is done by HW, so timestamp accuracy should not be effected by firmware version. Latency in general can be effected by firmware version.

Hello,

Pekka Varis said:

Paul Bacila said:

What is the setting for the DP83867-PHY (DP83867IRRGZ) related to latency?

Can you clarify this?

The question is about configuration of PHY related to Chapter 6 in https://www.ti.com/lit/an/snla240a/snla240a.pdf
 -> We can read exact configuration from the PHY in runtime.

The main missing information is MAC latency and how to isolate Rx and Tx parts. You refer to empirical measurement.

Pekka Varis said:

We don't have the RX and TX MAC latency documented or part of the explicit tests for AM65x. First step would be to measure the sum (you measured at 90ns on the AM65x IDK) and include it in the SDK documentation. I would think the RX MAC latency is higher than TX. The linuxptp discussion at https://sourceforge.net/p/linuxptp/mailman/linuxptp-users/thread/20190702031043.d32sm6kjufchnoz5@localhost/ points to about 1:3 relationship which would mean something like 23ns TX and 70ns RX.

If you truly need to know the value I'd say you need to empirically measure it for the board and configuration you are using. 

Do you have process how you measure the Rx a Tx parts of MAC delay?

Please clarify if exist recommended measurement how to measure exact values MAC Rx and Rx delays of PRU-ICSSG.

Regards
Lukas

Some PHY devices like DP83867 document the PHY delay, broken up in rx and tx. Looks like 88ns tx and 288ns rx for 1G operation of DP83867. As that is the case I'd use the values the PHY documents, only verify by measuring. One possibility would be to measure using cables of two different lengths and thus known propagation delay. Measure each cable and subtract out the known PHY delay and cable propagation delay to get what is left. Which I understood from this thread you have already done.

If the PHY does not document the delays I think splitting exactly out to tx and rx will be hard but I'd start with the 1:3 (tx:rx) ratio estimate.

If you need to you can read PHY registers with the Linux command "phytool", which I think you are already doing.

Paul Bacila said:

There were done some measurements and identified the following PortDelay Values for the Maxwell AM654x PG1/PG2/PG2.1 chips:

• PortDelay values for 100 MBit and RGMII Interface:
  • PortTxDelay (ns): 428
  • PortRxDelay (ns): 510

• PortDelay values for 1 GBit and RGMII Interface:
  • PortTxDelay (ns): 112
  • PortRxDelay (ns): 352

Which I think is the result of measurement I'm suggesting and you have already done.  

  Pekka

Hello,

I open new related task about PPS signal clarification, which is necessary for ingress and egress latency calibration.
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1198136/tmdx654idkevm-pru-icssg-pps-output-signal

After that we can continue with Calibration of ingress and egress latencies according to IEEE1588.

Lukas