This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMDX654IDKEVM: Clarification of frame transmission and receive paths with regard to latency and jitter on industrial Ethernet ports of the board

Part Number: TMDX654IDKEVM

Hello,

as I need to know the Egress latency and Ingress latency of PRU-ICSSG interfaces and as I was pointed to measure those by myself, I would like to clarify the theoretical operation here, mostly as basis for preparing meaningful measurement process.  I am mostly interested in jitter causes and mitigations, as our product is required to have 8 ns timestamp precision at MDI plane.

I have some assumptions and few questions that I would like to clarify.  For the assumptions, I would like to ask for confirmation whether those are correct.

Note:  Jitter less than 4 ns (as a sum over whole Ingress latency or whole Egress latency) can be considered zero jitter, as it is under the 4 ns resolution of the timestamping clock.

Assumption 1:
Both transmit and receive timestamping is implemented in PRU according to IEEE 802.3 Clause 90.
(I. e. timestamping plane is enter/exit generic reconciliation sublayer of the MAC and message timestamping point is SFD.)

Assumption 2:
Timestamping implementation in PRU uses free running clock counter that is independent on TX_CLK and RX_CLK signals of the xMII interface.

Assumption 3:
When crossing clock domains in PHY, resulting jitter can never be greater than one period of the target clock.

Assumption 4:
Processing in PHY, both sending and receiving, on 10Base-T, 100Base-TX and 1000Base-T (all Full Duplex) have zero jitter (excluding crossing of the clock domains).

Assumption 5:
PCB wiring and magnetics have zero jitter.

Assumption 6:
When transmitting MAC and transmitting PHY share clock signal source, crossing the MAC-PHY clock domain interface has constant latency (zero jitter).

Assumption 7:
When RX_CLK is driven by the line clock (from received data), there is zero added jitter in the PHY processing (thanks to no clock-domain crossing).

Question 1:
Can the PHY, during transmission driven by RGMII (driven by GTX_CLK from PRU) use this clock source for transmission on the medium for 10Base-T Full Duplex, resp. 100Base-TX Full Duplex operation?
(This seems theoretically very well possible at least for 10Base-T, as line clock is not continuous there and receiver clock is always synchronized at the start of the frame, during preamble.)

Question 2:
Can the PHY, during receiving, use link clock to drive the (RG)MII RX_CLK (and the whole decoding process)?  If not at all times, what are the conditions when this is/can be done?

Here I add an illustration of the whole path with Egress latency and Ingress latency marked.

Eth_frame_propagation.pdf

Thanks in advance.

Best Regards,
Jan Smrčina

  • Hello Jan Smrčina,

    Thank you for the query.

    I am reviewing the inputs and also checking internally with the experts. Please expect some delay in answering the query.

    Regards,

    Sreenivasa

  • Hello Jan Smrčina,

    Looks like there is an email thread that has been initiated to discuss the above requirement.

    I will update the highlights of the discussion as i receive them.

    Regards,

    Sreenivasa

  • Hello Kallikuppa Sreenivasa,

    I have opened e-mail discussion about the topic, but it does not map 1:1 to this thread.  I would prefer to clarify my assumptions and questions here in this thread, as the e-mail discussion is more about the implications of the answers I will get here and involves various people who do not need all the details from here.

    So please, keep this thread focused on the assumptions and questions presented here.

    Thank you!

    Best Regards,
    Jan Smrčina

  • Hello Jan Smrčina,

    Thank you for the note.

    I am checking internally and will have to reach out to the same team.

    Will update you based on the inputs i receive.

    Regards,

    Sreenivasa

  • Hello Sreenivasa,

    are there any news regarding this topic?

    Regards,
    Jan

  • Hello Jan Smrčina,

    Thank you for following on this.

    The team is returning after the yearend holidays this week.

    Let me check internally and update.

    Regards,

    Sreenivasa

  • Hello Sreenivasa,

    it's been a week already.  Time flies fast.

    Are there any news regarding this topic?

    Regards,
    Jan

  • Hi Jan

    We discussed this internally with Sreenivasa and Pekka and unfortunately we will not have the ability to provide any guidance on this level of questions and depth. 
    Our response on IDK and its feature set is similar to what Pekka provided in another thread from you 

    https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1123101/tmdx654idkevm-wrong-hw-timestamps---bad-precision-with-10mbit-and-100mbit .

    We do not have the ability to support the level of queries you have on this post. 

    Regards

    Mukul 

  • Jan

    Here is some guidance from   on your queries 

    Assumption 1:
    Both transmit and receive timestamping is implemented in PRU according to IEEE 802.3 Clause 90.
    (I. e. timestamping plane is enter/exit generic reconciliation sublayer of the MAC and message timestamping point is SFD.)

    [tley] While IEEE 802.3 is not an implementation specification on Ethernet MAC, it is correct that time stamps are taken with SFD reference point. In addition you have to understand various time domains on Ethernet. IEEE Ethernet does not specify time domain for MAC, bridge and Ethernet PHY. On ICSS_G there are various options to configure clock source for PRU (MAC, bridge), IEP timer (time stamp unit) and MII/RGMII (phy interface). In an ideal case all these clocks including the clock source for Ethernet PHY are one time domain. The IDK has hardware stuffing options to accomplish single time domain.

    Assumption 2:
    Timestamping implementation in PRU uses free running clock counter that is independent on TX_CLK and RX_CLK signals of the xMII interface.

    [tley] In case IEP timer is tuned to follow clock master then time stamp unit is not free-running. This should be marginal as IEP clock is typically tuned to compensate offsets of oscillators/cristal used on different boards. Typically < 50 ppm deviation.   

    Assumption 3:
    When crossing clock domains in PHY, resulting jitter can never be greater than one period of the target clock.

    [tley] As mentioned on TX direction PHY can be clocked with same source as MAC/bridge. We also have customers using denoising PLL with RX_CLK (25MHz fallback OSC on link loss) and provide network wide jitter in sub 10 ns range. This is similar to SYNC Ethernet from telecom standard. Not sure I understand what a period of the target clock is? (25 MHz, 125 MHz)

    Assumption 4:
    Processing in PHY, both sending and receiving, on 10Base-T, 100Base-TX and 1000Base-T (all Full Duplex) have zero jitter (excluding crossing of the clock domains).

    [tley] Correct for processing internal to ICSS_G. However TX_CLK direction is different on 100M vs Gig.

    Assumption 5:
    PCB wiring and magnetics have zero jitter.

    [tley] No, capacitive and inductive loads on high speed signals shape the edge of a signal which contributes to jitter. Maybe true, for your 0 jitter definition of < 4ns.

    Assumption 6:
    When transmitting MAC and transmitting PHY share clock signal source, crossing the MAC-PHY clock domain interface has constant latency (zero jitter).

    [tley] True in general. Phy may also have power-up/reset jitter in term of TX_CLK. DP83826 is optimized to have only 2 ns variation of TX_CLK with reset.

    Assumption 7:
    When RX_CLK is driven by the line clock (from received data), there is zero added jitter in the PHY processing (thanks to no clock-domain crossing).

    [tley] Yes, this mode will get you to < 10 ns jitter in a network.

    Question 1:
    Can the PHY, during transmission driven by RGMII (driven by GTX_CLK from PRU) use this clock source for transmission on the medium for 10Base-T Full Duplex, resp. 100Base-TX Full Duplex operation?
    (This seems theoretically very well possible at least for 10Base-T, as line clock is not continuous there and receiver clock is always synchronized at the start of the frame, during preamble.)

    [tley] This is a phy question and would need to be addressed by Ethernet team in TI.

    Please consider creating a separate post for their product for the subject matter experts on that team to address 

    Question 2:
    Can the PHY, during receiving, use link clock to drive the (RG)MII RX_CLK (and the whole decoding process)?  If not at all times, what are the conditions when this is/can be done?

    [tley] I do not understand the question. RX_CLK is a recovered clock from previous network node when link is established. RX_CLK is a phy output going into MII interface of ICSS_G.

     

  • Hello Mukul,

    Thank you very much, Thomas's comments were very helpful!

    I will clarify the PHY questions on the Interface forum.

    Additionally, Thomas stated that:

    In an ideal case all these clocks including the clock source for Ethernet PHY are one time domain. The IDK has hardware stuffing options to accomplish single time domain.

    I would very much like to have the entire Industrial Ethernet interface to work in single time domain (PRUICSSG + PHY).
    It would be even better if all PRUICSSGs, all PHYs and also Linux running on the A53 processor have the same time domain so I could avoid time synchronization shifting the clock all the time.

    Would you be able to provide some guidance on how to achieve this on the IDK?

    Thank you!

    Best Regards,
    Jan

  • Hello,

    any updates?

    Best regards,
    Jan

  • Jan

    Sorry for the delay - I am not sure we will have any guidance on how to customize the SDK and IDK hardware to achieve this. I will forward this post to Thomas Leyrer to see if he has any additional guidance on this topic and request him to directly respond here.

    Regards

    Mukul 

  • Jan,

      I see here three options.

    Default setting is RGMII interface with phy and SOC with own clock source. 

    DP83867 has also MII mode which I would recommend for 100 Mbit testing. Description how to change to MII mode is in chapter 7 of EVM users guide.  https://www.ti.com/lit/ug/spruim6a/spruim6a.pdf 

    Figure 3-4 of same document shows the clock tree for SoC and Ethernet Phy.

    Default option is that EPHYs and SOC run from different sources. There are 2 options.

    1. Use SOC clock source and route MCU_CLKOUT to clock buffer. Switch selection on clock buffer to MCU_CLKOUT.

    2. Use EPHY clock buffer source and route WKUP_OSC0_XIN from clock buffer to SoC. This option does not require any software change but only hardware mods with different resistor settings.

    We talk about clock synchronization in nano second range.  Linux OS is not sensitive to ns clock jitter. 

    I recommend to start with clock modification option 2 first - only hardware changes. 

    BR,

      Thomas

  • Thank you, Thomas, for your time and your insightful answer.

    We already have one board modified to use MII, but sending did not work and yesterday we were informed that MII mode is not supported by PRU FW provided for Linux.  Because of this lack of FW support, this modification is not usable for us, and we are currently planning to switch the modified board back to full RGMII configuration.

    I have to admit I was aware of the first possibility to switch the clock source, but somehow completely missed the second, better alternative.
    I think trying this out will be our next step.

    While Linux OS is not sensitive to ns clock jitter, time synchronization required for industrial control protocols are.  We are required to have 8 ns precision of timestamps because of the synchronization.  We have actually also other use cases requiring precise timestamps, all perfectly realizable with Linux userspace application.

    Best Regards,
    Jan

  • While the issue is still open, I have no further questions at the moment.
    It is now our turn to take action --- prepare, carry out and evaluate measurements, which is effort that has to be planned.  Therefore if I have further questions, I cannot say now when it will be.

    So, if the threads are closed and new questions arise, I will either start a related question, or ask internally for unlocking the thread.  Until then, feel free to lock this thread.