• State
  • Locked Locked
  • Replies 10 replies
  • Subscribers 99 subscribers
  • Views 393 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

DRA829V: Facing issues while enumeration of PICe

Charan cherry
Prodigy 150 points

Part Number: DRA829V
Other Parts Discussed in Thread: DRA829,

Hi,

I am successfully detected PCIe0 in EVM and unable to detect in custom board,when i compare registers of both, i observed the following registers difference

1)link control status register

In EVM ,negtotiated link width is 0x1 but in my board it is 0x2 even though i loaded same BSP image in both,

my hardware connection for lane 1 is not connected, lane 0 is connected and my endpoint is RTL8111.

2)link control status 2 register

TLS field is 0x1 in EVM , 0x4 in my board

HASD filed is 1 in EVM , 0x0 in my board

Thanks.

  • 0
    TI__Guru 61213 points

    Hi Charan,

    Good to see you on the forums again. 

    For the EVM and custom hardware, are they both running QNX, or are they running different OS like a Linux OS?

    Regards,

    Takuma

  • 0 in reply to Charan cherry
    TI__Guru 61213 points

    Hi Charan,

    Two questions:

    1. Is your custom hardware using the internal reference clock (aka, is the DRA829 providing a 100MHz reference clock to the PCIe slot/device on the custom board)? Or, is the custom board using the same design as TI EVM and it has a clock generator device that is separate from the SoC designed into the board, and that clock generator device is providing the reference clock to both SoC and PCIe slot?

    2. During PCIe enumeration is the link coming up? For example, in Linux driver from TI, there is a "Link up" log that will print out once the LINKSTATUS bit gets set. 

    In any case, if internal reference clock is in-use for the custom board and using QNX, maybe the reference clock is not enabled, or if it is enabled, there is some instability. And I ask these questions since these questions will also be relevant to the E2E thread that you have referenced.

    Regards,

    Takuma

  • 0
    Prodigy 150 points

    Hi Takuma,


    1) My custom board is using same design as TI EVM as it has a clock generator device that is separate from the SoC designed into the board, and that clock generator device is providing the reference clock to both SoC and PCIe slot.

    2) I am seeing the device in pci-tool -v when it is detected.

    3) I observed jitter in the clock generated from clock generator. What might be the issue ?

    4) How can I decide the problem is due to clock instability and how can I solve this for successful detection of pci device.

    Thanks.

  • 0 in reply to Charan cherry
    TI__Guru 61213 points

    Hi Charan,

    Let me loop in the hardware team to see if they can give suggestions. 

    Regards,

    Takuma

  • 0 in reply to Takuma Fujiwara
    TI__Intellectual 870 points

    Hi Charan,

    Thanks for the register comparison — it's very helpful. A quick note on what those values are telling us before getting into next steps:

    The RTL8111 is a x1 device, so a successfully trained link would always show Negotiated Link Width = x1. The x2 value you're seeing is the DRA829V's configured width, not a negotiated result — training never completed. For the TLS field, note that the Link Control 2 register uses a one-hot encoding: 0x1 = 2.5 GT/s, 0x2 = 5.0 GT/s, 0x4 = 8.0 GT/s (Gen3). Your custom board showing TLS = 0x4 is actually the Gen3 reset default — the correct starting value for the DRA829V. The EVM shows TLS = 0x1 (Gen1) because the driver successfully trained the link with the RTL8111 (a Gen1 device) and wrote TLS down to match. HASD = 0 is also a reset default — the EVM shows HASD = 1 because the driver set it after link-up.

    All three register differences tell the same story: the PCIe driver completed initialization on the EVM but never ran to completion on your custom board.

    Below are two suggestions:

    1. Read the LTSSM state register

    This will tell us exactly where training is breaking down. The key states to look for (per TRM Table 12-125):
    - 0x00–0x01 (Detect): The SERDES cannot detect a receiver — clock or physical layer issue
    - 0x02–0x03 (Polling): Receiver detected but cannot lock — marginal clock or signal integrity
    - 0x05–0x08 (Configuration): Polling passed but lane/width negotiation is failing
    - 0x10 (L0): Link is up

    Please share this value — it will tell us which direction to go next.

    2. Compare your CTRL_MMR0 SERDES and ACSPCIe register values against the EVM (per TRM section 12.2.4.3.1)

    Even with the same external clock generator hardware, there are CTRL_MMR0 registers that must be configured after reset to route the SERDES lanes to the PCIe controller and disable the ACSPCIe clock output buffer. Specifically:

    - `CTRLMMR_SERDES0_LNx_CTRL` — `LANE_FUNC_SEL` must be set to 0x1 for PCIe on your active lanes. Without this, the SERDES lanes are not routed to the PCIe controller at all.
    - ACSPCIe `CLK_SEL` — since your board uses an external clock generator, this should be set to 0x3 (disabled) so the DRA829V is not trying to drive the reference clock output while the external generator is also driving those lines.

    If your custom board uses a different device tree or board config file than the EVM, these registers may not be getting written.

    Regards,

    Jeff

  • 0
    Prodigy 150 points

    Hi Jeff,

    I am pasting some of the registers read in my custom board and in EVM with Address,Register Name,Register Value.

    For your suggestions, I am having 

    1) LTSSM state as Polling Configuration(0x03).

    2) LANE_FUNC_SEL as 0x1, Regarding ACSPCIE, please see below register dump. 

    Thanks.

    ******************* Custom Board Register dump ****************

    ./tmp/PCIe_Test_WS_R4 pcie2 dump

    x J721E PCIe Debug Tool x
    x Instance : PCIE2 (Custom board / SERDES_16G2) x
    x CFG base : 0x0E000000 x
    x LM base : 0x0E100000 x
    x SERDES : 0x05020000 (SERDES_16G2) x

    x PCIE2 (Custom board / SERDES_16G2)
    x Register Dump [CURRENT STATE]


    ****** CTRL_MMR ******

    [0x001040E8] SERDES_CTRL = 0x00000000
    Lane func [1:0] = 0 (PCIe mode OK)
    [0x001040A0] CTRLMMR_SERDES_LN0_CTRL = 0x00000001
    [0x001040A4] CTRLMMR_SERDES_LN1_CTRL = 0x00000001
    [0x00118090] CTRLMMR_ACSPCIE0_CTRL = 0x01000000
    [0x00118094] CTRLMMR_ACSPCIE1_CTRL = 0x01000003
    [0x00108078] CTRLMMR_PCIE_REFCLK_CLKSEL = 0x00000100
    [0x00108088] CTRLMMR_PCIE_CLKSEL = 0x00000000
    PLL lock [1] = 0 (not locked)
    [0x00104078] PCIE_CTRL = 0x00000183
    Enable [0] = 1 (Enabled)

    **** SERDES_16G2 WIZ [base=0x05020000] ******

    [0x05020404] SERDES_CTRL = 0x00000000
    POR_EN [31] = 0 (OK)
    [0x05020408] SERDES_TOP_CTRL = 0x1A000000
    REFCLK_MODE [31:30] = 0 (100MHz+ diff (PCIe correct))
    REFCLK_DIG_DIV[27:26]=2 (div4 for 100MHz (correct))
    [0x0502040C] SERDES_RST = 0xB1000000
    PHY_RESET_N [31] = 1 (Released (running))
    PLL0_REFCLK_SEL[28] = 1 (internal)
    REFCLK_DIG_SEL[25:24]=1 (check)
    [0x05020414] SERDES_CORE_STATUS = 0x80000000
    MACRO_PWR_EN_ACK[31]= 1 (Power-on complete)
    [0x0502048C] LANESTS0 = 0x00000003
    P0_PWR_EN_ACK [0] = 1 (Lane ready)

    ****** SERDES_16G2 Cadence PHY Core *******

    [0x05020080] CMN_PLLLC_STATUS_A = 0x00000095
    LOCKED [7] = 1 (PLL LOCKED)
    DCOCAL_DONE [6] = 0
    [0x0502E000] PHY_PMA_CMN_CTRL = 0x00002435
    cmn_plllc_locked[2] = 1
    cmn_ready [0] = 1 (CMN ready)
    [0x0502C01C] PHY_PLL_CFG = 0x00000011 (reset=0x11)
    SINGLE_LINK_PCIE[0] = 1 (correct)

    ****** PCIE2 (Custom board / SERDES_16G2) PCIe LM *****

    [0x0E000000] VendorID/DevID = 0x010017CD (RC accessible)
    [0x0E100000] PL_CONFIG_0 = 0x03000022
    LTSSM [29:24] = 0x03 Polling.Configuration
    RFC [15:8] = 0 (EP never responded)
    MLE [31] = 0
    [0x0E100004] PL_CONFIG_1 = 0x40808000
    [0x0E0000D0] LINK_CTRL_STS = 0x00210000
    DLLA [29] = 0 (not up)
    NLW [25:20] = x2 (NOT x1 -- mismatch with RTL8111!)
    NLS [19:16] = Gen1 2.5GT/s
    ASPM [1:0] = 0x0 (Disabled OK)
    [0x0E0000F0] LINK_CTRL_STS2 = 0x00010004
    TLS [3:0] = 0x4 (auto)
    HASD [5] = 0
    [0x0E100238] PHY_STATUS_1 = 0x00000000 (no errors)
    [0x0E10020C] LOCAL_ERR_STS = 0x00000000
    [0x0E100D00] LOCAL_ERR_STS2 = 0x00000000
    [0x0E100200] LANE_MAP = 0x00000000
    [0x0E100204] FTS_COUNT = 0x00000000 (EP FTS=0)

    ***** Root Cause Summary *****
    CTRL_MMR configured = YES
    SERDES POR_EN=0 = YES
    SERDES PHY_RESET_N=1 = YES
    SERDES MACRO_PWR_EN_ACK = YES
    SERDES PLL locked = YES
    SERDES Lane ready = YES
    PCIe RC accessible = YES
    LTSSM left Detect.Quiet = YES
    Link width x1 (RTL8111) = NO <-- NLW=x%u mismatch!

    #
    #
    # ./VATC/PCIe_Test_WS_R4 pcie2 poll

    x J721E PCIe Debug Tool x
    x Instance : PCIE2 (Custom board / SERDES_16G2) x
    x CFG base : 0x0E000000 x
    x LM base : 0x0E100000 x
    x SERDES : 0x05020000 (SERDES_16G2) x

    x PCIE2 (Custom board / SERDES_16G2)
    x LTSSM Live Poll (1000ms)

    [ 0ms] 0x03 Polling.Configuration
    Done. 1 transitions. Final LTSSM=0x03 Polling.Configuration

    #

    *****   EVM  Register dump  ************

    # ./tmp/PCIe_Test_WS_R4 pcie0 dump
    �╔�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═� ╗
    �║ J721E PCIe Debug Tool �║
    �║ Instance : PCIE0 (EVM / SERDES_16G0) �║
    �║ CFG base : 0x0D000000 �║
    �║ LM base : 0x0D100000 �║
    �║ SERDES : 0x05000000 (SERDES_16G0) �║
    �╚�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═� ╝

    �╔�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═� ╗
    �║ PCIE0 (EVM / SERDES_16G0)
    �║ Register Dump [CURRENT STATE]
    �╚�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═� ╝

    �─�─�─�─ CTRL_MMR �─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─ ─
    [0x001040E0] SERDES_CTRL = 0x00000000
    Lane func [1:0] = 0 (PCIe mode OK)
    [0x00104080] CTRLMMR_SERDES_LN0_CTRL= 0x00000001
    [0x00104084] CTRLMMR_SERDES_LN1_CTRL = 0x00000001
    [0x00118090] CTRLMMR_ACSPCIE0_CTRL = 0x01000000
    [0x00118094] CTRLMMR_ACSPCIE1_CTRL = 0x01000003
    [0x00108070] CTRLMMR_PCIE_REFCLK_CLKSEL = 0x00000100
    [0x00108080] CTRLMMR_PCIE_CLKSEL = 0x00000000
    PLL lock [1] = 0 (not locked)
    [0x00104070] PCIE_CTRL = 0x00000183
    Enable [0] = 1 (Enabled)

    �─�─�─�─ SERDES_16G0 WIZ [base=0x05000000] �─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─
    [0x05000404] SERDES_CTRL = 0x00000000
    POR_EN [31] = 0 (OK)
    [0x05000408] SERDES_TOP_CTRL = 0x1A000000
    REFCLK_MODE [31:30] = 0 (100MHz+ diff (PCIe correct))
    REFCLK_DIG_DIV[27:26]=2 (div4 for 100MHz (correct))
    [0x0500040C] SERDES_RST = 0xB1000000
    PHY_RESET_N [31] = 1 (Released (running))
    PLL0_REFCLK_SEL[28] = 1 (internal)
    REFCLK_DIG_SEL[25:24]=1 (check)
    [0x05000414] SERDES_CORE_STATUS = 0x80000000
    MACRO_PWR_EN_ACK[31]= 1 (Power-on complete)
    [0x0500048C] LANESTS0 = 0x00000003
    P0_PWR_EN_ACK [0] = 1 (Lane ready)

    �─�─�─�─ SERDES_16G0 Cadence PHY Core �─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─
    [0x05000080] CMN_PLLLC_STATUS_A = 0x00000094
    LOCKED [7] = 1 (PLL LOCKED)
    DCOCAL_DONE [6] = 0
    [0x0500E000] PHY_PMA_CMN_CTRL = 0x00002435
    cmn_plllc_locked[2] = 1
    cmn_ready [0] = 1 (CMN ready)
    [0x0500C01C] PHY_PLL_CFG = 0x00000011 (reset=0x11)
    SINGLE_LINK_PCIE[0] = 1 (correct)

    �─�─�─�─ PCIE0 (EVM / SERDES_16G0) PCIe LM �─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─� ─
    [0x0D000000] VendorID/DevID = 0x010017CD (RC accessible)
    [0x0D100000] PL_CONFIG_0 = 0x1000FF21
    LTSSM [29:24] = 0x10 L0 *** LINK UP ***
    RFC [15:8] = 255 (EP transmitting TS)
    MLE [31] = 0
    [0x0D100004] PL_CONFIG_1 = 0x40808000
    [0x0D0000D0] LINK_CTRL_STS = 0x00110000
    DLLA [29] = 0 (not up)
    NLW [25:20] = x1 (x1 correct for RTL8111)
    NLS [19:16] = Gen1 2.5GT/s
    ASPM [1:0] = 0x0 (Disabled OK)
    [0x0D0000F0] LINK_CTRL_STS2 = 0x00010004
    TLS [3:0] = 0x4 (auto)
    HASD [5] = 0
    [0x0D100238] PHY_STATUS_1 = 0x00000000 (no errors)
    [0x0D10020C] LOCAL_ERR_STS = 0x00000000
    [0x0D100D00] LOCAL_ERR_STS2 = 0x00000000
    [0x0D100200] LANE_MAP = 0x00000001
    [0x0D100204] FTS_COUNT = 0x00000000 (EP FTS=0)

    �─�─�─�─ Root Cause Summary �─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─�─
    CTRL_MMR configured = YES
    SERDES POR_EN=0 = YES
    SERDES PHY_RESET_N=1 = YES
    SERDES MACRO_PWR_EN_ACK = YES
    SERDES PLL locked = YES
    SERDES Lane ready = YES
    PCIe RC accessible = YES
    LTSSM left Detect.Quiet = YES
    Link width x1 (RTL8111) = YES

    #
    # ./tmp/PCIe_Test_WS_R4 pcie0 poll
    �╔�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═� ╗
    �║ J721E PCIe Debug Tool �║
    �║ Instance : PCIE0 (EVM / SERDES_16G0) �║
    �║ CFG base : 0x0D000000 �║
    �║ LM base : 0x0D100000 �║
    �║ SERDES : 0x05000000 (SERDES_16G0) �║
    �╚�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═� ╝

    �╔�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═� ╗
    �║ PCIE0 (EVM / SERDES_16G0)
    �║ LTSSM Live Poll (1000ms)
    �╚�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═�═� ╝
    [ 0ms] 0x10 L0 *** LINK UP ***
    Done. 1 transitions. Final LTSSM=0x10 L0 *** LINK UP ***

    #

  • 0 in reply to Charan cherry
    TI__Intellectual 870 points
    Hi Charan,

    Thanks for the detailed register dump. Your CTRL_MMR configuration is correct — all values match the EVM. The SERDES is fully initialized (PLL locked, lane ready, CMN ready). The problem doe not seem to be software configuration.

    Looking at PL_CONFIG_0, there is one clear difference between your custom board and the EVM:

    - Custom board: `0x03000022` → LTSSM = 0x03 (Polling.Configuration), NLC = x2, LS = 0 (training not complete)
    - EVM: `0x1000FF21` → LTSSM = 0x10 (L0, link up), NLC = x1, LS = 1 (training complete)

    Your PCIe controller is attempting a x2 link, while the EVM is configured for x1. The RTL8111 is a x1 device and only Lane 0 is connected, so x2 is incorrect. This matters because the PCIe spec requires all lanes that detected a Receiver to complete training sequence exchange before the LTSSM can advance immediately out of Polling.Active (PCIe Base Spec Section 4.2.6.2.1). With x2, if the unconnected Lane 1 is being included in training, this condition can never be met — forcing the controller into a slower timeout path that may prevent the link from training successfully.

    Please try the following, in order:

    1. Configure PCIe2 for x1 link width in your BSP or device tree

    This is the only configuration difference we can see between your board and the EVM. Check your device tree for the `num-lanes` property (or equivalent in QNX) for this PCIe instance and set it to 1.

    2. If the link still fails at x1, capture scope measurements during power-up

    We will need all three signals on the same timebase:
    - PERST# at the RTL8111 pin (should be low during power-up, then go high — must stay low at least 100ms after power rails are stable per PCIe CEM spec T_PVPERL)
    - 3.3V power rail to the RTL8111
    - REFCLK at the RTL8111 input (must be running and stable before PERST# deasserts — you mentioned observing jitter on the clock generator, so this is important to quantify)

    Note: the DRA829V's SERDES PLL is locked (CMN_PLLLC_STATUS_A LOCKED = 1), so the clock at the DRA829V side is sufficient for PLL lock. If there is a clock issue, it would be in the path to the RTL8111.

    Please try the x1 change first and let us know the result.

    Regards,
    Jeff
  • 0 in reply to Jeff Saied
    Prodigy 150 points

    Hi Jeff,

    I Changed to x1,but LTSSM state still in Polling Configuration.

  • 0 in reply to Charan cherry
    TI__Intellectual 870 points
    Hi Charan,

    Thank you for trying the x1 change. Can you please share an updated register dump. I would like to confirm the x1 change actually reflected in PL_CONFIG_0 (NLC [2:1] should now read 00 instead of 01). I also want to see whether RFC [15:8] has changed at all. The dump format from your previous run is fine. Can you also please check the following?
    ---
    1. Scope capture: REFCLK and PERST# at the RTL8111 (highest priority)

    You mentioned observing jitter on the clock generator. With the DRA829V's own SERDES PLL confirmed locked, any remaining clock quality problem is almost certainly in the path to the RTL8111. A poor REFCLK at the RTL8111 input would prevent it from ever entering Polling.Active — it would not send TS1 back to the DRA829V, and the TS2 handshake in Polling.Configuration would never complete.

    Please capture the following on the same oscilloscope timebase during a power-up/reset cycle:

    - REFCLK at the RTL8111 CLKP/CLKN input pins — should be a stable 100 MHz ±300 ppm differential signal, present and settled before PERST# deasserts.
    - PERST# at the RTL8111 PERST# pin — should assert cleanly on power-up with no glitches or bounce, then deassert once power is stable.
    - 3.3V rail at the RTL8111 VCC3.3 supply — should be stable at least 100 ms before PERST# deasserts (PCIe CEM spec T_PVPERL requirement).

    The sequence must be: 3.3V stable → REFCLK stable → PERST# deasserts. If REFCLK is not present and stable before PERST# deasserts, the RTL8111 cannot initialize.
    ---
    2. Verify Lane 0 signal path routing

    Please confirm the following from the schematic:

    - DRA829V SERDES Lane 0 TX+/TX- → RTL8111 RX+/RX- — correct lane, correct polarity (not swapped with Lane 1, not polarity-inverted)
    - RTL8111 TX+/TX- → DRA829V SERDES Lane 0 RX+/RX- — correct lane, correct polarity
    - AC coupling capacitors present on all four differential pairs (both TX and RX directions). Per PCIe Base Spec Table 8-8, CTX must be 176–265 nF.
    A single wrong connection here would block training regardless of software configuration.

    ---
    Regards,
    Jeff