This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

EVMK2EX: Link not coming up, finding clock propogation for PCIe-0 as root complex

Part Number: EVMK2EX
Other Parts Discussed in Thread: CDCM6208

I am experimenting with the EVMK2EX board to create a PCIe Root Complex using PCIe-0 through the AMC. We confirmed that Pin 80 and pin 81 (FCLKA-/A+) are receiving 10 ns clock from an RDK switch, using a x1 lane SMA connection. The clock signals from SMA's pin 13/14 are connected to AMC's FCLKA+/FCLKA-.

My VxWorks driver is failing to establish PCIe link. There may be several reasons for the failure, so I wanted to first confirm the Clock signal propagation from the SoC to AMC.

Kindly let me know how the correct clock signal can be configured. The TRM and Schematics refer to BMC (TI LMS2D93), but other than TRM's Section 2.7 sentence that the LMS2D93 registers are accessible through SPI interface, I haven't found a reference to how the MUX can select AMC's FCLKA+/A-. Please point me to appropriate references. 

Following diagrams are from the "EVMK2E Technical Reference Manual" spruin5.pdf

Figure 2.4 EVMK2E Clock Domains shows a Mux that controls which clock signal (from AMC / from CDCM6208) reaches SoC's PCIe-0_CLKP/N pair.

Based on eInfoChip's K2E_EVM_SCH_16_00175_03.pdf page 19, the IDT5V41068 PCI Clock Mux selects source of PCIE0CLKP_M/N_M pair.

Thanks,

Subodh

  • TI WIKI for PCIe driver has following interesting statements. I wish the "unsupported" code was available as a reference somewhere.

    Thanks!

    > Port 0 is known to work on customer boards using SATA interface. Internally verified using microTCA AMC chassis and using intel's e1000e NIC card on port 0. The driver for e1000e is supported by intel on its website and IS NOT supported by TI.

  • Hi, Subodh,

    I'll have hardware expert to review and answer your hardware related questions. Regarding the "unsupported" code, it is as described in the User's Guide that it is the driver for Intel's e1000e NIC card. The driver can be downloaded from Intel website.

    By the way, you should refer to ProcSDK's User Guide in 

    http://software-dl.ti.com/processor-sdk-linux/esd/docs/latest/linux/Foundational_Components/Kernel/Kernel_Drivers/PCIe/PCIe_Root_Complex.html

    MCSDK is not supported any more, and wiki page will be EOL by end of this year. All info in wiki pages should have been migrated to PLSDK pages.

    Rex

  • Subodh,

    I am confused by your clocking questions.  Are you trying to use the K2E EVM as a PCIe Root Complex or a PCIe Endpoint.  Normally, an Endpoint receives a clock from a Root Complex or from an alternate source.  Why are you trying to supply a clock to a root complex?

    PCIe can operate with narrow band clocks or with SSC clocks.  When using SSC clocks, both the RC and the EP must receive the same clock.  If both are using narrow band clocks, then they do not need to be synchronized.  Are you using SSC or NB?

    Are you trying to work in single lane or dual lane mode?  You might want to limit your testing to single-lane until you have your clocking configuration resolved.

    Tom

  • @Rex, thanks a lot for the link and updates.

    @Tom, thanks for your reply.

    Pls see the Table 6-2 of 66ak2e05.pdf pins PCIE0CLKN/P, and PCIE1CLKN/P are Input only. Pls also see the Fig 2.4 from EVMK2E TRM captured in my original post. Given those two, I don't understand how K2E's PCIE0 "root complex can supply the clock" as normally expected. 

    SSC vs NB clocks: We will be happy with NB at this point :-)

    Single lane only at this point.

    You're absolutely correct about focusing on minimal configuration until the clocking issue is resolved.

    and I am still reviewing code. Kindly post anything you come across.

  • Subodh,

    If you are operating with NB PCIe clocking, then the internal clock provided for the PCIe reference is sufficient.  You simply only need to make sure that the EP is also receiving a NB reference clock.  There is no need to supply an external clock to the EVM.

    Tom

  • Tom, I've connected the PCIe-0 through AMC to a Broadcom RDK. The link isn't coming up. My device driver code is derived from PCIe-1 Root Complex example of VxWorks. DTS (Device Tree file) is correctly modified with register values and clock domains.

    Any ideas about how to overcome that? Or should I post code and .dts snippets here? I don't want to post VxWorks code to avoid copyright issues.

  • Hi, Subodh,

    TI's K2E device tree has PCIe0 disabled by default. Only PCIe1 is enabled which is connected to the SATA controller. Make sure you have PCIe0 enabled.  It can be done by changing the status to "okay" the same way as for PCIe1 in dts file. 

    Please do not post vxWorks code in TI forum.

    Rex

  • Hi Rex, yes, I have changed the status of PCIe0 as "ok" in the .dts file. I've confirmed (using scope) that the device driver is reaching the Link Training stage.

    Thank you for confirming that I should not post VxWorks code here.

  • Hello and ,

    As Tom wrote above "normally RC" is expected to supply PCIe Clock to downstream devices (switch / end point). However, EVMK2E board's Clock Source generates 3 clocks of 100 MHz - SATA_CLKN/P (terminates at PCIe-1 to SATA Controller bridge onboard), and PCIE0CLKP/N, and PCIE1CLKP/N. Both PCIExCLK terminate at K2E. 

    Kindly confirm above following pages of schematics and please let me know if I am correct about that.

    Page 19: PCI CLOCK MUX and SOC Reference CLOCK

    Page 16: PCIe To SATA Controller

    Page 24: CLOCK SOURCE -2

    If my understanding is correct, then I have following questions for you:

    1. Is it possible - and if so how - for EVMK2E to provide clock to PCIe-0 Root Complex's downstream devices?
    2. If not, can an external clock be sufficient? Since it will be asynchronouse to the PCIe-0 data stream, will the Root complex work ok?
    3. With Intel's e1000e NIC card (example cited above) where was the clock source?
    4. In a different application, I want to use PCIe-0 as an End Point. Is it possible to use AMC FCLKA+/A- as PCIe-0 End Point's clock signals?

    Thank you very much.

  • Subodh,

    1.  This EVM is designed to either be a root complex where both devices use local, narrow band clocks or to be an end point where it either uses a local NB clock or it receives clock.  This EVM does not have standard support for providing a clock output.

    2.  Separate narrow band clocks that are asynchronous are acceptable.

    3.  I assume a NIC card is an endpoint that assumes that it will receive a clock from the root complex.

    4.  Yes, you can receive a clock over the AMC pins for operation as an endpoint.  Enumeration will be complicated since the device reset control is not connected over the AMC connector.  Also, most boot modes are too slow.  Developers normally have the endpoint code awake and waiting for the RC to start enumeration.

    Tom

  • Tom,

    Thank you for detail answers. I have only one question at this time.

    Does the K2E PCIe0 interface needs to be initialized to accept asynchronous or synchronous mode for narrow band clock?

  • Subodh,

    I do not believe their is a configuration difference.  We need to get confirmation from the software team.

    Tom

  • Hi, Subodh,

    You didn't mention vxWorks is Linux or RTOS version. In TI Linux, clocks for both PCIe ports are defined in the dts file and would be initialized.

    PCIe0

    clkpcie: clkpcie@2350028 {
    #clock-cells = <0>;
    compatible = "ti,keystone,psc-clock";
    clocks = <&chipclk12>;
    clock-output-names = "pcie";
    reg = <0x02350028 0xb00>, <0x0235000c 0x400>;
    reg-names = "control", "domain";
    domain-id = <3>;
    };

    pcie0: pcie@21800000 {
    compatible = "ti,keystone-pcie", "snps,dw-pcie";
    reg = <0x21800000 0x1000>, <0x21801000 0x1000>, <0x21802000 0x1000>;
    reg-names = "app", "dbics", "config";
    clocks = <&clkpcie>;
    clock-names = "fck";
    #address-cells = <3>;

    PCIe1:

    clkpcie1: clkpcie1@235006c {
    #clock-cells = <0>;
    compatible = "ti,keystone,psc-clock";
    clocks = <&chipclk12>;
    clock-output-names = "pcie1";
    reg = <0x0235006c 0xb00>, <0x02350048 0x400>;
    reg-names = "control", "domain";
    domain-id = <18>;
    };

    pcie1: pcie@21020000 {
    compatible = "ti,keystone-pcie", "snps,dw-pcie";
    reg = <0x21020000 0x1000>, <0x21021000 0x1000>, <0x21022000 0x1000>;
    reg-names = "app", "dbics", "config";
    clocks = <&clkpcie1>;
    clock-names = "fck";

    Rex

  • Subodh,

    Is correct understanding that K2E PCIE port 0 is directly connected to a Broadcom device without any PCIE switch in between?

    And the K2E PCIE clock is supplied with the onboard 100MHz NB?

    How the Broadcom device PCIe is provided? Does it also have a onboard clock? Or it is supplied by the K2E through the AMC/SMA PCIe cable connection? Is your connection cable also passing through the PCIE clock? Note, you can't let Broadcom device to receive clock from both.

    >>>>Does the K2E PCIe0 interface needs to be initialized to accept asynchronous or synchronous mode for narrow band clock?>>>>Yes, but I think it is enabled and initialized on Linux driver, if you can see the training sequence on the scope, it should be fine.  

    Regards, Eric

  • Thank you. Using VxWorks RTOS.

    I am setting up two experiments. One where K2E EP will directly connect with an Intel SBC RC. Second where K2E EP will connect with K2E RC through a pair of SMA cables and AMC cards.

    I will update this thread with my observations and questions.

    Thank you for excellent support. Kindly keep this thread open for another week.

  • Hi, Subodh,

    Yes, please update. I'll keep this thread open.

    Rex

  • I modified vxWorks driver such that EP enables link training and continues to print value of DEBUG0 every few microseconds. Please note, since I don't have access to Synopsys DW I am only relying on DEBUG0 here.

    Intel SBC + K2E EP: Not very successful with K2E EP directly connected to a Intel SBC's PCIe slot. K2E EP LTSSM was seen transitioning only between DETECT_QUIET -> DETECT_WAIT -> POLL_ACTIVE and back and forth between DETECT_WAIT and POLL_ACTIVE. We now believe this doesn't work because the Intel SBC may not allow bifurcation of less than x4 lanes. So, we are NOT pursuing this path any further.

    When the two K2E boards (one in RC and another in EP) are used, then the LTSSM on EP does proceed, in addition to above states to CFG_LINKWD_START state. However, later it seems to fall back to DETECT_WAIT. Both K2E device entries has num_lanes = 1, same as the physical lanes (6 SMA cables - TX, RX, REFCLK) connected across a pair of AMC to SMA Ultra 9000 boards.

    Following is the link for the "AMC SMA Ultra 9000" boards I am using.

    http://www.silicontkx.com/index.php?route=product/product&path=62&product_id=52

    Any ideas? Attached is the log of  EP DEBUG0 values


    DEBUG0_REG:0x00006300
    DEBUG0_REG:0x00005900
    DEBUG0_REG:0x0000c700
    DEBUG0_REG:0x00001700
    DEBUG0_REG:0x0000c406
    DEBUG0_REG:0x0c000607
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c001a06
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c006406
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c007906
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x5c006c06
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c00f747
    DEBUG0_REG:0x0c006206
    DEBUG0_REG:0x0c000607
    DEBUG0_REG:0x0c00f406
    DEBUG0_REG:0x0c00f747
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c008106
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c008f06
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x50008206
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c000506
    DEBUG0_REG:0x0c00f747
    DEBUG0_REG:0x0c00f747
    DEBUG0_REG:0x0c003406
    DEBUG0_REG:0x0c00bc42
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c004406
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c000007
    DEBUG0_REG:0x0c00c306
    DEBUG0_REG:0x0c004a07
    DEBUG0_REG:0x0c00fd06
    DEBUG0_REG:0x0c00f742
    DEBUG0_REG:0x0c00bc42
    DEBUG0_REG:0x0c007c06
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c001906
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c00f506
    DEBUG0_REG:0x0c000602
    DEBUG0_REG:0x0c000002
    DEBUG0_REG:0x0c00a406
    DEBUG0_REG:0x0c00f742
    DEBUG0_REG:0x0c006402
    DEBUG0_REG:0x0c00c106
    DEBUG0_REG:0x0c00bc42
    DEBUG0_REG:0x0c008506
    DEBUG0_REG:0x0c006402
    DEBUG0_REG:0x0c006402
    DEBUG0_REG:0x0c002506
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c001e06
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c005e06
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c001306
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c00aa06
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c004a02
    DEBUG0_REG:0x0c004a02
    
    

  • Hi,

    We tested K2E EVM (RC) and K2E EVM (EP) connection long ago, there was no problem and the code hasn't been changed:  pdk_k2e_x_x_xx\packages\ti\drv\pcie\example\sample\src

    >>>th K2E device entries has num_lanes = 1, same as the physical lanes (6 SMA cables - TX, RX, REFCLK) connected across a pair of AMC to SMA Ultra 9000 boards.>>>>>> Can you disconnect the SMA cable for REFCLK P/N? On the K2E EVM has onboard crystal generating 100MHz PCIE clock, if you let it pass to the other side via SMA cable, it may cause some problem. 

    Regards, Eric

  • Hi and

    Can one of you post here validation setup for K2E EVM (RC) and K2E EVM (EP) connection? I would love to know the part numbers of components used between the two AMC PCIe Edge connectors. If you don't have that readily accessible then a representative list will be great as well.

    I also found a list of validating cards on TI’s PCIe Root Complex page Based on that, I am thinking about purchasing x1 cards such as these two.

    https://www.newegg.com/broadcom-bcm5721/p/14U-004W-00003

    http://static.highspeedbackbone.net/pdf/Intel_EXPI9300PT_DataSheet.pdf

    Do you have a part number for x1 Ethernet card that can be used to prove K2E EVM RC? I ask because the TI’s PCIe Root Complex page mentions those cards were used to prove K2G (and not K2E) EVM.

    Thanks,

    Subodh

  • Subodh,

    We do not produce test boards for connecting two EVMs across PCIe.  You can use 3rd party hardware for this.  We recommend that you use the AMC SMA Ultra 9000 from http://www.silicontkx.com/.  These boards connect to the AMC connector and provide SMA breakout connections.  SMA cables can then be used to cross-connect the PCIe data pairs.  Both boards will operate using their local, narrow-band PCIe reference clock.

    Tom