• State
  • Locked Locked
  • Replies 13 replies
  • Subscribers 101 subscribers
  • Views 907 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMDS64EVM: PCIe link failure connected to PC mainboard

Grant Edwards
Prodigy 50 points
Part Number: TMDS64EVM
Other Parts Discussed in Thread: TMDS243EVM,

The board-to-board PCIe benchmark runs fine with a TMDS64EVM endpoint connect to a TMDS243EVM root complex using an rx/tx crossover PCIe cable (both eval boards are running R5 example apps).

When the TMDS64EVM endpoint is connected to a PC mainboard running Linux, the am64 endpoint never gets past the call to Pcie_waitLinkUp(). I've watched the LTSSM status value: it starts out alternating between 0x00 and 0x01. When the PC is rebooted and the PCIe bus is enumerated, the am64's LTSSM status then changes to 0x03 (POLL COMPLIANCE) and stays at that value until the R5 endpoint application is reloaded and restarted. Then it will start alternating between 0x00 and 0x01 again -- until the PC is rebooted and it again stops at 0x03.

We tried changing the resistors on p30 of the EVM schematic so that the am64 would use the PCIe reference clock provided by the PC mainboard (and reconnected the refclk pair in the cable — which had been disconnected). That made no difference: the link state machine still stops at 0x03 (POLL COMPLAINCE) when the PC host performs bus enumeration.

The pinout of the am243/am64 PCIe slot is the the same as a PC mainboard, correct? So the rx/tx crossover cable that works when the am64 endpoint is connected to the am243 root complex should be correct when it is used to connect the am64 endpoint to a PC mainboard?

What might be causing the link failure?

  • 0
    TI__Guru**** 168351 points

    Hi Grant,

    What OS do you use on TMDS64EVM? You mentioned "R5 endpoint application", so do you use MCU SDK running on R5 controlling the AM64x PCIe interface?

  • 0 in reply to Bin Liu
    Prodigy 50 points

    The am64 is running the R5 baremetal endpoint PCIe benchmark demo application.

  • 0 in reply to Grant Edwards
    TI__Guru**** 168351 points

    Hi Grant,

    Thanks for the update. I am routing your query to our MCU+ SDK team for comments.

  • 0
    Prodigy 50 points

    I've since been told that for the EVM to be detected, the PC must be powered off and then on instead of just doing a "reboot" from the Linux command line.

    When I do that, the link status continues to alternate between 0x00 and x01 forever. It never goes to state 0x03 they way it does when I "reboot" Linux.

  • 0 in reply to Grant Edwards
    Prodigy 50 points
    Grant Edwards said:
    I've since been told that for the EVM to be detected, the PC must be powered off and then on instead of just doing a "reboot" from the Linux command line.

    That was apparently not true.

    I was testing with a Dell Optiplex 3010 mini tower, and that apparently is too new/fast to work with the am64.  I plugged the am64 EVM into a 12 year old HP/Compaq machine, and the link came up immediately and the card was detected after doing a "reboot" of Linux.

    Now the problem is that the TI driver code in the MCU+ SDK refuses to work with the HP because it's gen1 and the demo app was configured for gen2. The TI drivers apparently don't support speed negotiation. You have to know at compile time whether you're going to plug into a gen1 or gen2 host?

  • 0 in reply to Grant Edwards
    Prodigy 50 points

    I never got the old HP machine to recognize the EVM a second time. After another 10-15 attempts, that PC died completely. The EVM reports that the link is up, but when rebooted (either power cycle or warm reboot), it never detected the EVM a second time.

  • 0 in reply to Grant Edwards
    Prodigy 50 points

    I've since tried with an 8 year old BIOSTAR mainboard (which supports PCIe up to 3.0). With it, I get the same alternating 0x00/0x01 LTSSM states that I did with the Dell.

    Can somebody give me a hint as to what brand/age/model of PC motherboard the TMDS64EVM is expected to work with?

    Did I post this in the wrong group?

  • 0 in reply to Grant Edwards
    TI__Guru* 83965 points

    Hi Grant

    I think the 2 threads are related 

    https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1213540/am6441-how-to-support-pcie-bus-enumeration-in-endpoint-mode

    I the above thread I had posted that currently all our testing with chip to chip, not PC to SoC.  The drivers we offer do not have the test coverage with PC motherboards, Host BIOS dependencies testing are not in scope of our offering. 

    We do have a few customers who have written their drivers from scratch and regressed it for their product needs on AM64/AM24

    Regards

    Mukul 

  • 0 in reply to Mukul Bhatnagar
    Prodigy 50 points
    Mukul Bhatnagar said:
    I the above thread I had posted that currently all our testing with chip to chip

    Yes, I'm aware of that. Unfortunately it doesn't matter to us whether the am64 PCIe works chip to chip or not.

    The only thing we care about is am64 to PC.

    That other thread was supposed to be about support for bus enumeration in EP mode.The MCU SDK documentation states that bus enumeration is not supported. That's presumably a separate issue from the link not coming up (which is what this thread was asking about).

    I'm now able to get a valid link and the Linux host will detect the board, but only with specific 10-12 year old PC motherboards that only support PCIe gen1. If I can keep that working, it should allow me to continue my evaluation (which will require bus enumeration support).

    What I had hoped to learn from that other thread:

    • What is missing from bus enumeration support?
    • How would one go about adding that it since the TRM doesn't document EP mode registers?
    • Where can I find documentation for EP mode registers?
    • Does the U-Boot PCIe EP driver code support bus enumeration?
    • Does the Linux EP driver code support bus enumeration?
  • 0 in reply to Grant Edwards
    TI__Guru* 83965 points

    Grant

    All fair and valid questions

    Will need some time to dig and assess. 

    On your hardware documentation/missing registers , we will be planning to add some of this in upcoming in the TRM, but we will see if we can post some relevant register info on this e2e to keep you moving.

    Regards

    Mukul 

  • 0 in reply to Mukul Bhatnagar
    TI__Mastermind 27050 points
    Grant Edwards said:
    Where can I find documentation for EP mode registers?

    Clarifying some of the registers for EP missing from the TRM here:

    The EP inbound ones are addresses are:

    ATU_FUNC0_WRAPPER_IB_EP_0_ADDR0 (0x400840)
    ATU_FUNC0_WRAPPER_IB_EP_0_ADDR1 (0x400844)
    ATU_FUNC0_WRAPPER_IB_EP_1_ADDR0 (0x400848)
    ATU_FUNC0_WRAPPER_IB_EP_1_ADDR1 (0x40084c)
    ATU_FUNC0_WRAPPER_IB_EP_2_ADDR0 (0x400850)
    ATU_FUNC0_WRAPPER_IB_EP_2_ADDR1 (0x400854)
    ATU_FUNC0_WRAPPER_IB_EP_3_ADDR0 (0x400858)
    ATU_FUNC0_WRAPPER_IB_EP_3_ADDR1 (0x40085c)
    ATU_FUNC0_WRAPPER_IB_EP_4_ADDR0 (0x400860)
    ATU_FUNC0_WRAPPER_IB_EP_4_ADDR1 (0x400864)
    ATU_FUNC0_WRAPPER_IB_EP_5_ADDR0 (0x400868)
    ATU_FUNC0_WRAPPER_IB_EP_5_ADDR1 (0x40086c)
    ATU_FUNC0_WRAPPER_IB_EP_6_ADDR0 (0x400870)
    ATU_FUNC0_WRAPPER_IB_EP_6_ADDR1 (0x400874)
    ATU_FUNC0_WRAPPER_IB_EP_7_ADDR0 (0x400878)
    ATU_FUNC0_WRAPPER_IB_EP_7_ADDR1 (0x40087c)
    And content is:

    ATU_FUNC0_WRAPPER_IB_EP_0_ADDR0 (0x400840)

    Provides bits 31:0 of the AXI address

     

    Bit

    Name

    Type

    Reset

    Description

    31:0

    data

    rw

    0x0

    Bits [31:0] of Address Register for BAR N

    ATU_FUNC0_WRAPPER_IB_EP_0_ADDR1 (0x400844)

    Provides bits 63:32 of the AXI address

     

    Bit

    Name

    Type

    Reset

    Description

    31:0

    data

    rw

    0x0

    Bits [63:32] of AXI Address Register for BAR N

     For BAR0 use 

    ATU_FUNC0_WRAPPER_IB_EP_0_ADDR0 (0x400840)
    ATU_FUNC0_WRAPPER_IB_EP_0_ADDR1 (0x400844)

    for address translation.
     
    For BAR1 use 
    ATU_FUNC0_WRAPPER_IB_EP_1_ADDR0 (0x400848)
    ATU_FUNC0_WRAPPER_IB_EP_1_ADDR1 (0x40084c)
    For BAR2-3 (64bit) use
    ATU_FUNC0_WRAPPER_IB_EP_2_ADDR0 (0x400850)
    ATU_FUNC0_WRAPPER_IB_EP_2_ADDR1 (0x400854)
    For BAR4-5 (64bit) use
    ATU_FUNC0_WRAPPER_IB_EP_4_ADDR0 (0x400860)
    ATU_FUNC0_WRAPPER_IB_EP_4_ADDR1 (0x400864)

    1.1.1.1          LM_I_REGF_LM_PCIE_BASE_I_PF_0_BAR_CONFIG_0_REG (0x100240)

    This register specifies the configuration of the BARs associated with the Physical Function 0

     

    Bit

    Name

    Type

    Reset

    Description

    31:29

    BAR3C

    rw

    0x0

    Specifies the configuration of BAR3. The various encodings are:

          000: Disabled

          001: 32bit IO BAR

          010-011: Reserved

          100: 32bit memory BAR, non prefetchable

          101: 32bit memory BAR, prefetchable

          110-111: Reserved

    28:24

    BAR3A

    rw

    0x5

    Specifies the aperture of the BAR 3 when it is configured as a 32-bit BAR.

          For 32-bit BAR 3, the valid encodings are:

          00000 = 128 B, 00001 = 256 B, 00010 = 512 B,  00011 = 1 KB,   00100 = 2 KB,   00101 = 4 KB, 00110 = 8 KB, 00111 = 16 KB,

          01000 = 32 KB, 01001 = 64 KB, 01010 = 128 KB, 01011 = 256 KB, 01100 = 512 KB, 01101 = 1 MB, 01110 = 2 MB,

          01111 = 4 MB, 10000 = 8 MB, 10001 = 16 MB, 10010 = 32 MB, 10011 = 64 MB, 10100 = 128 MB, 10101 = 256 MB,

          10110 = 512 MB, 10111 = 1 GB, 11000 = 2 GB

    23:21

    BAR2C

    rw

    0x0

    Specifies the configuration of BAR2. The various encodings are:

          000: Disabled

          001: 32bit IO BAR

          010-011: Reserved

          100: 32bit memory BAR, non prefetchable

          101: 32bit memory BAR, prefetchable

          110: 64bit memory BAR, non prefetchable

          111: 64bit memory BAR, prefetchable

    20:16

    BAR2A

    rw

    0x5

    Specifies the aperture of the 32-bit BAR 2 or 64bit BAR2-3.

          For 32-bit BAR 2, the valid encodings are:

          00000 = 128 B, 00001 = 256 B, 00010 = 512 B,  00011 = 1 KB,   00100 = 2 KB,   00101 = 4 KB, 00110 = 8 KB, 00111 = 16 KB,

          01000 = 32 KB, 01001 = 64 KB, 01010 = 128 KB, 01011 = 256 KB, 01100 = 512 KB, 01101 = 1 MB, 01110 = 2 MB,

          01111 = 4 MB, 10000 = 8 MB, 10001 = 16 MB, 10010 = 32 MB, 10011 = 64 MB, 10100 = 128 MB, 10101 = 256 MB,

          10110 = 512 MB, 10111 = 1 GB, 11000 = 2 GB

     

          For 64-bit BAR2-3, the valid encodings are:

         

          00000 = 128 B, 00001 = 256 B, 00010 = 512 B,  00011 = 1 KB,   00100 = 2 KB,   00101 = 4 KB, 00110 = 8 KB, 00111 = 16 KB,

          01000 = 32 KB, 01001 = 64 KB, 01010 = 128 KB, 01011 = 256 KB, 01100 = 512 KB, 01101 = 1 MB, 01110 = 2 MB,

          01111 = 4 MB, 10000 = 8 MB, 10001 = 16 MB, 10010 = 32 MB, 10011 = 64 MB, 10100 = 128 MB, 10101 = 256 MB,

          10110 = 512 MB, 10111 = 1 GB, 11000 = 2 GB, 11001 = 4 GB, 11010 = 8 GB, 11011 = 16 GB, 11100 = 32 GB,

          11101 = 64 GB, 11110 = 128 GB, 11111 = 256 GB

         

    15:13

    BAR1C

    rw

    0x0

    Specifies the configuration of BAR1. The various encodings are:

          000: Disabled

          001: 32bit IO BAR

          010-011: Reserved

          100: 32bit memory BAR, non prefetchable

          101: 32bit memory BAR, prefetchable

          110-111: Reserved

    12:8

    BAR1A

    rw

    0x5

    Specifies the aperture of the BAR 1 when it is configured as a 32-bit BAR.

          For 32-bit BAR 1, the valid encodings are:

          00000 = 128 B, 00001 = 256 B, 00010 = 512 B,  00011 = 1 KB,   00100 = 2 KB,   00101 = 4 KB, 00110 = 8 KB, 00111 = 16 KB,

          01000 = 32 KB, 01001 = 64 KB, 01010 = 128 KB, 01011 = 256 KB, 01100 = 512 KB, 01101 = 1 MB, 01110 = 2 MB,

          01111 = 4 MB, 10000 = 8 MB, 10001 = 16 MB, 10010 = 32 MB, 10011 = 64 MB, 10100 = 128 MB, 10101 = 256 MB,

          10110 = 512 MB, 10111 = 1 GB, 11000 = 2 GB

    7:5

    BAR0C

    rw

    0x4

    Specifies the configuration of BAR0. The various encodings are:

          000: Disabled

          001: 32bit IO BAR

          010-011: Reserved

          100: 32bit memory BAR, non prefetchable

          101: 32bit memory BAR, prefetchable

          110: 64bit memory BAR, non prefetchable

          111: 64bit memory BAR, prefetchable

    4:0

    BAR0A

    rw

    0x5

    Specifies the aperture of the 32-bit BAR 0 or 64bit BAR0-1.

          For 32-bit BAR 0, the valid encodings are:

          00000 = 128 B, 00001 = 256 B, 00010 = 512 B,  00011 = 1 KB,   00100 = 2 KB,   00101 = 4 KB, 00110 = 8 KB, 00111 = 16 KB,

          01000 = 32 KB, 01001 = 64 KB, 01010 = 128 KB, 01011 = 256 KB, 01100 = 512 KB, 01101 = 1 MB, 01110 = 2 MB,

          01111 = 4 MB, 10000 = 8 MB, 10001 = 16 MB, 10010 = 32 MB, 10011 = 64 MB, 10100 = 128 MB, 10101 = 256 MB,

          10110 = 512 MB, 10111 = 1 GB, 11000 = 2 GB

     

          For 64-bit BAR0-1, the valid encodings are:

         

          00000 = 128 B, 00001 = 256 B, 00010 = 512 B,  00011 = 1 KB,   00100 = 2 KB,   00101 = 4 KB, 00110 = 8 KB, 00111 = 16 KB,

          01000 = 32 KB, 01001 = 64 KB, 01010 = 128 KB, 01011 = 256 KB, 01100 = 512 KB, 01101 = 1 MB, 01110 = 2 MB,

          01111 = 4 MB, 10000 = 8 MB, 10001 = 16 MB, 10010 = 32 MB, 10011 = 64 MB, 10100 = 128 MB, 10101 = 256 MB,

          10110 = 512 MB, 10111 = 1 GB, 11000 = 2 GB, 11001 = 4 GB, 11010 = 8 GB, 11011 = 16 GB, 11100 = 32 GB,

          11101 = 64 GB, 11110 = 128 GB, 11111 = 256 GB

         

    1.1.1.1          LM_I_REGF_LM_PCIE_BASE_I_PF_0_BAR_CONFIG_1_REG (0x100244)

    This register specifies the configuration of the BARs associated with the Physical

    Function.

     

    Bit

    Name

    Type

    Reset

    Description

    31

    ERBC

    rw

    0x0

    Setting this bit to 1 enables the Resizable BAR Capability in the PCI Express

          Configuration Space of the associated Function. When the Resizable BAR Capability is enabled,

          the apertures of the memory BARs of the corresponding Function are no longer selected by the

          fields in this register, but by the setting of the registers in the Resizable BAR Capability

          Structure.

    30:24

    R24

    r

    0x0

    Reserved

    23:16

    R16

    r

    0x0

    Reserved

    15:13

    BAR5C

    rw

    0x0

    Specifies the configuration of BAR5. The various encodings are:

          000: Disabled

          001: 32bit IO BAR

          010-011: Reserved

          100: 32bit memory BAR, non prefetchable

          101: 32bit memory BAR, prefetchable

          110-111: Reserved

    12:8

    BAR5A

    rw

    0x5

    Specifies the aperture of the BAR 5 when it is configured as a 32-bit BAR.

          For 32-bit BAR 5, the valid encodings are:

          00000 = 128 B, 00001 = 256 B, 00010 = 512 B,  00011 = 1 KB,   00100 = 2 KB,   00101 = 4 KB, 00110 = 8 KB, 00111 = 16 KB,

          01000 = 32 KB, 01001 = 64 KB, 01010 = 128 KB, 01011 = 256 KB, 01100 = 512 KB, 01101 = 1 MB, 01110 = 2 MB,

          01111 = 4 MB, 10000 = 8 MB, 10001 = 16 MB, 10010 = 32 MB, 10011 = 64 MB, 10100 = 128 MB, 10101 = 256 MB,

          10110 = 512 MB, 10111 = 1 GB, 11000 = 2 GB

    7:5

    BAR4C

    rw

    0x0

    Specifies the configuration of BAR4. The various encodings are:

          000: Disabled

          001: 32bit IO BAR

          010-011: Reserved

          100: 32bit memory BAR, non prefetchable

          101: 32bit memory BAR, prefetchable

          110: 64bit memory BAR, non prefetchable

          111: 64bit memory BAR, prefetchable

    4:0

    BAR4A

    rw

    0x5

    Specifies the aperture of the 32-bit BAR 4 or 64bit BAR4-5.

          For 32-bit BAR 4, the valid encodings are:

          00000 = 128 B, 00001 = 256 B, 00010 = 512 B,  00011 = 1 KB,   00100 = 2 KB,   00101 = 4 KB, 00110 = 8 KB, 00111 = 16 KB,

          01000 = 32 KB, 01001 = 64 KB, 01010 = 128 KB, 01011 = 256 KB, 01100 = 512 KB, 01101 = 1 MB, 01110 = 2 MB,

          01111 = 4 MB, 10000 = 8 MB, 10001 = 16 MB, 10010 = 32 MB, 10011 = 64 MB, 10100 = 128 MB, 10101 = 256 MB,

          10110 = 512 MB, 10111 = 1 GB, 11000 = 2 GB

     

          For 64-bit BAR4-5, the valid encodings are:

         

          00000 = 128 B, 00001 = 256 B, 00010 = 512 B,  00011 = 1 KB,   00100 = 2 KB,   00101 = 4 KB, 00110 = 8 KB, 00111 = 16 KB,

          01000 = 32 KB, 01001 = 64 KB, 01010 = 128 KB, 01011 = 256 KB, 01100 = 512 KB, 01101 = 1 MB, 01110 = 2 MB,

          01111 = 4 MB, 10000 = 8 MB, 10001 = 16 MB, 10010 = 32 MB, 10011 = 64 MB, 10100 = 128 MB, 10101 = 256 MB,

          10110 = 512 MB, 10111 = 1 GB, 11000 = 2 GB, 11001 = 4 GB, 11010 = 8 GB, 11011 = 16 GB, 11100 = 32 GB,

          11101 = 64 GB, 11110 = 128 GB, 11111 = 256 GB

         
  • 0 in reply to Pekka Varis
    TI__Mastermind 27050 points
    Grant Edwards said:
    Does the Linux EP driver code support bus enumeration?

    Yes with lspci the EP enumerates as shown in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1213540/am6441-how-to-support-pcie-bus-enumeration-in-endpoint-mode/ . If the vendor id is not programmed (OTP efuses in PCIe boot or register write if booted otherwise), it shows up as Cadence.

  • 0 in reply to Pekka Varis
    Prodigy 50 points

    That demo isn't using the Linux EP driver.

    It's using the MCU+ SDK EP driver in the pci buffer transfer EP demo app.

    In my tests using that demo app, the board is detected as shown above, but enumeration does not work. No BAR is seen by the host, no PCIe memory space is allocated by the host, and the board is not accessible. I also see a lot of garbage when lspci -v is used to read the EVM's config space.