• State Resolved
  • Locked Locked
  • Replies 12 replies
  • Answers 1 answer
  • Subscribers 99 subscribers
  • Views 2817 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/AM5728: PCIe link fails

Jorge Miguel
Expert 1020 points
Part Number: AM5728

Tool/software: Linux

Hello,

We are stuck when trying to get to work our PCIe interface.

We designed a carrier board for de phytec AM5728 SOM (https://phytec.com/products/system-on-modules/phycore/am57x/)

In either TI AM5728 SDK or Phytech SDK, the two lane PCIe interfaces are routed to one PCIe connector. Given our need of two different PCIe cards, our carrier board has two different PCIe connectors, each one single lane.

In theory, routing TX0/RX0 to one connector and TX1/RX1 to another connector should do the work. Besides there is a 3xOutput PCIe Gen2 compliant clock generator which generates an appropriate clock to the PCIe cards and to the AM5728. There are also x2 PWR_GD signals, x2 Wake and 2x presence signals.

The problem that we have is that the link, at start time, is not usually linked. Sometimes it is but generally speaking does not succeded.

 

Here some images of my problem:

-          Yellow colour is the PWR_GD signal from the CPU to the PCIe card.

-          Green is TX0_P (from CPU to one of my PCIe connectors where a PCIe card is inserted). .Oscilloscope measurements are single ended

 

We can see that after PWR_GD goes high, there is some information exchange that continues when the link is established but stops when it fails. Furthermore I see that the electrical levels in this negotiation are not constant which really concerns me.

Example of my card (where each lane is routed to one PCIe connector) when link fails:

 

 

 

Example of my card when link succeed

 

 

 

 

 

Example of evaluation kit (where the two lanes are routed to one PCIe connector)

 

 Here we see that electrical levels are constant and there is no initial burst information. It is continuous.

Do you have any hint about what the problem could be?

Many thanks in advance.

Regards,

  • 0
    TI__Guru**** 393215 points
    Hi,

    Please post Linux version you are using and log files.
  • 0 in reply to Biser Gatchev-XID
    Expert 1020 points
    Hello,

    v04.03 Kernel linux 4.9.69

    logs refering to PCIe:

    when it links:

    ~# dmesg | grep pci
    [ 1.590329] dra7-pcie 51000000.pcie: Linked as a consumer to phy-4a094000.pciephy.1
    [ 1.590554] dra7-pcie 51000000.pcie: GPIO lookup for consumer (null)
    [ 1.590561] dra7-pcie 51000000.pcie: using device tree for GPIO lookup
    [ 1.590592] of_get_named_gpiod_flags: parsed 'gpios' property of node '/ocp/axi@0/pcie@51000000[0]' - status (0)
    [ 1.590719] OF: PCI: host bridge /ocp/axi@0/pcie@51000000 ranges:
    [ 1.792028] dra7-pcie 51000000.pcie: link up
    [ 1.792202] dra7-pcie 51000000.pcie: PCI host bridge to bus 0000:00
    [ 1.792215] pci_bus 0000:00: root bus resource [bus 00-ff]
    [ 1.792225] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
    [ 1.792234] pci_bus 0000:00: root bus resource [mem 0x20013000-0x2fffffff]
    [ 1.792243] pci_bus 0000:00: scanning bus
    [ 1.792272] pci 0000:00:00.0: [104c:8888] type 01 class 0x060400
    [ 1.792294] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
    [ 1.792307] pci 0000:00:00.0: reg 0x14: [mem 0x00000000-0x0000ffff]
    [ 1.792347] pci 0000:00:00.0: calling pci_fixup_ide_bases+0x0/0x5c
    [ 1.792390] pci 0000:00:00.0: supports D1
    [ 1.792397] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
    [ 1.792405] pci 0000:00:00.0: PME# disabled
    [ 1.792633] pci_bus 0000:00: fixups for bus
    [ 1.792656] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0
    [ 1.792766] pci_bus 0000:01: scanning bus
    [ 1.793060] pci 0000:01:00.0: [10b5:8112] type 01 class 0x060400
    [ 1.793868] pci 0000:01:00.0: calling pci_fixup_ide_bases+0x0/0x5c
    [ 1.794884] pci 0000:01:00.0: supports D1
    [ 1.794890] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
    [ 1.794972] pci 0000:01:00.0: PME# disabled
    [ 1.795372] pci_bus 0000:01: fixups for bus
    [ 1.795554] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 0
    [ 1.795561] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
    [ 1.795704] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
    [ 1.796395] pci_bus 0000:02: busn_res: can not insert [bus 02-ff] under [bus 01] (conflicts with (null) [bus 01])
    [ 1.796432] pci_bus 0000:02: scanning bus
    [ 1.796678] pci 0000:02:04.0: [1923:0100] type 00 class 0x028000
    [ 1.796920] pci 0000:02:04.0: reg 0x10: [mem 0x00000000-0x0001ffff]
    [ 1.797909] pci 0000:02:04.0: calling pci_fixup_ide_bases+0x0/0x5c
    [ 1.799050] pci_bus 0000:02: fixups for bus
    [ 1.799179] pci_bus 0000:02: bus scan returning with max=02
    [ 1.799187] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
    [ 1.799195] pci_bus 0000:02: busn_res: can not insert [bus 02] under [bus 01] (conflicts with (null) [bus 01])
    [ 1.799232] pci_bus 0000:02: [bus 02] partially hidden behind bridge 0000:01 [bus 01]
    [ 1.799262] pci_bus 0000:01: bus scan returning with max=02
    [ 1.799269] pci 0000:00:00.0: bridge has subordinate 01 but max busn 02
    [ 1.799281] pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 1
    [ 1.799290] pci_bus 0000:00: bus scan returning with max=01
    [ 1.799377] pci 0000:00:00.0: fixup irq: got 463
    [ 1.799384] pci 0000:00:00.0: assigning IRQ 463
    [ 1.799474] pci 0000:01:00.0: fixup irq: got 463
    [ 1.799480] pci 0000:01:00.0: assigning IRQ 463
    [ 1.799619] pci 0000:02:04.0: fixup irq: got 463
    [ 1.799625] pci 0000:02:04.0: assigning IRQ 463
    [ 1.799918] pci 0000:00:00.0: BAR 0: assigned [mem 0x20100000-0x201fffff]
    [ 1.799932] pci 0000:00:00.0: BAR 8: assigned [mem 0x20200000-0x202fffff]
    [ 1.799943] pci 0000:00:00.0: BAR 1: assigned [mem 0x20020000-0x2002ffff]
    [ 1.799957] pci 0000:01:00.0: BAR 8: assigned [mem 0x20200000-0x202fffff]
    [ 1.799969] pci 0000:02:04.0: BAR 0: assigned [mem 0x20200000-0x2021ffff]
    [ 1.800053] pci 0000:01:00.0: PCI bridge to [bus 02]
    [ 1.800142] pci 0000:01:00.0: bridge window [mem 0x20200000-0x202fffff]
    [ 1.800323] pci 0000:00:00.0: PCI bridge to [bus 01]
    [ 1.800334] pci 0000:00:00.0: bridge window [mem 0x20200000-0x202fffff]
    [ 1.800403] pcieport 0000:00:00.0: calling quirk_limit_mrrs+0x0/0x88
    [ 1.800568] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
    [ 1.800578] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
    [ 1.800585] pci 0000:02:04.0: Signaling PME through PCIe PME interrupt
    [ 1.800595] pcie_pme 0000:00:00.0:pcie001: service driver pcie_pme loaded
    [ 1.800722] aer 0000:00:00.0:pcie002: service driver aer loaded
    [ 1.801033] dra7-pcie 51800000.pcie_rc: Linked as a consumer to phy-4a095000.pciephy.2
    [ 1.801197] dra7-pcie 51800000.pcie_rc: GPIO lookup for consumer (null)
    [ 1.801204] dra7-pcie 51800000.pcie_rc: using device tree for GPIO lookup
    [ 1.801231] of_get_named_gpiod_flags: parsed 'gpios' property of node '/ocp/axi@1/pcie_rc@51800000[0]' - status (0)
    [ 1.801329] OF: PCI: host bridge /ocp/axi@1/pcie_rc@51800000 ranges:
    [ 2.802758] dra7-pcie 51800000.pcie_rc: phy link never came up
    [ 2.802912] dra7-pcie 51800000.pcie_rc: PCI host bridge to bus 0001:00
    [ 2.802924] pci_bus 0001:00: root bus resource [bus 00-ff]
    [ 2.802936] pci_bus 0001:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff])
    [ 2.802945] pci_bus 0001:00: root bus resource [mem 0x30013000-0x3fffffff]
    [ 2.802954] pci_bus 0001:00: scanning bus
    [ 2.802980] pci 0001:00:00.0: [104c:8888] type 01 class 0x060400
    [ 2.803000] pci 0001:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
    [ 2.803012] pci 0001:00:00.0: reg 0x14: [mem 0x00000000-0x0000ffff]
    [ 2.803046] pci 0001:00:00.0: calling pci_fixup_ide_bases+0x0/0x5c
    [ 2.803086] pci 0001:00:00.0: supports D1
    [ 2.803092] pci 0001:00:00.0: PME# supported from D0 D1 D3hot
    [ 2.803099] pci 0001:00:00.0: PME# disabled
    [ 2.803315] pci_bus 0001:00: fixups for bus
    [ 2.803337] pci 0001:00:00.0: scanning [bus 01-01] behind bridge, pass 0
    [ 2.803445] pci_bus 0001:01: scanning bus
    [ 2.803453] pci_bus 0001:01: fixups for bus
    [ 2.803467] pci_bus 0001:01: bus scan returning with max=01
    [ 2.803477] pci 0001:00:00.0: scanning [bus 01-01] behind bridge, pass 1
    [ 2.803487] pci_bus 0001:00: bus scan returning with max=01
    [ 2.803528] pcieport 0000:00:00.0: fixup irq: got 463
    [ 2.803534] pcieport 0000:00:00.0: assigning IRQ 463
    [ 2.803625] pci 0000:01:00.0: fixup irq: got 463
    [ 2.803631] pci 0000:01:00.0: assigning IRQ 463
    [ 2.803747] pci 0000:02:04.0: fixup irq: got 463
    [ 2.803753] pci 0000:02:04.0: assigning IRQ 463
    [ 2.803851] pci 0001:00:00.0: fixup irq: got 496
    [ 2.803857] pci 0001:00:00.0: assigning IRQ 496
    [ 2.803880] pci 0001:00:00.0: BAR 0: assigned [mem 0x30100000-0x301fffff]
    [ 2.803893] pci 0001:00:00.0: BAR 1: assigned [mem 0x30020000-0x3002ffff]
    [ 2.803905] pci 0001:00:00.0: PCI bridge to [bus 01]
    [ 2.803967] pcieport 0001:00:00.0: calling quirk_limit_mrrs+0x0/0x88
    [ 2.804132] pcieport 0001:00:00.0: Signaling PME through PCIe PME interrupt
    [ 2.804144] pcie_pme 0001:00:00.0:pcie001: service driver pcie_pme loaded
    [ 2.804270] aer 0001:00:00.0:pcie002: service driver aer loaded



    when it does not link:

    ~# dmesg | grep pci
    [ 1.590522] dra7-pcie 51000000.pcie: Linked as a consumer to phy-4a094000.pciephy.1
    [ 1.590748] dra7-pcie 51000000.pcie: GPIO lookup for consumer (null)
    [ 1.590755] dra7-pcie 51000000.pcie: using device tree for GPIO lookup
    [ 1.590786] of_get_named_gpiod_flags: parsed 'gpios' property of node '/ocp/axi@0/pcie@51000000[0]' - status (0)
    [ 1.590914] OF: PCI: host bridge /ocp/axi@0/pcie@51000000 ranges:
    [ 2.592402] dra7-pcie 51000000.pcie: phy link never came up
    [ 2.592572] dra7-pcie 51000000.pcie: PCI host bridge to bus 0000:00
    [ 2.592585] pci_bus 0000:00: root bus resource [bus 00-ff]
    [ 2.592595] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
    [ 2.592605] pci_bus 0000:00: root bus resource [mem 0x20013000-0x2fffffff]
    [ 2.592614] pci_bus 0000:00: scanning bus
    [ 2.592643] pci 0000:00:00.0: [104c:8888] type 01 class 0x000000
    [ 2.592655] pci 0000:00:00.0: ignoring class 0x000000 (doesn't match header type 01)
    [ 2.592687] pci 0000:00:00.0: calling pci_fixup_ide_bases+0x0/0x5c
    [ 2.592735] pci 0000:00:00.0: supports D1
    [ 2.592741] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
    [ 2.592749] pci 0000:00:00.0: PME# disabled
    [ 2.592980] pci_bus 0000:00: fixups for bus
    [ 2.593001] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 0
    [ 2.593008] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
    [ 2.593023] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
    [ 2.593136] pci_bus 0000:01: scanning bus
    [ 2.593144] pci_bus 0000:01: fixups for bus
    [ 2.593158] pci_bus 0000:01: bus scan returning with max=01
    [ 2.593166] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
    [ 2.593174] pci_bus 0000:00: bus scan returning with max=01
    [ 2.593263] pci 0000:00:00.0: fixup irq: got 463
    [ 2.593270] pci 0000:00:00.0: assigning IRQ 463
    [ 2.593284] pci 0000:00:00.0: not setting up bridge for bus 0000:01
    [ 2.593500] dra7-pcie 51800000.pcie_rc: Linked as a consumer to phy-4a095000.pciephy.2
    [ 2.593662] dra7-pcie 51800000.pcie_rc: GPIO lookup for consumer (null)
    [ 2.593669] dra7-pcie 51800000.pcie_rc: using device tree for GPIO lookup
    [ 2.593697] of_get_named_gpiod_flags: parsed 'gpios' property of node '/ocp/axi@1/pcie_rc@51800000[0]' - status (0)
    [ 2.593797] OF: PCI: host bridge /ocp/axi@1/pcie_rc@51800000 ranges:
    [ 3.595220] dra7-pcie 51800000.pcie_rc: phy link never came up
    [ 3.595455] dra7-pcie 51800000.pcie_rc: PCI host bridge to bus 0001:00
    [ 3.595467] pci_bus 0001:00: root bus resource [bus 00-ff]
    [ 3.595482] pci_bus 0001:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff])
    [ 3.595492] pci_bus 0001:00: root bus resource [mem 0x30013000-0x3fffffff]
    [ 3.595500] pci_bus 0001:00: scanning bus
    [ 3.595527] pci 0001:00:00.0: [104c:8888] type 01 class 0x060400
    [ 3.595548] pci 0001:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
    [ 3.595560] pci 0001:00:00.0: reg 0x14: [mem 0x00000000-0x0000ffff]
    [ 3.595594] pci 0001:00:00.0: calling pci_fixup_ide_bases+0x0/0x5c
    [ 3.595634] pci 0001:00:00.0: supports D1
    [ 3.595640] pci 0001:00:00.0: PME# supported from D0 D1 D3hot
    [ 3.595648] pci 0001:00:00.0: PME# disabled
    [ 3.595863] pci_bus 0001:00: fixups for bus
    [ 3.595884] pci 0001:00:00.0: scanning [bus 01-01] behind bridge, pass 0
    [ 3.595992] pci_bus 0001:01: scanning bus
    [ 3.596001] pci_bus 0001:01: fixups for bus
    [ 3.596014] pci_bus 0001:01: bus scan returning with max=01
    [ 3.596024] pci 0001:00:00.0: scanning [bus 01-01] behind bridge, pass 1
    [ 3.596033] pci_bus 0001:00: bus scan returning with max=01
    [ 3.596074] pci 0000:00:00.0: fixup irq: got 463
    [ 3.596080] pci 0000:00:00.0: assigning IRQ 463
    [ 3.596157] pci 0001:00:00.0: fixup irq: got 496
    [ 3.596163] pci 0001:00:00.0: assigning IRQ 496
    [ 3.596187] pci 0001:00:00.0: BAR 0: assigned [mem 0x30100000-0x301fffff]
    [ 3.596201] pci 0001:00:00.0: BAR 1: assigned [mem 0x30020000-0x3002ffff]
    [ 3.596214] pci 0001:00:00.0: PCI bridge to [bus 01]
    [ 3.596283] pcieport 0001:00:00.0: calling quirk_limit_mrrs+0x0/0x88
    [ 3.596451] pcieport 0001:00:00.0: Signaling PME through PCIe PME interrupt
    [ 3.596464] pcie_pme 0001:00:00.0:pcie001: service driver pcie_pme loaded
    [ 3.596591] aer 0001:00:00.0:pcie002: service driver aer loaded

    Thank you in advance,
    Regards
  • 0 in reply to Jorge Miguel
    TI__Mastermind 27890 points

    Hello Jorge,
    To start, please review SPRAAR7 to ensure that all layout guidelines were correctly implemented. Please note that there is a device-specific appendix that includes specific requirements for AM572x devices.

    A great majority of the PCIe issues we see can be traced to layout issues.

  • 0 in reply to -DK-
    TI__Guru 50170 points
    Hi, Jorge,

    Just checking to see if you made any progress on this issue and found any layout issues as suggested by DK. You may want to try different board or a TI EVM to see if the issue persists.

    If you don't have any more questions, I'll close this thread. If the issue is resolved, please click "resolved" button. Thanks!

    Rex
  • 0 in reply to Rex Chang
    Expert 1020 points

    Dear Rex,

    I am still struggeling with the same issue. My layout complies with SPRAAR7 layout guidelines.

    Can anyboady help with the logs in case some clue of the problem can be found?

    Many thanks for your support.

  • 0 in reply to Jorge Miguel
    TI__Mastermind 27890 points
    Hello Jorge,
    The traces for your board, both pass and fail, show what appears to be excessive noise on the interface. Note that this noise does not appear on the TI EVM trace you provided. This noise looks to be somewhat periodic in nature so my first suspicion is that it is cross-talk induced.

    My recommendation is to re-review your layout looking for instances where the keep-out and/or impedance matching rules defined in SPRAAR7 are not followed. If this re-review doesn't yield any likely noise sources or impedance deviations, please review the SoC power supply/grounding requirements to ensure that this noise is not being introduced as part of the PCB power delivery network.
  • 0 in reply to -DK-
    TI__Guru 50170 points
    Hi, Jorge,

    It seems the issue is related to the board manufacturing, and none TI related issue. I'll close this thread. If your issue is resolved after hardware rework, please click "Resolved". If you have other questions/issues, please submit a new thread. Thanks!

    Rex
  • 0 in reply to Rex Chang
    Expert 1020 points
    Hello,

    The PCIe link started to link when I configured my clock generator to a "no spread spectrum". It was the little spread spectrum configuration what was causing the link to fail.

    Thank you,
    Jorge
  • 0 in reply to Jorge Miguel
    Prodigy 200 points

    Hi Jorge,

    We are facing the same issue with 4.9.69 kernel. Can you please provide which register settings need to be done for No spread spectrum ?

    Thanks,

    Nirav

  • 0 in reply to Nirav Patel
    Expert 1020 points

    Hi Nirav,

    That is not an AM5728 register. It is a hardware configuration and it is related to the reference clock that is externally generated and connected to the AM5728 and PCIe external cards.

    Thus, first thing you have to to is checking wich PCIe clock scheme implements your harware:

    - Common Refclk Architecture
    - Separate Refclk Architecture
    - Data Clocked Refclk Architecture

    In my case, I designed my board following a "Common Refclk Architecture" (I do not know if AM5728 supports any other type) where an external PCIe clock synthesizer (IDT5V41066 to be precise) generates 4 clock outputs. One of them is connected to the AM5728 PCIe clock lines and two other outputs are connected to two different x1 PCIe cards. This way I have two different x1 PCIe Cards connected to the AM 5728 and each of the three devices has a common clock generated by a single IC.

    Well, this IC, and most of them has configuration options. One of them is to control the spread of the generated clock outputs (in my case through external resistors). That is what I changed, the configuration of my PCIe clock generator IC.

    Hope it helps

    Regards

  • 0 in reply to Jorge Miguel
    Prodigy 200 points

    Hi Jorge,

    Thanks for your quick reply. We are using Pericom clock generator IC and having common clock architecture with "No-spread" enabled via S[2:0]=011.

    Based on our log looks like PCIe-2 is not configured properly. Below is output log.

    [ 0.880843] OF: PCI: host bridge /ocp/axi@1/pcie@51800000 ranges:

    [ 0.880853] OF: PCI: No bus range found for /ocp/axi@1/pcie@51800000, using [bus 00-ff]

    [ 0.880885] OF: PCI: IO 0x30003000..0x30012fff -> 0x00000000

    [ 0.880909] OF: PCI: MEM 0x30013000..0x3fffffff -> 0x30013000

    [ 1.882257] dra7-pcie 51800000.pcie: phy link never came up

    [ 1.882414] dra7-pcie 51800000.pcie: PCI host bridge to bus 0001:00

    [ 1.882426] pci_bus 0001:00: root bus resource [bus 00-ff]

    [ 1.882439] pci_bus 0001:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff])

    [ 1.882449] pci_bus 0001:00: root bus resource [mem 0x30013000-0x3fffffff]

    [ 1.882491] pci 0001:00:00.0: ignoring class 0x000000 (doesn't match header type 01)

    [ 1.882765] PCI: bus0: Fast back to back transfers disabled

    [ 1.882778] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring

    [ 1.882897] PCI: bus1: Fast back to back transfers enabled

    [ 1.883064] pci 0001:00:00.0: not setting up bridge for bus 0001:01

    I am attaching my DTS file for your reference. Kindly review and let us know if any configuration is missing.

    Thanks,https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/dra7.7z

    Nirav

  • 0 in reply to Nirav Patel
    Expert 1020 points
    Hello Norav,

    I am from the HW part so I cannot assist with your device tree file.

    One more consideration. If you are using two different PCIe cards, each card needs a different power good signal. At the beginning I used a common one but two differents are required, otherwise only one card links up and the other file (similar to your problem)

    Regards,