• State Resolved
  • Locked Locked
  • Replies 8 replies
  • Answers 1 answer
  • Subscribers 101 subscribers
  • Views 1650 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/AM5728: PCI hot plug with the AM5728

Chris Welch66
Expert 1790 points
Part Number: AM5728

Tool/software: Linux

We are building a product based on the AM5728 following the EVM reference design and the TI Linux SDK.

I've been reading that the PCI host on the AM5728 doesn't support PCI hot plug:

http://processors.wiki.ti.com/index.php/Linux_Core_PCIe_EP_User%27s_Guide

This is a problem for us.  Our product involves plugging in custom PCI modules and we don't want our customers to have to restart to detect the module has been plugged in.

1) Can you confirm that PCI hot swap is NOT supported on the AM5728?

2) Is it possible to bring up a new device by reseting or manipulating the PCI host?

Thanks.

  • 0
    TI__Guru 50170 points
    Hi, Chris,

    Your understanding is correct. The hot plug is not supported. The EP needs to be powered up and ready before RC starts. To bring up a new device after RC is running, the kernel driver will need to assert reset to the device, and perform bus scan to enumerates the new device.

    Rex
  • 0 in reply to Rex Chang
    Expert 1790 points

    We are finding this doesn't work.  Specifically we have an FPGA that we are trying to hot plug.

    The FPGA is discovered and enumerated with a reset and bus scan, but its BAR memory is not allocated.

    We get the following errors from the kernel output:

    [ 2092.492384] pci 0000:07:00.0: BAR 3: no space for [mem size 0x00100000]

    [ 2092.492395] pci 0000:07:00.0: BAR 3: failed to assign [mem size 0x00100000]

    [ 2092.492404] pci 0000:07:00.0: BAR 2: no space for [mem size 0x00004000]

    [ 2092.492415] pci 0000:07:00.0: BAR 2: failed to assign [mem size 0x00004000]

    [ 2092.492424] pci 0000:07:00.0: BAR 1: no space for [mem size 0x00002000]

    [ 2092.492434] pci 0000:07:00.0: BAR 1: failed to assign [mem size 0x00002000]

    [ 2092.492443] pci 0000:07:00.0: BAR 0: no space for [mem size 0x00000400]

    [ 2092.492453] pci 0000:07:00.0: BAR 0: failed to assign [mem size 0x00000400]

    We don't have any issues with the FPGA if we do a reboot after it is made available, the BAR memory is setup correctly.

    We have not found anything other than a reboot and having the FPGA available prior to kernel launch works to get the bar memory allocated.

    Any suggestions?

  • 0 in reply to Chris Welch66
    TI__Guru 50170 points
    Hi, Chris,

    We had a similar discussion internally on an PCIe to PCI bridge set up. What we think the process should be as following:
    XIO powered.
    (Wait)
    RC asserts PCIe reset to XIO.
    (Wait)
    RC performs bus scan, enumerates "new" XIO bridge.
    RC writes XIO PCI PRST# bit. Downstream PCI devices are reset.
    (Wait)
    RC (re)performs bus scan, enumerates "new" PCI devices.

    In your case, you don't have the bridge in the middle and should be able to skip the process to enumerate the bridge.
    This is just a suggestion and we don't know if it works. This is out of our support scope.

    Rex
  • 0 in reply to Rex Chang
    Expert 1790 points
    Yes we have a bridge in our configuration. We find we have the BAR problem with any PCI device we attempt to hot plug, not just the FPGA.

    1) For the step "RC asserts PCIe reset to XIO" is this asserting a hardware reset and holding the FPGA in reset until the PCI bridge is enumerated, or is it just a normal reset pulse?

    2) We are performing the bus scan using "echo 1 > /sys/bus/pci/rescan". Is that correct for steps "RC performs bus scan, enumerates "new" XIO bridge" and "RC (re)performs bus scan, enumerates "new" PCI devices"?

    3) The step "RC writes XIO PCI PRST# bit. Downstream PCI devices are reset." I'm assuming this is done by having the bridge assert the PRST signal to the FPGA PCI bus?

    Sorry, I only have a basic understanding of PCI and just want to clarify the specifics.

    Thanks!
  • 0 in reply to Chris Welch66
    TI__Guru 50170 points
    Hi, Chris,

    The reset should be the PERST# and follows the PCI EXPRESS CARD ELECTROMECHANICAL SPECIFICATION.
    The rescan should be the driver performing it. We have not done rescan through sysfs.
    Please see disucssion in e2e thread, e2e.ti.com/.../2153006, for more info.

    Rex
  • 0 in reply to Rex Chang
    Expert 1790 points
    Ok, thanks for the quick response on all this. I'll work with your suggestions and see what we can do.
  • 0 in reply to Chris Welch66
    TI__Mastermind 27720 points
    Just chiming in here to remind everyone that Rex is correct in that PCIe hot plug is not supported.

    Attempting to hot plug PCIe on this (or any Sitara) device will result in damage to the add-in card, the SoC, or both. It's also important that no voltage be applied to any non-failsafe SoC pin when the SoC itself is unpowered. This includes the PCIe SerDes pins, so please ensure that externally powered PCIe add-in cards are not powered prior to the SoC being powered.

    Please refer to the datasheet for details on power sequencing requirements
  • 0 in reply to -DK-
    Expert 1790 points

    We are using a bridge so the SOC is not directly connected to the devices being plugged in.

    All the devices we are plugging in are unpowered.

    We got the mechanism working by doing the following:

    1) Initial startup

    root@map300:~# lspci

    0000:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)

    0000:01:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)

    0000:02:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)

    0000:02:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)

    0000:02:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)

    0000:02:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)

    0000:02:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)

    0001:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)

    1) Insert the devices, no change

    2) Remove the root complex host:

    echo 1 > "/sys/bus/pci/devices/0000:00:00.0/remove"

    root@map300:~# lspci

    0001:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)

    3) Rescan

    echo 1 > /sys/bus/pci/rescan

    and the devices are present:

    root@map300:~# lspci

    0000:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)

    0000:01:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)

    0000:02:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)

    0000:02:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)

    0000:02:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)

    0000:02:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)

    0000:02:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)

    0000:06:00.0 USB controller: Texas Instruments TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller (rev 02) <- test cassette

    0000:07:00.0 Unassigned class [ff00]: Device 04b3:0005 (rev 05) <- FPGA

    0001:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)

    And the BARs are properly allocated:

    root@map300:~# lspci -vv -s 0000:07:00.0

    0000:07:00.0 Unassigned class [ff00]: Device 04b3:0005 (rev 05)

           Subsystem: Device 04be:0006

           Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-

           Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

           Latency: 0, Cache Line Size: 64 bytes

           Interrupt: pin A routed to IRQ 0

           Region 0: Memory at 20406000 (32-bit, non-prefetchable) [size=1K]

           Region 1: Memory at 20404000 (32-bit, non-prefetchable) [size=8K]

           Region 2: Memory at 20400000 (32-bit, non-prefetchable) [size=16K]

           Region 3: Memory at 20300000 (32-bit, non-prefetchable) [size=1M]

           Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+

                   Address: 0000000000000000 Data: 0000

           Capabilities: [78] Power Management version 3

                   Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

                   Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

           Capabilities: [80] Express (v1) Endpoint, MSI 00

    This does have the consequence of disrupting other PCI devices that are present during the RC remove and rescan, however, we have control of when that happens and is acceptable for our use case.