AM5728: PCIe RC configuration

Amir Azadi

Part Number: AM5728
Other Parts Discussed in Thread: XIO2001,

I have build the system I have described here:

AM5728 PCIESS1 is connected to XIO2001 PCIe to PCI bridge and PCIESS2 is connected to PLX8609 PCIe switch on PCB.

Before I do any changes to PCIe sections in dts files, I have XIO2001 identified as below (SDK 06.00.00.07):

00:00.0 PCI bridge: Texas Instruments Device 8888 (rev 01)

01:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge

Then I changed the device tree files to enable PCIESS2 as below:

In the dra7.dtsi file I have commented the //status = "disabled"; under the axi@1 section.

In the am57xx-beagle-x15-common.dtsi file I've modified related section to enable pcie2_phy and pcie2_rc as follows:

&pcie2_phy {
	status = "okay";
};

&pcie1_rc {
	status = "ok";
	gpios = <&gpio2 2 GPIO_ACTIVE_LOW>; //PERSTn for XIO20001 on our board, was gpio2_8 for EVM
};

&pcie2_rc {
	status = "ok";
	gpios = <&gpio2 0 GPIO_ACTIVE_LOW>; //PERSTn for PLX8609 on our board
};

After above changes, the output of lspci is:

0000:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)
0001:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)

And I could not see XIO2001 after that. I've reversed the changes I've done, but I could not see XIO2001 again and the output is:

00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)

And the output of the dmesg | grep pci when both pcie are enabled is:

root@am57xx-evm:~# dmesg | grep pci
[    0.644428] dra7-pcie 51000000.pcie: Linked as a consumer to phy-4a094000.pciephy.1
[    0.644598] dra7-pcie 51000000.pcie: GPIO lookup for consumer (null)
[    0.644606] dra7-pcie 51000000.pcie: using device tree for GPIO lookup
[    0.644638] of_get_named_gpiod_flags: parsed 'gpios' property of node '/ocp/axi@0/pcie@51000000[0]' - status (0)
[    0.644688] dra7-pcie 51000000.pcie: WA for Errata i870 not applied
[    0.644784] dra7-pcie 51000000.pcie: host bridge /ocp/axi@0/pcie@51000000 ranges:
[    0.644826] dra7-pcie 51000000.pcie:    IO 0x20003000..0x20012fff -> 0x00000000
[    0.644856] dra7-pcie 51000000.pcie:   MEM 0x20013000..0x2fffffff -> 0x20013000
[    1.645148] dra7-pcie 51000000.pcie: Phy link never came up
[    1.645281] dra7-pcie 51000000.pcie: PCI host bridge to bus 0000:00
[    1.645296] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.645309] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    1.645320] pci_bus 0000:00: root bus resource [mem 0x20013000-0x2fffffff]
[    1.645356] pci 0000:00:00.0: [104c:8888] type 01 class 0x060400
[    1.645389] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff 64bit]
[    1.645450] pci 0000:00:00.0: supports D1
[    1.645458] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    1.656451] pci 0000:00:00.0: BAR 0: assigned [mem 0x20100000-0x201fffff 64bit]
[    1.656472] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    1.656811] pcieport 0000:00:00.0: Signaling PME with IRQ 172
[    1.656942] pcieport 0000:00:00.0: AER enabled with IRQ 172
[    1.657222] dra7-pcie 51800000.pcie: Linked as a consumer to phy-4a095000.pciephy.2
[    1.657332] dra7-pcie 51800000.pcie: GPIO lookup for consumer (null)
[    1.657339] dra7-pcie 51800000.pcie: using device tree for GPIO lookup
[    1.657370] of_get_named_gpiod_flags: parsed 'gpios' property of node '/ocp/axi@1/pcie@51800000[0]' - status (0)
[    1.657399] dra7-pcie 51800000.pcie: WA for Errata i870 not applied
[    1.657487] dra7-pcie 51800000.pcie: host bridge /ocp/axi@1/pcie@51800000 ranges:
[    1.657528] dra7-pcie 51800000.pcie:    IO 0x30003000..0x30012fff -> 0x00000000
[    1.657556] dra7-pcie 51800000.pcie:   MEM 0x30013000..0x3fffffff -> 0x30013000
[    2.657849] dra7-pcie 51800000.pcie: Phy link never came up
[    2.657975] dra7-pcie 51800000.pcie: PCI host bridge to bus 0001:00
[    2.657989] pci_bus 0001:00: root bus resource [bus 00-ff]
[    2.658003] pci_bus 0001:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x0000-0xffff])
[    2.658015] pci_bus 0001:00: root bus resource [mem 0x30013000-0x3fffffff]
[    2.658047] pci 0001:00:00.0: [104c:8888] type 01 class 0x060400
[    2.658077] pci 0001:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff 64bit]
[    2.658136] pci 0001:00:00.0: supports D1
[    2.658143] pci 0001:00:00.0: PME# supported from D0 D1 D3hot
[    2.668942] pci 0001:00:00.0: BAR 0: assigned [mem 0x30100000-0x301fffff 64bit]
[    2.668962] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    2.669257] pcieport 0001:00:00.0: Signaling PME with IRQ 174
[    2.669387] pcieport 0001:00:00.0: AER enabled with IRQ 174

My question is at least how to see XIO2001 and PLX8609 switch on lspci ?

I've checked all the related threads and it seems some other guys dealt with it but it is not clear how it could be done.

e2e.ti.com/.../764228
e2e.ti.com/.../758389
e2e.ti.com/.../1799986
e2e.ti.com/.../595948
e2e.ti.com/.../2476292
e2e.ti.com/.../597740

over 5 years ago

0 Bin Liu over 5 years ago

TI__Guru**** 167581 points

Hi Aidin,

I will look into it deeply, but at my first glance,

Aidin Haddadi said:
And I could not see XIO2001 after that. I've reversed the changes I've done, but I could not see XIO2001 again

Do you mean you reverted the DTS changes, but lspci command doesn't show XIO2001 device any more which was there before the DTS changes? If so, it sounds something very basic is wrong here.

0 Amir Azadi over 5 years ago in reply to Bin Liu

Expert 1105 points

Hi Bin

Thanks for the reply.

Bin Liu said:
Do you mean you reverted the DTS changes, but lspci command doesn't show XIO2001 device any more which was there before the DTS changes? If so, it sounds something very basic is wrong here.

Yes, and because I could not figure out what was wrong, I started with a fresh SDK. Now I am able to see XIO2001 again. But No sign of PLX8609 PCIe switch.

------------------------------------------------------------------------------------
root@am57xx-evm:~# lspci
00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)
01:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge
------------------------------------------------------------------------------------

0 Amir Azadi over 5 years ago in reply to Bin Liu

Expert 1105 points

Hi Bin Liu

Any updates?

0 Amir Azadi over 5 years ago in reply to Bin Liu

Expert 1105 points

Hi Bin Liu

Any updates?

0 Bin Liu over 5 years ago in reply to Amir Azadi

TI__Guru**** 167581 points

Hi Aidin,

As I mention before, something is fundamentally wrong that XIO2001 is not enumerated even after reverted your DTS changes. Please check if your dtb was deployed properly.

The running Linux has its device tree data in /sys/firmware/fdt. Copy it out of your system and use the dtc tool in kernel source tree to de-compile it and compare if your DTS change and restore is in place.

<kernel source reee>/scripts/dtc/dtc -I dtb <fdt-file-from-sys-firmware-fir>

0 Amir Azadi over 5 years ago in reply to Bin Liu

Expert 1105 points

Hi Bin Liu

Thanks for reply.

Bin Liu said:
something is fundamentally wrong that XIO2001 is not enumerated even after reverted your DTS changes.

No, you misunderstand it. I said:

Aidin Haddadi said:
I am able to see XIO2001 again. But No sign of PLX8609 PCIe switch.

This the output of lspci :

root@am57xx-evm:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)
0000:01:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge
0001:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)

and lspci -v :

root@am57xx-evm:~# lspci -v
0000:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 175
        Memory at 20100000 (64-bit, non-prefetchable) [size=1M]
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: None
        Memory behind bridge: None
        Prefetchable memory behind bridge: None
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport

0000:01:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge (prog-if 00 [Normal decode])
        Flags: fast devsel
        Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
        I/O behind bridge: None
        Memory behind bridge: None
        Prefetchable memory behind bridge: None
        Capabilities: [40] Subsystem: Device 0000:0000
        Capabilities: [48] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/16 Maskable- 64bit+
        Capabilities: [70] Express PCI-Express to PCI/PCI-X Bridge, MSI 00
        Capabilities: [100] Advanced Error Reporting

0001:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 177
        Memory at 30100000 (64-bit, non-prefetchable) [size=1M]
        Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
        I/O behind bridge: None
        Memory behind bridge: None
        Prefetchable memory behind bridge: None
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport

0 Bin Liu over 5 years ago in reply to Amir Azadi

TI__Guru**** 167581 points

Hi Aidin,

I didn't misunderstand it. I meant the scenario you described in the first post - XIO2001 disappears after you modified DTS to enable PCIe2 for the switch and even after reverted the DTS change.

What you referred is the scenario that you later posted in which you started with a fresh SDK then you can see XIO2001 bot not the switch. It is expected because the fresh SDK doesn't have PCIe2 RC enabled in DTS, right?

0 Amir Azadi over 5 years ago in reply to Bin Liu

Expert 1105 points

Hi Bin

Maybe I have described it in bad way. By fresh SDk I meant I did not apply extra changes rather than the required changes for enabling 2 PCIe RC. I think the output of lspci shows that the AM5728 can detect 2 RC ? Is that correct?

root@am57xx-evm:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)
0000:01:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge
0001:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)

What does these numbers mean?

0000:00:00.0 PCI bridge
0000:01:00.0 PCI bridge
0001:00:00.0 PCI bridge

0 Bin Liu over 5 years ago in reply to Amir Azadi

TI__Guru**** 167581 points

Hi Aidin,

Thanks for clarification. It is clear now.

To understand if it is really PCIe2 which causes the issue, have you checked

- Before enabling PCIe2-rc, plug the PEX8609 switch to PCIe1-rc to see if it gets detected?

- After enabled PCIe2-rc, plug the XIO2001 bridge to PICe2-rc to see if gets detected?

- Instead of XIO2001 and PEX8609, try to plug in PCIe-Ether or PCIe-USB card in all the scenarios above?

The number means PCI domain:bus:device.function

Processors

Processors forum

AM5728: PCIe RC configuration