Tool/software:
Hi there!
We noticed the intermittent issues only happen with the reboot command, not with hard reset.
We got help from Ben Lu in a previous thread.
> PCIe controller should work in RC mode, but not in EP mode.
We applied this feedback and disabled EP and did other tweaks (showing relevant DTS parts below).
Now the link is detected more often (so far 100% with the new changes, tests are still running).
Sometimes I'm getting this panic that seems to go away with pcie_aspm=off with the testing I'm doing.
Any concerns about keeping this kernel cmdline setting (pcie_aspm=off)?
[ 2.164649] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
[ 2.208300] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
[ 2.223331] davinci_mdio 8000f00.mdio: phy[1]: device 8000f00.mdio:01, driver TI DP83867
[ 2.239610] j721e-pcie-host f102000.pcie: host bridge /bus@f4000/pcie@f102000 ranges:
[ 2.247628] j721e-pcie-host f102000.pcie: IO 0x0068001000..0x0068010fff -> 0x0068001000
[ 2.256190] j721e-pcie-host f102000.pcie: MEM 0x0068011000..0x006fffffff -> 0x0068011000
[ 2.264750] j721e-pcie-host f102000.pcie: IB MEM 0x0000000000..0x0fffffffff -> 0x0000000000
[ 3.048439] sdhci-am654 fa00000.mmc: Power on failed
[ 3.084137] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit
[ 3.384501] j721e-pcie-host f102000.pcie: Link up
[ 3.389431] j721e-pcie-host f102000.pcie: PCI host bridge to bus 0000:00
[ 3.396136] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 3.401625] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] (bus address [0x68001000-0x68010fff])
[ 3.411110] pci_bus 0000:00: root bus resource [mem 0x68011000-0x6fffffff]
[ 3.418024] pci 0000:00:00.0: [104c:b010] type 01 class 0x060400
[ 3.424042] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0xfffffffff 64bit pref]
[ 3.431445] pci 0000:00:00.0: supports D1
[ 3.435455] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[ 3.444216] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 3.505999] Internal error: synchronous external abort: 0000000096000010 [#1] PREEMPT SMP
[ 3.506004] j721e-pcie-host f102000.pcie: LINK DOWN!
[ 3.519125] Modules linked in:
[ 3.522179] CPU: 1 PID: 9 Comm: kworker/u4:0 Not tainted 6.1.69-pwk-0405-bsp-yocto-ampliphy-am64x-pd23.2.1 #1
[ 3.532076] Hardware name: PHYTEC phyBOARD-Electra-AM64x RDK (DT)
[ 3.538158] Workqueue: events_unbound deferred_probe_work_func
[ 3.543995] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 3.550944] pc : pci_generic_config_read+0x38/0xe0
[ 3.555733] lr : pci_generic_config_read+0x24/0xe0
[ 3.560517] sp : ffff8000095736d0
[ 3.563819] x29: ffff8000095736d0 x28: ffff0000018c2800 x27: ffff8000093d6980
[ 3.570948] x26: ffff0000018c30e0 x25: 0000000000000001 x24: ffff8000094b53e0
[ 3.578076] x23: 0000000000000000 x22: ffff80000957375c x21: ffff0000018c3000
[ 3.585203] x20: 0000000000000004 x19: ffff8000095736f4 x18: ffffffffffffffff
[ 3.592329] x17: 6572202c295d3030 x16: 2d3030207375625b x15: ffff0000019aa48d
[ 3.599456] x14: ffffffffffffffff x13: ffff0000019aa48b x12: ffff0000018c2818
[ 3.600012] j721e-pcie-host f102000.pcie: LINK DOWN!
[ 3.606583] x11: ffff0000018c2818 x10: 000000000000002e x9 : ffff8000095737c0
[ 3.618655] x8 : 000000000000002e x7 : ffff8000095737c0 x6 : 0000000000000030
[ 3.625781] x5 : ffff800008590cd0 x4 : 0000000000000001 x3 : 0000000000000001
[ 3.632908] x2 : ffff80000bc00008 x1 : 000000000080000a x0 : ffff8000097d5100
[ 3.640035] Call trace:
[ 3.642473] pci_generic_config_read+0x38/0xe0
[ 3.646911] pci_bus_read_config_dword+0x80/0xe4
[ 3.650213] j721e-pcie-host f102000.pcie: LINK DOWN!
[ 3.651520] pci_read_config_dword+0x30/0x50
[ 3.660720] pci_cfg_space_size_ext+0x38/0xcc
[ 3.665067] pci_cfg_space_size+0x94/0xe0
[ 3.669069] pci_setup_device+0x118/0x734
[ 3.673070] pci_scan_single_device+0xcc/0x110
[ 3.677504] pci_scan_slot+0x6c/0x1d0
[ 3.681157] pci_scan_child_bus_extend+0x44/0x23c
[ 3.685851] pci_scan_bridge_extend+0x2e4/0x5d0
[ 3.690372] pci_scan_child_bus_extend+0x180/0x23c
[ 3.695153] pci_scan_root_bus_bridge+0x64/0xdc
[ 3.699674] pci_host_probe+0x18/0xc4
[ 3.703328] cdns_pcie_host_setup+0x5b0/0x990
[ 3.707678] j721e_pcie_probe+0x45c/0x590
[ 3.711680] platform_probe+0x68/0xe0
[ 3.715336] really_probe+0xbc/0x2dc
[ 3.718903] __driver_probe_device+0x78/0x114
[ 3.723252] driver_probe_device+0xd8/0x15c
[ 3.727427] __device_attach_driver+0xb8/0x134
[ 3.731861] bus_for_each_drv+0x7c/0xdc
[ 3.735689] __device_attach+0xac/0x1d0
[ 3.739516] device_initial_probe+0x14/0x20
[ 3.743691] bus_probe_device+0x9c/0xa4
[ 3.747518] deferred_probe_work_func+0x88/0xc0
[ 3.752039] process_one_work+0x1d0/0x320
[ 3.756043] worker_thread+0x2c8/0x444
[ 3.759783] kthread+0x10c/0x110
[ 3.763008] ret_from_fork+0x10/0x20
[ 3.766584] Code: 7100069f 54000340 71000a9f 54000180 (b9400001)
[ 3.772664] ---[ end trace 0000000000000000 ]---
[ 3.777269] note: kworker/u4:0[9] exited with irqs disabled
[ 3.782872] note: kworker/u4:0[9] exited with preempt_count 1
[ 4.527624] j721e-pcie-host f102000.pcie: LINK DOWN!
[ 29.779711] j721e-pcie-host f102000.pcie: LINK DOWN!
[ 29.951661] j721e-pcie-host f102000.pcie: LINK DOWN!
And here is the dmesg when things go well (always so far with pcie_aspm=off).
[ 2.231493] j721e-pcie-host f102000.pcie: host bridge /bus@f4000/pcie@f102000 ranges:
[ 2.239495] j721e-pcie-host f102000.pcie: IO 0x0068001000..0x0068010fff -> 0x0068001000
[ 2.248046] j721e-pcie-host f102000.pcie: MEM 0x0068011000..0x006fffffff -> 0x0068011000
[ 2.256607] j721e-pcie-host f102000.pcie: IB MEM 0x0000000000..0x0fffffffff -> 0x0000000000
[ 3.042369] sdhci-am654 fa00000.mmc: Power on failed
[ 3.077980] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit
[ 3.376416] j721e-pcie-host f102000.pcie: Link up
[ 3.381375] j721e-pcie-host f102000.pcie: PCI host bridge to bus 0000:00
[ 3.388087] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 3.393576] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] (bus address [0x68001000-0x68010fff])
[ 3.403047] pci_bus 0000:00: root bus resource [mem 0x68011000-0x6fffffff]
[ 3.409973] pci 0000:00:00.0: [104c:b010] type 01 class 0x060400
[ 3.415994] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0xfffffffff 64bit pref]
[ 3.423384] pci 0000:00:00.0: supports D1
[ 3.427394] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[ 3.436151] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 3.444521] pci 0000:01:00.0: [1e95:100e] type 00 class 0x010802
[ 3.450597] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[ 3.457439] pci 0000:01:00.0: reg 0x24: [mem 0x00000000-0x00001fff]
[ 3.464043] pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[ 3.479311] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 3.485972] pci 0000:00:00.0: BAR 0: no space for [mem size 0x1000000000 64bit pref]
[ 3.493711] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x1000000000 64bit pref]
[ 3.501809] pci 0000:00:00.0: BAR 14: assigned [mem 0x68100000-0x681fffff]
[ 3.508698] pci 0000:01:00.0: BAR 0: assigned [mem 0x68100000-0x68103fff 64bit]
[ 3.516025] pci 0000:01:00.0: BAR 5: assigned [mem 0x68104000-0x68105fff]
[ 3.522817] pci 0000:00:00.0: PCI bridge to [bus 01]
Here are the relevant parts of the DTS. Disable EP, and also separate pcie from USB, using default PHY for USB2 only. Feedback welcome. We're using 1 lane for pcie.
&serdes0 {
assigned-clock-parents = <&pcie_refclk0>, <&pcie_refclk0>, <&pcie_refclk0>;
serdes0_pcie_usb_link: phy@0 {
reg = <0>;
cdns,num-lanes = <1>;
#phy-cells = <0>;
cdns,phy-type = <PHY_TYPE_PCIE>;
resets = <&serdes_wiz0 1>;
};
};
&serdes0_pcie_usb_link {
cdns,phy-type = <PHY_TYPE_PCIE>;
};
&serdes_refclk {
clock-frequency = <100000000>;
};
&serdes_ln_ctrl {
idle-states = <AM64_SERDES0_LANE0_PCIE0>;
};
&pcie0_ep {
status = "disabled";
};
&pcie0_rc {
status = "okay";
pinctrl-names = "default";
pinctrl-0 = <&pcie0_pins_default>;
reset-gpios = <&main_gpio0 37 GPIO_ACTIVE_HIGH>;
phys = <&serdes0_pcie_usb_link>;
phy-names = "pcie-phy";
num-lanes = <1>;
max-link-speed = <2>; /* Gen2 = 5.0 GT/s */
};
&usbss0 {
ti,usb2-only;
};
&usb0 {
status = "okay";
dr_mode = "host";
maximum-speed = "high-speed";
pinctrl-names = "default";
pinctrl-0 = <&main_usb0_pins_default>;
// USB2 only, no need to specify more as the internal PHY will be used.
};
