This thread has been locked.
If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.
Hi,experts
Please help analyze a system exception and how to solve it.
We have a board that uses TDA4VM as PCIE RC and FPGA as EP, and uses echo 1 > /sys/bus/pci/rescan to detect ep devices.
And access the BAR address through devmem2, but a system exception occurred.
Thank you.
I refer to this issue:e2e.ti.com/.../4668074
If 0xbf000000 is filtered out, no exception occurs that reads or writes on the BAR of the EP. However, the written data does not take effect, and the EP node cannot receive read or write signals.
SDK version,08.06.01
root@tda4vm-sk:~# devmem2 0x10100000 w 1 /dev/mem opened. Memory mapped at address 0xffffa8f61000. Read at address 0x10100000 (0xffffa8f61000): 0x00000000 Write at address 0x10100000 (0xffffa8f61000): 0x00000001, readback 0x00000001 root@tda4vm-sk:~# devmem2 0x10100000 /dev/mem opened. Memory mapped at address 0xffff9e8ce000. Read at address 0x10100000 (0xffff9e8ce000): 0x00000000
Hi Ljc,
So far, I have not been able to reproduce the issue on my setup.
Attached are my logs for comparison in which I have TDA4VM as the RC and TDA4VL as an EP:
root@j7-evm:~# dmesg | grep -i bar [ 3.003004] pci 0001:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x00ffffff 64bit] (contains BAR0 for 4 VFs) [ 3.075127] pci 0001:01:00.1: VF(n) BAR0 space: [mem 0x00000000-0x00ffffff 64bit] (contains BAR0 for 4 VFs) [ 3.147222] pci 0001:01:00.2: VF(n) BAR0 space: [mem 0x00000000-0x00ffffff 64bit] (contains BAR0 for 4 VFs) [ 3.219315] pci 0001:01:00.3: VF(n) BAR0 space: [mem 0x00000000-0x00ffffff 64bit] (contains BAR0 for 4 VFs) [ 3.358045] pci 0001:00:00.0: BAR 14: assigned [mem 0x18100000-0x1c7fffff] [ 3.365081] pci 0001:01:00.0: BAR 7: assigned [mem 0x18400000-0x193fffff 64bit] [ 3.372559] pci 0001:01:00.1: BAR 7: assigned [mem 0x19400000-0x1a3fffff 64bit] [ 3.380034] pci 0001:01:00.2: BAR 7: assigned [mem 0x1a400000-0x1b3fffff 64bit] [ 3.387510] pci 0001:01:00.3: BAR 7: assigned [mem 0x1b400000-0x1c3fffff 64bit] [ 3.394988] pci 0001:01:00.0: BAR 5: assigned [mem 0x18100000-0x181fffff] [ 3.401929] pci 0001:01:00.1: BAR 5: assigned [mem 0x18200000-0x182fffff] [ 3.408869] pci 0001:01:00.2: BAR 5: assigned [mem 0x18300000-0x183fffff] [ 3.415811] pci 0001:01:00.3: BAR 5: assigned [mem 0x1c400000-0x1c4fffff] [ 3.422752] pci 0001:01:00.4: BAR 5: assigned [mem 0x1c500000-0x1c5fffff] [ 3.429692] pci 0001:01:00.5: BAR 5: assigned [mem 0x1c600000-0x1c6fffff] [ 3.436632] pci 0001:01:00.0: BAR 4: assigned [mem 0x1c700000-0x1c71ffff] [ 3.443572] pci 0001:01:00.1: BAR 4: assigned [mem 0x1c720000-0x1c73ffff] [ 3.450512] pci 0001:01:00.2: BAR 4: assigned [mem 0x1c740000-0x1c75ffff] [ 3.457452] pci 0001:01:00.3: BAR 4: assigned [mem 0x1c760000-0x1c77ffff] [ 3.464392] pci 0001:01:00.4: BAR 4: assigned [mem 0x1c780000-0x1c79ffff] [ 3.471333] pci 0001:01:00.5: BAR 4: assigned [mem 0x1c7a0000-0x1c7bffff] [ 3.478275] pci 0001:01:00.0: BAR 3: assigned [mem 0x1c7c0000-0x1c7c3fff] [ 3.485216] pci 0001:01:00.1: BAR 3: assigned [mem 0x1c7c4000-0x1c7c7fff] [ 3.492156] pci 0001:01:00.2: BAR 3: assigned [mem 0x1c7c8000-0x1c7cbfff] [ 3.499096] pci 0001:01:00.3: BAR 3: assigned [mem 0x1c7cc000-0x1c7cffff] [ 3.506036] pci 0001:01:00.4: BAR 3: assigned [mem 0x1c7d0000-0x1c7d3fff] [ 3.512976] pci 0001:01:00.5: BAR 3: assigned [mem 0x1c7d4000-0x1c7d7fff] [ 3.519916] pci 0001:01:00.0: BAR 2: assigned [mem 0x1c7d8000-0x1c7d83ff] [ 3.526856] pci 0001:01:00.1: BAR 2: assigned [mem 0x1c7d8400-0x1c7d87ff] [ 3.533797] pci 0001:01:00.2: BAR 2: assigned [mem 0x1c7d8800-0x1c7d8bff] [ 3.540739] pci 0001:01:00.3: BAR 2: assigned [mem 0x1c7d8c00-0x1c7d8fff] [ 3.547680] pci 0001:01:00.4: BAR 2: assigned [mem 0x1c7d9000-0x1c7d93ff] [ 3.554620] pci 0001:01:00.5: BAR 2: assigned [mem 0x1c7d9400-0x1c7d97ff] [ 3.561560] pci 0001:01:00.0: BAR 1: assigned [mem 0x1c7d9800-0x1c7d99ff] [ 3.568501] pci 0001:01:00.1: BAR 1: assigned [mem 0x1c7d9a00-0x1c7d9bff] [ 3.575441] pci 0001:01:00.2: BAR 1: assigned [mem 0x1c7d9c00-0x1c7d9dff] [ 3.582381] pci 0001:01:00.3: BAR 1: assigned [mem 0x1c7d9e00-0x1c7d9fff] [ 3.589322] pci 0001:01:00.4: BAR 1: assigned [mem 0x1c7da000-0x1c7da1ff] [ 3.596262] pci 0001:01:00.5: BAR 1: assigned [mem 0x1c7da200-0x1c7da3ff] [ 3.603204] pci 0001:01:00.0: BAR 0: assigned [mem 0x1c7da400-0x1c7da4ff] [ 3.610145] pci 0001:01:00.1: BAR 0: assigned [mem 0x1c7da500-0x1c7da5ff] [ 3.617086] pci 0001:01:00.2: BAR 0: assigned [mem 0x1c7da600-0x1c7da6ff] [ 3.624026] pci 0001:01:00.3: BAR 0: assigned [mem 0x1c7da700-0x1c7da7ff] [ 3.630967] pci 0001:01:00.4: BAR 0: assigned [mem 0x1c7da800-0x1c7da8ff] [ 3.637907] pci 0001:01:00.5: BAR 0: assigned [mem 0x1c7da900-0x1c7da9ff] root@j7-evm:~# root@j7-evm:~# root@j7-evm:~# devmem2 0x18100000 /dev/mem opened. Memory mapped at address 0xffffb2cb0000. Read at address 0x18100000 (0xffffb2cb0000): 0xA0A0A0A0 root@j7-evm:~# devmem2 0x18200000 /dev/mem opened. Memory mapped at address 0xffff88b70000. Read at address 0x18200000 (0xffff88b70000): 0x00000000 root@j7-evm:~# devmem2 0x18200000 w 0x1 /dev/mem opened. Memory mapped at address 0xffffa4020000. Read at address 0x18200000 (0xffffa4020000): 0x00000000 Write at address 0x18200000 (0xffffa4020000): 0x00000001, readback 0x00000001 root@j7-evm:~# devmem2 0x18200000 /dev/mem opened. Memory mapped at address 0xffffab4f0000. Read at address 0x18200000 (0xffffab4f0000): 0x00000001 root@j7-evm:~#
It is strange that a bus error pops up when accessing BAR0. I will be trying out some other experiments to see if I can reproduce these errors on my setup, but it might be that this is an issue on the FPGA-side, in which whatever address these BARs are mapped to are non-writable or protected from writes somehow. The reason being, the E2E you referenced is a fix for ignoring "UR responses (aka, unsupported request)" triggering a system abort when it should not trigger a system abort. If the patch made the error go away, most likely an UR response was being generated from the FPGA-side.
So my suggestion is to see if the FPGA memory location this BAR is mapped to can be accessed or not.
Regards,
Takuma
Hi Takuma san,
Is there any way to delimit whether the problem is TDA4VM or FPGA, such as whether we can set serdes loopback, or whether there are send counts, failure counts, and so on?
Regards,
Ljc
Hi Ljc,
So far, I have tested TDA4VM as RC and two different devices as EP. A TDA4VL as EP, and a SSD as an EP. The logs from when using TDA4VL as EP are in my previous post which shows that I can access BAR.
The following logs are from when I have SSD as an EP:
root@j7-evm:~# dmesg | grep pci [ 1.516364] j721e-pcie 2900000.pcie: host bridge /bus@100000/pcie@2900000 ranges: [ 1.524036] j721e-pcie 2900000.pcie: IO 0x0010001000..0x0010010fff -> 0x0010001000 [ 1.532312] j721e-pcie 2900000.pcie: MEM 0x0010011000..0x0017ffffff -> 0x0010011000 [ 1.540590] j721e-pcie 2900000.pcie: IB MEM 0x0000000000..0xffffffffffff -> 0x0000000000 [ 2.555563] j721e-pcie 2900000.pcie: PCI host bridge to bus 0000:00 [ 2.566425] pci_bus 0000:00: root bus resource [bus 00-0f] [ 2.572030] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] (bus address [0x10001000-0x10010fff]) [ 2.581720] pci_bus 0000:00: root bus resource [mem 0x10011000-0x17ffffff] [ 2.588769] pci 0000:00:00.0: [104c:b00d] type 01 class 0x060400 [ 2.594908] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x4 may corrupt adjacent RW1C bits [ 2.604777] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x4 may corrupt adjacent RW1C bits [ 2.614666] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0xe8 may corrupt adjacent RW1C bits [ 2.624626] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2.634592] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x92 may corrupt adjacent RW1C bits [ 2.644552] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0xb2 may corrupt adjacent RW1C bits [ 2.654536] pci 0000:00:00.0: supports D1 [ 2.658630] pci 0000:00:00.0: PME# supported from D0 D1 D3hot [ 2.664501] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x84 may corrupt adjacent RW1C bits [ 2.676102] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring [ 2.684293] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2.694251] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2.704209] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2.715655] pci_bus 0000:01: busn_res: [bus 01-0f] end is updated to 01 [ 2.722425] pci 0000:00:00.0: PCI bridge to [bus 01] [ 2.727697] pcieport 0000:00:00.0: PME: Signaling with IRQ 56 [ 2.734001] j721e-pcie 2910000.pcie: host bridge /bus@100000/pcie@2910000 ranges: [ 2.741672] j721e-pcie 2910000.pcie: IO 0x0018001000..0x0018010fff -> 0x0018001000 [ 2.749947] j721e-pcie 2910000.pcie: MEM 0x0018011000..0x001fffffff -> 0x0018011000 [ 2.758222] j721e-pcie 2910000.pcie: IB MEM 0x0000000000..0xffffffffffff -> 0x0000000000 [ 2.872971] j721e-pcie 2910000.pcie: Link up [ 2.881828] j721e-pcie 2910000.pcie: PCI host bridge to bus 0001:00 [ 2.888234] pci_bus 0001:00: root bus resource [bus 00-0f] [ 2.893838] pci_bus 0001:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x18001000-0x18010fff]) [ 2.903706] pci_bus 0001:00: root bus resource [mem 0x18011000-0x1fffffff] [ 2.910750] pci 0001:00:00.0: [104c:b00d] type 01 class 0x060400 [ 2.916948] pci 0001:00:00.0: supports D1 [ 2.921042] pci 0001:00:00.0: PME# supported from D0 D1 D3hot [ 2.928522] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring [ 2.936796] pci 0001:01:00.0: [144d:a809] type 00 class 0x010802 [ 2.942967] pci 0001:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit] [ 2.950202] pci 0001:01:00.0: 15.752 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link) [ 2.979055] pci_bus 0001:01: busn_res: [bus 01-0f] end is updated to 01 [ 2.985830] pci 0001:00:00.0: BAR 14: assigned [mem 0x18100000-0x181fffff] [ 2.992858] pci 0001:01:00.0: BAR 0: assigned [mem 0x18100000-0x18103fff 64bit] [ 3.000343] pci 0001:00:00.0: PCI bridge to [bus 01] [ 3.005416] pci 0001:00:00.0: bridge window [mem 0x18100000-0x181fffff] [ 3.012435] pcieport 0001:00:00.0: enabling device (0000 -> 0002) [ 3.018768] pcieport 0001:00:00.0: PME: Signaling with IRQ 59 [ 3.025094] j721e-pcie 2920000.pcie: host bridge /bus@100000/pcie@2920000 ranges: [ 3.032762] j721e-pcie 2920000.pcie: IO 0x4400001000..0x4400010fff -> 0x0000001000 [ 3.041040] j721e-pcie 2920000.pcie: MEM 0x4400011000..0x4407ffffff -> 0x0000011000 [ 3.049316] j721e-pcie 2920000.pcie: IB MEM 0x0000000000..0xffffffffffff -> 0x0000000000 [ 3.180047] j721e-pcie 2920000.pcie: Failed to init phy [ 3.185411] j721e-pcie: probe of 2920000.pcie failed with error -110 [ 8.649332] nvme nvme0: pci function 0001:01:00.0 [ 2762.555316] pci_generic_config_write32: 48 callbacks suppressed [ 2762.555322] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2762.572469] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2762.582211] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2762.591940] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2762.603167] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2762.614479] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2762.624211] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2762.633941] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2868.475275] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2868.486521] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2868.496260] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2868.505992] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2868.517228] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2868.527018] pci 0001:01:00.0: [144d:a809] type 00 class 0x010802 [ 2868.533052] pci 0001:01:00.0: reg 0x10: [mem 0x18100000-0x18103fff 64bit] [ 2868.540438] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0xe8 may corrupt adjacent RW1C bits [ 2868.550252] pci 0001:01:00.0: 15.752 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link) [ 2868.565396] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0xd0 may corrupt adjacent RW1C bits [ 2868.575579] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0xd0 may corrupt adjacent RW1C bits [ 2868.599181] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2868.608952] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 2868.618697] pci 0001:01:00.0: BAR 0: assigned [mem 0x18100000-0x18103fff 64bit] [ 2868.626293] nvme nvme0: pci function 0001:01:00.0 [ 3072.087293] pci_generic_config_write32: 1 callbacks suppressed [ 3072.087300] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 3072.104339] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 3072.114078] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 3072.123807] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 3072.135107] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 3072.144910] pci 0001:01:00.0: [144d:a809] type 00 class 0x010802 [ 3072.150958] pci 0001:01:00.0: reg 0x10: [mem 0x18100000-0x18103fff 64bit] [ 3072.158073] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0xe8 may corrupt adjacent RW1C bits [ 3072.167881] pci 0001:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x2 link at 0001:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link) [ 3072.182928] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0xd0 may corrupt adjacent RW1C bits [ 3072.193104] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0xd0 may corrupt adjacent RW1C bits [ 3072.204502] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 3072.214248] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0x3e may corrupt adjacent RW1C bits [ 3072.223991] pci 0001:01:00.0: BAR 0: assigned [mem 0x18100000-0x18103fff 64bit] [ 3072.231586] nvme nvme0: pci function 0001:01:00.0 root@j7-evm:~# devmem2 0x18100000 /dev/mem opened. Memory mapped at address 0xffff86eb0000. Read at address 0x18100000 (0xffff86eb0000): 0x3C033FFF root@j7-evm:~# devmem2 0x18100000 w 0x1 /dev/mem opened. Memory mapped at address 0xffffac5d0000. Read at address 0x18100000 (0xffffac5d0000): 0x3C033FFF Write at address 0x18100000 (0xffffac5d0000): 0x00000001, readback 0x00000001 root@j7-evm:~# devmem2 0x18100000 /dev/mem opened. Memory mapped at address 0xffff8db10000. Read at address 0x18100000 (0xffff8db10000): 0x3C033FFF root@j7-evm:~#
Here, it has very similar behaviors as described by you, where I can read access the BAR memory address, but writing to it does not cause any changes. Looking through some NVMe SSD specification, these registers are RO (read-only). And with this same SSD, when I write to a RW (read/write) register such as the INTMS register, I am able to modify this register like so:
root@j7-evm:~# devmem2 0x1810000C /dev/mem opened. Memory mapped at address 0xffffb0f60000. Read at address 0x1810000C (0xffffb0f6000c): 0x00000000 root@j7-evm:~# devmem2 0x1810000C w 0x1 /dev/mem opened. Memory mapped at address 0xffffbd870000. Read at address 0x1810000C (0xffffbd87000c): 0x00000000 Write at address 0x1810000C (0xffffbd87000c): 0x00000001, readback 0x00000001 root@j7-evm:~# devmem2 0x1810000C /dev/mem opened. Memory mapped at address 0xffff8f560000. Read at address 0x1810000C (0xffff8f56000c): 0x00000001 root@j7-evm:~#
For reference, this is the NVMe specification documentation I found online in-case you would like to replicate this experiment: https://www.nvmexpress.org/wp-content/uploads/NVM-Express-1_1.pdf
Based on these experiments, I am thinking something similar is happening with the FPGA, where the register you are trying to write to are protected somehow. If the PCIe device can easily be replaced, my recommendation would be to see if you can replace the FPGA with some standard device like a SSD as an EP to connect with TDA4VM to confirm if the TDA4VM is set up correctly. Otherwise, experiment with the FPGA to see if there is a memory address that is known to be able to modify.
Regards,
Takuma
