DRA829V: ITS queue timeout occurs when enabling all 4 PCIe interfaces

Hi Romain,

We have had another customer developer had a similar issue, but no resolution for the error was found. Internally at TI, we have failed to reproduce this issue on our boards.

In any case, this commit somewhat explains behavior of the indexes and what the error log means, so it would be a good read: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/drivers/irqchip/irq-gic-v3-its.c?h=ti-linux-6.12.y&id=a050fa5476d418fc16b25abe168b3d38ba11e13c. It looks like to_idx is progressing, but linear_idx is stuck (aka, no delta in rd_idx). Try printing out the indexes to see if we can see what leads to the linear_idx not progressing.

Romain Naour said:

No issues so far but ITS queue timeout occurs when rebooting

Otherwise, does issue only occur when manually trying to do a soft reboot using the reboot command? And without reboot command, PCIe is working as intended?

Regards,

Takuma

Hello Takuma,

Thanks for the info about ITS logs.

Indeed, ITS queue timeout only occur after using reboot command.
PCIe interfaces seems to work so far but we are still in the board bring up phase, all 4 PCIe interfaces are not yet used at the same time.

I'll try to debug linear_idx probably next week.

Best regards,
Romain

Hello Takuma,

Here a small update about this issue to keep this thread open.

I did a test with latest ti-linux 6.12.y, but I noticed the same behavior during reboot.

I noticed something in dmesg:

[    0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19
[    0.000000] ITS@0x0000000001820000: allocated 524288 Devices @880800000 (flat, esz 8, psz 64K, shr 0)

git.kernel.org/.../

"irqchip: gicv3-its: Add limitation to page order

When required size of Device Table is out of the page allocator's
capability, the whole ITS will fail in probing. This actually is
not the hardware's problem and is mainly a limitation of the kernel
page allocator. This patch will keep ITS going on to the next
initializaion stage with an explicit warning."

I'm not sure this "Device Table too large" log can be related to this issue.

Best regards,
Romain

Hi Romain,

Thanks for the update. Not sure if that ITS log is the issue, or part of the issue. Those logs will get printed out always, regardless of whether there are issues with PCIe or not. But perhaps the ITS cmd queue only gets printed out in some corner case that is hit due to this warning issue.

In any case, I did an experiment to see if number of PCIe devices is causing issue by using a PCIe switch with the kernel panic workaround. I was not able to reproduce the ITS cmd queue messages when rebooting:

Arago 2025.01 j721e-evm ttyS2

j721e-evm login: root
[   31.233207] audit: type=1006 audit(1741187631.852:15): pid=1243 uid=0 subj=kernel old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=2 res=1
[   31.247169] audit: type=1300 audit(1741187631.852:15): arch=c00000b7 syscall=64 success=yes exit=1 a0=8 a1=ffffc793d658 a2=1 a3=1 items=0 ppid=1 pid=1243 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=2 comm="(systemd)" exe="/usr/lib/systemd/systemd-executor" subj=kernel key=(null)
[   31.274858] audit: type=1327 audit(1741187631.852:15): proctitle="(systemd)"
[   31.291322] audit: type=1334 audit(1741187631.912:16): prog-id=18 op=LOAD
[   31.298425] audit: type=1300 audit(1741187631.912:16): arch=c00000b7 syscall=280 success=yes exit=8 a0=5 a1=ffffd1039358 a2=90 a3=0 items=0 ppid=1 pid=1243 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=2 comm="systemd" exe="/usr/lib/systemd/systemd" subj=kernel key=(null)
[   31.325258] audit: type=1327 audit(1741187631.912:16): proctitle="(systemd)"
[   31.332373] audit: type=1334 audit(1741187631.912:17): prog-id=18 op=UNLOAD
root@j721e-evm:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Device b00d
0001:00:00.0 PCI bridge: Texas Instruments Device b00d
0001:01:00.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:02:00.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:02:02.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:02:03.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:02:04.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:02:06.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:02:07.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:02:08.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:02:0a.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:02:0b.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:02:0c.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:02:0e.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:02:0f.0 PCI bridge: ASMedia Technology Inc. Device 812b (rev 01)
0001:03:00.0 Non-Volatile memory controller: Phison Electronics Corporation PS5013-E13 PCIe3 NVMe Controller (DRAM-less) (rev 01)
0001:06:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. NV2 NVMe SSD TC2200 (DRAM-less)
0001:09:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
0001:0c:00.0 Network controller: Intel Corporation Wi-Fi 5(802.11ac) Wireless-AC 9x6x [Thunder Peak] (rev 29)
0002:00:00.0 PCI bridge: Texas Instruments Device b00d
root@j721e-evm:~# reboot
root@j721e-evm:~#          Stopping topping Session c2 of User root...
[  OK  ] Removed slice Slice /system/modprobe.
[  OK  ] Stopped target Graphical Interface.
[  OK  ] Stopped target Multi-User System.
[  OK  ] Stopped target Login Prompts.
[  OK  ] Stopped target Host and Network Name Lookups.
[  OK  ] Stopped target RPC Port Mapper.
[  OK  ] Stopped target Sound Card.
[  OK  ] Stopped target Timer Units.
[  OK  ] Stopped Daily rotation of log files.
[  OK  ] Stopped Daily Cleanup of Temporary Directories.
[  OK  ] Stopped target System Time Set.
[  OK  ] Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
         Stopping Save/Restore Sound Card State...
         Stopping Job spooling tools...
         Stopping Avahi mDNS/DNS-SD Stack...
         Stopping containerd container runtime...
         Stopping Periodic Command Scheduler...
         Stopping emptty display manager...
         Stopping Getty on tty1...
         Stopping irqbalance daemon...
         Stopping Reboot and dump vmcore via kexec...
         Stopping LLDP daemon...
         Stopping Netperf Benchmark Server...
         Stopping Telephony service...
[  OK  ] Stopped Expand the rootfs partition to full size of the boot device..
         Stopping Serial Getty on ttyS2...
         Stopping Simple Network Management Protocol (SNMP) Daemon....
         Stopping strongSwan IPsec IKEv1/IKEv2 daemon using ipsec.conf...
         Stopping Load/Save OS Random Seed...
         Stopping TEE Supplicant on teepriv0...
         Stopping Telnet Server...
[  OK  ] Stopped Job spooling tools.
[  OK  ] Stopped Avahi mDNS/DNS-SD Stack.
[  OK  ] Stopped Periodic Command Scheduler.
[  OK  ] Stopped irqbalance daemon.
[  OK  ] Stopped Telephony service.
[  OK  ] Stopped strongSwan IPsec IKEv1/IKEv2 daemon using ipsec.conf.
[  OK  ] Stopped TEE Supplicant on teepriv0.
[  OK  ] Stopped LLDP daemon.
[  OK  ] Stopped Netperf Benchmark Server.
[  OK  ] Stopped Simple Network Management Protocol (SNMP) Daemon..
[  OK  ] Stopped Getty on tty1.
[  OK  ] Stopped Serial Getty on ttyS2.
[  OK  ] Stopped containerd container runtime.
[  OK  ] Stopped Save/Restore Sound Card State.
[  OK  ] Stopped Reboot and dump vmcore via kexec.
[  OK  ] Stopped Load/Save OS Random Seed.
[  OK  ] Stopped Telnet Server.
[  OK  ] Stopped Session c2 of User root.
[  OK  ] Removed slice Slice /system/getty.
[  OK  ] Removed slice Slice /system/serial-getty.
[  OK  ] Removed slice Slice /system/tee-supplicant.
         Stopping User Manager for UID 0...
[  OK  ] Stopped User Manager for UID 0.
         Stopping User Runtime Directory /run/user/0...
[  OK  ] Unmounted /run/user/0.
[  OK  ] Stopped User Runtime Directory /run/user/0.
[  OK  ] Removed slice User Slice of UID 0.
[  OK  ] Stopped emptty display manager.
[  OK  ] Stopped Session c1 of User weston.
         Stopping User Login Management...
         Stopping User Manager for UID 1000...
[  OK  ] Stopped User Manager for UID 1000.
[  OK  ] Stopped User Login Management.
[  128.172348] kauditd_printk_skb: 8 callbacks suppressed
[  128.172359] audit: type=1334 audit(1741187728.792:20): prog-id=14 op=UNLOAD
         Stopping Permit User Sessions...
         Stopping User Runtime Directory /run/user/1000...
[  OK  ] Stopped Permit User Sessions.
[  OK  ] Unmounted /run/user/1000.
[  OK  ] Stopped User Runtime Directory /run/user/1000.
[  OK  ] Removed slice User Slice of UID 1000.
[  OK  ] Stopped target Network.
[  OK  ] Stopped target Remote File Systems.
         Stopping Network Configuration...
[  128.375734] ti_cpsw_proxy_client virtio1.ti.ethfw.ethdevice.-1.34: 01:80:c2:00:00:0e is reserved for EthFw
[  OK  ] Stopped Network Configuration.
[  OK  ] Stopped target Preparation for Netw[  128.423186] audit: type=1334 audit(1741187729.044:21): prog-id=17 op=UNLOAD
ork.
[  OK  ] Stopped IPv6 Packet Filtering Framework.
[  OK  ] Stopped IPv4 Packet Filtering Framework.
         Stopping Multiple Spanning Tree Protocol Daemon...
[  OK  ] Stopped Generate network units from Kernel command line.
[  OK  ] Stopped Multiple Spanning Tree Protocol Daemon.
[  OK  ] Stopped target Basic System.
[  OK  ] Stopped target Path Units.
[  OK  ] Stopped Dispatch Password Requests to Console Directory Watch.
[  OK  ] Stopped Forward Password Requests to Wall Directory Watch.
[  OK  ] Stopped target Slice Units.
[  OK  ] Removed slice User and Session Slice.
[  OK  ] Stopped target Socket Units.
[  OK  ] Closed Avahi mDNS/DNS-SD Stack Activation Socket.
[  OK  ] Closed Docker Socket for the API.
[  OK  ] Closed dropbear.socket.
[  OK  ] Closed PC/SC Smart Card Daemon Activation Socket.
[  OK  ] Closed Network Service Netlink Socket.
         Stopping D-Bus System Message Bus...
[  OK  ] Stopped D-Bus System Message Bus.
[  OK  ] Closed D-Bus System Message Bus Soc[  128.680320] audit: type=1334 audit(1741187729.300:22): prog-id=13 op=UNLOAD
ket.
[  OK  ] Stopped target System Initialization.
         Stopping Network Name Resolution...
         Stopping Network Time Synchronization...
[  OK  ] Stopped Update is Completed.
[  OK  ] Stopped Rebuild Dynamic Linker Cache.
[  OK  ] Stopped Rebuild Journal Catalog.
[  OK  ] Stopped Network Name Resolution.
[  OK  ] Stopped Network Time Synchronization.
[  128.811694] audit: type=1334 audit(1741187729.432:23): prog-id=12 op=UNLOAD
[  128.818740] audit: type=1334 audit(1741187729.432:24): prog-id=11 op=UNLOAD
[  OK  ] Stopped Apply Kernel Variables.
[  OK  ] Closed Process Core Dump Socket.
[  OK  ] Stopped Load Kernel Modules.
[  OK  ] Stopped Create System Files and Directories.
[  OK  ] Stopped target Local File Systems.
         Unmounting /run/media/boot-mmcblk1p1...
         Unmounting /run/media/kingston-250g-1-nvme1n1...
[  128.939766] EXT4-fs (nvme1n1): unmounting filesystem 2335c232-731d-4105-84d7-0935c3a99b94.

         Unmounting /run/media/ssd-nvme2n1p1...
         Unmounting Temporary Directory /tmp...
[  128.996327] EXT4-fs (nvme2n1p1): unmounting filesystem 838a8b75-1872-47d6-948f-a8345f37b0ec.
         Unmounting /var/volatile...
[  OK  ] Unmounted /run/media/boot-mmcblk1p1.
[  OK  ] Unmounted /run/media/kingston-250g-1-nvme1n1.
[  OK  ] Unmounted /run/media/nvme0n1p1.
[  OK  ] Unmounted /run/media/ssd-nvme2n1p1.
[  OK  ] Unmounted Temporary Directory /tmp.
[  OK  ] Unmounted /var/volatile.
[  OK  ] Stopped target Swaps.
[  OK  ] Reached target Unmount All Filesystems.
[  OK  ] Stopped File System Check on /dev/mmcblk1p1.
[  OK  ] Stopped File System Check on /dev/nvme0n1p1.
[  OK  ] Stopped File System Check on /dev/nvme1n1.
[  OK  ] Stopped File System Check on /dev/nvme2n1p1.
[  OK  ] Removed slice Slice /system/systemd-fsck.
[  OK  ] Stopped target Preparation for Local File Systems.
[  OK  ] Stopped Create Static Device Nodes in /dev.
[  OK  ] Stopped Create System Users.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Stopped Create Static Device Nodes in /dev gracefully.
[  OK  ] Reached target System Shutdown.
[  OK  ] Reached target Late Shutdown Services.
[  OK  ] Finished System Reboot.
[  OK  ] Reached target System Reboot.
[  129.309797] audit: type=1334 audit(1741187729.928:25): prog-id=16 op=UNLOAD
[  129.317114] audit: type=1334 audit(1741187729.928:26): prog-id=15 op=UNLOAD
[  129.324538] audit: type=1334 audit(1741187729.940:27): prog-id=8 op=UNLOAD
[  129.331727] audit: type=1334 audit(1741187729.940:28): prog-id=7 op=UNLOAD
[  129.338877] audit: type=1334 audit(1741187729.940:29): prog-id=6 op=UNLOAD
[  129.375701] watchdog: watchdog0: nowayout prevents watchdog being stopped!
[  129.382595] watchdog: watchdog0: watchdog did not stop!
[  129.400106] systemd-shutdown[1]: Using hardware watchdog 'K3 RTI Watchdog', version 0, device /dev/watchdog0
[  129.409955] systemd-shutdown[1]: Modifying watchdog hardware timeout is not supported, reusing the programmed timeout.
[  129.420791] systemd-shutdown[1]: Watchdog running with a hardware timeout of 1min.
[  129.468809] systemd-shutdown[1]: Syncing filesystems and block devices.
[  129.485425] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[  129.504966] systemd-journald[120]: Received SIGTERM from PID 1 (systemd-shutdow).
[  129.517378] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[  129.533801] systemd-shutdown[1]: Unmounting file systems.
[  129.541011] (sd-remount)[1321]: Remounting '/' read-only with options ''.
[  129.581543] EXT4-fs (mmcblk1p2): re-mounted eb38c320-367b-4a35-8fc1-93b90f10c045 ro. Quota mode: none.
[  129.593766] systemd-shutdown[1]: All filesystems unmounted.
[  129.599359] systemd-shutdown[1]: Deactivating swaps.
[  129.604427] systemd-shutdown[1]: All swaps deactivated.
[  129.609646] systemd-shutdown[1]: Detaching loop devices.
[  129.617717] systemd-shutdown[1]: All loop devices detached.
[  129.623297] systemd-shutdown[1]: Stopping MD devices.
[  129.628489] systemd-shutdown[1]: All MD devices stopped.
[  129.633795] systemd-shutdown[1]: Detaching DM devices.
[  129.639045] systemd-shutdown[1]: All DM devices detached.
[  129.644438] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[  129.654077] watchdog: watchdog0: nowayout prevents watchdog being stopped!
[  129.660946] systemd-shutdown[1]: Failed to disable hardware watchdog, ignoring: Device or resource busy
[  129.670325] watchdog: watchdog0: nowayout prevents watchdog being stopped!
[  129.677183] watchdog: watchdog0: watchdog did not stop!
[  129.691315] systemd-shutdown[1]: Syncing filesystems and block devices.
[  129.698038] systemd-shutdown[1]: Rebooting.
[  130.824930] reboot: Restarting system

U-Boot SPL 2025.01-00410-g70667128cb5b (Apr 04 2025 - 18:20:14 +0000)

So, I do not think issue is with number of PCIe devices.

A different theory would be whether the size of BAR address allocated is what is causing the issue. Perhaps utilizing a larger memory space for BAR is reaching some limit for the ITS to do address translation. So, could you send the full logs from "lspci -vvv"?

Regards,

Takuma

Hello Takuma,

Thanks for testing.

I did a try by switching ARM64 pages size from 4k to 64k. With that the "ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19" is gone but I still have the ITS timeout during reboot. I also tried with the latest 6.17 stable kernel but the PCIe doesn't work yet.

In your experiment, you only have 3 PCIe interfaces enabled (RC). In this case, there is no ITS timeout.

0000:00:00.0 PCI bridge: Texas Instruments Device b00d
0001:00:00.0 PCI bridge: Texas Instruments Device b00d
0002:00:00.0 PCI bridge: Texas Instruments Device b00d

Indeed, the number of PCIe endpoint doesn't seems to be the issue.
It's really when 4 PCIe (RC) interfaces are enabled.

Note:
On EVM board, the serdes3 is usually used for usb0 USB3.0.
On our custom board, the usb0 is actually on the serdes0 lane1 and the PCIe0 is using lane0, I had to backport the upstream patch "phy: cadence: Sierra: Add PCIe + USB PHY multilink configuration":
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=6767df73f2d36e7d1cf0eb3c4d9469c7e9fe9824
With that, the usb0 is working as USB3.0 host controller.

Here is the result of lspci -vvv:

0000:00:00.0 PCI bridge: Texas Instruments Device b00d (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 811
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: 10100000-101fffff [size=1M]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 0000000001000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00000000
		PBA: BAR=0 offset=00000008
	Capabilities: [c0] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <8us
			ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (ok), Width x1 (ok)
			TrErr- Train- SlotClk- DLActive- BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL+ CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled, ARIFwd+
			 AtomicOpsCtl: ReqEn- EgressBlck-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn+ NFERptEn+ FERptEn+
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [4c0 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
		VC2:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=2 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
		VC3:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=3 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
	Capabilities: [5c0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 01
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [640 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000001, Page Request Allocation: 00000000
	Capabilities: [900 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=26us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=255us LTR1.2_Threshold=561152ns
		L1SubCtl2: T_PwrOn=300us
	Kernel driver in use: pcieport
	Kernel modules: pci_endpoint_test

0000:01:00.0 Non-Volatile memory controller: Micron Technology Inc Device 5411 (rev 01) (prog-if 02 [NVM Express])
	Subsystem: Micron Technology Inc Device 0100
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 810
	Region 0: Memory at 10100000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (downgraded), Width x1 (downgraded)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
		LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: Upstream Port
	Capabilities: [d0] MSI-X: Enable+ Count=9 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [e0] MSI: Enable- Count=1/8 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [f8] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [110 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=10us PortTPowerOnTime=300us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=561152ns
		L1SubCtl2: T_PwrOn=300us
	Capabilities: [128 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [1e0 v1] Data Link Feature <?>
	Capabilities: [200 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [340 v1] Physical Layer 16.0 GT/s <?>
	Capabilities: [378 v1] Lane Margining at the Receiver <?>
	Kernel driver in use: nvme
	Kernel modules: nvme

0001:00:00.0 PCI bridge: Texas Instruments Device b00d (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 815
	Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: 18100000-181fffff [size=1M]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 0000000001040000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00000000
		PBA: BAR=0 offset=00000008
	Capabilities: [c0] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <8us
			ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x1 (ok)
			TrErr- Train- SlotClk- DLActive- BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL+ CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled, ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn+ NFERptEn+ FERptEn+
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [4c0 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
		VC2:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=2 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
		VC3:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=3 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
	Capabilities: [5c0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 01
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [640 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000001, Page Request Allocation: 00000000
	Capabilities: [900 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=26us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Kernel driver in use: pcieport
	Kernel modules: pci_endpoint_test

0001:01:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: 18100000-181fffff [size=1M]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
	Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: Device 0000:0000
	Capabilities: [48] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
		Bridge: PM- B3+
	Capabilities: [50] MSI: Enable- Count=1/16 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [70] Express (v2) PCI-Express to PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (ok), Width x1 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000

0001:02:00.0 Unassigned class [ff00]: Hilscher GmbH CIFX 50E-DP(M/S)
	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 814
	Region 0: Memory at 18100000 (32-bit, non-prefetchable) [size=64K]
	Kernel driver in use: netx
	Kernel modules: uio_netx

0002:00:00.0 PCI bridge: Texas Instruments Device b00d (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 819
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: 00200000-003fffff [size=2M]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 0000000001080000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00000000
		PBA: BAR=0 offset=00000008
	Capabilities: [c0] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <8us
			ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (ok), Width x1 (ok)
			TrErr- Train- SlotClk- DLActive- BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL+ CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled, ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn+ NFERptEn+ FERptEn+
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [4c0 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
		VC2:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=2 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
		VC3:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=3 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
	Capabilities: [5c0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 01
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [640 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000001, Page Request Allocation: 00000000
	Capabilities: [900 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=26us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=255us LTR1.2_Threshold=287744ns
		L1SubCtl2: T_PwrOn=26us
	Kernel driver in use: pcieport
	Kernel modules: pci_endpoint_test

0002:01:00.0 Network controller: Qualcomm QCNFA765 Wireless Network Adapter (rev 01)
	Subsystem: Qualcomm QCNFA765 Wireless Network Adapter
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin ? routed to IRQ 850
	Region 0: Memory at 4400200000 (64-bit, non-prefetchable) [size=2M]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=32/32 Maskable+ 64bit-
		Address: 01080400  Data: 0000
		Masking: fe023c00  Pending: 00000000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (ok), Width x1 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp+ ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [148 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [158 v1] Transaction Processing Hints
		No steering table available
	Capabilities: [1e4 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [1ec v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=70us PortTPowerOnTime=0us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=287744ns
		L1SubCtl2: T_PwrOn=26us
	Kernel driver in use: ath11k_pci
	Kernel modules: ath11k_pci

0003:00:00.0 PCI bridge: Texas Instruments Device b00d (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 823
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: 00100000-001fffff [size=1M]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 00000000010c0000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00000000
		PBA: BAR=0 offset=00000008
	Capabilities: [c0] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x2, ASPM L1, Exit Latency L1 <8us
			ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (downgraded), Width x2 (ok)
			TrErr- Train- SlotClk- DLActive- BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL+ CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled, ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn+ NFERptEn+ FERptEn+
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [4c0 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
		VC2:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=2 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
		VC3:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=3 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
	Capabilities: [5c0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 01
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [640 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000001, Page Request Allocation: 00000000
	Capabilities: [900 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=26us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Kernel driver in use: pcieport
	Kernel modules: pci_endpoint_test

0003:01:00.0 Memory controller: Xilinx Corporation Device 7022
	Subsystem: Xilinx Corporation Device 0007
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Region 0: Memory at 4410100000 (32-bit, non-prefetchable) [disabled] [size=32K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 1, Latency L0s <64ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x2, ASPM L0s, Exit Latency L0s unlimited
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (ok), Width x2 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00

Note:
I had to apply the TI patch to disable the kernel panic for pcie switch (errata i2086 for J721E/DRA829/TDA4VM) to use the
"Texas Instruments XIO2001 PCI Express-to-PCI Bridge [104c:8240]" device. The "XIO2001 PCI Express-to-PCI Bridge" is
part of the "Hilscher GmbH CIFX 50E-DP" device.

Here is /proc/iomem:

# cat /proc/iomem

0011c000-0011c2b3 : pinctrl-single
00600000-006000ff : 600000.gpio gpio@600000
00601000-006010ff : 601000.gpio gpio@601000

01800000-0180ffff : GICD
01900000-019fffff : GICR

02000000-020000ff : 2000000.i2c i2c@2000000
02010000-020100ff : 2010000.i2c i2c@2010000
02110000-021103ff : 2110000.spi spi@2110000
02200000-022000ff : 2200000.watchdog watchdog@2200000
02210000-022100ff : 2210000.watchdog watchdog@2210000
02800000-0280001f : serial
02810000-0281001f : serial

02900000-02900fff : 2900000.pcie intd_cfg
02907000-029073ff : 2900000.pcie user_cfg
02910000-02910fff : 2910000.pcie intd_cfg
02917000-029173ff : 2910000.pcie user_cfg
02920000-02920fff : 2920000.pcie intd_cfg
02927000-029273ff : 2920000.pcie user_cfg
02930000-02930fff : 2930000.pcie intd_cfg
02937000-029373ff : 2930000.pcie user_cfg

04104000-041040ff : 4104000.cdns-usb cdns-usb@4104000
04114000-041140ff : 4114000.cdns-usb cdns-usb@4114000
04200000-042fffff : 4200000.video-encoder video-encoder@4200000
04300000-043fffff : 4300000.video-decoder video-decoder@4300000
04a00000-04a0ffff : 4a00000.dss common_m
04a20000-04a2ffff : 4a00000.dss vidl1
04a30000-04a3ffff : 4a00000.dss vidl2
04a50000-04a5ffff : 4a00000.dss vid1
04a60000-04a6ffff : 4a00000.dss vid2
04a70000-04a7ffff : 4a00000.dss ovr1
04a80000-04a8ffff : 4a00000.dss vp1
04a90000-04a9ffff : 4a00000.dss ovr2
04aa0000-04aaffff : 4a00000.dss vp2
04ab0000-04abffff : 4a00000.dss ovr3
04ac0000-04acffff : 4a00000.dss vp3
04ad0000-04adffff : 4a00000.dss ovr4
04ae0000-04aeffff : 4a00000.dss vp4
04e00000-04e011ff : 4e00000.crypto crypto@4e00000
04e10000-04e1007c : 4e10000.rng rng@4e10000
04f40000-04f4001f : a000000.dp-bridge j721e-intg
04f80000-04f80fff : 4f80000.mmc mmc@4f80000
04f88000-04f883ff : 4f80000.mmc mmc@4f80000
04fb0000-04fb0fff : 4fb0000.mmc mmc@4fb0000
04fb8000-04fb83ff : 4fb0000.mmc mmc@4fb0000
05000000-0500ffff : 5000000.serdes serdes
05010000-0501ffff : 5010000.serdes serdes
05020000-0502ffff : 5020000.serdes serdes
05030000-0503ffff : 5030000.serdes serdes
05050000-0505ffff : 5050000.serdes torrent_phy
05c00000-05c07fff : 5c00000.r5f
05c10000-05c17fff : 5c00000.r5f
05d00000-05d07fff : 5d00000.r5f
05d10000-05d17fff : 5d00000.r5f
05e00000-05e07fff : 5e00000.r5f
05e10000-05e17fff : 5e00000.r5f
05f00000-05f07fff : 5f00000.r5f
05f10000-05f17fff : 5f00000.r5f
06000000-0600ffff : 6000000.usb otg
06010000-0601ffff : xhci
  06010000-0601ffff : xhci-hcd.11.auto xhci
06020000-0602ffff : 6000000.usb dev
06400000-0640ffff : 6400000.usb otg
06410000-0641ffff : xhci
  06410000-0641ffff : xhci-hcd.12.auto xhci
06420000-0642ffff : 6400000.usb dev
0a000000-0a0309ff : a000000.dp-bridge mhdptx
0a030a00-0a030a3f : 5050000.serdes dptx_phy
0b020000-0b021fff : b020000.interrupt-controller interrupt-controller@20000
0b120000-0b121fff : b120000.interrupt-controller interrupt-controller@20000
0c000000-0c1fffff : c000000.ethernet cpsw_nuss

0d000000-0d7fffff : 2900000.pcie reg
0d800000-0dffffff : 2910000.pcie reg
0e000000-0e7fffff : 2920000.pcie reg
0e800000-0effffff : 2930000.pcie reg
10000000-10000fff : 2900000.pcie cfg

10011000-17ffffff : pcie@2900000        disk
  10100000-101fffff : PCI Bus 0000:01
    10100000-10103fff : 0000:01:00.0
      10100000-10103fff : nvme

18000000-18000fff : 2910000.pcie cfg    cifx
18011000-1fffffff : pcie@2910000
  18100000-181fffff : PCI Bus 0001:01
    18100000-181fffff : PCI Bus 0001:02
      18100000-1810ffff : 0001:02:00.0
        18100000-1810ffff : netx

28590000-285900ff : 2b800000.ringacc proxy_gcfg
285c0000-285c00ff : 285c0000.dma-controller gcfg
2a500000-2a53ffff : 2b800000.ringacc proxy_target
2a800000-2a83ffff : 285c0000.dma-controller rchanrt
2aa00000-2aa3ffff : 285c0000.dma-controller tchanrt
2b000000-2b3fffff : 2b800000.ringacc fifos
2b800000-2bbfffff : 2b800000.ringacc rt
30e00000-30e00fff : 30e00000.spinlock spinlock@30e00000
310d0000-310d03ff : 310d0000.cpts cpts
31120000-311200ff : 3c000000.ringacc proxy_gcfg
31150000-311500ff : 31150000.dma-controller gcfg
31f80000-31f801ff : 31f80000.mailbox mailbox@31f80000
31f81000-31f811ff : 31f81000.mailbox mailbox@31f81000
31f82000-31f821ff : 31f82000.mailbox mailbox@31f82000
31f83000-31f831ff : 31f83000.mailbox mailbox@31f83000
31f84000-31f841ff : 31f84000.mailbox mailbox@31f84000
32400000-324fffff : 32c00000.mailbox rt
32800000-328fffff : 32c00000.mailbox scfg
32c00000-32cfffff : 32c00000.mailbox target_data
33000000-3303ffff : 3c000000.ringacc proxy_target
33d00000-33dfffff : 33d00000.interrupt-controller interrupt-controller@33d00000
34000000-340fffff : 31150000.dma-controller rchanrt
35000000-350fffff : 31150000.dma-controller tchanrt
36600000-36600dff : 36600000.iommu
36610000-36610dff : 36600000.iommu
38000000-383fffff : 3c000000.ringacc fifos
3c000000-3c3fffff : 3c000000.ringacc rt
40300000-403003ff : 40300000.spi spi@40300000
40a00000-40a0001f : serial
40b00000-40b000ff : 40b00000.i2c i2c@40b00000
41000000-41007fff : 41000000.r5f
41010000-41017fff : 41000000.r5f
41400000-41407fff : 41400000.r5f
41410000-41417fff : 41400000.r5f
41c00000-41cfffff : 41c00000.sram sram@41c00000
42040000-4204034f : 42040000.temperature-sensor temperature-sensor@42040000
42050000-4205034f : 42040000.temperature-sensor temperature-sensor@42040000
42110000-421100ff : 42110000.gpio gpio@42110000
42120000-421200ff : 42120000.i2c i2c@42120000
43000014-43000017 : 43000014.chipid chipid@43000014
43000300-4300030f : 42040000.temperature-sensor temperature-sensor@42040000
4301c000-4301c177 : pinctrl-single
44083000-44083fff : 44083000.system-controller debug_messages
46000000-461fffff : 46000000.ethernet cpsw_nuss
64800000-6487ffff : 64800000.dsp
64e00000-64e0bfff : 64800000.dsp
70000000-707effff : 70000000.sram sram@70000000
80000000-9e7fffff : System RAM
9e800000-a8ffffff : reserved
a9000000-a9ffffff : System RAM
aa000000-abbfffff : reserved
abc00000-abffffff : System RAM
ac000000-ac0fffff : reserved
ac100000-fce1bfff : System RAM
  dd600000-dd614fff : reserved
  dd810000-de4fffff : Kernel code
  de500000-de70ffff : reserved
  de710000-de8dffff : Kernel data
  dfab0000-dfab0fff : reserved
  dfab1000-dfab1fff : reserved
  dfab2000-dfac9fff : reserved
  dffff000-dfffffff : reserved
  f8e03000-fce1afff : reserved
fce1c000-fce1cfff : reserved
fce1d000-fce1dfff : System RAM
fce1e000-fce2efff : reserved
  fce1e000-fce1efff : reserved
fce2f000-fce40fff : System RAM
fce41000-fce42fff : reserved
fce43000-fce43fff : System RAM
fce44000-fce63fff : reserved
fce64000-fdeb0fff : System RAM
fdeb1000-fdeb1fff : reserved
fdeb2000-ffeaffff : System RAM
ffeb0000-ffebffff : reserved
ffec0000-ffffffff : System RAM
880000000-8ffffffff : System RAM
  880011000-880011fff : reserved
  880030000-88003ffff : reserved
  880040000-88004ffff : reserved
  880050000-88005ffff : reserved
  8db800000-8dfdfffff : reserved
  8dfe85000-8dff44fff : reserved
  8dff45000-8dff45fff : reserved
  8dff46000-8dff6dfff : reserved
  8dff6f000-8dff6ffff : reserved
  8dff70000-8dff72fff : reserved
  8dff73000-8dff84fff : reserved
  8dff85000-8ffffffff : reserved
  
4400000000-4400000fff : 2920000.pcie cfg   Wifi
4400011000-4407ffffff : pcie@2920000
  4400200000-44003fffff : PCI Bus 0002:01
    4400200000-44003fffff : 0002:01:00.0
      4400200000-44003fffff : ath11k_pci

4410000000-4410000fff : 2930000.pcie cfg   FPGA
4410011000-4417ffffff : pcie@2930000
  4410100000-44101fffff : PCI Bus 0003:01
    4410100000-4410107fff : 0003:01:00.0

4d80800000-4d80847fff : 4d80800000.dsp
4d80e00000-4d80e07fff : 4d80800000.dsp
4d80f00000-4d80f07fff : 4d80800000.dsp
4d81800000-4d81847fff : 4d81800000.dsp
4d81e00000-4d81e07fff : 4d81800000.dsp
4d81f00000-4d81f07fff : 4d81800000.dsp

While the extra reboot time due ITS timeout is annoying, it doesn't prevent the SoC from rebooting.
But we don't understand what's going on, and if it's a log we can ignore for now.
(it also happen with poweroff)

Best regards,
Romain

Hi Romain,

Romain Naour said:

It's really when 4 PCIe (RC) interfaces are enabled.

The EVM has 3 PCIe ports, so unable to test hardware-wise. But do you see issues with ITS when no device is connected to the PCIe port as well, when there are 4 ports enabled?

Regards,

Takuma

Hi Takuma,

It seems you are close to reproduce it on TI EVM board.
I did additional testing with several PCIe devices.

On the customer board, there are ITS timeout when:
- NVMe (pcie0) + wifi (pcie2) + FPGA (pcie3) (pcie1 slot is empty)
- NVMe (pcie0) + cifx (pcie1) + FPGA (pcie3) (pcie2 slot is empty)
- FPGA (pcie3) + wifi (pcie0) + wifi2 (pcie2) (ITS timeout difficult to reproduce every time) (pcie1 slot is empty)
- NVMe (pcie0) + wifi (pcie2) + FPGA (pcie3) (pcie1 slot is empty, FPGA bitstream erased*)

wifi is an gen3 PCIe device
wifi2 is an gen1 PCIe device

* for all other tests the FPGA bitstream initialize an PCIe EP interface.
The last test case is interesting since we can test it on the TI EVM board (see bellow).

Otherwise, there is no ITS timeout when:
- FPGA (pcie3) only
- FPGA (pcie3) + NVMe (pcie0)
- FPGA (pcie3) + wifi (pcie2)
- FPGA (pcie3) + wifi (pcie2) + cifx (pcie1)
- FPGA (pcie3) + wifi (pcie2) + cifx (pcie1) + wifi2 (pcie1)
- NVMe (pcie0) + cifx (pcie1) + FPGA (pcie3)
- NVMe (pcie0) + cifx (pcie1) + wifi (pcie2) (pcie3 disabled from the devicetree)

I tried with another NVMe disk without improvement.

Since I actually have the TI J721e EVM board, I was able to reproduce the issue!
Here is the log I recorded with minicom on the serial console (Yocto kirkstone based image):

j721e-evm-4-PCIe-ITS-timeout.txt

To do so, we have to disable the USB0 port and enable the 4th PCIe port on the devicetree.

From 646e7d7df309f66e96636989b2463d7c9eeb0017 Mon Sep 17 00:00:00 2001
From: Romain Naour <romain.naour@smile.fr>
Date: Thu, 9 Oct 2025 12:08:47 +0200
Subject: [PATCH] try 4 PCIe EVM

---
 .../dts/ti/k3-j721e-common-proc-board.dts     | 31 ++++++++++---------
 1 file changed, 16 insertions(+), 15 deletions(-)

diff --git a/arch/arm64/boot/dts/ti/k3-j721e-common-proc-board.dts b/arch/arm64/boot/dts/ti/k3-j721e-common-proc-board.dts
index 15102143035f5..fd64f1a3a3936 100644
--- a/arch/arm64/boot/dts/ti/k3-j721e-common-proc-board.dts
+++ b/arch/arm64/boot/dts/ti/k3-j721e-common-proc-board.dts
@@ -495,37 +495,38 @@ &serdes_ln_ctrl {
 	idle-states = <J721E_SERDES0_LANE0_PCIE0_LANE0>, <J721E_SERDES0_LANE1_PCIE0_LANE1>,
 		      <J721E_SERDES1_LANE0_PCIE1_LANE0>, <J721E_SERDES1_LANE1_PCIE1_LANE1>,
 		      <J721E_SERDES2_LANE0_PCIE2_LANE0>, <J721E_SERDES2_LANE1_PCIE2_LANE1>,
-		      <J721E_SERDES3_LANE0_USB3_0_SWAP>, <J721E_SERDES3_LANE1_USB3_0>,
+		      <J721E_SERDES3_LANE0_PCIE3_LANE0>, <J721E_SERDES3_LANE1_PCIE3_LANE1>,
 		      <J721E_SERDES4_LANE0_EDP_LANE0>, <J721E_SERDES4_LANE1_EDP_LANE1>,
 		      <J721E_SERDES4_LANE2_EDP_LANE2>, <J721E_SERDES4_LANE3_EDP_LANE3>;
 };
 
-&serdes_wiz3 {
-	typec-dir-gpios = <&main_gpio1 3 GPIO_ACTIVE_HIGH>;
-	typec-dir-debounce-ms = <700>;	/* TUSB321, tCCB_DEFAULT 133 ms */
-};
-
 &serdes3 {
-	serdes3_usb_link: phy@0 {
+	serdes3_pcie_link: phy@0 {
 		reg = <0>;
 		cdns,num-lanes = <2>;
 		#phy-cells = <0>;
-		cdns,phy-type = <PHY_TYPE_USB3>;
+		cdns,phy-type = <PHY_TYPE_PCIE>;
 		resets = <&serdes_wiz3 1>, <&serdes_wiz3 2>;
 	};
 };
 
+&pcie3_rc { /* fpga */
+	status = "okay";
+	// pinctrl-names = "default";
+	// pinctrl-0 = <&main_ringgb_pcie3_pins_default>;
+	// reset-gpios = <&main_gpio0 72 GPIO_ACTIVE_HIGH>; /* (Y24) PRG0_PRU1_GPO9.GPIO0_72 - GPO_FPGA_RSTn */
+
+	phys = <&serdes3_pcie_link>;
+	phy-names = "pcie-phy";
+	num-lanes = <2>;
+};
+
 &usbss0 {
-	pinctrl-names = "default";
-	pinctrl-0 = <&main_usbss0_pins_default>;
-	ti,vbus-divider;
+	status = "disabled";
 };
 
 &usb0 {
-	dr_mode = "otg";
-	maximum-speed = "super-speed";
-	phys = <&serdes3_usb_link>;
-	phy-names = "cdns3,usb3-phy";
+	status = "disabled";
 };
 
 &usbss1 {
-- 
2.51.0

The nvme disk populate the PCIe slot PCIe_M.2_INTERFACE J47 pcie2, while the wifi module populate the PCIe slot x1LANE PCIe Interface J11 pcie0.
pci1 and pcie3 are enabled from the devicetree but the slot is empty.

root@j721e-evm:~# lspci -nn
0000:00:00.0 PCI bridge [0604]: Texas Instruments Device [104c:b00d]
0000:01:00.0 Network controller [0280]: Qualcomm Technologies, Inc QCNFA765 Wireless Network Adapter [17cb:1103] (rev 01)
0001:00:00.0 PCI bridge [0604]: Texas Instruments Device [104c:b00d]
0002:00:00.0 PCI bridge [0604]: Texas Instruments Device [104c:b00d]
0002:01:00.0 Non-Volatile memory controller [0108]: Micron Technology Inc 2550 NVMe SSD (DRAM-less) [1344:5416] (rev 01)
0003:00:00.0 PCI bridge [0604]: Texas Instruments Device [104c:b00d]

I'm running out of time for further investigation but at least you should be able to reproduce it.

Best regards,
Romain

Hello Takuma,

I did some further testing with several kernels 6.1.151-rt54+, 6.12 and 6.17.1 kernels with similar results.

With "PCI: j721e: Add support to build as a loadable module" added, we can trigger such ITS timeout while removing the pci_j721e_host driver:

# lspci -nn
0000:00:00.0 PCI bridge [0604]: Texas Instruments Device [104c:b00d]
0000:01:00.0 Non-Volatile memory controller [0108]: Micron Technology Inc Device [1344:5411] (rev 01)
0001:00:00.0 PCI bridge [0604]: Texas Instruments Device [104c:b00d]
0002:00:00.0 PCI bridge [0604]: Texas Instruments Device [104c:b00d]
0002:01:00.0 Network controller [0280]: Qualcomm QCNFA765 Wireless Network Adapter [17cb:1103] (rev 01)
0003:00:00.0 PCI bridge [0604]: Texas Instruments Device [104c:b00d]

lsmod:

pci_j721e_host         16384  0
pci_j721e              12288  1 pci_j721e_host
pcie_cadence_host      16384  1 pci_j721e_host
pcie_cadence           12288  2 pcie_cadence_host,pci_j721e_host

# rmmod pci_j721e_host

[   34.545172] pci_bus 0003:01: busn_res: [bus 01] is released
[   34.545321] pci_bus 0003:00: busn_res: [bus 00-ff] is released

[   34.561436] wlP2p1s0: deauthenticating from 46:fe:3b:77:75:c0 by local choice (Reason: 3=DEAUTH_LEAVING)
[   34.971908] pci_bus 0002:01: busn_res: [bus 01] is released
[   34.984049] pci_bus 0002:00: busn_res: [bus 00-ff] is released

[   36.444707] ITS queue timeout (23008 22944)
[   36.444714] ITS cmd 0xffff8000084156b0 failed
[   37.882042] ITS queue timeout (23072 22944)
[   37.882051] ITS cmd 0xffff800008415640 failed
[   39.323850] ITS queue timeout (23104 22944)
[   39.323859] ITS cmd 0xffff800008413d60 failed

[   39.323973] pci_bus 0001:01: busn_res: [bus 01] is released
[   39.328970] pci_bus 0001:00: busn_res: [bus 00-ff] is released
[   40.907427] ITS queue timeout (23168 22944)
[   40.907434] ITS cmd 0xffff8000084156b0 failed
[   42.342928] ITS queue timeout (23232 22944)
[   42.342936] ITS cmd 0xffff800008415640 failed
[   43.779602] ITS queue timeout (23296 22944)
[   43.779608] ITS cmd 0xffff8000084156b0 failed
[   45.214579] ITS queue timeout (23360 22944)
[   45.214586] ITS cmd 0xffff800008415640 failed
[   46.653549] ITS queue timeout (23424 22944)
[   46.653557] ITS cmd 0xffff8000084156b0 failed
[   48.102638] ITS queue timeout (23488 22944)
[   48.102644] ITS cmd 0xffff800008415640 failed
[   49.544791] ITS queue timeout (23520 22944)
[   49.544799] ITS cmd 0xffff800008413d60 failed
[   50.989615] ITS queue timeout (23584 22944)
[   50.989623] ITS cmd 0xffff8000084156b0 failed
[   52.424533] ITS queue timeout (23648 22944)
[   52.424540] ITS cmd 0xffff800008415640 failed
[   53.877601] ITS queue timeout (23680 22944)
[   53.877609] ITS cmd 0xffff800008413d60 failed
[   53.877836] pci_bus 0000:01: busn_res: [bus 01] is released
[   53.887103] pci_bus 0000:00: busn_res: [bus 00-ff] is released

I noticed that if we remove each pcie device manually one by one in a specific order, then we don't trigger any ITS timeout:

# echo 1 > /sys/devices/platform/bus@100000/2900000.pcie/pci0000:00/0000:00:00.0/remove
# echo 1 > /sys/devices/platform/bus@100000/2910000.pcie/pci0001:00/0001:00:00.0/remove
# echo 1 > /sys/devices/platform/bus@100000/2930000.pcie/pci0003:00/0003:00:00.0/remove
# echo 1 > /sys/devices/platform/bus@100000/2920000.pcie/pci0002:00/0002:00:00.0/remove

# rmmod pci_j721e_host
pci_bus 0003:00: busn_res: [bus 00-ff] is released
pci_bus 0002:00: busn_res: [bus 00-ff] is released
pci_bus 0001:00: busn_res: [bus 00-ff] is released
pci_bus 0000:00: busn_res: [bus 00-ff] is released

But If we remove in the same order that the kernel usually does we got ITS timeout.

echo 1 > /sys/devices/platform/bus@100000/2930000.pcie/pci0003:00/0003:00:00.0/remove
echo 1 > /sys/devices/platform/bus@100000/2920000.pcie/pci0002:00/0002:00:00.0/remove

echo 1 > /sys/devices/platform/bus@100000/2910000.pcie/pci0001:00/0001:00:00.0/remove
ITS queue timeout 

echo 1 > /sys/devices/platform/bus@100000/2900000.pcie/pci0000:00/0000:00:00.0/remove
ITS queue timeout 

rmmod pci_j721e_host
[  100.067097] pci_bus 0003:00: busn_res: [bus 00-ff] is released
[  100.067903] pci_bus 0002:00: busn_res: [bus 00-ff] is released
[  100.073590] pci_bus 0001:00: busn_res: [bus 00-ff] is released
[  100.073855] pci_bus 0000:00: busn_res: [bus 00-ff] is released

Note: With Ftrace enabled we can see ITS cmd names:

ITS cmd its_build_inv_cmd failed
ITS cmd its_build_discard_cmd failed
ITS cmd its_build_mapd_cmd failed

Do you think it could be related to the J721e SoC errata i2101: GIC: ITS Misbehavior ?

"This will cause all ITS requests on that particular Device ID, for which read access was requested to fail."

"That issue only prevents more than 5 pci-endpoint-test devices to be tested simultaneously."
https://lore.kernel.org/linux-omap/b70bd865-920d-f736-9e6c-a6bb4442476f@ti.com/

"since both the platforms use GIC ITS it would need modifications
in the irq-gic-v3-its.c driver. I could add the workaround mentioned in
the errata document and send the patch for review to see what others think."
https://lore.kernel.org/all/78b4c4df-0854-fe28-53bd-7629b0357637@ti.com/

It seems no gic-v3-its workaround was send on the kernel mailing list.

Both AM65 and J721e have the same issue reported on their errata:

https://www.ti.com/lit/er/sprz452i/sprz452i.pdf

"Note: This workaround is not implemented in Processor SDK Linux since it needs
modifications to Arm® provided / maintained generic driver for GIC in Linux."

It would be interesting to check if the same issue could be reproduced on the AM69 (j784s4), it seems doable with the SK-AM69 board.

Thought ?

Best regards,
Romain

Hi Romain,

Thanks for doing a deep dive on this issue. So far only J721e has been showing this ITS issue I agree with you that this may be due to errata i2101. 

Let me see if we can find workaround for this errata. As a warning, we are entering Diwali season, so some engineers will be out of office, so please expect a few days of delays.

Regards,

Takuma