PROCESSOR-SDK-J721E: PCIe: Bridge: Linux Kernel Panic

Hi,

All 6 functions have to to be created on the endpoint device.

Please have a look at this ticket for more info.

Let me know if you need further help.

Regards,
Tanmay

Hi Tanmay,

Thanks for your answer! And we'll try at least to replicate and see if those steps works and we'll let you know.

However, that that seams a workaround and not a proper fix. We do not expect any user space configuration to get the switch working  as on the Sitara board.

Is that something on the roadmap to be solved?

Regards,

Paulo

Hi, Tanmay,

I am facing the same kernel panic issue.

My setup is: Jacinto 7 as a PCIe host (Root Complex) and PCIe Endpoint with a Realtek PCIe NIC RTL8111H

I think the ticket you mentioned is a different case with our issue. The ticket is about how to config Jacinto 7 as endpoint device. But our setup's endpoint is a third-party device. 

Could you help to investigate this issue?

kernel boot and panic log:

[    1.026490] j721e-pcie 2900000.pcie: host bridge /bus@100000/pcie@2900000 ranges:

[    1.026511] j721e-pcie 2900000.pcie:       IO 0x0010001000..0x0010010fff -> 0x0010001000

[    1.026522] j721e-pcie 2900000.pcie:      MEM 0x0010011000..0x0017ffffff -> 0x0010011000

[    1.026532] j721e-pcie 2900000.pcie:   IB MEM 0x0000000000..0xffffffffffff -> 0x0000000000

[    1.137304] j721e-pcie 2900000.pcie: Link up

[    1.137309] j721e-pcie 2900000.pcie: Link up

[    1.137375] j721e-pcie 2900000.pcie: PCI host bridge to bus 0000:00

[    1.137379] pci_bus 0000:00: root bus resource [bus 00-ff]

[    1.137384] pci_bus 0000:00: root bus resource [io  0x0000-0xffff] (bus address [0x10001000-0x10010fff])

[    1.137388] pci_bus 0000:00: root bus resource [mem 0x10011000-0x17ffffff]

[    1.137418] pci 0000:00:00.0: [104c:b00d] type 01 class 0x060400

[    1.137423] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x4 may corrupt adjacent RW1C bits

[    1.137509] pci 0000:00:00.0: supports D1

[    1.137512] pci 0000:00:00.0: PME# supported from D0 D1 D3hot

[    1.137530] pci 0000:00:00.0: reg 0x224: [mem 0x00000000-0x003fffff 64bit]

[    1.137534] pci 0000:00:00.0: VF(n) BAR0 space: [mem 0x00000000-0x00ffffff 64bit] (contains BAR0 for 4 VFs)

[    1.139524] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring

[    1.161305] SError Interrupt on CPU1, code 0xbf000000 -- SError

[    1.161311] CPU: 1 PID: 66 Comm: kworker/1:2 Not tainted 5.10.162-rt79-g02a1d48fd7 #3

[    1.161314] Hardware name: Texas Instruments K3 J721E SoC (DT)

[    1.161320] j721e-pcie 2900000.pcie: LINK DOWN!

[    1.161321] Workqueue: events deferred_probe_work_func

[    1.161334] pstate: 20000085 (nzCv daIf -PAN -UAO -TCO BTYPE=--)

[    1.161338] pc : pci_generic_config_read+0x3c/0xe0

[    1.161346] lr : pci_generic_config_read+0x24/0xe0

[    1.161349] sp : ffff800011883720

[    1.161350] x29: ffff800011883720 x28: ffff000801be6800

[    1.161354] x27: 0000000000000000 x26: 0000000000000001

[    1.161357] x25: ffff800011256bf8 x24: 0000000000000000

[    1.161360] x23: ffff800011883864 x22: 0000000000000087

[    1.161363] x21: ffff000801be7000 x20: 0000000000000004

[    1.161366] x19: ffff8000118837a4 x18: 0000000000000000

[    1.161368] x17: 00000000f6ae94a3 x16: 00000000e360973d

[    1.161372] x15: ffff000800d9b260 x14: ffffffffffffffff

[    1.161374] x13: ffff000801ab9a1c x12: ffff000801ab929a

[    1.161377] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f

[    1.161380] x9 : 00000000b00d104c x8 : 0000000080b5111d

[    1.161383] x7 : 000000000000ea60 x6 : ffff800011883864

[    1.161386] x5 : ffff000801be6800 x4 : 000000000000000b

[    1.161388] x3 : 0000000000000001 x2 : ffff800013c00008

[    1.161391] x1 : 0000000000000000 x0 : ffff8000116b5000

[    1.161394] Kernel panic - not syncing:

[    1.161395] Asynchronous SError Interrupt

[    1.161397] CPU: 1 PID: 66 Comm: kworker/1:2 Not tainted 5.10.162-rt79-g02a1d48fd7 #3

[    1.161400] Hardware name: Texas Instruments K3 J721E SoC (DT)

[    1.161406] Workqueue: events deferred_probe_work_func

[    1.161410] Call trace:

[    1.161411]  dump_backtrace+0x0/0x1a0

[    1.161420]  show_stack+0x18/0x68

[    1.161424]  dump_stack+0xd0/0x12c

[    1.161430]  panic+0xcc/0x360

[    1.161434]  nmi_panic+0x94/0x98

[    1.161439]  arm64_serror_panic+0x78/0x84

[    1.161442]  do_serror+0x38/0x98

[    1.161447]  el1_error+0x90/0x110

[    1.161449]  pci_generic_config_read+0x3c/0xe0

[    1.161453]  cdns_ti_pcie_config_read+0x18/0x38

[    1.161459]  pci_bus_read_config_dword+0x80/0xd8

[    1.161462]  pci_bus_generic_read_dev_vendor_id+0x34/0x1b0

[    1.161465]  pci_bus_read_dev_vendor_id+0x4c/0x70

[    1.161468]  pci_scan_single_device+0x7c/0xf8

[    1.161470]  pci_scan_slot+0x40/0x120

[    1.161472]  pci_scan_child_bus_extend+0x54/0x298

[    1.161475]  pci_scan_bridge_extend+0x29c/0x580

[    1.161477]  pci_scan_child_bus_extend+0x1e4/0x298

[    1.161480]  pci_scan_root_bus_bridge+0x64/0xd8

[    1.161482]  pci_host_probe+0x18/0xc8

[    1.161485]  cdns_pcie_host_setup+0x51c/0x8f8

[    1.161489]  j721e_pcie_probe+0x494/0x820

[    1.161492]  platform_drv_probe+0x54/0xa8

[    1.161498]  really_probe+0xec/0x3e0

[    1.161501]  driver_probe_device+0x58/0xb8

[    1.161504]  __device_attach_driver+0xb8/0xe0

[    1.161507]  bus_for_each_drv+0x78/0xc8

[    1.161510]  __device_attach+0xf8/0x188

[    1.161513]  device_initial_probe+0x14/0x20

[    1.161516]  bus_probe_device+0x9c/0xa8

[    1.161519]  deferred_probe_work_func+0x88/0xc0

[    1.161522]  process_one_work+0x1a0/0x328

[    1.161527]  worker_thread+0x1f8/0x420

[    1.161531]  kthread+0x174/0x198

[    1.161534]  ret_from_fork+0x10/0x34

Regards!

Johnson Xu

Hi Tanmay,

For Johnson's case issue, I have provided more details below.

sw3.5 and sw3.6 configuration:

Pcie ROOT complex device node:

Pcie root complex scan pcie end point device, and crashed in pci_generic_config_read.

After debugging the addr value, then found the value is the equal to host’s config addr: 0xffff8000116b5000

It seems like some configuration issues in PCIE bridge between TDA4 host (root complex mode) & third party endpoint device. Could you provide some suggestions for this?

Thanks,

Kevin

Hello,
Your use-case is with RT-Linux or Non-RT ?

Hi,

I'm using non-RT linux! And actually we perform the tests with prebuilt images both from jacinto and sitara.
That happens still exists on the latest yocto dunfell. Re-statment that on sitara the pci it works.

Would it be possible for you to try with Non-RT ?
Also, Is RT kernel must for your use-case ?

RT is not our case... and we would line to avoid linux-rt.

ok, Could you please try with Non RT.

Thanks,

Achal Verma

Hi, Achal

    I'm using RT Linux SDK:  PROCESSOR-SDK-LINUX-RT-J721E

Regards!

Johnson

Hi Achal,

Based on the discussions, Paulo used non-RT linux & Johnson used RT linux, both of them get this issue, it seems not related to the RT kernel.

Kind Regards,

Kevin

Thanks Kevin for clarification.

In Paulo case, non RT linux is being used and setup is J721E connected to 3rd party switch.

From logs, as mentioned during enumeration issue occurs while discovering available functions by reading vendor IDs.

Also with Sitara device which has designware PCIe IP, it works right ?

Give me sometime, to look into it, I might also have to run some experiments though we don't have the same peripheral.

Also if you can share the complete dmesg logs and lspci -vvv output in both working and non-working case.

That is right!

For the J721E, obviously we do not have lspci log (because it crashes on boot at enumeration step), but for the AM64x (TMDS64EVM) we have the following log:

00:00.0 PCI bridge: Texas Instruments Device b010 (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 44
	Region 0: Memory at <unassigned> (64-bit, prefetchable)
	Bus: primary=00, secondary=01, subordinate=0a, sec-latency=0
	I/O behind bridge: 68002000-68003fff [size=8K]
	Memory behind bridge: 68100000-685fffff [size=5M]
	Prefetchable memory behind bridge: 0000000068600000-00000000689fffff [size=4M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 0000000001000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
		Vector table: BAR=0 offset=00000000
		PBA: BAR=0 offset=00000008
	Capabilities: [c0] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <8us
			ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (ok), Width x1 (ok)
			TrErr- Train- SlotClk- DLActive- BWMgmt+ ABWMgmt+
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL+ CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCap: CRSVisible-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range B, TimeoutDis+, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt+, EETLPPrefix+, MaxEETLPPrefixes 1
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, LN System CLS Not Supported, TPHComp-, ExtTPHComp-, ARIFwd-
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn- NFERptEn- FERptEn-
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [4c0 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
		VC2:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=2 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
		VC3:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=3 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
	Capabilities: [900 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=26us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [a20 v1] Precision Time Measurement
		PTMCap: Requester:- Responder:+ Root:+
		PTMClockGranularity: 4ns
		PTMControl: Enabled:- RootSelected:-
		PTMEffectiveGranularity: Unknown
	Kernel driver in use: pcieport
	Kernel modules: pci_endpoint_test

01:00.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 43
	Region 0: Memory at 68500000 (32-bit, non-prefetchable) [size=256K]
	Bus: primary=01, secondary=02, subordinate=0a, sec-latency=0
	I/O behind bridge: 68002000-68003fff [size=8K]
	Memory behind bridge: 68100000-684fffff [size=4M]
	Prefetchable memory behind bridge: 0000000068600000-00000000689fffff [size=4M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [68] Express (v2) Upstream Port, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x8, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (strange), Width x1 (downgraded)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: Routing+ 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
			 AtomicOpsCtl: EgressBlck-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8725
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=8
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending- InProgress-
			Port Arbitration Table <?>
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] Multicast
		McastCap: MaxGroups 64, ECRCRegen+
		McastCtl: NumGroups 1, Enable-
		McastBAR: IndexPos 0, BaseAddr 0000000000000000
		McastReceiveVec:      0000000000000000
		McastBlockAllVec:     0000000000000000
		McastBlockUntransVec: 0000000000000000
		McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
	Capabilities: [b00 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport

01:00.1 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
	Subsystem: PLX Technology, Inc. Device 87d0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 0
	Region 0: Memory at 68540000 (32-bit, non-prefetchable) [disabled] [size=8K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [68] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x8, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (strange), Width x1 (downgraded)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, TPHComp-, ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?>
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>

01:00.2 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
	Subsystem: PLX Technology, Inc. Device 87d0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 0
	Region 0: Memory at 68542000 (32-bit, non-prefetchable) [disabled] [size=8K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [68] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x8, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (strange), Width x1 (downgraded)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, TPHComp-, ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?>
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>

01:00.3 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
	Subsystem: PLX Technology, Inc. Device 87d0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 0
	Region 0: Memory at 68544000 (32-bit, non-prefetchable) [disabled] [size=8K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [68] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x8, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (strange), Width x1 (downgraded)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, TPHComp-, ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?>
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>

01:00.4 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca)
	Subsystem: PLX Technology, Inc. Device 87d0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 0
	Region 0: Memory at 68546000 (32-bit, non-prefetchable) [disabled] [size=8K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [68] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed unknown, Width x8, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s (strange), Width x1 (downgraded)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, TPHComp-, ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?>
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>

02:01.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 45
	Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: 68002000-68002fff [size=4K]
	Memory behind bridge: 68100000-682fffff [size=2M]
	Prefetchable memory behind bridge: 0000000068600000-00000000687fffff [size=2M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 0000000001000820  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 5GT/s, Width x2, ASPM L1, Exit Latency L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x1 (downgraded)
			TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise-
			Slot #1, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL+ CmdCplt+ PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, ARIFwd+
			 AtomicOpsCap: Routing+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
			 AtomicOpsCtl: EgressBlck-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8725
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=8
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending- InProgress-
			Port Arbitration Table <?>
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] Multicast
		McastCap: MaxGroups 64, ECRCRegen+
		McastCtl: NumGroups 1, Enable-
		McastBAR: IndexPos 0, BaseAddr 0000000000000000
		McastReceiveVec:      0000000000000000
		McastBlockAllVec:     0000000000000000
		McastBlockUntransVec: 0000000000000000
		McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport

02:02.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 46
	Bus: primary=02, secondary=04, subordinate=05, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: fff00000-000fffff [disabled]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 0000000001000840  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 5GT/s, Width x2, ASPM L1, Exit Latency L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x2 (ok)
			TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #2, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, ARIFwd+
			 AtomicOpsCap: Routing+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
			 AtomicOpsCtl: EgressBlck-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8725
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] Multicast
		McastCap: MaxGroups 64, ECRCRegen+
		McastCtl: NumGroups 1, Enable-
		McastBAR: IndexPos 0, BaseAddr 0000000000000000
		McastReceiveVec:      0000000000000000
		McastBlockAllVec:     0000000000000000
		McastBlockUntransVec: 0000000000000000
		McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport

02:03.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 47
	Bus: primary=02, secondary=06, subordinate=06, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: fff00000-000fffff [disabled]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 0000000001000860  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #3, Speed 5GT/s, Width x2, ASPM L1, Exit Latency L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x0 (downgraded)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #3, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, ARIFwd+
			 AtomicOpsCap: Routing+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
			 AtomicOpsCtl: EgressBlck-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8725
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending+ InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] Multicast
		McastCap: MaxGroups 64, ECRCRegen+
		McastCtl: NumGroups 1, Enable-
		McastBAR: IndexPos 0, BaseAddr 0000000000000000
		McastReceiveVec:      0000000000000000
		McastBlockAllVec:     0000000000000000
		McastBlockUntransVec: 0000000000000000
		McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport

02:04.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 48
	Bus: primary=02, secondary=07, subordinate=07, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: fff00000-000fffff [disabled]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 0000000001000880  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #4, Speed 5GT/s, Width x2, ASPM L1, Exit Latency L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x0 (downgraded)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #4, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, ARIFwd+
			 AtomicOpsCap: Routing+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
			 AtomicOpsCtl: EgressBlck-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8725
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending+ InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] Multicast
		McastCap: MaxGroups 64, ECRCRegen+
		McastCtl: NumGroups 1, Enable-
		McastBAR: IndexPos 0, BaseAddr 0000000000000000
		McastReceiveVec:      0000000000000000
		McastBlockAllVec:     0000000000000000
		McastBlockUntransVec: 0000000000000000
		McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport

02:08.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 49
	Bus: primary=02, secondary=08, subordinate=08, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: fff00000-000fffff [disabled]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 0000000001000900  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #8, Speed 5GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x0 (downgraded)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #8, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, ARIFwd+
			 AtomicOpsCap: Routing+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
			 AtomicOpsCtl: EgressBlck-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8725
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=8
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending+ InProgress-
			Port Arbitration Table <?>
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] Multicast
		McastCap: MaxGroups 64, ECRCRegen+
		McastCtl: NumGroups 1, Enable-
		McastBAR: IndexPos 0, BaseAddr 0000000000000000
		McastReceiveVec:      0000000000000000
		McastBlockAllVec:     0000000000000000
		McastBlockUntransVec: 0000000000000000
		McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport

02:09.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 50
	Bus: primary=02, secondary=09, subordinate=09, sec-latency=0
	I/O behind bridge: 68003000-68003fff [size=4K]
	Memory behind bridge: 68300000-684fffff [size=2M]
	Prefetchable memory behind bridge: 0000000068800000-00000000689fffff [size=2M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 0000000001000920  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #9, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x0 (downgraded)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise-
			Slot #9, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd Off, Power+ Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL+ CmdCplt+ PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, ARIFwd+
			 AtomicOpsCap: Routing+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
			 AtomicOpsCtl: EgressBlck-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8725
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=8
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending+ InProgress-
			Port Arbitration Table <?>
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] Multicast
		McastCap: MaxGroups 64, ECRCRegen+
		McastCtl: NumGroups 1, Enable-
		McastBAR: IndexPos 0, BaseAddr 0000000000000000
		McastReceiveVec:      0000000000000000
		McastBlockAllVec:     0000000000000000
		McastBlockUntransVec: 0000000000000000
		McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport

02:0a.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 51
	Bus: primary=02, secondary=0a, subordinate=0a, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: fff00000-000fffff [disabled]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 0000000001000940  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #10, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (downgraded), Width x0 (downgraded)
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #10, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, NROPrPrP-, LTR+
			 10BitTagComp-, 10BitTagReq-, OBFF Via message, ExtFmt-, EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-, ARIFwd+
			 AtomicOpsCap: Routing+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
			 AtomicOpsCtl: EgressBlck-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8725
	Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
		LaneErrStat: 0
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=1 RefClk=100ns PATEntryBits=1
		Arb:	Fixed+ WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending+ InProgress-
		VC1:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending+ InProgress-
	Capabilities: [e00 v1] Multicast
		McastCap: MaxGroups 64, ECRCRegen+
		McastCtl: NumGroups 1, Enable-
		McastBAR: IndexPos 0, BaseAddr 0000000000000000
		McastReceiveVec:      0000000000000000
		McastBlockAllVec:     0000000000000000
		McastBlockUntransVec: 0000000000000000
		McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport

03:00.0 RAM memory: Xilinx Corporation Device 1016
	Subsystem: Xilinx Corporation Device 3249
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin ? routed to IRQ 551
	Region 0: Memory at 68100000 (32-bit, non-prefetchable) [size=1M]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 0000000001000c00  Data: 0000
	Capabilities: [58] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (ok), Width x1 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Kernel driver in use: hbm-fpga
	Kernel modules: hbm_fpga_core_driver

04:00.0 PCI bridge: Pericom Semiconductor PCI Express to PCI-XPI7C9X130 PCI-X Bridge (rev 04) (prog-if 00 [Normal decode])
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 43
	Bus: primary=04, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled]
	Memory behind bridge: fff00000-000fffff [disabled]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
	Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [80] PCI-X bridge device
		Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=conv
		Status: Dev=04:00.0 64bit- 133MHz- SCD- USC- SCO- SRD-
		Upstream: Capacity=16 CommitmentLimit=16
		Downstream: Capacity=16 CommitmentLimit=16
	Capabilities: [90] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [a8] Subsystem: Device 0000:0000
	Capabilities: [b0] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr+ FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (ok), Width x2 (downgraded)
			TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
	Capabilities: [f0] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [150 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-

Additional information, not sure if it is relevant:

* With yocto dunfell we tried linux-ti-staging as well as linux-ti-mainline, the problem still exists

* With yocto kirstone linux-ti-staging and linux-ti-mainline, the problem still exists

HI,

I have posted below but I am having a similar issue.

e2e.ti.com/.../dra829v-pcie-rescan-fails

I haven't resolved it either.

Is there anything I can do to resolve this?

Best regards,
Takumi

Hi Achal,

From their log, it seems pci_generic_config_read is using, but I think the default one should go to pci_generic_config_read32 instead. Do you think this might be the problem? If so, what situations could cause this?

Kind Regards,

Kevin

hi, Achal

       The following patch can resolve PCIE driver crash issue:

diff --git a/drivers/pci/controller/cadence/pcie-cadence-host.c b/drivers/pci/controller/cadence/pcie-cadence-host.c
index 0a79e553e..584022b3f 100644
--- a/drivers/pci/controller/cadence/pcie-cadence-host.c
+++ b/drivers/pci/controller/cadence/pcie-cadence-host.c
@@ -13,6 +13,8 @@
 
 #include "pcie-cadence.h"
 
+#define LINK_RETRAIN_TIMEOUT (5 * HZ)
+
 static u64 bar_max_size[] = {
 	[RP_BAR0] = _ULL(128 * SZ_2G),
 	[RP_BAR1] = SZ_2G,
@@ -79,6 +81,27 @@ static struct pci_ops cdns_pcie_host_ops = {
 	.write		= pci_generic_config_write,
 };
 
+static int cdns_pcie_host_training_complete(struct cdns_pcie *pcie)
+{
+	u32 pcie_cap_off = CDNS_PCIE_RP_CAP_OFFSET;
+	unsigned long end_jiffies;
+	u16 lnk_stat;
+
+	/* Wait for link training to complete. Exit after timeout. */
+	end_jiffies = jiffies + LINK_RETRAIN_TIMEOUT;
+	do {
+		lnk_stat = cdns_pcie_rp_readw(pcie, pcie_cap_off + PCI_EXP_LNKSTA);
+		if (!(lnk_stat & PCI_EXP_LNKSTA_LT))
+			break;
+		usleep_range(0, 1000);
+	} while (time_before(jiffies, end_jiffies));
+
+	if (!(lnk_stat & PCI_EXP_LNKSTA_LT))
+		return 0;
+
+	return -ETIMEDOUT;
+}
+
 static int cdns_pcie_host_wait_for_link(struct cdns_pcie *pcie)
 {
 	struct device *dev = pcie->dev;
@@ -120,6 +143,9 @@ static int cdns_pcie_retrain(struct cdns_pcie *pcie)
 		cdns_pcie_rp_writew(pcie, pcie_cap_off + PCI_EXP_LNKCTL,
 				    lnk_ctl);
 
+		ret = cdns_pcie_host_training_complete(pcie);
+		if (ret)
+			return ret;
 		ret = cdns_pcie_host_wait_for_link(pcie);
 	}
 	return ret;
diff --git a/drivers/pci/controller/cadence/pcie-cadence.h b/drivers/pci/controller/cadence/pcie-cadence.h
index 56fe4f843..f7a740124 100644
--- a/drivers/pci/controller/cadence/pcie-cadence.h
+++ b/drivers/pci/controller/cadence/pcie-cadence.h
@@ -11,7 +11,7 @@
 #include <linux/phy/phy.h>
 
 /* Parameters for the waiting for link up routine */
-#define LINK_WAIT_MAX_RETRIES	10
+#define LINK_WAIT_MAX_RETRIES	30
 #define LINK_WAIT_USLEEP_MIN	90000
 #define LINK_WAIT_USLEEP_MAX	100000
 
diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
index 875d50c16..4d07bc830 100644
--- a/drivers/pci/setup-res.c
+++ b/drivers/pci/setup-res.c
@@ -20,6 +20,7 @@
 #include <linux/ioport.h>
 #include <linux/cache.h>
 #include <linux/slab.h>
+#include <linux/delay.h>
 #include "pci.h"
 
 static void pci_std_update_resource(struct pci_dev *dev, int resno)
@@ -96,6 +97,8 @@ static void pci_std_update_resource(struct pci_dev *dev, int resno)
 				      cmd & ~PCI_COMMAND_MEMORY);
 	}
 
+	usleep_range(90000, 120000);
+
 	pci_write_config_dword(dev, reg, new);
 	pci_read_config_dword(dev, reg, &check);
 

     Thanks!

Johnson Xu

Hi,

I used Johnson's patch to test connectivity with the TDA4VM EVM and a third party PCIeSW EVM.

However, it did not resolve the error that occurred when accessing the PCIeSW downstream registers.

Best regards,
Takumi

Hi,

I've tested the patch suggest by Johnson and does not work. And as Takumi we have a PCIe Switch.

Regards.

Hi Paulo,

Can you post the full kernel crash log, including all the logs leading up to the kernel crash? We would like to know if the link has gone up before the crash happens.

Regards,

Takuma

Hi Takuma,

Here follows the full log:

U-Boot SPL 2021.01 (May 07 2023 - 03:04:07 +0000)
Model: Texas Instruments K3 J721E SoC
Board: J721EX-PM2-SOM rev A
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
Trying to boot from MMC2
Starting ATF on ARM64 core...

NOTICE:  BL31: v2.8(release):v2.8-896-gc194aa0c6
NOTICE:  BL31: Built : 03:02:04, May  7 2023
I/TC:
I/TC: OP-TEE version: 3.21.0-42-g426790bd3 (gcc version 9.2.1 20191025 (GNU Toolchain for the A-profile A
rchitecture 9.2-2019.12 (arm-9.10))) #1 Sun May  7 03:02:57 UTC 2023 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
I/TC: HUK Initialized
I/TC: Activated SA2UL device
I/TC: Fixing SA2UL firewall owner for GP device
I/TC: Enabled firewalls for SA2UL TRNG device
I/TC: SA2UL TRNG initialized
I/TC: SA2UL Drivers initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2021.01 (May 07 2023 - 03:05:56 +0000)
Model: Texas Instruments K3 J721E SoC
Board: J721EX-PM2-SOM rev A
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
Detected: J7X-BASE-CPB rev A
Detected: J7X-VSC8514-ETH rev E2
Trying to boot from MMC2


U-Boot 2021.01 (May 07 2023 - 03:05:56 +0000)

SoC:   J721E SR1.1 GP
Model: Texas Instruments K3 J721E SoC
Board: J721EX-PM2-SOM rev A
DRAM:  4 GiB
Flash: 0 Bytes
MMC:   sdhci@4f80000: 0, sdhci@4fb0000: 1
Loading Environment from FAT... *** Warning - bad CRC, using default environment

In:    serial@2800000
Out:   serial@2800000
Err:   serial@2800000
am65_cpsw_nuss ethernet@46000000: K3 CPSW: nuss_ver: 0x6BA00101 cpsw_ver: 0x6BA80100 ale_ver: 0x00293904
Ports:1 mdio_freq:1000000
Detected: J7X-BASE-CPB rev A
Detected: J7X-VSC8514-ETH rev E2
cdns,sierra serdes@5000000: sierra probed
Net:   eth0: ethernet@46000000port@1
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc1 is current device
SD/MMC found on device 1
Failed to load 'boot.scr'
574 bytes read in 8 ms (69.3 KiB/s)
Loaded env from uEnv.txt
Importing environment from mmc1 ...
19147264 bytes read in 765 ms (23.9 MiB/s)
111990 bytes read in 14 ms (7.6 MiB/s)
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Loading Device Tree to 000000008fee1000, end 000000008fffffff ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd080]
[    0.000000] Linux version 5.10.168-g991c5ce91e (oe-user@oe-host) (aarch64-oe-linux-gcc (GCC) 9.5.0, GN
U ld (GNU Binutils) 2.34.0.20200910) #1 SMP PREEMPT Wed May 24 11:19:02 UTC 2023
[    0.000000] Machine model: Texas Instruments K3 J721E SoC
[    0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002800000 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a0000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a0000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a0100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@a0100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a1000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a1000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a1100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@a1100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a2000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a2000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a2100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@a2100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a3000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a3000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a3100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@a3100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a4000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a4000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a4100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@a4100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a5000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a5000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a5100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@a5100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a6000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node c66-dma-memory@a6000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a6100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node c66-memory@a6100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a7000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node c66-dma-memory@a7000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a7100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node c66-memory@a7100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a8000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node c71-dma-memory@a8000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a8100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node c71-memory@a8100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000ac000000, size 2 MiB
[    0.000000] OF: reserved mem: initialized node r5f-virtual-eth-queues@ac000000, compatible id shared-d
ma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000ac200000, size 30 MiB
[    0.000000] OF: reserved mem: initialized node r5f-virtual-eth-buffers@ac200000, compatible id shared-
dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000008ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x000000009e7fffff]
[    0.000000]   node   0: [mem 0x000000009e800000-0x00000000a8ffffff]
[    0.000000]   node   0: [mem 0x00000000a9000000-0x00000000a9ffffff]
[    0.000000]   node   0: [mem 0x00000000aa000000-0x00000000abbfffff]
[    0.000000]   node   0: [mem 0x00000000abc00000-0x00000000abffffff]
[    0.000000]   node   0: [mem 0x00000000ac000000-0x00000000adffffff]
[    0.000000]   node   0: [mem 0x00000000ae000000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000880000000-0x00000008ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000008ffffffff]
[    0.000000] cma: Reserved 512 MiB at 0x00000000e0000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 22 pages/cpu s51288 r8192 d30632 u90112
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: EL2 vector hardening
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] CPU features: detected: Spectre-BHB
[    0.000000] CPU features: detected: ARM erratum 1742098
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 1032192
[    0.000000] Kernel command line: console=ttyS2,115200n8 earlycon=ns16550a,mmio32,0x02800000 mtdparts=4
7040000.spi.0:512k(ospi.tiboot3),2m(ospi.tispl),4m(ospi.u-boot),256k(ospi.env),1m(ospi.sysfw),256k(ospi.e
nv.backup),57088k@8m(ospi.rootfs),256k(ospi.phypattern);47034000.hyperbus:512k(hbmc.tiboot3),2m(hbmc.tisp
l),4m(hbmc.u-boot),256k(hbmc.env),1m(hbmc.sysfw),-@8m(hbmc.rootfs) root=PARTUUID=7b02bd7a-02 rw rootfstyp
e=ext4 rootwait
[    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000dc000000-0x00000000e0000000] (64MB)
[    0.000000] Memory: 3271236K/4194304K available (11200K kernel code, 1162K rwdata, 4308K rodata, 1920K
 init, 431K bss, 398780K reserved, 524288K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
[    0.000000]  Trampoline variant of Tasks RCU enabled.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 960 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001900000
[    0.000000] ITS [mem 0x01820000-0x0182ffff]
[    0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
[    0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19
[    0.000000] ITS@0x0000000001820000: allocated 524288 Devices @880800000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: using LPI property table @0x0000000880030000
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000880040000
[    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_n
s: 440795210634 ns
[    0.000002] sched_clock: 56 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns
[    0.008406] Console: colour dummy device 80x25
[    0.012965] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS
 (lpj=800000)
[    0.023637] pid_max: default: 32768 minimum: 301
[    0.028392] LSM: Security Framework initializing
[    0.033146] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.040722] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.049511] rcu: Hierarchical SRCU implementation.
[    0.054568] Platform MSI: msi-controller@1820000 domain created
[    0.060809] PCI/MSI: /bus@100000/interrupt-controller@1800000/msi-controller@1820000 domain created
[    0.070115] EFI services will not be available.
[    0.074834] smp: Bringing up secondary CPUs ...
I/TC: Secondary CPU 1 initializing
I/TC: Secondary CPU 1 switching to normal world boot
[    0.088053] Detected PIPT I-cache on CPU1
[    0.088075] GICv3: CPU1: found redistributor 1 region 0:0x0000000001920000
[    0.088087] GICv3: CPU1: using allocated LPI pending table @0x0000000880050000
[    0.088121] CPU1: Booted secondary processor 0x0000000001 [0x411fd080]
[    0.088169] smp: Brought up 1 node, 2 CPUs
[    0.117515] SMP: Total of 2 processors activated.
[    0.122320] CPU features: detected: 32-bit EL0 Support
[    0.127575] CPU features: detected: CRC32 instructions
[    0.141527] CPU: All CPU(s) started at EL2
[    0.145723] alternatives: patching kernel code
[    0.150733] devtmpfs: initialized
[    0.159528] KASLR disabled due to lack of seed
[    0.164169] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 76450417851000
00 ns
[    0.174132] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    0.187283] pinctrl core: initialized pinctrl subsystem
[    0.192942] DMI not present or invalid.
[    0.197177] NET: Registered protocol family 16
[    0.202372] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations
[    0.209673] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.217724] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.226030] thermal_sys: Registered thermal governor 'step_wise'
[    0.226033] thermal_sys: Registered thermal governor 'power_allocator'
[    0.232637] cpuidle: using governor menu
[    0.243404] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.250361] ASID allocator initialised with 65536 entries
[    0.275349] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.282209] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.289056] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.295903] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.303344] cryptd: max_cpu_qlen set to 1000
[    0.309382] k3-chipinfo 43000014.chipid: Family:J721E rev:SR2.0 JTAGID[0x1bb6402f] Detected
[    0.318200] vsys_3v3: supplied by evm_12v0
[    0.322574] vsys_5v0: supplied by evm_12v0
[    0.327369] iommu: Default domain type: Translated
[    0.332515] SCSI subsystem initialized
[    0.336659] mc: Linux media interface: v0.10
[    0.341082] videodev: Linux video capture interface: v2.00
[    0.346713] pps_core: LinuxPPS API ver. 1 registered
[    0.351787] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.361125] PTP clock support registered
[    0.365143] EDAC MC: Ver: 3.0.0
[    0.368948] FPGA manager framework
[    0.372463] Advanced Linux Sound Architecture Driver Initialized.
[    0.379046] clocksource: Switched to clocksource arch_sys_counter
[    0.385364] VFS: Disk quotas dquot_6.6.0
[    0.389394] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.398883] NET: Registered protocol family 2
[    0.403617] IP idents hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.412214] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[    0.420972] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.429171] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear)
[    0.436968] TCP: Hash tables configured (established 32768 bind 32768)
[    0.443760] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.450655] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.458046] NET: Registered protocol family 1
[    0.462769] RPC: Registered named UNIX socket transport module.
[    0.468823] RPC: Registered udp transport module.
[    0.473627] RPC: Registered tcp transport module.
[    0.478431] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.485015] NET: Registered protocol family 44
[    0.489559] PCI: CLS 0 bytes, default 64
[    0.493925] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    0.504275] Initialise system trusted keyrings
[    0.508886] workingset: timestamp_bits=46 max_order=20 bucket_order=0
[    0.517011] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.523270] NFS: Registering the id_resolver key type
[    0.528458] Key type id_resolver registered
[    0.532729] Key type id_legacy registered
[    0.536846] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.543693] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.551329] 9p: Installing v9fs 9p2000 file system support
[    0.576363] Key type asymmetric registered
[    0.580550] Asymmetric key parser 'x509' registered
[    0.585549] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    0.593109] io scheduler mq-deadline registered
[    0.597734] io scheduler kyber registered
[    0.603478] pinctrl-single 4301c000.pinctrl: 94 pins, size 376
[    0.609654] pinctrl-single 11c000.pinctrl: 173 pins, size 692
[    0.618760] ti-pat 31010000.pat: Found PAT Rev 1.0 with 16384 pages
[    0.625192] debugfs: Directory '31010000.pat' with parent 'regmap' already present!
[    0.633169] ti-pat 31011000.pat: Found PAT Rev 1.0 with 16384 pages
[    0.639588] debugfs: Directory '31011000.pat' with parent 'regmap' already present!
[    0.647525] ti-pat 31012000.pat: Found PAT Rev 1.0 with 16384 pages
[    0.653942] debugfs: Directory '31012000.pat' with parent 'regmap' already present!
[    0.661870] ti-pat 31013000.pat: Found PAT Rev 1.0 with 2048 pages
[    0.668194] debugfs: Directory '31013000.pat' with parent 'regmap' already present!
[    0.676144] ti-pat 31014000.pat: Found PAT Rev 1.0 with 2048 pages
[    0.682469] debugfs: Directory '31014000.pat' with parent 'regmap' already present!
[    0.692204] Serial: 8250/16550 driver, 10 ports, IRQ sharing enabled
[    0.700476] arm-smmu-v3 36600000.iommu: ias 48-bit, oas 48-bit (features 0x00001faf)
[    0.708679] arm-smmu-v3 36600000.iommu: allocated 65536 entries for cmdq
[    0.715709] arm-smmu-v3 36600000.iommu: allocated 32768 entries for evtq
[    0.724155] arm-smmu-v3 36600000.iommu: msi_domain absent - falling back to wired irqs
[    0.737299] brd: module loaded
[    0.743661] loop: module loaded
[    0.747400] megasas: 07.714.04.00-rc1
[    0.753238] tun: Universal TUN/TAP device driver, 1.6
[    0.758703] igbvf: Intel(R) Gigabit Virtual Function Network Driver
[    0.765114] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.771187] sky2: driver version 1.30
[    0.775569] VFIO - User Level meta-driver version: 0.3
[    0.781376] i2c /dev entries driver
[    0.785936] sdhci: Secure Digital Host Controller Interface driver
[    0.792254] sdhci: Copyright(c) Pierre Ossman
[    0.796994] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.803439] ledtrig-cpu: registered to indicate activity on CPUs
[    0.809784] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[    0.817184] optee: probing for conduit method.
I/TC: Reserved shared memory is enabled
I/TC: Dynamic shared memory is enabled
I/TC: Normal World virtualization support is disabled
I/TC: Asynchronous notifications are disabled
[    0.821745] optee: revision 3.21 (426790bd)
[    0.838144] optee: dynamic shared memory is enabled
[    0.847600] optee: initialized driver
[    0.852743] NET: Registered protocol family 17
[    0.857361] 9pnet: Installing 9P2000 support
[    0.861755] Key type dns_resolver registered
[    0.866274] Loading compiled-in X.509 certificates
[    0.879873] ti-sci 44083000.dmsc: ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
[    0.949476] davinci-mcasp 2ba0000.mcasp: IRQ common not found
[    0.957282] omap_i2c 40b00000.i2c: bus 0 rev0.12 at 100 kHz
[    0.963389] omap_i2c 40b10000.i2c: bus 1 rev0.12 at 100 kHz
[    0.969839] omap_i2c 42120000.i2c: bus 2 rev0.12 at 100 kHz
[    0.976321] pca953x 3-0020: supply vcc not found, using dummy regulator
[    0.983142] pca953x 3-0020: using no AI
[    1.011931] pca953x 3-0022: supply vcc not found, using dummy regulator
[    1.018734] pca953x 3-0022: using AI
[    1.023004] gpio-480 (CTRL_PM_I2C_OE): hogged as output/high
[    1.028979] gpio-481 (MCASP/TRACE_MUX_S0): hogged as output/low
[    1.035221] gpio-482 (MCASP/TRACE_MUX_S1): hogged as output/high
[    1.041715] omap_i2c 2000000.i2c: bus 3 rev0.12 at 400 kHz
[    1.048178] omap_i2c 2010000.i2c: bus 4 rev0.12 at 400 kHz
[    1.058267] omap_i2c 2020000.i2c: bus 5 rev0.12 at 400 kHz
[    1.064583] pca953x 6-0020: supply vcc not found, using dummy regulator
[    1.071411] pca953x 6-0020: using no AI
[    1.099747] omap_i2c 2030000.i2c: bus 6 rev0.12 at 400 kHz
[    1.105847] omap_i2c 2040000.i2c: bus 7 rev0.12 at 100 kHz
[    1.111821] omap_i2c 2050000.i2c: bus 8 rev0.12 at 100 kHz
[    1.118098] pca953x 9-0020: supply vcc not found, using dummy regulator
[    1.124917] pca953x 9-0020: using no AI
[    1.151379] omap_i2c 2060000.i2c: bus 9 rev0.12 at 400 kHz
[    1.157840] ti-sci-intr bus@100000:bus@28380000:interrupt-controller2: Interrupt Router 137 domain cre
ated
[    1.167823] ti-sci-intr bus@100000:interrupt-controller0: Interrupt Router 131 domain created
[    1.176634] ti-sci-intr bus@100000:main-navss:interrupt-controller1: Interrupt Router 213 domain creat
ed
[    1.186503] ti-sci-inta 33d00000.interrupt-controller: Interrupt Aggregator domain 209 created
[    1.208734] j721e-pcie 2900000.pcie: host bridge /bus@100000/pcie@2900000 ranges:
[    1.216408] j721e-pcie 2900000.pcie:       IO 0x0010001000..0x0010010fff -> 0x0010001000
[    1.224687] j721e-pcie 2900000.pcie:      MEM 0x0010011000..0x0017ffffff -> 0x0010011000
[    1.232964] j721e-pcie 2900000.pcie:   IB MEM 0x0000000000..0xffffffffffff -> 0x0000000000
[    1.352140] j721e-pcie 2900000.pcie: Link up
[    1.356559] j721e-pcie 2900000.pcie: PCI host bridge to bus 0000:00
[    1.362967] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.368572] pci_bus 0000:00: root bus resource [io  0x0000-0xffff] (bus address [0x10001000-0x10010fff
])
[    1.378262] pci_bus 0000:00: root bus resource [mem 0x10011000-0x17ffffff]
[    1.385316] pci 0000:00:00.0: [104c:b00d] type 01 class 0x060400
[    1.391455] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x4 may corrupt adjacent RW1C
bits
[    1.401404] pci 0000:00:00.0: supports D1
[    1.405499] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    1.411386] pci 0000:00:00.0: reg 0x224: [mem 0x00000000-0x003fffff 64bit]
[    1.418415] pci 0000:00:00.0: VF(n) BAR0 space: [mem 0x00000000-0x00ffffff 64bit] (contains BAR0 for 4
 VFs)
[    1.430435] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.438720] pci 0000:01:00.0: [10b5:8725] type 01 class 0x060400
[    1.444883] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0003ffff]
[    1.451537] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[    1.457865] pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at
 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[    1.473268] pci 0000:01:00.1: [10b5:87d0] type 00 class 0x088000
[    1.479435] pci 0000:01:00.1: reg 0x10: [mem 0x00000000-0x00001fff]
[    1.485917] pci 0000:01:00.1: enabling Extended Tags
[    1.491257] pci 0000:01:00.2: [10b5:87d0] type 00 class 0x088000
[    1.497420] pci 0000:01:00.2: reg 0x10: [mem 0x00000000-0x00001fff]
[    1.503901] pci 0000:01:00.2: enabling Extended Tags
[    1.509236] pci 0000:01:00.3: [10b5:87d0] type 00 class 0x088000
[    1.515399] pci 0000:01:00.3: reg 0x10: [mem 0x00000000-0x00001fff]
[    1.521880] pci 0000:01:00.3: enabling Extended Tags
[    1.527216] pci 0000:01:00.4: [10b5:87d0] type 00 class 0x088000
[    1.533379] pci 0000:01:00.4: reg 0x10: [mem 0x00000000-0x00001fff]
[    1.539862] pci 0000:01:00.4: enabling Extended Tags
[    1.545271] SError Interrupt on CPU1, code 0xbf000000 -- SError
[    1.545273] CPU: 1 PID: 23 Comm: kworker/1:1 Not tainted 5.10.168-g991c5ce91e #1
[    1.545274] Hardware name: Texas Instruments K3 J721E SoC (DT)
[    1.545275] Workqueue: events deferred_probe_work_func
[    1.545278] pstate: 20000085 (nzCv daIf -PAN -UAO -TCO BTYPE=--)
[    1.545279] pc : pci_generic_config_read+0x3c/0xe0
[    1.545280] lr : pci_generic_config_read+0x24/0xe0
[    1.545281] sp : ffff800011653710
[    1.545282] x29: ffff800011653710 x28: ffff000801a88800
[    1.545284] x27: 0000000000000000 x26: 0000000000000001
[    1.545287] x25: ffff80001128bca8 x24: 0000000000000005
[    1.545289] x23: ffff800011653854 x22: 0000000000000087
[    1.545291] x21: ffff000801a89000 x20: 0000000000000004
[    1.545293] x19: ffff800011653794 x18: 0000000000000000
[    1.545295] x17: 0000000000000000 x16: 000000002e48a1e9
[    1.545297] x15: ffff000800144c90 x14: ffffffffffffffff
[    1.545300] x13: ffff000801a9da1c x12: ffff000801a9d326
[    1.545302] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[    1.545304] x9 : 00000000b00d104c x8 : 0000000080b5111d
[    1.545306] x7 : 000000000000ea60 x6 : ffff800011653854
[    1.545308] x5 : ffff000801a88800 x4 : 000000000000000b
[    1.545310] x3 : 0000000000000001 x2 : ffff800013c00008
[    1.545312] x1 : 0000000000000000 x0 : ffff8000116f5000
[    1.545315] Kernel panic - not syncing: Asynchronous SError Interrupt
[    1.545316] CPU: 1 PID: 23 Comm: kworker/1:1 Not tainted 5.10.168-g991c5ce91e #1
[    1.545317] Hardware name: Texas Instruments K3 J721E SoC (DT)
[    1.545318] Workqueue: events deferred_probe_work_func
[    1.545320] Call trace:
[    1.545321]  dump_backtrace+0x0/0x1b8
[    1.545322]  show_stack+0x18/0x38
[    1.545323]  dump_stack+0xe8/0x124
[    1.545324]  panic+0x198/0x360
[    1.545325]  nmi_panic+0xac/0xb0
[    1.545326]  arm64_serror_panic+0x78/0x84
[    1.545327]  do_serror+0x38/0x98
[    1.545328]  el1_error+0x90/0x110
[    1.545329]  pci_generic_config_read+0x3c/0xe0
[    1.545330]  cdns_ti_pcie_config_read+0x18/0x38
[    1.545331]  pci_bus_read_config_dword+0x80/0xe0
[    1.545332]  pci_bus_generic_read_dev_vendor_id+0x34/0x1b0
[    1.545333]  pci_bus_read_dev_vendor_id+0x4c/0x70
[    1.545334]  pci_scan_single_device+0x80/0x100
[    1.545335]  pci_scan_slot+0x74/0x120
[    1.545336]  pci_scan_child_bus_extend+0x54/0x298
[    1.545337]  pci_scan_bridge_extend+0x2a0/0x588
[    1.545338]  pci_scan_child_bus_extend+0x1e4/0x298
[    1.545339]  pci_scan_root_bus_bridge+0x64/0xd8
[    1.545340]  pci_host_probe+0x18/0xc8
[    1.545341]  cdns_pcie_host_setup+0x534/0x970
[    1.545342]  j721e_pcie_probe+0x498/0x828
[    1.545343]  platform_drv_probe+0x54/0xa8
[    1.545344]  really_probe+0xec/0x400
[    1.545345]  driver_probe_device+0x58/0xb8
[    1.545346]  __device_attach_driver+0xb8/0xe0
[    1.545347]  bus_for_each_drv+0x78/0xd0
[    1.545348]  __device_attach+0xfc/0x190
[    1.545349]  device_initial_probe+0x14/0x20
[    1.545350]  bus_probe_device+0x9c/0xa8
[    1.545351]  deferred_probe_work_func+0x88/0xc0
[    1.545351]  process_one_work+0x1bc/0x358
[    1.545352]  worker_thread+0x1f8/0x440
[    1.545353]  kthread+0x140/0x160
[    1.545354]  ret_from_fork+0x10/0x30
[    1.545363] SMP: stopping secondary CPUs
[    1.545364] Kernel Offset: disabled
[    1.545366] CPU features: 0x28040022,20006008
[    1.545366] Memory Limit: none

If you need anything else let me know.

Regards

Hi Paulo,

Based on those logs, they do seem a little different from Johnson's case. Main difference between the case that you are seeing and Johnson is that the link goes down and fails after entering retraining for Johnson's case, but in your case the link is up and I think it is failing at or after the initialization of the 5th function - which I also assume is the last function for the PCIe device being connected.

The reasoning for this is these two lines:

[    1.539862] pci 0000:01:00.4: enabling Extended Tags
[    1.545271] SError Interrupt on CPU1, code 0xbf000000 -- SError

And the picture from the "designware" driver, in which I think there are functions 0 to 4 for the device that is failing to initialize.

Can you confirm/de-confirm if that sounds reasonable?

With this in mind, there is currently a silicon errata for DRA829 regarding PCIe that might be related to this issue:

There was a workaround previously proposed in this separate E2E: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/949996/tda4vm-pcie-endpoint-mode-errors-when-adding-more-than-1-physical-function-to-endpoint-node, and I think my colleague Tanmay also linked a separate E2E in the past that points to this issue.

Could you try out these workaround until we can have a patch integrated into our SDK?

Regards,

Takuma

Hi Takuma,

Our device, has 4 DMA endpoints and those the driver is able to detect, when it goes for the next function (which is a bridge) it fails.

(Not sure if this information helps)

-  Regarding the errata you mention,we've already found it some time ago, and we even look to this patch https://lore.kernel.org/linux-arm-kernel/20230118170454.GA225634@bhelgaas/T/, which sounded similar to us! (Didn't work).

- We even disable the ARI capability on the switch itself by specific hardware registers! (Didn't work)

- The "Designware" driver on sitara board (TMDS64EVM) apperantly had similar issues some years ago [PATCH] PCI: dwc: Added link up check in map_bus of dw_child_pcie_ops (lwn.net) on a different device thought!!,With that we're able to properly find the 4 DMA endpoints and able to pass through the bridge and find the other endpoints. That patch does not apply seamless to cadence driver and we didn't investigate further.

P.S: The link you mention, it is broken for me.

Regards,

Paulo

Hi Paulo,

The Linux kernel patch was a good try, but looking through it, it seems that it was for when the TDA4VM-side is configured as EP since it modifies pcie-cadence-ep.c.

The link that I posted in my previous post that indeed got broken somehow should have a patch for when the TDA4VM is configured as RC. Could you see if this link works and try out the patch there: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/949996/tda4vm-pcie-endpoint-mode-errors-when-adding-more-than-1-physical-function-to-endpoint-node

Regards,

Takuma

Hi Takuma,

I've a look on the patch purposed on the link you suggested:

- The patch does not apply right way, you need to perform some small changes (It seams the patch was for 5.4 and we're with 5.10). I just ugly hacked on top of it:

From 9ce4a3cd145fd563b8bd7f583fa88e85a40acc0f Mon Sep 17 00:00:00 2001
From: Kishon Vijay Abraham I <kishon@ti.com>
Date: Thu, 22 Oct 2020 16:48:09 +0530
Subject: [PATCH] PCI: cadence: Enable ARI on Cadence RP

Since Cadence RP support ARI, set ARI bit in device Capabilities 2
register.

Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com>
---
 .../controller/cadence/pcie-cadence-host.c    |  4 ++++
 drivers/pci/controller/cadence/pcie-cadence.h | 24 +++++++++++++++++++
 2 files changed, 28 insertions(+)

diff --git a/drivers/pci/controller/cadence/pcie-cadence-host.c b/drivers/pci/controller/cadence/pcie-cadence-host.c
index 946ad78b77a4..e379be1fd4f5 100644
--- a/drivers/pci/controller/cadence/pcie-cadence-host.c
+++ b/drivers/pci/controller/cadence/pcie-cadence-host.c
@@ -103,6 +103,10 @@ static int cdns_pcie_host_init_root_port(struct cdns_pcie_rc *rc)
 	cdns_pcie_rp_writeb(pcie, PCI_CLASS_PROG, 0);
 	cdns_pcie_rp_writew(pcie, PCI_CLASS_DEVICE, PCI_CLASS_BRIDGE_PCI);
 
+	value = cdns_pcie_rp_readl(pcie, CDNS_PCIE_RP_PCIE_CAP2);
+	value |= PCI_EXP_DEVCAP2_ARI;
+	cdns_pcie_rp_writel(pcie, CDNS_PCIE_RP_PCIE_CAP2, value);
+
 	return 0;
 }
 
diff --git a/drivers/pci/controller/cadence/pcie-cadence.h b/drivers/pci/controller/cadence/pcie-cadence.h
index 54435689b25e..818b34a45d1e 100644
--- a/drivers/pci/controller/cadence/pcie-cadence.h
+++ b/drivers/pci/controller/cadence/pcie-cadence.h
@@ -111,6 +111,7 @@
  */
 #define CDNS_PCIE_RP_BASE	0x00200000
 
+#define CDNS_PCIE_RP_PCIE_CAP2	0xe4
 
 /*
  * Address Translation Registers
@@ -403,6 +404,29 @@ static inline void cdns_pcie_rp_writew(struct cdns_pcie *pcie,
 	writew(value, addr);
 }
 
+static inline void cdns_pcie_rp_writel(struct cdns_pcie *pcie,
+				       u32 reg, u16 value)
+{
+	void __iomem *addr = pcie->reg_base + CDNS_PCIE_RP_BASE + reg;
+
+	/*if (pcie->ops && pcie->ops->write) {*/
+		cdns_pcie_write_sz(addr, 0x4, value);
+		/*return;*/
+	/*}*/
+
+	/*writel(value, addr);*/
+}
+
+static inline u32 cdns_pcie_rp_readl(struct cdns_pcie *pcie, u32 reg)
+{
+	void __iomem *addr = pcie->reg_base + CDNS_PCIE_RP_BASE + reg;
+
+	/*if (pcie->ops && pcie->ops->read)*/
+		return cdns_pcie_read_sz(addr, 0x4);
+
+	/*return readl(addr);*/
+}
+
 /* Endpoint Function register access */
 static inline void cdns_pcie_ep_fn_writeb(struct cdns_pcie *pcie, u8 fn,
 					  u32 reg, u8 value)
-- 
2.17.1

- Anyway purposed patch does not work, here it goes the log:

U-Boot SPL 2021.01 (Jun 06 2023 - 23:18:19 +0100)
Model: Texas Instruments K3 J721E SoC
Board: J721EX-PM2-SOM rev A
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
Trying to boot from MMC2
Starting ATF on ARM64 core...

NOTICE:  BL31: v2.9(release):v2.9.0-84-ga0f3b552c
NOTICE:  BL31: Built : 23:16:11, Jun  6 2023
I/TC:
I/TC: OP-TEE version: 3.21.0-176-g145953d55 (gcc version 9.2.1 20191025 (GNU Toolchain for the A-profile
Architecture 9.2-2019.12 (arm-9.10))) #1 Tue Jun  6 22:17:08 UTC 2023 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
I/TC: HUK Initialized
I/TC: Activated SA2UL device
I/TC: Fixing SA2UL firewall owner for GP device
I/TC: Enabled firewalls for SA2UL TRNG device
I/TC: SA2UL TRNG initialized
I/TC: SA2UL Drivers initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2021.01 (Jun 06 2023 - 23:20:23 +0100)
Model: Texas Instruments K3 J721E SoC
Board: J721EX-PM2-SOM rev A
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
Detected: J7X-BASE-CPB rev A
Detected: J7X-VSC8514-ETH rev E2
Trying to boot from MMC2


U-Boot 2021.01 (Jun 06 2023 - 23:20:23 +0100)

SoC:   J721E SR1.1 GP
Model: Texas Instruments K3 J721E SoC
Board: J721EX-PM2-SOM rev A
DRAM:  4 GiB
Flash: 0 Bytes
MMC:   sdhci@4f80000: 0, sdhci@4fb0000: 1
Loading Environment from FAT... *** Warning - bad CRC, using default environment

In:    serial@2800000
Out:   serial@2800000
Err:   serial@2800000
am65_cpsw_nuss ethernet@46000000: K3 CPSW: nuss_ver: 0x6BA00101 cpsw_ver: 0x6BA80100 ale_ver: 0x00293904
Ports:1 mdio_freq:1000000
Detected: J7X-BASE-CPB rev A
Detected: J7X-VSC8514-ETH rev E2
cdns,sierra serdes@5000000: sierra probed
Net:   eth0: ethernet@46000000port@1
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc1 is current device
SD/MMC found on device 1
Failed to load 'boot.scr'
574 bytes read in 8 ms (69.3 KiB/s)
Loaded env from uEnv.txt
Importing environment from mmc1 ...
19147264 bytes read in 766 ms (23.8 MiB/s)
111990 bytes read in 14 ms (7.6 MiB/s)
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Loading Device Tree to 000000008fee1000, end 000000008fffffff ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd080]
[    0.000000] Linux version 5.10.168-g991c5ce91e (oe-user@oe-host) (aarch64-oe-linux-gcc (GCC) 9.5.0, GN
U ld (GNU Binutils) 2.34.0.20200910) #1 SMP PREEMPT Sun Jun 11 08:09:39 UTC 2023
[    0.000000] Machine model: Texas Instruments K3 J721E SoC
[    0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002800000 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a0000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a0000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a0100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@a0100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a1000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a1000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a1100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@a1100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a2000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a2000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a2100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@a2100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a3000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a3000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a3100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@a3100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a4000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a4000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a4100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@a4100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a5000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@a5000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a5100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-memory@a5100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a6000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node c66-dma-memory@a6000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a6100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node c66-memory@a6100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a7000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node c66-dma-memory@a7000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a7100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node c66-memory@a7100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a8000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node c71-dma-memory@a8000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a8100000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node c71-memory@a8100000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000ac000000, size 2 MiB
[    0.000000] OF: reserved mem: initialized node r5f-virtual-eth-queues@ac000000, compatible id shared-d
ma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000ac200000, size 30 MiB
[    0.000000] OF: reserved mem: initialized node r5f-virtual-eth-buffers@ac200000, compatible id shared-
dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000008ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x000000009e7fffff]
[    0.000000]   node   0: [mem 0x000000009e800000-0x00000000a8ffffff]
[    0.000000]   node   0: [mem 0x00000000a9000000-0x00000000a9ffffff]
[    0.000000]   node   0: [mem 0x00000000aa000000-0x00000000abbfffff]
[    0.000000]   node   0: [mem 0x00000000abc00000-0x00000000abffffff]
[    0.000000]   node   0: [mem 0x00000000ac000000-0x00000000adffffff]
[    0.000000]   node   0: [mem 0x00000000ae000000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000880000000-0x00000008ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000008ffffffff]
[    0.000000] cma: Reserved 512 MiB at 0x00000000e0000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 22 pages/cpu s51288 r8192 d30632 u90112
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: EL2 vector hardening
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] CPU features: detected: Spectre-BHB
[    0.000000] CPU features: detected: ARM erratum 1742098
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 1032192
[    0.000000] Kernel command line: console=ttyS2,115200n8 earlycon=ns16550a,mmio32,0x02800000 mtdparts=4
7040000.spi.0:512k(ospi.tiboot3),2m(ospi.tispl),4m(ospi.u-boot),256k(ospi.env),1m(ospi.sysfw),256k(ospi.e
nv.backup),57088k@8m(ospi.rootfs),256k(ospi.phypattern);47034000.hyperbus:512k(hbmc.tiboot3),2m(hbmc.tisp
l),4m(hbmc.u-boot),256k(hbmc.env),1m(hbmc.sysfw),-@8m(hbmc.rootfs) root=PARTUUID=82d1b7da-02 rw rootfstyp
e=ext4 rootwait
[    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000dc000000-0x00000000e0000000] (64MB)
[    0.000000] Memory: 3271236K/4194304K available (11200K kernel code, 1162K rwdata, 4308K rodata, 1920K
 init, 431K bss, 398780K reserved, 524288K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
[    0.000000]  Trampoline variant of Tasks RCU enabled.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 960 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001900000
[    0.000000] ITS [mem 0x01820000-0x0182ffff]
[    0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
[    0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19
[    0.000000] ITS@0x0000000001820000: allocated 524288 Devices @880800000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: using LPI property table @0x0000000880030000
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000880040000
[    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_n
s: 440795210634 ns
[    0.000002] sched_clock: 56 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns
[    0.008392] Console: colour dummy device 80x25
[    0.012952] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS
 (lpj=800000)
[    0.023623] pid_max: default: 32768 minimum: 301
[    0.028379] LSM: Security Framework initializing
[    0.033133] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.040709] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.049502] rcu: Hierarchical SRCU implementation.
[    0.054563] Platform MSI: msi-controller@1820000 domain created
[    0.060803] PCI/MSI: /bus@100000/interrupt-controller@1800000/msi-controller@1820000 domain created
[    0.070106] EFI services will not be available.
[    0.074824] smp: Bringing up secondary CPUs ...
I/TC: Secondary CPU 1 initializing
I/TC: Secondary CPU 1 switching to normal world boot
[    0.088048] Detected PIPT I-cache on CPU1
[    0.088071] GICv3: CPU1: found redistributor 1 region 0:0x0000000001920000
[    0.088082] GICv3: CPU1: using allocated LPI pending table @0x0000000880050000
[    0.088117] CPU1: Booted secondary processor 0x0000000001 [0x411fd080]
[    0.088162] smp: Brought up 1 node, 2 CPUs
[    0.117507] SMP: Total of 2 processors activated.
[    0.122312] CPU features: detected: 32-bit EL0 Support
[    0.127567] CPU features: detected: CRC32 instructions
[    0.141521] CPU: All CPU(s) started at EL2
[    0.145718] alternatives: patching kernel code
[    0.150720] devtmpfs: initialized
[    0.159572] KASLR disabled due to lack of seed
[    0.164214] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 76450417851000
00 ns
[    0.174178] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    0.187045] pinctrl core: initialized pinctrl subsystem
[    0.192706] DMI not present or invalid.
[    0.196949] NET: Registered protocol family 16
[    0.202152] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations
[    0.209454] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.217505] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.225805] thermal_sys: Registered thermal governor 'step_wise'
[    0.225807] thermal_sys: Registered thermal governor 'power_allocator'
[    0.232409] cpuidle: using governor menu
[    0.243179] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.250136] ASID allocator initialised with 65536 entries
[    0.274927] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.281787] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.288634] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.295481] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.302906] cryptd: max_cpu_qlen set to 1000
[    0.308948] k3-chipinfo 43000014.chipid: Family:J721E rev:SR2.0 JTAGID[0x1bb6402f] Detected
[    0.317762] vsys_3v3: supplied by evm_12v0
[    0.322131] vsys_5v0: supplied by evm_12v0
[    0.326901] iommu: Default domain type: Translated
[    0.332068] SCSI subsystem initialized
[    0.336211] mc: Linux media interface: v0.10
[    0.340613] videodev: Linux video capture interface: v2.00
[    0.346244] pps_core: LinuxPPS API ver. 1 registered
[    0.351317] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.360655] PTP clock support registered
[    0.364673] EDAC MC: Ver: 3.0.0
[    0.368462] FPGA manager framework
[    0.371979] Advanced Linux Sound Architecture Driver Initialized.
[    0.378562] clocksource: Switched to clocksource arch_sys_counter
[    0.384876] VFS: Disk quotas dquot_6.6.0
[    0.388906] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.398377] NET: Registered protocol family 2
[    0.403109] IP idents hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.411677] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[    0.420435] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.428634] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear)
[    0.436430] TCP: Hash tables configured (established 32768 bind 32768)
[    0.443224] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.450120] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.457518] NET: Registered protocol family 1
[    0.462235] RPC: Registered named UNIX socket transport module.
[    0.468297] RPC: Registered udp transport module.
[    0.473101] RPC: Registered tcp transport module.
[    0.477904] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.484486] NET: Registered protocol family 44
[    0.489031] PCI: CLS 0 bytes, default 64
[    0.493395] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    0.503771] Initialise system trusted keyrings
[    0.508395] workingset: timestamp_bits=46 max_order=20 bucket_order=0
[    0.516547] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.522772] NFS: Registering the id_resolver key type
[    0.527950] Key type id_resolver registered
[    0.532222] Key type id_legacy registered
[    0.536338] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.543186] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.550824] 9p: Installing v9fs 9p2000 file system support
[    0.576529] Key type asymmetric registered
[    0.580714] Asymmetric key parser 'x509' registered
[    0.585711] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    0.593270] io scheduler mq-deadline registered
[    0.597896] io scheduler kyber registered
[    0.603627] pinctrl-single 4301c000.pinctrl: 94 pins, size 376
[    0.609806] pinctrl-single 11c000.pinctrl: 173 pins, size 692
[    0.618882] ti-pat 31010000.pat: Found PAT Rev 1.0 with 16384 pages
[    0.625304] debugfs: Directory '31010000.pat' with parent 'regmap' already present!
[    0.633281] ti-pat 31011000.pat: Found PAT Rev 1.0 with 16384 pages
[    0.639701] debugfs: Directory '31011000.pat' with parent 'regmap' already present!
[    0.647638] ti-pat 31012000.pat: Found PAT Rev 1.0 with 16384 pages
[    0.654055] debugfs: Directory '31012000.pat' with parent 'regmap' already present!
[    0.661980] ti-pat 31013000.pat: Found PAT Rev 1.0 with 2048 pages
[    0.668309] debugfs: Directory '31013000.pat' with parent 'regmap' already present!
[    0.676250] ti-pat 31014000.pat: Found PAT Rev 1.0 with 2048 pages
[    0.682579] debugfs: Directory '31014000.pat' with parent 'regmap' already present!
[    0.692262] Serial: 8250/16550 driver, 10 ports, IRQ sharing enabled
[    0.700491] arm-smmu-v3 36600000.iommu: ias 48-bit, oas 48-bit (features 0x00001faf)
[    0.708709] arm-smmu-v3 36600000.iommu: allocated 65536 entries for cmdq
[    0.715736] arm-smmu-v3 36600000.iommu: allocated 32768 entries for evtq
[    0.724128] arm-smmu-v3 36600000.iommu: msi_domain absent - falling back to wired irqs
[    0.737295] brd: module loaded
[    0.743696] loop: module loaded
[    0.747423] megasas: 07.714.04.00-rc1
[    0.753210] tun: Universal TUN/TAP device driver, 1.6
[    0.758705] igbvf: Intel(R) Gigabit Virtual Function Network Driver
[    0.765110] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.771187] sky2: driver version 1.30
[    0.775547] VFIO - User Level meta-driver version: 0.3
[    0.781333] i2c /dev entries driver
[    0.785867] sdhci: Secure Digital Host Controller Interface driver
[    0.792190] sdhci: Copyright(c) Pierre Ossman
[    0.796929] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.803370] ledtrig-cpu: registered to indicate activity on CPUs
[    0.809711] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[    0.817113] optee: probing for conduit method.
I/TC: Reserved shared memory is enabled
I/TC: Dynamic shared memory is enabled
I/TC: Normal World virtualization support is disabled
I/TC: Asynchronous notifications are disabled
[    0.821679] optee: revision 3.21 (145953d5)
[    0.838082] optee: dynamic shared memory is enabled
[    0.847537] optee: initialized driver
[    0.852682] NET: Registered protocol family 17
[    0.857301] 9pnet: Installing 9P2000 support
[    0.861695] Key type dns_resolver registered
[    0.866214] Loading compiled-in X.509 certificates
[    0.879875] ti-sci 44083000.dmsc: ABI: 3.1 (firmware rev 0x0008 '8.6.3--v08.06.03 (Chill Capybar')
[    0.949085] davinci-mcasp 2ba0000.mcasp: IRQ common not found
[    0.956863] omap_i2c 40b00000.i2c: bus 0 rev0.12 at 100 kHz
[    0.962966] omap_i2c 40b10000.i2c: bus 1 rev0.12 at 100 kHz
[    0.969397] omap_i2c 42120000.i2c: bus 2 rev0.12 at 100 kHz
[    0.975887] pca953x 3-0020: supply vcc not found, using dummy regulator
[    0.982706] pca953x 3-0020: using no AI
[    1.011443] pca953x 3-0022: supply vcc not found, using dummy regulator
[    1.018248] pca953x 3-0022: using AI
[    1.022516] gpio-480 (CTRL_PM_I2C_OE): hogged as output/high
[    1.028492] gpio-481 (MCASP/TRACE_MUX_S0): hogged as output/low
[    1.034734] gpio-482 (MCASP/TRACE_MUX_S1): hogged as output/high
[    1.041228] omap_i2c 2000000.i2c: bus 3 rev0.12 at 400 kHz
[    1.047694] omap_i2c 2010000.i2c: bus 4 rev0.12 at 400 kHz
[    1.057761] omap_i2c 2020000.i2c: bus 5 rev0.12 at 400 kHz
[    1.064080] pca953x 6-0020: supply vcc not found, using dummy regulator
[    1.070906] pca953x 6-0020: using no AI
[    2.082723] pca953x 6-0020: failed writing register
[    2.087763] pca953x: probe of 6-0020 failed with error -16
[    2.093793] omap_i2c 2030000.i2c: bus 6 rev0.12 at 400 kHz
[    2.099905] omap_i2c 2040000.i2c: bus 7 rev0.12 at 100 kHz
[    2.105904] omap_i2c 2050000.i2c: bus 8 rev0.12 at 100 kHz
[    2.112184] pca953x 9-0020: supply vcc not found, using dummy regulator
[    2.118999] pca953x 9-0020: using no AI
[    2.146890] omap_i2c 2060000.i2c: bus 9 rev0.12 at 400 kHz
[    2.153355] ti-sci-intr bus@100000:bus@28380000:interrupt-controller2: Interrupt Router 137 domain cre
ated
[    2.163337] ti-sci-intr bus@100000:interrupt-controller0: Interrupt Router 131 domain created
[    2.172149] ti-sci-intr bus@100000:main-navss:interrupt-controller1: Interrupt Router 213 domain creat
ed
[    2.182017] ti-sci-inta 33d00000.interrupt-controller: Interrupt Aggregator domain 209 created
[    2.204258] j721e-pcie 2900000.pcie: host bridge /bus@100000/pcie@2900000 ranges:
[    2.211932] j721e-pcie 2900000.pcie:       IO 0x0010001000..0x0010010fff -> 0x0010001000
[    2.220209] j721e-pcie 2900000.pcie:      MEM 0x0010011000..0x0017ffffff -> 0x0010011000
[    2.228485] j721e-pcie 2900000.pcie:   IB MEM 0x0000000000..0xffffffffffff -> 0x0000000000
[    2.347759] j721e-pcie 2900000.pcie: Link up
[    2.352178] j721e-pcie 2900000.pcie: PCI host bridge to bus 0000:00
[    2.358586] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.364191] pci_bus 0000:00: root bus resource [io  0x0000-0xffff] (bus address [0x10001000-0x10010fff
])
[    2.373883] pci_bus 0000:00: root bus resource [mem 0x10011000-0x17ffffff]
[    2.380937] pci 0000:00:00.0: [104c:b00d] type 01 class 0x060400
[    2.387078] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x4 may corrupt adjacent RW1C
bits
[    2.397027] pci 0000:00:00.0: supports D1
[    2.401123] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    2.407010] pci 0000:00:00.0: reg 0x224: [mem 0x00000000-0x003fffff 64bit]
[    2.414039] pci 0000:00:00.0: VF(n) BAR0 space: [mem 0x00000000-0x00ffffff 64bit] (contains BAR0 for 4
 VFs)
[    2.426011] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.434298] pci 0000:01:00.0: [10b5:8725] type 01 class 0x060400
[    2.440462] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0003ffff]
[    2.447118] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[    2.453464] pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at
 0000:00:00.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[    2.468867] pci 0000:01:00.1: [10b5:87d0] type 00 class 0x088000
[    2.475035] pci 0000:01:00.1: reg 0x10: [mem 0x00000000-0x00001fff]
[    2.481517] pci 0000:01:00.1: enabling Extended Tags
[    2.486853] pci 0000:01:00.2: [10b5:87d0] type 00 class 0x088000
[    2.493017] pci 0000:01:00.2: reg 0x10: [mem 0x00000000-0x00001fff]
[    2.499498] pci 0000:01:00.2: enabling Extended Tags
[    2.504834] pci 0000:01:00.3: [10b5:87d0] type 00 class 0x088000
[    2.510997] pci 0000:01:00.3: reg 0x10: [mem 0x00000000-0x00001fff]
[    2.517479] pci 0000:01:00.3: enabling Extended Tags
[    2.522816] pci 0000:01:00.4: [10b5:87d0] type 00 class 0x088000
[    2.528980] pci 0000:01:00.4: reg 0x10: [mem 0x00000000-0x00001fff]
[    2.535463] pci 0000:01:00.4: enabling Extended Tags
[    2.540871] SError Interrupt on CPU1, code 0xbf000000 -- SError
[    2.540873] CPU: 1 PID: 23 Comm: kworker/1:1 Not tainted 5.10.168-g991c5ce91e #1
[    2.540874] Hardware name: Texas Instruments K3 J721E SoC (DT)
[    2.540876] Workqueue: events deferred_probe_work_func
[    2.540878] pstate: 20000085 (nzCv daIf -PAN -UAO -TCO BTYPE=--)
[    2.540879] pc : pci_generic_config_read+0x3c/0xe0
[    2.540880] lr : pci_generic_config_read+0x24/0xe0
[    2.540881] sp : ffff800011653710
[    2.540882] x29: ffff800011653710 x28: ffff0008019df000
[    2.540884] x27: 0000000000000000 x26: 0000000000000001
[    2.540886] x25: ffff80001128bca8 x24: 0000000000000005
[    2.540888] x23: ffff800011653854 x22: 0000000000000087
[    2.540891] x21: ffff0008019df800 x20: 0000000000000004
[    2.540893] x19: ffff800011653794 x18: 0000000000000000
[    2.540895] x17: 0000000000000000 x16: 00000000207c5d25
[    2.540897] x15: ffff000800144c90 x14: ffffffffffffffff
[    2.540899] x13: ffff000801a95a1c x12: ffff000801a95326
[    2.540901] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
[    2.540903] x9 : 00000000b00d104c x8 : 0000000080b5111d
[    2.540905] x7 : 000000000000ea60 x6 : ffff800011653854
[    2.540907] x5 : ffff0008019df000 x4 : 000000000000000b
[    2.540909] x3 : 0000000000000001 x2 : ffff800013c00008
[    2.540911] x1 : 0000000000000000 x0 : ffff8000116f5000
[    2.540914] Kernel panic - not syncing: Asynchronous SError Interrupt
[    2.540915] CPU: 1 PID: 23 Comm: kworker/1:1 Not tainted 5.10.168-g991c5ce91e #1
[    2.540917] Hardware name: Texas Instruments K3 J721E SoC (DT)
[    2.540918] Workqueue: events deferred_probe_work_func
[    2.540919] Call trace:
[    2.540920]  dump_backtrace+0x0/0x1b8
[    2.540921]  show_stack+0x18/0x38
[    2.540922]  dump_stack+0xe8/0x124
[    2.540923]  panic+0x198/0x360
[    2.540924]  nmi_panic+0xac/0xb0
[    2.540925]  arm64_serror_panic+0x78/0x84
[    2.540926]  do_serror+0x38/0x98
[    2.540927]  el1_error+0x90/0x110
[    2.540928]  pci_generic_config_read+0x3c/0xe0
[    2.540929]  cdns_ti_pcie_config_read+0x18/0x38
[    2.540930]  pci_bus_read_config_dword+0x80/0xe0
[    2.540931]  pci_bus_generic_read_dev_vendor_id+0x34/0x1b0
[    2.540932]  pci_bus_read_dev_vendor_id+0x4c/0x70
[    2.540933]  pci_scan_single_device+0x80/0x100
[    2.540934]  pci_scan_slot+0x74/0x120
[    2.540935]  pci_scan_child_bus_extend+0x54/0x298
[    2.540936]  pci_scan_bridge_extend+0x2a0/0x588
[    2.540937]  pci_scan_child_bus_extend+0x1e4/0x298
[    2.540938]  pci_scan_root_bus_bridge+0x64/0xd8
[    2.540939]  pci_host_probe+0x18/0xc8
[    2.540940]  cdns_pcie_host_setup+0x584/0x970
[    2.540941]  j721e_pcie_probe+0x498/0x828
[    2.540942]  platform_drv_probe+0x54/0xa8
[    2.540943]  really_probe+0xec/0x400
[    2.540944]  driver_probe_device+0x58/0xb8
[    2.540945]  __device_attach_driver+0xb8/0xe0
[    2.540945]  bus_for_each_drv+0x78/0xd0
[    2.540946]  __device_attach+0xfc/0x190
[    2.540947]  device_initial_probe+0x14/0x20
[    2.540948]  bus_probe_device+0x9c/0xa8
[    2.540949]  deferred_probe_work_func+0x88/0xc0
[    2.540949]  process_one_work+0x1bc/0x358
[    2.540950]  worker_thread+0x1f8/0x440
[    2.540951]  kthread+0x140/0x160
[    2.540952]  ret_from_fork+0x10/0x30
[    2.540963] SMP: stopping secondary CPUs
[    2.540964] Kernel Offset: disabled
[    2.540965] CPU features: 0x28040022,20006008
[    2.540966] Memory Limit: none

Regards,

Paulo

Hi Paulo,

Got it. Can I get around a week to look into this deeper? My suspicion is still with errata i2086, mainly due to the behavior where the initialization of the last function are the last logs for the kernel crash. It could be that with the latest SDK some things have changed that makes the old patches no longer valid.

Regards,

Takuma

Hi Takuma,

Sure, appreciated! If you need test let me know!

Regards,

Paulo 

Hi Paulo,

As an update, I ordered a PCIe switch with two ports which should come in around a week on June 20, so that I can see if I can reproduce the issue on our EVM board. Additionally, I ordered a generic PCIe device that should exhibit the issue outlined in errata i2086 which should arrive tomorrow - mainly for testing our old patches with new SDKs - this will be a temporary way to potentially reproduce the issue while I wait for the PCIe switch to get delivered.

Regards,

Takuma

Hi Takuma,

Sounds great!  Looking forward.

Thanks for the support,

Paulo

Hi Paulo,

As an update, I received the PCIe switch card today and will be starting initial experiments with the switch.

Regards,

Takuma

Hi Paulo,

We found some interesting behaviors for the PCIe switch. When using the DRA829V TI EVM, we also see some kernel errors panic.

However, when using a similar processor, the DRA821 TI EVM, the board is able to enumerate the PCIe switch. Issue looks to be very specific to the DRA829V EVM, and when using a PCIe switch. This is most likely a separate issue from the silicon errata I suggested in the past, since the switch I have obtained does not enumerate as having multiple physical functions.

We will be looking into this and I will keep you updated if we find any other interesting results/experiments/fixes. If you do not hear back from me and want an update, please let me know by posting in this E2E and I will let you know the latest.

Regards,

Takuma

Hi Takuma,

Thank you for being looking at this and at same time replicating the behavior!

Well, unfortunately the DRA821 does not have C7 DSPs which does suit us, otherwise we could tried that one! For that reason I'll appreciate yours update on the status.

Regards,

Paulo 

Hi Paulo,

Thank you for your patience. We are still in the process of debugging the issue - although we do have a few ideas on where the issue could be. As mentioned yesterday, we have experimented on DRA821 and found that the PCIe switch is detected correctly. Some new findings we had was that for AM69A, another device that has the same PHY as DRA821 also detects the switch correctly.

Based on this information, I think the issue is specific to DRA829/TDA4VM, since this device has a different PHY, and different driver for the PHY. The difference in drivers are quite large so it will take a few days to list up the differences and do some experiments to further narrow down where the issue could be. Again, thank you for your patience.

Regards,

Takuma

Hi Takuma,

Ok! Good to know!
Yes, I understand that this can take some time!

Thanks for the updates,

Paulo

Hi Paulo,

Thank you for your continued patience.

Regards,

Takuma

Hi Takuma,

An update from our side.

Today we performed some tests with the J784S4XG01EVM development board and the PCIe enumeration was working out of the box.

We just tried the prebuilt image from linux sdk.

Regards,

Paulo

Hi Paulo,

Thanks for the update! 

The behavior seen on your end is the same as the behavior seen on our end. Issue is specifically for J721E/TDA4VM/DRA829 with a PCIe switch. From some quick tests I did, all other platforms including J7200/DRA821, J784S4/AM69, and J721S2/AM68 do not exhibit the kernel panic issue.

The main difference between the platforms that work and does not work is a difference in PHY for SERDES. The issues might be in the PHY specific code, or the difference in PHY is causing some unexpected behavior in code that is shared between the different platform. We are still in the process of narrowing down where the issue could be.

Regards,

Takuma

Hello Takuma,

Thank you for all the effort!!!

On behalf of Paulo which is not available at the moment. I carried on with tests.
The patch that you provided to us works properly. Now we are able enumerate the PCI bridge installed.

For your information, for this test we used the yocto-kirkstone so we needed to modify slightly the patch.

We are aware that the patch is a workaround but considered this thread as closed.

Regards,
Joao Lima

Hi Joao,

Awesome! That is great to hear.

Regards,

Takuma