PCI root complex differences between DM8148 and DM8168 ?

Christian Schroeder

Hello all, hello dear PSP team

I'm are trying to integrate a third party pcie ip (lancero, www.logicandmore.com/Lancero.html)
to connect an arria2-gx FPGA to the dm8148 using pci express Gen1/x1.
I tried the same endpoint device and the same driver with EZSDK 5.03 on 2 different boards,
DM8168 EVK (DIMM3) and DM8148 EVK. The main difference is that the test on the DM8168 board
runs (still with limitations) but with the DM8148 board we get always a complete board freeze
immediately after we started the test applicaltion (pure loading of the driver works in both
cases). I'm using MSI interrupts.

Are there any undocumented differences between the pcie root complexes for the EZSDK 5_03_00_09
in case of DM8168 and DM 8148 or are there any known issues on the DM8148 EVK ??? We suspect
some interrupt issue but board freezes are really hard to debug ...

Any comments

Greetings

Christian

over 13 years ago

0 HemantPedanekar over 13 years ago

TI__Expert 5685 points

Christian,

There is no difference apart from DM8148 having x1 and DM8168 having x2 link width.

Can you provide more information such as logs (refer Root Complex Driver User Guide), lspci -xvv output on both setups , what is the driver for endpoint you are loading? Is it some custom driver or already present in kernel?

Hemant

0 Christian Schroeder over 13 years ago in reply to HemantPedanekar

Prodigy 50 points

Dear Hemant,

the setups are completely identical for dm8148 and dm8168. We are using the Lancero SGDMA driver as endpoint (http://www.logicandmore.com/Lancero.html). The problem we have can be described the following way:

The board freezes after initiating any DMA transfer. We disabled the interrupts after the first interrupt that occurs. After loading the driver we ran a very small DMA transfer and the 8148 is now stuck in an endless interrupt loop (the DM8168 does not and works stable):

schedule_work(engine=0xcbcd8300->work))
Disabling interrupts after IRQ #0.
schedule_work(engine=0xcbcd8300->work))
Disabling interrupts after IRQ #0.
schedule_work(engine=0xcbcd8300->work))
Disabling interrupts after IRQ #0.
schedule_work(engine=0xcbcd8300->work))
Disabling interrupts after IRQ #0.

Could it be that the MSI interrupts are configured level-triggered instead of edge triggered ? Is there some difference in PCIe initialization (e.g. by u-boot) ?

lspci -xvv: (see log)

Fullscreen 8750.dm8148.log Download

root@dm814x-evm:~# lspci -xvv
00:00.0 Class 0604: Device 104c:8888 (rev 01)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Region 0: Memory at <ignored> (32-bit, non-prefetchable)
        Region 1: Memory at <ignored> (32-bit, prefetchable)
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        Memory behind bridge: 20000000-22ffffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Latency L0 <2us, L1 <64us
                        ClockPM- Surprise- LLActRep+ BwNot-
                LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
00: 4c 10 88 88 47 01 10 00 01 00 04 06 10 00 01 00
10: 00 00 00 51 08 00 00 80 00 01 01 00 f0 00 00 00
20: 00 20 f0 22 f0 ff 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 30 01 01 00

01:00.0 Class ff00: Device 1172:0004 (rev 01)
        Subsystem: Device 1172:0004
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 48
        Region 0: Memory at 20000000 (32-bit, non-prefetchable) [disabled] [size=32M]
        Region 1: Memory at 22000000 (32-bit, non-prefetchable) [disabled] [size=64K]
        Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [78] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [80] Express (v1) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s, Latency L0 unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        Capabilities: [100 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
00: 72 11 04 00 40 01 10 00 01 00 00 ff 10 00 00 00
10: 00 00 00 20 00 00 00 22 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 72 11 04 00
30: 00 00 00 00 50 00 00 00 00 00 00 00 30 01 00 00

Best regards

Christian

0 HemantPedanekar over 13 years ago in reply to Christian Schroeder

TI__Expert 5685 points

Christian,

There should not be any difference on MSI.

Regarding DMA, is this DMA from Endpoint or EDMA from DM816x/DM816x device?

While I try to replicate the scenario (using an off the shelf card), can you use legacy interrupt for the time being (the Lancero driver needs to support this)?

Thanks.

Hemant

0 Christian Schroeder over 13 years ago in reply to HemantPedanekar

Prodigy 50 points

Dear Hemant

falling back to legacy PCI interrupt handling in the driver (disable interrupts when they occur, and re-enable an interrupt after the source has been serviced) solved the problem partially.

On the DM8168 this works now for MSI and non-MSI interrupts, on the DM8148 this works only for non-MSI interrupts. Using MSI interrupts the board still freezes. This is the reason for the assumption of differences in the MSI handling in the DM81xx pcie root complexes of both processors.

The test application sends the data from DM81xx to the endpoint device (ArriaII-GX eval board) and the endpoint bypasses the data back to the DM81xx.

Regards, Christian

0 HemantPedanekar over 13 years ago in reply to Christian Schroeder

TI__Expert 5685 points

Christian,

Ok, so at least legacy interrupts work.

I will try with a card supporting MSI to check if this could be some s/w issue like incorrect interrupt number being used or EOI requirement.

Hemant

0 HemantPedanekar over 13 years ago in reply to HemantPedanekar

TI__Expert 5685 points

An update:

I think the issue is with difference in clearing the MSI interrupt across DM816x and DM814x (as you guessed).

In file arch/arm/mach-omap2/pcie-ti81xx.c, change the following line in function ack_msi() -->

__raw_writel(~(1 << (msi_num & 0x1f)), reg_virt + MSI0_IRQ_STATUS);

to -->

__raw_writel((1 << (msi_num & 0x1f)), reg_virt + MSI0_IRQ_STATUS);

(Of course this needs to be handled specifically for DM814x, for DM816x, the first method - write 0 to clear still holds).

Will get back to you after tests and update if the above fixes the issue.

Thanks.

Hemant

0 Leon Woestenberg over 13 years ago in reply to HemantPedanekar

Prodigy 50 points

Hello Hemant,

I am the developer of the Linux device driver of the Lancero PCI Express SGDMA solution for Altera FPGA's and working with Christian to solve this issue. Thank you for looking into this. We have not tested your proposal yet.

You wrote:

> (Of course this needs to be handled specifically for DM814x, for DM816x, the first method - write 0 to clear still holds).

Does this mean the DM816x has "write 0 to clear interrupts" behaviour and the DM814x has "write 1 to clear interrupts" behaviour? If so, is this due to different PCIe Root Complex versions or interconnect?

Apart from scanning both datasheets, are there any other differences?

With best regards,

Leon.

0 HemantPedanekar over 13 years ago in reply to Leon Woestenberg

TI__Expert 5685 points

Leon,

I will get back with concrete answer on the further differences question in a day or two.

Meanwhile, regarding the interrupt ack behavior, I am still verifying the modifications suggested earlier, but if confirmed, yes, it will be due to the difference in PCIe h/w versions being used in DM8168 v/s DM8148.

Thanks.

Hemant

0 Christian Schroeder over 13 years ago in reply to HemantPedanekar

Prodigy 50 points

Thanks Hemant,

just tested your proposal on DM814x and ist seems that the solution you suggested works now.

Best regards.

Christian

0 HemantPedanekar over 13 years ago in reply to Christian Schroeder

TI__Expert 5685 points

Christian,

The patch is available now at http://arago-project.org/git/projects/?p=linux-omap3.git;a=commitdiff;h=23a129439d647abb73cb3ada28335f4e8a4554eb

Hemant

Processors

Processors forum

PCI root complex differences between DM8148 and DM8168 ?