AM5748: Access to PCIe from DSP

Part Number: AM5748
Other Parts Discussed in Thread: SYSBIOS

I have a question according connection between PCIe and DSP.

I have found some example from pdk how to use PCIe driver on am574x board. After some changes I was able to run it on the board. However when I tried to get access to some PCIe registers (eg. VID/DID) Sitara raised an error:
[ 15.637427] omap-iommu 40d01000.mmu: iommu fault: da 0x51800000 flags 0x0

And in this moment DSP crashed and then remoteproc reload it. So after some investigations I recognized this is caused by lack of entry in DSP resource table. After adding below entries:

I am able to run DSP and get access to memory which caused mmu fault.

However now I have another problem after trying to read VID/DID register:

VID and DID are equal to 0. In the same time linux generates below error:

So I thought it was connected to interrupt controller defined in dra7.dtsi file under pcie@51800000 section. Removing this interrupt did not helped. Do you have any idea what I can do now?

Regards
Bernd

over 4 years ago

0 Stanley Liu over 4 years ago

TI__Guru 50045 points

Hi,

The L3 error was triggered from DSP read access to PCIE2.

This L3 error typically happens if PCIE2 was not clocked when DSP tried to access it.

I assume PCIE2 is enabled in Linux Kernel so maybe DSP tries to access it before it is enabled during Linux kernel boot.

Regards,
Stanley

0 Marcin Hujdus over 4 years ago in reply to Stanley Liu

Prodigy 50 points

Hi,

I have tried to add 10 second delay in DSP code during linux booting, before DSP make any access to PCIe however it does not change anything. Same error

44000000.ocp:L3 Standard Error: MASTER DSP1_MDMA TARGET PCIE2 (Read Link): At Address: 0x00001000 : Data Access in User mode during Functional access

still occures.

We are basing on example from: pdk_am57xx_1_0_17 and ipc_3_50_04_08.

BR,

Marcin

0 Grzegorz Stawarz over 4 years ago

Prodigy 70 points

Hello,

We fixed 44000000.ocp:L3 Standard Error: MASTER DSP1_MDMA TARGET PCIE2 (Read Link): At Address: 0x00001000 : Data Access in User mode during Functional access error, but now we are facing another problem.

When we call this method (from TI example for AM57X):

statusCmd.ioSp = 1;
statusCmd.memSp = 1;
statusCmd.resp = 1;

setRegs.statusCmd = &statusCmd;

if ((status = Pcie_writeRegs(conf.handle, location, &setRegs)) != pcie_RET_OK) {
Console_printf("Could not write register!");
}

On FPGA we are receiving cyclic RX data with random data and on A15 below error occurs:

omap_l3_noc 44000000.ocp: L3 application error: target 5 mod:1 (unclearable)
omap_l3_noc 44000000.ocp: L3 debug error: target 5 mod:1 (unclearable)

What can be a reason of this L3 errors.

BR,

Grzegorz

0 Marcin Hujdus over 4 years ago in reply to Grzegorz Stawarz

Prodigy 50 points

Update,

We have solved this issue with L3 application error but we still are not able to send data over PCIe to FPGA. What is strange driver written in linux works like a charm when we try to send data. Problem occurs only on DSP.

0 BO over 4 years ago in reply to Marcin Hujdus

TI__Genius 12760 points

Hi Jian,

can you please look at that?

Regards Bernd

0 jian35385 over 4 years ago in reply to BO

TI__Mastermind 23125 points

BO,

I am trying to catch up how you are using the PCIe. I assume you let Linux to bring up the PCIe address space, then let DSP to use the PCIe space. right?

If so, can you list what is the exposed FPGA memory regions seen by Linux? you can find it out by lspci command in Linux.

Then from DSP, I assume you have updated the DSP MMU and pointed to the same memory region the FPGA exposed?

Jian

0 Marcin Hujdus over 4 years ago in reply to jian35385

Prodigy 50 points

Hi Jian,

Some more explanation from our side.

After confirmation of working A15 <-PCIe-> FGPA communication, we have decided to change our design and allow only DSP to communicate with FPGA. So now A15 Linux only knows about connected FPGA but do not touch it and does not load any driver to communicate with it.

DSP image is loaded with remoteproc driver in late-attach mode. We are basing on pdk_am57xx_1_0_17/packages/ti/drv/pcie/example/ but with disabled EDMA.

We are able to read VID/DID on DSP, write and read registers with Pcie_* API.

Pcie_getMemSpaceRange returns:

0x20000000 (the same region which is described in SoC documentation).

and now I am comparing it with linux lspci -vv result:

Region 0: [virtual] Memory at 20400000 (32-bit, non-prefetchable) [size=1M]
Region 1: [virtual] Memory at 20200000 (32-bit, non-prefetchable) [size=2M]

Shoould those regions be the same? If yes how can i get this shift (0x0040_0000 and 0x0200_0000). Those address should be read from PCIe config space. But when I try to use pcieType1Bar32bitIdx_t and Pcie_readRegs, I am receiving: 0x90000008

BR,

0 jian35385 over 4 years ago in reply to Marcin Hujdus

TI__Mastermind 23125 points

Marcin, could you paste the complete output of the lspci -v, then lspci-vv? Also can you read the EP's BAR0-3 from the Linux side? you can do so by devmem2 command, to the address of the PCIe outbound address plus standard pcie structure offset.

I did similar exercise on AM65 device previously, where upon linux enumeration of the PCIE, I connect CCS to another core in SOC, access the remote memory space on the EP.

sorry for the delayed response

Jian

0 Marcin Hujdus over 4 years ago in reply to jian35385

Prodigy 50 points

Jian,

Thank you for your response. Below you can find three outputs of lspci: with loaded dsp image, with loaded our driver and case when any of those binaries is loaded. Also we've added debug trigger on rx lane in vivado tool (FPGA toolchain).

I am little afraid that I do not understand correctly:

> you can do so by devmem2 command, to the address of the PCIe outbound address plus standard pcie structure offset.

NO DSP LOADED, NO LINUX LOADED:

root@am57xx-evm:~# lspci -v
00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 173
Memory at 20100000 (64-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: None
Memory behind bridge: 20200000-204fffff [size=3M]
Prefetchable memory behind bridge: None
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: pcieport

01:00.0 RAM memory: Xilinx Corporation Device 7021
Subsystem: Xilinx Corporation Device 0007
Flags: fast devsel, IRQ 255
Memory at 20400000 (32-bit, non-prefetchable) [disabled] [size=1M]
Memory at 20200000 (32-bit, non-prefetchable) [disabled] [size=2M]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [100] Device Serial Number 00-00-00-01-01-00-0a-35

root@am57xx-evm:~# lspci -vv
00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 173
Region 0: Memory at 20100000 (64-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: None
Memory behind bridge: 20200000-204fffff [size=3M]
Prefetchable memory behind bridge: None
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 0000000093943000 Data: 0000
Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x1 (downgraded)
TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
RootCmd: CERptEn+ NFERptEn+ FERptEn+
RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
Kernel driver in use: pcieport

01:00.0 RAM memory: Xilinx Corporation Device 7021
Subsystem: Xilinx Corporation Device 0007
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 255
Region 0: Memory at 20400000 (32-bit, non-prefetchable) [disabled] [size=1M]
Region 1: Memory at 20200000 (32-bit, non-prefetchable) [disabled] [size=2M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35

root@am57xx-evm:~# [ 128.096005] NET: Registered protocol family 15
root@am57xx-evm:~#
root@am57xx-evm:~#
root@am57xx-evm:~# lspci -xxx
00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)
00: 4c 10 88 88 47 05 10 00 01 00 04 06 10 00 01 00
10: 04 00 10 20 00 00 00 00 00 01 ff 00 f0 00 00 00
20: 20 20 40 20 f0 ff 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ac 01 01 00
40: 01 50 03 5a 00 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 81 00 00 30 94 93 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 00 42 00 01 80 00 00 1f 28 00 00 22 3c 73 00
80: 48 00 12 f0 00 00 00 00 c0 03 40 00 08 00 00 00
90: 00 00 00 00 1f 04 00 00 00 00 00 00 06 00 00 00
a0: 02 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

01:00.0 RAM memory: Xilinx Corporation Device 7021
00: ee 10 21 70 40 01 10 00 00 00 00 05 10 00 00 00
10: 00 00 40 20 00 00 20 20 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 ee 10 07 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00
40: 01 48 03 78 08 00 00 00 05 60 80 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 10 00 02 00 02 8e 00 00 10 28 01 00 12 f4 03 00
70: 40 00 12 10 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00
90: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

root@am57xx-evm:~# devmem2 0x20400000 w 0xdeadbabe
/dev/mem opened.
Memory mapped at address 0xb6fd9000.
Read at address 0x20400000 (0xb6fd9000): 0xFFFFFFFF
Write at address 0x20400000 (0xb6fd9000): 0xDEADBABE, readback 0xDEADBABE
root@am57xx-evm:~# devmem2 0x20400000 w 0xdeadbeef
/dev/mem opened.
Memory mapped at address 0xb6f4d000.
Read at address 0x20400000 (0xb6f4d000): 0xFFFFFFFF
Write at address 0x20400000 (0xb6f4d000): 0xDEADBEEF, readback 0xDEADBEEF
root@am57xx-evm:~# devmem2 0x20400000 w 0xdeadbeef
/dev/mem opened.
Memory mapped at address 0xb6f60000.
Read at address 0x20400000 (0xb6f60000): 0xFFFFFFFF
Write at address 0x20400000 (0xb6f60000): 0xDEADBEEF, readback 0xDEADBEEF
root@am57xx-evm:~#

triger on vivado: no visible

DSP LOADED, DRIVER NO LOADED:

root@am57xx-evm:~# lspci -v
00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 173
[virtual] Memory at 20100000 (64-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
I/O behind bridge: 00000000-00000fff [size=4K]
Memory behind bridge: 00000000-000fffff [size=1M]
Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: pcieport

01:00.0 RAM memory: Xilinx Corporation Device 7021
Subsystem: Xilinx Corporation Device 0007
Flags: bus master, fast devsel, latency 0, IRQ 255
Memory at 20400000 (32-bit, non-prefetchable) [size=1M]
Memory at 20200000 (32-bit, non-prefetchable) [size=2M]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [100] Device Serial Number 00-00-00-01-01-00-0a-35

root@am57xx-evm:~# lspci -vv
00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01) (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 173
Region 0: [virtual] Memory at 20100000 (64-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
I/O behind bridge: 00000000-00000fff [size=4K]
Memory behind bridge: 00000000-000fffff [size=1M]
Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: CorrErr+ NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x1 (downgraded)
TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP+ Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
RootCmd: CERptEn- NFERptEn- FERptEn-
RootSta: CERcvd+ MultCERcvd- UERcvd- MultUERcvd-
FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
Kernel driver in use: pcieport

01:00.0 RAM memory: Xilinx Corporation Device 7021
Subsystem: Xilinx Corporation Device 0007
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 255
Region 0: Memory at 20400000 (32-bit, non-prefetchable) [size=1M]
Region 1: Memory at 20200000 (32-bit, non-prefetchable) [size=2M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0001000000000000 Data: 0000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35

root@am57xx-evm:~# lspci -xxx
00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)
00: 4c 10 88 88 46 05 10 00 01 00 00 00 00 00 01 00
10: 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ac 01 00 00
40: 01 50 03 5a 00 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 80 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 00 42 00 01 80 00 00 11 28 00 00 22 3c 73 00
80: 08 00 12 f0 00 00 00 00 c0 03 40 00 00 00 00 00
90: 00 00 00 00 1f 04 00 00 00 00 00 00 06 00 00 00
a0: 02 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

01:00.0 RAM memory: Xilinx Corporation Device 7021
00: ee 10 21 70 46 05 10 00 00 00 00 05 10 00 00 00
10: 00 00 40 20 00 00 20 20 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 ee 10 07 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ac 01 00 00
40: 01 48 03 78 08 00 00 00 05 60 80 00 00 00 00 00
50: 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 10 00 02 00 02 8e 00 00 10 28 00 00 12 f4 03 00
70: 00 00 12 10 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00
90: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
root@am57xx-evm:~# devmem2 0x20400000 w 0xdeadbabe
/dev/mem opened.
Memory mapped at address 0xb6f7a000.
Read at address 0x20400000 (0xb6f7a000): 0xFFFFFFFF
Write at address 0x20400000 (0xb6f7a000): 0xDEADBABE, readback 0xDEADBABE
root@am57xx-evm:~# devmem2 0x20400000 w 0xdeadbabe
/dev/mem opened.
Memory mapped at address 0xb6fb4000.
Read at address 0x20400000 (0xb6fb4000): 0xFFFFFFFF
Write at address 0x20400000 (0xb6fb4000): 0xDEADBABE, readback 0xDEADBABE
root@am57xx-evm:~# devmem2 0x20400000 w 0xdeadbabe
/dev/mem opened.
Memory mapped at address 0xb6fd7000.
Read at address 0x20400000 (0xb6fd7000): 0xFFFFFFFF
Write at address 0x20400000 (0xb6fd7000): 0xDEADBABE, readback 0xDEADBABE
root@am57xx-evm:~#

triger on vivado: no visible

When we try to transmit data from DSP to FPGA to address 0x20400000, pci receives system error:

01:00.0 RAM memory: Xilinx Corporation Device 7021
00: ee 10 21 70 46 05 10 40 00 00 00 05 10 00 00 00
10: 00 00 40 20 00 00 20 20 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 ee 10 07 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ac 01 00 00
40: 01 48 03 78 08 00 00 00 05 60 80 00 00 00 00 00
50: 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 10 00 02 00 02 8e 00 00 10 28 0a 00 12 f4 03 00
70: 00 00 12 10 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00
90: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

NO DSP LOADED, DRIVER LOADED:

root@am57xx-evm:~#
root@am57xx-evm:~#
root@am57xx-evm:~# lspci -v
00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 173
Memory at 20100000 (64-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: None
Memory behind bridge: 20200000-204fffff [size=3M]
Prefetchable memory behind bridge: None
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: pcieport

01:00.0 RAM memory: Xilinx Corporation Device 7021
Subsystem: Xilinx Corporation Device 0007
Flags: bus master, fast devsel, latency 0, IRQ 215
Memory at 20400000 (32-bit, non-prefetchable) [size=1M]
Memory at 20200000 (32-bit, non-prefetchable) [size=2M]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [100] Device Serial Number 00-00-00-01-01-00-0a-35

root@am57xx-evm:~# lspci -vv
00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 173
Region 0: Memory at 20100000 (64-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: None
Memory behind bridge: 20200000-204fffff [size=3M]
Prefetchable memory behind bridge: None
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR+ <PERR-
BridgeCtl: Parity+ SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 0000000093943000 Data: 0000
Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x1 (downgraded)
TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
AtomicOpsCtl: ReqEn- EgressBlck-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
RootCmd: CERptEn+ NFERptEn+ FERptEn+
RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
Kernel driver in use: pcieport

01:00.0 RAM memory: Xilinx Corporation Device 7021
Subsystem: Xilinx Corporation Device 0007
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR+ <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 215
Region 0: Memory at 20400000 (32-bit, non-prefetchable) [size=1M]
Region 1: Memory at 20200000 (32-bit, non-prefetchable) [size=2M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 0000000093943000 Data: 0001
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr+ FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35

root@am57xx-evm:~# lspci -xxx
00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)
00: 4c 10 88 88 47 05 10 00 01 00 04 06 10 00 01 00
10: 04 00 10 20 00 00 00 00 00 01 ff 00 f0 00 00 60
20: 20 20 40 20 f0 ff 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ac 01 01 00
40: 01 50 03 5a 00 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 81 00 00 30 94 93 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 00 42 00 01 80 00 00 1f 28 00 00 22 3c 73 00
80: 48 00 12 f0 00 00 00 00 c0 03 40 00 08 00 00 00
90: 00 00 00 00 1f 04 00 00 00 00 00 00 06 00 00 00
a0: 02 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

01:00.0 RAM memory: Xilinx Corporation Device 7021
00: ee 10 21 70 46 05 10 40 00 00 00 05 10 00 00 00
10: 00 00 40 20 00 00 20 20 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 ee 10 07 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ac 01 00 00
40: 01 48 03 78 08 00 00 00 05 60 81 00 00 30 94 93
50: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00
60: 10 00 02 00 02 8e 00 00 10 28 0b 00 12 f4 03 00
70: 40 00 12 10 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00
90: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

root@am57xx-evm:~# devmem2 0x20400000 w 0xdeadbeef
/dev/mem opened.
[ 1114.225490] [EXIT] init_hpg:533
[ 1126.200007] omap_l3_noc 44000000.ocp: L3 application error: target 5 mod:1 (unclearable)
Memory mapped at address 0xb6f9d000.
Read at address 0x2040000[ 1126.211460] omap_l3_noc 44000000.ocp: L3 debug error: target 5 mod:1 (unclearable)
0 (0xb6f9d000): 0xFFFFFFFF
Write at address 0x20400000 (0xb6f9d000): 0xDEADBEEF, readback 0xDEADBEEF
root@am57xx-evm:~# devmem2 0x20400000 w 0xdeadbabe
/dev/mem opened.
Memory mapped at address 0xb6f63000.
Read at address 0x20400000 (0xb6f63000): 0xFFFFFFFF
Write at address 0x20400000 (0xb6f63000): 0xDEADBABE, readback 0xDEADBABE

triger on vivado: visible

What is interested in last case: we do not getting L3 application error when we try to transmit data directly from driver to FPGA.

BR,

0 jian35385 over 4 years ago in reply to Marcin Hujdus

TI__Mastermind 23125 points

Marcin,

Could you explain more details differences between:

NO DSP LOADED, NO LINUX LOADED
NO DSP LOADED, DRIVER LOADED

I see you have some lspci log even under the first bullet. so not sure what you mean by "NO LINUX LOADED"

Sorry to circle back to your original problem. I thought you mentioned:

>>After confirmation of working A15 <-PCIe-> FGPA communication, we have decided to change our design and allow only DSP to communicate with FPGA. So now A15 Linux only knows about connected FPGA but do not touch it and does not load any driver to communicate with it.

So I assume your first method of letting Linux driver to enumerate the PCIe driver space, then let DSP to read/write to the space was working, but then you want to switch to DSP-side of baremetal driver completely. Is this the correct understanding?

If so, I don't believe the DSP driver code has the probing function like Linux driver does, where it detects the memory space the FPGA offered. The function Pcie_getMemSpaceRange() does not return the probed FPGA windows. instead, you need to add code to write then read back the remote BAR MASK registers, then calculate the memory space, then enable the remote memory region by writing to the the command register of the PCIe structure. some pseudo code to enable memory region like below:

get_epval = *((volatile uint32_t *) (uintptr_t)(PCIE_WINDOW_CFG_BASE+0x0004U));

put_val = get_epval | 0x02U ; // Enable memory space

*((volatile uint32_t *) (uintptr_t)(PCIE_WINDOW_CFG_BASE+0x0004U)) = put_val;

get_epval = *((volatile uint32_t *) (uintptr_t)(PCIE_WINDOW_CFG_BASE+0x0004U));

PCIE_logPrintf (" RC set CMD_STS: wrote 0x%08X, readback 0x%08X\n", put_val, get_epval);

regards

Jian

0 Marcin Hujdus over 4 years ago in reply to jian35385

Prodigy 50 points

Jian,

My bad, I mean:

> NO DSP LOADED, NO LINUX LOADED

it should be: NO DSP IMAGE IS LOADED, NO LINUX DRIVER IS LOADED

> NO DSP LOADED, DRIVER LOADED

It means that we do not load DSP image, but we load our custom driver.

jian35385 said:
So I assume your first method of letting Linux driver to enumerate the PCIe driver space, then let DSP to read/write to the space was working, but then you want to switch to DSP-side of baremetal driver completely. Is this the correct understanding

You are understading it correct.

jian35385 said:
Pcie_getMemSpaceRange() does not return the probed FPGA windows

When we checked the return addres it was: 0x2000_0000, memory address desdribed in document: AM574x technical reference manual rev, b (spruih8b).

jian35385 said:
instead, you need to add code to write then read back the remote BAR MASK registers, then calculate the memory space, then enable the remote memory region by writing to the the command register of the PCIe structure.

So you mean that first of all we have to set the bar mask register and after that enable memory for read/write? However what do you mean by calculating memory space? Is any document available which describes it?

BR,

0 Grzegorz Stawarz over 4 years ago in reply to Marcin Hujdus

Prodigy 70 points

Hello Jian,

We have question regarding inbound and outbound addresses.

What are:

ibCfg.ibBar = PCIE_BAR_IDX_RC; /* Match BAR that was configured above*/

ibCfg.ibStartAddrLo = PCIE_IB_LO_ADDR_RC;

ibCfg.ibStartAddrHi = PCIE_IB_HI_ADDR_RC;

ibCfg.ibOffsetAddr = (uint32_t)pcieConvert_CoreLocal2GlobalAddr ((uintptr_t)dstBuf.buf);

 ibCfg.region        = PCIE_IB_REGION_RC;

In our code ibOffsetAddr equals 0x40800000. Is it expected value or proper value for this case?

PCIE_IB_LO_ADDR_RC and PCIE_IB_HI_ADDR_RC are both set to 0x0.

Should we use offsets or base addresses with shift in those cases?

BR,

0 Grzegorz Stawarz over 4 years ago in reply to Grzegorz Stawarz

Prodigy 70 points

Hello Jian,

any info or update in our case?

BR,

0 Grzegorz Stawarz over 4 years ago in reply to Grzegorz Stawarz

Prodigy 70 points

Hello Jian,

We resolved communication problem on DSP side. We are able to write data from DSP to FPGA with PCIE bus.

But now we are struggling with opposite communication. We want to read data from FPGA on DSP side.

When we are trying to read data from FPGA we are receiving error:

root@am57xx-evm:~# devmem2 0x20200100 w
/dev/mem opened.
[   46.822550] [EXIT] hpg_proc_release:37
[   62.722263] omap_l3_noc 44000000.ocp: L3 application error: target 5 mod:1 (unclearable)
Memory mapped at address 0xb6f36000.
Read at address  0x2020010[   62.734326] omap_l3_noc 44000000.ocp: L3 debug error: target 5 mod:1 (unclearabl                                                           e)
0 (0xb6f36100): 0xFFFFFFFF

Do you know what can cause such error?

BR,

0 jian35385 over 4 years ago in reply to Grzegorz Stawarz

TI__Mastermind 23125 points

Grzegorz,

Can you issue two commands:

devmem2 0x20200100

devmem2 0x20200100 w 0xABCDABCD

and let me know what happens?

thanks
Jian

0 Grzegorz Stawarz over 4 years ago in reply to jian35385

Prodigy 70 points

Hello Jian,

We resolved current problem with PCIe. We let the Linux configure PCIe config space, and on DSP we are only opening handler.

And now, communication between DSP and FPGA works.

But now we are struggling with multithreading on DSP. Can you give us some info how multithreading works on sitara on DSP, and how we should handle task creation?

Now when we create few tasks, only one runs, that one with the highest priority. We are trying to configure somehow scheduler, we are trying different interfaces from TI sysbios, but still without success.

We tried to add flag to compilation:

BIOS.customCCOpts = "--multithread"

But it causes a lot of linker errors. We tried to add missing modules( e.g. xdc.useModule('ti.sysbios.family.c64p.EventCombiner'); ), but it changed nothing.

e.g errors:

ti_sysbios_family_c64p_EventCombiner_Module_startup__E       /home/user/Sitara_PS863/hpg_dsp/DRA7XX_linux_elf/ex41_forwardmsg/dsp1/bin/debug/configuro/package/cfg/Dsp1_pe66.oe66                                              
 ti_sysbios_family_c64p_EventCombiner_disableEvent__E         /home/user/Sitara_PS863/hpg_dsp/DRA7XX_linux_elf/ex41_forwardmsg/dsp1/bin/debug/configuro/package/cfg/Dsp1_pe66.src/ipc/ipc.ae66<Ipc.obj>                         
 ti_sysbios_family_c64p_EventCombiner_dispatchPlug__E         /home/user/Sitara_PS863/hpg_dsp/DRA7XX_linux_elf/ex41_forwardmsg/dsp1/bin/debug/configuro/package/cfg/Dsp1_pe66.src/ipc/ipc.ae66<Ipc.obj>                         
 ti_sysbios_family_c64p_EventCombiner_dispatch__E             /home/user/Sitara_PS863/hpg_dsp/DRA7XX_linux_elf/ex41_forwardmsg/dsp1/bin/debug/configuro/package/cfg/Dsp1_pe66.src/ipc/ipc.ae66<Ipc.obj>                         
 ti_sysbios_family_c64p_EventCombiner_enableEvent__E          /home/user/ti/edma3_lld_2_12_05_30E/packages/ti/sdo/edma3/drv/sample/lib/tda2xx-evm/66/debug/edma3_lld_drv_sample.ae66<sample_cs.oe66>                            
 ti_sysbios_family_c64p_Exception_Module_startup__E           /home/user/Sitara_PS863/hpg_dsp/DRA7XX_linux_elf/ex41_forwardmsg/dsp1/bin/debug/configuro/package/cfg/Dsp1_pe66.oe66                                              
 ti_sysbios_family_c64p_Exception_handler__I                  /home/user/Sitara_PS863/hpg_dsp/DRA7XX_linux_elf/ex41_forwardmsg/dsp1/bin/debug/configuro/package/cfg/Dsp1_pe66.src/sysbios/sysbios.ae66<c64p_Exception_asm.obj>  
 ti_sysbios_family_c64p_Hwi_Instance_finalize__E              /home/user/Sitara_PS863/hpg_dsp/DRA7XX_linux_elf/ex41_forwardmsg/dsp1/bin/debug/configuro/package/cfg/Dsp1_pe66.oe66                                              
 ti_sysbios_family_c64p_Hwi_Instance_init__E                  /home/user/Sitara_PS863/hpg_dsp/DRA7XX_linux_elf/ex41_forwardmsg/dsp1/bin/debug/configuro/package/cfg/Dsp1_pe66.oe66

Shall we create another e2e.ti thread, or can we continue in this?

Thanks in advance,

0 Grzegorz Stawarz over 4 years ago in reply to Grzegorz Stawarz

Prodigy 70 points

Hello Jian,

current update:

We resolved multithreading problem, so in this case help is not needed.

But now we are trying to run EDMA and transmit data, we are receiving error:

[ 21.718957] omap_l3_noc 44000000.ocp: L3 application error: target 5 mod:1 (unclearable)
[ 21.727117] omap_l3_noc 44000000.ocp: L3 debug error: target 5 mod:1 (unclearable)

During debugging(Code Compose Studio), we found that error occurs when we are trying to save data into ESR register, in method EDMA3_DRV_enableTransfer, file edma3_drv_basic.c, line 1924. It look that we don't have access to such region.

Shall we add some memory regions to DSP resource table? If yes, then which are proper? Because we are trying to use regions below:

And still we have such error.

From the other hand, before we started adding memory regions into DSP resource table, we had an error:

21.580621] omap-iommu 40d02000.mmu: iommu fault: da 0x401d1400 flags 0x0
[ 21.587443] remoteproc remoteproc2: crash detected in 40800000.dsp: type mmufault
[ 21.594963] omap-iommu 40d02000.mmu: 40d02000.mmu: errs:0x00000002 da:0x401d1400 pgd:0xfef2897d *pgd:px00000000
[ 21.606636] remoteproc remoteproc2: handling crash #1 in 40800000.dsp
[ 21.613319] remoteproc remoteproc2: recovering 40800000.dsp
[ 21.625577] remoteproc remoteproc2: stopped remote processor 40800000.dsp
[ 21.634515] remoteproc remoteproc2: powering up 40800000.dsp
[ 21.647981] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 5287160
[ 21.655962] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[ 21.661845] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
[ 21.680122] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 21.690571] remoteproc remoteproc2: registered virtio0 (type 7)
[ 21.698727] remoteproc remoteproc2: remote processor 40800000.dsp is now up

Caught in the same line in EDMA3_DRV_enableTransfer, in file edma3_drv_basic.c, line 1924.

Thanks in advance Jian :)

Processors

Processors forum

AM5748: Access to PCIe from DSP