Hello, I have a question about inbound address translation on DM816x.
I have a DM816X running linux 2.6.37 setup as pcie Root Complex connected to this is a xilinx FPGA configured as End-Point (EP). The FPGA implements two non-prefetchable memory regions and is accessible from the RC (dm816x) for both read and write without any issues.
However, we are trying to have the FPGA read from the DM816X memory but the DM816X does not appear to claim the read request. No completion returns to the FPGA. Two BARs are configured on the DM816X, BAR0 the default access to the DM816X registers and BAR1 mapped to DDR memory.
Here's output from lspci -v:
root@ace:~# lspci -v
00:00.0 Class 0604: Device 104c:8888 (rev 01)
Flags: bus master, fast devsel, latency 0
I/O ports at 20200000 [size=4K]
I/O ports at 20000008 [size=1M]
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Memory behind bridge: 20100000-201fffff
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
01:00.0 Class 0500: Device 10ee:0007
Subsystem: Device 10ee:0007
Flags: bus master, fast devsel, latency 0
Memory at 20100000 (32-bit, non-prefetchable) [size=32K]
Memory at 20108000 (32-bit, non-prefetchable) [size=128]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [58] Express Endpoint, MSI 00
Capabilities: [c0] <chain broken>
In the DM816X the inbound translation is setup as:
IB_BAR0 = 1, IB_START_LO=0x20000000 IB_START0_HI=0 IB_OFFSET0=0xC0000000
or from the register DUMP:
root@ace:~# md host 0x51000300 16
/usr/sbin/md host 0x51000300 16 3
0x51000300 : 0x00000001 0x20000000 0x00000000 0xC0000000
0x51000310 : 0x00000000 0x00000000 0x00000000 0x00000000
0x51000320 : 0x00000000 0x00000000 0x00000000 0x00000000
0x51000330 : 0x00000000 0x00000000 0x00000000 0x00000000
Inbound Address Translation is enabled (OCP is configured by default as Smart Idle/Smart Standby):
root@haivision-ace:~# md host 0x51000000 8
/usr/sbin/md host 0x51000000 8 3
0x51000000 : 0x4E310900 0x00000A07 0x00010000 0x00000000
0x51000010 : 0x00000000 0x00010000 0x00000000 0x00000000
At boot, linux generates the following:
[ 23.350000] pci 0000:00:00.0: [104c:8888] type 1 class 0x000604
[ 23.360000] pci 0000:00:00.0: reg 10: [mem 0x51000000-0x51000fff]
[ 23.370000] pci 0000:00:00.0: reg 14: [mem 0xc0000000-0xc00fffff pref]
[ 23.370000] pci 0000:01:00.0: [10ee:0007] type 0 class 0x000500
[ 23.380000] pci 0000:01:00.0: reg 10: [mem 0x00000000-0x00007fff]
[ 23.390000] pci 0000:01:00.0: reg 18: [mem 0x00000000-0x0000007f]
[ 23.440000] pci 0000:01:00.0: supports D1 D2
[ 23.440000] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot
[ 23.450000] pci 0000:01:00.0: PME# disabled
[ 23.500000] PCI: bus1: Fast back to back transfers disabled
[ 23.500000] pci 0000:00:00.0: BAR 1: assigned [mem 0x20000000-0x200fffff pref]
[ 23.510000] pci 0000:00:00.0: BAR 1: set to [mem 0x20000000-0x200fffff pref] (PCI address [0x20000000-0x200fffff])
[ 23.520000] pci 0000:00:00.0: BAR 8: assigned [mem 0x20100000-0x201fffff]
[ 23.530000] pci 0000:00:00.0: BAR 0: assigned [mem 0x20200000-0x20200fff]
[ 23.530000] pci 0000:00:00.0: BAR 0: set to [mem 0x20200000-0x20200fff] (PCI address [0x20200000-0x20200fff])
[ 23.540000] pci 0000:01:00.0: BAR 0: assigned [mem 0x20100000-0x20107fff]
[ 23.550000] pci 0000:01:00.0: BAR 0: set to [mem 0x20100000-0x20107fff] (PCI address [0x20100000-0x20107fff])
[ 23.560000] pci 0000:01:00.0: BAR 2: assigned [mem 0x20108000-0x2010807f]
[ 23.570000] pci 0000:01:00.0: BAR 2: set to [mem 0x20108000-0x2010807f] (PCI address [0x20108000-0x2010807f])
[ 23.580000] pci 0000:00:00.0: PCI bridge to [bus 01-01]
[ 23.590000] pci 0000:00:00.0: bridge window [io disabled]
[ 23.590000] pci 0000:00:00.0: bridge window [mem 0x20100000-0x201fffff]
[ 23.600000] pci 0000:00:00.0: bridge window [mem pref disabled]
[ 23.600000] PCI: enabling device 0000:00:00.0 (0000 -> 0003)
Bus mastering by the FPGA is set with the setpci command:
setpci -s 00:01:00.00 4.l=0x146
The DM816X receives requests because if I purposely generate a read request from the FPGA with one of the FPGA's addresses (0x20100000) I get the following in the DM816X's extended capability registers:
root@ace:~# md host 0x51001100 16
/usr/sbin/md host 0x51001100 16 3
0x51001100 : 0x00010001 0x00100000 0x00000000 0x00062030
0x51001110 : 0x00002000 0x00000000 0x000001F4 0x00000001
0x51001120 : 0x0100120F 0x20100000 0x00000000 0x00000007
0x51001130 : 0x00000000 0x00000000 0x00000000 0x00000000
A TLP read request to address 0x20100000 -> Generates an Unsupported Request Error (UR_ERR_ST) @ 0x51001104, Advisory Non-Fatal Error @0x51001110 and First Error Ptr 0x14 @ 0x51001118.
Thank you for your attention.
Philip
