• State Resolved
  • Locked Locked
  • Replies 4 replies
  • Subscribers 99 subscribers
  • Views 3067 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

linux and pcie-nonprefetch address translation doesn't work?

n-Houtteilip
Prodigy 220 points

I'm running the 8168 as Root Complex and I have a pci device (fpga) where the link is up, configuration space accesses work. The device has a single BAR BAR0 of size 256 bytes.

PCI is configured correctly in the kernel, the pci link is up and configuration accesses to the device work.

However address remapping doesn't function correctly!

Both ioremap and pci_ioremap_bar return virtual addresses that cause exceptions when accessed. The same exception occurs with devmem2.

The exception typically looks like:

# devmem2 0x20000000 w
/dev/mem opened.ti81xx_pcie: Data abort: address = 0x400c1000 fsr = 0x1018 PC = 0x00008940 LR = 0x40181188
Memory mapped
ti81xx_pcie: Data abort 0x400c1000 -> 0x000c1000
at address 0x400Unhandled fault: Precise External Abort on non-linefetch (0x1018) at 0x400c1000
c1000.
Bus error

In the device driver case:

BAR0 Resource Start/End/Flags/Size :: 20000000/200000ff/40200/100
PCI: enabling device 0000:01:00.0 (0140 -> 0142)
FPGA: Remap BAR 0 20000000 -> d7696000
pci_ioremap_bar( 20000000 ) -> virtual:d7696000 physical:97696000

Is there something extra required to get physical address 0x20000000 into the MMU?

Evidence:

pci config of 8168(RC) and PCI endpoint.

# lspci -s 0:0:0.0 -v
00:00.0 Class 0604: Device 104c:8888 (rev 01)
Flags: bus master, fast devsel, latency 0
Memory at 20100000 (32-bit, non-prefetchable) [size=4K]
Memory at <ignored> (32-bit, prefetchable)
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Memory behind bridge: 20000000-200fffff
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting

# lspci -s 0:1:0.0 -v
01:00.0 Class 05ti81xx_pcie: Data abort: address = 0xd70220c0 fsr = 0x1008 PC = 0xc005ac50 LR = 0xc005abfc
ti81xx_pcie: Data abort 0xd70220c0 -> 0x970220c0
ti81xx_pcie: Handled PCIe abort
00: Device 10ee:0007
Subsystem: Device 10ee:0007
Flags: fast devsel, IRQ 368
Memory at 20000000 (32-bit, non-prefetchable) [size=256]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [58] Express Endpoint, MSI 00
Capabilities: [c0] <chain broken>
Kernel driver in use: pci_unit_test


Link state:

# devmem2 0x51001728 w
/dev/mem opened.
Memory mapped at address 0x40149000.
Value at address 0x51001728 (0x40149728): 0x3000B11

Output address translation:

# devmem2 0x51000200 w
/dev/mem opened.
Memory mapped at address 0x400c2000.
Value at address 0x51000200 (0x400c2200): 0x20000001
root@haiVision-ace:~# devmem2 0x51000208 w
/dev/mem opened.
Memory mapped at address 0x400a1000.
Value at address 0x51000208 (0x400a1208): 0x20800001

  • 0
    Guru 322000 points

    I have a pci device (fpga) where the link is up, configuration space accesses work.

    # lspci -s 0:1:0.0 -v
    01:00.0 Class 05ti81xx_pcie: Data abort: address = 0xd70220c0 fsr = 0x1008 PC = 0xc005ac50 LR = 0xc005abfc
    ti81xx_pcie: Data abort 0xd70220c0 -> 0x970220c0
    ti81xx_pcie: Handled PCIe abort
    00: Device 10ee:0007
    ...
    Capabilities: [c0] <chain broken>

    This doesn't look as if configuration space accesses work.

    However address remapping doesn't function correctly!

    BAR0 Resource Start/End/Flags/Size :: 20000000/200000ff/40200/100
    PCI: enabling device 0000:01:00.0 (0140 -> 0142)
    FPGA: Remap BAR 0 20000000 -> d7696000
    pci_ioremap_bar( 20000000 ) -> virtual:d7696000 physical:97696000

    What do the values in the last line mean? pci_ioremap_bar takes an index, not a bus address. Where do you get the "physical" value from?

  • 0 in reply to Clemens Ladisch
    Prodigy 220 points

    I'm getting a memory exception with lspci but not when accessing pci configuration space. In fact any type of memory exception is reported as a PCIE "Data abort". However I can read/write to pcie config. space with setpci without any error or warning messages.

    root:~# setpci -s 0:1:0.0 10.l
    20000000
    root:~# setpci -s 0:1:0.0 4.w
    0142
    root:~# setpci -s 0:1:0.0 4.w=0

    01:00.0 Class 0500: Device 10ee:0007
    Subsystem: Device 10ee:0007
    Flags: fast devsel, IRQ 368
    Memory at 20000000 (32-bit, non-prefetchable) [disabled] [size=256]
    Capabilities: [40] Power Management version 3
    Capabilities: [48] MSI: Enable+ Count=1/1 Maskable- 64bit+
    Capabilities: [58] Express Endpoint, MSI 00
    Capabilities: [c0] <chain broken>
    Kernel driver in use: pci_unit_test
    00: ee 10 07 00 00 00 10 00 00 00 00 05 10 00 00 00
    10: 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00
    20: 00 00 00 00 00 00 00 00 00 00 00 00 ee 10 07 00
    30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 00 00 00

    As for the 'last line: pci_ioremap_bar ( 20000000 ) -> virtual:d7696000 physical:97696000

    20000000 == BAR0 value

    d7696000 == virtual address returned by pci_ioremap_bar

    97696000 == virt_to_phys or virtual address returned by pci_ioremap_bar

  • 0 in reply to n-Houtteilip
    Guru 322000 points

    I'm getting a memory exception with lspci but not when accessing pci configuration space.

    But lspci does access configuration space.

    Capabilities: [c0] <chain broken>

    This looks like a bug in your FPGA device.

    What is the output of "lspci -s 1:0 -vvv -xxx"?

  • 0 in reply to Clemens Ladisch
    Prodigy 220 points

    FPGA problem involving how the BARs were mapped.