This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

DM8148 - PCIe/DMA - Cannot write to DMA memory space

Issue:

PCIe is able to read and write from the root complex (DM8148) to an endpoint device.  However, the endpoint is unable to write to the DM8148's DMA memory space.

Background:

My customer is in their design verification stage and has verified that the PCIe data is being sent from the end point device to the DM8148.  We have also verified that the inbound memory range of the PCIe (from the devices.c file) includes the DMA memory space (DMA: 0xffc00000 - 0xffe00000 2 MB) as seen below.  They are using the default memory map that comes with EZSDK for their board that is also using 1GB of memory.

{
               /* Inbound memory window */
               .name           = "pcie-inbound0",
               .start          = PHYS_OFFSET,
               .end            = PHYS_OFFSET + SZ_2G - 1,
               .flags          = IORESOURCE_MEM,
},

Question:

How do we enable PCIe to write to the kernal's DMA memory space. 

Link to documentation:

EZSDK Memory Map:
PCIe Root Complex User Guide:
  • Hello,

    I'll try to address that issue soon. I'll let you know, when I have something.

    BR

    Vladimir

  • 6724.Pcie Logs.zip

    Vladimir,

    Thanks for your support.  Here are some boot and console logs. Also, attached are some waveform captures that show that there is actual PCIe data on the bus.  Hope this helps.

    ------------------------------KERNEL BOOT -------------------------------------

    Memory: 364MB 274MB = 638MB total
    Memory: 642160k/642160k available, 62352k reserved, 280576K highmem
    Virtual kernel memory layout:
        vector  : 0xffff0000 - 0xffff1000   (   4 kB)
        fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
        DMA     : 0xffc00000 - 0xffe00000   (   2 MB)
        vmalloc : 0xd7000000 - 0xf8000000   ( 528 MB)
        lowmem  : 0xc0000000 - 0xd6c00000   ( 364 MB)
        pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
        modules : 0xbf000000 - 0xbfe00000   (  14 MB)
          .init : 0xc0008000 - 0xc0038000   ( 192 kB)
          .text : 0xc0038000 - 0xc0490000   (4448 kB)
          .data : 0xc0490000 - 0xc04d2100   ( 265 kB)
     
     
    ti81xx_pcie: Invoking PCI BIOS...
    ti81xx_pcie: Setting up Host Controller...
    ti81xx_pcie: Register base mapped @0xd7020000
    ti81xx_pcie: Starting PCI scan...
    PCI: bus0: Fast back to back transfers disabled
    PCI: bus1: Fast back to back transfers disabled
    pci 0000:00:00.0: BAR 8: assigned [mem 0x20000000-0x200fffff]
    pci 0000:01:00.0: BAR 1: assigned [mem 0x20000000-0x2000ffff]
    pci 0000:01:00.0: BAR 1: set to [mem 0x20000000-0x2000ffff] (PCI address [0x20000000-0x2000ffff])
    pci 0000:01:00.0: BAR 2: assigned [mem 0x20010000-0x2001ffff]
    pci 0000:01:00.0: BAR 2: set to [mem 0x20010000-0x2001ffff] (PCI address [0x20010000-0x2001ffff])
    pci 0000:01:00.0: BAR 3: assigned [mem 0x20020000-0x20021fff]
    pci 0000:01:00.0: BAR 3: set to [mem 0x20020000-0x20021fff] (PCI address [0x20020000-0x20021fff])
    pci 0000:01:00.0: BAR 0: assigned [mem 0x20022000-0x20022fff]
    pci 0000:01:00.0: BAR 0: set to [mem 0x20022000-0x20022fff] (PCI address [0x20022000-0x20022fff])
    pci 0000:00:00.0: PCI bridge to [bus 01-01]
    pci 0000:00:00.0:   bridge window [io  disabled]
    pci 0000:00:00.0:   bridge window [mem 0x20000000-0x200fffff]
    pci 0000:00:00.0:   bridge window [mem pref disabled]
    PCI: enabling device 0000:00:00.0 (0140 -> 0143)
     
    dvm_pci_driver: initialising
    revision:0:  dev->dev.kobj.name: "0000:01:00.0"
    PCI BAR0: 20022000 : bar0_len: 00001000
    PCI BAR0 remap: d70f2000
    PCI BAR1: 20000000 : bar1_len: 00010000
    PCI BAR1 remap: d7100000
    PCI BAR2: 20010000 : bar2_len: 00010000
    PCI BAR2 remap: d7120000
    PCI BAR3: 20020000 : bar3_len: 00002000
    PCI BAR3 remap: d70f8000
    0000:01:00.0: Memory Allocated: ffc00000 DMA Address: 95d00000
    PCI: enabling device 0000:01:00.0 (0144 -> 0146)
     
    ----------------------------------------------------------------------------------------------------
     
    ----------------------------------- PCIe Tests ------------------------------------------------------
    --- IO ----
    root@dm814x-dvm:~/tools/PCIe# ./pcie_test
    Revision              : 44564da0
    SDI A/B Control       : 00000000
    SDI A Status          : 80008027
    SDI B Status          : 80008810
    Video Source          : 00000000
    MPEG Stream Control   : 00000000
    Audio Monitor Control : 00000000
    MISC                  : 00000009
     
    ---- DMA ----
    root@dm814x-dvm:~/tools/PCIe# ./pcie_dmatest
     
    dvm_fpgacntl_fd=3
    dvm_mpegcntl=4
    Wr: DMA TS Count 10
    Rd: DMA TS Count 0
    Rd: DMA Physical Address 95d00000
    Wr: DMA Address pointer 95d00000
    Rd: DMA Address pointer 95d00000
    Rd: DMA CPU Virtual Address ffc00000
    Wr: Enable DMA/IRQ 8001
    Rd: Enable DMA 8001
    mpeg_put_cmd: sent 0000b800
    mpeg_put_cmd: Bit-31 cleared at delta count 45: got 0x3800
    mpeg_get_pid:
    mpeg_put_cmd: sent 0000b000
    mpeg_put_cmd: Bit-31 cleared at delta count 54: got 0x3000
    Press Enter to capture active PIDs
    mpeg_get_pid:
    mpeg_put_cmd: sent 0000b000
    mpeg_put_cmd: Bit-31 cleared at delta count 53: got 0x3000
     
     >>> Active PID <<<
    PID: 8000
    PID: 8100
    PID: 9001
    PID: 9011
    PID: 9100
    Press Enter for TS count
    TS count: 0
     
    Press Enter to enable stream to Host
    mpeg_put_cmd: sent 00008800
    mpeg_put_cmd: Bit-31 cleared at delta count 1: got 0x0800
    mpeg_put_cmd: sent 00009000
    mpeg_put_cmd: Bit-31 cleared at delta count 1: got 0x1000
    root@dm814x-dvm:~/tools/PCIe#
     
    ------------------ TTY Console (printk from driver during interrupt processing) ------------------------------
    root@dm814x-dvm:~# cpu_addr: ffc00000 lastwrp: 95d00000 tsdata: 00000000
    cpu_addr: ffc00100 lastwrp: 95d00200 tsdata: 00000000
    cpu_addr: ffc00200 lastwrp: 95d00300 tsdata: 00000000
    cpu_addr: ffc00300 lastwrp: 95d00400 tsdata: 00000000
    cpu_addr: ffc00400 lastwrp: 95d00500 tsdata: 00000000
    cpu_addr: ffc00500 lastwrp: 95d00600 tsdata: 00000000
    cpu_addr: ffc00600 lastwrp: 95d00700 tsdata: 00000000
    cpu_addr: ffc00700 lastwrp: 95d00800 tsdata: 00000000
    cpu_addr: ffc00800 lastwrp: 95d00900 tsdata: 00000000
    cpu_addr: ffc00900 lastwrp: 95d00a00 tsdata: 00000000
    cpu_addr: ffc00a00 lastwrp: 95d00b00 tsdata: 00000000
    cpu_addr: ffc00b00 lastwrp: 95d00c00 tsdata: 00000000
    cpu_addr: ffc00c00 lastwrp: 95d00d00 tsdata: 00000000
    cpu_addr: ffc00d00 lastwrp: 95d00e00 tsdata: 00000000
    cpu_addr: ffc00e00 lastwrp: 95d00f00 tsdata: 00000000
    cpu_addr: ffc00f00 lastwrp: 95d01000 tsdata: 00000000
    root@dm814x-dvm:~#
  • Hello, Mike

    I just wanted to ask you: you said that the customer goes for the 1GB, but the following line is present:

    .end            = PHYS_OFFSET + SZ_2G - 1,

    Can we clarify this?

    I am investigating this issue and I'll also try to involve someone else here.

    BR

    Vladimir

  • We've changed the offset to SZ_1G but that did not fix the issue. 

    We are able to IOREAD/IOWRITE to the BAR physical memory locations:

     BAR 1: [mem 0x20000000-0x2000ffff]
     BAR 2: [mem 0x20010000-0x2001ffff]
     BAR 3: [mem 0x20020000-0x20021fff]
     BAR 0: [mem 0x20022000-0x20022fff]

     However, MEMWRITE fails when writing to the physical DMA memory locations.  The following output occurs after the MEMWrite

                   Virtual (read)       Physical (target)  Data(result)
    cpu_addr: ffc00100 lastwrp: 95d00200 tsdata: 00000000
    cpu_addr: ffc00200 lastwrp: 95d00300 tsdata: 00000000
    cpu_addr: ffc00300 lastwrp: 95d00400 tsdata: 00000000
    cpu_addr: ffc00400 lastwrp: 95d00500 tsdata: 00000000
    cpu_addr: ffc00500 lastwrp: 95d00600 tsdata: 00000000
    cpu_addr: ffc00600 lastwrp: 95d00700 tsdata: 00000000
    cpu_addr: ffc00700 lastwrp: 95d00800 tsdata: 00000000
    cpu_addr: ffc00800 lastwrp: 95d00900 tsdata: 00000000
    cpu_addr: ffc00900 lastwrp: 95d00a00 tsdata: 00000000
    cpu_addr: ffc00a00 lastwrp: 95d00b00 tsdata: 00000000
    cpu_addr: ffc00b00 lastwrp: 95d00c00 tsdata: 00000000
    cpu_addr: ffc00c00 lastwrp: 95d00d00 tsdata: 00000000
    cpu_addr: ffc00d00 lastwrp: 95d00e00 tsdata: 00000000
    cpu_addr: ffc00e00 lastwrp: 95d00f00 tsdata: 00000000
    cpu_addr: ffc00f00 lastwrp: 95d01000 tsdata: 00000000
  • My customer added a full scan of the 2Mbyte on the DMA region to see if any data had been received. This was just to check to see if the DMA had written to some other section. The entire 2Mbyte is all '0' after the DMA.
  • Another debugging update:
     
    1) Attempting to write to the virtual memory address assigned to DMA behaves the same as physical address memory write. The data reads back as all '0'.
     
    2) Changing the transfer size to 46 words instead of 47. This caused a root complex fault and kernel trap. This implies that the root complex is receiving the PCIe MEMwr transaction layer packet.
  • Another debugging update:

    Advisory 2.1.6 in the errata states.
    Details: Data is corrupted if 16-Byte multiples and 16-Byte boundary transfer rules are not met.
    Workaround: The User must ensure that all EDMA transfers to PCIESS slave are aligned to 16-Byte boundaries and any misaligned accesses are performed as single access.
     
    To account for this we will  attempt to transfer 48 words instead of 46 in order to be 16 byte aligned. 

  • We received feedback that the customer has solved this issue. The customer took into account the restrictions described here:

    Chapter 19.2.3.3 Address Alignment Requirements from the DM814x TRM

    Thank you very much.

    BR

    Vladimir