EDMA3 Upstream (from DSP to host) doesn't work

Bernd Erbe said:

but no data can be seen on host (only zeros)

You mean, the data present in 0x600F8000 is overwritten with zeros?

For the upstream, can you just try with the source same as the dst in downstream. (i.e. 0x84002000 instead of 0x80002000) ?

Make sure if theaddress 0x80002000 is accessible by EDMA.

Hello Radhesh,

thanks for your quick answer. I've to clarify the scenario a little bit:

-I start the DMA upstream from the host (data at destination address, e.g. 0x600F8000, is zero)
-I'll setup the ParamSet in the DSP and start the DMA -> after the DMA has finished, the expected data can be seen at 0x600F8000
-sending a DMA-finished-IRQ to the host, the mapped memory is released and the data at 0x600F8000 is set to zero

However, the data (which can be temporary seen at 0x600F8000) not arrives at the host (we have a test program so we can see the data block at the host; this block is not touched by the DMA, that means if we store a pattern there, it stays the same -> so no data arrives). It is the same behaviour regardless which src address is used (0x84002000 or 0x80002000). Both addresses can be used by EDMA because downstream works for both addresses.

Best Regards,
Bernd

Bernd,

Please help me understand your scenario:

Do you use EDMA3 on DSP side for both "Upstream" and "Downstream" transfer please?

If so, it means the DSP initializes the DMA transfer and it is EDMA read from PCIe data space (mapped to host memory) and write to DDR in DSP for "Upstream".

And it is EDMA read from DDR in DSP and write to PCIe data space (mapped to host memory) for "Downstream", is it correct?

Could you also explain a bit about "the mapped memory is released and the data at 0x600F8000 is set to zero" please?

How the mapped memory is released in your setup please? 

When you check the data arrived at host, are your checking 0x01FF8000 (assuming 0x600F8000 mapped to 0x01FF8000 in host) or other place please?

 And are your able to check the data in host after data transfer completes in DSP and before "mapped memory is released" please?

Hello Steven,

I'll try to clarify your questions.

1) Do you use EDMA3 on DSP side for both "Upstream" and "Downstream" transfer please?

Yes

2) If so, it means the DSP initializes the DMA transfer and it is EDMA read from PCIe data space (mapped to host memory) and write to DDR in DSP for "Upstream". And it is EDMA read from DDR in DSP and write to PCIe data space (mapped to host memory) for "Downstream", is it correct?

Correct, the DSP initializes the DMA transfers. Also the other terms are correct, but I call it vice versa: for Upstream (from DSP to host) I read from DDR and copy it to PCIe data space, for Downstream (from host to DSP) I read from PCIe data space and copy it to DDR.

3) Could you also explain a bit about "the mapped memory is released and the data at 0x600F8000 is set to zero" please? How the mapped memory is released in your setup please?

We also develop the driver for the used components (also for the C6678). Therefore we've have a handshaking between the driver and the DSP when a DMA is set up, e.g. for downstream: driver is telling the DSP that there's data to read -> driver initializes everything (prepares a scatter-gatter-table etc.) -> then gives the DSP the first entry of the table -> the DSP handles every entry of the table

For upstream, the memory where the data has to be written by the DSP, is premapped by the driver only during this process and then released when the DMA has finished (then it is not visible in the PCIe data space anymore). But this is no cause of an error as we also have tried to keep the mapping. Behaviour was the same.

4) When you check the data arrived at host, are your checking 0x01FF8000 (assuming 0x600F8000 mapped to 0x01FF8000 in host) or other place please?

Exact, I'll check the outbound register (0x21800200, e.g. has to be 0x01F00001) and then the address 0x600F8000. There I can see the data (so the DMA has copied it to this address), but for example the array from our testprogram remains with the preset pattern.

5) And are your able to check the data in host after data transfer completes in DSP and before "mapped memory is released" please?

Yes, we are able to keep the mapping and then check the array (which should be the address seen at the DSP, e.g. 0x01FF8000). We are not able to read directly from the internal memory of the host, so maybe the data is somewhere else...

We know that the driver (and its mechanism) works well for other components (e.g. Altera FPGA's) and also for the DSP downstream everything is fine. But it seems that the upstream data never arrives or the DSP is not allowed to modify the internal memory. I've also tried it with a simple memcpy to the given address or modifying the value directly in the debugger (window memory browser; this works for example for every bar over pcie), but always the data is not seen on the host.

Best Regards,
Bernd

Bernd,

Thanks for your clarification.

It looks like the issue is not due to the DMA setup since you tried the memcpy and emulator access. 

From the DSP point of view, both of the Downstream and Upstream are outbound PCIe transactions (outbound read for downstream and outbound write for upstream).

For upstream, are you able to send data to the same host memory region (0x01FF7000) from the same DDR region (0x84002000) which has been verified to be accessible by DSP in downstream case?

And what is the BAR value and the BAR window size on your host side please?

I am wondering if anything different on the host side.

Hi Steven,

we're not sure if the memcpy and emulator access will work for writing into internal memory of the host (we assume, but no other device has done this so we can't say if this is not working, the internal ram is not accessible; all other devices are only using a DMA for this).

Exact, for the DSP both directions are outbound transactions so I assume that the address translation should be correct (but for the driver both directions are also the same: calculate the scatter-gather-table and give the DSP the first entry).

I've tested the upstream from both addresses (new one and the one verified by downstream), but the behaviour is the same (DMA seems to complete, but no data present on host). The bar configuration is as follows:

BAR 0 (default, PCIeConfigSpace)
BAR 1 (L2SRAM, 0x00800000, 512K)
BAR 2 (MCMS, 0x0C000000, 4M)
BAR 3 (DDR3, 0x80000000, 16M)
BAR 4 (SRIOConfigSpace, 0x02900000, 4M)

Problem is that we"ve no possibility to check if an outbound write works correct over PCIe. Checking the registers of the CC- & TC-Block of the EDMA, no error can be seen and the registers have the same values for both upstream and downstream (except the changed addresses).
So we're searching for any other debug possibility...

Best Regards,
Bernd

Bernd,

For outbound transaction from DSP, please take a look at the BAR setup on the HOST side, not on the DSP side.

We should make sure the BAR on HOST side could accept the PCIe packets and the host could forward these packets to its internal destination memory (like the inbound translation mechanism in C66x DSP PCIe module, but the host may use other mechanism).

Since you tested both the downstream address and new address for upstream, it may be better to verify the HOST internal memory is writable first (not only readable). 

Hello Steven,

the BAR configuration was read out on the host side via 'lspci', so it is already the BAR setup seen and used by the host. What's the coherence between the outbound write and the BAR setup? I thought the outbound write goes 'directly' into the internal memory address (given by the host to the DSP) independent from the BAR's.

For your second hint, that's our problem: we don't know how to verify if the internal memory can be written by the DSP. Other components can with DMA, but from the DSP no method has worked (as mentioned above a memcpy, debugger access and DMA)

Best Regards,
Bernd

Bernd,

You are right that RC/host will accept packets as long as the PCIe address is outside of the base/limit ranged defined by non-prefetchable memory, prefetchable memory, and I/O Base/Limit register sets. Although we can use BAR on RC side to make a window to accept the packets from EP (at least applicable for C66x PCIe module in RC mode), but it is not mandatory.

For the original question, I am wondering if it is a cache coherency issue on DSP side. Is the memory region in DDR being cached by L1D/L2 in your setup please? If so, you need to write-back those data to DDR memory from cache, before you do the transfer from DDR to the host memory via PCIe, otherwise the host may only see the stall data in DDR which could be 0s. 

Hello Steven,

thanks for clarifying the coherence between BAR's and PCIe. With the cache I already had some trouble with other issues; so here I disabled the cache for the whole DDR3 and also checked it in the debugger that the data is visible.

// Disable caching for DDR3 (0x80000000) and MSI-Data (0x21800000)
for(iIndexI = 128; iIndexI < 144; iIndexI++)
{
        CACHE_disableCaching(iIndexI);
}

Best Regards,
Bernd