OMAP L138 EDMA3 and SPI0 issue

Hi
Questions
1) Is it an occassional dropped word or always a specific word at a specific time instance? Trying to figure out if it is an initialization/synchronization type issue or truly a priority/contention type issue
2) What other traffic do you have in the system and TC1?
3) Does the "dropped word" issue go away if do not have other traffic on TC1 or reduce SPI clock frequency>
4) What is the source/destination for other traffice - is this also going in/out of L2? If so can you try to put this on SHRAM or DDR?

See the SCR section in the following wiki
processors.wiki.ti.com/.../AM1x_SoC_Architectural_Overview
to see if you have any common source/destination bridges that may be causing a choke point
Hope this helps

Hi Mukul,

1) It is only occasionally and is random.

2) We have McBSP0 and McBSP1 running at 50Mbit/s on TC1. CC1 is doing QDMA transfers between L2 and mDDR and mDDR and L2

3) SPI frequency reduction is not an option (it is a function of the SPI master). I can try and take out the TC1 traffic, but it is difficult as we have a dual OMAP design with designated tasks to each core. I can possibly reduce some of the traffic to see if it makes a difference.

4) The EDMA McBSP traffic is using un-cached mDDR, but our QDMA transfers are transferring from/to L2. 

My experience is the SPI as master with EDMA rather than your case of McBSP as slave. Not quite sure if you are using L2 or DDR to/from the McBSP. I found that using DDR resulted in pauses in a continous transfer of words over SPI. I am guessing the DDR is busy doing a refresh and pauses the DMA. Using L2 removed the pause. In my case, a irregular SPI master clock is annoying but functional. In your case, the pause might cause an receiver overflow.

Thanks.
It appears that you may have a contention issue

Few suggestions to try out
1) Try to move your SPI traffice to SHRAM instead of L2 ( I think L2 is getting heavier traffic from QDMAs) - or if you need SPI in L2, try to move your QDMA traffic to SHRAM instead (instead of L2)
2) Try to throttle your QDMA traffic - is the requirement on this link time critical? If not , make sure you are breaking the QDMA transfers to smaller data chunks per transfer request - if you have a big transfer size (ACNT*BCNT) going out per event - it maybe back to back transfers to L2 that may be starving your SPI /EDMA traffic due to head of line blocking somewhere (even though you have the TC0 at higher priority).
3) If CC1-TC0 is only going to be used only for memory to memory transfer (and no other DMA etc) *and* your system is ok with lower throughput, you can also play with the TC.RDRATE register - this register setting puts dead cycles between every read issued by TC - it can be detrimental for overall TC performance but perhaps acceptable to your system.
4) It is important to understand/isolate the offending / contention traffic - is the McBSP transfers or the QDMA transfers or both? If it is the QDMA transfers - it is likely L2 that is your common choke point, if it is the McBSP transfer it is likely the BR3 and BR4 in the system interconnect, where you could have potential head of line blocking in the bridges.

OOPS

I did not mention that the clock of SCR is the same as DSP/ARM clock. 375-476MHz clock (see the data sheet www.ti.com/.../omap-l138.pdf )

Ran

OOPS number 2

I did not mention that the clock of SCR is HALF of   DSP/ARM clock. 375-476MHz divide by 2 clock (see the data sheet www.ti.com/.../omap-l138.pdf )

Ran

Hi Norman,

Thanks, we are already using L2. mDDR has indeed shown to have problems for SPI transfers.

Hi Mukul,

Some of these suggestions will take some time:
1) Does the EDMA3 cache snooping work for L3 SHRAM? If not, moving to SHRAM, I would have to think about cache instructions to be added in. SHRAM in our system is cached
2) The QDMA traffic is not THAT time critical, although still part of a real time system. Transfers are set up as AB transfers as bytes, which is probably fairly inefficient. I'll switch to word transfers. How can I throttle QDMA
3) I've tried TC.RDRATE - it does not seem to make a difference
4) OK, is there any priorities that can be assigned to BR3 and BR4?

Sorry, 2) should read:
The QDMA traffic is not THAT time critical, although still part of the real time system. Transfers are set up with ACNT equal to size of the transfers and the BCNT as 1 (transfers are less than 64k. How should I set up the QDMA to throttle the transfers?

Hi Pirow

On 1, no DSP caches will not stay coherent if another master touches Shared RAM/DDR (manual cache operations will be needed)

On 2, It appears like for QDMA event you are transferring a big ACNT (BCNT=1)? If so, you are likely using the TC very "efficiently" and you may need to insert some dead cycles between transfers , by either manipulating RDRATE for this TC ( looks like this is not helping) - or further break down your transfers to smaller chunks - easiest way to do this would be to use a DMA channel (instead of QDMA) and have chaining enabled so that with a single ESR write and intermediate chaining you break your ACNT x BCNT transfers into smaller transfer chunks using ACNT = (smaller than current size) , BCNT = total transfer size (this will also be the number of TRPs that go out to TC instead of one big TRP per chained event).

On 3, see above

On 4, the priorities that you assign to the various masters - in your case the TC priorities via the MSTPRI registers is what is "globally" seen and respected on all interconnect components (Bridges and SCRs) "if" there is an opportunity to arbitrate. But remember, if the higher priority request did not show up at the time your lower priority QDMA request showed up you could have several lower priority access chunks slip in the bridge fifo - where it is just first in first out. The default arbitration size is 16 byte chunks for EDMA TCs ( i.e. the tc internally breaks the request to default burst size/DBS chunks)


You may want to see if playing with the QDMA memory x'fers TC default burst size (via CFGCHIP0 in SYSCFG) makes any differences. It could work in favor or against - depending on the nature of your SPI traffic etc.


Memory re-assignment and breaking your memory to memory transfers into smaller chunks by changing the EDMA programming are likely going to have the most impact to improve system traffic and reduce chances of head of line blocking etc.

Some confusion on my part. Are the SPI-->L2 transactions related to the L2-->mDDR transactions? Does the same packet travel through one then the other? If so, is the dropped word detected in L2 or mDDR? I've been assuming master to slave only. Is there slave to master as well?

Hi Norman,

Yes. Basically there is a fast ping-pong buffer in L2 to receive incoming SPI words (each buffer is about 600us deep). Once the DMA engine has finished one buffer it triggers an interrupt that triggers a QDMA transfer to transfer the just buffered block out to mDDR (a deep receive buffer). Another semaphore driven process then brings in the the next block for processing from mDDR to L2 (using QDMA). 

Hi Norman

If you are confused on the path of contention or where the "common" end point is - i have tried to quickly draw out the SPI0 and QDMA traffic paths ( I didn't draw the McBSP path and did not pay much emphasis on the direction of the arrows src/dst etc). 

Hope this helps. spi_edma.pdf

Hi Mukul,

Yes, except there is one more contention path, as the DMA controller is also doing SPI TX from L2 (double buffered).

Mukul, thanks for the diagram. DMA has never been my strong point.

Pirow, Is it possible that the ping pong buffer are getting overwritten before being transferred out? Perhaps your dropped word is actually a word from the next packet. Admitedly, is that happened, you would likely have more than one dropped word.

Hi Norman, the SPI words received are actually synchronised pairs and the buffers are of even length. We are losing exactly one SPI word, i.e. one half of a synchronised pair. I'll add in some explicit checks to see that the buffer that the DMA engine is writing to is not the same as the buffer that is being copied out to mDDR.

Hi Norman,

I've confirmed that it is NOT the ping-pong buffer mechanism becoming un-synchronized. I've added a assert check to check the current destination address of the DMA engine for SPI0 RX and the assert did not fire when the dropped word occurred.

Hi Mukul,

I've shifted the ping-pong buffers to L3 SHRAM, but this does not fix the issue. On this issue, is the L3 SHRAM actually lower latency than L2 for the DMA engine to write to? It seems that for L2 there is the internal C6000 busses to traverse and also L2 cache coherency?

I'll try the QDMA breakup and the transfer controller default burst size.

Hi Pirow
Hmm. I find this a little strange that moving to Shared RAM did not change your failure/failure rate at all.
It would almost seem like the QDMAs from L2 to DDR are not causing the issue , because by moving the SPI rx/tx to SHRAM you should now have completely independent paths

Perhaps it is the McBSP traffic and how it interleaves in BR3/4 that is causing issues with SPI traffic , if so , before trying the QDMA break up (which is likely a good thing in the longer term anyways) - can you confirm that if you were to disable your QDMA or McBSP traffic your SPI will run fine and w/o errors? Only a single data corruption, even though random - could be some sort of software/synchronization issue too?

from a topology perspective the latency to the boundary of L2 and SHRAM is roughly the same for the TC/EDMA, the only advantage is likely when DSP has to access L2 memory it is "closer" to DSP compared to DSP having to go over the interconnect to fetch data from SHRAM etc.