EDMA to McASP and UART causes McASP to fail to start, bizarrely affected by memory map

I will ask the PSP team to answer this question.

Hi,

1.     If the “driver is timing out trying to set RSRCLR in the GBLCTL register”. Can you please confirm that the clocks are coming in from master device to the Mcasp when this problem is observed?

2.     Can you please attach the map file for both the cases

a.     when you are facing the Mcasp problem and

b.     when the Mcasp works fine

3.     Is it possible to send a simple test project to reproduce this issue?

4.     Also the BIOS version you are using is older. Can you please use the same version that the PSP package was tested with. you should be able to see the details in the release note.

Thanks & Regards,

Imtiaz SMA

Firstly, many thanks for your prompt response.

We've updated DSP BIOS to 5_41_03_17 as suggested, unfortunately that made no difference. To answer your other question, yes we can confirm that the master device clock was running. In terms of a test project, that is going to be difficult in the short term (though I appreciate the desire for it.)

However, we have made some progress by moving the memory around and we now suspect this to be something to do with the DMA scheduling.
Having managed to break into the code following a failure, we've found that there was an error in the EDMA handle relating to a missed transfer event - when processing the SIO_Issue, Mcasp_localEdmaCallback was called with the status set to EDMA3_RM_E_CC_DMA_EVT_MISS.

I can't make the full memory map available for you I'm afraid but an overview of the memory layout is at the bottom of the post. In terms of DMA bandwidth, it is reasonably low. We have two DMA channels, one servicing a McASP (data in) and one servicing a UART (data out). For the McASP, this is receiving data from an audio ADC which is running at 2 channel, 32 bit words, 8 khz sampling, leading to a total throughput of approx 512 kbit/s and this is copied into data buffers ( in SDRAM). The UART is idle.

Our theory is that SDRAM is causing a bottleneck and that the processor (which has priority over the McASP DMA requests) is reading data from the SDRAM and excluding the DMA, causing the missed event.

We tried swapping the transfer controllers (so that the McASP was using TC 0 and the UART TC 1) but that made no difference. However we have moved the buffers that the DMA is trying to write the McASP data to into internal L2 ram and this appears to have cured the problem.

A few technical details:

Memory configuration
DSP/Bios objects in SDRAM
All 'BIOS Data' located in L2 apart from .stack (SDRAM)
All 'BIOS Code' located in SDRAM
Compiler Sections:
.text, .bss, .far, .cinit, .const .printf   in SDRAM
.switch, .pinit, .data, .cio in L2
No separate load addresses specified

L2 cache enabled: base 0x11820000, length 0x20000

L2 memory base: 0x11800000, length 0x20000
SDRAM: base 0xC0000000, length, 0x800000

SDRAM speed: 133Mhz 16bit 4 bank
Processor speed: 300Mhz
Before moving the buffers, we found that when located at 0xC06CDA28 -  0xC06DDA28 then the system fails (we have 4 * 16k buffers) (SDRAM starts at)
Moving the buffers back by 8k in the memory range (by shrinking a buffer located earlier in the range) causes the DMA to work. The buffer that we shrink is completely unrelated to the McASP / UART / DMA code.
From a back of the envelope calculation,  it doesn't look like these buffers are located anywhere near a bank boundary of the SDRAM. The internal structure of the SDRAM is 4 bank, 4096 row * 256 column per bank, 16 bit words. However, the working theory is that something else that is critical moves across a bank boundary and this causes another part of the system to slow down enough to cause problems.

A final observation is that stepping through the initialisation code line by line causes no problems, again suggesting some sort of timing issue?

The concern at the moment is that we haven't truly fixed the problem and that instead, we've worked around it (by indirectly changing the timing etc.)  It may be that our memory configuration is the problem and that the SDRAM is a bottleneck. If so, then this is something that we can bear in mind in future. However, it may be that the issue is actually in the DMA configuration, and that changing the memory configuration helps it in this case, but the problem may come back and so any advice would be gratefully received.

Hi Stephen

Thank you for the detailed information.

The EVENT miss error occurs when a new EDMA sync event request comes when there a request pending in TC and in this case it looks like EDMA has write latency problems with SDRAM.

I would suggest to check if the SDRAM read/write performance/latency is normal, particularly in the buffer address range you mentioned (as you mentioned that changing the buffer address made a difference). 

Regards

Vichu

Hi Stephen

Have you also tweaked the value of EMIFB.BPRIO  register

Here is another thread where BPRIO setting made a difference

http://e2e.ti.com/support/dsp/tms320c6000_floating-point_dsps/f/115/p/39441/138663.aspx#138663

Regards

Mukul

Hi Mukul,

I'm a colleague of Stephen's working on the same project.  We have tried changing the values of BPRIO (within the range 0x20 to 0x30) but it doesn't seem to have made a difference.  If we do detect a DMA error - can you suggest how we would reset the state of McASP via the driver? And if it isn't possible via the driver, how we could do it by writing the registers directly without conflicting with the state the driver expects McASP to be in?

Kind regards,

Gavin Lightfoot.

Hi Gavin

I have requested the driver team to address your query on the feasability of doing a reset of the state of McASP on getting a DMA error.  I need to spend some additional time re-reading the description Stephen had put on the earlier posts. Can you please confirm that not much has changed from what was previously described as the problem and your system setup? By DMA error, is it stillt the Event Missed error on the DMA?

Regards

Mukul

Hi Mukul,

We have changed the hardware to run the McASP as master rather than slave, but apart from the clocking configuration the rest of the system is essentially the same.  We have a simple loop calling SIO_reclaim followed by SIO_issue. When things suddenly go wrong - currently this seems to happen when there is a lot of UART(EDMA) traffic - SIO_reclaim returns zero (assuming that this is caused by the DMA error) and then subsequent calls to SIO_reclaim return -9 (-SYS_ETIMEOUT) and calls to SIO_issue return 6 (SYS_EBADIO). These are often the last errors we see just before the system spontaneously freezes or resets.

Regards,

Gavin.

Gavin and Stephen,

1)      You mention “that the processor (which has priority over the McASP DMA requests)” … Can you modify the priority level such that the processor is lower priority than the McASP?

2)      Can you play with the BPRIO setting … try 0x10, 0x4, 0x1?  Any effect?

3)      Is the system completely stable with the McASP buffer in on-chip-sram?

4)      Are you certain that there are no other EDMA transfers in play in the system? 

5)      Is this strictly a startup problem?  I.e., once the McASP is operating in steady state, does it run OK?

Thanks,

Kyle

Hi Kyle,

To answer your questions:

1) Changing the priority of the CPU to lower than the DMA had no effect

2) BPRIO settings (0x10, 0x20, 0x28, 0x30, 0x80) had no effect

3) The McASP buffer has always been in IRAM where we've been seeing these problems.

4) We also have a DMA-driven UART

5) The system works on boot, but after a few minutes when we start reasonably heavy traffic on one of the (interrupt driven) UARTs it falls over.

However: we have found that by defining Mcasp_LOOPJOB_ENABLED when compiling the McASP driver, the problem seems to disappear. Could you tell me whether this is always expected to be defined?

Kind regards,

Gavin.

Hi Gavin,

Gavin Lightfoot said:

However: we have found that by defining Mcasp_LOOPJOB_ENABLED when compiling the McASP driver, the problem seems to disappear. Could you tell me whether this is always expected to be defined?

If "Mcasp_LOOPJOB_ENABLED " is set during the compile time, the driver will be built with feature of loop job buffer. The loop job buffer is a buffer which will be used when there is no request to the driver from the application. So at any point of time, the driver (EDMA) will have either the loop job buffer or the actual valid buffer requested from the application to process.

In your case, whenever you enable the loop job feature, there is no chance of getting the EDMA missed event (if at all it is causing the problem in the non loop job mode) and this will prevent the SIO_reclaim() to come back with the value '0'. Is it ok to you to use the driver with loop job enabled? 

In the non loop job mode (with no "Mcasp_LOOPJOB_ENABLED "), there might be EDMA missed event being generated which is causing the problem. [I am not 100 % sure at this moment].

Conduct these experiments by disabling the "Mcasp_LOOPJOB_ENABLED" and let me know the result: 

1. Could you please put the break point in the "mcaspIsrInput()" and "mcaspIsrOutput()" to see whether there is any other error which is causing the issue?

2. Also put breakpoint in the edma call back function "Mcasp_localEdmaCallback()" under the condition "if (EDMA3_RM_XFER_COMPLETE != status)". This will give you the clear picture of what is the error which you are getting from the EDMA.

        if (EDMA3_RM_XFER_COMPLETE != status)
        {
            /* Clear the EDMA Error bits                                      */
            chanHandle->tempPacket->status = IOM_EBADIO;

            /* There is a data transfer error in EDMA transfer. Return        *
             * actualSize equal to 0 represent that there is an error in the  *
             * packet transfer                                                */
            chanHandle->tempPacket->size = 0;

            EDMA3_DRV_clearErrorBits(
                chanHandle->edmaHandle,
                chanHandle->xferChan);
        }
        else
        {

               ...................

         }

Thanks and Regards,

Sandeep K