Intermittent McASP failure Issue

Thanks Brad,

The edma3MemCpy change didn't change the problem BUT it got me thinking about other memcpy functions.

I've been using the stdc lib memcpy() in a few SWI's, and memset(), so I changed these to be word copies like the edma3MemCpy() instead.

It seems to have fixed my problem.... not 100% convinced yet :)

I'll verify your response once I've done some more testing and reproduced the issue with stdc lib memcpy back in.

Thanks

Steven

Hi Steve,

Any updates? 

Best regards,
Brad

Hi Brad,

Hard to verify this one. Without long soak testing... which we are doing.

But changing all my memcpy()'s and memset()'s seems to have fixed the problem.  
Within the context that my previous test cases, which caused the problem within about 10mins, now last overnight at least.
I may have just moved a timing issue to somewhere else...

I'm going to verify your answer and include my memcpy() and memset() in this post incase anyone wants them.
They seems to have helped when needing to do memcpy()'s or memset()'s within SWI's.

These functions were created from tips/code I got from this web page: http://www.danielvik.com/2010/02/fast-memcpy-in-c.html

Thanks
Steven

void* memcpy(void* dst, const void* src, unsigned int len)
{
    uint8_t* dst8 = (uint8_t*)dst;
    const uint8_t* src8 = (const uint8_t*)src;

    /* Byte copy to word align dst8 */
    switch ( ((unsigned int)dst8) & 0x3 ) {
        case 1: if ( len ) { *dst8++ = *src8++; --len; }
        case 2: if ( len ) { *dst8++ = *src8++; --len; }
        case 3: if ( len ) { *dst8++ = *src8++; --len; }
    }

    /* Store remaining and reduce len to word count */
    unsigned int remaining = len & 0x3;
    len >>= 2;

    if ( len ) {
        uint32_t* dst32 = (uint32_t*)dst8;
        /* Word align src8 and store shift offset value */
        const unsigned int shift = ((unsigned int)src8) & 0x3;
        const uint32_t* src32 = (const uint32_t*)(((unsigned int)src8) & ~0x3);

        if ( shift ) {
            const unsigned int shr = shift * 8;
            const unsigned int shl = (4 * 8) - shr;
            /* dst32 and src32 have different alignments
             * so do sum shift magic while copying */
            uint32_t srcWord = *src32++;
            uint32_t dstWord;

            while ( len-- ) {
                dstWord = srcWord >> shr;
                srcWord = *src32++;
                dstWord |= srcWord << shl;
                *dst32++ = dstWord;
            }

            /* Convert src32 into src8 accounting for shifted address */
            src8 = (const uint8_t*)(src32 - 1);
            src8 += shift;
        } else {
            /* dst32 and src32 are aligned so simple word copy */
            while ( len-- ) {
                *dst32++ = *src32++;
            }

            /* Convert aligned src32 to src8 */
            src8 = (const uint8_t*)src32;
        }

        /* Convert aligned dst32 to dst8 */
        dst8 = (uint8_t*)dst32;
    }

    /* Byte copy remaining */
    switch ( remaining ) {
        case 3: *dst8++ = *src8++;
        case 2: *dst8++ = *src8++;
        case 1: *dst8++ = *src8++;
    }

    return dst;
}
 

void* memset(void* dst, int c, unsigned int len)
{
    uint8_t* dst8 = (uint8_t*)dst;

    /* Byte copy to word align dst8 */
    switch ( ((unsigned int)dst8) & 0x3 ) {
        case 1: if (len--) *dst8++ = c;
        case 2: if (len--) *dst8++ = c;
        case 3: if (len--) *dst8++ = c;
    }

    unsigned int remaining = len & 0x3;

    if ( len > 3 ) {
        uint32_t* dst32 = (uint32_t*)dst8;
        uint32_t val = c & 0xFF;

        /* Repeat byte into 32bit word */
        val |= val << 8;
        val |= val << 16;

        /* reduce len to word count and copy */
        len >>= 2;
        while ( len-- ) {
            *dst32++ = val;
        }
        /* Byte copy remaining */
        dst8 = (uint8_t*)dst32;
    }

    switch ( remaining ) {
        case 3: *dst8++ = c;
        case 2: *dst8++ = c;
        case 1: *dst8++ = c;
    }

    return dst;
}

Thanks for the reply and the additional info.  If you get a minute please provide another update.  Hopefully it will still be going strong after running all weekend!

Hi Brad,

Sorry to say my issue is not solved.  The memcpy's have help but just moved the timing of the issue.

I've narrowed it down a bit by increasing the number of samples per channel requested per McASP buffer request, from 64 from 128.
When buffers of 64(samples) * 8(channels) * 4(32bit) are issued to the McASP the time it takes before the driver moves into the error state is within about 1-2 minutes.
If I increase the samples to 128 the driver is rarely showing the error. I've only seen the error once so far, one days testing, but once is more than enough.

I'll need to break my app down into smaller pieces to get to the bottom of this issue.

Some info in case it rings any bells.

I have two apps that I run on the gpp side, the full audio network streaming and a test app that stores and plays the audio from files.
Both these apps use the same interface library that wraps/hides the DSPLink and DSP app loading.  This library just presents interfaces to pull/push the audio to/from the DSP.

Now when the samples are 64 per channel my test app will usually run quite well but still fail after about 10mins where are the full app using the same parameters will fail within a 1min.
Which seems to suggest to me that the gpp has some influence over the McASP failure.
Which really can only be the DSPLink transfers.
I'm using MSGQ with ZCPY and the DSP is DMA master for these transfers and the PCI bus is share for all peripherals. ie network device.
I've been told, I didn't set this part up so sorry if I get the term wrong, that the EDMA has 2 priority ordered channels and DSPLink usually uses 0. but this has been changed so DSPLink uses the lower priority channel 1 and McASP is using 0.

So my current thoughts are the EDMA driver moves into an error condition caused by a McASP request conflicting with a DSPLink request and maybe a PCI mastering issue.
This is the area I'll be investigating after finishing up so cleanup work.
I'm not very familiar with the PSP lower layers so it'll take a bit, any suggestion welcome.

I'd also like to point out that if I only enable 1 serialiser (2 channels) then is problem new presents itself. Been running these tests for the last 1 week with no errors.

On this point if only 1 serialiser is enabled I can SIO_create() the SIO_INPUT McASP before the SIO_OUTPUT McASP without a problem.
But with more that 1 serialiser enabled I MUST SIO_create() the SIO_OUTPUT driver before the SIO_INPUT, otherwise 45% of the time the DSP started move into this error condition with 2 seconds of SIO_issue()'ing the buffers to the driver.

Any thoughts appreciated.

Steven

Please post your McASP registers, EDMA parameter RAMs (active + links), and EDMA registers. I'd like two dumps actually -- one while things are in a good state and one after things fail. 

Hi Brad,

Sorry I'll have to get these to you in a couple of weeks, I have to move on to some other high priority work just now.
This bug is a background thing I've been trying to resolve.

Also the sad thing is I've not been doing this work with CCS, just makefiles and tools from CCSv3.3.
I'm building on top of PSP work done last year. 

Late next I'll be setting up CCSv4, or maybe I'll try out v5, and get some usefully source level debugging going and get all these registers.

Thanks for your help so far... I'll be back.

Steven

Steven,

Thanks a lot for sharing this valuable information.  Most of our devices in the last couple years have already corrected that particular erratum so I apologize that it didn't come to mind sooner!  That certainly explains the behavior you're seeing.  I've run into the issue a few times and I the observed issue depends on which driver is being used (e.g. some of my customers wrote their own).  I've seen this behavior manifest itself as channel swapping/rotation on some other devices because the delay in servicing the McBSP was long enough to cause a sample to be missed.

This erratum is now resolved in all of our current/new devices (good news!).  Furthermore, TI now generally builds a FIFO on the front end of all the serial ports to reduce the hard real-time deadlines.  So I'm glad you were able to get everything figured out and I wanted to share the good news that for future TI DSP designs this sort of thing will be a bit easier to avoid!

Best regards,
Brad