• State Resolved
  • Locked Locked
  • Replies 7 replies
  • Subscribers 97 subscribers
  • Views 1011 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CPU DMA interrupt stops occuring even though dma's are still running.

Shawn Steenhagen
Expert 1545 points

CPU: 5535, CCS v4.2, CGTOOLS 4.3.9, BIOS: 5.41.13.42 Platform:5535ezdsp

We are having trouble with the aggregate DMA interrupt.

We have a SW system that is running the Codecs at 48Khz Stereo and streaming the adc data to an SD Card.

We are using the dmas for the rx/tx codec transfers and for the transfers of data to/from the sd card (mmc0), so we essentially have three dma sources that can create a CPU DMA Interupt.

dma1:ch0-3 are used for codec. dma0ch0 for sd writes, dm0ch1 for sd reads.

the dma_isr( ) is pretty straight forward, here is some psuedo code:
dma_isr()
{

ifrValue = CSL_SYSCTRL_REGS->DMAIFR;
CSL_SYSCTRL_REGS->DMAIFR = ifrValue;  // clear/acknowledge all latched events imediately.


if (ifrvalue shows rx/tx codec event has happened)
{
    service the rx/tx code interrupt
}

if (ifrvalue shows sd card event has happend)
{
   service the sd card interupt.
}
}//done


The system runs fine for the most part, but after 1/2 to hour of continuous running, will suddenly just stop. By stop I mean we stop getting all dma interrupts.

When I halt the system with the emulator, this is what I see.

IER0: 0x0130  (dma int (0x0100) is enabled)
IFR0: 0xE010  (no DMA int pending).
DMAIFR = 0x00F0 (shows all 4 codec dma events have happened)
DMAIER = 0x00F0 (shows all 4 codec dma events are enabled).

So somehow the codec dma events failed to trigger a CPU DMA INTERUPT in the IFR0.

If I continue running the emulator, I see that the dmas are still running, and all my other threads are still running, its just that we have stopped getting the CPU DMA interrupt.

I put in observation variables at the end of my dma_isr( ) and I know that on occasion, while I am in the dma_isr for one type of dma event, a new DMAIFR bit has been set for one of the other events. In this situation, I see the the DMA interrupt bit of IFR0 is also set. Lets call this situation a latentDMA event.  For example I might be servicing a sd event and I'll see at the end of the isr that a new DMAIFR bit is set for the codec dma event.  I view this as a normal situation, as I expect upon exit of the dma_isr( ), once the DMA INT bit of IER0 is re-enabled, that it will generate another dma_isr to service this new one.

On even rarer occasion, I see at the end of my dma_isr( ) that a new DMAIFR bit has been set as described above, HOWEVER the DMA interupt bit of IFR0 is NOT set. Lets call this situation a missedDMA Event.

When the system breaks, is see that in the last dma_isr( ) a missedDMA Event occured.

However, a missedDMA Event didn't always break the system. For example in one run, out of 533535 dma isrs, there were 54 missed DMA events. The last one that occured broke the system. In that missed event, we were servicing the sd dma event when a new Rx codec dma event came in.

So something in the timing and/or mechanism of acknowledging the DAMIFR so that the next DMA event will trigger a CPU DMA INT is broken.  Its seems I have done all one should do on the SW side of things: acknowledge the events that occured, service them, return.

Are there any known pipeline issues, silicon issues, SW bugs anything that can explain why the aggregate DMA interrupt architecture can ocassionaly fail if another DMA interupt comes in while we are in the dma_isr?

What is the timing of a DMAIFR bits being set to the eventually setting of the CPU DMA interrupt bit in IFR0?  what is the silicon logic that determines when a new DMAIFR event will create a new CPU DMA INT?

Any help will be greatly appreciated.

-Shawn

  • 0
    TI__Mastermind 30830 points

    Shawn,

    Some of your questions regarding how internal hardware signals ripple through the device are beyond the scope of the BIOS Forum.

    The description of  these symptoms plus the similar issue you're having with the TINT interrupt makes me wonder whether you might be clocking your CPU a little faster than it should be or if not, perhaps the clock signals are a little glitchy.

    Alan

  • 0 in reply to Alan DeMars
    Expert 1545 points

    Alan,

    thanks for your comments.  Knowing that the issue may be at a low level, I also posted the same question on the c5000 forum, but also posted here since it was a bios app and wanted to see if any other bios users had seen a similar issue.

    The app is running on an ezdsp at 100Mhz, so nothing out of the ordinary with clocking.

    If any other thoughts come to mind, please let me know.

    thanks

    S

  • 0 in reply to Alan DeMars
    Expert 1545 points

    Alan,

    Here is an update.  I modified the bios config such that within the dma_isr( ), I have disabled other interrupt sources so that dma_isr( ) will not be interuptable.  This seems to have fixed the problem. I successfully ran for about 19hrs with no failures! (enough data to almost fill a 16G sd card).  I stopped the emu just to check on some of my metrics.

    Total dma_isrs: 17,620,985
    rx dma isrs: 6,320,876
    sd dma isrs: 11,300,685
    latent Dma Events: 2040
    Missed Dma Events: 0!

     So it seems:

    If I have other interrupt sources enabled during the dma_isr( ), we create the rare situation of missing dma interrupts if a new dma event occurs before dma_isr completion.

    If interrupts are essentially disabled during the dma_isr( ), we never miss a dma interrupt if a new dma event occurs before the dma_isr( ) completion.

    Do you know of any requirement/restriction that would dictate that the dma isr needs to be NOT interrupt-able?

    So at this point there is either :

    (a) Something wrong with BIOS in the way its HWI dispatcher works that breaks the aggregate DMA interrupt if other interrupts are enabled, a new HWI comes in (interupting the dma_isr), and then another dma event occurs before the dma isr has completed.

    OR

    (b) Some timing issue within the CPU itself that breaks the aggregate DMA interrupt if interrupts are enabled within the dma_isr, it gets interrupted, and then another dma event occurs before the dma isr has completed.

    Thoughts?

    -S

  • 0 in reply to Shawn Steenhagen
    TI__Mastermind 30830 points

    Do any of your interrupt handlers write to the IFR register? If so, they may be inadvertently clearing the subsequent dma interrupt.

    Alan

  • 0 in reply to Alan DeMars
    Expert 1545 points

    Hi Alan,

    That is a fair question.  In the isrs I've been dealing with, I don't do anything with the IFR.  But there are some low level drivers/library functions that I link with whose source code I haven't seen.  Let me check into those and get back to you.

    -Shawn

  • 0 in reply to Alan DeMars
    Expert 1545 points

    Alan,

    I did my investigation and posed my questions based on the initial information I had at the time, which was that none of our ISRs were manipulating the IFR. 

    But after you and a couple of other people raised this question again, we dug deeper and found a rouge line of code in one of our ISRs that was manipulating the IFR, explaining all of the observations I made during my investigation.

    We've modified that ISR and we are re-running the tests with this new code.  I expect all will go well, but will know for sure after about 24hrs.

    Thank  you for providing suggestions and insightful questions that led us to finally solve this issue.

    -Shawn