This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMS320C6414: McBSP EDMA - Issue

Part Number: TMS320C6414

Hello TI - experts,

could you please assist me in the following topic, in fact an issue.

Application description

DSP-application running on C6414 (TMS320C6414TCLZ of last silicon revision 2.0) makes use of all three McBSP ports driving EDMA.
RX- and TX (REVT, XEVT) get handled simultaneously, using common ping-pong technique. DSP/BIOS is not used, only CSL and EDMA_INT(errupt) out of which the whole signal processing is performed. EDMA_INT is the only interrupt in the application.
Instead of EDMA-interrupt, application uses the interrupt to the CPU (XINT) that the event generates:
Transmit interrupt (XINT) mode - is generated by a new frame synchronization.

McBSPs running in TDM mode (128 timeslots each), get clocked completely externally. Clock for all three McBSPs is of the same origin:

FPGA spreads the same serial bitclock and frameclock as external clocks to all three McBSPs.

Shared EDMA_INT interrupt is used for all transfer-complete (tcc) interrupts. Out of it, EDMA CIPR (Interrupt pending register) is monitored in order to detect the transfer completion. More precisely, for certain time (XINT) interrupt waits for the TX and RX indication of completion of all 3 (McBSPs) transfer complete codes. 

As proposed in SPRU234C EDMA Ref-Guide (paragraph 1.15.1) the ISR checks (waits) for all pending interrupts and continues until all the posted interrupts are serviced.

Speaking of (E)DMA, besides McBSP, in an application there’s only one more peripheral also utilizing EDMA:
Host Port Interface (HPI), operating wordwise, requiring no software-setup. 

Initial EDMA-setup Transfer Request Queues Q1 (3 x McBSP) and Q2 (HPI) are used (PQAR and TRCTL registers using default setup).
Element synchronized 1-dimensional transfers are used. QDMA is not used.
Issue Description

On some boards, it happens randomly (6-10 times/24 hours) that completion of all 6 McBSP transfers (Rx & Tx for all 3 ports) takes longer time than usually! 

In the worst case, already next frame arrives leading to its loss!
What happens?

On affected boards, both RX & TX transfers for all three McBSPs get sometimes very delayed (RX transfers the most).
EDMA Channel Interrupt Pending Register (CIPRL) transfer complete codes (TCCs: 6 of them) do not get set! 

Measurements performed Of course we have performed measurements of serial clocks. At the moment of issue-occurence and some time before, no clock miss-behaviour could be observed.
Attempts made to handle the issue

Since requests on a single priority level are serviced serially and in order; whereas, requests on different priority levels can be serviced concurrently, I tried to change the usage of EDMA queues in order to benefit from parallelism.

I changed the initial EDMA-setup by utilizing more queues for McBSPs in the following way:
Q0: McBSP0 Rx, Tx; Q1: McBSP1 Rx, Tx; Q2: HPI, McBSP2 Rx, Tx).

Unfortunately, in spite of these changes, the failure still occurs. 

My Question:

What else could I do in order to narrow down the failure? What could cause the issue?

Thanks!

Mladen

  • The team is notified. They will post their feedback directly here.

    BR
    Tsvetolin Shulev
  • Please let me append two images as an illustration of the issue.

    Yellow signal on the very bottom is a signal showing the processing time of an ISR.
    Other signals are McASP0 BCLKX (Bitclock), FSX (FrSyncTx), DX (TX-data)

    The first image illustrates a good-case showing that ISR needs some 350 ns to process (including waiting for all 3 x 2 events to get pending in CIPR):

    In ISR it is waited for  all 6 events to get pending in CIPR which in normal case, occurs almost simultaneously. 

    The second image illustrates an issue - case: for some reason ISR needs much more time to process (846 instead of 350 ns):

    The next frame gets completely lost (see yellow signal on the very right), remains high. 

  • Mladen,

    Could you clarify several things about your system operation, please?

    malden2507 said:
    all three McBSP ports driving EDMA. RX- and TX (REVT, XEVT) get handled simultaneously, using common ping-pong technique.

    The common ping-pong technique that I have seen used is to have a full buffer of data ready to transmit by the time the previous buffer of data has been transmitted, and to have one buffer of receive data being processed while another buffer is allowed to fill. This technique allows an entire 128-channel frame period for all processing to be completed. Your method seems to be different, so could you explain your method of ping-pong buffering, please?

    malden2507 said:
    DSP/BIOS is not used, only CSL and EDMA_INT(errupt) out of which the whole signal processing is performed. EDMA_INT is the only interrupt in the application.

    This description sounds like the entire application on the DSP is servicing the EDMA_INT, so there is nothing else that could interrupt the signal processing code. Is that correct?

    What is the result of the 'whole signal processing'? Does this consume the latest receive data and generate the next data to be transmitted? If so, is that new transmit data expected to be sent out during the same data frame in which the signal processing is done? That is what I imply from the fact that running the ISR for too long causes the whole 128-channel transmit frame to fail.

    To help you identify and debug the problem, I suggest that there are at least 4 areas that could cause this:
    1. Something in the algorithm being processed takes extra time once-in-a-while and that is what causes the lengthened ISR time. This is unlikely because you would have figured this out long ago.
    2. The EDMA internal operation gets stalled because of other things going on in the system. This could be from cache activity or from memory conflicts at the memory endpoints. This is unlikely from your system description, but something to consider.
    3. Some interaction is occurring between the ISR and the next set of EDMA transfers. Some critical timing window is not met, so an EDMA event gets missed and the data transfer does not occur until the next frame.
    4. The EDMA speed or the DSP speed are not able to keep up with the McBSP data rates.

    An 'easy' thing to try for narrowing down the source of the problem is to vary the clock rate of the McBSP data. If you run it 10-25% slower and the problem continues or if the problem goes away completely, either result will help to find or eliminate a source of the problem.

    You could also try varying the speed of the DSP, although that is usually more difficult, especially making it run faster since memory timing also changes.

    It would also be good to try reducing the processing requirement in the signal processing code that is performed during the ISR. Good tests would be to simply comment out portions of the code, and to simply reduce the number of channels processed. The advantage of reducing the number of channels being processed is that the code will still have the same memory signature as before and you can watch for variation in the ISR duration even with the shorter processing time.

    What is the data length of each of the TDM channels?

    What memory target endpoints are you using (internal, external)?

    What else is running, if anything, while the ISR is executing?

    What other uses of the EDMA could be happening at the same time as the ISR?

    That is my start of questions and comments and suggestions. My usual recommendations would be to make the ISR run faster, or make the EDMA more efficient, or rework the code system architecture to allow more time for the ISR. But your clarifications and test results will help to point out more clearly where to look.

    Regards,
    RandyP

  • Hello Randy,

    here answers to your questions.

    • The ping-pong technique:
      Yes, we indeed do use the ping-pong technique, but of buffer depth of only one, from the perspective of single TDM-channel or timeslot (1D transfer is used for 128-channel TDM, i.e.1 sample per channel only - for the sake of audio latency). So for example, during one 128-sample frame is read, another one is filled by EDMA.

    • Interrupt  generation:
      I've made a mistake in the very first post (that I've meanwhile revisewd, see removed/edited parts in blue).
      Instead of EDMA-interrupt, an application uses the interrupt to the CPU (XINT) that the event generates.
      Transmit interrupt (XINT) mode - is generated by a new frame synchronization.

    • There is nothing else that could interrupt the signal processing code. Is that correct?
      Yes: the XINT interrupt is of the highest priority (there's only one more software-interrupt of lower priority).

    • The issue can be described this way:
      although a new-TX-frame interrupt occurs, sometimes/rarely no events can be detected in
      EDMA Channel Interrupt Pending Register (CIPRL) - transfer complete codes (TCCs: 3x2 of them) do not get set!

    Meanwhile, we have an assumption that possibly external clocks get messed.

    Regards,
    Malden

  • Hello Randy,

    please allow following question (related to the same topic):

    What do you think about the following idea of mine?

    C6000 McBSP manual mentions an unexpected receive/transmit frame synchronization as well as offers control bits RSYNCERR/XSYNCERR to issue a dedicated miss-synchronisation interrupt.

    In SPCR (Serial Port Control Register for McBSP) we normally use XINTM (Transmit Interrupt Mode) bits:
    2h: XINT is generated by a new frame synchronization.
    Thus we have a XINT interrupt for each (externally supplied) frame.

    My prepared “test-feature” consists in preparing the special setup for only one of the ports, say McBSP2 to let XSYNCERR get “visible”.

    I’ve prepared a following “test-feature”.

    Idea Using the XINT-setup for XSYNCERR in order to inspect/evaluate the quality of externally supplied clocks. 
    In other words, to monitor/detect a synchronisation issues. 
    Implementation
    details

    SPCR2 (Serial Port Control Register for McBSP2) XINTM-section is configured in a following way: 
    3h: XINT is generated by XSYNCERR

    A dedicated interrupt routine is prepared, mapped to IRQ_EVT_XINT2 for it letting it increment some (prepared) global variable on its occurence. To evaluate if interrupt occurred at first.
    Later I intend to toggle some test-pin that can be used for measurements using an oscilloscope.
    Final implement. test

    Manually set XSYNCERR / SPCR2 register, then deselected it afterwards.
    And really: the prepared XINT-interrupt occurred (incrementing my test variable)! 

    Real-time

    Unfortunately, the same implementation does not manage to detect the clock misbehaviour at realtime. Why?
    Prepared interrupt simply does not get triggered.

    I think that following prevents XSYNCERR-interrupt from occuring:

    XFIG (Transmit Frame IGnore bit) in Transmit Control Register (XCR) must be cleared!

    Here XFIG meaning:
    0 Transmit frame-synchronization pulses after the first pulse restarts the transfer.
    1 Transmit frame-synchronization pulses after the first pulse are ignored. 

    Please confirm if I understood all well and if XINTM-setup for XSYNCERR could be used to inspect the quality of externally supplied clocks. 

    Regards,
    Malden

  • Hello Randy,

    everything works well, as I've expected. 

    SPCR (Serial Port Control Register) XSYNCERR (bit 19) or RSYNCERR (bit 3) can be used to simulate a McPSP port-issued error condition. 

    As if such a failure occures fo real, upon McBSP detection of frame sync error (expected frame sync pulse. 

    Thus the XSYNCERR feature of McBSP can be seen as an excellent measuring feature!

    XSYNCERROR used to drive a XINTM interrupt; in conjunction with some indication (via say dedicated GPIO pin) can be used to support measuring the external measuring equipment (say triggering the oscilloscope) and repair the external clock issue.

    Back to our primary topic: The McBSP XSYNCERR feature shows that external  clocks our DSP uses were not stable as they should have been. 

    Regards,
    Mladen

  • Mladen,

    Thank you for the status update, and great job getting to the source of the problem. We are glad there were measurement features that helped.

    Please mark you post above as a Verified Answer so this thread will be closed and future readers will know which post is the most important to find the answer.

    Regards,
    RandyP