TMS320C6657: EDMA3 ping pong buffer issue to identify which buffer EDMA3 is actively writing.

The team is notified. They will post their feedback directly here.

BR
Tsvetolin Shulev

Hi Eric,

Thanks for the response.  Here are my answers to your questions.

Can you explain what kinds of EDMA code is used? EDMA LLD?  CSL or register level of code?

[SQ]: CSL.

Why the EDMA generate an interrupt?

[SQ]: Transfer completion interrupt.

Is it because of transfer completion?

[SQ]: Yes.

The EDMA transfer parameters are in the parameter set (like OPT, SRC, DST, ACNT, BCNT etc ...). How do you know which parameter set is used?

[SQ]: I use LINK in the parameter set to reload ping/pong buffers.

Are the EDMA is configured as static or not?

[SQ]: No, linked.

If it is not static, the parameters would disappear after transfer, how can you read them back after transfer (to determine destination address)?

[SQ]: with linked parameter sets, transfer completion cause the parameter set reload.  By reading the DST, I should be able to tell the buffer EDMA just completed.

If it is static, the address is still there why you can't read it back?  [SQ]: NA.

Another way, since it is a ping/pong buffer, the destination address is fixed at address A or B. You can use a flag to determine which buffer is written already with new data, then just process this buffer.

[SQ]: Yes, I can use an alternating flag.  But dynamically determine it would be better.  Since system could overload and skip/lose an interrupt, such as breakpoint hit.

But sometimes, the DST show A buffer address which means EDMA is writing into it.  But in actuality, EDMA is writing into the B buffer.  That is my problem!

Thank you.

Steve

Hi,

In the EDMA, OPT has configuration for when the interrupt generated: Intermediate transfer completion interrupt enable and Transfer complete interrupt enable. Is possible that you used Intermediate transfer completion interrupt enable, can you check?

"The only difference is that McBSP EDMA3 uses 3 dimension transfer, whereas EMIF16 uses 2 dimensions. Both uses AB-sync.", are you able to make mcbsp 2-D or EMIF 3-D to see if the difference caused the issue?

Regards, Eric

Hi Eric,

I do use transfer completion interrupt (XFR_COMPLETE_INT_ENABLE), not intermediate transfer completion interrupt(XFR_COMPLETE_INT_ENABLE).

I cannot make EMIF16 EDMA 3 dimension.  Because McBSP has frame sync continuously trigger EDMA, whereas EMIF16 need an interrupt chain for continuous transfer.

It is straight forward that when EDMA is working on a ping buffer (indicating by the DST).  It is not accessing pong buffer.

I believe that it is an obvious EDMA3 bug.  I need TI to confirm it and provide a work around.

Steven Ji has provide good support to me a few years ago.  Is he still at TI doing support?

Thank you.

Regards,

Steve

Hi,

Do you use TI 6657 EVM or some customized board? If you feel this is a bug, do you have details of setup and CCS project for us to reproduce your issue?

Regards, Eric

As Eric mentioned , you will likely need to send us something that is reproducible on the EVM to debug this further. I would be very surprised if there is a bug around this - this is a 10+ year old IP and we have not heard of any bug in this area -

>But sometimes, the DST show A buffer address which means EDMA is writing into it.  But in actuality, EDMA is writing into the B buffer.  That is my problem!

Can you further elaborate on this

Ping Buffer - DST A -  Transfer Complete - Link to Pong Buffer / DST B loaded (automatically in HW) - In ISR check if DST B loaded - do something  (process DST A data) - Trigger DST B transfer - Transfer Complete - Link to Ping Buffer / DST A loaded - in ISR check if DST A loaded - so something (process DSTB data) - Trigger DST A transfer.

 

Can you also confirm that TCCMODE is not set to early completion? How do you know that EDMA is writing to buffer B ?

 

PS Steven Ji is no longer with TI 

 

Hi Eric,
We use a customized board similar to the TI 6657 EVM.
Regards,
Steve

Hi Mukul,
I setup EDMA for ping pong buffers with the following parameter sets.
80124004 70004000 00400100 108C3D80 01000000 000043A0 00010000 00000001 (active)
80124004 70004000 00400100 108C3D80 01000000 000043A0 00010000 00000001 (ping)
80124004 70004000 00400100 108C7D80 01000000 00004380 00010000 00000001 (pong)
As a transfer complete interrupt happens, HW reloads the ping buffer, and I re-trigger another EDMA transfer.
CSL_edma3HwChannelControl(EMIF16hChannel,CSL_EDMA3_CMD_CHANNEL_SET,NULL);
Then at the processing code, I read the active DST. If it is 108C3D80, CPU will process pong (108C7D80) buffer.
Otherwise, CPU will process ping (108C3D80) buffer. So EDMA and CPU should always working on different buffers without conflict.
But sometimes, DST indicates EDMA is writing in ping (108C3D80), and actually it is writing pong (108C7D80).
There isn't much else to elaborate.
Thank you.
Regards,
Steve

Hi Steve
Thanks for sharing the PARAM dump - I see that TCCMODE is set to normal completion
I am out next week so will not be able to look at this more carefully , but I guess a few things to check/comment

1) What is the syncrhonization between your ISR and processing code? How do you know that when you snapshot the param DST address in processing code, it has not already link reloaded the ping buffer? The set/ESR write in ISR to link reload on completion is done in hardware?
Could it be that the transfer is faster compared to when you get to look at the param value? If you think this could be happening, you could try bigger ACNT/BCNT as an experiment (assuming your EMIF16 interface allows it) to see if bigger transfers = more time to finish transfer make it harder to hit the condition of ping reloaded?

Hi Mukul,

Thanks for the quick reply.  I am answering your questions so they will be ready when you come back.

1) What is the synchronization between your ISR and processing code?  [SQ]: I use Semaphore_post().

How do you know that when you snapshot the param DST address in processing code, it has not already link reloaded the ping buffer? [SQ]: In fact, the parameter set has reloaded in processing code.  I read DST in ISR and in processing code.  They are always different from one another.

The set/ESR write in ISR to link reload on completion is done in hardware? [SQ]: Yes.

Could it be that the transfer is faster compared to when you get to look at the param value? [SQ]: No.  EMIF16 is for audio input and has constant transfer rate of 8192 bytes per 1333us, evenly spread out.

If you think this could be happening, you could try bigger ACNT/BCNT as an experiment (assuming your EMIF16 interface allows it) to see if bigger transfers = more time to finish transfer make it harder to hit the condition of ping reloaded?  [SQ]: I use two dimensional AB synchronization (see my previous posting of parameter set for details).  The ACNT/BCNT seem not decrement.  The reading of ACNT/BCNT is always initial setting values.

Quote

Hi Mukul,

I post my reply immediately and now I look at it is not there.  I will rewrite it.  E2E use to email me when I post.  Now it doesn't do it anymore.  

1) What is the synchronization between your ISR and processing code?  [SQ]: I use Semaphore_post().  

How do you know that when you snapshot the param DST address in processing code, it has not already link reloaded the ping buffer?  [SQ]: I look at this DST in ISR and then in processing code.  They are always different.  So I know it has been reloaded.  From ISR read to processing code read is about 17us.

The set/ESR write in ISR to link reload on completion is done in hardware? [SQ]: Yes.

Could it be that the transfer is faster compared to when you get to look at the param value?  [SQ]: No, EMIF16 receives 8192 bytes every 1333us and is spread out evenly (audio data).

If you think this could be happening, you could try bigger ACNT/BCNT as an experiment (assuming your EMIF16 interface allows it) to see if bigger transfers = more time to finish transfer make it harder to hit the condition of ping reloaded?  [SQ]:  It is hard to change.  EMIF16 receives 64 channels of 48KHz audio data.

Thank you.

Regards,

Steve

Thanks Steve, sorry for the delay in response, i was out last week.

I know Eric mentioned that it would be great if you could share something that is reproducible on TI platform for us to help debug/root cause.

However, with the description of your setup , not sure how easily you can accomplish that - can you confirm if there is a way for you to show this on a TI EVM with stripped down code?

How do you know that when you snapshot the param DST address in processing code, it has not already link reloaded the ping buffer?  [SQ]: I look at this DST in ISR and then in processing code.  They are always different.  So I know it has been reloaded.  From ISR read to processing code read is about 17us.

This is still not clear to me.  I am trying to see if there is a race condition somewhere where there is a possibility of stale data being read from PARAM status or some notion of transfer completion , when it is still not complete - because your setup in software is slightly different then typical use-case (and I dont see people use just the PARAM status to judge transfer completion - even though it is a savvy way to do it). 

Hi Mukul,

Thanks for the reply.

I might not have been clear of the sequence of reading the DST.

1)  Submit EDMA transfer with CHANNEL_SET.

2)  EDMA completes and generate an interrupt.

3)  In the ISR, I read DST, which is the buffer A, resetmit another EDMA transfer, and set a semaphore to processing task.

4)  In processing task, I read DST, which is the buffer B.

5)  Processing task assumes EDMA is working on buffer B.  Hence processing buffer A.  But often times, EDMA is actually writing to buffer A.  That is my problem.

Please let me know of any questions.

Thank you.

Regards,

Steve

Steve,

Not sure if you can add some debug trace into your ISR and processing code, I wanted to have a log of the timestamp the ISR is entered, followed by the DST read, then a timestamp of processing code is entered, then a read of DST. For the timestamp, your can use TSCL if the issue can be reproduced very often. Otherwise, you need to use TSCL and TSCH to avoid overflow.

My idea is:
unsigned int debug_buffer[big number], index = 0;
1) debug_buffer[index++] = TSCL; //when you entered ISR
2) debug_buffer[index++] = DST; // save your DST which just completed the transfer, reading from ISR, how soon you read the DST in ISR? I thought parameter reloaded already happened? Or how do you know this DST is before or after reload?
3) debug_buffer[index++] = TSCL; //when you entered processing code
4) debug_buffer[index++] = DST; // save your DST reading from processing code

Given the ISR is triggered periodically, we can get a large array of debug_buffer. Then we expect the both ISR and processing code entered at the regular interval, and DST in 2) toggle and DST in 4) toggle and DST in 2) is always different from DST in 4). Can you confirm this or you can catch up some anomaly in such debug trace.

Suppose you didn't see anything wrong in this trace (may be you did similar trace to confirm that DST toggled in ISR and processing code), how do you know that "Processing task assumes EDMA is working on buffer B. Hence processing buffer A. But often times, EDMA is actually writing to buffer A. "

Regards, Eric

Hi Eric,

I have done the test you are asking.

In the ISR, the DST read is A, (the ISR only takes 0.5 us).  ISR signals processing task through semaphore.  17us later, the processing task DST read is B.  It is very consistent.  My measurement of timing is very accurate using Oscilloscope.

Thank you.

Regards,  Steve

Steve,

Thanks! So can you confirm from you log, you saw something like:

ts1, A, ts2, B
ts3, B, ts4, A
ts5, A, ts6, B
ts7, B, ts8, A
....

In the processing work you read DST is B so you processed the A buffer. How you know that sometimes (do you have an idea how often?) EDMA is writing to buffer A actually? E.g., you take two snapshots of buffer A content and you found it changed? Or you read the DST again you found it is buffer A?

I knew you read the data from an EMIF interface, let say if you simply change the EDMA source to a different location, e.g. DDR3 or MSMC, do you still have this issue?

Regards, Eric