DM814x interrupts being held off

Hi Andrew,

Are you using EZSDK? If yes, which version?

Are you using DM8148 TI EVM or DM814x custom board?

BR
Pavel

Hi Pavel,

ti-syslink_2.21.00.03
ti-ipc_1.25.01.09
ti-xdctools_3.24.03.33
ti-cgt6x_7.2.2
ti-sysbios_6.34.03.19

All of the above match the versions of components listed in the validation section of this: http://software-dl.ti.com/dsps/dsps_public_sw/sdo_sb/targetcontent/syslink/2_21_00_03/exports/syslink_2_21_00_03/syslink_2_21_00_03_ReleaseNotes.html

Running on a custom DM814x board (based on EVM).

BR

Andrew

Andrew,

I will check this with syslink team. Note that Syslink is no longer supported.

BR
Pavel

Andrew,

You claim to observe the same issue as reported in the referenced thread. However, we don't think the two are related.

In this thread, you indicate the EDMA interrupt is "scheduled to fire every 750 microseconds". How is this done? Are you using a periodic clock function to trigger an interrupt, or is the EDMA using a chained transaction, or is there a peripheral driving the data flow? It might be helpful to understand this better.

Your post make it sound like the EDMA ISR is calling MessageQ_alloc() and MessageQ_free(). These are blocking calls and should not be called from interrupt context.

BR
Pavel

The "root" event that causes the EDMA interrupt is a McASP service. There a some chained EDMA transactions that occur as well, so the EDMA channel that causes the interrupt event is not the same one that is servicing the McASP.

EDMA ISR posts an event to a Mailbox that a task blocks on. EDMA ISR is only a few lines of code and does not call any blocking function.

How did you conclude that my report was/is unrelated to the parent post? You must have concluded that my used case does not match for some reason.

Thanks,

Andrew

Are the sequence diagrams for what occurs when MessageQ_alloc() and MessageQ_free() are called? I assume SYSLINK should only block for a short amount of time?

I've started working with UIA logging to try to figure out what is going on.
What are the following:
#Hwi.ti_sdo_ipc_family_ti81xx_InterruptDsp_intShmStub()
#Swi.ti_sdo_ipc_transports_TransportShm_swiFxn()
?
I'm seeing the above Hwi takes 18 microseconds to run and the Swi takes 30 microseconds to run. Is there a way to run IPC in polled mode from the DSP side? And/or, can the TransportShm be run as a task?

Thanks,
Andrew

Andrew Elder said:

The "root" event that causes the EDMA interrupt is a McASP service. There a some chained EDMA transactions that occur as well, so the EDMA channel that causes the interrupt event is not the same one that is servicing the McASP.

EDMA ISR posts an event to a Mailbox that a task blocks on. EDMA ISR is only a few lines of code and does not call any blocking function.

How did you conclude that my report was/is unrelated to the parent post? You must have concluded that my used case does not match for some reason.

So, it looks like the ISR is not held off but rather the DSP task which is blocked on the mailbox. The next step is to identify what the DSP task is doing which is taking longer than expected. Try instrumenting the task to identify the point of latency.

I believe the second forum thread was related to Linux side thread preemption while holding a GateMP instance on which the DSP was busy waiting. This caused the DSP task to spin until the original Linux thread could resume execution and ultimately release the gate which the DSP was busy waiting on.

 

Pavel Botev said:

I had understood the original thread to mean the EDMA ISR was being held off. But now that it seems the EDMA ISR runs promptly, but actually the DSP *task* is taking too long, then maybe the two threads are indeed related.

 

To your original question: "Why changing the MessageQ's gate protection will help". On Linux, when user-mode code calls GateMP_enter() it will ultimately end up in the kernel calling Gate_enterSystem(). The default behavior is to acquire a mutex object to serialize access to critical code. Once inside the critical section, the thread acquires a hardware spinlock (by busy waiting on it). After acquiring the spinlock, the thread may be preempted by another thread (doing something completely different, maybe even in another process). Eventually, the original thread will resume again and finally leave the critical section. During this entire period, if the DSP is busy-waiting on the same spinlock, it could wait for a very long time.

My requirements are to have one task (on the DSP) that has a deadline to start and a deadline to finish. Other different tasks do not have any deadlines and communicate with ARM using SYSLINK. The task with the critical deadline is signaled to run from a HWI.

How can I structure my code to guararntee to meet the required deadlines? It seems it is not possible. I'm seeing average utilization of 80%, but sometimes the deadline is missed.

If I do the following thought experiment, I conclude that ARM side Linux operations cause indeterminate DSP task operation.

- critical task is signaled to run on DSP

- ARM sends SYSLINK message

- DSP is interrupted Hwi.ti_sdo_ipc_family_ti81xx_InterruptDsp_intShmStub() is called

- ARM calls GateMP_enter() and acquires hardware spinlock

- ARM kernel thread is preempted. ARM linux kernel threads are off doing "stuff"

- DSP SWI ti_sdo_ipc_transports_TransportShm_swiFxn() runs and calls GateMP() enter. SWI "spins" waiting for GateMP_enter() to complete.

- At this point the DSP's critical task is delayed for an unbounded amount of time by the Linux kernel. Linux kernel needs to release hardware spinlock and DSP's SWI needs to acquire it, then release it and complete the SWI before the critical task can be scheduled to run again.

Am I missing something in the above discussion? Is my best/only alternative to convert my critical task to run as a hardware interrupt?

Thanks,

Andrew

Any comments from TI?

Andrew Elder said:

My requirements are to have one task (on the DSP) that has a deadline to start and a deadline to finish. Other different tasks do not have any deadlines and communicate with ARM using SYSLINK. The task with the critical deadline is signaled to run from a HWI.

How can I structure my code to guararntee to meet the required deadlines? It seems it is not possible. I'm seeing average utilization of 80%, but sometimes the deadline is missed.

If I do the following thought experiment, I conclude that ARM side Linux operations cause indeterminate DSP task operation.

- critical task is signaled to run on DSP

- ARM sends SYSLINK message

- DSP is interrupted Hwi.ti_sdo_ipc_family_ti81xx_InterruptDsp_intShmStub() is called

- ARM calls GateMP_enter() and acquires hardware spinlock

- ARM kernel thread is preempted. ARM linux kernel threads are off doing "stuff"

- DSP SWI ti_sdo_ipc_transports_TransportShm_swiFxn() runs and calls GateMP() enter. SWI "spins" waiting for GateMP_enter() to complete.

- At this point the DSP's critical task is delayed for an unbounded amount of time by the Linux kernel. Linux kernel needs to release hardware spinlock and DSP's SWI needs to acquire it, then release it and complete the SWI before the critical task can be scheduled to run again.

Am I missing something in the above discussion? Is my best/only alternative to convert my critical task to run as a hardware interrupt?

Thank you for the detailed response Pavel. I can in fact convert my task to be completely non-blocking and I have done that in fact, so I don't think I need further help.

Were the issues I describe detailed in the SYSLINK documentation somewhere, because if they were, I think I missed it? I think the SYSLINK team may have dropped the ball on this one.

I assume TI's newer IPC framework is written so the situation I describe cannot occur with default build options?

Thanks,

Andrew

IPC 3.x should also include a single writer and single reader transport example that used a circular buffer and notify event and no GateMP calls. That way this issue goes away.