task switch in DSP/BIOS on C6747 with C/C++

Fawad Nackvi

Hello,

I have an issue with context switch and thread priorities on my C6747 running DSP/BOIS. I noticed that when I added a new low-priority thread in C++ to my existing c code the thread does not get switched out for a higher priority thread. Any idea why this may be the case? I changed my c++ processing code from FFTs to a simple burn-cycles test code and I still see the same thing.

Using CGT 7.4.18

Thanks,

Fawad

over 9 years ago

RandyP over 9 years ago

TI__Guru* 84110 points

Fawad,

This is not completely clear to me.

Did you add a new low-priority thread dynamically in your C++ code?

What were you expecting for the new thread relative to a higher priority thread?

Can you show some code that is doing what you are describing, please?

Regards,
RandyP

Fawad Nackvi over 9 years ago in reply to RandyP

Prodigy 150 points

Randy,

The new c++ thread was created statically thru the tcf file. The main processing thread takes 50% CPU and runs every 2.6 msecs. It posts a semaphore every 43 msec. The new c++ thread waits for the semaphore and then runs for 8 msecs. In the time the low priority c++ thread is running I expect it to be switched out by the main processing thread to continue the audio processing. However I do not see that. All other threads in the system seem to behave fine and follow the thread priority rules. But the c++ seems to appear to work as the highest priority thread. The priorities are :

main processing thread : 14

c++ thread : 1

ROV also reports the correct thread priorities.

So only thing different in my system is the use of C++ code to implement the FFT. So hence I am assuming that's where the problem lies.

Thanks,

Fawad

RandyP over 9 years ago in reply to Fawad Nackvi

TI__Guru* 84110 points

Fawad,

[I will ask a Moderator to move this thread to the BIOS forum. We will continue the discussion there with BIOS experts available to jump in.]

By 'thread', do you mean task for all of these, or are some of them PRDs or SWIs?

The main processing thread posts a semaphore to unblock the new c++ thread. At that point, since the main processing thread is higher priority, it should continue running until it hits a blocking point.

What is the blocking point that will block the main processing thread?
Does the new c++ thread incorrectly start immediately when the semaphore is posted or does it wait until the main processing thread has reached its blocking point?

What trigger or event causes the main processing thread to unblock?

Regards,
RandyP

Fawad Nackvi over 9 years ago in reply to RandyP

Prodigy 150 points

Randy,

Yes I think I should have posted this in the BIOS forum. I am using thread to mean a task.

The main thread does run to completion (it posts the semaphore for the c++ processing thread at the start of it's processing). So this part is fine. The problem happens when the next time the main processing thread needs to run. It is woken up with its own lock via a SIO_reclaim

nmadus = SIO_reclaim(inStream, (Ptr *)&rcv, NULL);

But during the time the low priority thread is doing its processing the main processing thread is shut out until the low priority thread is finished.

Further testing has shown putting a TSK_sleep(1) in the low priority thread allows the main processing thread to start up correctly. As if a tight loop (or FFTs) in the low priority thread is stopping a context switch! Code is optimized with -O2

Thanks,

Fawad

Fawad Nackvi over 9 years ago in reply to Fawad Nackvi

Prodigy 150 points

Fawad Nackvi over 9 years ago in reply to Fawad Nackvi

Prodigy 150 points

So the top plot is the main processing routine and is active when the signal is high (so around 40% CPU usage). The bottom curve is the low priority thread that stops the main thread from running!

Fawad Nackvi over 9 years ago in reply to Fawad Nackvi

Prodigy 150 points

So with a single TSK_sleep (1) in the low priority thread I get the main thread posting correctly. You can also see the low priority thread time jitter slightly (take longer when the main thread has to run too). This is what I expected without having to TSK_sleep(). So why do I need the sleep?

RandyP over 9 years ago in reply to Fawad Nackvi

TI__Guru* 84110 points

Fawad,

You will need to show some code to help us figure out what is going on. Where is the TSK_sleep inserted, in a loop or once? Are interrupts disabled at all during the low priority task? Can other BIOS things happen like PRDs or such?

Regards,
RandyP

Fawad Nackvi over 9 years ago in reply to RandyP

Prodigy 150 points

Randy,

The code is rather complex and only runs on custom HW. But I could simplify it. That could take a few days. The TSK_sleep is in the low priority thread and the low priority thread looks something like this:

void FBR()
{
volatile Uint32 iii, delay;
int k;
for (;;) // loop forever
{
// wait for FFT buffer to be posted
SEM_pend(&semFBR, SYS_FOREVER);
gpioLedProcessing1On();

TSK_sleep(1);
delay =200000;
for ( iii = 0 ; iii < delay ; iii++ ){ };

gpioLedProcessing1Off();
}
}

I also see other high priority threads that are locked out while this low priority thread runs to completion.

Fawad

RandyP over 9 years ago in reply to Fawad Nackvi

TI__Guru* 84110 points

Fawad,

We do not need executable code, so please do not spend a couple days trying to make something. This pseudo version helps a lot, and we may need to dig deeper into the details. I assume the delay loop is replacing your actual code.

This definitely acts like interrupts are being turned off. Were you able to check on HWIs or PRDs?

From your description earlier, control should always pass from the main processing thread to FBR when the main processing thread reaches its SIO_reclaim() blocking point. That would mean that immediately after that, FBR does a TSK_sleep(1) and then does what it always does.

Will TSK_sleep(1) only get called that once, immediately after getting the semaphore?

If you move the TSK_sleep(1) call to before the LED-on call, how much does it push out the start of the FBR LED pulse from the end of the main processing thread's LED pulse?

As a random thing to try, change the FBR priority from 1 to 4, something still lower than all the rest. Or try changing it to 13, just 1 below the main processing thread.

Whatever causes the SIO_reclaim() to complete / unblock the main processing thread, that is not happening. And if the other higher priority tasks also are having their triggers being blocked, it is pretty indicative of HWIs being disabled.

You may need to show us some parts of your code to help figure this out, or go in there looking for something that might disable interrupts.

Can you save the assembly output (-k compiler option) for FBR with the burn-cycle test code and show that to us? If it uses a tight loop it might also be stopping interrupts.

It would be worthwhile to look at all of the config code in the tcf, although I do not know of anything there that could do this.

Regards,
RandyP

Ashish Kapania over 9 years ago in reply to RandyP

TI__Mastermind 21645 points

Hi Fawad,

One other thing to try would be to get the code back into the old state where the problem occurs and put a breakpoint between the GPIO toggle calls. When the core hits the breakpoint, check to see if interrupts are globally enabled (CSR.GIE bit) and if tasking is enabled in the ROV Task view.

Best,
Ashish

Fawad Nackvi over 9 years ago in reply to RandyP

Prodigy 150 points

Randy,

So i did some more testing. It looks like the issue is not interrupts being turned off in a tight loops. I compiled the FBR routine with no optimization, -g and

#pragma FUNC_INTERRUPT_THRESHOLD(FBR, 1)

Using any settings like -O2, -g produced the same result. i.e. lower priority thread running to completion. The FBR code header looks like this:

;*   SOFTWARE PIPELINE INFORMATION
;*...
;*
;*
;*      Searching for software pipeline schedule at ...
;*         ii = 15 Schedule found with 2 iterations in parallel
;*      Done
;*
;*      Loop is interruptible
;*
;*
;*      Minimum safe trip count       : 1

i have tried moving around the TSK priorities but that also does not make a difference.

I have another high priority UART thread that locks on a GIO_submit mutex and that also fails to run while this the FBR low priority thread is running.

Any other ideas I could try?

Thanks.

Ashish Kapania over 9 years ago in reply to Fawad Nackvi

TI__Mastermind 21645 points

Hi Randy,

You can try putting a breakpoint in the low priority thread. Once the breakpoint is hit, you can look at TSK view in ROV and determine whether the scheduler is looked as well as whether other higher priority tasks are in fact ready.

Best,
Ashish

Fawad Nackvi over 9 years ago in reply to Ashish Kapania

Prodigy 150 points

Randy and Ashish,

I put a breakpoint near the end of the low priority thread. The ROV showed the high priority thread was in a blocked state, waiting for a semaphore. This behavior is so bizarre that it makes no sense. But I have no idea why this is happening.

Thanks,

Fawad

Ashish Kapania over 9 years ago in reply to Fawad Nackvi

TI__Mastermind 21645 points

Who posts the semaphore for the higher priority thread ?

Did you check to see if tasking and interrupts were enabled ? You can check if tasking is enabled from ROV and check if interrupts are enabled by looking at CSR register's GIE bit.

Best,
Ashish

Fawad Nackvi over 9 years ago in reply to Ashish Kapania

Prodigy 150 points

Ashish,

Yep you are right. Stopping in multiple points in the low priority thread shows GIE is disabled. So this seems to be the problem. What is disabling it during the thread run and how do I enable interrupts.

The high priority thread is an audio i/o using a SIO call:

nmadus = SIO_reclaim(inStream, (Ptr *)&rcv, NULL);

Another high priority thread fails with UART in and uses A GIO call :
status = GIO_submit(hUart_IN,IOM_READ,buf,&lenRx,NULL);

Thanks,

Fawad

Ashish Kapania over 9 years ago in reply to Fawad Nackvi

TI__Mastermind 21645 points

Hi Fawad,

Its difficult to tell what code is disabling interrupts. There are 2 things you can try:

1) Step through low priority thread and see which section disables interrupts. This can effect timing so if a race condition is occurring then you will not be able to see it.
2) Add a breakpoint at the end of the low priority task, enable PC trace and run till breakpoint is hit. The PC trace obtained will show the execution flow and should help figure out what is causing interrupts to get disabled. On the C674x, I believe the unified breakpoint manager (UBM) can be used to enable PC tracing. I haven't tried getting a PC trace on a C674x before so am not sure how to do it. You may have to post on the device E2E forum to get help with enabling trace.

Wiki page on tracing:
processors.wiki.ti.com/.../DA8x_devices

Best,
Ashish

Fawad Nackvi over 9 years ago in reply to Ashish Kapania

Prodigy 150 points

Ashish,

There is no part of my low priority thread that disables interrupts. If I replace the main code in the low priority thread with a "waste cycles" routine I still have GIE=0 in the thread. If I put a HWI_enable() in my thread after after the sem_pend(...) the correct context switching happens. But I guess this is not the right way to fix the problem.

I will try and follow your instruction above to debug further. But if you can think of another reason why only one thread (TASK) is entered with interrupts disabled please let me know.

Thanks,

Fawad

RandyP over 9 years ago in reply to Fawad Nackvi

TI__Guru* 84110 points

Fawad,

Please try this:

Create a new task and a new semaphore and use a new file.
Set the priority the same as your current low priority thread, 1.
Comment out the semaphore post in the main thread and add a new line just like it that posts the new semaphore.
Put the "waste cycles" code in the new task.

Let's see how that runs as a test. There 'should' be no difference, but then the current low priority task 'should' leave interrupts alone.

Regards,
RandyP

Fawad Nackvi over 9 years ago in reply to Fawad Nackvi

Prodigy 150 points

Ashish,

There is one more piece of information I want to tell you. When I stop in my low priority thread of "waste cycles" I can see that GIE can be either 1 or 0 (equal probability). GIE is always 1 in all other tasks.

Fawad

Ashish Kapania over 9 years ago in reply to Fawad Nackvi

TI__Mastermind 21645 points

Hi Fawad,

After replacing the main code in the low priority thread with waste cycles, the only API your function calls is SEM_pend() or is there any other API call ? Why I ask is because an interrupt disable before SEM_pend() will presist even after SEM_pend() returns.

I waiting to see what you learn from the PC trace and the short experiment Randy suggested.

Best,
Ashish

Fawad Nackvi over 9 years ago in reply to Ashish Kapania

Prodigy 150 points

Ashish,

I am looking into the PC trace.

But the main thread (high priority) is an audio input output thread using the mcasp. I am using the mcasp driver from the PSP 1.30.01 for the C6747. There is a number of _disable_interrupts in that code. The mcasp is run in dma mode and so I have to link in the edma drivers. From some further testing I can see the issue lies with either the mcasp code or the edma library. The last part of the main thread does a SIO_issue. The SIO issue calls into the mcasp driver to submitchannel (data) to the DMA to output on the mcasp. As the SIO_issue is a asynchronous call I guess there is a context switch some time during the time the mcasp driver has to set up the dma. I can't prove this yet but a simple TSK_sleep) of 1 msec after SEM_pend in the low priority thread fixes the problem. As if the TSK_sleep gave enough time for the mcasp DMA to be programmed.

Fawad

Ashish Kapania over 9 years ago in reply to Fawad Nackvi

TI__Mastermind 21645 points

Hi Fawad,

I just checked with my colleagues who are familiar with DSP/BIOS scheduler and can confirm that if the higher priority main thread disables interrupts and then switches to a lower priority task, interrupts will stay disabled when the low priority task starts running. In other words, the interrupt status (enabled or disabled) is maintained across a task switch. So, if you can prove the mcasp library is disabling interrupts, then you have found the reason why low priority thread is not getting pre-empted.

Best,
Ashish

Processors

Processors forum

task switch in DSP/BIOS on C6747 with C/C++