RTOS/TMS320F28335: Stack Overflow in Kernel Idle Task

Freddy Gauss

Part Number: TMS320F28335
Other Parts Discussed in Thread: SYSBIOS

Tool/software: TI-RTOS

Here's a new type of bug that I haven't encountered before and I'm having some trouble debugging it. I've got a small project, currently it has a main Task which adds items to a queue when the queue is empty and an HWI which pops items off the queue until it is empty. After a short while I get a stack overflow in the kernel idle task (ti.sysbios.knl.Task.IdleTask). Unfortunately that's not a task that I defined and have no idea how to debug it. As I understand it, if no idle task is defined the kernel has its own idle task. If I disable the queue operations the stack overflow disappears.

One odd observation is that the stack size doesn't creep up slowly in the idle task, it suddenly overflows.

Idle Task "ti.sysbios.knl.Task.IdleTask"

stack base at 0x8000, normal steady state (peak) usage is 142 words. After a while this one overflows into the stack space of "My Task"

My Task

stack base at 0x8100, normal steady state (peak) usage is 120words.

A simplified version of my code is below:

typedef struct QUEUE_REC {
    Queue_Elem elem;
    Uint16 data;
} queue_rec_t;

queue_rec_t queue_rec_buf[16];
queue_rec_t *rec_tmp_ptr;

void epwm_1_hwi_48(void) {
    if(!Queue_empty(queue_hndl)) {
        rec_tmp_ptr = Queue_get(queue_hndl);
        temp_uint16 = rec_tmp_ptr->data
    }

//    ... do stuff with temp_uint16 ...

    EPwm1Regs.ETCLR.bit.INT = 1;
    return;
}


void main_task() {
    Task_sleep(100);

    for(;;) {
        queue_rec_buf[0].data = 0;
        Queue_put(queue_hndl, &(queue_rec_buf[0].elem));
        queue_rec_buf[1].data= 1;
        Queue_put(queue_hndl, &(queue_rec_buf[1].elem));
        queue_rec_buf[2].data = 2;
        Queue_put(queue_hndl, &(queue_rec_buf[2].elem));
        queue_rec_buf[3].data = 3;
        Queue_put(queue_hndl, &(queue_rec_buf[3].elem));

        while(!Queue_empty(queue_hndl)) {
            Task_sleep(100);
        }
    }
}

EDIT: I forgot to add version info, they're below

Compiler: v18.1.2.LTS
SYS/BIOS 6.73.1.01
XDCtools: 3.50.8.24_core

And an unrelated question, should I still be using SYS/BIOS or should I be transitioning to TI-RTOS (for C2000)?

over 6 years ago

0 ToddMullanix over 6 years ago

TI__Guru* 96960 points

Hi Freddy,

First, I'd recommend you stay with SYS/BIOS. The TI-RTOS for C2000 is really more for Concerto devices (C28 + M3). We have active development on SYS/BIOS in regards to C2000. We do not have active development on TI-RTOS for C2000.

I tried your code (+ a Queue_create) and it worked as expected. Note: I called Queue_get in the task after all the Queue_put calls.

How did you create the Queue?

How big is the Idle stack?

From your description it sounds like some corruption is going on. How are the queue_rec_bufs managed? For example, once the ISR gets one, what does it do with it? Typically I like to have a "free" Queue and a "full" Queue. The Task would get from the free queue, fill it in as needed and then place it on the full queue. The ISR would get from the full queue, process it, and then put it in the free queue. This avoids issues with over-running or under-running by any of the threads.

Todd

0 Freddy Gauss over 6 years ago in reply to ToddMullanix

Intellectual 300 points

Hi Todd,

Thanks for giving it a go!

The queue was created statically by adding the instance to app.cfg, the relevant section is below. The handle, which appears to be a 'Queue_Elem' type is allocated to reside in data memory at 0x8A62.

BIOS.runtimeCreatesEnabled = false;
var queue0Params = new Queue.Params();
queue0Params.instance.name = "queue_hndl";
Program.global.queue_hndl = Queue.create(queue0Params);

The idle stack appears to be 256 words, according to the ROV viewer, I'm not sure where it's defined.

Well, queue_rec_bufs aren't really managed. The way the code works (AFIK) is that the HWI is constantly going,. Every time it is called it checks to see if the queue is empty. If it is not empty, the HWI dequeues an element and operates on it. In the mean-time, the main task might be adding to it. I guess this queue might be be more of a FIFO operation.

Am I misusing the queue structure or its intended purpose? My application has a long sequence of data on which it has to operate. What I was hoping to do here is to have a small FIFO buffer that can be added to so I can avoid having to allocate enough memory for an entire data sequence. My intention was to have the HWI always just pop a new item off the front of the queue when available and a task to keep that queue full.

In hindsight your approach seems good, it may be worthwhile for me to switch to that structure.

I've been running this code for over an hour now without a stack overflow, which is odd. The last couple times I tried I got an overflow within 5 minutes :/

0 Freddy Gauss over 6 years ago in reply to Freddy Gauss

Intellectual 300 points

Some new information has come to light.

The Idle task that is defined by SYS / BIOS runs one function only, 'ti_sysbios_hal_Hwi_checkStack', which as the name implies just checks the HWI stack for overflows. It does this by checking that the filler data (0xBEBE) is still in the stack. The HWI stack lives at 0x0000 and eventually overflows, the Idle task overflow appears to be a side-effect of this first overflow.

Now, I'm not sure how to go about debugging the HWI stack overflow, there's a lot going on there. First, it does appear to be a genuine overflow as the stack appears to be fairly full under normal operation, but with what? It has a few symbols defined but I have no idea what, they are the following.

0x0000    __ISA__, _stack
0x0013    __TARG__
0x0037    __PLAT__
0x005D    __TRDR__
0x0084    __ASM__
0x0100    _STACK_END

Eventually the stack gets used up past _STACK_END, which is where I get the stack overflow error. I believe there is something not quite right in the HWI stack as there are sections of 0xBEBE markers interleaved with what looks like actual data. Under normal operation, should there be holes left in the stack?

What are these symbols and what code uses them? Is the solution simply to increase HWI stack space?

0 ToddMullanix over 6 years ago in reply to Freddy Gauss

TI__Guru* 96960 points

I would bump up the system stack size. In the .cfg, set Program.stack.

Program.stack = 1024;

In case you have not seen this, it might be helpful: training.ti.com/debugging-common-application-issues-ti-rtos

Todd

0 Freddy Gauss over 6 years ago in reply to ToddMullanix

Intellectual 300 points

Hi Todd,

It appear that my problem was simply a standard stack overflow. There were no bugs in the code (as it seems), the HWI stack was simply too small and would occasionally peak too high. Increasing it from 256 to 512 did the trick. The link you sent was very helpful as well, thank you!

Do you know what the above mentioned symbol definitions are (__ISA__, __TARG__, etc.)? It's not a problem, really, but I would like to know what's using that memory if possible. I wasn't able to find anything about them in the compiler documentation. Are they defined by SYS / BIOS or some other libraries I'm using?

0 ToddMullanix over 6 years ago in reply to Freddy Gauss

TI__Guru* 96960 points

Hi Freddy,

I'm glad you resolved the issue. The defines are not from SYS/BIOS. You might want to open a new thread in the compiler forum and ask them.

Todd

C2000™︎ microcontrollers

C2000 microcontrollers forum

RTOS/TMS320F28335: Stack Overflow in Kernel Idle Task