When all tasks go to sleep, code jumps (crashes?) to FXN_F_selfLoop

Slight update...hardware interrupts stopped working too. The flags are still being set in IFR, and IER is still set properly, but the interrupt routine is never called. I've done a temporary workaround...instead of other tasks waking this task up, I set it to poll some variables that the other tasks (started by interrupts) use to wake up this thread. It works, but isn't ideal and I would like to properly fix this.

It probably will not matter, but which specific C2000 device are you using?

Every application is different and how priorities are handled is different. Even though I do not fully understand how you are starting and managing your tasks, you do not seem to be doing anything that should cause the bad behavior that you get.

The BIOS scheduler should always be running, and it should be able to handle the absence of any active tasks. In fact, it is perfectly valid to build a BIOS project that only uses HWIs or only HWIs and SWIs.

To help us figure out a fix for your case, it would be helpful to understand a little more about your intended program flow. To start, the interaction between TSK/SWI/HWI and any IDL tasks might be useful.

I'm using the F2808. The project itself involves getting some F2808's connecting via SPI to do some parallel processing. I'm currently implementing a subset of MPI, which is when I noticed the behavior.

I have a total of 9 tasks (excluding idle) that basically all try and sleep whenever they aren't doing anything in order to increase performance (I found time slicing is very expensive). The tasks are assigned to two different priority levels, and time slice with other threads of the same priority.

FollowUpMonitor - this task basically provides PRD like functionality while providing a higher level management functionality. This task runs off of a PRD (the PRD itself raises this tasks priority from -1).

ProcessInboundMessages - this task processes any inbound messages and provides inbound buffer capabilities. This task is woken anytime a message arrives on the SPI (interrupt driven)

ProcessOutboundMessagesonPortx - this task processes any outbound messages on port x and provides outbound buffer capabilities (there are four total for the four SPI ports) and is woken up by any other task that needs to transmit something on the given port

MPISendService - in charge of coordinating MPI transmissions. This is woken up as a result of an MPI_Send() or MPI_ISend() call

MPIReceiveService - in charge of coordinating MPI receives. This is woken up whenever a data transfer specific to MPI is received by the ProcessInboundMessages task

UserService - this is the MPI "program" and is woken up once the system has finished initializing itself. This is the task that I set to repeatedly yield instead of sleeping to keep the system stable. Ideally this task would sleep whenever a blocking MPI send/receive is called.

I only use the idle task to pump RTDX info (aka I don't do anything custom with it)

I don't have any interrupts enabled except for the four SPI receive interrupts (fyi, I'm using the SPI FIFOs, and the interrupt is the FIFO level interrupt).

I have three other PRDs: one periodically performs some house keeping (handles transmission timeouts, etc), another refreshes a seven segment display, and the last drives a heartbeat LED

BryanHughes said:

Slight update...hardware interrupts stopped working too. The flags are still being set in IFR, and IER is still set properly, but the interrupt routine is never called. I've done a temporary workaround...instead of other tasks waking this task up, I set it to poll some variables that the other tasks (started by interrupts) use to wake up this thread. It works, but isn't ideal and I would like to properly fix this.

Did your code clear GIE and not re-enabled it?

I double checked and no, it's not an issue with GIE. I just enable it during initialization and then don't touch it...I only mess with PIE enables, and I haven't updated the interrupt code in months, and this behavior didn't appear until a few days ago.

I'm looking into using semaphores, but it will take a while to vet the changes. I'll get back to you when I have more info.

I've switched everything over to semaphores: TSK_sleep() and TSK_setpri() calls have been replaced with SEM_pend() calls. My code got a nice speed boost, but it still crashes.

I have had a new problem crop up: kernel view crashes CCS. Not sure if it's related, but at this point in time who knows. I created a new thread over at http://e2e.ti.com/forums/t/11312.aspx

RandyP said:

• This is good news for TI because you are less likely to be worrying about a DSP/BIOS bug in the interrupt processing, which is what the "all tasks at priority -1" problem was pointing to.
• This is good news for you since you get a speed boost. And it is a good datapoint for me to remember when considering code optimization techniques.
• Did you do a search on "kernel object viewer" and "KOV" to see if there are any other reports on this? We had problems with KOV in CCS 2, but I thought it was working well in CCS 3. I have flagged your KOV posting to watch the progress there. I have not used it much so I have no useful opinion.
• Even though changing to semaphores was a time-consuming effort that did not solve your problem, it may make it easier to debug. And at least you got the speed boost.
• Since you are using a common system approach (i.e. one that I understand) with the SEM_pend/post calls, we are pretty much back at the beginning of trying to find why you are crashing.

RandyP said:

This implies to me that you searched the code for changes to GIE and found none. Even though Brad specifically asked about "your code" affecting GIE, he was most likely looking for what the value is for GIE when the crash occurs.

Do you have CCS access when it crashes so you can inspect registers? It is implied that you do have this access since you said that IFR and IER are still behaving. If IFR is set and IER is set and GIE is set, then you should immediately branch to an interrupt routine.

RandyP said:

What did change in your code "a few days ago"? That is usually where the problem is.

Can you add a slow PRD function that you can use to detect a heartbeat? When your code crashes, you could put a breakpoint inside the PRD function to see if you can get to it, and that will tell a little more about what is going on.

You could also set a breakpoint on FXN_F_selfLoop then look at the Call Stack to find out how it got there.

Sorry, INTM is the correct bit.  It's the same purpose though opposite polarity of what we call "GIE" in some of our other devices.  When an interrupt occurs INTM will automatically be set to 1 in the hardware (so that interrupts won't nest automatically) as you execute your ISR.  If you are executing code in your ISR that goes awry then perhaps you would end up in this state.  ?  Is your convolution code that you changed executed from an ISR?

Are you making any BIOS function calls from your ISR, e.g. SEM_post?  If so, you must use "the dispatcher".

My interrupt code is pretty brief. I do call SEM_post() and SEM_count() and I am using the dispatcher (e.g. bios.HWI.instance("HWI_INT6").useDispatcher = 1;). Below is my interrupt code. If it would be helpful, I can attach my project code, although it is kinda large (~7500 lines).

void PortARXInterrupt()
{
    Uint16 i, flit[NUM_CHARACTERS_IN_FLIT];

    // Buffer the flit
    for(i = 0; i < NUM_CHARACTERS_IN_FLIT; i++)
        flit[i] = (spiaRegisters.SPIRXBUF & 0x0FFF);

    if(flit[0] < 0x0F00)
    {
        // Add the port to the left-most 4 bits of the first flit characer
        flit[0] += PORTA << 12;

        // Enque the flit if there is room and wake up the processing thread
        Uint16 i;

        // If there is room, que the flit
        if(!globals.processing.inboundFlitQueFull)
        {
            // Copy the flit to the que
            for(i = 0; i < NUM_CHARACTERS_IN_FLIT; i++)
                globals.processing.inboundFlitQue[globals.processing.inboundFlitQueHead][i] = flit[i];
           
            // Check if the buffer is full
            if(globals.processing.inboundFlitQueHead == globals.processing.inboundFlitQueTail - 1 ||
                (globals.processing.inboundFlitQueTail == 0 && globals.processing.inboundFlitQueHead == INBOUND_FLIT_BUFFER_SIZE - 1))
            {
                // Assert all MARB pins to prevent anything else from being transmitted
                gpioCtrlRegisters.GPADIR.bit.MARBA = 1;
                gpioDataRegisters.GPADAT.bit.MARBA = 1;
                gpioCtrlRegisters.GPADIR.bit.MARBB = 1;
                gpioDataRegisters.GPADAT.bit.MARBB = 1;
                gpioCtrlRegisters.GPADIR.bit.MARBC = 1;
                gpioDataRegisters.GPADAT.bit.MARBC = 1;
                gpioCtrlRegisters.GPADIR.bit.MARBD = 1;
                gpioDataRegisters.GPADAT.bit.MARBD = 1;

                // Set the inbout flit que full flag
                globals.processing.inboundFlitQueFull = true;

                // Update the statistics
                globals.statistics.inboundQueFullCount++;

                // Raise an error
                LogWarning(__FILE__,__LINE__);
                return ERROR;
            }

            // Inrement the index
            globals.processing.inboundFlitQueHead++;
            if(globals.processing.inboundFlitQueHead == INBOUND_FLIT_BUFFER_SIZE)
                globals.processing.inboundFlitQueHead = 0;

            // Update the statistics
            globals.statistics.currentInboundQueLoad++;
            if(globals.statistics.currentInboundQueLoad > globals.statistics.maxInboundQueLoad)
                globals.statistics.maxInboundQueLoad = globals.statistics.currentInboundQueLoad;

            // Wake up the processing thread
            if(SEM_count(&ProcessInboundFlitsSem) == 0)
                SEM_post(&ProcessInboundFlitsSem);

            return SUCCESS;
        }
        else
        {
            // Wake up the processing thread
            if(SEM_count(&ProcessInboundFlitsSem) == 0)
                SEM_post(&ProcessInboundFlitsSem);

            // Set the error
            LogWarning(__FILE__,__LINE__);
            return ERROR;
        }
    }

    // Reset the SPI to prevent things from getting shifted around
    spidRegisters.SPICCR.bit.SPISWRESET = 0;
    spidRegisters.SPICCR.bit.SPISWRESET = 1;

    // Acknowledge the interrupt and re-enable it
    pieRegisters.PIEACK.bit.ACK6 = 1;
    spiaRegisters.SPIFFRX.bit.RXFFINTCLR = 1;
}

void LogError(char* file, Uint16 line)
{
    // Update the statistics
    globals.statistics.numErrors++;
       
#if PRINT_ERRORS == true
    LOG_printf(&LOG_error,"Error: %s at line %d.",file,line);
#endif
}

Note: PRINTER_ERRORS is currently false and NUM_CHARACTERS_IN_FLIT is 13, so the for loops are pretty short.