TMS320F280039: repetitive and permanent hardware fault on PIE handler diagnostics

Hi Luca,

I am looping in the expert for our diagnostics library to take a look at your issue. Please expect a response from them in the next 1-2 days.

Best Regards,

Delaney

So you're referring to the STL_PIE_RAM_testHandler() test failing? Does it fail for one specific value of the interruptNumber parameter or multiple?

If you trigger the interrupt manually (setting the PIEIFR bit) outside of the STL_PIE_RAM_testHandler() (which runs with a simulated redundant vector table mismatch and triggers an error handler rather than the intended ISR) does it execute?

Does totally erasing the flash before loading your application fix the issue or is only the sequence you describe above that works so far?

Whitney

Hi Whitney,

I tried the handler testing with a few of the EPWM units' interrupts, and it always gave the same results.

Note that the test/result of STL_PIE_RAM_testHandler() on the PFIEFR extends to any other interrupt I use: none of them is vectored anymore. In some of the initial attempts, I commented out the test functions to allow my code to run. However, no other interrupt was serviced anymore (SPI, ADC, EPWM).

Erasing the Flash did the trick, too. I didn't realize it for a long while, as my code was initially loaded into RAM by the debugger probe. Later on, I focused on programming the code in Flash (still with almost all of it copied to and run from RAM).

Note that even running from RAM, I could have programmed some part of the Flash once, to see if the board was able to run without the debugger.
After many tries, I can imagine that it kept repeating the following: power on, starting my older test code from Flash, debugging from RAM (repeated many times with new code), hours/days/weeks of this routine, without touching Flash anymore ..., at some point, the interrupts aren't serviced at all (power off/on, disconnetting the board from debugger, storing it for some days, nothing changes it) - at this point re-Flashing does fix it.

[EDIT]

Now, this seems unrelated, but I had a possible clue from running HWBIST testing, when my PIE also isn't vectoring interrupts as above.

In fact, in those cases, even the HWBIST fails. The MCU falls stuck in a BootROM wait point (in the 0x3FEEC9 – 0x3FEEF9 range, ITRAP ISR).  Reading "Application Note - C2000 Hardware Built-In Self-Test" doc. SPRACA7A, I found this final note:

If the CPU comes back, but it vectors into the BootROM or flash, one possible reason is that the PIE is not
enabled. HWBIST executes a CPU reset upon completion, but if PIE is not enabled, then the CPU vectors to
the BootROM instead of the Diagnostic Library STL_HWBIST_restoreContext() code.

On the PIE handler test, I checked if the PIECTRL.ENPIE is set before running STL_HWBIST_runMicro(), and it is '1' as expected.

May it be that I have another PIE setting preventing a proper PIE handler test?

Is it only impacting PIE interrupts? If you force a non-PIE interrupt (like timer 1 or 2 or even an NMI), does it work?

Things that would impact interrupt execution of all interrupts would be ENPIE as you pointed out or them getting masked in the CPU INTM bit.

Do all the various interrupt registers (PIEIFR, PIEIER, CPU IFR, CPU IER, etc...) look as expected when this condition occurs?

Whitney

I hadn't tried it before with non-PIE interrupts, but I see now that even those aren't served - I tried it with CPU TIMER2.

Registers always look to be fine: ST1.INTM = 0, IER.INT14 = 1, TIMER2 is running, IFR.INT14 is however 0

I modified my application so that it was limited to something like the following, where the timer functions are extracted from the self-test application example and modified for TIMER2

int main(void)
{
Device_init();
Device_initGPIO();

SysCtl_setWatchdogMode(SYSCTL_WD_MODE_INTERRUPT);

DINT;

Interrupt_initModule();
Interrupt_initVectorTable();

self_test_timer_config(1000u); // ******************************

STL_PIE_RAM_configHandler(&regular_pieVectError);

sys_init_LED0(); // GPIO line setup
sys_init_LED1(); // GPIO line setup

EINT;
ERTM;

while (!self_test_timer_has_timedout()) // ******************************
{
sys_set_LED0(1u); // GPIO, LED on
SysCtl_delay((DEVICE_SYSCLK_FREQ / 5000u) * 125u);
sys_set_LED0(0u); // GPIO, LED off
SysCtl_delay((DEVICE_SYSCLK_FREQ / 5000u) * 125u);
}

do {
sys_set_LED0(1u);
sys_set_LED1(0u);
} while (true); // ******************************

// ... the rest of the code is then ignored

...
}

volatile bool timer2_OutFlag;
volatile uint16_t timer2_OutCount;
volatile uint16_t timer2_isrTimeout;

#define TIMER2_RES (100u)
#define TIMER2_PERIOD (DEVICE_SYSCLK_FREQ / TIMER2_RES)

#pragma CODE_SECTION(timer2_ISR,".TI.ramfunc")
__interrupt void timer2_ISR(void)
{

if(timer2_isrTimeout++ >= timer2_OutCount)
{
timer2_isrTimeout = 0u;
timer2_OutFlag = true;
}
}

static void self_test_timer_config(uint16_t msTimeOut)
{
Interrupt_register(INT_TIMER2, &timer2_ISR);

timer2_OutCount = msTimeOut / TIMER2_RES;
timer2_OutFlag = false;
timer2_isrTimeout = 0u;

CPUTimer_setPeriod(CPUTIMER2_BASE, (TIMER2_PERIOD - 1));
CPUTimer_setPreScaler(CPUTIMER2_BASE, 0);
CPUTimer_stopTimer(CPUTIMER2_BASE);
CPUTimer_reloadTimerCounter(CPUTIMER2_BASE);
CPUTimer_setEmulationMode(CPUTIMER2_BASE,
CPUTIMER_EMULATIONMODE_STOPAFTERNEXTDECREMENT);
CPUTimer_enableInterrupt(CPUTIMER2_BASE);
CPUTimer_startTimer(CPUTIMER2_BASE);

Interrupt_enable(INT_TIMER2);
}

static bool self_test_timer_has_timedout(void)
{
return timer2_OutFlag;
}

Note that with the code above, I did these tests, in sequence (starting from a non-working interrupt condition):

• System with failing interrupts - Compiled it for Flash; the application loops forever waiting for the TIMER2 interrupt
• System with failing interrupts - Compiled it for RAM; the application loops forever waiting for the TIMER2 interrupt
• Whole Flash erase with UniFlash - Still running the compiled for RAM binary: the application serves TIMER2 interrupt, ending in the second and last loop
• Running again the compiled for Flash binary; the application serves TIMER2 interrupt

Given that the code is now oversimplified, there shall be a ridiculous error that I added somewhere, and I still can't see it.

What are your boot mode pins set to on your board? The fact that erasing the flash can fix the issue, makes me wonder if your device is booting to flash, executing some code that is getting your device into a weird state and then affecting the behavior of your application after you connect your debugger and load it in CCS.

Assuming they are set to Flash boot at the moment, if you change your them to wait boot and the problem goes away (you power cycle your board, it sits in the "wait" function instead of running whatever is in Flash), it would point to that theory, although we'd still have to narrow down what specifically is the previously programmed application that's causing the issue.

Whitney

I double-checked the boot pins, both on board and on schematics. GPIO24 and GPIO32 have a pull-up, and testing them from software in main() are at level `1`.
Note that on board GPIO24 is used as an output from the MCU driving a logic gate; in my test code, I left it as an input.

The application that can generate all the good and bad results is a single one. I tailored it, and I attach it here: beware, it's a Frankenstein, but it does its job.
From the original board, I used and configured just two signalling lines, GPIO54 and GPIO56. They both are connected to LEDs, with a `0` turning the LED on.f28003x_mini.zip

Here is what it can do, in a sequence:

1. Start with an erased Flash
2. Running it from RAM (CPU1_RAM configuration): everything works, the diagnostics are complete, the application ends in a blocking loop at the end of main()
3. Running it from Flash (CPU1_FLASH): the PIE handler diagnostic ends into BootROM wait point at 0x3FEEC9
4. Restart debug, run again CPU1_FLASH: interrupts aren't served anymore, the function main() is stuck in the loop waiting for TIMER2 interrupt to be served; IER.INT14 = 1, IFR.INT14 = 0
5. Power cycle: without debugging, I see LEDs stuck to ON, as when it runs into the wait point - I assume it's there again
6. Start again debugging of CPU1_RAM: interrupts aren't being served, resulting in the same outcome as point 4.

The only hypothesis I have right now is that this application, when running from Flash, can do something unrecoverable by restarting the application in RAM with the debugger.

----

One or two questions about the diagnostic library. In the function STL_PIE_RAM_testHandler() there's access to a  PIEIFRn register to force the interrupt handling: shouldn't it be fenced in an atomic access to prevent any race condition?
In the above function, is the interrupt mask used to set PIEIERn and PIEIFRn, zeroing any other IER/IFR bits?

That memcpy is for the HWBIST context restore functions, so after HWBIST runs the CPU is in an unknown state and so you run the context restore functions stored at address 0x0 to set it right again, so it might be possible there's something lingering from the unrestored CPU state. 

Usually when you're debugging, it's the GEL file doing the device initialization (unless you're doing something to force it to run through boot ROM). It's possible that the GEL file is missing something.

Whitney