CMSIS throwing NANs when interrupted

Benjamin Fitzpatrick said:

every so often the arm_rms_f32 function throws QNAN

How delightful - SW's version of past, always dreaded, hardware "intermittent!"

Truth in advertising - have only scanned CMSIS - rely instead upon (one hopes) logic, past related experience - and assorted witchcraft...

a) Can you improve upon, "every so often?"  Substantial clue emerges should some pattern be recognized - when massaged/coaxed or teased out. 

b) Might the last value - or past several values - have caused a computational irregularity - thus triggering QNAN?

c) Might elevating the priority of your, "arm_fir_f32" interrupt reduce and/or eliminate this issue? 

d) Might the rate of your call to, "arm_rms_f32()" influence the QNAN occurrence? 

e) Might a "safer" time window exist for your call to, "arm_rms_f32()?"  (i.e. when most "likely" troublesome suspects have just completed)

f) Might some collision or conflict be occurring during the pack and/or unpack of your (assumed) data buffer?

g) Use of IntMasterDisable - bit like 105mm Howitzer - when small Colt may suffice!  By selective disabling of say "half" of your potential interrupts - might you be able to narrow the, "offender?"

h) While not "satisfying" - perhaps upon the detection of, "QNAN" - you could generate a, "re-calculation" - while discarding the QNAN result.

We were taught that missile may fly when paperwork pile exceeds missile's height.  Perhaps something here warrants inclusion - that pile...

btw - thanks your kind mention - always satisfying when non-trivial solution can be, "coaxed out..."

Kindly note - have added to/rearranged above suggestion list.  (09:45 CST - 23 Jun '13.)

@cb1-

One more point to add to your previous list :

i) what is amount of stack provided to this application? calling a floating point routine in interrupt means some f.p registers are saved on stack - and the function called has one more nested level, also floating point. Maximum registers saved on stack is 32 registers 32 bits each. Of coarse not all will be moved on stack, but must be taken into account.

Petrei

@Petrei-

Indeed.  And your suggestion far faster - easier to implement - and test - than this reporter's.  (laundry list)

That said/acknowledged - review & some/better compliance w/key issues listed - may lead to improved, more robust program execution...  (and - critically - points the way toward solution - which expanded stack does not (should it fail...))

Thank you for your advice, both of you.

To clarify for cb1, we are pretty sure that the 'every so often' occurrence of getting a NaN is that it happens when we get interrupted in the middle of the routine. I have not conclusively proved this, but it fits with the IntMasterDisable making the problem go away.

I tried the strategy of disabling specific interrupts, but it was not successful. I disabled the interrupts doing floating point math first (this reduced the frequency, but not much), then added in the timer interrupts, then added in the generic processor interrupts (0-15). After that the problem happened very infrequently, but it did still happen. My suspicion is that we have additional interrupts running that I am not aware of, and that any interrupt doing floating point math or not can cause a NaN from the RMS function. I'm not sure how to get a list of all the interrupts that are enabled.

I have also tried increasing the stack size, but this didn't seem to have an effect - we still failed after several minutes of operation.

I tried that again with IntMasterDisable, and was able to run long-term without incident. No question that it is the Howitzer of solutions, but the Colts don't seem to be working right now...

Benjamin Fitzpatrick said:

suspicion is that we have additional interrupts running...of which I'm unaware

Would not each/every enabled interrupt be listed w/in, "Start-Up" file?  And - have a servicing interrupt - w/in your code?

Perhaps by creating a code test for "NaN" (w/in your existing code block) - and inserting a break-point therein - cause may better reveal...

I'm pretty sure that in CCS5 or at least for the LM4/TM4C line, they have removed the 'start-up' file. For instance, the interrupts 0-15 which seem to be processor interrupts we found in documentation - they do not appear in the app configuration or any other file that we've been able to find. 

As far as needing code to service the interrupts, all of the interrupts we have code for were disabled, that was the first thing I tried. Yet we still see behavior with all of those interrupts disabled that we do not see when IntMasterDisable is called.

I did create a test for NaN and put a breakpoint in it - this is how I can tell that IntMasterDisable is working, but that disabling the individual interrupts aren't. However, since the breakpoint seems to be after the NaN is generated, I can't tell what (if any) interrupt is causing it to return.

I'm not certain if I can instrument the code for the CMSIS library without breaking some of the optimizations they did (I've looked at it and it's pretty arcane), but I suppose that for figuring out where the NaN originates, this is probably the best option. I'll definitely give it a try as soon as I can, but it probably won't be today. Thanks so much for helping cb1, I can't believe TI hasn't hired you to run this place yet...

Benjamin Fitzpatrick said:

I'm pretty sure that in CCS5 or at least for the LM4/TM4C line, they have removed the 'start-up' file. For instance, the interrupts 0-15 which seem to be processor interrupts we found in documentation - they do not appear in the app configuration or any other file that we've been able to find.

In a Cortex M4F there is a difference between Vector number (offset into the vector table) and interrupt number (bit in VNIC interrupt vector registers)

Vector numbers 0 to 15, which are Processor Exceptions, don't have entries in the NVIC interrupt registers

Vector number 16 is interrupt number 0 (bit in NVIC interrupt vector register)

Vector number 17 is interrupt number 1

And so on...

Benjamin Fitzpatrick said:

I tried a solution that the estimable cb1_mobile threw out for another problem we were having and I now call the functions FPUEnable() and FPULazyStackingEnable() at the start of the code, before I enable Sys/BIOS.

1) At processor reset the ASPEN (Automatic State Preservation Enable) and LSPEN (Lazy State Preservation Enable) bits are set in the Floating-Point Context Control (FPCC) register. This is line with the documented reset values given in the LM4F120H5QR datasheet.

2) The ASPEN and LSPEN bits are still set when main is called. FPULazyStackingEnable sets the ASPEN and LSPEN bits, so as they are already set at processor reset adding a call to FPULazyStackingEnable won't change the behaviour.

3) When one of my SYS/BIOS tasks stopped at a breakpoint, i.e. after BIOS_start had been called, the ASPEN and LSPEN bits were shown as cleared. This means automatic floating point state presevation has been disabled, which may explain the problems with CMIS throwing NANs when interrupted.

I looked at the SYS/BIOS properties exposed in the CCS GUI and couldn't see any obvious settings which control floating point state presevation so will have to read the manuals / study the source code....

Chester Gillon said:

When one of my SYS/BIOS tasks stopped at a breakpoint, i.e. after BIOS_start had been called, the ASPEN and LSPEN bits were shown as cleared. This means automatic floating point state presevation has been disabled, which may explain the problems with CMIS throwing NANs when interrupted.

I will add some task floating-point operations to the SYS/BIOS task to try and check for floating point corruption.

Again - great effort/detail/awareness by Monsieur Gillon. 

That said - poster reports, "every so often the arm_rms_f32 function throws QNAN!"

That "every so often" restriction - at least to this reporter - appears to confound (or be outside) the 2 critical bit discovery you have unveiled...

Poster holds steady as regards, "IntMasterDisable" preventing this unwanted QNAN.  Perhaps the critical overlap of that fact - w/in your new findings - may prove fruitful...

Unfortunate that "officialdom" has thus far avoided this "unfruited" plain...

They are supposed to release CMSIS for Cortex-M4 in this thread:

http://e2e.ti.com/support/microcontrollers/tiva_arm/f/908/p/261081/960090.aspx

Regards

PAk SY said:

They are supposed to release CMSIS for Cortex-M4 in this thread:

http://e2e.ti.com/support/microcontrollers/tiva_arm/f/908/p/261081/960090.aspx

a) The CMSIS source code v3.01 downloaded from ARM. In this version the ARM provided core_cm3.h, core_cm4.h, core_cm4_simd.h, core_cmFunc.h and core_cmInst.h files already have #if statements to detect use of the TI CCS compiler by checking for the predefined symbol __TMS470__

b) The TI provided cmsis_ccs.h installed from sw01291

c) In the CCS project properties added the predefined symbol ARM_MATH_CM4 to specify the build the CMSIS library for a Cortex M4.

The only issue during compilation was that two compiler specific intrinsics required by some of the ARM supplied source files were missing from the TI supplied cmsis_ccs.h. See my post in the Problem compiling the CMSIS library in CCS thread for the suggested additions to cmsis_ccs.h

Edit: I also ran the CMSIS arm_fft_bin_example and arm_fir_example examples on a LM4F120H5QR and the tests passed - the examples compare the results calculated by the CMSIS functions against an expected result.

Benjamin Fitzpatrick said:

Anyone know if my problem is that there are FPU instructions being used in both regular code and the interrupt handler, that my FPU-using function (arm_rms_f32) is being interrupted, or something else?

I noticed the following in Bios_6_35_01_29_release_notes.html :

Defects Fixed in SYS/BIOS 6.35.01.29 GA (Fixes since SYS/BIOS 6.35.00.20):
ID Headline
SDOCM00099886 Interrupt dispatcher in M4F targets does not restore FPSCR properly

Chester Gillon said:

I noticed the following in Bios_6_35_01_29_release_notes.html :

Defects Fixed in SYS/BIOS 6.35.01.29 GA (Fixes since SYS/BIOS 6.35.00.20):
ID Headline
SDOCM00099886 Interrupt dispatcher in M4F targets does not restore FPSCR properly

[/quote]Comparing bios_6_34_02_18\packages\ti\sysbios\family\arm\m3\Hwi_asm.sv7M (from a CCS 5.3 installation) and bios_6_35_01_29\packages\ti\sysbios\family\arm\m3\Hwi_asm.sv7M (from a CCS 5.4 installation) shows a difference in:

- How _ti_sysbios_family_arm_m3_Hwi_dispatch__I pops fpscr

- How _ti_sysbios_family_arm_m3_Hwi_pendSV__I pops fpscr

Using CCS 5.4.0.00091 and SYS/BIOS 6.35.01.29 created a project for a LM4F120H5QR which:

a) Had a "low priority" task which initially called the arm_fir_example_f32 and arm_fft_bin_example_f32 CMSIS examples a number of times.

b) The "low priority" task then changed to continuously called arm_fir_example_f32, while a "higher priority" task called arm_fft_bin_example_f32 when activated by a sempahore given from a SYS/BIOS counter function.

The idea was to call CMSIS examples, which check results from CMSIS functions against expected results, from two SYS/BIOS tasks to see if any errors were detected. Note that the test DIDN'T call the TivaWare FPUEnable or FPULazyStackingEnable functions, so that all the floating point support was handled by SYS/BIOS.

When compiled using the SYS/BIOS 6.35.01.29 from CCS 5.4 this test ran for 20 minutes without detecting any errors.

The Hwi_asm.sv7M used was then changed to the version from SYS/BIOS 6.34.02.18, i.e. without the fix for SDOCM00099886. This then caused arm_fir_example_f32 to detect an error on the 2nd iteration. Therefore, the fix made for SDOCM00099886 in SYS/BIOS 6.35.01.29  does appear to fix an error where floating point operations can be affected by interrupts.

Poster repeatedly reports, "every so often the arm_rms_f32 function throws QNAN!"

Diagnosis - minus proper regard of symptoms/case history - concerns... 

cb1_mobile said:

Poster repeatedly reports, "every so often the arm_rms_f32 function throws QNAN!"

Diagnosis - minus proper regard of symptoms/case history - concerns...

I agree that since I don't have the actual source code any diagnosis isn't necessarily accurate. However, my line of thinking was:

1) The reported problem was that if interrupts are enabled then every so often the arm_rms_f32 function throws QNAN.

2) In the original system I don't know how often the arm_rms_f32 function is called, or how often interrupts occur. The relative rate of the calls to arm_rms_f32 .vs. rate of interrupt may lead to a failure "every so often".

3) Lacking details for 2, a SYS/BIOS test was created which had two tasks perfoming CMSIS calculations and checking the results, with regular clock interrupts enabled. The theory was that if there was a SYS/BIOS problem with the floating point context not being saved correctly across task switches or interrupts then the test would highlight the problem.

With the latest SYS/BIOS 6.35.01.29 no test failures were observed.

4) The release notes for SYS/BIOS 6.35.01.29 reports that that version has corrected the defect "SDOCM00099886 Interrupt dispatcher in M4F targets does not restore FPSCR properly". If the Floating-Point Status Control (FPSCR) is not restored properly after an interrupt then an interrupted floating point operation could return incorrect results.

When the interrupt dispatcher in the previous SYS/BIOS 6.34.02.18 was used the CMSIS test then detected failures.

Therefore, my suggestion is for the poster to check which version of SYS/BIOS is in use:

a) If a version prior to SYS/BIOS 6.35.01.29 is in use, then try upgrading to SYS/BIOS 6.35.01.29 to see if the fix in the interrupt dispatcher makes the "every so often" failure go away without having to disable interrupts around the arm_rms_f32 function.

b) If SYS/BIOS 6.35.01.29 is in use then the problem is something else, and needs further investigation. 

As past stated here - your work is detailed and very often tightly focused.  And - I've memorialized & appreciated those facts.

It is the "every so often" (loose end) which does concern.  Unfortunate that op has gone silent - especially in light of your constructive efforts.

Having some past, professional success wrt remote, tech diagnosis - I reacted to and challenged poster's, "every so often."  And he responded w/out adding requested specificity.  That said - have been at this long enough to suspect that, "every so often" usually ranges between 1 event out of 20, or 50 or even far higher.  (again - that's past experience speaking)

Data you gathered suggests (to me) that the errant QNAN results would occur at a rate beyond, "every so often." 

As I stated in opening/answering post - intermittent problems - be they HW or SW - are often the most difficult.  (especially when presented w/imprecision!)

I am in high agreement that upgrade to newest SYS/BIOS appears to make sense - but remain reluctant to, "bet the farm" on its successful outcome...

And - myself/others would have found your analysis more compelling if some "interweave was attempted" - better linking the "QNAN randomness" to your findings.  There was none - and this silence/avoidance stood in contrast to the otherwise great detail & effort w/in your report - motivating my comment...

Hello everyone,

Apologies for 'going silent' the last week - I was on vacation, and while your analyses are amazing and thorough, the beach was more attractive at the time ;)

Now that I'm back on the case, I can report that I am using TI-RTOS 1.1.0.25, which seems to include Sys/BIOS 6.34.04.22 (this is what's in the 'products' folder inside the TI-RTOS folder). This seems to be significantly out of date with your versions, but after checking for an update in CCS I don't see one. I'll prod our TI reps and find out what the story is there.

As for the 'every so often' part of the problem, I'll try and shed some light, though unfortunately as it's intermittent I can't offer you too many hard numbers. Our application runs a 'main loop' which performs non-safety-critical tasks, and this loop runs as often as it can (it's just a while(true) loop). As interrupts, we process values from the ADCs and perform safety-critical calculations. Our RMS processing function, the one that's throwing QNAN, is part of our main loop but operates on a 1 millisecond delay timer, so it won't process RMS values any more often than once per millisecond. In practice, it's probably fairly close to every millisecond because our processor isn't very busy.

We have four interrupts, two of which perform floating point operations. One implements our timer code which we use to implement long-duration timers, a 32-bit timer isn't much good if you want to wait for hours or days, so once a millisecond we 'tick' several timers. Another interrupt just notifies us when a hardware GPIO goes high - this feeds into our state machine but does nothing else, it can happen at any time though. Once per millisecond we also process ADC values and do FIR filter calculations and floating point math on them, and then once every 800 microseconds we process other ADC values and do FIR calculations and floating point math on them. None of these interrupts are synchronized to each other - that is, it's possible for all four of them to fire at the same time just out of happenstance. It's also possible none of them will ever fire at the same time. In addition, I think there are BIOS interrupts which are hidden from me, possibly to implement the timers that fire the interrupts, or possibly ADCs.

When I have all the interrupts enabled, the QNAN seems to crop up in the code very quickly, I would say that it's rare for the code to run for more than a second or two between throwing a QNAN. Since all but the hardware GPIO interrupts are on a timer, they are happening fairly rapidly, but I can't tell you whether or not they are happening *during* the RMS processing function. As I disable interrupts, including interrupts 0-15 which I don't have code for and in theory don't 'use', the frequency of QNAN becomes less and less, to the point when I have disabled everything I can think of it takes more than a minute for a QNAN to occur. Or, if I disable the master interrupt, QNAN never occurs.

It is important to note that when I 'disable' interrupts I am only doing that for the duration of the RMS processing function and then immediately re-enabling them. Several folks I have discussed this with here thought I meant I was disabling them completely for the application, but that would be ludicrous since it would mean many important pieces of code would never run, and it would ruin the troubleshooting.

Thank you all for your amazing insights and the time you have put into this problem! I never expected so much to be done when I was out of the office or I would have posted something about my vacation. If there's anything else I can tell you about my interrupts occurring or their schedules, or if you have any ideas of how to see whether I am being interrupted or not during the RMS loop, I will gladly provide if I can. I have to say that Chester's solution of FPULazyStacking not being enabled seems to make sense, but I'd like to root-cause it as well.

Oh, and I did another test right before I left based on something cb1 said - I tried instrumenting the CMSIS library itself to see if I could discover where the QNAN was actually being generated. The code is an unrolled loop, which looks like this:

for each 'block':

    in = pointer++;

    sum += in * in;

   // above code x4 because the loop is 4x unrolled

With the above code, I instrumented both 'in' and 'sum' to see what the values were when the QNAN was generated. It occurred right in the middle of a block, which was not the first nor last block. The 'in' value was 0 (which should not be the case, there were no 0.0 values in the input buffer I handed the function), and the 'sum' value generated from multiplying 0*0 was QNAN. So, it seems something horrible happened instead of the right thing.

Welcome back.  Appreciated your instrumenting the code - attempting to coax/tease out some further clue...

That illegal "in" value of 0 - does this not suggest a faulty pointer?  (however - would expect this to more normally occur @ initial read - not w/in the middle)  Is it possible that someway/somehow - your input buffer was "unstable" during that data read?  Or that one/multiple of your, "Interrupts temporarily disabled" - "poisoned" your pointer?

Do the reads on each side (i.e. prior to and post) the QNAN event fetch the buffer content properly - or does the process "screech to a halt" - upon QNAN?

Far earlier - asked if the QNAN (upon detection) could be discarded - and your processing allowed to continue.  (if the pointer had been "spoiled" - seems doubtful that your operation could have "auto-recovered" - but we do not know...)

I've missed - or the "size" of each "block" - was not presented.  (or presented days ago...kindly provide)

Might the QNAN's "favoring" - "right in the middle of a block" - signal some overflow - or some regularity - which points toward a solution?

Suggest that you create a, "secure, fixed buffer" - which disallows any/all updates - and re-run your test.  (eliminates any, "buffer in transition" issues)

And of course - if newer/better RTOS release exists - do try...

Update: little experience w/"this" CMSIS or vendor restricted RTOS.  That said - might CMSIS and/or RTOS have "expected" - or been targeted toward - MCUs w/larger SRAM?  As current LX4F and rebrand thus far constrain SRAM to 32KB - might this 32KB be impacting?  (Certain, past LM3S devices far exceeded - your "mid-block" QNAN occurrence triggers this line of thought.)

Benjamin Fitzpatrick said:

I have to say that Chester's solution of FPULazyStacking not being enabled seems to make sense, but I'd like to root-cause it as well.

How are the interrupts configured in your SYS/BIOS:
- HWI or SWI?
- What is the interrupt priority?

[I found that on a Cortex M4, if a HWI interrupt is defined with a priority of less than disablePriority then the interrupt is a "Zero Latency" interrupt handler which is severely restricted in which SYS/BIOS functions can be called from the handler]

Haven't had a chance to try cb1's solutions yet, but it looks like I can answer Chester's questions fairly easily....

• All interrupts are HWI. The SWI module isn't enabled in our app.cfg.
• The interrupt priority for all interrupts appears to be the default, which seems to be -1. 

Follow on to yesterday's (10:24 CST) suggestions:

Upon the detection of, "in == 0," (which triggers QNAN) cannot you capture the pointer?  Would not this prove most beneficial?

Beyond one single pointer capture - might you capture those pointers bit prior to "middle of block" - which you report as "usual occurrence" of this issue?

Another method - seed a safe buffer (i.e. buffer nowhere else "known/accessed") w/ a regular pattern of valid data.   Review then of your converted data should reveal pointer's "correctness" - up to the QNAN.

Believe the facts/data you've harvested/presented "justify" bit more probing... 

Benjamin Fitzpatrick said:

but it looks like I can answer Chester's questions fairly easily....

• All interrupts are HWI. The SWI module isn't enabled in our app.cfg.
• The interrupt priority for all interrupts appears to be the default, which seems to be -1. 

I am unfamiliar with 'disablePriority', but I would also indicate that we don't call any Sys/BIOS functions in our interrupt handlers.

Benjamin Fitzpatrick said:

I wonder if these are also affected if we are in a "Zero Latency" situation, or whether stacking does not work properly there?

I can certainly change these interrupt priorities and see if it has any effect