SPI Communications Fails At 8 hours Of Operation And Thereafter

Timothy Ball said:

fails at almost exactly 8 hours of operation.

Might our diagnosis improve if:

a) you clarify if this unit (or units) have "ever" operated properly beyond this 28K seconds "hiccup"

b) is yours a, "single board anomaly?"   If not - what percentage of units tested exhibit this failure?

c) how do you determine the "guilty party" as "two" MCUs are involved?

d) at/around that 28K second mark - does the program execute any unique operations - which may cause this misfortune?   (on either MCU side)

e) we assume that system is fully/properly powered - that "battery discharge" caused "brown-out" is not occurring.

f) while you report that SPI commo breaks - have all (or most) other segments of the program continued as normal?   How have you (or have you) tested?

g) always delightful when execution is described as, "flawless."   Do you really know?   Famed auto-maker thought they knew - had extensively tested - yet their "flawless" claim was voided during live, courtroom demonstration.   And they were found, "Guilty" and paid a mighty sum! 

Proving that software performs flawlessly - under each/every condition (some most unlikely) presents a real challenge (beyond most all small firms) -  you may wish to withdraw or retreat from such claim...   (burden is very much upon you when you so claim - credit UCLA law for this insight...)

Suspect this fact finding may lead to "more & properly focused info" - always helpful in such a diagnosis...

[edit] 09:12 CST)  I composed prior to seeing your 09:05 post - reading/reviewing now...

Thank you - quick & reasonably complete response - that's both helpful & appreciated.

I agree with your striving - hoped to alert you to the "danger" of "over-claiming/stating."

Google should reveal the live, courtroom demonstration which proved that (inadequate software design) caused, "Unintended Acceleration."   Expert witness was able to repeatedly create those conditions (which while rare) caused that U.A.   Jury was able to understand - that's not always an easy feat when "tech" is on trial.

To your case:

Your (b) suggests that this is still, "single board anomaly" (sba) - does it not?    My firm performs such diagnostics regularly - we (avoid) as much as possible, "sba."   (we've little interest in "rushed prototypes, failed soldering, or misplaced components" all too often exhibited by sba.)   (we're fortunate in being able to "pick/choose")

We need you to better define, "SPI Commo fails" - that can mean many things - can it not?

Can you examine the "SPI payload" at/around the failure?   Might the payload itself "break" the SPI link?   We are great believers/proponents of KISS - often it pays to reduce the data transfer "size/type" to the minimum - to eliminate this as causal nexus.

It would greatly aid your diagnosis if we could some way/how "shorten" this 28K second "trigger."   While you've locked in on this duration - might the number of SPI transactions - instead be suspect?   (granted - I'm reaching)

You claim that little unique/special happens just prior to SPI loss.   Our experience teaches that this - most always - is especially fertile ground for your analysis.   It may prove useful if you'd create very basic code (blink several Leds regularly - bypassing the bulk of your existing code) and repeating that 28K duration test.   This attempts to eliminate any/all "unexpected/unwanted" intrusions upon your SPI by "other" program operations.   Is this clear?

When issues occur w/such timed regularity - as you know - often some improperly managed/anticipated "roll-over" proves the culprit.   All counters, timers and similar must be reviewed for such sensitivity.

Breaking your SPI may indicate a change in system clock - you need to monitor the critical SPI select and clock signals - both prior to and after - SPI failure.

Perhaps this qualifies as a reasonable start?   

Timothy Ball said:

... appears that an interrupt edge, that tells the Tiva that the Hercules is ready for communications, is being missed.

Is this a HW generated interrupt?   If so - has the signal been monitored for "rail to rail swing" and proper rise/fall times - especially at/near the failure mark?

Don't believe either MISO or MOSI to be causative - data would be "scrambled" but commo should persist.

I remain concerned that the SPI clock may have changed - at either end.  Believe that (really) has to be properly monitored.

Assume that - as you suggest - the Tiva "misses" a Hercules "edge."    Should not the Tiva then - in the absence of such, "Request to Send" signal from the Hercules - generate a, "May I Send" command?   (mimics the far older RS232/485 "RTS/CTS" interlocked communication protocol)

Good software should anticipate the "loss of expected signals" - and exhibit some (are you there?) recovery mechanism...

I was wondering if the issue could be related to errata SPMZ850C TM4C129x SSI#03 which states:

SSI1 can Only be Used in Legacy Mode

Revision(s) Affected: 1 (all modules). 2 and 3 (only SSI1).

Description: Bi-, quad-, and advance-modes of operation do not function correctly on the specified

SSI module(s). As a result, any affected module can only be used for legacy operation.

Workaround(s): On revision 2 devices, use SSI0, SSI2, or SSI3 for bi-, quad-, and advance-mode

operation. Only use SSI1 for legacy-mode operation.

The SSI1 is being used in the default legacy-mode with:

SSIConfigSetExpClk(SSI1_BASE, system_clock_hz, SSI_FRF_MOTO_MODE_1, SSI_MODE_MASTER, 100000, 8);

.

This is the vendor's "special (often hidden) inside info" area.    Best that he/they respond...   (I've protested lack of, "Publication of "KNOWN" issues!)
 
Were you "unaware" of this errata?    And are you using the device in one of those "suspect/illegal" modes?   (pity if so)

Does it not seem (bit) strange that you'd run correctly for 28K seconds - and (only then) dreaded errata, "sinks its fangs?"

Single board anomalies prove the, "bane of our existence!"   Really - who cares?   Hard to "be efficient" when "everything" is suspect!

In a (rare) disagreement - I have difficulty (imagining/justifying) a "race" which occurs with such, "clockwork precision" and after "so long" an interval!   (unlikely)

CB1:

I will look up the video that you referred to. We own two Saturns that were recalled for the ignition shut off issue that was known about over which people were killed. I will never understand why such matters are ignored till it is too late. Nor do I understand why people are not prosecuted and imprisoned for falsifying records while cutting corners over which others are killed such as that documented at the link below:

http://tdbconsulting.org/LessonsLearned.htm#CuttingCorners

The SPI issue may possibly be a single board anomaly (sba) and I will be running with other boards too to test.

In an effort to shorten the 28K second trigger, the FreeRTOS function xTaskIncrementTick(); was also executed many times to dramatically speed up the RTOS time, but the error does not occur after the faked 28K time period.

The payload is always 32 8-bit words.

I agree 100% with KISS and #ifdef out other portions of code in an effort to isolate the issue. However, it is time consuming while making one change at a time, for which determining if the change was the issue takes 8 hours to surface.

Yes, the term “failure” can mean many things. The Hercules Technical Reference Manual includes the following graphic for MIBSPI communications:

The two graphics below show the signals on the oscilloscope after the 28K fail time. The signals (top to bottom) are identified below:

1. ATTN (Attention) (YELLOW) set by Hercules GPIO to indicate when Ready for a SPI transfer with Tiva

2. SPISCS (CYAN) controlled by Tiva master SPI

3. SPIENA (MAGENTA) controlled by Hercules slave MIBSPI

4. SPICLK (BLUE) controlled by Tiva master SPI

Note: MOSI and MISO are toggling as well.

The graphic below shows the first of the 32 bytes transfer:

The ATTN (Ready) signal is controlled by the Hercules, which will set the pin high when ready for a SPI transfer with Tiva and will clear the pin when not ready (i.e. busy with a transfer). However, before doing so, the Hercules must receive a Notification interrupt from it's MIBSPI when a transfer is finished. Therefore, it is possible that the Hercules is not responding because that Notification interrupt edge was not activated or detected. However, apparently, it may be possible that an earlier edge of the ATTN interrupt to the Tiva (from the Hercules) was not activated or detected or cleared which leads to the concern below as stated in the TivaWare Peripheral Driver Library about interrupts:

Note:

Because there is a write buffer in the Cortex-M processor, it may take several clock cycles

before the interrupt source is actually cleared. Therefore, it is recommended that the interrupt

source be cleared early in the interrupt handler (as opposed to the very last action) to avoid

returning from the interrupt handler before the interrupt source is actually cleared. Failure to

do so may result in the interrupt handler being immediately reentered (because the interrupt

controller still sees the interrupt source asserted).

Will be adding inline assembly NOPs.

Below is the HALCoGen created interrupt code for the Notification interrupt mibspi1LowLevelInterrupt() in which interrupt clearing is not performed after an interrupt (but evidently not required as needed for Tiva interrupts):

The graphic below shows the signals before failure with the Hercules responding after getting a transfer group finished Notification followed by ATTN deactivation and activation:

Yes, I am considering possible rollovers but it appears none are occurring (e.g. 28700 seconds = 28700000 RTOS ticks = 1B5ED60 Hex).

Thank you very much for your suggestions,

Tim

Timothy Ball said:

...possible that the Hercules is not responding because that Notification interrupt edge was not activated or detected. However, apparently, it may be possible that an earlier edge of the ATTN interrupt to the Tiva (from the Hercules) was not activated or detected or cleared

Scope/LA caps are quite clear - thanks for that.   Sentence(s) above are similar to mine - crack staff (here) often emit WTF?

The logic w/in your quote seems reasonable - still unexplained is the timed consistency of the "break" in SPI commo.

Cannot you "count" each/every of those critical "handshake" signals (perhaps via a separate, independent MCU - in no way held hostage/dependent by your existing code) and via that means determine, "if, when and which" critical signal, "went missing?"   Is it not probable that each legitimate handshake transaction reveals a "balance" of "Commands/Responses?"    I'm suggesting that you employ a HW connection to each every such signal - from both (existing) MCUs into a 3rd one - which must, "sit & count."   Review of that 3rd MCU - at time of failure - attempts to identify the culprit.   And that 3rd MCU may "halt" the process - immediately when the "out of balance" is detected!

Still advise that you, "reduce/bypass" most all existing, non SPI code - and "accept the 28K second interval" and determine if that impacts!   Your desire for "instant identification" of the fault cause seems "anti-KISS" - recall, "One small step for man..."

BP101 said:

Asynchronous serial data transfer clocking can become an issue always void in error free synchronous data exchange

Poster is using SPI - which is clocked, synchronous transfer - I believe.

And - to your point of different xtal caps @ each MCU proving causative - with SPI - only the Master (Tiva here) is generating that clock!

The suggestion of a thin, coax feed cannot hurt - yet the consistency of 28K seconds seems to argue against the (sudden) appearance of such a "well timed" noise source.    And - I immediately asked if any such "special program drive and/or event" occurred at/near that 28K seconds.   None was identified...

Anthony F. Seely said:

Is there a good picture in one of these posts that explains what occurs when there is a failure?

Greetings Anthony - great to have you here & assisting.

Post today @ 2:36 by o.p. Timothy may meet your requirement of, "good picture."    Four LA caps appear - 2 good - 2 post failure.

Like you - in my initial post (1st response here) I suspected timer/counter "roll-over."    Never would I have thought of 28K second WDG - though.

The consistency of that 28K seconds, "sucked me in."   And to my mind - such consistency rules out noise - especially as I asked poster to identify anything special (such as motor/actuator drive) at/around that 28K point.   (none was identified/admitted)

I further suggested that poster reduce his code to basics - in the attempt to eliminate all "intrusions" which may upset the SPI.  

Appreciate your ideas - surely hope poster will "share" final cause & fix...

I like your detail & logic - yet "nothing" w/in your review "keys into" the spectacular consistency - at/around the 28K second mark.    

If poster has a possible config issue - as you posit - how & why would it evidence itself (only) 28K seconds out?    (don't believe you've fully/properly (yet) absorbed that fact...)    In the first responding post - I specifically asked if any, "untoward/special events" occurred at/around this 28K second mark.    (such as the multiple-DMA alarm - you sounded - poster cited "no such occurrences!)

You guys are - of course - trained to (most always) key upon, "MCU-centric" issues.    Yet - the consistency of this issue - so far out timewise - attracts myself/others.   May I suggest that rather than seizing upon set-up/config (i.e. MCU innards) you look for means to "speed and/or advance" the occurrence of this issue?   (to far sooner than 28K seconds - that's a long time to "test & wait!")

Poster reports use of an RTOS - I'd like him to some way/how "kill that" - the more variables which can be eliminated - the better the odds of a "fix."

And of course 2 dissimilar MCUs (even from the same vendor) adds to the joy of this diagnosis.   

To my simple (yet curious) mind - a search both this forum and Herc. - for, "issues revealing hours out" seems indicated.   Being in this biz for awhile - its very hard to imagine most any issue being, "a FIRST!"    (and insiders seem not to have fully/properly engaged this (usually) helpful channel...)

CB1:

 As suggested, I will increment a counter for each and every signal for the Tiva and Hercules and constantly report those values.

However, note the following number of SPI transfers that completed when the failure occurred at 28K seconds for the 5 overnight tests performed so far:

TEST   TIME IN SECONDS    TRANSFERS

 1       28,784.204       2,227,832

 2       28,784.152       4,111,502

 3       28,700.878       2,125,224

 4       28,780.628       2,131,034

 5       28,782.352       2,131,323

VARIANCE:    83.326       1,986,278

The difference in the number of SPI transfers during that time is due to a difference in the operations performed while testing.

Since the failure appears to be independent of the number of transfers, and appears to only be based on time, this leads one to wonder if it is possible that the Hercules SafeRTOS is failing.

The Tiva FreeRTOS is evidently not failing because it continues to make SPI requests to the Hercules which does not respond, apparently, either due to an interrupt issue or SafeRTOS issue.

To confirm this theory, code will be included in the SafeRTOS task loop to toggle an LED. If the LED gets stuck, the SafeRTOS must be stuck.

Thanks,

Tim

BP101:

 I agree that FreeRTOS (used on the Tiva) or SafeRTOS (used on Hercules) could be doing something that I am not aware of.

I will also observe SPIENA and SPICLK drift which may be caught while incrementing a counter for each and every signal as suggested by CB1.

 Thank you,

Tim

Thank you, Tim - that data helpful/appreciated.    Following based upon your post: 08:59, 17 Apr 2015.

Do note test #2 - that cannot be a "real/proper" transfer count - can it?   (nearly double the count of all others!)

In the attempt to both, "Speed & immediately capture the first (likely) issue occurrence" (that's a valid goal - is it not) can't you implement (on the independent, monitoring MCU) a, "TRANSFER COUNT ARRIVAL TIMER" which will trip/trigger upon the passage of "excessive time" without any transfer signal's arrival?    I base this theory in the belief that yours (likely) operates much as past, "Command/Response" mutual handshakes - which should occur w/in a (limited) and (predictable) time-period.

As to your Time/Transfer chart: would it not prove useful to HALT that test - slightly before 28K7 seconds - and compare transfer numbers (then) at that (pre-failure) time?

Excelsior

One more "test/troubleshoot" idea dawns.    (and I really need to do (some) "for obscene profit" work)

Earlier I suggested that you "Kill the RTOS" - so that less "gremlins" litter your field of battle.     And - that's yet to happen.    Yet - if we could separate RTOS issues from all others - would not that be useful?

Here's a "back door" approach which may work - cannot you, "Reset  both Tiva's & Herc's SPI peripherals" at regular intervals - which should render (both) SPI ends, "free from any frequency drift or (other) potentially destructive, accumulative issues?"   (after all - they're the most likely - n'est pas?)

That done - and your noted failure continuing - does that not, "Raise RTOS (either side) as, "Prime Suspect?"    While requiring minimal effort - your part...   Seems quite the plan - to this reporter...

Doubtful - we don't usually source our legal material from "youtube." My reference was to the Japanese auto maker - I directed you to Google, "UA" - techniques/methods employed by the prosecution's expert witness (likely) serve as a model for small firms - such as yours & mine.

May we note that your case has gone, "way cold" - borderline "unresponsive" (thus unkind) to the time/effort expended by your, "helpers!"

Hello Anthony:

As requested, the CS and ENA signals are given in the graphic below.

CS is the second signal in CYAN and ENA is the third signal in MAGENTA:

The yellow trace is the active high signal from Hercules (slave) that informs the Tiva (master) that it (Hercules) is ready for another transfer. The Hercules sets that bit after the MIBSPI transfer of the group gives a Notification (interrupt) as described previously.

The DMA is not being used, only the MIBSPI with 32 8-bit words.

Yes, the CS is implemented as a GPIO because experience has shown the Tiva SPI CS is apparently not compatible with the MIBSPI and because of comments such as that at:

e2e.ti.com/.../309096

I am checking all the error bits of both the Tiva SPI and Hercules MIBSPI as indicated previously. I have been able to get the errors to trigger while debugging but they do not appear during the 8 hour period or after when the Hercules SafeRTOS fails.

The CS from the Tiva GPIO is tied to the Hercules MIBSPI CS. When a Notification interrupt occurrs, which indicates that the MIBSPI transferred group of 32 8-bit words is finished, the Hercules clears the ATTN GPIO bit and only sets it after it has read the data with mibspiGetData().

Thank you,

Tim

Timothy Ball said:

I will also try the HALT etcetera as you (cb1) suggested.

I suspect that the "greatest value" vendor's insiders may provide is the identification of a method to, "Speed this issue's occurrence!"   Waiting 8 hours to note, "Success/Failure" falls low on the time/cost/morale scale.

What will you/others do when such issues/failures take "YEARS" to develop?    This is (and remains) your critical path as I see it - and has (thus far - but for moi) been underserviced.    (to be kind)

Quote above recites "Halt" yet that was not my intent.    Quickly/smartly/regularly I argued for, "SPI Peripheral Reset" - on both sides (perhaps sync'ed) so that many/most SPI Peripheral "drifts" and/or "accumulating errors" could be greatly reduced.   I don't believe such brief, coordinated "peripheral reset" qualifies as a, "Halt."

cb1- said:

f) while you report that SPI commo breaks - have all (or most) other segments of the program continued as normal?   How have you (or have you) tested?

@marc_rjr:   You may note that your question had been asked - previously!

OP reported that (only) SPI communication was lost...    (post has gone on & on - fog of battle "hides" past data...)

Timothy Ball said:

I was able to speed up the Tiva FreeRTOS time by calling the FreeRTOS xTaskIncrementTick() function repeatedly.

That's great!   

Can you imagine how our world would be impacted (think medical research - drug qualification etc.) if methods to, "Speed yet still greatly anticipate outcomes" were not "de rigeur?"    Glad to see your attention (rightly) turned in that direction.

May I continue in the belief that my, "More Limited - Less Brutal" (Reset of the offending Peripheral (SPI this case)) usually proves superior?   (i.e. Sledgehammer not always required!    And such brutality - more likely to generate, "Unintended Consequences...")

One wonders too about the ability to periodically, "Reset the RTOS" - on both ends.    (should such "drift or accumulating error" be in play - one or both RTOS)

UPDATE:

Now using SafeRTOS vTaskIncrementTick() (versus FreeRTOS xTaskIncrementTick()) to speed up SafeRTOS time to get SafeRTOS to fail in about 1 minute instead of 8 hours.
Stay Tuned for further updates.
Thank you,
Tim

UPDATE:

SafeRTOS failing at Code Composer Studio (CCS) Disassembly (no source available) code depicted below:

Good Work!   Do you know what this means?  Is there some error condition above this that branches to $C$18? 

It doesn't look to me that this is related to the call to 'vApplicationTickHook' because I think the LDMFD that follows that call is a 'return'.    But maybe they expect you to implement some error handler in portasm.asm and if you don't then it'll just return to the branch self ?

UPDATE: Issue in vTaskIncrementTick():

In regard to memory allocation with SafeRTOS as documented by Wittenstein:

Each task and queue created consumes a small amount of RAM. Under FreeRTOS the required RAM is automatically dynamically allocated at run time. SAFERTOS does not permit dynamic memory allocation so the required RAM must instead be statically allocated at compile time, then manually passed into the xTaskCreate() and xQueueCreate() API functions.

I'm afraid I have not read through this thread to see if this has been covered or not - so apologies if there is duplication:  

Are you a SafeRTOS customer of WITTENSTEIN high integrity systems?  Or are you using an evaluation version of SafeRTOS?  If you are using an evaluation version then there is a time limit of 8 hours before the system must be reset to get another 8 hours.