Tiva tm4c129 unusual hard fault

Tim Cooper1 said:

...multiple cards;  happens on some cards with great frequency; not observed on other cards. 

Few here (i.e. none) can compete w/Amit in this MCU specific realm - thus outsiders are forced to steer towards general test/troubleshooting - groomed over the ages...

The fact that you report "some cards" fail - but not all - may prove worthwhile.   That said - might you (somewhat) define "some?"   (i.e. as you know; 1% vs. 20% "failure rate" leads to far different "suspects.")   I present (below) items which usually - but not always - have resolved similar issues @ our small, tech firm & our clients:

• were "all" cards built at the same time - via the same vendor - using the same assembly methods & equipment?
• were all "critical" components held constant during such builds?   Did you supply all of the components or rely upon CM?  (contract mfgr)
• it's likely that your "cards" are part of a (bigger) interconnected system.   How have you eliminated external signals, connections, noise from being causative?
• how critical is the "SysTickValue?"   The SysTick is just 24 (not 32) bits.   Might it have "rolled over" - and that fact missed?
• usually - but not always - with the speed you're running & the number of "demanding" peripherals - there is potential for a "race" condition

What to do:  (imho)

• dial down your MCU clock's speed (temporarily, during test) so that factor may be probed  (running @ 50MHz seems a good beginning)
• you note that there's a "2mS" delay prior to your accessing the external SPI peripheral.   Such indicates that you have (some) concern about the "readiness" of that peripheral - or other system matters?   How did that concern arise?   And have you monitored that "2mS" to confirm that it is "proper" during failure?
• you "assert some GPIOs" to enable "an" external device.   (I read that as one device)   Why several GPIOs for (one) device?   Are you quite sure that it is (only) the external SPI peripheral which proves the culprit?   (there is conflict w/in your language, here)
• systematically - one by one - eliminate, "USB, serial I/O" to learn if such reduction eliminates your issue.   Serial Port - if via UART - should avoid the introduction of any delays - especially those occurring w/in interrupt handlers.
• focus intently upon the correct powering, interconnect & proper set-up of that external SPI device.   Should that SPI device be located, "off board" might the "swap" of SPI devices - between "known good" and "known failed" boards provide insight?

Firm and this reporter are serious believers in "KISS."   We always (and only) focus upon and exhaustively test each peripheral (one by one) trying to "break it."   Only when (we believe) that peripheral (or external device) to be "bullet-proof" will we admit it into our (otherwise) "known good" program and/or board mix.

All other methods - especially so when many peripherals & external devices are "in play" - have received our (kind) designation, "Maximum Randomness!"   And - when that's allowed (or worse) encouraged - the fault you characterize as "unusual" - rises so often as to "defy that characterization..."   (i.e. becomes quite "predictable!")

Hello Amit,

I reduced the PLL output frequency to 60 MHz;  this showed similar behaviour.   I will continue to test and get the other info as I can.

Thanks!

Tim Cooper1 said:

removing a device and having it work is not a conclusive indicator

No one here has made such (conclusive) claim - yet (some) indicator trumps (no) indicator!     (i.e. effectively the present condition...)

Might the date codes on the MCUs differ between "pass/fail?"

Hello Amit,

Additional info, with another failure mode.  This test was run at 120 MHz. This mode is also observed frequently:

REGISTERS:

Program received signal SIGINT, Interrupt.

0x0001c108 in __udivdi3 ()

(gdb) bt

#0  0x0001c108 in __udivdi3 ()

#1  0x000c55b8 in ?? ()

Backtrace stopped: frame did not save the PC

(gdb) info registers

r0             0x4437d 279421

r1             0x0 0

r2             0x124f80 1200000

r3             0xe000e018 -536813544

r4             0x4438f 279439

r5             0x42 66

r6             0x20023fac 537018284

r7             0x0 0

r8             0x0 0

r9             0x0 0

r10            0x0 0

r11            0xfffffffe -2

r12            0x4c 76

sp             0x20023f68 0x20023f68 <pui32Stack+16232>

lr             0xfffffff1 -15

pc             0x1c108 0x1c108 <__udivdi3+500>

xPSR           0x21000003 553648131

STACK starting at sp - 0x20:

(gdb) x/64xw 0x20023f68-0x20

0x20023f48 <pui32Stack+16200>: 0xffffffff 0xffffffff 0xffffffff 0xffffffff

0x20023f58 <pui32Stack+16216>: 0xffffffff 0xffffffff 0xffffffff 0x2002be84

0x20023f68 <pui32Stack+16232>: 0x0004437d 0x00000000 0x00124f80 0xe000e018

0x20023f78 <pui32Stack+16248>: 0x0000004c 0x00029411 0x00032c48 0x21000003

0x20023f88 <pui32Stack+16264>: 0xffffff38 0x0000010a 0x20023fac 0x000c55b8

0x20023f98 <pui32Stack+16280>: 0x0005a6d0 0x000297d7 0x00000000 0x00000010

0x20023fa8 <pui32Stack+16296>: 0x20035094 0x000e0b42 0x20028258 0x20024a70

0x20023fb8 <pui32Stack+16312>: 0x20024a78 0x00029801 0x00005254 0x00025529

0x20023fc8 <pui32Stack+16328>: 0x20027e80 0x00000054 0x00000000 0x00000000

0x20023fd8 <pui32Stack+16344>: 0x000018e0 0x00000000 0x00000000 0x00000000

0x20023fe8 <pui32Stack+16360>: 0x00000000 0x0001c729 0x000018e0 0x0001c463

0x20023ff8 <pui32Stack+16376>: 0x00000000 0xffffffff 0x00000000 0x00000001

0x20024008 <__ctype_ptr__>: 0x00054398 0x20024010 0x00000000 0x00054520

FAULT REGISTERS:

(gdb) x/1xw 0xe000ed28      

0xe000ed28: 0x00010000

(gdb) x/1xw 0xe000ed2c

0xe000ed2c: 0x40000000

(gdb) x/1xw 0xe000ed34

0xe000ed34: 0xe000edf8

(gdb) x/1xw 0xe000ed38

0xe000ed38: 0xe000edf8

PC where failure occurred:

   0x0001c0f8 <+484>:    mov    r10, r0
   0x0001c0fa <+486>:    mov    r0, r7
   0x0001c0fc <+488>:    bl    0x1a1dc <__aeabi_uidivmod>
   0x0001c100 <+492>:    ldr    r2, [sp, #0]
   0x0001c102 <+494>:    mul.w    r2, r2, r10
   0x0001c106 <+498>:    orr.w    r1, r5, r1, lsl #16
   0x0001c10a <+502>:    cmp    r2, r1
   0x0001c10c <+504>:    bls.n    0x1c11a <__udivdi3+518>
   0x0001c10e <+506>:    adds.w    r1, r1, r8
   0x0001c112 <+510>:    add.w    r0, r10, #4294967295
   0x0001c116 <+514>:    bcc.n    0x1c1f2 <__udivdi3+734>
   0x0001c118 <+516>:    mov    r10, r0
   0x0001c11a <+518>:    ldr    r5, [sp, #4]
   0x0001c11c <+520>:    orr.w    r0, r10, r11, lsl #16
   0x0001c120 <+524>:    subs    r1, r1, r2
   0x0001c122 <+526>:    umull    r2, r3, r0, r5
   0x0001c126 <+530>:    cmp    r1, r3
   0x0001c128 <+532>:    bcc.n    0x1c144 <__udivdi3+560>
   0x0001c12a <+534>:    lsl.w    r6, r6, r4

Now, an unusual observation is that r2 = the timer reload value I use, and r3 = address of tick value register;  I find it unlikely that these would occur inside the divide function.

I will perform the PIOSC test later today.

Thanks!

Hello Tim,
I'm not sure if you can debug this on the level I'm going to mention, but here goes.....
From your Opening Post:
"when the processor locked up, ctl-c and GDB reported the PC at 0x1c22a"

So the 32bit store (STR) instruction at 0x1c22a locked the CPU up. It would do that if it went to store outside of the RAM block. In TMC123GH6PM the RAM runs from 0x20000000 to 0x20040000. The instructions immediately prior to that show R3 being setup as a pointer for the store close to where the RAM ends at 0x20033448. As you mention that R3 is much higher than expected would it be fair to say that it is the advancing R3 that eventually crashes the CPU?

If so there's a 'leak' somewhere that allows this. As to finding it....

Is there a way to trap the R3 value as it runs away? Is this possible if the code is compiled from C and would the R3 use be the same each time? Amit and CB maybe able to help there.
Other thoughts:
1) Breakpoint it and run between break points simply watching R3. Go backwards in the initialisation in steps looking for where R3 'breaks free'. Depending on the system you maybe able to do this single stepping or with a full speed run and stopping it dead with a breakpoint.

2) Modify the code to trap itself when either R3, or the variable it represents runs away. (what is being stored in RAM?)

The trace information I can't read properly but they mention the timer/s. The problem also seems to occur after setting up the GPIO after which there's a 2mS delay. The problem also seems to vary somewhat from board to board. If possible, I would lengthen that 2mS time to say 200mS to rule it out for now. Also, does the system have a way to establish that the external hardware has powered up fully before using it? It sounds like a borderline timing issue.

"2mS: that is the manufacturer's spec regarding the time between when power is applied to the external device (via on-board switchable power supply: part draws ~100mA) and the time when configuration, via SPI, should occur."

And we know what manufacturers specs mean sometimes don't we :)


Also, another thought; this is provoked by one of Amit's posts from way back. Make sure that you wait for each and every module you use on the TM4Cx to become available for use and that you also reset it. So the order would be, and correct me Amit if I'm wrong:

1) Enable clocking to module
2) Enable the module
3) Reset it
4) Wait for the module to become ready (check its flag in system control)

These steps need doing for all of them that are used. The datasheet mentions this for some modules but not others. Doing so for all of them will prevent some wild goose chases and at times some unpredictable behaviour further down the line which is hard to trace.

Hope that helps Tim.

One may consider, "Test-Driven Development" as well - which would briefly "follow" the above by measuring & confirming key output signals - to insure that the peripheral device is (truly) "up & functional."

Even "elaborate" thought process - and especially lesser "cut/paste" (i.e. "borrowed code") - benefits from such systematic test/verification - the earlier the better...