CCS/TM4C1294NCPDT: TM4C: Open for advice how to debug/trace for difficult bug

richard payne

Part Number: TM4C1294NCPDT

Tool/software: Code Composer Studio

I have coded up firmware based on TI-RTOS with USB which interface (SPI) with external dual ADC (24 bits) which captures data into RAM, process data stream and placed into USB firmware from the TI library.

It was running very well but it tends to crash after 4-48 hours (unpredictable timing) while streaming data to Laptop.

I have not yet ruled out whether this issue is USB related or not.

I welcome any suggestions on how to debug this issue, how to trap these crashes, especially working with 3rd party library and my firmware. Is there a useful strategy to look into RTOS, USB to detect this event.

I have been working on TI-RTOS from CC2640R2 and this device for 4-5 years so I'm pretty experienced but I have not done much for some advanced debugging level that I'm learning more.

In case there som bugs in TI library, what the best course of action and how to avoid bug fixed being overwritten from an update?

One thing I do notice the RTOS and TI library for TM4C is quite old 2017 and has not been updated for some time, I can't help wondering what this really means for the future of the TM4C family.

over 5 years ago

0 cb1_mobile over 5 years ago

Guru 117855 points

Greetings,

richard payne said:
Is there a useful strategy to look into RTOS, USB to detect this event.

Perhaps these initial methods will aid: (Note: my firm & I are 'outsiders' - deploy ARM Cortex MCUs from multiple vendors - thus employ 'FREE RTOS' [always/only - vendor agnostic])

while the timing of these crashes proves 'unpredictable' - might there exist (other) events/captures - which reveal (some) commonality? (i.e. Data Ram captures, data-stream processing, and/or USB transfers.)
might this be a, 'Single-Board Anomaly?' Does this issue persist across multiple of your boards and/or systems?
might noise sources - via power devices and/or EMI/RFI - even power supply 'peaks/dips' have been (positively) ruled out?
should the 'USB' - receiving the ADC's data-stream - be resident w/in a 'PC' - should not that (too) [PC] prove suspect? (thus switched w/another)
have you lowered the rate of each individual process - (i.e. via 'KISS directed - One & only one change at a time') - and carefully noted resulting, 'Crash Frequency?'
alternatively (and in opposition) you may 'Raise the rate of each process (individually, please) - noting the (hopefully increased) arrival & impact of such 'Crashes.' Via the 'combination of 'slowed & raised' data rates - it is expected that key/critical 'data rates' may be 'more quickly & optimally recognized!' Via this particular method (increased data rate) we intend to 'heighten and/or speed failure's onset' - which most always VASTLY SPEEDS & BETTER PINPOINTS Diagnosis!
you may, via 'separate/external' means, 'Time & Record' the arrival of each event! This (very) well enables your 'Recognition of the (likely) 'Onset of Trouble' - along w/identifying the 'prime suspect.' (my small firm has implemented such a, 'Diagnostic Event Capture System' via multiple ARM Cortex M7 MCUs coupled to (vast & fast) logging memory)
you've noted a '4-hour' period of (apparently) always, 'Successful Operation.' What is 'most likely to occur' - at/around/beyond that 'magic (4 hour) number?' You may try a general system pause & reset - prior to the elapse of 4 hours! (in case your (combined) system (somehow) cannot fully/properly accommodate 'longer continuous operation periods.') (We've 'cured' several 'client issues' in 'Just this Manner!' Sometimes operational success - minus absolute knowledge of 'causal nexus' - proves sufficient. And is far faster & (substantially) lower in cost...)

Ideally - it proves more desirable to, "Design-In Added Robustness" - than to "Trap these Crashes."

And - do consider - does it 'prove best' to employ (alone) the 'Known Failing System' to attempt a proper & efficient (yet) 'SELF DIAGNOSIS?' And a (potential) 96 Hour one - at that! Really?

0 Bob Crosby over 5 years ago

TI__Guru 72500 points

richard payne said:
I welcome any suggestions on how to debug this issue, how to trap these crashes, especially working with 3rd party library and my firmware. Is there a useful strategy to look into RTOS, USB to detect this event.

If you have enough flash space, you might want to recompile the TivaWare library with DEBUG defined so the library includes the ASSERT checks. You need to define the __error__() function, at least something like:

#ifdef DEBUG
void
__error__(char *pcFilename, uint32_t ui32Line)
{
    while(1);
}
#endif

If flash space is tight, you might want to just include some of the functions in your project and build with DEBUG defined.

richard payne said:
In case there som bugs in TI library, what the best course of action and how to avoid bug fixed being overwritten from an update?

It is our sincere hope that as we release a new version of the TivaWare library we only fix bugs, but I understand the concern. If you find a bug, please let us know as soon as possible. If you include the source code of a TivaWare function, it will compile and use that code and not pull the function from the library.

richard payne said:
One thing I do notice the RTOS and TI library for TM4C is quite old 2017 and has not been updated for some time, I can't help wondering what this really means for the future of the TM4C family.

We are actually really close to releasing an updated version of the TivaWare library. (Yes, CB1, you will believe it when you see it.) I am also expecting some increased marketing for the TM4C devices in 2020. The good thing is that the library is actually pretty stable. Most of the corrections in the next release are in the ASSERT statements.

0 Chester Gillon over 5 years ago in reply to Bob Crosby

Guru 92251 points

Bob Crosby said:
I am also expecting some increased marketing for the TM4C devices in 2020.

Is there going to a reference to the overlap between TM4C129 and MSP432E devices?

From a cursory glance it appears you can get members of the TM4C129 and MSP432E device families which have the same pinout / amount of memory / peripherals but the differences are:

a. The equivalent MSP432E device has a reduced set of published silicon errata.

b. Two different sets of software - TivaWare for TMC129 devices and the SimpleLink SDK for MSP432E devices.

0 Chester Gillon over 5 years ago

Guru 92251 points

richard payne said:
I have coded up firmware based on TI-RTOS

Have you seen Debugging Common Application Issues with TI-RTOS?

0 cb1_mobile over 5 years ago in reply to Bob Crosby

Guru 117855 points

Bob Crosby said:
you will believe it when you see it.

Sainted parents past assured the (pending) arrival of (both) 'Santa & the Easter Bunny' ... neither (ever) arrived! (Yet my 'only child' holiday presents were (discovered) 'stock-piled' w/in parents' closet ... had the 'famed duo' (Santa & Rabbit friend) arrived early?)

At one point it was (almost) assured here that, 'Improved & more challenging "Peripheral Code Examples" would be included.' If the 'promised/teased' (update) consists primarily of 'Assert statement corrections' - many here will be, 'Beyond Disappointed!' (existing examples are of the most basic nature - never coming close to (either) the more 'useful/general/complex' (& never/ever, 'corner cases') - and have remained (highly) static for (beyond) TEN Years now...)

Might some 'blame' for such (wished for - yet delayed) library 'improvements' attach to:

the 'welcoming' of multiple sensors - along w/their 'highly specific coding' - which distract & divert key forum staff from their 'Central' MCU Mission?'
the repeated request (and acceptance) of 'poster's engineering design & principle education' - which strays (SO FAR) from the forum's MCU-Centric intent?

Some wounds may be ... 'self-inflicted' - the cure may be, 'Increased & Sharpened Focus' upon the Central (MCU) MISSION!

0 Bob Crosby over 5 years ago in reply to cb1_mobile

TI__Guru 72500 points

Hi CB1,

Yes there are improved peripheral code examples done by Ralph and Charles. Most corrections to the driver library functions are corrections to the ASSERT statements.

0 twelve12pm over 5 years ago

Genius 4455 points

richard payne said:

In case there som bugs in TI library, what the best course of action and how to avoid bug fixed being overwritten from an update?

If you are referring to making changes in TivaWare and/or TI-RTOS yourself, and then having your changes overwritten by an update from TI, then the tool you need is a version control system.

At my company we use Subversion, which is free, easy to learn/use, convenient, and safe, to manage firmware source code as well as electronic design files in a central repository. This system ensures that people don't accidentally overwrite each others' changes.

With version control, you would import TI's stock TI-RTOS / TivaWare source code with no changes. This would be called "trunk." Then you would form a "branch" for your company's customizations. All of your company's customizations go into the branch, not into trunk. When TI issues an update, you apply it to "trunk" and then "merge" it to your branch. Provided that TI hasn't changed the same areas of code that you have customized, this will preserve both TI's and your changes in your branch. (If TI changes the same parts of the code that you have customized, the version control system will NOT overwrite your changes with TI's changes; it will prompt you to look at what is being changed and let you decide what to do for each instance.) Thus the "trunk" is always synchronized to TI's code with no customizations and the branch always contains your customizations, even as you bring in updates from TI.

Having said all of the above, if you do fix bugs in TI code, you should send your fixes to TI so that they can be fixed for everyone. But in case you make customizations that don't make sense for TI, you can manage them as I described above.

To learn more about version control and Subversion: http://svnbook.red-bean.com