This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC3220SF: Store (UART) logs offline for later debugging

Part Number: CC3220SF

Dear TI,

I have an application running on CC3220MODASF device on a custom PCB. Up to now it was mainly used in development environments with easy access to the devices. The devices seem to work fine most of the time, but they still show some instability when running a long time (they are suddenly unresponsive, and need a reset).  For some cases (e.g. no internet or lost server connection) I have already used (softy configured) Watchdog timer that will reset the device after a while, but apparently there are still some states/cases where the device stops functioning/crashes.

The main problem is that I don't know the reason that they are unresponsive. One reason could be a memory leakage that causes the device to run out of memory, causing it to be unresponsive. This is something that I should be able to analyze with the "Runtime Object View" tool on CCS even for shorter periods of time. I have planned to do this, but I would also like to analyze the last executions/ (UART) print statements of the device in order to check what it was doing to find the cause of the crash. However, this can only be done if I connect a debugging device before the application is actually crashed. In practice this is not feasible because the 'crashes' are too infrequent, so I am looking for alternatives. I have already looked into logging via MQTT, but the problem is that it would rely on an internet connection in order to log.

The most stable way that I can think of right now, is storing the logs into a persistent file on the device itself. This way I can take the crashed device and read out all the data (still have to figure out how that would be possible though) from the device storage. There is currently no (extra) external storage attached to the device on the custom PCB, so the only option is to do logging via the NWP on the SFLASH in that case. However, I am a bit afraid that the (most important) last few logs might not be writable to the device before it crashes. In that case there is no benefit to use this method at all.

I could not find a document/example that would tackle this problem or showing the possibilities to store the (UART) log print statements for monitoring/after-analysis, therefore I am asking on the forum what is the best suggested solutions are to tackle this problem.

Best regards,
MJ

  • Hi,

    Generally there is no reasonable solution of your problem especially when you not have connected other hardware which can store this data.

    Common way how to save information from error states like a hard-fault is change hard fault handler. Inside this handler you save this information into RAM and after that restart MCU. At startup code you will read this RAM and in case there will be some interesting information you will store them permanently. Unfortunately this approach is not possible with CC3220SF because ROM bootloader clear at boot time previous content of RAM from security reasons.

    Another option can be using as permanent storage part of XIP (on chip) flash. But this will work for very limited amount of data. Definitely not any kind of UART log.

    Usage sFlash to storing any kind of log is not a good idea. Because this approach will damage your sFlash at short time. Do not forget the sFlash have limited number of write cycles.

    Jan

  • If you are able to catch an exception (such Jan mentioned) you can copy the relevant information to the local flash (or to a raw section of flash on the SFLASH). Since you need just the last couple of messages, this should be ok.

    Br,

    Kobi 

  • Dear Jan and Kobi,

    Thanks for your replies.

    Just to be clear: If the devices would always automatically restart, then I wouldn't be so interested in offline logging, so assuring a reboot for the device is the higher (ultimate) goal here, so a solution for that is also welcome (instead of going through the difficulties of logging this data). I started this topic to discover how I can analyze this problem, because I just don't know how to start debugging.

    My first thoughts to approach this problem is described in the first post, so store these logs to get the reason why it crashed and then fix that problem. So I don't want to store the full UART log, but only the last few messages.

    I am a bit lost in the several API's available in order to store the logs and a lot more on how to catch the (hardware) fault. Especially catching seems tricky to me. From my observations, the device seems to crash completely (no UART output when I do connect it, and my Watchdog timer does not seem to work either).
    If there is no (easier) way to assure a reboot, then I have the following questions with regard to the logging:

    1) (How) would I be able to catch the hard fault handler, and store the data? Since this would be the most relevant message of all, plus maybe a couple of the last UART lines.

    2) Related with 1: Which API should I use for storing? You have the non-volatile storage (NVS) driver from TI Drivers and the FileSystem module from the Host Driver. However, can I still use these API's from the hard fault handler?

    3) Related with 2: With the secured flash access: only the NWP can write to that memory right? what if the NWP causes the problem such that I am unable to write it to the filesystem?

  • Hi,

    I can give you a few hints and ideas but not a complete solution. Real implementation will be up to you.

    • description how to use own hard-fault (exception) handler at TI-RTOS you find here
    • storing data from UART log will need some effort. My recommendation is to allocate at RAM space for  circular buffer. Together with writing to UART you will store data into this buffer. This buffer will hold a few kB of last UART printfs / logs. In case of hard-fault (or maybe NWP error events) you will store values from RAM into XIP flash (of chip flash at CC3220SF)
    • XIP is a on-chip parallel flash. It is splitted into two banks. In case of your code does not exceed 512kB, you can use 2nd bank for storing some temporary data. Be aware that XIP flash have limited write cycles as any other flash memory. Reading data from XIP flash can be done directly via pointer (XIP flash is part of CC3220SF address space). For writing you need to use Driverlib API ( \source\ti\devices\cc32xx\driverlib\flash.h ). You cannot use VNS driver. Description about XIP flash you find at TRM chapter 21.
    • Yes. Access into sFlash is done via NWP. Communication with NWP is done via SimpleLink drvier. Usage of SimpleLink driver in hard-fault handler is definitely not possible. That means you cannot save information into sFlash directly at hard-fault handler. But you can save information into XIP flash and after fresh boot you can copy this content from XIP into real file inside sFlash.

    Jan

  • Dear Jan,

    Thanks for your tips. This is definitely helpful. I would also recommend others to learn a bit more about TI-RTOS system modules, because a lot of existing functionality could help you to achieve your goals. You can find more info included in the SDK under "documents" (e.g. here: dev.ti.com/.../node )

    For example I think the LoggingSetup module with LoggerStreamer2 module would help implementing/using the circular buffer as suggested by Jan, since it allows an user defined handler.

    I parked the idea of the tracking the UART logs right now, because other things have more priority right now.
    However, I did come across a "Flight Recorder" tutorial by TI with the previous version of the LoggerStreamer: http://processors.wiki.ti.com/index.php/System_Analyzer_Tutorial_5A

    So right now I have only one question remaining. I succeeded to execute my Hook function in case of an HWI error (tested it with a printf statement), however when I am trying to reset my device from this function with following code:

    void hwiExcHookFunc(){
        PRCMMCUReset(true);
        // MAP_PRCMHibernateCycleTrigger();
    }

    The device does break the JTAG connection, but does not not reboot for both lines. Why is that the case, and how can I solve that?

  • Hi,

    I am not pretty sure why driverlib API does not restart MCU at your case.  Described API does restart application MCU + peripherals but not whole SoC. If you need restart whole MCU, you can use this way (at your case probably without sl_Stop()).

    Jan

  • Thanks Jan. That indeed solved my problem.