CCS/LAUNCHXL-CC1310: How do I debug stuff like this?

It seems to be faulting in Idle thread context, going through the stack manually and cross-checking the symbol table (why doesn't CCS do this for me?) suggests that the backtrace is as follows:

my_fault_handler2(NULL)
ti_sysbios_family_arm_m3_Hwi_excHandler__I(NULL, 0)
ti_sysbios_knl_Idle_funcList__C
ti_sysbios_knl_Task_schedule__I
ti_sysbios_family_arm_m3_Hwi_dispatch__I
PowerCC26XX_standbyPolicy
ti_sysbios_knl_Idle_funcList__C
Power_idleFunc
ti_sysbios_knl_Idle_run__E
ti_sysbios_knl_Idle_loop__E
ti_sysbios_knl_Task_exit__E

So, why is TI-RTOS crashing in the idle thread? I don't even have any user idle functions in there..

Curiously, DFSR points to ti_uia_loggers_LoggerStopMode_Instance_State_0_hdr__A - 1 and MMAR points to ti_uia_loggers_LoggerStopMode_Instance_State_0_packetArray__A although I've no idea what the relevance of either of these symbols are, other than they appear to be related to some sort of logging facility that I'm not (to my knowledge) currently using.

Why are their addresses in those registers? DFSR isn't supposed to be an address and the value it holds isn't legal according to the datasheet.. MMAR isn't mentioned in the datasheet but perhaps it's actually the MMFAR register which points to the location of a memory fault of some kind.

R1-R9 are pointing t0 ti_sysbios_family_arm_m3_Hwi_excHandlerAsm__I (0x1001c9fd), R10 points to ti_sysbios_family_arm_m3_Hwi_pendSV__I (0x1001ca3b) - not sure why so many point to the same place?

and most of the other registers look like nonsense...

Any ideas? Even further steps for debugging or getting CCS to tell me something useful would be really helpful

Michael Moon said:

Even single-stepping through, when I try to 'step into' Semaphore_pend in my task, it throws a fault and stops at the breakpoint in my fault handler.

The default debug properties for a CCS project under Program/Memory Load Options is that:

a) Interrupts are disabled when Assembly Stepping or when Source Stepping.

b) Interrupts are not disabled when Running.

If you change the CCS project properties to NOT disable interrupts when Assembly Stepping or Source Stepping does that avoid the SYS/BIOS exception?

The reason for asking is that such a change to the CCS project properties prevented a similar crash in Step Into (F5) and Step Over(F6) not working on CCS 6.1.3.00033. In the referenced thread, the cause of the SYS/BIOS exception wasn't identified.

Michael Moon said:

CCS seems unable to give any sort of usable backtrace, so I've no clue where it's actually crashing, other than "somewhere in the call chain initiated by Semaphore_pend".

Are you using the GNU or TI ARM compiler?

There is an outstanding bug where as of CCS 7.1 when using the GNU ARM compiler the stack backtrace is not always complete - see CCS 6.2.0.00050 doesn't display complete stack backtrace when a Cortex-A15 SYS/BIOS program using the GNU compiler terminates due to an error or calling BIOS_exit()

I am using the TI compiler.

I've left the interrupts-during-stepping setting at its default value which matches what you describe; I will check the linked article for further info. In my case however, the fault occurs both while stepping and while running so I don't believe that that's my issue.

Image

If I manually edit the stack pointer register (SP), I can get partial backtraces by stepping over functions CCS doesn't have symbols for, but I'm still terribly confused about why it's crashing like this in the first place..

Furthermore, the addresses often don't line up with the functions and often the function arguments look very wrong as well.

No matter how far up the stack I walk, all the function calls seem to be deep within the TI RTOS kernel and I can't fathom what the actual problem is..

The few object pointers and things I feed in from my code seem to be intact although I'm never entirely certain what's supposed to be in them - documentation on that is very sparse and reading the source code to find out gets rather tiring as often there's chains of indirected functions and all sorts of shenanigans between something being set and the same thing being used. Things passed between functions within TI RTOS seem fairly broken a lot of the time, although I've no idea how or why that's happening..

For example, in Hwi_dispatchC in the above screenshot, hwi apparently should be a pointer to a Hwi_Object but the pointer address is 0x40081000 - the address of the IOCFG0 register - why?

I2CCC26XX_hwiFxn apparently wants an argument that's a pointer to I2C_Handle, but instead it simply gets passed the interrupt number - why?

Why does CCS list the address if I2CCC26XX_hwiFxn as 0x6F6A when the symbol table written to Debug/output.map says this is inside the function "ti_uia_loggers_LoggerStopMode_write1__E" and I2CCC26XX_hwiFxn apparently isn't even listed in either the 'Symbols sorted by address' or 'symbols sorted by name' section ?

In short, how on earth am I supposed to find out what the real problem is when nothing CCS tells me makes any sense?

Michael Moon said:

Why does CCS list the address if I2CCC26XX_hwiFxn as 0x6F6A when the symbol table written to Debug/output.map says this is inside the function "ti_uia_loggers_LoggerStopMode_write1__E" and I2CCC26XX_hwiFxn apparently isn't even listed in either the 'Symbols sorted by address' or 'symbols sorted by name' section ?

That sounds like there is a mismatch between the symbols being used by the CCS debugger and the contents of the Debug/output.map file.

When you start a debugging session is CCS programming the Debug/output.out file into flash?

Can you look at the CCS Debug Configurations and post the image of the Program tab for your project as that should show what program CCS will program into flash when starting a debug session.

Yes, when I change my code, the behaviour of the board changes as expected - except for all these seemingly random random crashes!

Image - debug configuration looks ok to me

I disabled my crash handler functions (which I only used in an attempt to debug these faults in the first place) and turned on all the debugging stuff I could find, yet CCS still refuses tell me what the actual problem is..

Now my crashes look like image, but that's still basically useless.

If I try to edit the stack pointer in core registers to show me the reported stack pointer at crash time, nothing happens - it won't even update the UI element, but if I go to edit it again it'll show the value I put in, until I press enter at which point it reverts to the old value and does nothing else.

I've tried the instructions here to no avail - image is the result. Note that SP and LR are still showing the old values because CCS ignores me when I put new ones. Furthermore, my symbol table contains nothing at all in the 0x1000xxxx range, and I can't find much information about what's supposed to be there as the CC1310 TRM seems to be lacking a proper memory map.

As can be seen here, all my tasks are sleeping - and this crash apparently occurs in TI-RTOS's Idle thread! I have not added any custom idle functions, I only use Idle for the built-in power saving stuff. Also note that neither of my stacks have overflowed.

The various addresses evident in the system stack (0x20004C00-0x20004FFF) are various TI-RTOS internal stuff, power and clock modules according to the symbol table.

Without being able to change the stack pointer at will, and with having to look up each address manually in the symbol table, interpreting the raw stack is pretty hard work, and doesn't seem to give any useful information even when I slog through it.

The most infuriating part is I'll change something totally innocuous and unrelated, then things will suddenly work fine for a while and I can get some actual work done, then after another little while I'll run straight back into this again and get completely stalled while I stumble around randomly changing stuff until the code suddenly runs again for no apparent reason. I'm quite adept with debugging stuff using another embedded toolchain (gcc/gdb) but this TI toolchain simply won't give me the information I need, and constantly points at TI-RTOS internals.

Any and all help is appreciated because at this stage, all the information I possess suggests that it's TI-RTOS itself that's crashing, and there seems to be some sort of insurmountable disconnect between CCS debug view and the build toolchain which makes discovering otherwise incredibly difficult.

Michael Moon said:

Furthermore, my symbol table contains nothing at all in the 0x1000xxxx range, and I can't find much information about what's supposed to be there as the CC1310 TRM seems to be lacking a proper memory map.

I found the thread What is the address range of CC1310 memory ? which says the CC1310 has a 128Kbyte ROM starting at address 0x10000000.

The Loading TI-RTOS in ROM Symbols section of the TI 15.4 Stack SDK User's Guide says all TI-RTOS kernel code in ROM starts with address 0x1001xxxx and shows how to load symbols in the CCS debugger for the TI-RTOS functions in ROM.

If you look in your TI-RTOS installation there should be a packages\ti\sysbios\rom\cortexm\cc13xx\golden\CC13xx\rtos_rom.map linker map file containing listing the addresses of the symbols in the TI-RTOS kernel code in ROM.

Michael Moon said:

As can be seen here, all my tasks are sleeping - and this crash apparently occurs in TI-RTOS's Idle thread!

Does your .cfg file for your project use the ROM module ti.sysbios.rom.ROM, and if so does your application code use any of the the RAM or flash areas which the ROM code uses?

The thread RTOS/CC1310: Jumping between two applications in flash notes that the TI-RTOS code in ROM expects some RAM and flash data at fixed addresses, and presumably strange crashes will occur if the application attempts to use the same RAM / flash areas as the ROM code.

Thanks Chester,

While trying to get more debug information I had removed ti.sysbios.ROM from my configuration (to get stack checking and at least attempt to see what these 0x100xnnnn symbols are), but that didn't seem to help at all - things are apparently still jumping into rom anyway.

The second thread you linked says "The linker command files for both projects will place the ROM referenced BIOS objects at the addresses required." - since I'm only trying to run a single application, the linker should do this job for me without problems right? My linker script is the default from the empty_min example upon which I built my project.

Could it be that TI-RTOS is using ROM calls even though I've told it not to, and the linker is failing to reserve that memory for the ROM code because I've disabled it, thus causing all this madness?

I had a look at that rtos_rom.map file, it's very curious -

Why does it have a bunch of symbols in user flash (0x0 - 0x20000)? If the corresponding object file also has them, could that be confusing my debuggerwhen I load the rtos_rom.xem3 object file?

Why is there nothing in the 0x1000nnnn range? That's where I frequently find my code spinning when I pause things (even if it seems to be working fine), and where it frequently seems to crash. The memory map says this is part of the ROM area, yet there's no symbols for this region that I can find.

Since CCS seems to be unable to stack-walk past symbols it doesn't recognise, this makes debugging extremely difficult.

Why does it have symbols in RAM outside the 0x20000100-0x20000600 range that your thread says is ROM .bss section? Could that also be confusing my debugger when I load the rtos_rom.xem3 object file?

Michael Moon said:

Why is there nothing in the 0x1000nnnn range? That's where I frequently find my code spinning when I pause things (even if it seems to be working fine), and where it frequently seems to crash. The memory map says this is part of the ROM area, yet there's no symbols for this region that I can find.

I think the 0x1000nnnn range includes the ROM version of CC13xx Driver Library.

E.g. looking at the online documentation for rom.h there are the macros ROM_HAPI_TABLE_ADDR and ROM_API_TABLE which point at tables on ROM functions in the 0x1000nnnn range.

Is there some way to feed CCS some symbols for this area? It seems like the vast majority of the time I'm trying to debug stuff, the PC will go into this area and CCS becomes totally lost.. I can single step through and sometimes it pops out somewhere else and sometimes it'll just go back to 'Running' state even though I'm trying to step

Even when things are working (relatively) normally, the debug info from CCS seems somewhat messed up..

See image for example - Why is my (currently running) task not listed in the ROV Task viewer? Why is there an extra symbol in the Disassembly view? Why is the stack pointer apparently pointing at unused stack space? Why can't I see either of the listed function addresses in the stack? (and yes I know about how the address LSB is used strangely on ARM cortex architecture)

I'm trying to work out why this task appears to stop at a Task_sleep() call - this is the view from a breakpoint placed on that Task_sleep call...

Michael,

Did you create the IMU_Task? If so, how did you create it? [statically in the *.cfg file? Dynamically using Task_create(), or using Task_construct()?]

For a long time, constructed Task threads would not show up in ROV (I'll have to double check whether or not this is still the case). If created with Task_construct(), this may be the reason it's not showing up in ROV.

Steve

Michael Moon said:

Why is there an extra symbol in the Disassembly view?

If you mean the $C$CON670 symbol, symbols of that name are static labels used by the compiler for such things as loops. They don't show up in the map file.

Michael Moon said:

Why is the stack pointer apparently pointing at unused stack space?

From the IMU_Task::Task_Function(unsigned int) prologue there are 9 registers pushed on the stack and then 0x94 bytes of stack space allocated. Given the current SP is 0x20000560 that means at the function entry the SP would have been 0x20000560 + 9 * 4 + 0x94 = 0x20000618 which straddles the symbol ti_sysbios_knl_Task_instance_State_0_stack_A at address 0x20000600 shown in the memory browser.

Where does the symbol ti_sysbios_knl_Task_instance_State_0_stack_A come from, and is there the possibility that the stack areas of different tasks is overlapping leading to corruption?

@Steven,

> Did you create the IMU_Task? If so, how did you create it? [statically in the *.cfg file? Dynamically using Task_create(), or using Task_construct()?]

I'm using a mixture of Task_create and Task_construct.

I need multiple instances of some tasks, excluding static instantiation unless I set up multiple static buffers, which prevents me trimming memory usage in the case where I choose only a specific subset of tasks to start at runtime.

Other tasks always run, so I'm using static instantiation for those. I can trivially flip them to dynamic and grow the heap if TI-RTOS limits me to only using a single method of creating tasks (a limitation I haven't seen in the documentation).

> For a long time, constructed Task threads would not show up in ROV (I'll have to double check whether or not this is still the case). If created with Task_construct(), this may be the reason it's not showing up in ROV.

Really? Wow, is that documented somewhere? Any timeline for that getting fixed?

There have been a few occasions where I could find some of my tasks in the ReadyQ but not other views, but currently they seem to be vanishing from there as well.

@Chester,

> If you mean the $C$CON670 symbol, symbols of that name are static labels used by the compiler for such things as loops

I meant ti_sysbios_family_arm_cc26xx_Timer_setNextTick__E__mangled()

The PC is after this symbol, yet the backtrace does not show this as being the current function.

Furthermore, the disassembly preceding it suggests that it doesn't belong here - IMU_Task::Task_Function doesn't appear to have a return of any kind before it.

This looks like a bogus symbol, possibly from rtos_rom.xem3 which I apparently have to add to have even a sliver of hope of debugging the stuff in 0x1001nnnn memory range.

Perhaps I should objcopy rtos_rom.xem3 to only include symbols from 0x1001nnnn and use that instead?

> Where does the symbol ti_sysbios_knl_Task_instance_State_0_stack_A come from

Inside TI-RTOS presumably.

> is there the possibility that the stack areas of different tasks is overlapping leading to corruption?

Only if the toolchain's linker is horribly broken. I'm not even trying to do anything fancy with memory layout.

After removing all use of malloc and free, I seem to be having profoundly fewer crashes, but here's a similar one that popped up while I was stepping through my code while trying to find why i2c_transfer never returns after a while:

(from ROV Hwi Exception viewer)
Decoded exception,
Decoded,Undefined Hwi: 197
Exception context,
$addr,0x0
$type,ti.sysbios.family.arm.m3.Hwi.ExcContext
threadType,ti.sysbios.BIOS.ThreadType_Hwi
threadHandle,0xacf5
threadStack,0x1001c9fd
threadStackSize,268552701
r0,0x1001c9fd
r1,0x1001c9fd
r2,0x1001c9fd
r3,0x1001c9fd
r4,0x1001c9fd
r5,0x1001c9fd
r6,0x1001c9fd
r7,0x1001c9fd
r8,0x1001c9fd
r9,0x1001c9fd
r10,0x1001ca3b
r11,0xc335
r12,0xc500
sp,0x101c90f
lr,0x85007500
pc,0xd101cf0e
psr,0xffc6
ICSR,0xa5c5
MMFSR,0x10a6cc61
BFSR,0x3f56c93
UFSR,0x2900371
HFSR,0x33119f11
DFSR,0x8020003f
MMAR,0x320002a
BFAR,0x55000333
AFSR,0x10df1404
Exception call stack,
0    <symbol is not available>,PC = 0xD101CF0E FP = 0x0101C90F
Decoded exception,
Decoded,Undefined Hwi: 197
Exception context,
$addr,0x0
$type,ti.sysbios.family.arm.m3.Hwi.ExcContext
threadType,ti.sysbios.BIOS.ThreadType_Hwi
threadHandle,0xacf5
threadStack,0x1001c9fd
threadStackSize,268552701
r0,0x1001c9fd
r1,0x1001c9fd
r2,0x1001c9fd
r3,0x1001c9fd
r4,0x1001c9fd
r5,0x1001c9fd
r6,0x1001c9fd
r7,0x1001c9fd
r8,0x1001c9fd
r9,0x1001c9fd
r10,0x1001ca3b
r11,0xc335
r12,0xc500
sp,0x101c90f
lr,0x85007500
pc,0xd101cf0e
psr,0xffc6
ICSR,0xa5c5
MMFSR,0x10a6cc61
BFSR,0x3f56c93
UFSR,0x2900371
HFSR,0x33119f11
DFSR,0x8020003f
MMAR,0x320002a
BFAR,0x55000333
AFSR,0x10df1404
Exception call stack,
0    <symbol is not available>,PC = 0xD101CF0E FP = 0x0101C90F

Debugging this sort of thing is maddening, because all three of SP,PC,LR are well outside valid regions of the memory map, and TI-RTOS/CCS apparently doesn't even understand what sort of exception this is.

0x1001C9FD is the address of ti_sysbios_family_arm_m3_Hwi_excHandlerAsm__I() which doesn't seem helpful at all.

The SP from 'Core Registers' seems ok, but CCS can only backtrace a couple of functions, then chokes on 0xFFFFFFFD which it apparently can't recognise as the exception descriptor.

Furthermore, CCS won't let me update the SP,LR,PC registers so I can manually step over that part of the trace - I can write new values but it reverts to the previous one when I press enter.

This requires me to backtrace manually, thus:

SP -> 0x00000001    0x1001BBD9    0x20003A3C    0xFFFFFFFD    0x4009202C    0x00001378    0xFFFFFFFF    0xFFFFFFFF    0xFFFFFFFF    0xFFFFFFFF    0xFFFFFFFF
0xFFFFFFFF    0x00000000    0x10000000    0x00000000    0x20003A3C    0x00006D6D    0x1001C93D    0x1001C950    0x41000000    0x00000001    0x00000000    0x4008218C    0xF3BFFF3A
0x00006D6D    0x000077EF    0x000077EE    0x01000000    0x20003A3C    0x00000000    0x00001378    0xFFFFFFFF    0xFFFFFFFF    0x0000C69B    0x20003A3C    0x1001BD15    0x20003A3C
0xFFFFFFFF    0xFFFFFFFF    0x1001C20D    0x20003A3C    0x1001B425    0x00000000    0x00000000    0x1001C207    0x1001B425    0x1001BE05    0xBEBEBEBE (end of ti_sysbios_knl_Task_Instance_State_0_stack__A)

0x1001BBD9 is ti_sysbios_family_arm_m3_Hwi_excHandler__I()
0x20003A3C is ti_sysbios_knl_Task_Module_State_0_readyQ__A
0x00001378 is (according to Debug/configPkg/linker.cmd) ti_sysbios_knl_Idle_funcList__C
0x00006D6D is ti_uia_loggers_LoggerStopMode_write4__E
0x1001C93D is somewhere inside ti_sysbios_family_arm_m3_Hwi_dispatch__I()
0x1001C950 is "nvic_icsr"
0x000077EF is inside PowerCC26XX_standbyPolicy()
0x000C69B is inside Power_idleFunc
0x1001BD15 is inside ti_sysbios_knl_Idle_run__E
0x1001C20D is inside ti_sysbios_knl_Idle_loop__E
0x1001B425 is ti_sysbios_knl_Task_exit__E which I presume to be the handler for Task function return
0x1001C207 is ti_sysbios_knl_Idle_loop__E
0x1001BE05 is ti_sysbios_knl_Task_enter__I
and 0xBEBEBEBE is the canary value that startup seems to fill memory with.

It's worth noting that I have NOT added any idle functions of my own - this is 100% TI-RTOS internal stuff.

As far as I can tell, this says it threw a hardware exception of some kind in the idle thread during TI-RTOS's on-idle power saving...

I guessed that maybe CCS expects some handler to fill a struct somewhere rather than examining the stack directly on a HW exception, so I switched my handlers for default ones.

Now the Hwi/Exception info looks slightly more sensible, it suggests that PendSV is *not* triggering at tirtos_cc13xx_cc26xx_2_21_00_06/products/bios_6_46_01_37/packages/ti/sysbios/family/arm/m3/Hwi_asm.sv7M:226 and the PC is falling through to the nvic_icsr symbol and faulting when it tries to execute the word there which is a pointer rather than code.

Why this might be I have absolutely no idea - this is *deep* in TI-RTOS internals and all I'm currently trying to do is a simple I2C_transfer()..

Michael Moon said:

Could it be that TI-RTOS is using ROM calls even though I've told it not to, and the linker is failing to reserve that memory for the ROM code because I've disabled it, thus causing all this madness?

I don't have a CC1310 at the moment, but was using a LAUNCHXL-CC2650 which has the same ROM structure to investigate detecting when the TI-RTOS ROM code is called. In the CCS debugger set two watchpoints to detect reads from the TI-RTOS ROM code at addresses 0x1001000 ... 0x1001FFFF (maximum range of one watchpoint is 0x8000 bytes):

When these watchpoints were set and the ROM configuration code block in the .cfg file was commented out the watchpoints were not triggered, which confirmed that the TI-RTOS ROM functions were not being used.

Whereas when the ROM configuration was enabled in the .cfg file the watchpoints were triggered.

Can you try setting the same watchpoints on your project which should have disabled use of the TI-RTOS ROM and see if they trigger?

I'm currently reinstalling CCS from scratch because, after trying to update to 7.1, the cfg viewer barfed quite completely and now build is spitting "cannot find driverlib/sys_ctrl.h" when trying to make BIOS.o despite cc13xxware being in the include path list and providing that file.. I'm hoping that a fresh install might help with these incessant and bizarre toolchain failures.

It's going to take a while, the CCS installer provides zero progress indication and last time I installed, I left it overnight so I've no idea what the typical install time is. It's been going for a few hours so far with no suggestion of when it might complete.

OK, fresh CCS install working, seems to be vastly fewer crashes in my code, but still having weirdnesses..

My task said it was going to wait for 2.5 seconds but then never woke up again, so I paused execution to have a dig with the ROV.

In Task view, my task is simply gone. Previously, when I've installed Task hooks, none of the task exit hooks ever triggered when my tasks vanished.

I went to have a look at the semaphores that my tasks use for synchronisation, and found this - Why does the ROV think that bit of memory is unreadable when the Memory Browser can see it just fine?

I've set the BIOS to non-ROM code, yet when I paused, the SP was still in the 0x1000nnnn region, which so far has eluded all my attempts at finding out what's here. I had to step several times for it to pop out into viewable power saving code.

So, when stuff like this happens, does anyone have any idea how to find out why my task suddenly vanished?