CCS/AM6548: R5F debug problems

Dominic,

While Garrett is helping you with the NIMU NDK setup on R5F on the other thread, we can limit the discussion here to CCS/Trace related issues. 

I have looped in CCS/emulation experts to comment on the Trace features that they are currently supporting for this device and for CCS IDE based issues. There is currently no plan to support CToolsLib for this device. The support for that is limited to AM57x, Keystone I and Keystone II multicore ARM+DSP devices. Is the CtoolsLib software being used as part of any of your existing designs.

I would like to understand the usecase and provide our development team required details so approrpiate priority can be assigned to availability of this tool for AM654x device.

Regards,

Rahul

Hello Rahul,

thanks for your quick feedback.

Yeah, it would be great to hear from your emulation experts. We're currently using an XDS200 probe. The trace port isn't an option since our custom hardware has severe size limitations. We're limited to what fits into the ETB, but for now I'm really only interested in "what was the last instruction before the processor died", and that should easily fit into the 16 KB CT-TRB of the MCU debug cell (?).

> "There is currently no plan to support CToolsLib for this device"

This is kind of unexpected, since the TRM specifically mentions the CToolsLib in chapter 13.7 Application Support

I haven't used CToolsLib before, but since tracing seems to be (possibly unnecessarily) limited in CCS I tried to find a way to access the trace functionality myself. Our application is running on the R5F core, and that core "dies", i.e. it becomes inaccessible to the debug tools. But the SOC is still well, i.e. I can connect to the DMSC for example, and use that to view memory. My hope was being able to setup R5F core trace using either CToolsLib or CCS, and then since I can't access the trace data with CCS find some other way to extract the trace data.

Regards,

Dominic

Dominic, 

A few clarifications:

Dominic Rath said:

We're limited to what fits into the ETB, but for now I'm really only interested in "what was the last instruction before the processor died", and that should easily fit into the 16 KB CT-TRB of the MCU debug cell (?).

Any JTAG Debug Probe can capture data via the ETB. However, the most critical part is to trap the core execution immediately after the crash, in order to avoid overwriting the prior history stored on the ETB. If the issue is causing an Exception on the core, you can always use the Exception traps built into CCS - something mentioned at this thread. Otherwise, if the issue is in the middle of your code you will have to go through an iterative process of trial and error to isolate the point where the problem is happening - you can use Watchpoints to inspect for invalid read/write operations on suspicious arrays and/or memory mapped registers. 

Dominic Rath said:

> "There is currently no plan to support CToolsLib for this device"

This is kind of unexpected, since the TRM specifically mentions the CToolsLib in chapter 13.7 Application Support

I would have to double check this statement, but CToolsLib is a software package that hasn't seen an update in a long time. There is a chance this was a Copy/Paste error but I will confirm. 

Regards,

Rafael

Hello Rafael,

I'm aware of these basic debugging techniques, but our problem is that the R5F randomly locks up in a way that makes it look like it's not powered up, see the screenshots that I posted. It is NOT taking an exception, at least none of the breakpoints I set ever triggered. Since we've seen this behaviour where the R5F becomes unavailable for debugging in three totally different test cases, one being TI's own NIMU example, and since this is always random (i.e. most likely depending on some very specific timing conditions) I started looking into the tracing capabilities of the AM65x.

The problem with the "core trace" available in CCS is that it wont download the collected trace data, since I can't halt the core anymore.

The latest issue is becoming somewhat urgent, so I started looking into debug peripherals located at 0x4c3c... and 0x4c3d.... It would have been really helpful if TI documented the addresses and exact version of debug peripherals such as the ETM and ETB, but I managed to locate these myself.

I'm using the M3_DMSC to access the SoC even after the R5F locked up. From the ETB (CT-TBR) registers I can tell that the R5F is NOT executing any code, so halting the core/trace shouldn't be an issue in my case. I managed to manually dump the ETB content and used bin2tdf and td to decode the trace, but I'm not sure yet if this is really the END of what the processor executed, or if there's some data still in the ETM/ETB pipeline. I finally located the ETM yesterday at address 0x4c3d01c000, and I hope to be able to stop the trace and flush all data that way.

Regarding CToolsLib:

It would be great if you could check what TI's plans are regarding this. The TRM describes some very interesting scenarios where trace data is exported using DMA or using the USB peripherals, but I suppose without the CToolsLib this is going to be almost impossible for the average user.

It would be great if you could answer my specific questions regarding trace:

• Is there some documentation on the use of tracing with the AM65x, CCS 9 and an XDS200 debug probe?

• What is this "FIFO overflow" that we're getting when trying to use "traffic profiling" to trace MCU->Main domain transactions, and how can we avoid it?

• Is there a way to access the R5F core trace data in case the R5F becomes inaccessible for debugging? (I'm working on getting there manually, but this still seems to be an unnecessary shortcoming of CCS)

• Any documentation on the CT-TBR would be great, since this appears to be something TI specific that merely implements the ARM ETB architecture?

• Any documentation on this part of the SoC would be nice, too: MCU_ARMSS0_SOC_REGION_SHGY39R4XFIGZ5JTB 0x4C3D000000 0x4C3D0FFFFF 1 MB

Regards,

Dominic

Dominic Rath said:

I managed to manually dump the ETB content and used bin2tdf and td to decode the trace, but I'm not sure yet if this is really the END of what the processor executed, [...] I finally located the ETM yesterday at address 0x4c3d01c000, and I hope to be able to stop the trace and flush all data that way.

After first halting the ETM and then halting the ETB I was able to see that the processor really executed a few more instructions. The last instruction traced is a PC relative load of some constant (code is in DDR RAM in our case). The next instruction would read the VIM.ACTIRQ register.

I then tried accessing the R5F core via the R5 debug registers at 0x4c3d010000, and after setting the "Cancel memory request" bit in the DBGDSCR the processor entered debug state in CCS. The R5F is then "usable" from within the debugger, but it dies as soon as I try to read the VIM registers. We ran a few more tests and I verified that I can read MCU SRAM at 0x41c00000 and MSMC SRAM at 0x70000000, but I can't read the VIM at 0x40f80000 or the RAT at 0x40f90000. I'm assuing that these peripherals live on a different interface (R5F private peripherals?) and not on the "main interconnect", since I can't read these registers from other processors (i.e. the DMSC). Accessing the VIM using the R5F works fine before the system died, so it looks like something happens that causes further accesses to the VIM (and possibly other peripherals) to freeze the processor.

It would be nice if anyone from TI could comment on what we found out so far. Answers on my previous questions would still be welcome.

Regards,

Dominic

... I just verified that the first problem that we encountered which caused this kind of processor freeze fails in exactly the same place, i.e. within           ti_sysbios_family_arm_v7r_keystone3_Hwi_dispatchIRQC__I trying to read from ACTIRQ.

Like with our current problem this only happens when the application gets loaded via SBL, but not when loaded via the debugger (see the other thread linked in my first post).

Dominic,

Dominic Rath said:

It would be great if you could check what TI's plans are regarding this.

The dev team confirmed that cToolsLib will be ported to this device, but the work hasn't started yet. At the time we are tentatively planning to intercept SDK releases later this year. 

Dominic Rath said:

Is there some documentation on the use of tracing with the AM65x, CCS 9 and an XDS200 debug probe?

I am not entirely sure what you mean - basically the Trace information is gathered directly from the ETB via plain JTAG commands, as XDS200 does not support any other method.

Dominic Rath said:

What is this "FIFO overflow" that we're getting when trying to use "traffic profiling" to trace MCU->Main domain transactions, and how can we avoid it?

Usually overflows are caused by excessive traffic between the data collectors and the sink of this data (trace buffer in this case) - there are limits to the data throughput that are heavily dependent on the device and its configuration, but I couldn't find a way to configure parameters relevant to this scenario. I will keep looking. 

Dominic Rath said:

Is there a way to access the R5F core trace data in case the R5F becomes inaccessible for debugging? (I'm working on getting there manually, but this still seems to be an unnecessary shortcoming of CCS)

From your posts it seems you are already doing this, but not from the IDE.The reason the IDE does not have this function natively may be historical as to how the tool was implemented (focused on the legacy DSPs of the time) and not with a direct memory mapped port such as the DAP of the ARM subsystems. I completely agree this is a shorcoming of the Trace interface of CCS, but a definitive fix may take time to implement.

Dominic Rath said:

Any documentation on the CT-TBR would be great, since this appears to be something TI specific that merely implements the ARM ETB architecture?

I would have to get back to you on that. 

Dominic Rath said:

Any documentation on this part of the SoC would be nice, too: MCU_ARMSS0_SOC_REGION_SHGY39R4XFIGZ5JTB 0x4C3D000000 0x4C3D0FFFFF 1 MB

I am not sure what this memory region refers to. I will defer that to the Processor experts. 

Regards,

Rafael

Dominic,

The data collection procedure looks ok. 

Dominic Rath said:

but I can't read the VIM at 0x40f80000 or the RAT at 0x40f90000. I'm assuing that these peripherals live on a different interface (R5F private peripherals?) and not on the "main interconnect",

Looking at page 602 of the device's Technical Reference Manual (SPRUID7) I don't see these memory addresses listed there, thus can't be accessed from another core - but most probably from the DAP. I then found section 6.3.3.3.2 that says in no other terms that these registers are private to the R5F. 

Unfortunately I am not versed enough on the device details to provide a thorough explanation about these registers and the possible root causes a data access could cause the R5F core to shut down. I will have to rely on the Processor experts for that. 

Regards,

Rafael

P.S. I accidentally hit the "TI thinks answered" button. Sorry. 

Hello Rafael,

thank you for your replies.

desouza said:

The dev team confirmed that cToolsLib will be ported to this device, but the work hasn't started yet. At the time we are tentatively planning to intercept SDK releases later this year. 

Thank you. Not the answer I was hoping for, but it's good to hear that this is on the roadmap.

desouza said:

I am not entirely sure what you mean - basically the Trace information is gathered directly from the ETB via plain JTAG commands, as XDS200 does not support any other method.

I was thinking of some kind of examples on how to use all the trace features supported by the AM65x, after all it offers a lot more than just the core trace. I guess I was able to figure out most of the stuff for myself, but I was rather lucky that I found the memory transaction logging in the memory profiling view for example. Could this be a bug in CCSv9? Without more documentation these very useful features are most likely going to go unnoticed.

I also had trouble using bin2tdf and td. I ended up saving a tdf file using CCS, then cutting the header and footer from that file and adding my own 16 KB trace data in between. This could of course be something that is only worth documenting once there is a CToolsLib for the AM65x.

desouza said:

Usually overflows are caused by excessive traffic between the data collectors and the sink of this data (trace buffer in this case) - there are limits to the data throughput that are heavily dependent on the device and its configuration, but I couldn't find a way to configure parameters relevant to this scenario. I will keep looking. 

It would be nice to know what these limits are. Without some example of what should be possible with this functionality it is hard to tell if I'm just not using it the right way, or if I'm really hitting a hardware limitation. Could it be that the "memory profiling" that I'm using is causing unnecessary overhead? With CCS v8 there was a separate memory transaction logging view.

desouza said:

Looking at page 602 of the device's Technical Reference Manual (SPRUID7) I don't see these memory addresses listed there, thus can't be accessed from another core - but most probably from the DAP. I then found section 6.3.3.3.2 that says in no other terms that these registers are private to the R5F. 

Unfortunately I am not versed enough on the device details to provide a thorough explanation about these registers and the possible root causes a data access could cause the R5F core to shut down. I will have to rely on the Processor experts for that. 

Yeah, now that I'm starting to have a clue of what is going wrong inside the processor it would be great if the processor experts could comment on this issue.

Since you seem to understand a lot about the debug architecture: Do you know how the processor-specific memory view works, i.e. through which parts of the debug system the memory requests are routed? If I select the R5 core and then enter an address in the memory browser I get different data then from the M3_DMSC, so it looks like these views all use the respective core to access memory. Is there a MEM-AP in the AM65x? Figure 13-1 . SoC Debug Framework Overview to me looks like that. As far as I understand it that would allow me to access the system interconnect without relying on one of the cores, providing a lot more flexibility, and would remove the need to reprogram the DMSC RAT in order to see the peripherals I need for debugging.

Regarding the access path to VIM/RAT:

Thanks for mentioning chapter 6.3.3.3.2. It clearly says that the VIM and RAT are private to the R5F and that these live on the 32bit VBUSP interface, so it's obvious that I can't access them using other cores. But I can access these registers using the R5F before the processor freezes, so whatever is causing these problems seems to lock up this bus, or at least some part of it. The last thing I checked today at the office was that the R5F in the AM6548 has a AHB peripheral interface at 0x40000000 with a size of b01111 - I believe that means it is 16 MB. This would mean that the VIM and RAT live on the same interface as all the other MCU peripherals. I'll check tomorrow if these are affected by the same problem. So far I've only checked addresses outside the peripheral interface.

Regards,

Dominic

Dominic Rath said:

The last thing I checked today at the office was that the R5F in the AM6548 has a AHB peripheral interface at 0x40000000 with a size of b01111 - I believe that means it is 16 MB. This would mean that the VIM and RAT live on the same interface as all the other MCU peripherals. I'll check tomorrow if these are affected by the same problem. So far I've only checked addresses outside the peripheral interface.

Once the R5F is stuck it can't access any memory in the 0x40000000-0x40ffffff range, i.e. on the AHB peripheral interface. Even in this case it is still possible to access the other peripherals (except VIM and RAT of course) that live on this bus from other masters, for example I can read from the ADC controller using the M3.

Dominic,

Dominic Rath said:

Since you seem to understand a lot about the debug architecture: Do you know how the processor-specific memory view works, i.e. through which parts of the debug system the memory requests are routed? If I select the R5 core and then enter an address in the memory browser I get different data then from the M3_DMSC, so it looks like these views all use the respective core to access memory. Is there a MEM-AP in the AM65x? Figure 13-1 . SoC Debug Framework Overview to me looks like that. As far as I understand it that would allow me to access the system interconnect without relying on one of the cores, providing a lot more flexibility, and would remove the need to reprogram the DMSC RAT in order to see the peripherals I need for debugging.

Please apologize for the delay; unfortunately the intrinsic details vary from device to device and at that level I am not knowledgeable enough about the complete routing of the debug signals and their interaction with the internal buses on the AM65x device. I would ask you to check with the device experts on the Sitara forum. 

Regards,

Rafael