CCS/AM3352: Debug trace issue

Hi Chris,

can you please provide exact steps to re-create the issue including...

• A PATCH showing the exact changes you made to the Kernel
• The version of CCS you use
• The exact steps you followed until you tried to execute Enable_ETB_CortexA8_Trace
• What is the behavior you expect

Regards, Andreas

Hello Andreas,

Thank you for responding.  I will need to respond to your requests in multiple messages, and I also have some additional debugging information.

The problem starting the trace is intermittent for me, and on some days it works with no issues.

The behavior that I expect is to be able to have Tools / Hardware Trace Analyzer / PC Trace start the trace on the Cortex A8.  When it is working, I am able to view the Program Trace as shown in this screen shot.

Thanks,
Chris Norris

Hello Andreas,

Here is the patch I used to keep the JTAG clocks active.  This forum editor won't let me attach files, so I have pasted it inline here.

This patch keeps the debugSS clock alive, it clocks the JTAG macro and enables
access to the SoC via JTAG after the kernel booted.

Signed-off-by: Lothar Felten <lothar.felten@gmail.com>
---
diff -Naur linux-orig/arch/arm/mach-omap2/omap_hwmod_33xx_data.c linux-52c4aa7cdb93d61f8008f380135beaf7b8fa6593/arch/arm/mach-omap2/omap_hwmod_33xx_data.c
--- linux-orig/arch/arm/mach-omap2/omap_hwmod_33xx_data.c	2015-10-02 17:30:56.000000000 +0200
+++ linux-52c4aa7cdb93d61f8008f380135beaf7b8fa6593/arch/arm/mach-omap2/omap_hwmod_33xx_data.c	2016-08-15 11:28:55.017617612 +0200
@@ -208,6 +208,7 @@
 	.name		= "debugss",
 	.class		= &am33xx_debugss_hwmod_class,
 	.clkdm_name	= "l3_aon_clkdm",
+	.flags		= (HWMOD_INIT_NO_IDLE|HWMOD_INIT_NO_RESET), /* keep debugSS clock alive for JTAG */
 	.main_clk	= "trace_clk_div_ck",
 	.prcm		= {
 		.omap4	= {

Chris,
I just experimented with the current CCSv9.3 and a XDS560v2 debugger and also had issues getting ETB to work properly. Sometimes it worked, and sometimes it didn't, resulting in errors while executing Enable_ETB_CortexA8_Trace().

Let me follow up on this internally and report back.

Also what CCS version are you using?

Regards, Andreas

Hi Andreas,

I am on Code Composer 9.3.0.00012.  Thanks for looking into this.

Here are some screens showing other symptoms.  When the trace enable is working, I can enter the addresses for CM_PER_L4FW_CLKSTCTRL and FW_DBGSS_MRM_PERM_REGION0_LOW into the Expressions window for CS_DAP_DebugSS core.  CM_PER_L4FW_CLKSTCTRL has the value of 0x102 which I have set in the Registers window on CortexA8 core, and FW_DBGSS_MRM_PERM_REGION0_LOW has the value of 0xffffffff that was set in enableDebugSSFirewall() of the AM335x_trace_dapdebugss.gel.

When the script is not working, I am no longer able to access these same registers from the CS_DAP_DebugSS core.  This screen shot shows the register access failing.

It seems like the trace enable script works the first time I've ever connected to my Sitara board with Linux running.  It seems like the script (and register access thru Expressions window) always fails the second and subsequent attempts when Linux is running.  Power cycling both the XDS110 and the board doesn't seem to clear the problem.

Interestingly, if I reboot and pause with u-boot running and before the kernel is running, I am able to get register access from the CS_DAP_DebugSS core again.  I can enable Program Trace while u-boot is running then let it finish the boot into kernel and the trace keeps running.  This is my workaround for now to use the trace.

Since the register access from CS_DAP_DebugSS works under u-boot but not from the kernel, it seems like there is something else needed to regain the register access.

I noticed in my own reading and other posts that CM_PER_L4FW_CLKSTCTRL and many details about the interconnect are not documented in this chip's TRM, so not sure where to go next with my debugging.

Will keep posting other observations to this topic, and appreciate any insight from your team.

Cheers,
Chris Norris

Hi Chris,

Chris Norris said:

It seems like the script (and register access thru Expressions window) always fails the second and subsequent attempts when Linux is running.  Power cycling both the XDS110 and the board doesn't seem to clear the problem.

In the few tests I did I noticed a similar pattern. For me part of the "restart" sequence included re-starting CCS, and I don't think I had seen failures after CCS was re-started, but then, I've only done a few test runs. Let me know if you can correlate this with anything you do.

Chris Norris said:

Since the register access from CS_DAP_DebugSS works under u-boot but not from the kernel, it seems like there is something else needed to regain the register access.

Agreed, I also have not seen issues with U-Boot during the runs I did. The Kernel was always where the problem was. Also my setup is different from yours - I'm using CCS v9.3 under Linux (what are you using??) with a very different debugger (XDS560). So this would point at something going wrong inside the chip.

Chris Norris said:

Will keep posting other observations to this topic, and appreciate any insight from your team.

Yes. I've reached out to one of our experts, will see what they recommend as next steps.

Thanks and Regards,
Andreas

Hi Andreas,

My CCS is running on Windows.

For source debugging using CCS on Windows, I built on Ubuntu from VM or WSL then copied the kernel sources over to my workspace.

Thx,
Chris

Chris,

talking to one of our experts he pointed out there was a similar issue in the past on a different device, and it turned out to be MMU related. Things got re-mapped in the Kernel and the debugger lost the ability to access certain debugSS registers. Let me experiment with this later this week when I'm back in the office to see if a similar concern applies here, and check if there is a way to work around it if possible.

Regards, Andreas

Chris,

just a quick update. I've since gotten a chance doing some further investigation and talked to some more people here and tried a few things in the Kernel for example disabling the PMU framework (CONFIG_ARM_PMU) and manually forcing the clocks/power controls for the debugSS to enabled state by making direct low-level hwmod API calls but have not been successful in making the Kernel ETB connectivity working reliably. I have some more things to try and I'll work on this some more next week as time permits. Will keep you posted.

Regards, Andreas

Hi Andreas,

Thank you for the updates!

Still interested in starting the trace from Code Composer when Linux is up.

On another related path, I'm looking at OpenCSD and seeing if I can get any of the /sys/bus/coresight stuff working.  Haven't gotten very far there.

Chris N.

Hi Chris,

I've not had bandwidth to look into this further but here is one of the things I had planned in case you want to pitch in some. Basically, we know ETB connectivity works well in U-Boot. However once in Kernel we can't invoke it anymore, if not invoked previously. So something during the startup and initialization of the Kernel is causing an issue here, and if we can identify the "what" we will have a better idea about how to fix or workaround it. So far the theory.

So specifically I would trap program execution at various stages in the following functions:

init/main.c:start_kernel()

and

linux/arch/arm/kernel/setup.c:setup_arch()

Then it may also make sense to look at the following arch-specific function referenced by below structure:

#ifdef CONFIG_SOC_AM33XX
static const char *const am33xx_boards_compat[] __initconst = {
        "ti,am33xx",
        NULL,
};

DT_MACHINE_START(AM33XX_DT, "Generic AM33XX (Flattened Device Tree)")
        .reserve        = omap_reserve,
        .map_io         = am33xx_map_io,
        .init_early     = am33xx_init_early,
        .init_machine   = omap_generic_init,
        .init_late      = am33xx_init_late,
        .init_time      = omap3_gptimer_timer_init,
        .dt_compat      = am33xx_boards_compat,
        .restart        = am33xx_restart,
MACHINE_END
#endif

You could use simple while(1){} loops to trap execution or debugger breakpoints at various points during startup, and then keep trying ETB connectivity, effectively bi-secting where the issue is.. As you can see, a little labor some, but I think a good way to narrow things down.

Regards, Andreas

Hi Chris,

did you already get a chance to bisect this a bit to see what might be causing the issue?

Thanks and Regards,
Andreas