Debugging syslink.ko with CCSV5.2

Hi John,

If you do not need to debug the initialization of the module, the easiest method is to use the attached shell script to generate a GEL file.

http://e2e.ti.com/cfs-file.ashx/__key/communityserver-discussions-components-files/81/7558.ccs_5F00_modsyms.sh

sh ccs_modsyms.sh  syslink ccs_mod.gel  /netshare/staging/syslink.ko 

Then, from CCS, you should load the ccs_mod.gel file

 Here are the steps:

This will bring the GEL Files view.

In the GEL files view, right click to bring up the context menu and select load GEL to load the gel file you created

 Once loaded, you will see:

Then, you can activate the script from the Scripts menu item.

This will load the module symbol and relocate the sections.

Please note that although process awareness is not supported in v5, module awareness is still supported.

The steps are:

1. You need to load vmlinux
2. Set up the OS Aware Debug Options under Tools->Generic Debugger Options
3. Enable OS Aware debug.
4. Run insmod on the target

Raymond

Raymond Pang said:

Hi John,

If you do not need to debug the initialization of the module, the easiest method is to use the attached shell script to generate a GEL file.

http://e2e.ti.com/cfs-file.ashx/__key/communityserver-discussions-components-files/81/7558.ccs_5F00_modsyms.sh

sh ccs_modsyms.sh  syslink ccs_mod.gel  /netshare/staging/syslink.ko 

Then, from CCS, you should load the ccs_mod.gel file

 

FWIW, this script didn't work for me.  (CCS5.2 on fully-updated Ubuntu v10.10, for an OMAP3530 target).  When I run the generated GEL script an additional copy of the module is added for each section parsed.

Say the module whose symbols are to be loaded is named mymod.ko.  The result of the script execution are 15 copies of mymod.ko shown in CCS's Modules window.  Each of these 15 copies contains the symbols for a given section (one copy for .ARM.exidx, another for .data, etc.).  Put another way, the symbols for a given module are not collapsed into a single instance of that module.

At first blush this might seem like a cosmetic problem.  After all, the symbols are still loaded and associated with appropriate given addresses, right?  Uh, no.  The list shown in the Modules window scrolls *constantly* and furiously in rotation.  The 16 modules shown (vmlinux + 15 instances of mymod.ko) scroll so fast that you can't select one for removal.  The only recourse to stop the full-tilt scrolling of the  Modules window is to select Remove All Symbols from the Run menu.

If I edit the script down to just the .text and .data sections, on script execution I have 3 modules shown in the Modules window (vmlinux + mymod.ko + mymod.ko) and they scroll just as furiously and just as permanently.  (Since they all fit in the window, I guess "scroll" is a bad term.  It is more like they are highlighted/select repeatedly in sequence.)  CCS5.2 is just not happy with the results of the GEL_SymbolShowSection() call.

The only way I can get symbols for my kernel module is to load the module, then Run>>Load>>Add Symbols for the module, specifying the Code and Data addresses found in the /proc filesystem.  It's a slow and painful process since those addresses change on each load of the module.

FYI.

Hi Steve,

I try it on 5.2 and our latest snapshot. I cannot reproduce what you have observed.

Can you post your gel file ?  Also, what is the Linux kernel version running on your target ?

Using Add symbol may not work as not all code and data sections are relocated by the same offset.

-Raymond

Kernel version 2.6.26, as seen below.

[code]

# cat uppermac.gel
menuitem "Linux Aware Debug"
hotmenu load_module_symbol()
{
GEL_TextOut( "Loading /lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko\n" );
GEL_SymbolAdd( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko" );
GEL_TextOut( "Relocate section .ARM.exidx to 0xbf0f3e9c\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".ARM.exidx",0xbf0f3e9c,0xbf0f3e9c);
GEL_TextOut( "Relocate section .ARM.exidx.exit.text to 0xbf0f3e10\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".ARM.exidx.exit.text",0xbf0f3e10,0xbf0f3e10);
GEL_TextOut( "Relocate section .ARM.exidx.init.text to 0xbf0f3e08\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".ARM.exidx.init.text",0xbf0f3e08,0xbf0f3e08);
GEL_TextOut( "Relocate section .bss to 0xbf0f4124\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".bss",0xbf0f4124,0xbf0f4124);
GEL_TextOut( "Relocate section .data to 0xbf0f3f6c\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".data",0xbf0f3f6c,0xbf0f3f6c);
GEL_TextOut( "Relocate section .exit.text to 0xbf0f3d58\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".exit.text",0xbf0f3d58,0xbf0f3d58);
GEL_TextOut( "Relocate section .gnu.linkonce.this_module to 0xbf0f3fdc\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".gnu.linkonce.this_module",0xbf0f3fdc,0xbf0f3fdc;
GEL_TextOut( "Relocate section .init.text to 0xbf0f8000\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".init.text",0xbf0f8000,0xbf0f8000);
GEL_TextOut( "Relocate section .note.gnu.build-id to 0xbf0f3ddc\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".note.gnu.build-id",0xbf0f3ddc,0xbf0f3ddc);
GEL_TextOut( "Relocate section .rodata to 0xbf0f3e00\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".rodata",0xbf0f3e00,0xbf0f3e00);
GEL_TextOut( "Relocate section .rodata.str1.4 to 0xbf0f3e18\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".rodata.str1.4",0xbf0f3e18,0xbf0f3e18);
GEL_TextOut( "Relocate section .strtab to 0xbf0f8ba0\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".strtab",0xbf0f8ba0,0xbf0f8ba0);
GEL_TextOut( "Relocate section .symtab to 0xbf0f8090\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".symtab",0xbf0f8090,0xbf0f8090);
GEL_TextOut( "Relocate section .text to 0xbf0f3000\n" );
GEL_SymbolShowSection( "/lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko",".text",0xbf0f3000,0xbf0f3000);
GEL_TextOut( "Loading completed.");
}

[/code]

Perhaps some process is not completing.  After I run the script CPU utilization by ccstudio goes to 100+% (this on a dual-core machine).  It drops back down after doing a "Remove All Symbols".

Steve, John,

I've taken over the investigation of this issue from Raymond. We are having trouble reproducing this it would be really helpful if you could provide

1) a screenshot of the module view in the bad state

2) collect a debug server log ( http://processors.wiki.ti.com/index.php/Troubleshooting_CCSv5#CCS_Diagnostic_Logs)

Thanks,

Mandeep

Hello, Mandeep.

0. This is the initial state of the debugger, immediately after launching the configuration.  Symbols from the running Linux kernel are loaded as part of the launch.

1. Ready to run this generated GEL script:

4705.uppermac.gel

2. Console after running the script.  No errors seen but CPU is now at 100+% utilization (on a dual-core CPU).

3. Switch back to the Modules window to see this:

4. CPU still at 100+% utilization.  That long list of modules is not now visibly moving but something is going on.  Any attempt to select one of those uppermac.ko module results in it being instantly deselected (the highlight bar flickers on then off).  I'll expand a couple of those modules just to see what's in them.  Also, there is now no content beneath the vmlinux module, though there was earlier.  CCS is really screwed up now.

5. Maybe there is actual content (functions, variables, etc.) beneath those module but it just not being shown?  No, clicking on a blank slot just gives the same select-then-instant-deselect behavior.  I work my way down the list until I get a reaction.  Now I have information in the window to the right.  The highlight bar is scrolling madly through the last 3 modules in the list, though it looks like a normal selection in this image.  Note that only the data address is shown to the right, with the code being zero.  Clicking on another of the modules in the list gets me a view with a code address and data being zero.

6. CPU usage by ccstudio is now at 116% and shows no sign of stopping.  After doing a "Remove All Symbols" the Modules window is empty and the CPU use has fallen to 40%.

I am attaching the log output you asked for..  This 9.3MB ZIP file expands to a 131MB log file:

8510.ccs-stdout-log.zip

Please let me know if there is any other info I can provide.

Steve,

Thanks for the quick reply. I'll take some time to analyze the log and see if I can figure something out. You host is a linux machine right ?

Thx,

Mandeep

Steve,

Can you try running the script after closing both the disassembly and registers view and see it has any effect?

Thx,

Mandeep

Yes, that's right.  Fully-updated x86 Ubuntu v10.10.

CCSv5.2 was installed via the downloaded CCS5.2.0.00069_linux.tar.gz, and no updates have been applied to CCS since installation.

>Can you try running the script after closing both the disassembly and registers view and see it has any effect?

No different in behavior seen.

Tried again after also removing all breakpoints from the symbols to be loaded.  Still no difference.

Steve,

One more thing. Can you try running script without loading linux image first.

Mandeep

Also,

Is this " /lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko" or any part of this path a sym link? I can reproduce the issue that way.

Mandeep

> Is this " /lib/modules/2.6.36-3256/kernel/drivers/xmax/uppermac.ko" or any part of this path a sym link? I can reproduce the issue that way.

On the host system, yes, there a symlink.  There is no symlink involved on the target.

The actual kernel module is buried deep in a source tree on my development machine and copied to the target at that actual location seen above.  I created a symlink on my host machine so that the module would be seen at the "same" location on both host and target systems.  That "xmax" directory seen above is the symlink to a directory that contains both the binary kernel module and its source files.

This was done for the benefit of the GEL script which is generated on the target and includes the full path to the kernel module.  When copied to the host for execution in CSS that full path is again used to read the module.  Hence the symlink.

> One more thing. Can you try running script without loading linux image first.

No difference in behavior.

I start with an empty Modules window, run the script and again see 15 copies of the same module, with the same flaky behavior (gaps in the list, rolling highlight, pegged CPU, etc.).

Hi Steve,

For the shell script, you are supposed to specify the host path of the module.ko.

i,e, you run it as

sh ccs_modsyms.sh  uppermac  ccs_mod.gel  <the location of uppermac.ko in your host build pc> 

 You do not have to use symbolic link to match the host and target path names.

-Raymond

> sh ccs_modsyms.sh  uppermac  ccs_mod.gel  <the location of uppermac.ko in your host build pc>

The good news: it works perfectly.

The bad news: apparently I'm an idiot. 

Thanks.

Steve. You are being too hard on yourself. There is clearly an CCS issue here as well. I've created a CQ to track this. If you PM me your email address I can add you to the customer stakeholder so you get updated when a appropriate fix is available.

Thanks,
Mandeep Deol