AM263P4-Q1: [ERROR] Issue with debugger when launching from CCS - Trouble Setting Breakpoint with the Action "Process CIO" at 0x70215094

Hello Seb,

Sebastien Chapeau said:

Could you please point me out into the right direction ?

The error mentioned has several potential root causes. The error is often related to the CIO buffer on the target. For more details on this buffer, please see:

https://software-dl.ti.com/codegen/docs/tiarmclang/compiler_tools_user_guide/compiler_manual/using_runtime_support/the-c-i-o-functions-stdz0536526.html?highlight=cio#device-driver-level-i-o-functions

When the debugger has issues reading/writing to the buffer, or the contents of the buffer get corrupted, then the error you see can happen. Check the C I/O buffer. Make sure the .cio section is properly allocated in the correct memory location. Make sure that the debugger has read/write access to that range (debugger memory map). Make sure that memory range is not tampered with by the running application or otherwise.

You mentioned:

Sebastien Chapeau said:

The problem arised when updating the application memory mapping to the representative one we intend to use for production.

This likely impacted where the .cio section is placed. See where it was originally and where it is now. 

I will bring this thread to the attention of the device experts for further suggestions.

Thanks

ki

Hi

Thanks for your answer.

I had a look into the projects searching for the .cio section.

In the linker command file, there is nothing related. Nor in the makefile either.

When looking into the map files, the only thing I could relate to CIO is _CIOBUF_ as part of lisysbm library which is effectively linked to the project.

I can see that for all 3 cores a size of 288 bytes is allocated.

The place it is allocated in falls into the OCM execution range for each cores.

See screen shot below:

Core 0-1

Core 2

Core 3

I double check that MPU setup for all core for OCM is same with the following (which is very crude for now):

The size is bigger than the physical OCM on the AM263P4 device however, with the linker setup I'm not expecting this to cause any issue.

I also read the content referred by link in the post. It does not seem to really apply to the AM263P4 or at least it is abstracted by linking the library.

Will dig further ...

Could you let me know if I'm looking into the right place or if anything seems odd ?

Thanks

Regards

Seb

Hi

Digging further, I can't find the beginning of an answer.

On top of the steps I've done above I made sure I am using the the AM263Px_ZCZ_C debug config for each core + system project.

But no change to the behaviour.

An observation though ...

Trying to play with the internal memory with the debugger after halting core 3, I can observe that I can't overwrite the memory content at address 0x70215094 onwards.

The possibility to overwrite memory with the debugger seems to stop at 0x70214A00. Before that address, content can be overwritten albeit not always.

Starting 0x70214A00, can't seem to be able to change any value with the debugger.

Not sure why but it seems to indicate limitation in what address the debugger can access into the OCM but It does not seem to relate to a 2MB boundary from an AM263x device.

Note whilst writing this post I was trying several other addresses. I can overwrite some around 0x70215094 but it is very much random in what can be overwritten.

Definitively can't alter this area:

Don't know if those observations help but I'm really stuck with this.

Regards

Seb

Hi Seb,

I have started looking into this issue. I am trying to reproduce the same on my end. If possible, can you please share your CCS project as a zip? If you cannot do that due to security concern, can you please share just the example.syscfg of your project and some information about what each core of your project does. Are you running FreeRTOS on all 4 cores or it is baremetal? Any SDK example that you picked and modified to run the above tests? 

Regards,
Shaunak

Hi

The project is purely bare metal on all cores.

It does not do anything fancy. Just flashing LEDs but on the dev board.

However, it is intended to provide a representative structure about how things will run with core 0-1 in lockstep and core 2 & 3 working individually as well as memory partitioning.

See attached the project.

DevOnLaunchpad.zip

Note this morning I was trying to reduce the size of the memory used on core 0-1 to go down to 684kBytes. 

I got no problem at launch with the debugger creating the breakpoints and halting on main for each core.

This seems to indicate an issue with the debugger and the memory range it can access.

Note at 684kBytes, the application wasn't running correctly which I will look into. But at least it seems to be a different problem.

Thanks a lot for looking into this.

Regards

Seb

Hi

Update my side.

If all running sections for each cores are set at 684kBytes. Then debugger can write the breakpoint and halt all cores at main.

Though the behaviour when debugging is very inconsistent.

I can step through core 3 code but it will disconnected when letting the code run freely.

The application works for sure. When launching the debug session and disconnecting the debugger when all core are halted at main, this is resulting in all core starting OK and the application running fine.

I can't do that if core 0 is set at 1.3M (2*684kBytes).

It would really point to a tool problem...

I would appreciate help as it is blocking.

Thanks

Regards

Seb

Hi Seb,

I tried using your ZIP and building it in CCS. It is giving me a lot of build errors on every core. Attaching the screenshot below:

The address range having issue seems to be OCRAM Bank4 but the syscfg MPU settings show it has a read access. Not sure why it is crashing. Can you please help me by sharing a ZIP i can use to debug the issue.

Regards,
Shaunak

Hi 

For some reason when zipping, the link to the content within common multicore folder was lost by CCS configuration.

In each core project, you should have this line into the include options:

..\..\Multicore_LedBlinkingApp_am263px-lp_system_nortos\Common_Multicore

Please add that line. I also fixed the project (ie readded that line and checked the way the folders are linked) checked it compiles OK and I get the issue stated above.

See attached a new zip file.

Hopefully the issue does not reproduce.

If it does, maybe I'm doing something wrong with the tool.

Thanks for your help

Regards

Seb

0511.DevOnLaunchpad.zip

Hi Seb,

Thanks for sharing a new zip, appreciate your patience.

I tried running with the new zip file, I was able to build the project.

Sharing my observations below:

1. On building and loading the binaries on respective cores, I got the following logs in the CCS terminal and the program works as expected without crashing.

I see LEDs flashing on the AM263Px-LP in a sequence as expected.

2. My content for the CIO_BUFFER

When I halt the R5F core for core3, I was able to overwrite memory via the memory browser post the address "0x70215094"

Please help me understand if I missed out some steps to reproduce the issue. Can you share exactly how the memory was increased for the non-working case (I assume the ZIP attached is for working case, since I see no issues with the above zip) ? I see that the OCRAM section for Core-0 is set to 0x156000.

Regards,
Shaunak

Hi

OK. You have done the right things. The zip attached was the non working case.

You should see a size of 0x156000 for core 0 and an OCM size of 0xAB000 for core 2 & 3.

OCM memory allocation for all 3 cores are contiguous from 0x70000000.

With this project, my side, I get either of the following:

After launch core 0 et 2 are halted automatically - Core 3 stays running. Halting Core 3 manually I get :

In that case I can overwrite CIO @ 0x70215094 / set to 0 here:

Second behaviour after launch core 0 et 2 are halted automatically - Core 3 stays running. Halting Core 3 manually I get :

In that case I cannot write CIO  @ 0x70215094 - Tried to set it to 0 but value does not change.

Invariably Core3 does not work correctly and I get either one or the other behaviour described above.

That's quite problematic if you cannot reproduce the issue. What can I check that could be different between our setup ?

The launch pad I work with is an LP-AM263P / SL# 08241710217

I'm using CCS Theia 1.5.1.3 with the following products:

Any difference your side ?

We need to get to the bottom of this one as it is stopping me to work with the intended memory partitioning.

Thanks in advance for your help.

Regards

Seb

Hi Seb,

Sebastien Chapeau said:

Any difference your side ?

I'm using the same  SDK version and AM263Px-LP as you. The only difference I notice is that you're using CCS Theia and I'm using CCS studio IDE (https://www.ti.com/tool/CCSTUDIO). I'm not really the tools expert to know if this could be the problem.

Sebastien Chapeau said:

With this project, my side, I get either of the following:

After launch core 0 et 2 are halted automatically - Core 3 stays running. Halting Core 3 manually I get :

I observed the quoted issue as well. What I do is, power cycle the device, connect to all the cores, and issue a CPU reset via CCS and then load my .out files to the respective cores.

1. In my case, after CPU reset -> Loading binaries to the cores, when the program is halted at main (all 3 R5F cores), I can overwrite the CIO buffer

2. If I let the application run for some time, then halt only the core-3, i'm still able to overwrtite the CIO buffer.

3. If I halt all the cores after letting them run for some time, even then i'm able to overwrite the CIO buffer.

I also tried playing around with addresses in the memory browser, I was able to change the values even after the 0x70215094 address.

This might sound like a vague request, but can you once try using CCS Studio (https://www.ti.com/tool/download/CCSTUDIO/12.8.1)? That's the only difference I could find between our setups.

Regards,
Shaunak

Hi Shaunak

I will try tomorrow and let you know.

CCS as we know is normally meant to be deprecated and replaced by CCS THEIA.

What is the plan currently as to when the replacement will be affective with the same level of maturity ?

Thanks

Seb

Hi Seb,

Thanks for being patient, I know it can be frustrating facing such issues.

Sebastien Chapeau said:

CCS as we know is normally meant to be deprecated and replaced by CCS THEIA.

Yes that is the long term plan.

Sebastien Chapeau said:

What is the plan currently as to when the replacement will be affective with the same level of maturity ?

This will come with the MCU_PLUS_SDK release 10.02, targeted around April 2025

Regards,
Shaunak

Hi

Just tried CCS 12.8.1 on the same "non" working project I sent you.

The first time, It ran straight away. No issues in launching the debug session. All cores halt at main.

The program executes ok. Seems I can step through the code as required and alter the CIO location

BUT it only worked once!

Since I'm back with the same problem.

I've tried many time but end always up in the same situation with the the 3rd core in a running state into an exception.

I tried to follow your process:

- Connect to core 3

- Do a CPU reset

- Go to Run / Load / Load Program and try to load the .out for Core 3

==> It lead to the following error:

I'm not sure if I'm doing something wrong there.

Am I following the same process as you ? It seems you can get away with this method but my side, is it stubbornly doing the same thing on core 3.

As you can reproduce partly the behaviour, would you have an answer at what causes the problem ? 

I think what I'm doing with the memory partition is correct as I could (and you could too) see the code running ok.

Also, in CCS 12, I now have the following error messages caused by the post build step which I'm unsure about:

C:/ti/ccs1281/ccs/utils/bin/gmake -C C:\TiTest\DevOnLaunchpad\Multicore_LedBlinkingApp_am263px-lp_system_nortos -f makefile_system_ccs_bootimage_gen OUTNAME=Multicore_LedBlinkingApp_am263px-lp_system_nortos PROFILE=Debug MCU_PLUS_SDK_PATH=C:/ti/mcu_plus_sdk_am263px_10_00_00_35 CG_TOOL_ROOT=C:/ti/ti-cgt-armllvm_4.0.0.LTS CCS_INSTALL_DIR=C:\ti\ccs1281\ccs CCS_IDE_MODE=desktop
gmake[1]: Entering directory 'C:/TiTest/DevOnLaunchpad/Multicore_LedBlinkingApp_am263px-lp_system_nortos'
Boot multi-core image: Debug/Multicore_LedBlinkingApp_am263px-lp_system_nortos.appimage ...
C:\ti\ccs1281\ccs/tools/node/node C:/ti/mcu_plus_sdk_am263px_10_00_00_35/tools/boot/multicoreImageGen/multicoreImageGen.js --devID 55 --out Debug/Multicore_LedBlinkingApp_am263px-lp_system_nortos.appimage ../MultiCore_LedBlinkingApp_am263px-lp_r5fss0-0_nortos_ti-arm-clang/Debug/MultiCore_LedBlinkingApp_am263px-lp_r5fss0-0_nortos_ti-arm-clang.rprc@0 ../MultiCore_LedBlinkingApp_am263px-lp_r5fss1-0_nortos_ti-arm-clang/Debug/MultiCore_LedBlinkingApp_am263px-lp_r5fss1-0_nortos_ti-arm-clang.rprc@2 ../MultiCore_LedBlinkingApp_am263px-lp_r5fss1-1_nortos_ti-arm-clang/Debug/MultiCore_LedBlinkingApp_am263px-lp_r5fss1-1_nortos_ti-arm-clang.rprc@3 >> Debug/temp_stdout_Debug.txt
C:\ti\ccs1281\ccs/tools/node/node C:/ti/mcu_plus_sdk_am263px_10_00_00_35/tools/boot/multicoreImageGen/multicoreImageGen.js --devID 55 --out Debug/Multicore_LedBlinkingApp_am263px-lp_system_nortos.appimage_xip ../MultiCore_LedBlinkingApp_am263px-lp_r5fss0-0_nortos_ti-arm-clang/Debug/MultiCore_LedBlinkingApp_am263px-lp_r5fss0-0_nortos_ti-arm-clang.rprc_xip@0 ../MultiCore_LedBlinkingApp_am263px-lp_r5fss1-0_nortos_ti-arm-clang/Debug/MultiCore_LedBlinkingApp_am263px-lp_r5fss1-0_nortos_ti-arm-clang.rprc_xip@2 ../MultiCore_LedBlinkingApp_am263px-lp_r5fss1-1_nortos_ti-arm-clang/Debug/MultiCore_LedBlinkingApp_am263px-lp_r5fss1-1_nortos_ti-arm-clang.rprc_xip@3 >> Debug/temp_stdout_Debug.txt
Boot multi-core ELF image: Debug/Multicore_LedBlinkingApp_am263px-lp_system_nortos.mcelf ...
python C:/ti/mcu_plus_sdk_am263px_10_00_00_35/tools/boot/multicore-elf/genimage.py --core-img=0:../MultiCore_LedBlinkingApp_am263px-lp_r5fss0-0_nortos_ti-arm-clang/Debug/MultiCore_LedBlinkingApp_am263px-lp_r5fss0-0_nortos_ti-arm-clang.out --core-img=2:../MultiCore_LedBlinkingApp_am263px-lp_r5fss1-0_nortos_ti-arm-clang/Debug/MultiCore_LedBlinkingApp_am263px-lp_r5fss1-0_nortos_ti-arm-clang.out --core-img=3:../MultiCore_LedBlinkingApp_am263px-lp_r5fss1-1_nortos_ti-arm-clang/Debug/MultiCore_LedBlinkingApp_am263px-lp_r5fss1-1_nortos_ti-arm-clang.out --output=Debug/Multicore_LedBlinkingApp_am263px-lp_system_nortos.mcelf --merge-segments=true --tolerance-limit=0 --ignore-context=false --xip=0x60000000:0x68000000 --xlat="" --max_segment_size=8192
makefile_system_ccs_bootimage_gen:70: recipe for target 'all' failed
gmake[1]: Leaving directory 'C:/TiTest/DevOnLaunchpad/Multicore_LedBlinkingApp_am263px-lp_system_nortos'
makefile:15: recipe for target 'system-post-build' failed
process_begin: CreateProcess(NULL, python C:/ti/mcu_plus_sdk_am263px_10_00_00_35/tools/boot/multicore-elf/genimage.py --core-img=0:../MultiCore_LedBlinkingApp_am263px-lp_r5fss0-0_nortos_ti-arm-clang/Debug/MultiCore_LedBlinkingApp_am263px-lp_r5fss0-0_nortos_ti-arm-clang.out --core-img=2:../MultiCore_LedBlinkingApp_am263px-lp_r5fss1-0_nortos_ti-arm-clang/Debug/MultiCore_LedBlinkingApp_am263px-lp_r5fss1-0_nortos_ti-arm-clang.out --core-img=3:../MultiCore_LedBlinkingApp_am263px-lp_r5fss1-1_nortos_ti-arm-clang/Debug/MultiCore_LedBlinkingApp_am263px-lp_r5fss1-1_nortos_ti-arm-clang.out --output=Debug/Multicore_LedBlinkingApp_am263px-lp_system_nortos.mcelf --merge-segments=true --tolerance-limit=0 --ignore-context=false --xip=0x60000000:0x68000000 --xlat= --max_segment_size=8192, ...) failed.

gmake[1]: *** [all] Error 2
gmake: [system-post-build] Error 2 (ignored)

Can you please clarify what causes the problem ?

Thanks

Seb

Hi

Any Update on this please ?

Thanks

Seb

Hi

May I chase on any update on this please ?

Note, I also tried to launch a project less debug by right clicking on the target configuration and load the code manually for each core.

I'm still having the same problem with Core 3 whereas this method works for the other 2 cores...

Looking forward to reading from you.

Thanks

Regards

Seb

Hi Seb,

I'm not sure what is causing the issue here. I try to reproduce it but fail.

Sebastien Chapeau said:

BUT it only worked once!

I am still trying to figure out what the problem here might be.

Regards,
Shaunak