PROCESSOR-SDK-AM64X: M4F IRAM memory issue

Hello Gergely,

The SBL sets the value 0xBF30BF30 at the address 0x400 while initializing the M4F IRAM with valid reset vector and wait instruction. This is shown below

This ensures the M4F core is in a valid state on the reset & release of the core.

Gergely Korcsák said:

Our linker.cmd matches the hello_worls projects linker file, but I still notice that the .sysmem heap segment is usually linked after the .text codesegment in our case, while in the example it becames the first segment from 0x200-0x8200, but as I understand, that shouldn't cause this issue.

Coming to this, I also understand this shouldn't be the cause. But just for the confirmation, can you please group the .text & .sysmem sections together in that order like shown below

SECTIONS
{
    /* This has the M4F entry point and vector table, this MUST be at 0x0 */
    .vectors:{} palign(8) > M4F_VECS

    GROUP
    {
        .text:   {} palign(8)     /* This is where code resides */
        .sysmem: {} palign(8)     /* This is where the malloc heap goes */
    } > M4F_IRAM

    .bss:    {} palign(8) > M4F_DRAM     /* This is where uninitialized globals go */
    RUN_START(__BSS_START)
    RUN_END(__BSS_END)

    .data:   {} palign(8) > M4F_DRAM     /* This is where initialized globals and static go */
    .rodata: {} palign(8) > M4F_DRAM     /* This is where const's go */
    .stack:  {} palign(8) > M4F_IRAM     /* This is where the main() stack goes */

    /* Sections needed for C++ projects */
    .ARM.exidx:     {} palign(8) > M4F_IRAM  /* Needed for C++ exception handling */
    .init_array:    {} palign(8) > M4F_IRAM  /* Contains function pointers called before main */
    .fini_array:    {} palign(8) > M4F_IRAM  /* Contains function pointers called after main */

    /* General purpose user shared memory */
    .bss.user_shared_mem (NOLOAD) : {} > USER_SHM_MEM
    /* this is used when Debug log's to shared memory are enabled, else this is not used */
    .bss.log_shared_mem  (NOLOAD) : {} > LOG_SHM_MEM
    /* this is used only when IPC RPMessage is enabled, else this is not used */
    .bss.ipc_vring_mem   (NOLOAD) : {} > IPC_VRING_MEM
}

This should ensure that the .text section comes before the .sysmem section. Afterwards, if by any chance the firmware works, we can try to reason why it actually works and not the other way.

Thanks!

Hi Prashant,

grouping the sections doesn't change behavior. We still witness FW crash.

Hello Marc,

Thanks for doing the testing!

So, this rules the ordering of the sections. Next, can we have a look at the MPU configuration and experiment with different settings. At times, these MPU configuration settings have been the source of bugs. The default M4F Hello World example defines the two MPU configurations out of which one is of particular interest.

I would like to know if you have the same MPU configurations in your firmware? If no, could you please have the same MPU configurations as the Hello World example and then try running your firmware. If yes, can you experiment with different options of Region Attributes like Strongly Ordered to see if this changes the behaviour.

Thanks!

Hi,

The original MPUs from the sample are there, but we also added one MPU config for access to MSRAM on main domain. We use shared memory here.

The RATs are as in the sample; so the MSRAM MPU accesses via CONFIG_ADDR_TRANSLATE_REGION3 in the local address which is identical to the system address.

Hi Marc,

Could you please try running the firmware with different Region Attributes settings like Strongly Ordered for CONFIG_MPU_REGION1. I would also like to know if the firmware is trying to access MSRAM at the point of failure.

There is one more thing: Did you have this firmware working before but after some changes, it broke?

Thanks!

I changed to strongly ordered, but the crash is still there.

the crash occurs in a function which will also access the shared memory, but it doesn't crash upon accessing this shared mem and for some lines of code the access to shared mem works.

When looking at the mixed source/disassembly code; there the CCS loses track of the source-assembly linkage at some point in this function close to the crash. It doesn't display any source anymore.

When changing the start address of IRAM from 0x200 to 0x2000 with exactly the same code; then all is fine and source/assembly shows correct and no crash happens.

I shared a movie of the debug session good+bad with our TI application engineer; as well as the .map files; but I cannot share this here on the forum.

Hello Marc,

Thank you for all the testing & sharing the screen recordings of the observations! It helps in ruling out the suspected causes.

Though I can't reproduce the issue on my side, this time I have dig deep into the library source code. There, I see that we are resetting the Stack Pointer (SP) manually in the startup function (_c_int00) code of M4F core. This seems somewhat unusual.

So, can we once comment the code that sets that SP as shown below in the M4F NORTOS kernel library code and try running the firmware with these changes.

After the changes, rebuilt the libraries for the right profile or we can build the M4F NORTOS kernel library only for the right profile since we have done changes only in that library code. Assuming we are building for debug profile, please run the below command from the MCU+ SDK installation directory to build the M4F NORTOS kernel library only.

gmake -s -C source\kernel\nortos -f makefile.am64x.m4f.ti-arm-clang PROFILE=debug all

Once the library is built then rebuild the example and try running the same. Let me know if the issue still persists.

Thanks & Regards,

Prashant

Hi,

Unfortunately the change didn't solve the issue.

I shared the screencapture video via our application team.

Looking forward to more suggestions!

Hello Marc,

I realize that in my very first response, I suggested to group the sections (.text & .sysmem), in the Linker Command file like this:

GROUP
{
    .text:   {} palign(8)     /* This is where code resides */
    .sysmem: {} palign(8)     /* This is where the malloc heap goes */
} > M4F_IRAM

However, this ordering was already there with your builds. What I wanted to suggest is to group the sections (.text & .sysmem) like this:

GROUP
{
    .sysmem: {} palign(8)     /* This is where the malloc heap goes */
    .text:   {} palign(8)     /* This is where code resides */
} > M4F_IRAM

This ensures that the .sysmem comes before the .text section.

Actually, this ordering can be of relevant here as the Hard Fault is occuring somewhere around 0xFF(x) and if .text comes before .sysmem then this address belongs to .text section. However, if the .text section does not cover this address, then the Hard Fault does not occur. So, I guess somehow somewhere in .text something is getting overwritten when .text covers that address.

So, could you please try the above ordering which I wanted you to suggest in the very first response. Sorry for the mishap. Let me know if this works.

Hoping for the best!!

Regards,

Prashant

Hi Prashant,

I tested your proposal and indeed; in this case I don't see the hard fault.

But is it really solving the problem or just masking it? Maybe now the heap in .sysmem gets overwritten instead of the .text code?

The workaround is similar to what I did earlier; namely let the IRAM start from 0x2000 instead of 0x200; which of course reduces the amount of available code memory.

Overall I have the same feeling, namely that something is overwriting memory around address 0xFFx . Could it be SYSFW? Could it be related to SBL? Is there any code being used from SBL after the image is in place and execution starts on the seperate cores? I would believe not?

Hi Marc,

Marc Schouteeten said:

I tested your proposal and indeed; in this case I don't see the hard fault.

This definitely suggests whatever section covering 0xFF(x) is somehow gets overwritten around same 0xFF(x).

Marc Schouteeten said:

Overall I have the same feeling, namely that something is overwriting memory around address 0xFFx . Could it be SYSFW? Could it be related to SBL? Is there any code being used from SBL after the image is in place and execution starts on the seperate cores? I would believe not?

I also believe the Sysfw & SBL has nothing to do here. It must be the M4F code, most probably the startup code that executes before main, which overwrites the memory around 0xFF(x).

There is actually a way to debug this startup code and find if it is overwriting the memory. The way to do this is shown in the below attached screen recording:

Following this, could you please try verifying the program at each statement of this startup function. This verification checks if the sections are intact in the memory or not. If they are it reports "Verification Successful" otherwise reports "Failures" with the address at which the change is detected.

Let me know if you get a verification failure in this startup function.

Regards,

Prashant

Hi Prashant,

the verification fails in the Dpl_init(), so already after jump to main.

Hi Marc,

This verification failure is expected as it is caused by the HwiP_init call which initializes the interrupt handlers in the .vectors section. Even, I see the same verification failure.

So, we want to verify only the .text section and ignore all other sections. To do that, we would need to create a new ELF file from the original ELF file. The steps would be:

• Take the original ELF file and copy paste it to preserve the original one.
• Use the objcopy tool to keep only the .text section in the copied ELF file. The command for this is:
  

  tiarmobjcopy --only-section=.text ${Copied ELF File Path}

• Confirm if the copied ELF file now really contains only .text section. Ignore sections (.symtab, .strtab & .shstrtab) if present. The command is:
  

  tiarmreadelf.exe -S ${Copied ELF path file}

These tiarm tools comes with the TI ARM CLANG compiler and are available at ${TI_ARM_CLANG_PATH}/bin.

Now, after the original ELF file has been loaded on the core, we will use the copied ELF file for verification. We know the .text section content is same in both ELF files. Please ignore verification failures, if any, because of the other sections (.symtab, .strtab & .shstrtab).

For example,

Here, I can ignore this verification error as the corresponding address does not belong to .text section. And we know the verification of .text must be successful as other sections in the copied ELF file come later.

The above can be tried to see what is causing the change. I was also thinking before finding out what caused the change, we should first verify if the .text section indeed is overwritten. For that, let the program run into Hard Fault then do the verification step using the copied ELF file. If the verification fails in .text section, then we can debug it otherwise we have look for something else.

Regards,

Prashant

Hi Prashant,

Since HwiP_init() installs the interrupt handlers; this is causing the verification of course.

However even without starting execution; it seems the disassembly is already not correct. So after CPU reset and reload of the program, even before stepping into the _c_int00() and main(), I'm seeing the following (with missing source code in the mixed disassembly window :

Dear Prashant,

when I load my program as always with the CSS debug button and then verify against the stripped ELF; then I have a verification failure at 0xFFC. In this case the program will crash when executing code from the function which overlaps with this location.

When I follow the instruction in your video and in GEL menu I first disable verification and auto run; then this memory location 0xFFC is NOT overwritten and the code runs correctly.

kind regards,
Marc

Hi Prashant,

yes, this solves the issue. Thank you!

Hi Marc,

If I may ask, I would really like to understand what really is causing the issue in your case and take this thread to its end. As I understand it, you replied that there are no verification errors in the startup code in case the verification & auto run are turned off. So, there still is the mystery of how the value is getting overwritten at your end.

So, if you permit, may I ask for some more help to find the root cause at your end.

Thanks!

Sure, I have proposed our FAE to setup an online meeting tomorrow; that would maybe be the most efficient?