TMS320F28388D: CPU1 gets stuck after power on

Update:

We have confirmed that CPU1 hit a ITRAP situation by removing GEL files with emulator. The following are the events cause ITRAP.

How can I confirm which event is the right reason?

Another strange thing is that If we try to add some codes in function "Interrupt_illegalOperationHandler" for debug, then CPU1 runs normally.

static void Interrupt_illegalOperationHandler(void)
{
    //
    // Something has gone wrong.  The CPU has tried to execute an illegal
    // instruction, generating an illegal instruction trap (ITRAP).
    //
    ESTOP0;
    for(;;)
    {
        ;
    }
}

Hi,

Did you try stepping into the code to locate the source? One quick thing to check would be the stack memory region in the linker cmd file. The stack memory should be assigned to the lower 64K RAM region only.

Thanks
Vasudha

Hi,

The stack memory is assigned from 0x400 to 0x800 which belongs to M1 RAM. As I mentioned above, the CPU1 works well if we run the code using debugger. The abnormal occurrence only happened in standalone mode. so I can't step into the code. 

Besides, the chip is secured by DSCM, which means we can't watch the CPU registers(specially, the SP register is 0x0000) and variables in the program when CPU1 gets stuck. The PC register is accessible and it points to the function "Interrupt_illegalOperationHandler". Then we unlock the DSP using CCS, strange things happened. The PC register changed to an illegal address with content 0x0000 and the SP register changed to 0x0202. We found that once we unlock the chip, the PC and SP value will change, no matter the CPU is stack or not. If this is true, how can we know the context data exactly at the moment when the CPU gets stuck.

Another strange thing is that, most chips in our products work fine. There is only a few chips do not work properly occasionally. All the software and hardware configuration of these chips are the same.

We have done many tests and get the following surmises:

1. For the sake of simplicity, let's call the "CPU1 stack problem" as PROBLEM. 

2. The PROBLEM may be not a only software problem but related to some hardware setting.

3. The PROBLEM may be related to the execution of function "memset". we checked the disassembly code. It seems OK.

4. The most strange things is that if we added or removed some codes in the program, then the PROBLEM will appear or disappear. We don't know which code and where it is placed will affect the PROBLEM. As I posted above, even if we just added a code as below, the PROBLEM maybe disappeared. So we think the PROBLEM may be related to the FLASH.

asm("		NOP"); /* Wait one cycle */

Hi Vesgine,

This post is assigned to me today.

I will review and get back to you in a day or two.

Thanks and regards,

Vamsi

Hi Vamsi,

Glad to hear that. It seems a tough problem for us. We have done many other tests. My team member will explain the details in the next. There are many weird things that we can't understand. We suspect that maybe there is a problem with some batches of the chips. Looking forward to your reply.

Thanks and regards,

Vesgine

Hi Vesgine,

Please see below snapshot taken from my FAQ link: e2e.ti.com/.../faq-flash---how-to-modify-an-application-from-ram-configuration-to-flash-configuration

Could you check if any of these are affecting your application?

Thanks and regards,
Vamsi

Hi Vamsi,

Thanks for your reply.

We have complied all the rules you mentioned except the following one:

"All the sections mapped to Flash are aligned on a 128-bit boundary using ALIGN() directive in the Flash based linker command file"

In my application, the FLASH sections are aligned on a 64-bit boundary using ALIGN(4). We can test it later. Could you tell us why the sections in FLASH must be aligned on a 128-bit boundary.

BTW, we didn't find any problem in our earlier products which used TMS320F28377D with FLASH aligned on a 64-bit boundary.

Hi Vamsi,

Let me introduce more details about this problem.

1.Here are some "useful things" I've tried to get CPU1 working:

  1) Use the compiler's optimization features to minimize code space. At first the .text area used FLASH0-8, and after optimization, only FLASH0-6 was used；

  2) No longer start CPU2 by deleting "Device_bootCPU2()"

2.Here are some "useless" attempts:

  1) Update the compiler version to the latest；

  2) Adjust RAM and FLASH allocations in the .cmd files

  3) Add an while(1) loop at the beginning of CPU2 application, or simply erase the entire CPU2 FLASH. Can we assume that not CPU2's code, but CPU2's boot process affects CPU1?

3.Here are some noteworthy phenomena:

  1) As Vesgine metioned, the most strange things is that if we added or removed some codes in the program, then the PROBLEM will appear or disappear. This has caused us a lot of trouble in analyzing the problem, so I would like to ask you if there are other ways to get more useful information besides changing the code, such as through stacks or registers.

  2) It seems that an illegal interrupt occurs when a SB instruction is executed.

      Here is the context saved after entering the illegal interrupt. We can see that the Return Address is "0xade05". Can we tell from this that an exception occurs when fetch 0xad0e4?

     Then I used different programming methods to implement the same loop assignment, only to find that it entered an illegal interrupt after SB executed.

  3) We use 0x400-0x800 as the stack space.  When an exception occurs, there is a lot of data other than the small number of areas where the program performed normal operations. Can you get more useful information from it?

Hi Shawn,

Could you share the linker command file and the map file?

If I understand correctly, the issue occurred even when the CPU2 has an infinite for loop in flash and CPU2 booted to flash - correct?

Thanks and regards,

Vamsi

Hi Vamsi,

I found the specification about FLASH-prefetch mechanism from TRM as shown below. And YES, we enabled cache and prefetch mechanism to improve performance of code during initialization. 

If the contents in FLASH are aligned on 64-bit boundary, will this trigger an ITRAP?

Hi Vamsi,

To your question, yes, even if CPU2 only executes an infinite loop, it will cause CPU1 to enter an illegal interrupt.

In addition, I tried changing Align(4) to Align(8) in the CMD file and found it worked. To verify this, I tried adding other loops to the code to make sure that Align(8) solved the problem, and found that in some cases CPU1 still entered an illegal interrupt.

For example:

I added an infinite loop after "Device_bootCPU2()" to the CPU1 code and adjusted the CMD file to Align(8), where CPU1 would theoretically stop in that infinite loop after waking up CPU2, but I found that there was still a chance of entering an illegal interrupt. Then I observed the execution of the interrupts by adding global variables, and found none of the interrupts were executed except “Interrupt_illegalOperationHandler”, and the contents of the stack seemed to confirm this.

Please help us further analyze this problem and we are looking forward to your reply.

Thanks and regards,

Shawn

Hi Shawn and Vesgine,

Please see below from the device errata (https://www.ti.com/lit/pdf/sprz458) regarding prefetch.  This is the reason I asked for your linker cmd file and map file.  Default linker command files from C2000Ware have these locations as reserved.  Please check if you are using these locations in your linker cmd file or in the application - these locations should not be used.

Thanks and regards,
Vamsi

Hi Shawn,

I reviewed the files that you sent offline and I don't see your application's linker cmd file using the reserved locations at the end of the flash memory.

Can you check if there is anything mapped accidentally to the reserved locations in your map file (can happen if a section is defined but not mapped in the linker command file)?

Thanks and regards,
Vamsi

Hi Vamsi,

Thanks for your reply.

I checked the map files of CPU1 and CPU2, and found no program in the reserved FLASH area.

Hi Shawn,

In your linker command file, you have several ramfunc sections for different purposes.  Did you ensure that all of them are copied to RAM before functions in those sections are called in your application?

Thanks and regards,
Vamsi

Hi Vamsi,

I checked the ramfunc sections, all of them are copied to RAM at the very beginning of the code, and none of these programs had been called when CPU1 got stuck, so I think that's probably not the cause of the problem.

Thanks and regards,

Shawn

Hi Shawn,

This does not seem like a flash issue to me.  Few things to check:

1) Can you check if there is a flash ECC error as well?  

2) Please ensure that the stack is not overflowing in to some other RAM block overwriting code or data.  You can fill in the stack space with a known value before executing the application and later see the stack space to know whether the entire space is used up or not.

3) In the event of an ITRAP, you can read the next PC location pushed to stack.

    MOVL     ACC,*-SP[2] ;

    The immediate previous address (ACC-1) would be the address of the instruction that caused ITRAP. 

4) Do you have watchdog enabled?  If yes, are you serving it periodically using an ISR?  Or is it serviced in the linear code?

I will assign this to our system control expert to help you further.  Please expect a reply from them next week.

Thanks and regards,
Vamsi

After going through all the post, I have few addition queries -

• I see below comment -

vesgine said:

We have confirmed that CPU1 hit a ITRAP situation by removing GEL files with emulator. The following are the events cause ITRAP.

Does it mean, you are able to reproduce this issue with debugger connected ? 

• Are you having nested interrupt in this application code ?
• I see even if you have while(1) in CPU2 code, this issue exist. what if you keep CPU2 in reset. Do you still see this issue ?
• When ITRAP happens, have you checked the error flag in NMISHDFLG register. If not, can you please read this register value inside ITRAP handler.

Regards,

Vivek Singh

Hi Vamsi,

Thank you very much for your help. I will check 1) & 2)  today.

I've already check 3), and post some imformation in the previous reply. For example, here is the context saved in the stack when ITRAP happened, and the "Return Address" is highlighted. I found it point to a statement in the "memset" function. I repeated this test, and they all ended up in the same place. Then I tried to replace this "memset" function with my own code, and found the ITRAP still happend. What they all have in common is that ITRAP happened after the execution of the assembly instruction "SB", which I think might be a clue.

For 4), ITRAP happened before I enabled the watchdog. That's why it got stuck in ITRAP.

Thanks and regards,

Shawn

Vivek Singh said:

I see even if you have while(1) in CPU2 code, this issue exist. what if you keep CPU2 in reset. Do you still see this issue ?

The issue disappeared if we keep CPU2 in reset by removing the code "Device_bootCPU2()" in CPU1. But we can't confirm this is the reason of the issue. Because we changed the CPU1 code. As we mentioned before, changing CPU1 code will affect the issue no matter what the code is.

Hi Vivek,

Thanks for your replay.

For your questions:

1. We can't reproduce the problem when we run it through the debugger. We can only reproduce it in the standalone mode, and then attach an debugger to check the stack and registers.

2. Yes, we are using nested interrupt in this application code, but most of the interrupts were not enabled when ITRAP happened, and none of the interrupts were executed except illegalOperation_ISR.

3. If I keep CPU2 in reset by remove "Device_bootCPU2()" in CPU1 code, the system will run normally.

4. I will check NMISHDFLG today and get back to you soon.

Thanks and regards,
Shawn

Hi Vamsi,

1) I read the "Flash0EccRegs" inside the ITRAP handler, and there seems to be no FLASH ECC error.

2) I tried to fill the stack with "0x5A5A" first, and found only a small portion was used and no overflow occurred.

HiVivek,

Here is the content of NMISHDFLG when ITRAP happened.

Hi Shawn,

Thank you for checking the flash ECC registers and the stack overflow.  

Vivek will guide you further on root-causing the ITRAP occurring from SB.

Thanks and regards,
Vamsi

Hi Vivek,

We still have no good idea to investigate the cause of the problem, and we hope to get your assistance as soon as possible. 

Looking forward to your reply. 

Regards,

Shawn Yu

Can you provide the snapshot of nested interrupt code ? Also will it be possible for you to provide a sample code (CCS project) which we can run on TI board to reproduce this issue ?

Regards,

Vivek Singh

Hi Vivek,

I will send the snapshots to you offline.

In previous tests, we found the following phenomena:

1. Using the same program, most of our chips have no problem, only a small part of the chip occasionally abnormal, and very few chips often abnormal.
2. Even with the worst chips, if we add or delete some code, the anomalies don't happen anymore.

Since we cannot provide you with a complete project, it may be difficult to reproduce the problem on your side.  Can we take this approach: if you have any ideas, I can help you verify them.

Thanks and regards,

Shawn

I agree. Such issues are very difficult to reproduce. Let me look at the interrupt code and get back to you if I find something.

Regards,

Vivek Singh

Hi,

We have below usage note in the device errata -

In your iISR, you are enabling the interrupt using EINT just before exiting the ISR. I am not sure what is the use of that. Can you please remove that and see if that helps. 

Regards,

Vivek Singh

We have consided this suggetstion in the errata, and that is "DINT" at the end of the ISR, not "EINT".

Got it. I only saw the function which has EINT. Can you check all the interrupt routine to make sure they all have DINT?

Earlier when you provided the value of NMISHDFLG register value, did you read the NMISHDFLG register in debugger after error happened or read it in your SW inside ITRAP handler ? 

Also on the memset() function, is that function called in your application ? If yes, what exactly that function is doing. You mentioned that even if you replace that function with some other code, issue still exist but you also mentioned that if you make any modification in code then issue goes away. 

Also you mentioned that with debugger connected, you do not see the issue. In that case are you reloading the code after debugger connection ? If yes, can you just run the test instead of loading the code. Code is loaded in flash so it should work without loading. 

Regards,

Vivek Singh

Hi Vivek,

Sorry to reply so late, I've been busy with other thiings this week.

For your questions:

1. I have added DINT to the end of all the nested interrupts, and the problem still there;

2. I copied the NMISHDFLG to a global variable inside the ITRAP handler;

3.  Yes, we used the memset() to restore the specified variable to its default value. I tried other code to implement the same functionality as memset(), and found the issue still exist. However, when I try to add or remove code elsewhere, the issue may disappear；

4. Yes, I reloaded the code after debugger connection. I will try as you suggest and get back to you with the result.

Thank you very much.

Shawn