Concerto - C28 NMI + ITRAP on reading data from L1

Fabio,

Fabio Mannino said:

bit C28FLUNCERR = 1 e NMIINT=1 (spruh22b pag. 194). After running the NMI handler the code runs into ITRAP and halts there (since I put an endless loop).

from above description it looks like an ECC error in Flash not RAM. I beleive the RAM-INIT code is pretty stable and well tested. When you run the code with debugger connected, the GEL scripts might be disabling the ECC on flash, but when you run stand-alone ECC is enabled. To make sure that this is not flash memory realted can you send an IPC command to C-Boot ROM to write to ECC disabling address (*0x4300 = 0x0), you will have to send a 16 bit protected write of value 0x00 to the location 0x4300. Please refer to IPC command supported by C-Boot ROM in device TRM, boot ROM section.

you will have to send this command to C-Boot ROM, before sending the boot mode IPC command. Thats the tricky part. Below is simpler way to do this. Implement your own unsigned short IPCMtoCBootControlSystem(unsigned long ulBootMode) function, in your M3 app...like below. Copy paste the function code from driverlib/ipc_util.c in your application and edit as below.

unsigned short $$$_YOUR_OWN$$_IPCMtoCBootControlSystem(unsigned long ulBootMode)
{
    // Wait until C28 control system boot ROM is ready to receive MTOCIPC INT1
    // interrupts
    while ((HWREG(MTOCIPC_BASE +
                  IPC_O_CTOMIPCBOOTSTS) &
            CBROM_BOOTSTS_CTOM_CONTROL_SYSTEM_READY)!=
           CBROM_BOOTSTS_CTOM_CONTROL_SYSTEM_READY)
    {
    }

-----ECC disable code to add ------

    //RAM INIT for M1, and CTOM MsgRAM - M0 RAM INIT is done by C-BootROM
    IPCLiteMtoCDataWrite_Protected(IPC_FLAG1,
                                 0x4300,
                                 0,
                                 IPC_LENGTH_16_BITS, IPC_FLAG32);

    //wait until C-BootROM acks
    while(HWREG(MTOCIPC_BASE + IPC_O_MTOCIPCFLG) & IPC_FLAG1) ;

    //CHECK IF pass or fail
    if(HWREG(MTOCIPC_BASE + IPC_O_MTOCIPCFLG) & IPC_FLAG32)
    {  
        //still set - so command failed.
        return 1;

    }

-----ECC disable code end ------

    //Initialize Control Subsystem memories
    if(RAMControlInitM1MsgRam())

....

....

} // func end

call this function in place of the "IPCMtoCBootControlSystem" in your M3 app, if it works then you know the problem is because of error on flash memory. Else we will have to debug further.

Best Regards

Santosh

one more thing to to do is, can you step into the function RAMControlInitL0L3Ram() and see if it is executing all the way through? you could do this without doing abvoe suggested modifications...if you find the functions is executing proper...then go ahead and do the above tests.

Best Regards

santosh

Hi Fabio,

Addition to what Santosh has suggesteed, please check if following errata is applicable to your case.

---------------------------------------------------

Advisory C28x Flash: The SBF and BF Instructions Will Not Execute From Flash
Revision(s) Affected 0
Details When the flash prefetch is enabled, the SBF and BF instructions will not be correctly
fetched from the Flash.
Workaround(s) Compile the application code with the –me option to avoid the SBF and BF instructions.
This will be fixed in the next revision of the silicon.

---------------------------------------------------

Complete errata document can be accessed at link below. 

http://www.ti.com/general/docs/lit/getliterature.tsp?literatureNumber=sprz357c&fileType=pdf

Regards,

Vivek Singh

Vivek Singh said:

Hi Fabio,

Addition to what Santosh has suggesteed, please check if following errata is applicable to your case.

---------------------------------------------------

Advisory C28x Flash: The SBF and BF Instructions Will Not Execute From Flash
Revision(s) Affected 0
Details When the flash prefetch is enabled, the SBF and BF instructions will not be correctly
fetched from the Flash.
Workaround(s) Compile the application code with the –me option to avoid the SBF and BF instructions.
This will be fixed in the next revision of the silicon.

---------------------------------------------------

Complete errata document can be accessed at link below. 

http://www.ti.com/general/docs/lit/getliterature.tsp?literatureNumber=sprz357c&fileType=pdf

Regards,

Vivek Singh

Hello Vivek, I had already seen this errata and tryied the -me option, but it did not solve. Actually I looked at the generated assembler and SB/SBF instructions where still there, so I'm not sure I put the -me option in the right place of CCS project properties dialog box (the Command field of Build->C2000 Compiler page). I didn't find a suitable option in the various settings pages. Can you tell me where exactly shall I set the -me option please ?

Tomorrow I will try the Santosh suggestion and will update you.

Thanks

Fabio

Hi Fabio,

There is option under  CCS Build -> C2000 Compiler ->  Advanced Options -> Runtime Model Options, where you can disable C28x fast branch instructions.

Following is the snapshot of the same.

Is this the one you tried earlier?

Regards,

Vivek Singh

Vivek Singh said:

Hi Fabio,

There is option under  CCS Build -> C2000 Compiler ->  Advanced Options -> Runtime Model Options, where you can disable C28x fast branch instructions.

Is this the one you tried earlier?

Regards,

Vivek Singh

Hello, it was not the place I tried earlier, but activating the -me as shown above did not changed the behavior.

Regards

Fabio

Santosh Athuru said:

one more thing to to do is, can you step into the function RAMControlInitL0L3Ram() and see if it is executing all the way through? you could do this without doing abvoe suggested modifications...if you find the functions is executing proper...then go ahead and do the above tests.

Best Regards

santosh

Hello Santosh,

I wrote a my own function for C28 boot as you suggested. It returns 0 if everything is fine, 1 if either RAMControlInitM1MsgRam or RAMControlInitL0L3Ram fail and 2 if writing the 0x4300 address fail. The function returns 0 so everything seems to be done correctly.

I run more tests reading the values of some useful registers when C28 runs into ILLEGAL trap and this is the result:

NmiIntruptRegs.NMIFLG.all = 5

FlashEccRegs.UNC_ERR_ADDR = 0

FlashEccRegs.ERR_STATUS.all = 0

RAMErrRegs.CCUNCREADDR = 00009122

0x9122 is the address in RAM (L1) where the code is reading when CNMIFLG change to 5 (from the initial value 0, of course) and corresponds to the address of a C struct that appears to be partially cleared to zero when the code is executing the function that generates the trap. Unfortunately I don't have a screenshot of debugger showing the struct values, but it has some fields = 0 and some other not (clearly garbage). When the code reads the garbage fields the trap is triggered.

As shown above the NMI register reports a FLASH RAM uncorrectable error but no evidence of this error is shown by Flash ECC registers, while the RAM address that triggers the trap is correctly stored into CCUNCREADDR.

The code that generate the trap is the following:

            |Uint32 getStatusValue(Uint16 index)
          27|{
  P:0010AC30|FE02            _getStat.:addb    sp,#0x2
  P:0010AC31|9641                      mov     *-sp[0x1],al
          28|        return *(pStatusVars[index]);
  P:0010AC32|8F009508                  movl    xar4,#0x9508
  P:0010AC34|0E41                      movu    acc,*-sp[0x1]
  P:0010AC35|FF30                      lsl     acc,#0x1
  P:0010AC36|560100A4                  addl    @xar4,acc
  P:0010AC38|8AC4                      movl    xar4,*+xar4[0]
  P:0010AC39|06C4                      movl    acc,*+xar4[0]
          29|}
  P:0010AC3A|FE82______________________subb____sp,#0x2
  P:0010AC3B|0006                      lretr
            |
     

the trap is generated when the MOVL in line 0010AC39 is executed.

Thanks for supporting.

Fabio

Fabio,

I did not read through the entire thread.  I notice that there is a Flash ECC error occurence shown by NMIFLG and hence wanted to ask few questions to eliminate the Flash ECC issue.

Are you using CCS Flash plugin to program the Flash?

If yes, which version of CCS are you using?

Are you programming any passwords in the password locations of Flash using Plugin GUI or in application code?  If yes, are you programming ECC for the password locations?  

Thanks and regards,
Vamsi

Fabio,

CNMIFLG = 5 means RAM uncorrectable error and not Flash uncorrectable error.

That is the reason you see RAM error address register getting updated but not Flash uncorrectable address register.

Thanks and regards,

Vamsi

Vamsi Gudivada said:

Are you using CCS Flash plugin to program the Flash?

If yes, which version of CCS are you using?

hello Vamsi,

I don't use CCS Flash plugin. I have a Lauterbach debugger and I'm using the scripts provided with it. The RAM zero-setting at startup in debug environment are part of such scripts (and as said above, this removes the ITRAP when code is started from debug).

Vamsi Gudivada said:

CNMIFLG = 5 means RAM uncorrectable error and not Flash uncorrectable error.

sorry, you're right.

I (repeatedly !) confused them reading the TRM. I corrected the post above, now it correctly summarizes what we have seen so far.

About the password location, I'm not programming any password.

Thanks and regards

Fabio

Hi Fabio,

Just wanted to know if the issue you were facing got resolved.

Regards,

Vivek Singh

Hello Vivek,

I modified the CodeStartBranch code in order to clear RAM L1,L2,L3 before calling _c_int00:

    .ref _c_int00
    .global code_start

LRAM_init:
        movl XAR0,#0x9000
        mov AL,#0
        mov AH,#0x2FFF
        rpt AH || mov *XAR0++,AL
        LB _c_int00                                ;Branch to start of boot.asm in RTS library

    .sect "codestart"

code_start:
    LB LRAM_init

This seems solving the problem and is reasonable since this way we initialize the ECC before starting the C environment, but this should be already done by C28 BOOT ROM before calling the LRAM_init (requested by IPCMtoCBootControlSystem, which is called by M3 application to boot C28).

In other words, as far as I understood, the above modification of CodeStartBranch should be not needed, since the job should be already done by M3 IPC calls to C28 boot ROM.

Thanks

Fabio

Fabio,

does the error happen in stand-alone execution mode or when you run wiht debugger connected. As you mentioned the IPCMtoCBootControlSystem should take care of RAM-INITs when in stand-alone mode. When debugger is connected and you are running M3 and C28x in debug context, the C28x RAM-INITs are to be taken care of by GEL files.

Also, it is true that you wouldn't need to to do RAM-INIT from code_start that is what I'm worried of. Once thing you can try is in your own implemented function in M3 application (the func that send IPC to RAM-INIT C28x) after the function returns and before sending the BOOT MODE IPC command, add some delay and see if that fixes the issue as well.

Best Regards

Santosh

Hello Santosh,

the error happens in stand-alone mode. The debugger (Lauterbach, that uses a GEL-like scripting system) correctly initializes the RAM before running the code. If I remove the RAM initialization from the script I get the error also in debug mode.

I will try to introduce a delay as you described, but why should it fix the issue if the IPCMtoCBootControlSystem already uses the IPC flags to perform the correct handshaking with BOOT ROM ?

As mentioned in a previous post, I checked the return value of the M3 functions that interact with C28 BOOT ROM and no error is reported.

Thanks and best regards

Fabio

Fabio,

from your earlier explanation, where some part of the data structure is not all properly initialized, I was suspecting that RAM-INIT has begun but not completed, it shouldn;t take more 2048 cycles for RAM-INIT to complete. That is very unlikely but lets rule that out.

we also need to verify that the code/function that does the RAM-INIT is not buggy. Can you check if M3 is sending the IPC write and reads to correct locations to C-Boot ROM? you verified that the function is not returning any erros earlier but lets check the address and values also.

I don't yet understand why would adding another RAM-INIT at code_start solve the issue? is your application accessing any C28x RAM (via IPC) before it sends the boot mode command?

Best Regards

Santosh

Hello Santosh,

I put a long delay in IPCMtoCBootControlSystem before the boot command (just after the switch (ulBootMode) {...}) but the trap is still run.

How can I check the M3 is using the correct location of C-Boot ROM ? the code of RAMControlInitL0L3Ram is not available.

Besides that, I had tried to use the code listed in Concerto TRM spruh22b chapter 6.7.2 "Master Subsystem Application Procedure to Initialize the Control Subsystem RAM
Using IPC", which should be - more or less - the same code used by driverlib for RAMControlInitxxxx functions, and the behavior is not changed.

My M3 application does not access any C28 RAM via IPC before booting the C28. The only things done are some initializations to assign GPIOs and a part of shared ram to C28 for write access.

Thanks and best regards

Fabio

Thanks Fabio,

Thanks for testing it out. Can you try blinky flash example in control Suite and see if the problem is repeating? I will try to run this as well on F28M35x control card and see if I can reproduce the error. Besides that we are going to re-review the function and see if we can find any thing.

can you also list down the exact steps you follow and how you are aware (the symptoms) that the code is not running without having debugger connected?

Best Regards

Santosh

hi Santosh,

I had already tried the blinky example in the past weeks but unfortunately I couldn't reproduce the issue. The traps are triggered reading L1. A code like this:

void ramScan(void)
{
    Uint16 dummy,i;

    for(i=0; i<0x3000; i++)
        dummy = *(Uint16 *)(0x9000+i);
}

called at the very beginning of main() (just after the code needed to init Flash memory, IPC and interrupts to assign NMI and ILLEGAL trap handlers) may or may not trigger the traps. Few lines of code that reads an array placed in L1 may or may not run the traps. Writing on L1 before reading the same data makes the traps disappear.

To be precise, the traps ar not random: "may or may not" means that if the overall code is arranged in such a way that the traps are triggered, they are always triggered every time I run the system. If the code arrangement does not trigger the traps, they never happen during application execution. Changing/removing/moving even a single line of code - regardless it is related or not with L1 reading - can change the behavior.

Eveng changing the code of NMI and ILLEGAL handler may let the traps happen or not. Basically seems that placing differently the code in FLASH has something to do with the issue.

I admit this is strange and hard to explain but it is what happens (investigated by two people for several days on two different control cards with F28M35H52C1 and reported at least on a third board).

The symptoms are that the C28 core enters both NMI and ILLEGAL handlers. I am sure of that since, for example, I put in the handlers some code to send an IPC command back to M3 core that let it write a string to UART connected to a terminal. If I remove the GEL-like debugger script code for L1 init I can reproduce the issue in debug and see that the code jumps into trap handlers. I also tried to remove any IPC-related code just having the C28 core show the traps blinking a led and the behavior is the same, thus I can state that IPC is not related with issue.

The issue happens if C28 .ebss section is mapped to L1 memory.

If needed I can try to give you more details.

Thanks and best regards

Fabio

Fabio,

I have reviewed the code again today and found below changes could be done for the RAM INIT functions to function more accurately. I will try linking blinky example to L1 RAM and see if I can reproduce the issue. But please apply below changes to your RAM INIT functions on M3 and see if there is any change in error.

Replace lines

if(HWREG(MTOCIPC_BASE + IPC_O_MTOCIPCDATAR) &

(CRINITDONE_RAMINITDONE_M1 | CRINITDONE_RAMINITDONE_CTOMMSGRAM))

{

with

if(HWREG(MTOCIPC_BASE + IPC_O_MTOCIPCDATAR) &

(CRINITDONE_RAMINITDONE_M1 | CRINITDONE_RAMINITDONE_CTOMMSGRAM) == (CRINITDONE_RAMINITDONE_M1 | CRINITDONE_RAMINITDONE_CTOMMSGRAM))

{

and replace lines

if(HWREG(MTOCIPC_BASE + IPC_O_MTOCIPCDATAR) &

(CLxRINITDONE_RAMINITDONE_L0| CLxRINITDONE_RAMINITDONE_L1|

CLxRINITDONE_RAMINITDONE_L2| CLxRINITDONE_RAMINITDONE_L3 ))

with lines

if(HWREG(MTOCIPC_BASE + IPC_O_MTOCIPCDATAR) &

(CLxRINITDONE_RAMINITDONE_L0| CLxRINITDONE_RAMINITDONE_L1|

CLxRINITDONE_RAMINITDONE_L2| CLxRINITDONE_RAMINITDONE_L3 ) ==

(CLxRINITDONE_RAMINITDONE_L0| CLxRINITDONE_RAMINITDONE_L1|

CLxRINITDONE_RAMINITDONE_L2| CLxRINITDONE_RAMINITDONE_L3 ))

The above modifications does a more throrough check for errors. I'm not yet 100% sure that this will solve the error, but as I mentioned I will try to reproduce the error on my end and let you know.

Best Regards

Santosh

Fabio,

I got a chance to verify the driver lib RAM INIT function and it is woring fine. I'm able to clear the all the RAM(s). Can you repeat the below steps and let me know how it goes. Please so the below if the code modifications suggested above doesn't change anything.

1.> Edit you Target configuration and take out the GEL files from M3 and C28x .

2.> Set boot mode pins to SCI (option 2) or Parallel (0x0)

3.> power off and power ON the board. In your CCS debug perspective close all the watch/expressions, memory windows that can look at any flash address.

4.> Now connect CCS to both M3 and C28x

5.> Dont' do any debugger resets (because boot ROM would have run and initialized the subsystems)

6.> Load your application in M3 RAM or Flash and run upto the RAM INIT functions

7.> on C28x (don't do any debugger reset), just look at the current instruction in dis-assembly (it must be at IDLE instruction) and now open different memory windows to location '0x500'  (M1 RAM) , '0x8000 (L0 RAM), 0x x9000 (L1 RAM), 0xA000 (L2 RAM), 0xB000 (L3 RAM)

8.> all the above memory windows must have some garbage or randam data.

9.> now just RUN C28x (by this you will be running C-Boot ROM)

10.> on M3 from where you hit the break point , run over the RAM INIT functions and HALT M3 afte rthe functions return

11.> Now select C28x core and look at the memory windows. Are they all Zeroes out?

Please let me know the result at the end of step 11, this should help us provide right direction for debug.

I tried all the above steps with a blinky_dc_m3 project from control Suite... (just make sure you call the RAM init functions in the file). I see that all the RAM-INITs are happening fine.

Best Regards

santosh