TMS320F28388D: Can't access flash after issuing many flash API async programming commands

Part Number: TMS320F28388D
Other Parts Discussed in Thread: UNIFLASH, , C2000WARE

Hello,

I have been working with the flash API since a long time and it has been working great all along until now. I have been working with the flash API in a synchronous manner, using the Fapi_issueProgrammingCommand() function always followed by a while(Fapi_checkFsmForReady() != Fapi_Status_FsmReady).  In order to make my flash access less time consuming, I have decided to separate the calls to Fapi_issueProgrammingCommand() and the check that the Flash Memory Controller is ready for the next command so I can issue a programming command and then poll the FMC until it's ready to receive a new command in a non blocking way. And it has been also working great until my program ran through a path that writes many data in flash in which I forgot to add the new function checking for the readyness of the flash api for a new command. At this point I basically did a for loop in which there is essentially Fapi_issueProgrammingCommand() functions and of course no while(Fapi_checkFsmForReady() != Fapi_Status_FsmReady). That would look like this:

for ( i=0; i<200;i++) {
    Fapi_StatusType apiResult = Fapi_issueProgrammingCommand(
        (uint32_t *)address, FLASH_AlignedBuffer, 4, 0, 0, Fapi_AutoEccGeneration);
	}

My program was running on core1 with all functions loaded in RAM during initialization. So for now, the program still runs but all the flash access are denied. The program is not writing to the flash anymore and I can not program my core1 either.  I tried the operation on 2 TMS320F28388D. on the first one, I just lost access to every flash control you can think of either through UniFlash or CCS. But I can still connect via JTAG (run the code step by step for exemple) but I can't erase the flash nor reprogram it even though I can still read it. I have checked everything I could think of: trying to unlock the DCSM but it seems it is not locked, using the erase function of CCS On-Chip flash, using the erase function of UniFlash. When I try to upload code, I have these outputs from the GEL file:

C28xx_CPU1: Error initializing flash programming: Interface returned from dll, but flash is not available on this device.

C28xx_CPU1: GEL Output:
... DCSM Initialization Start ...
C28xx_CPU1: GEL Output:
... DCSM Initialization Done ...
C28xx_CPU1: GEL Output:
CPU2 is out of reset and configured to wait boot.
(If you connected previously, may have to resume CPU2 to reach wait boot loop.)
C28xx_CPU1: GEL Output:
CM is out of reset and configured to wait boot.
(If you connected previously, may have to resume CM to reach wait boot loop.)
C28xx_CPU1: Loader: One or more sections of your program falls into a memory region that is not writable. These regions will not actually be written to the target. Check your linker configuration and/or memory map.
C28xx_CPU1: Trouble Removing Breakpoint with the Action "Finish Auto Run" at 0x18368: (Error -1066 @ 0x18368) Unable to set/clear requested breakpoint. Verify that the breakpoint address is in valid memory. (Emulation package 9.2.0.00002)

And any operations in the CCS On-Chip Flash utility results in a pop up informing me that : "Target is not connected or does not support current Flash operation" whereas I am connected via JTAG and running the code step by step. Another interesting thing is that the code does not run when I am not connected via JTAG. The thing is, to run my code in debug mode, I have to run the "emu boot Flash" script. When I power up my microcontroller, it is not able to boot by itself anymore. The core2 is still fully functionnal, I can load code into it and everything runs fine.

On the 2nd TMS320F28388D, after running the bad code, it seemed all right so i just unplugged the jtag connection. Since that moment, it is impossible to connect to that microcontroller, the code does not run (probably the same booting problem as the other one) but I can't even connect to any of the two core in jtag anymore.

I am running out of ideas but I need to get these two microcontrollers back to work so any hint would be greatly appreciated.

Best regards, 

Quentin

  • Quentin,

    Can you check your map file to see if anything is mapped to the security configuration area in OTP?

    On the board where you are able to connect, after connect, issue a debug reset, hit resume button (to run the BootROM), then halt after couple of seconds and then try flash operations.  Let me know if that works.

    Thanks and regards,

    Vamsi

  • Hi Vamsi,

    There is nothing mapped to the security configuration area in OTP. The code that created the problem is essentially the same that worked before. I just changed the sequence that writes data in flash.

    I already reset the board, tried many sequences to get the flash operations to work but nothing did it. I still tried again on your advice but it still does not work.

  • Quentin,

    Can you send a snap shot of the DCSM OTP space and flash start address?

    As you already know, the sequence that you changed is not valid - when a flash program operation is active, you can not issue another program command.  Hence, we need to check what happened in your case.

    Thanks and regards,
    Vamsi

  • Vamsi,

    Here is the DCSM OTP space from the memory browser view. If you prefer any other visualization let me know.

     

    And here is the flash start address : 

    As seen in the .map file, that's where the codestart is located

    Thanks and regards,

    Quentin

  • Quentin,

    Which CCS and Uniflash versions are you using?

    Can you check for updates in CCS and install if any for TI C2000 Device support and Debug Server flash?

    Thanks and regards,
    Vamsi

  • Vasmi, 

    I am using CCS 10.1.0 and Uniflash 6.1.0 (I would love not to update anything in CCS since I am working on this project for at least the last 6 month and I don't wanna change my compilation chain that have been working very well until now)

    This is the list of available updates. Is there any of these related to TI C2000 Device support or Debug Server Flash?

    Thanks and regards,

    Quentin

  • Quentin,

    No need to update the compiler.  Looks like you are using recent versions of CCS and Uniflash. 

    1) How are the boot mode pins configured?  Please try with wait boot if not already and see if that helps.

    2) What values do you see at the addresses 0x700B0 and 0x700B1?  Asking since you got the error "C28xx_CPU1: Error initializing flash programming: Interface returned from dll, but flash is not available on this device.".  This tells me that the flash tool is not able to read the OTP correctly.  

    Thanks and regards,

    Vamsi

  • Hi Vamsi,

    1) I am booting in Flash/USB mode (GPIO72 = 1, GPIO84 =1). I did try the wait boot mode ( GPIO72 = 0, GPIO84 = 1) bit it did not change anything. Checking the XRsn pin on the board I can't connect anymore, I saw that it was constantly reseting (i.e. Going low periodically, I don't remember what period but something very short), and trying different boot mode pins would not change anything.

    2) here are the values I read at the adresses 0x700B0 and 0x700B1 : 000 0716

    It seems I can read the OTP correctly.

    Thanks and regardsn

    Quentin

  • Quentin,

    1) XRSn reset should not happen in wait boot mode.  Can you check whether or not your power supply is stable and is able to meet the erase/program power demands?

    2) These locations does not read back the correct value.  I need to check this internally.  Power supply issue or wait-state issue can cause this.  

    3) Which emulator are you using? Try re-installing the drivers if possible.

    Thanks and regards,

    Vamsi

  • Vamsi,

    I am now working on the board I can connect to, I changed the microcontroller on the other card. 

    1) If I boot in flash mode, the program runs, if I boot in wait mode, it keeps reseting (i.e. toggling the XRSn pin). The power supply is stable. My 3.3V is stable.

    2) Ok, if you need more info let me know

    3) I am using the Emulation package 9.3.0.00042 which I updated after the problem occured. Isn't this supposed to update the drivers?

    thanks and regards

    Quentin

  • Quentin,

    Thank you for the info.  

    1) Ok. Do you have a valid code programmed in flash along with ECC?

    2) Ok. When the application runs fine in flash boot mode (as you said above), does it show the same values (you mentioned 000 0716 earlier) at addresses 0x700B0 and 0x700B1?  Or is that different?  Please check and let me know.

    3) Which emulator are you using?  XDS200? XDS110?  

    Now that you noticed that issue with your code, can you proceed with the correct code on other new devices that you have?  

    Thanks and regards,
    Vamsi

  • Vamsi,

    1) I have a valid code programmed in flash

    2) The values I showed you for the adresses 0x700B0 and 0x700B1 are the values I can see when running the valid code programmed in my flash booting in flash boot mode. This is the only way I can access the flash memory browser through JTAG

    3) I am using the XDS100v2 emulator on the TMS320F28388D controlcard.

    I proceeded with correct code on other device and I have no problem. So i can say for sure that the problem is that I issued many asyncronous program command to the api flash without waiting for it to be ready to receive a new command.

    Thanks and regards,

    Quentin

  • Quentin,

    Thank you for the info.

    I would suggest to continue with your fixed code.  If you want to debug this further, we may need you to submit the device for Failure Analysis.  Let me know if you want to proceed for that.

    If not, we can consider this as closed.

    Thanks and regards,
    Vamsi

  • Vamsi,

    I need that microcontroller to get back to a functionnal mode. Moreover, as I am working with the flash Api I also need to understand why issuing several asynchronous command made my flash unaccessible anymore.

    I'll submit the informations needed for further Failure Analysis.

    Thanks and regards,

    Quentin

  • Quentin,

    Regarding the flash API usage: Flash API guide clearly mentions that the user application should wait for the current command to be completed before using another command.  If not, the next command may modify the registers and state machine modes as per the new command - basically this is not the way to use the flash wrapper.  Below is copied from the flash API guide.

    I asked our boot ROM expert to take a look at your DCSM OTP space configuration to see if anything related to boot settings is programmed incorrectly.

    Regarding FA (Failure analysis): Please note that the FA process may erase the flash.  Also, you may want to mention that you need the device back since you mentioned you need it.

    Thanks and regards,

    Vamsi

  • Hi Vamsi,

    Let me know what your boot ROM expert reports after taking a look at the OTP space configuration.

    I shall return the microcontroller for Failure Analysis?

    Thanks and regards,

    Quentin

  • Quentin,

    Our boot expert will take a look mostly tomorrow.  

    You can submit for FA after our boot expert takes a look at it.

    Thanks and regards,

    Vamsi

  • Quentin,

    I reminded our boot and security experts to review the OTP snaps.

    Thanks and regards,
    Vamsi

  • Quentin,

    Our security expert asked for a snap of the DCSM config register space at 0x5F000 - 0x5F200.  

    Please provide that snap.

    Thanks and regards,
    Vamsi

  • Vamsi,

    thanks for coming back to me.

    Here are the requested snaps:

    Thanks and regards,

    Vamsi

  • Quentin,

    Thank you.  We will review and get back to you.

    Regards,
    Vamsi

  • Quentin,

    I discussed with our security expert.  We noticed that you have a non default JTAGKEY0/1 - that should be ok since you bypassed configuring JLM_ENABLE settings on this device.  

    However, for the device on which you are not able to connect:  If JLM_ENABLE is configured unintentionally, then JTAGPSWD also might have got configured by accident (just guessing) in your application (check your linker cmd/map file as well to see if anything is mapped to those locations in any of your application versions that you used when this issue happened).  

    I will assign this post to our security expert and he will suggest an experiment for you to try.

    Thanks and regards,
    Vamsi

  • Quentin,

    Based on the DCSM memory snaps which you published, it looks like you are scanning a JTAGKEY which is non-default. If the JLM_ENABLE is configured to a non-default value, then the scanned in JTAGKEY should match with the JTAGPSWD for the target/device to be connected. This could be one of the reason why you are not able to connect on the other device which .

    I would suggest you to try the following assuming you only have configured the JLM_ENABLE and have not disturbed the JTAGPSWD from its default value.
    1. Please follow the steps as per section 4.1 from the following application note (www.ti.com/.../spracs4) and update the JTAGKEY with the default values shown below.

    JTAGKEY[31:00] 0xffffffff
    JTAGKEY[63:32] 0x2bffffff
    JTAGKEY[95:64] 0x4bffffff
    JTAGKEY[127:96] 0x3fffffff


    2. Save the ccxml and then relaunch the target configuration file.

    Let us know if this will help you connect to the target.

    Just reiterating that this needs to be tried on the device which you are unable to connect.

    Thanks & Regards
    Pramod

  • Pramod,

    I have worked with the jtaglock before so I also had this idea. I already tried to connect with the default password for the jtaglock but it does not change anything. Moreover I already changed the microcontroller so I can not do any more testing on this device.

    Thanks anyway for your reply,

    Quentin

  • Quentin,

    Should I consider this as closed then?  Or do you have any other questions?

    Please let us know.

    Thanks and regards,

    Vamsi

  • Vamsi,

    If you don't have any other idea about how to get my microcontroller back to normal functionning mode, you can close this thread.

    Thanks and regards,

    Quentin

  • Quentin,

    You said you already changed the microcontroller and that you can't test it anymore.  Hence, asked if we can close this.

    If you are able to connect the debugger and run some code, we can do some experiments.  If you are not able to establish JTAG connection, I feel it is locked.  If you prefer to submit for FA, please proceed for that.  And notify them that you did not lock it but you are still not able to connect the debugger to it.  They will try and see if they can connect.  If they are able to connect, they may do tests that may erase and reprogram the device.  Meaning, your application code in flash will be erased (if they are able to connect and erase).

    Thanks and regards,
    Vamsi  

  • Vamsi,

    I changed the microcontroller I was not able to connect to anymore.

    I am still trying to get the other one back to work. It's the one I can connect to via jtag and that keeps running the program containning the wrong flash api calls. All the memory snapshots I sent you are from this microcontroller.

    I just want to be able reprogram it and keep using this microcontroller as a development station.

    Thanks and regars,

    Quentin

  • Quentin,

    On the device that you are able to connect, could you try running the C2000Ware flash programming example?  Note that this example uses flash linker cmd file.  Since you are not able to load to flash, I suggest to modify the example to load to RAM and then execute it to see if it is able to erase the flash fine.

    Thanks and regards,
    Vamsi

  • Vamsi,

    I won't be at the office until wednesday, I'll try what you suggest and get back to you.

    Thanks and regards,

    Quentin

  • Quentin,

    Sure, I will keep the thread in pause until then. Let me know how it goes.

    Thanks and regards,

    Vamsi

  • Vamsi,

    I modified the .cmd file in the flashapi_ex1_programming project so it does not use the flash :

    Still when I try to load the program, I get these errors : 

    Regards,

    Quentin

  • Quentin,

    Regarding the File loader: Verification failed error:  It still looks as if the codestart is mapped to flash in your linker cmd file.  Can you check your linker cmd?  

    Map it to RAM. Please take a look at C2000Ware_3_04_00_00\device_support\f2838x\common\cmd\2838x_RAM_lnk_cpu1.cmd

    Regarding the error initializing flash programmer:  You can ignore this for now.  Basically, flash programmer is not able to read the OTP correctly to identify the device (which is due to the issue that we discussed earlier).  

    Thanks and regards,
    Vamsi

  • Vamsi,

    Mapping the code start in RAM made the exemple possible to run.

    On the first attempt, I reached the Exemple_EraseSector() function line 235. Within the function, the oReturnCheck was equel to Fapi_Status Succes but the Fapi_getFsmStatus() returned a value of 0xC10.

    Since that moment, when I try to run that exemple again, I can't go further than the call to Fapi_setActiveFlashBank() (l. 222) returning the Fapi_Error_InvalidHclkValue error code.

    Regards,

    Quentin

  • Quentin,

    What is the wait state configuration that you are using?  

    And what is the operating frequency that you configured the PLL for? and what is the frequency value that you passed to the flash API?

    Thanks and regards,

    Vamsi

  • Vamsi,

    Since I am using the revision A of the control card, the microcontroller is clocked by at 20MHz oscillator. Following the instructions in the exemple, I added the "USE_20MHZ_XTAL" to the project. It's the only thing I changed.

    So as already configured in the exemple, I am using a wait state value of 3, PLL is configured for 200MHz operating frequency. And the frequency value passed to the flash API is 200 which seems correct since the value must be passed to the Fapi_initializeAPI() function in MHz.

    Regards,

    Quentin