Unable to erase flash on TMS570 (TMS570LS20216, F035)

Jairo,

Do you have a way to try erasing the part using a native windows 7 environment?

The XDS100V2 emulates JTAG via bit banging, This is under a virtual machine a slow process that can end up with error while trying to program/erase the flash memory.

For the linux question, I need to run some test and see.

Jairo,

Is this just 1 device or a few devices?  Have they been cycled (program/erase) a significant # of times or stressed somehow or is it / are they 'fresh'?

The win7 VM might be a problem becuause of the older CCS and the XDS100v2 - the USB emulation that happens when you run in a VM is sometimes enough to cause the emulation driver to fail / timeout.

Not sure about ubuntu 14.04.   We've got a Ubuntu 12.10 machine in our lab and I can do a quick sanity check w. a 20216 on this ...

@Anthony, This is on a few devices, 4 or 5 different ones, but same board design.

@Jean-Marc, I'm sure I can a native Windows machine. I'll try that and report back.

Jairo,

We tried on Ubuntu 12.10 native and it is working with our USB board.

In your case, Ubuntu is running native or virtual?

Native Ubuntu 14.04

I haven't been able to check with a native Windows machine, but I did just try with Ubuntu 12.10, running CCS 5.5. This setup is used to program a different code base on boards with a TMS570 and it works fine with those projects and boards. When I tried my board, I had the same failure. 

Also, I would like to note that I find it unlikely that we've worn out the Flash. We've reprogrammed some of these a few times. Worst case scenario I would say < 100 times.

Could it be related to the code that I'm loading? The code I've successfully loaded and which I'm attempting to load now has Flash and RAM ECC turned on. Could ECC checking be interfering with the flash erase?

It doesn't sound like an ECC issue because you're getting *through* the erase eventually - right?  It is just that its taking more attempts than normal and you're getting errors initially... Like the flash is 'degrading' in performance rather than 'failing' like you might expect with an ECC error.

Our spec for W/E cycles between -40/105 with 15yrs of data retention is 1000 cycles so if you're at 100 then you are far away from that.

A long long time ago I had a customer that was doing EEPROM emulation on a small array which had a 100K w/e spec and we found there was a bug in their code that was 'looping' around write&erase so that they pretty much wore out the EEPROM within a couple cycles as viewed at the system level - so do need to ask if you think there is any possibility of that sort of problem with your code.  But if you don't have the Flash API built into your application I think we can probably rule that out.  

Since the problem seems tied to your board and/or software more than it does to the PC/emulator combination - maybe we should ask about the board.   Have you checked the power rails? There is a dedicated pin VCCP for the flash pump.  It requires more current during program/erase than it does during read so is it possible that you're not supplying enough current on this pin?

Also would be good to confirm the state of the Flash TP1, TP2 pins - they should be no-connects and not tied to anything like GND or power...

 EDIT:  Also what Freq are you feeding to the part (XTAL?) did you change from 16MHz.

It doesn't sound like an ECC issue because you're getting *through* the erase eventually - right?

Yes, eventually it gets through. 

 But if you don't have the Flash API built into your application I think we can probably rule that out.  

We don't use the flash API in this particular project.

 Have you checked the power rails? There is a dedicated pin VCCP for the flash pump.  It requires more current during program/erase than it does during read so is it possible that you're not supplying enough current on this pin?

I connected an oscilloscope to the 3.3V rail, set to single-capture with the trigger level at varying levels. When I set the trigger at 3.12V, I got a capture on 1 out of 4 erase attempts. On the scope, there was no apparent dip. When I set it to 3.04V, I was unable to capture anything out of the 6 times I attempted to erase. It looks as though we're staying within the recommended operating voltage [1] either way. 

Also what Freq are you feeding to the part (XTAL?) did you change from 16MHz.

We didn't change it from 16 MHz. We have a 16MHz crystal connected to the OSCIN/OSCOUT pins.

This morning I got a new error message while erasing: CortexR4: Flash operation timed out waiting for the algorithm to complete. Operation cancelled. Is it possible that a timeout in erase algorithm is too close to expiring each time? Is there a way to increase this timeout value? 

--

[1] The TMS570 datasheet recommends 3.0-3.6V

EDIT: The 3.3V rail on our board also feeds the VCCP on the TMS570

I just ran a new test on the board where once I got "through" erasing the board, I loaded a non-ECC project binary to the part. Subsequent (non-ECC) program operations have been successful on the part. I've gone through 5+ times without it failing. Erasing all of the flash goes quick. Our project only uses Bank 0 (per the linker script), so it probably doesn't actually do an erase operation on banks 1-3 since they are already erased. Does an ECC binary change this?

From what I understand it seems that for ECC enabled binaries, each program operation has to erase/write the whole flash device to write in the ECC bits (even if they're the same from a previous load) since they aren't 0xFF. Is that correct? Does erasing this part of the flash word add time to the erase/write cycle? Could this be part of the issue?

Hi Jairo,

I'll have to check into the timeout question but I don't think there is a straightforward way to adjust this through uniflash or through CCS.  

When you say that VCCP triggers but doesn't apparently 'dip' what exactly does this look like on the scope - is it steady averaging near 3.12V (instead of 3.3V?) and it happens to trigger because of noise or is it sitting around 3.3V and it triggers 3.12V but you don't see any samples that low?  

Also where did you attach the scope probe - was it at the device VPP pin?  That's where you'd need to be probing to take into account any issues that might occur due to routing.  

You might also try tacking a 0.1uF capacitor on to the VPP pin to see if this helps - whatever is quicker.  This might just rule the VCCP pin out as a problem rather than solving anything but right now my instinct says this is the most likely source of problems.

When you say that VCCP triggers but doesn't apparently 'dip' what exactly does this look like on the scope - is it steady averaging near 3.12V (instead of 3.3V?) and it happens to trigger because of noise or is it sitting around 3.3V and it triggers 3.12V but you don't see any samples that low?   

The latter is correct. The average, per the oscilloscope is 3.35V +/- 0.060 V noise. 

The probe is not attached to the VCCP pin, it's on a breakout board. Unfortunately we don't have a test point right at the VCCP pin, I'll see if I can solder a jumper. Also, there is already quite a bit of capacitance on the 3.3V rail. That pin, specifically has a 0.01uF cap next to it. 

Hi Jairo,

Strike my last reply then - our messages crossed and I didn't see the data point about the ECC.

Let me do some homework on the ECC part of the question - might take a day or two as I probably need to talk to the flash API owner ...

Jairo,

Also - let me see if I can recap the datapoints you've got now:

1) If your ECC enabled binary is already in flash,  then erasing is slow / takes multiple attempts / fails.

2) Once you remove the ECC enabled binary and program a non-ecc binary - you can cycle this quickly without any timeouts between program/erase.

3) sounds like VCCP pin probably isn't the issue based on 1) 2).

Is it possible for you to send us the binaries - If I can recreate your results before I talk to the flash library owner this would probably speed things up a lot.  

Also, when you were programming the non-ecc binary - did you have the option to auto-generate ECC during flash programming turned on?   There is this method and there is the linker method.  The former uses on-chip hardware to calculate the ECC value (same hardware that checks it) so it's sort of idiot proof whereas the linker ECC has options that you need to tell the linker about.

Another good thing to do might be to check your linker .cmd file - what are the ECC settings that you've got?

1), 2) are accurate and I would agree with 3).

I'm generating ECC data via the linker script; I don't think the option to auto-gen ECC exists in CCS 6.0 anymore.

These are the contents of my linker command script:

// Address of Flash ECC region
#define FLASH_ECC_ORIGIN  (0x00400000)

MEMORY
{
    STACKS  (RW) : origin=0x08000000 length=0x00001500
    RAM     (RW) : origin=0x08001500 length=0x00026B00

    // Embedded Flash (F035) with vfill specifier, used for ECC generation
    VECTORS (X)  : origin=0x00000000 length=0x00000020  fill=0xFFFFFFFF
    FLASH0  (RX) : origin=0x00000020 length=0x0007FFE0 vfill=0xFFFFFFFF
    FLASH1  (RX) : origin=0x00080000 length=0x00080000 vfill=0xFFFFFFFF
    FLASH2  (RX) : origin=0x00100000 length=0x00080000 vfill=0xFFFFFFFF
    FLASH3  (RX) : origin=0x00180000 length=0x00080000 vfill=0xFFFFFFFF

    // ECC sections
    ECC_VEC      : origin = (FLASH_ECC_ORIGIN+(start(VECTORS)>>1))
                   length = (size(VECTORS)>>1)
                   ECC = {input_range=VECTORS}

    ECC_FLA0     : origin = (FLASH_ECC_ORIGIN+(start(FLASH0)>>1))
                   length = (size(FLASH0)>>1)
                   ECC = {input_range=FLASH0}

    ECC_FLA1     : origin = (FLASH_ECC_ORIGIN+(start(FLASH1)>>1))
                   length = (size(FLASH1)>>1)
                   ECC = {input_range=FLASH1}

    ECC_FLA2     : origin = (FLASH_ECC_ORIGIN+(start(FLASH2)>>1))
                   length = (size(FLASH2)>>1)
                   ECC = {input_range=FLASH2}

    ECC_FLA3     : origin = (FLASH_ECC_ORIGIN+(start(FLASH3)>>1))
                   length = (size(FLASH3)>>1)
                   ECC = {input_range=FLASH3}

}

ECC
{
    algoR4F035 : address_mask = 0x003ffff8 /* Address Bits 21:3 See spnu489c.pdf, p279 */
                 hamming_mask = R4         /* Use the built-in R4 mask */
                 parity_mask  = 0x0c       /* Set which ECC bits are even/odd parity (spnu489c.pdf, Table 8-4) */
                 mirroring    = F035       /* TMS570 are built with F035 */
}

/*----------------------------------------------------------------------------*/
/* Section Configuration                                                      */

SECTIONS
{
    .intvecs : {} > VECTORS
    .text    : {} > FLASH0
    .const   : {} > FLASH0
    .cinit   : {} > FLASH0
    .pinit   : {} > FLASH0
    .bss     : {} > RAM
    .data    : {} > RAM

}

Jairo,

Ok this looks fine.  The only difference I see when comparing to the wiki page is that you don't have the algorithm specified in the ECC={} line,  but then the assembly manual says you don't need this if only one ECC algorithm is defined.  It may not hurt to try adding it but it shouldn't help.

EDIT:  ECC on download is still in CCS6 -  It is in the Tools->On Chip Flash menu and there is a checkbox Auto ECC Generation.  Not sure what would happen if you have that checked at the same time you have a linker generated ECC file.   (I see that it is not available for the LS20216 so scratch this - it's only available for newer products... )

I am still not getting a clear picture of what the difference in *ERASE* would be though due to ECC being programmed in the flash - because erase is by sector and erases flash data and ECC in parallel. 

Probably would be a good idea if you can send your .out files to do that.  If you can't then I could try to recreate.

What are you doing for the file that doesn't have ECC then - are you using a different linker command file without the ECC statements.

Jairo,

I may have reproduced the issue with a simple HalCoGen program that turns on ECC, using your linker command file.  So hold off on figuring out if you can send your .out files.  Need to try it a few more times.  Then I'll send the proj. to you and see if it acts the same way on your board.

I had done a 'Load Program" and "Reload Program" several times initially and didn't have any issue at alll.  Almost figured I hadn't reproduced the problem but then I opened the "On-Chip- Flash" tab and hit "Erase Flash".   Then I got hit with:

CortexR4: Erasing Flash memory...

CortexR4: Flash Programmer: Error erasing Bank 3, Sector 1. Operation Cancelled.

Subsequent erase did complete.  I'm going to try again a few times through the "Load Program" menu and not the Erase flash menu (the load program menu of course erases flash before programming, but since I didn't have any trouble this way I want to see if that was just a quirk or not...)

When you erase and it fails are you erasing through the "Erase Flash" button or just normally as a part of a program load?

Jairo,

Once I got the stick in a failing state it stayed that way - and I got erase fails no matter how I asked it to erase.

I was able to erase using the nowFlash utility that we no longer support (it has been replaced with Uniflash). 

Do you have one of our kits / DVDs thought - nowFlash might be on the disk and you could install / try from there - and that could be a workaround.

Also what code are you using to enable ECC?   I was using the .asm code from halCoGen:

_coreEnableFlashEcc_

        mrc   p15, #0x00, r0,         c1, c0,  #0x01
        orr   r0,  r0,    #0x02000000
        mcr   p15, #0x00, r0,         c1, c0,  #0x01
        bx    lr

    .endasmfunc

But the above code doesn't match what is published in our TRM - there are two additional bits that the TRM shows.  Need to track down whether this might be the cause - may be unrelated....

Great! Looks like we're converging on the same error.

I've erased both ways, as part of the program operation and through the "Erase Flash" button. I've also tried through Uniflash with the same outcome.

Our code to enable ECC is based on the HalCoGen output, although it has been modified.

I can probably dig up one of the DVDs; the issue would be that my development workstation runs Linux, and if I remember correctly nowFlash is a Windows application. 

Hi Jairo,

I changed the enable ecc procedure to match our TRM and I can still create the error.

Sorry for all the trouble we've caused you.  Unfortunately I'm at a point where I need to pass the test back to the developers to analyze further.   And the only workaround that seems viable in nowflash.  You are right that it only supports windows.  Hopefully you can figure out a way to work through this but if not let me know.

I put nowflash's last installer back up temporarily on this share for you to download.  Share should be open till Aug 31 2014.

https://txn.box.com/nfl342r3temp

(also in case anyone else has the same issue).

Hello Anthony, any updates on this issue?

Jairo,

There has been a lot of internal communication on this - but I haven't seen a resolution yet.

EDIT: there is a CQ ticket # for this issue SDSCM00050563.  So it's being formally tracked by the developers. The status of the ticket can be monitored here:  SDOWP by searching for SDSCM00050563.   Right now there are no external updates though as the problem is still being analyzed (trying to understand the differences between the tool that works - nowFlash - and Uniflash/CCS).

Anthony,

I think I may be seeing a similar problem on the TMS570LC4357. The LC4357 includes an L2FMC module with always-enabled ECC. The datasheet shows a maximum erase time of four seconds. My test program that follows the SPNU501F recommended erase flowchart takes 56 seconds to complete. I attempted to reproduce the problem using the Uniflash version 3.1.0.00026 linked from the HDK product page. Unfortunately Uniflash does not include a target configuration for the TMS570LC4357.

Is the timeout problem specific to the TMS570LS20216, or does it affect all chips with ECC flash?

Is there a version of nowFlash or Uniflash that supports the TMS570LC4357?

Thanks.

Matt Wronkiewicz

Hi Jairo,

Have you got any resolution for your reported issue? I am getting the same issue with same device LS20216 and trying to find out what could be the reason.

Thanks

Sanjeev

No resolution yet.  Based on Anthony's response it doesn't look like the ticket has received any updates.

Anthony, have you received any internal updates regarding this issue?

--

The link to the ticket tracker above seems to be broken, but you can see their bug tracker here: 

https://cqweb.ext.ti.com/cqweb/#/SDO-Web/SDOWP/RECORD/SDSCM00050563&noframes=true&format=HTML&recordType=IncidentReport

If it asks for username/password use "readonly" for both (without the quotes).

--

Jairo

Jairo,

No, also I am not setup to get automatic copies of replies to this thread so thanks for the heads up email.

I can inquire about the ticket - but the latest status would be in the CQ.

Did you try the Nowflash utility already?    If the BOX link doesn't work for you let me know.   It unfortunately is set to expire every week and I can't figure out how to override that - so if you didn't download it let me know and I can refresh the link.

Hi Anthony,

I have the same download issue with 570LS20216 device. I like to try with nowflash as you suggested.

When I click on the link it says "404 Page Not Found".

Thanks

Sanjeev

Hi Jairo,

Can you please let me know what frequency oscillator you are using in your custom board?

In our case, we replaced 20MHz oscillator with 12MHz it started to work but I am not sure why because 570LS20216 device manual says it supports 5MHz - 20MHz. 

Thanks

Sanjeev

Hi Sanjeev,

I reactivated the link so you should be able to download it again.

Hi Sanjeev, we are using a 16MHz oscillator on our board.

Hi Jairo,

We had the same issue because we were using wrong PLL multiplier value(0x59) in the PLL cntrl register.

Check your PLL control register settings per manual and try.

In our case, we are using 20MHz oscillator and manual says the multiplier value must be greater than 0x95.

Thanks

Sanjeev