• State
  • Locked Locked
  • Replies 5 replies
  • Subscribers 83 subscribers
  • Views 95 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMS320F28075: RAMUNCERR at field

Hang Yang
Expert 4156 points

Part Number: TMS320F28075

Hi Experts,

My customer is have RAMUNCERR error issues at field.

The ERROR happens randomly occur on products that deployed in different locations, so far the issue has never occur on the same device for a second time, and the issue is never reproduced in the lab.

The issue will cause a reset (due to NMIWD). Cusotmer would know about that there has been a reset occured on certain product, and they will go to the field to check it. The device would function normally after the reset, therefore by the time their FAE reach the product, the issue is not there anymore. They read the NMIFLG, and found that the RAMUNCERR flag is set.

There are about 20 events happend year to date, so they are now paying attention to the issue and try to find the root cause.  

  • Given the fact that the device would reset when the issue occur, is there any register that is preserved after reset, so that they can read it and know about what happend before the reset? (Apart from NMIFLG because they already read it). This will help customer get preperared for the field diagnosis when the next event happens.
  • Where the RAM is getting initalized? - One guess is that the RAM is not initalized after POR, and the data and ECC does not align. Accessing such location would certainly cause a UNCERR. All the variable are initlized in Cint00, so variables should be safe. However, there might be a pointer access that goes wild and read unexpected address. This would not cause UNCERR is the address has been initialized. So we are checking weather in ROM or user code that the full range of RAM are initilized, could you pleae help checking if the ROM initilize the full range of RAM?
  • What might be the casue of the RAMUNECC? In all we are looking for every possible cause UNCERR, so that we can deploy some kind of probe the verify it.

 

In the meantime, they can barely change the code of the shipped products because great amount of them do not have OTA function.

Regards,

Hang

 

  • 0
    TI__Expert 4156 points

    BTW, we've added some function to save the address of ECC error in the NMI, however, due to the limit in the OTA avaliability, this method can only apply to a very limited amount of device and we might not have enough sample to ensure the error can happen on the device that carries this function.

  • 0
    TI__Guru** 117461 points

    Hi Hang,

    Hang Yang said:
    Given the fact that the device would reset when the issue occur, is there any register that is preserved after reset, so that they can read it and know about what happend before the reset? (Apart from NMIFLG because they already read it). This will help customer get preperared for the field diagnosis when the next event happens.

    No, we do not have any other register. Is user has NMI Handler then they can dump the address and other info in some flash location and use it later for debug.

    Hang Yang said:
    Where the RAM is getting initalized? - One guess is that the RAM is not initalized after POR, and the data and ECC does not align.

    All the RAMs get initialized in ROM code so this should not happen unless they are corrupting some RAM location during diagnostic testing.

    Hang Yang said:
    What might be the casue of the RAMUNECC? In all we are looking for every possible cause UNCERR, so that we can deploy some kind of probe the verify it.

    This is difficult to say without getting more info.

    Vivek Singh

  • 0 in reply to Vivek Singh
    TI__Expert 4156 points

    Hi Vivek,

    Vivek Singh said:
    Is user has NMI Handler then they can dump the address and other info in some flash location and use it later for debug.

    They don't have this function in the NMI in the deployed product, and due to the limit in the OTA availability, this method can only apply to a very limited amount of device. 

    Vivek Singh said:
    This is difficult to say without getting more info.

    At this stage, we are still gathering info. However, given the nature of this problem, it would take a lot of time and cost to do one experiment. They need to wait for the issue to occur and check the machine on-site. Which is the reason they would like to be well prepared and understand the known possible cause of RAMUNCERR, so that they can check with purpose when the event happens. 

    Could you please help checking again on the possible cause of RAMUNCERR?

    Regards,

    Hang 

  • 0 in reply to Hang Yang
    TI__Guru** 117461 points

      Do you have any suggestion ?

  • 0 in reply to Vivek Singh
    TI__Guru* 86290 points

    I would look at two factors in the system:

    1)Elevation: Higher elevations will be exposed to more high energy neutrons caused by cosmic rays interacting with the Earth's atmosphere.  This neutron create an energy transfer that can cause a bit flip in the RAM.  The FMEDA for this device https://www.ti.com/drr/opn/C2000-SAFETI-AUTOMOTIVE-INDUSTRIAL-PACKAGE has a placeholder to enter the average elevation and will give an expected FIT rate for the RAM based on this.

    2)Power/ground plane noise from the system: Noise from the control system(switching of FETs, motor EMI, etc) can cause disruptions to the power/ground plane of the PCB, inducing transients to the MCU, which in turn can impact the device.  Think of this as not meeting the DS specs for VDDIO/VDD under certain conditions, basically operating the device out of spec for a brief period of time.  If this co-incides with an access to the RAM, the the read can be potentially disrupted.  The content of the RAM itself may not be impacted at all, i.e. if you performed another read it may not trip an ECC event.

    For #1, this might be the case if the customer are running their application at a very different altitude than where they developed the system.

    For #2, I would have expected customer to see this during their system development, but there are alot of factors that can contribute to this, including PCB design, decaps and decap placement, etc.  If that has changed since their development then maybe they would not see it during their validation effort. 

    Another culprit could be noise from some other component in the system that is not part of the customer's system but part of the end equipment.  Again, something disturbing the power/ground plane of the MCU while its accessing the RAM.

    From a historical perspective, is this a new system deployed for the first time to the field, or is this something that has been in production for awhile and customer is just starting to see this?

    Best,

    Matthew