AM5728: AM5728 CPU failures

Stephen,

Chip Failure Analysis is not a prototype debug tool.  These types of latent failures are normally due to some type of electrical over-stress.  Let's start with some basic checks:

1. Are you using one of the mandatory PMIC implementations?
   1. If so, which one?
2. Have you implemented this such that the power-up sequencing AND the power down sequencing requirements are met?
   1. Do you have a hold-up circuit and a detection circuit that will tell the PMIC to initiate the shut-down sequence before supply collapse?
   2. Did you implement the clamp circuits, as needed, to make sure the delta between the 3.3V IO rails are never more than 2.0V above the 1.8V bias supply?
   3. Please supply scope captures showing both the power-up and power-down sequencing.
3. Are you making sure that there is no unexpected leakage paths?
   1. The IO pins are not failsafe - i.e., you cannot drive any of the signal pins when the supply for that pin is not stable and at the rated value.
   2. How is this managed both during start-up and shut-down?
4. From the basic level, what about assembly issues?
   1. You mention 3 out of 15 cards have failed catastrophically and 2 others have interface issues.  Were all 15 fully functional at one time?
   2. What percentage of the board peripherals do you have supported by test software?
5. Does you software, or a port of it, run on the EVM?  This could rule out a software cause of this issue.

Tom

Hi Tom,

The design is based on the TI schematics TI_AM572XEVM_REV_A3a.

1. We are using the TPS6590378ZWSR PMIC

2. a) I have been looking at the power sequencing and do not see any violations - I am including a few screen shots. There are no additional hold-up circuits other than what is on the EVM.

b) there are clamp circuits in the original design and I have verified that they are working

c) I've supplied a few here - if you want to see more I'm happy to include them

3. a) the entire design is run on battery and there are no other power domains. IE when the PMIC shuts off power there is no path to power a device that can drive the CPU

b) see a)

4. a) Not all were functional, we have one that has had SDIO issues from the start but other than that the remaining boards were originally working.

b) We are writing our own test software but it is written from the board functional perspective for manufacturing.

5. We used the X15 EVM's for our initial proof of concept and the software was developed on the test platform before we spun the boards. Our Linux guy isn't here at the moment, but I don't believe there is anything other than our custom hardware that would prevent our standard distribution to run on an X15.

I initially suspected contamination in the assembly process, but our CM swears that the parts came out clean when they were reworked. Another possibility is static damage, our environment is dryer than I would like and we have had problems before.

Thanks!

LF1.pdfLF2.pdfLF3.pdf

Stephen,

A few more questions:

• I am concerned that the slow decay to the 3V3 supply could corrupt the sequence but I do not see any violations in these pictures.
• Do you have a supply supervisor circuit that detects main power cut-off and initiates a PMIC shut-down, such as a TPS3808-33?
• How are you holding the PMIC VCC1 supply above 2.7V for the required 1.1ms after PMIC shut-down starts?
• Does your design have any clamps (such as using a TLVH431) between 3.3V and 1.8V?  If so, protecting which 3.3V VDDSHVx IO supply inputs?  Normally there is one on the VDDSHV8 supply rail and one on the other VDDSHVx supplies sourced from 3.3V.

The AM572x GP EVM was a very early low-cost EVM design which may not have had all of the protection circuits implemented.  The TMDXIDK5718 EVM at

www.ti.com/.../TMDXIDK5718 is a later design with all of these items implemented.

Tom

Stephen,

I have another concern.  You said that this is a battery-powered device.  How does it connect to other devices?  In general, these other devices have voltages and some type of reference to earth ground.  You are probably also connecting items like emulators and oscilloscopes that also impose an earth ground reference.  These types of development situations have been known to incur large currents through ground loops that destroy circuits.  In fact, we had this problem with the AM572x GP EVM when operated with certain external power supplies.  The solution may be to make sure that digital ground is strongly connected to earth ground through a strap or cable prior to attaching an emulator or scope probe.

Tom

* the same supervisor circuit is present on our design as the EVM, which includes the TPS3808-33.
* the VCC1 pin is connected to the PS_3V3 rail, I don't have a picture of the transition, but I'll take one soon to see what the typical hold up is. 3V3 is the output of the load switch of PS_3V3.
* the clamp circuits from 3V3 to 1V8, and SHV5 and 1V8 are also present in the design (they use the TLVH431)

I'll take a look at the schematics you referenced over the weekend and see if there are any significant differences.

The board we have designed either provides power to the peripherals or is connected by either ethernet interfaces. There are no powered devices that externally attach to our design.

We haven't had to use an emulator yet, but we do routinely attach scopes to the board. I'll think more about ground loop problems...

Do you have a UART console or Ethernet port? If not, how are you interacting with the board?

When you power the board off, are you running a "poweroff" command or are you just yanking out the power (battery)? I don't recall off the top of my head how this specific PMIC behaves in the case of sudden power loss. I've seen plenty of designs however where the behavior is dramatically different in the case where power is yanked out. If you are doing that often, and if the power supply isn't able to gracefully shut down, you might be damaging the device a little more each time you do it. Hence after a couple weeks of development they start failing.

Stephen,

UART cables, Ethernet cables, Scope probes, Emulators all can be sources of leakage current that can flow through the ground paths.  These currents can be surprisingly large and can easily damage the processor since it is probably the most voltage sensitive device on the board.  The digital ground needs to have a low-impedance path to earth ground before console ports or debug connectors are attached.

Tom

Hi Brad,

we have several UARTs in use in the product, there are two external ports which use a SN74LVC2G241 to interface to the FTDI usb-to seral cables we have. There is one internal GPS connection to a UBLOX device which is also 3.3V device.

I have spent some time looking at having the battery being yanked without a graceful shutdown, and so far I haven't seen any out of spec conditions. The quality testers have told me they try to use the power switch but occasionally they pull the battery first. I'll look more carefully to see if there is something not right when the battery is pulled.

Just to be clear, the power switch initiates a poweroff sequence in Linux.

If the issue is related to debug peripherals & scopes and leakage currents, I can certainly change behavior (and add ground straps) when debugging boards. I still won't know for sure what happened to the processors that I have replaced.

So far I am going to look at:
* power off sequences when yanking the battery
* go over the www.ti.com/.../TMDXIDK5718 schematics to look for updates and differences
* measure the hold up time for the PMIC when power is removed to make sure it meets spec.

I think I have found something suspicious.

I was measuring the hold up on the PMIC VCC1 input and it seems to not be meeting spec. I have attached a screen shot of the problem when I have pulled the battery from the system.

PMIC_RESET_IN (pin K9) is driven by a voltage monitor on the 3.3V regulator. It trips at 3.0V, which should initiate the power off sequence. The delta T from the time the reset is asserted and the 3.3V rail decaying to 2.7V is only 228us. This is no where near the 1.1ms previously mentioned.

It would be possible to generate a new PMIC_RESET_IN based on the battery voltage alone, but its Vin low is 7V (the first cursor in the PDF) and that would only buy me a few hundred more microseconds.

Can you point me to which document describes the value of VSYS_LO based on the PMIC that we are using? I couldn't find that since it an OTP register and not in the general spec.

I'm also looking for the 1.1ms hold up parameter - which spec is this in? Thanks!

8547.PMIC.pdf

Tom,
The VCC1 pin is attached to the PS_3V3 rail which is regulated from the battery - it is always on as long as the battery is attached. The red scope trace is the 3.3V connection to VCC1. The PS_3V3 net has 200uf of capacitance and is before the load switch for the system 3V3 rail.

One difference between the two designs is that the TMDXIDK5718 uses 5V (VMAIN) for VCC1 and the SMPS inputs, the X15 schematics uses the PS_3V3 for regulation and the VCC1 input. PS_3V3 looks to be the equivalent to your VMAIN net.

I'm going to look at adding capacitance to increase the hold up time, but I'm not sure I'll be able to reach the 1.6ms duration without an enormous amount of bulk.

Stephen,

There still needs to be a diode between the battery lead and the input to the the PS_3V3 switcher.  All bulk capacitance needs to be on the load side of the diode.  The SENSE input to the TPS3808 needs to connect to the battery lead.  When the battery is removed, the voltage on the SENSE input will drop instantly, long before the input to the PS_3V3 decays.

You will probably not need as much capacitance as you are thinking.  The PMIC RESET_OUT will go low as soon as the battery voltage drops.  This signal will drive PORz low thus stopping all active power consumption.

Tom

Tom,

Great - now I understand! I modified a board to monitor the battery input and used a spare schottky (probably not the final one, but close enough for bench work) in the power feed to isolate the input from the bulk and it looks great. Now there is more holdup than I need.

Scope trace attached. PMIC2.pdf

Thanks Tom, we appreciate the great feedback on the problem!