AM3352: Custom board power failures

We have been trying to figure out a problem with a Sitara AM3352-based design for a weeks without success - problem description and known debugging info so far are provided below. Appreciate it if anyone has any insights, pointers.

Problem description

The basic problem is that the device reboots with a power-on-reset; there is no discernible pattern to the reboots, time between reboots ranges from 30 minutes to 36 hours, over several hundred reboots tested with about 8 units. Kernel panics, software crashes etc. are ruled out as causes, processor reports power-on reset and indeed, as would be seen from the detailed info below, the processor does get a power-on reset.

Design background

The device design is almost exactly the same as the latest production release Beaglebone Black Rev C. Basic design elements:

• Processor: AM3352BZCZD30
• PMIC: TPS65217C
• 4GB eMMC flash
• 4Gb DDR3-1866 SDRAM
• USB0 used as OTG port
• USB1 hooked up to an up-stream hub, which feeds Ethernet and USB host ports
• TI WiLink based Wifi/BT/GPS module interfaced over SDIO and UART
• Miscellaneous I2C and serial port peripherals

The system software is Android N.

The AC input to the PMIC is fed from an external regulated 5V supply. The USB input can be provided, but by default the USB power path in the PMIC is disabled. There is no battery (note: I am aware of the VIN brownout issue in the battery-less case with TPS65217, as will be seen below, our problem does not match that scenario).

Power rail configurations of the PMIC are exactly as in the Beaglebone Black.

Detailed debug info

The trace below shows the conditions leading up to the problem.

Channel 1 (yellow):  3.3V rail from LDO4 of the PMIC
Channel 2 (blue): PGOOD signal from PMIC (which gets fed to the processor reset)
Channel 3 (purple): SYS output from the PMIC
Channel 4 (green): 5V AC input to the PMIC

As is seen from the traces, the problem begins with the SYS output dropping linearly from 5V all the way to 1.8V --- along the way the 3.3V LDO of course starts tracking the SYS output since it is fed from it. After dropping to 1.8, there is a kind of shoulder where it dips further below. PGOOD follows this further dip, but still stays high. Eventually (right edge of the trace capture), PGOOD is de-asserted, which resets the processor, after which all rails return to normal, the processor boots again and life goes on.

AC input to the PMIC is solid throughout. 

This is different from the VIN brownout issue addressed in an app note and discussed elsewhere: a) there is no VIN brownout, and b) that issue leads to the PMIC falsely detecting the presence of a battery and not detecting the AC input recovery, thus staying in the OFF state and not recovering without a power cycle. We do not have this issue, the system recovers itself every single time, there is no "lockup."

By default the PMIC power path configuration is: AC enabled, USB disabled, both AC sink and USB sink enabled. The current limit on the AC input is set at maximum 2.5A.

Some points from debugging so far:

1. There is indirect indication that the AC input current limit is being reached, and current draw demands exceed the 2.5A limit. We can reproduce the same reboot behavior and traces above by putting a large external constant current load on the SYS output which exceeds the 2.5A limit.
2. Obviously, system consumption under normal conditions is far below, of the order of few hundred mA under normal conditions, so some fault condition triggers this high current demand.
3. When PGOOD eventually gets de-asserted, and the processor is reset, the condition clears, and the system recovers, so this indicates the fault could be cleared by resetting the processor state. It does not persist.
4. In our indirect load test, with the load applied between SYS and ground, the SYS output gets driven all the way to ground; however, the above trace in the real devices shows that the SYS voltage drop stops at about 1.8V. Perhaps this indicates a higher voltage rail shorting to 1.8V somehow.
5. Of course, there is nothing obvious in the design that shorts any power rails together, so any such condition would have to a transient event created in a running system

If any further information, schematic snippets, software/driver details etc are needed on specific questions or areas of the design, I'll be happy to supply.

If anyone has any insights or information on the following questions first, any help is much appreciated:

• has anyone seen this kind of behavior or has had a similar experience with Beaglebone Black or similar designs?
• would the above be the expected behavior under the TPS65217 PMIC's AC current limit  conditions?
• is it reasonable to infer a power rail short from a higher rail into 1.8V from the above behavior?
• any other pointers inferred from above...

Thanks,

Best regards,
Anand.