BQ25896: Stops outputting VSYS and won't exit "unknown mode"

Hi Florian,

Thank you for reaching out via E2E. Can you please help to answer a couple questions about your system. 

-What do you use as the pullup rail for I2C lines (SCL and SDA)? 

-Do you set REG09 bit 3 BATFET_DLY = 0b or 1b before turning off the Q4 BATFET to enter ship mode?

Best Regards,

Garrett 

Hi Garrett,
thank you for your response!

SCL and SDA are pulled up by 10k resistors. The voltage used for these pullups actually comes from SYS. It is getting converted to 3V by a different IC.

BATFET_DLY is 0b before entering shipping mode.

We actually do not know if we entered shipping mode on purpose in this error scenario. At least the device isn't supposed to. The only reason entering shipping mode there would be a firmware bug on our side.

Two actions could cause the device to enter shipping mode on purpose:
1. User enters shipping mode (which was not the case)

2. Battery voltage is too low (which was not the case)

We never had problems waking up from shipping mode when entering on purpose in these cases.

The beta testers said that the device suddenly stopped working. The battery wasn't anywhere near empty (some said 50%, some 75%).

Hi Florian, 

Thank you for the further explanation and answering my questions. I was initially under the impression you were purposely entering ship mode based on the thread title. I now understand that is not the case. Can you help to answer a few additional questions. 

-You say you observed the issue after not looking at the device for ~5 days. What are all register values on the BQ25896 when it is last working as expected? 

-You mention you are not intentionally entering ship mode and the battery voltage is above the over discharge threshold (max 2.5V). Is your expectation that the battery will continue to power components connected to SYS for the ~5 days where the device is not touched? 

-Based on you observations something is disconnecting BAT from SYS. Is there any protection circuit in your system which can disconnect battery from SYS output?

We need your help to try to catch when the battery is initially disconnected from SYS to determine exactly what is happening.  

Best Regards,

Garrett 

Hi Garrett,
you're right, the title was misleading. I changed it now.

Garrett Kreger said:

You say you observed the issue after not looking at the device for ~5 days. What are all register values on the BQ25896 when it is last working as expected? 

Unfortunately there is no simple way for me to read the registers since the connected microcontroller isn't powered in that case. It would be possible by connecting another microcontroller with special firmware, but if it's not absolutely necessary, I'd like to avoid doing that. I could give you a list of the register values as I'm setting them on a restart. There's only 1 bit that's possibly getting modified on purpose during runtime (except from shipping mode) if I'm not mistaken and that is OTG_CONFIG in REG03.

Garrett Kreger said:

-You mention you are not intentionally entering ship mode and the battery voltage is above the over discharge threshold (max 2.5V). Is your expectation that the battery will continue to power components connected to SYS for the ~5 days where the device is not touched? 

It should power them as long as the battery lasts. There no such thing as a power saving / sleep mode in our device. But the expected runtime (even without touching the device) is just about 30 hours. What we cannot tell is what actually happened right after the "crash" (if SYS stopped outputting in the exact moment of the crash or later on).

Garrett Kreger said:

-Based on you observations something is disconnecting BAT from SYS. Is there any protection circuit in your system which can disconnect battery from SYS output?

There is a fuse between BAT and the battery, but the fuse was good in every case. We always measured a voltage of about 3.3V on the BAT pin of BQ25896 on these devices.

Best regards,
Florian

Hi Florian,

If the charger IC is completely shut down (i.e. disconnect both VBUS and VBAT power sources), then re-apply adaptor, will the device work normally?

Thanks,

Ning.

Hi Ning,

in the error case we don't know. We only have one device left in that state and we're protecting it as if it's the holy grail, since maybe we could do important measurements on it.

in the normal case that works, yes.

We will try it if we get another device that's in this error state!

Thank you,
Florian

Hi Florian, 

Thank you for answering our various questions. Can you help to elaborate on one point you make. While the device is in battery only operation OTG mode may be enabled is that correct?

Additionally for reference can you provide the list of register values you do set at restart. 

Regards,

Garrett

Hi Garrett,

yes, that's correct. OTG mode is enabled for battery >= 75% and disabled for < 75%. As i said, some customers reported 75% and some 50% (there are only 25% steps). In that range it is quite likely OTG mode was enabled/disabled. There is a hysterisis for switching it on an off, so no "on/off - massacre" should happen.
I tested switching OTG mode on/off and in general it's working fine, I found no issues. But maybe it won't in some cases.
Maybe it's something like: 75% => 50% => OTG disable => error.

Yes. At the startup we perform a register reset by setting REG_RST to 0b1 in REG14. After that the registers are getting set like that:
BG25896_registers.xlsx

Hi Florian, 

Thank you for the further explanation and providing the initial register settings.

What I believe is happening is the BATFET is somehow being turned off while in battery only mode. You mention you are not purposely trying to enter ship mode, but a condition such as high load current or low battery voltage are possible causes for BATFET to turn off while operating from battery only (i.e. no adaptor connected). 

It would be helpful to understand how the load profile changes in the system with and without OTG/boost mode enabled. Are large current spikes expected? Are drops in BAT pin voltage due to load current transients ever observed? 

Regards,

Garrett 

Hi Garrett,

You're correct, the problem definitely happened in battery only mode.

We don't expect voltage drops or current spikes when enabling OTG/boost mode. In every error case no external device was connected to the supply that is created by enabling OTG/boost mode.
I tested that assumption by measuring the battery voltage and current (20mOhm shunt) with an oscilloscope. Neither voltage nor current did change at all, when enabling/disabling OTG/boost mode.

We also measured the current in the same state the device was, when running into the problem. We did not see any remarkable changes at all. Constant 220-270mA.
Don't get me wrong, we didn't actually see the current when running into the problem, but we do not expect any load current transients at all.

Thank you,
Florian

Hi Florian, 

Thank you for the explanation of your setup and conducting the tests relating to VBAT and current draw for us. Given this information we are doing some investigation on our end and we will get back to you. 

Best Regards,

Garrett 

Hi Garrett,

thank you for your help so far!
FYI: I won't be in office next week.

Best regards,
Florian

Hi Florian, 

Understood, thanks for letting us know. 

Best Regards,

Garrett 

Hi Garrett,

I'm back!

Have you got any new information on this topic for me?

Best regards,
Florian

Hi Florian, 

We have done some testing on our end to see if it was possible to observe BATFET turn off during battery only operation due to load transients as well as enabling/disabling OTG. Unfortunately, we have not been able to observe such behavior. It is difficult to determine a root cause when moment error occurs is not captured. 

Regarding your initial report of the device getting stuck in an "unknown state" the behavior appears to match an issue on this device where it can get "locked up" due to entering shipmode in the middle of an I2C communication. Typically this will happen when the I2C pullup rail is SYS. In this situation the BATFET turning off can potentially lead to an I2C communication being incomplete, which will cause the device to not be able to exit this state by plugging in an adapter. 

On the few units where you see the behavior my best guess is something currently unknown is causing the BATFET to turn off. When the BATFET turn off happens at a certain timing the BQ25896 can get "locked up" and an adapter plug in does not work to start up the converter. 

Best Regards,

Garrett 

Hi Garrett,

thank you for your explanation!

Just to make sure I understood you correctly:
There is a known bug which results in the same behavior, but the root of the problem seems to be another for our devices. The actual root in our case is unknown.
Is that correct?

Important information:
BATFET should not intentionally be turned off by our firmware at the moments the problem has happened. The only reason for such a behavior would be another unknown issue (f.e. firmware bug on our side). If we turned BATFET off intentionally the device never got stuck (we have done that without any issues for 1000+ times by now).

Is there any workaround to avoid the problem? We're still stuck and exposed to great pressure.

Can we give you more information to resolve the issue?

Best Regards,
Florian

Hi Florian, 

Let me help to clarify. 

There is a known bug which results in the behavior you reported where BQ25896 device gets in a locked up state and an adapter plug in won't power on the outputs as expected. The root cause for this bug is an incomplete I2C communication at the time the device enters shipmode. A verified fix to this issue is to set BATFET_DLY = 1 and ensure your MCU host sends no I2C communications after setting BATFET_DIS = 1 to enter shipmode. 

Given all the information provided my best guess as to what is happening in your system is that on the 5 devices where you see the issue the BATFET happened to turn off in the middle of an I2C communication. 

For your situation it is difficult to resolve because you have reported you are not intentionally turning off BATFET.  Since it is unknown when BATFET will turn off it is much more difficult to implement a workaround.

Florian Moser said:

Can we give you more information to resolve the issue?

To resolve the issue we need to determine what is causing BATFET to turn off. If we understand what is causing BATFET to turn off then you can actively avoid putting the device in such a condition. 

The key information we would need for your end to find a workaround is for you to be monitoring the device(i.e. monitoring/logging parameters such as VBAT,VBUS, VSYS, and battery discharge current) when the unintended BATFET turn off occurs.

Best Regards,

Garrett  

Hi Garrett,

thank you for clarifying, now i got it!

The last days we observed a behavior which possibly could turn BATFET off unintentionally (we needed time to confirm that).
On a very rare occasion we saw some I2C messages are getting misinterpreted by our microcontroller (BQ25896 sent them correctly). What we suspect is that our microncontroller could also send I2C messages incorrectly which could lead to turning off BATFET.
It was a timing issue which we fixed by reducing the I2C clock.
Maybe that was the root of the problem...

What we are planning to do now is to update the firmware on a huge amount of devices to test if that fixes it.
I'll keep you updated on that, but these tests will take some time.

We will also implement the logging parameters. The problem with that is that logging is not possible for production devices. We can only observe the logs on our development devices. It's very unlikely to happen given the small number of development devices we have.

Best Regards,
Florian

Hi Florian, 

Thank you for the update. Interesting to hear that you may have identified the root cause for the BATFET turning off. Please update me in the thread once you have results with the firmware update, but no rush.

If you no longer observe BATFET being unintentionally turned off when device is powered off battery for a long period of time then you should also avoid the issue of device entering the locked up state described in my last couple of responses. 

Best Regards,

Garrett