TPS23880: Unexpected reading errors on I2C bus

Hi Jiri, sorry for the delay here as we currently are heavily queued on apps support. Are you testing on your own board or the EVM? If your own board, have you tried communicating on I2C on the EVM under the same conditions just see if you can replicate the issue? Thanks! 

Hi Darwin,

Testing was made on our own board. During the experiments, the signal integrity on both sides of the galvanic barrier was verified. At the same time, the communication was decoded by the oscilloscope to eliminate the possibility of misinterpretation in our SW. Theoretically, we can try to replicate the experiment using EVM, but currently my colleague from the SW department is on vacation, so I have to wait for him to return to work. However, we are sure that the communication is clear, not disturbed and at the level of I2C transactions does not meet expectations.

To explain how we discovered the problem - while testing our equipment, we observed occasional readings of unexpectedly low values from the current measurement register. First we tested whether there is a current drop on the tested POE device. When we verified that the power supply of the POE device is uninterrupted and stable, we focused on communication with the POE controller. After many experiments, we observed that the unexpected value when reading the registers corresponds to the value that is currently in the interrupt register. Another finding is that some registries are more prone to this misread than others.

Hi Jiri,

Thank you for your response.

As FYI I would like to let you know that our PSE expert is out of the office until the week of 8/30 and his response will be delayed until that time.

Best Regards,

Brett Colteaux

Hi Jiri, sorry for the delay here. This is quite unusual. Can you send the schematic for review? And just to know the scope of the issue, is this happening for all boards you have?

Hi Darwin,

part of schematic is attached. 

Issue was observed on all our boards which was tested.

Now my colleague preparing same test on TI EVM board.

Hi Jiri,

Thanks for updating and it would be interesting to see if the EVM also exhibits this behavior. If you can't replicate on the EVM, a couple of comments that catch my eye on the schematic, the 1.5k ohm pull ups seem low, typically I see 5k resistors. Also, the ISOW7841 looks like the minimum recommended voltage is 3V. This is pretty close to the 3.3V rail whereas we've used ISO7731 which has a minimum recommended voltage of 2.25V with plenty of margin for the 3.3V rail.

These comments might help with the debugging if you find the EVM working okay. It's the main hardware differences I can point to.

Hi Darwin,

Thanks for the suggestions, but 3V3 power supply rail is stable with a big power reserve. And of course - one of the first what we checked was signal integrity of I2C. The described behavior was monitored on an oscilloscope with I2C bus decoding. Therefore, we can say that I2C communication is clean without disturbing (from analog point of view) but the controller responds unexpectedly (looking at the decoded communication sequence). As soon as I have new information I will write to you.

Unlocking

Thank you for unlocking the thread.

We are able to reproduce the problem described using evaluation module BOOST-PSEMTHR-007 with daughterboard TPS23880EVM-008.
The EVM is connected via USB Interface Adapter (J2) to I2C bus of our device, the daughterboard has set I2C address to 20h.
We use I2C Register Access Configuration A (8-bit access).

Test case where we are able to reproduce behaviour the fastest way:
We switch on the EVM, switch on all PoE channels via Diagnostic/Manual mode by writing the following registers via I2C:
Address 20h:
  OPERATING MODE Register (12h) value 55h
  DISCONNECT ENABLE Register (13h) value 00h
  Power Priority / 2P-PCUT Disable Register (15h) value 0Fh
  2x FOLDBACK SELECTION Register (40h) value F0h
  POWER ENABLE Register (19h) value 0Fh
The same for address 21h.
Then we run code in our device, which reads DEVICE ID Register (43h) from I2C address 20h in the infinite loop and check if its value is 0010,0001b (21h).
Check of the value is OK most of the reads, but approximately once in 10 minutes we read 83h. In 10 minutes we perform around 500000 read cycles.
We suppose that value 83h corresponds to the content of INTERRUPT Register (00h) - it contains 83h after switch on of PoE channels and when we clear some bits by reading corresponding event register, we read this value with our read loop.

For faster reproducing of the problem all PoE channels need to be switched on and we read DEVICE ID Register, which we can simply check and the problem appears in shortest time.

Hi Ales,

Once the issue is reproduced, does the 0x43 register recover or it stays with 0x83? What's the value of register 0x41? Did you get any other faults from register 0x06,0x08,0x0A? Thanks. 

Hi Penny,

my colleague Ales is out of the office until the 3th Jan. But I will try to answer some of your questions.

Retrieving an unexpected value is limited to a single read. The expected value 0x21 is already returned during the next reading cycle. The error rate for the ID register is approximately 1 / 500k reads.

The problem is not limited to the ID register. Reading the DEVICE ID register is only the easiest way to induce fault reading (because we know what value is to be reading). But originally we discovered the error by reading current measurement register, when our test software reported current drops on the POE load. Based on measurements and subsequent analysis we found that the current and voltage on the load are stable, and the problem is caused by reading the wrong value from register. During these measurements, we captured I2C communication on an independent device. During the analysis of the communication, Aleš found that the unexpected value correlates with the current value of the interrupt register. I think he talked about the fact that different registries reading fails with different probabilities. In other words, some registries fail more often than others.

Thanks and have a nice day

Hi Jiri,

Thanks for your update. I will wait for Ales for answers to other questions. I will need his confirmation about 0x41 register value and fault register value to determine if there's any SRAM fault happened during this faulty stage. One thing I would recommend to add some delays between each read and see if the issue is reproducible. Thanks. 

Hi Penny,

There are answers to your questions:

- Register recovery:
As Jiri written, after reproducing of the issue, the 0x43 register recovers to default value 0x21 in the next read. Issue is repeating with average period 500k reads.

- Value of register 0x41:
For register 0x41 we observed the same issue - default value is 0x00, we read the value 0x83 once after several hundred thousand reads, register recovers to default value 0x00 in the next read.

- Faults from register 0x06, 0x08, 0x0A:
For registers 0x06, 0x08 we observed the same issue like with registers 0x43 and 0x41 - reading the value 0x83 once after several hundred thousand reads of expected value.
For register 0x0A we observed the issue too, for this register was the period of reading wrong value even shorter - we observed reading the value 0x83 once after several thousand reads.

We checked some more registers and we didn't observe changed value for register 0x1B - we read right default value 0x55
for more than 5 million reads.

We added delay 100 ms between reads and tried with register 0x0A, which has shorter period - issue is still reproducible
with the similar rate once after several thousand reads (only the time of test run is longer).

Thanks and have a nice day.

Hi Ales,

Looks like you didn't load the SRAM code into TPS23880. Can you please try to load the SRAM code to TPS23880 and see if the issue still exists? You can go to TPS23880 product folder and request access to the SRAM code.  Please follow this app note and reference code to load the SRAM code. Thanks. 

Hi Penny,

I loaded SRAM code into TPS23880, now value of register 0x41 is 0x02, and performed the same reading tests.
The issue still exists, but there is a difference for some registers:
 - Registers 0x43, 0x41 behave the same - reading wrong value once after several hundred thousand reads of expected value.
 - Register 0x0A behaves the same - reading wrong value once after several thousand reads of expected value.
 - For registers 0x06 and 0x08 I didn't observe changed value.

Thanks and have a nice day.

Thanks for the update Ales.

When you read the wrong value, is the wrong value always 0x83?  If that's the case, can you add a workaround that if 0x83 is read, do another read to double check the value? Thanks. 

Hi Penny,

We found that the wrong value corresponds with the value of the register 0x00 - it is value 0x83 during mentioned test setup, but it can contain other values, e.g. after clearing its bits it will contain 0x00 and we will read wrong value 0x00.
I added check for 2-byte registers with current (0x30) and voltage (0x32) and I observed the same issue like with registers 0x43 and 0x41 - once after several hundred thousand reads there was one byte changed, e.g. for the register 0x30 was right value 0x0000, wrong value 0x0083 and for the register 0x32 was right value 0x3803 and wrong value 0x3883.

A workaround based on another read may be possibly used, but it will be more complicated in this case.

Is it possible to reproduce this issue in your facilities?

Thanks and have a nice day.

Hi Ales,

My apologies. Our team is very tight on bandwidth and we are not able to create the scripts and run this test. In short term, is it possible to check if the register values includes 0x83 in lower nibbles or higher nibbles to identify the issue and do a second register read. Thanks.

Best regards,

Penny

Hi Penny,

as mentioned earlier, the unexpected value read from the register is the same as the value of the interrupt register. The value 0x83 is in the interrupt register after the chip is turned on, but during operation the value can be any other. Therefore, validating value if 0x83 is readed from register is not an acceptable fix of problem. For example - if we handle all pending interrupts, unexpected read value will be 0x00.

Jiri

Hi Jiri,

Is it possible to run the test with TPS23881? It is a newer device which is pin to pon with TPS23880 (only need to change Rsense to 200mohm). The FW should be compatible except the current reading factor and the SRAM code. Thanks.

Best regards,

Penny

Hi Penny,

Yes, we can run test with TPS23881, if You can send me TPS23881 sample. The component is currently unavailable anywhere from standard distributors.

Jiri

Hi Jiri,

Can you work with your local TI representative to submit a ZS order request on TI.com? Thanks. 

Hi Jiri,

I have contacted TI local representatives and they should reach out to you to support the sample request. Thanks. 

Best regards,

Penny