TPS65987D: I2C bus unstable. Is this SMBbus?

Some more info:

Connected logic analyser with infinite memory instead of scope with 'dead moments'. Also change driving SW to create an increasing gap between message blocks. Now I can reproduce effect perfectly. If there is a gap of ~20msec between blocks of bytes the PD controller will return a NACK at some time. Can be at first byte after gap, can also be after first giving normal ACK for fist bytes in block. Then suddenly return NACK..

Some screenshots:

(1) Typical register 9 write with 64 bytes (66 write bytes total):

• Start condition follwed by first block
• Gap (preparing next block + creating some logging on result)
• 2nd block + creating some logging
• Stop condition

When the gap < ~20msec all is fine.

(2) SW was made where gap (which has a lot of jitter) is increased by 1msec after each message that succeeded.

(gap in blue annotation, messages are repeated with increaseing gap until 'breaking point')

(3a) When gap is to big ACK is no longer given after some time:

In this case a bus-idle period of 16.15msec is fine, 18.03msec cause a 'delayed' fail

(3b) Normal end. Each byte in payload has the first nibble as an incrementing number. The 2nd nibble is always 0101 (0x05) so ACK can be verified easier.

(3c) However after a long enough gap, suddenly no ACK is given anymore. Not even after a number of bytes were ACK'd ok. This is the burst above:

(4) a more typical (and expected) failure (in case there is indeed SMBbus instead of I2C) is this instance. In this case the idle-gap is longer:

Whole message:

Last packet, first byte fails:

Black-box behavior speculation:

It seems that:

• Bus has indeed SMBbus behavior
• The idle-detection timeout is 16~18msec
• It takes about ~6 msec before the bus actually releases (SDA pull-low FET disabled)
• The time-out function is seperate from data communication. As a result in the 'dead time' above, the bus operates normally. After this dead time the bus stops for no appearent readon

(5) dead time

Open question:

Can someone confirm the speculation above or provide any other information on this instability?

(note: this is experienced with a non-configured device in PTCH mode, configuring fails. Not verified with configured device)

Mark,

The none of our PD controllers implement full SMBbus behavior, but they do implement clock stretching to allow higher priority tasks to complete or for bus translations to happen.  We will hold SCL low for various periods of time during different commands to support internal timing.

If the device is in PTCH mode, then the only traffic that the device is expecting are the commands and data related to the PBMx flow, so any other traffic may cause issues.

Is this process happening in the PBMx flow?

What do you have your timeout set to?

Your logic analyzer appears to be a Saleae.   If this is the case, can you post the .sal file of the traffic that is failing.  This will make it much easier for me to look into what is happening.

Regards,

Chuck

Hi Chuck,

The first paragraph starts with a strange sentence, but I expect that you mean to say is that the PD controllers have some SMBbus function implemented. So it isn't strictly I2C (NXP UM10204 specificiation) and not strictly SMBbus. Holding SCL low for clock stretching is valid I2C behavior but isn't the causing any problems here. The problem here is that there is no ACK coming from the PD controller.

The device is in PTCH mode, I don't know what you mean with PBMx flow. This is not mentioned in the TRM (SLVUBH2B). However I tried several random I2C bulk writes that fail in similar way, can write ~150 bytes when I limit the size of gaps as much as possible. Still seems to fail in these cases due to the gaps as I can not fully control these. In this case I reduced the test set to valid write commands of register 0x09. So I assume this should rule higher level traffic issues?

The time-out is something I wasn't aware of. I assume you refer to register 0x27, bit 14:12. The device has no EEPROM connected and only POR'd so I assume the value of 0x27 is still 0 (not at the lab right now). The table in the TRM shows 50msec as default and 25ms as POR value.... But 25 msec seems to be in the same ball-park. The function is not described futher, but from my experiments 'time-out' detection is ~18msec and 'time-out+activation' is indeed ~25msec. This smells very much as the problem I am experiencing.

When I am back in the lab I will try with some other setting. I am expecting that the issue I am running into will change timing behavior with this setting. In that case we can confirm that the PD controller has SMBbus like behavior and we are not running into some mis-understood problem that will bite back later when we modify our I2C driver. A further improvement for us will be to set the time out to as long as possible.

As a side note, also for other readers of this post, pure I2C behavior is to have no time out. Would be nice if this would be an option, also state clearly in documentation that this is not true I2C but has a time out. At this time it states in the datasheet that minimum I2C speed as slave is 0kHz which is DC which implies no time-out... Not having a time out has major benifits on I2C communication that requires buffering or a multitaksing MCU. It also allows debugging the SW as it is possible to use breakpoints without corrupting on-going I2C communication. It is clear why TI has put a time-out in their implementation. As anyone may pull low SDA or SCL there is no bus recovery mechanism when SCL is stuck low by some misbehaving device.

The analyser used is indeed a saleae, however I could not find a way to post this file. From the form I can only post images or such. This file is rejected.

Thanks so far, I'll update you when I have checked the I2C time-out setting.

Mark

Mark,

The clock stretching that I am talking about is defined in the NXP UM specification.  It is an optional feature, but I have not seen a MCU that does not work correctly with it.

If you have no Flash or EEPROM, then you are using what we refer to as the PBMx flow to load the patch and configuration the device.

You should be able to attach the .sal file by clicking the Insert button on the bottom of the edit window and selecting the Image/Video/File option and attaching the file.  I can debug this issue quickly once I have that file.

Regards,

Chuck

Hi Chuck,

In that case yet we are using in PBMx flow.

I tried uploading the file but it failed (already tried in my original post). When placing it in a zip file it uploads.

The file contains 3 seperate tests in which I generate messages with increasing delay between the blocks. I stop when I receveive a  NACK.

The 2nd stop condition on the bus is something I force from master side as a recovery mechanism which is I do on any I2C error as I am not sure at which point the I2C communication stops. This can be ignored.

I work in the EU, so in ~ 12hrs I will be back in the lab to test the I2C time-outs but I am quite confident this is the root cause/solution.

TPS65987 I2C SMBbus like fail.zip

Hi Chuck,

Cause confirmed & Solution found !

In short, my black box assumption seems to be correct. There is a time-out on I2C. This can be changed up to 1 second so in practical cases this will not be a problem.

Writing following data before staring configuration sets time-out to 150msec:

I2C.WriteRepRead(0x23, // [U8] Device_address
 {0x27,0x0E,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x50}, // [U8-1D] Write_data
  0); // [I32] Read_count

However there are some notes to be made...

(1) The datasheet mentions I2C 0 kHz operation, IMHO this turns out to be false and should have note that there is a timeout.

(2) The time-out isn't pure. It detects a timeout ~6msec shorter than set and then waits 6msec before it kicks in. In this dead-man-walking period the I2C is responding normally which makes it hard to find out it is a time-out

(3) When the device is not configured (PBMx flow?) register 0x27 returns 'no data' suggesting it doesn't exist. However it can be written and the I2C time-out is operating normally

A deeper dive into my experiments and results (see attached zip file):

First baseline, increasing gap between blocks as before, still fails after ~20msec) Device unconfigured - 6 attempts (after POR).sal

When reading/setting/writing register 0x27 when device is unconfigured mode, reading register 0x27 returns length 0x00  Device unconfigured -Set I2C timeout 1sec (read-write-read 0x27).sal

Now with I2C timeout set to 1second the device works fine with gaps up to 1 second Device unconfigured - many attempts (I2C timeout 1sec).sal

Changing time out to 150msec causes fail after 143msec Device unconfigured -Set I2C timeout 150msec (write).sal Device unconfigured - 2 attempts (I2C timeout 150msec).sal

Protoworkbench:

- log with configuration command for 150msec: Configure I2C to 150msec - log.txt

- Memory dump unconfigured device: Memory dump - unconfigured.txt

Device configuration now works fine and I2C remains working OK Device configured I2C timeout 1sec.sal

So thanks for your support. Setting I2C time-out is doing the trick for us.

(I will leave the tread open for you to close and make some final notes on your behalf)

Kind regards,

Mark

2022-03-11 I2C timeout setting experiments.zip

Mark, 

I will review these captures early next week and get back to you with my findings.

Regards,

Chuck