TMS320F28034: Possible cause(s) of I2C bus hang-up

Hi Murdock,

From your description it sounds like a POR while reading from the EEPROM is possible. Some questions to help me better understand the situation below:

1. Are all of your devices are on the same board (F28034 master and the two slave devices) and therefore all powered from the same source? Is it possible to test with only one slave on the I2C bus to see if it's specific to the EEPROM/Accelerometer?

2. Are you running your application code from flash, if so do you have access to the JTAG to debug your program within CCS? It would be helpful to check the status of the I2C related registers at different times, assuming a POR isn't happening and the debugger can actually stay connected.

3. Are you able to capture logic waveforms on the I2C bus or probe with an oscilloscope? The waveforms would be helpful for seeing what's happening on the bus before/after the SDA line is being held.

If a POR is indeed occurring regularly and holding the I2C bus (SDA line) as a result, it's probably better to narrow down the cause and fix it rather than countering it within software. However, there are some ways of recovering within software as mentioned in the I2C tips you linked.

1. For master devices that mux the SCL/SDA pins with GPIO, the easiest thing is to configure the pins for GPIO operation and toggle SCL until the slave releases SDA. At this point you should be able to resume normal operation.

2. Many master devices don't mux SCL/SDA with GPIO since the I2C I/O cells are often special open drain cells. A workaround has been reported to work even on these devices. By configuring the I2C for "free data format" and then reading a byte the I2C will immediately start sending clocks to input data (rather than trying to send an address). This can be used to free up the bus.

3. Some slave devices can reset their I2C interface when the bus is hanging (e.g. the LTC4151, after 33 ms). Switch it on if that is not the default behavior.

Option (3) might be a good practice to try first, seeing if setting the IRS bit frees up the I2C bus. Otherwise option (1) could be implemented within your application after a POR.

Hope this helps,
Kevin

Kevin

Thanks for responding.

The I2C bus is only 2-3 inches long and all on the same PCB.  All of the devices on the I2C bus are powered by the same 3.3V LDO regulator.  This board is already in production and as a consequence the JTAG pins are not brought out on the PCB so unfortunately there is no way to attach to JTAG.  The board is normally enclosed in a metal box.  We have a custom designed CAN bus bootloader to allow the production units to be programmed following test via CAN.  We can read registers and monitor some points on the board via this CAN connection, but it is similar to debugging with a UART connection in that it is not real time.  The application runs out of on chip Flash and responds to CAN inquires in the background.  We have opened the metal enclosure on several units and probed various test points.

We have not seen any electrical noise on the input to the 3.3V  LDO regulator or any electrical noise on the 3.3V signal, so we think that the POR/BOR reset is unlikely (although possible).

Resetting the Microcontroller in the Middle of I2C Communications Hung the I2C Bus

(SCL is in Yellow on the bottom and SDA is in Red on the top)

Reset of the Microcontroller Occurs on the Right and SCL is OK (high), but SDA is left asserted (low)

This scope photo shows how we were able to hang up the I2C bus my manually resetting the TMS320F28034 during an I2C read of a serial EEPROM.

We were unable to create a free form packet to send out and reset the I2C bus using the I2C peripheral.  The I2C peripheral would see the I2C bus as busy and not transmit the reset packet.  We ended up having to disable the I2C peripheral, turning the SCL and SDA pins to GPIO, setting them as inputs to confirm that SDA was low, then converting them to GPIO outputs and bit-banging a packet: [START] [1] [1] [1] [1] [1] [1] [1] [1] [NACK] [STOP].  This was found to reset the I2C bus.  Just sending clock pulses did not do it.  The following scope photo shows the hung I2C bus, the bit-banged I2C reset, I2C bus unhung, and I2C communications started again.

I2C Reset Routine Clearing a Hung I2C Bus (SDA low (red) & SCL high (yellow))

Note: The I2C routine uses GPIO pins to bit-bang so the SCL bit-bang frequency is a little slower (~100kHz) than the microcontroller’s SCL (~160kHz). The I2C Reset Routine changes the SDA and SCL pins from I2C function to GPIO’s and sends out the routine. GPIO outputs are not open drain (like for I2C) and you can see where the master is driving the SDA line while the slave is holding it low (see the small pulses). Eventually the slave releases the SDA line and it goes high. At the end it changes the GPIO’s to inputs and reads the bus (both signals high). At that point it then changes the GPIO pins back to the I2C peripheral functions SCL and SDA and enables the I2C peripheral again.

At this point, we do not see a likely HW cause of this problem.  Granted we do see how a POR/BOR while operating could cause this problem if it occurred at the magic point of an I2C slave sending data consisting of a "0", but we have not been able to  see or cause such a problem on a board operating within it's design parameters.  

Since the I2C control registers are not write protected and might be overwritten by errant SW, is it possible to operate the I2C peripheral in a manner to cause such a problem?

If you were in the middle of reading data on the I2C bus and the I2C peripheral was disabled by writing a "0" into the IRS bit in the I2CMDR (I2C Mode Register) would this cause the I2C peripheral to stop in the middle of an I2C transmission/reception or would the I2C peripheral complete what it was doing before it could be disabled?

We are trying to determine if a SW issue/problem/error (such as a memory leak or errant pointer) that occurs on a pseudo irregular basis might over write an I2C control register and cause the same problem as a reset.  It turns out that the I2C software module used in the SW sets up the I2C control registers every time the routine is called (rather than only once on initialization).  Unless the I2C bus is reading a "0" from a slave when the I2C peripheral is disabled, it would not hang up the bus.  If the SW goes further into the weeds, the microcontroller might be reset by an NMI type reset or by the internal watch-dog (expecting to be touched every 6-12ms).  In that way the resetting of the microcontroller itself is not actually causing the bus to hang up, while writing over the I2C control register at the right instant may have caused it.  Because this is a production unit, the system does not monitor things like what type of reset occurred and how often is a reset occurring.  It is possible that the unless the application is instrumented and monitored, periodic resets by the watch-dog may not be noticed.  Hanging the I2C bus however will result in a system fault and would be noticeable.

Murdock

Hi Murdock,

It will be difficult to really know what's going on within the device's I2C module without having access to the register values. Maybe debugging over the CAN connection could be useful if you can periodically retrieve some register values (Like I2CSTR and I2CMDR).

The scope shot you provided is from your forced POR test, correct? Does the I2C hung case in your actual application look the same as what is shown, SCL stays high and is no longer driven by the master? Could be the I2C module is getting reset during the communication (I2CMDR.IRS being set to 0) since you said the registers are being touched each time the routine is being called.

1. Can you share what all is happening within your I2C routine and when in your application it's being called? What I2C registers are being set each time the routine is ran and to what values? Are they being set mid-communication? Also would help if you could provide your initial I2C init.

2. Probing the XRS pin while your application is running would give insight into whether or not watchdog resets are occurring (or if something else is driving it low).

When toggling SCL by itself on/off, you saw SDA stay low and it did not release? Only when you toggled both SDA/SCL it worked? Seems odd.

3. Can you share what accelerator and EEPROM you are using? They handle up to fast-mode at least?

best,
Kevin

Murdock,

I'd like to add some more info.

Resetting the I2C peripheral (I2CMDR.IRS is set to 0 and then 1) in-between transfers will clear the BB bit and the module will not know the state of the bus (I.E. if it's busy or not) until a START or STOP condition is detected. I imagine this same response is seen, though I'd have to test to be certain, when the I2C module is reset in the middle of receiving a byte. The byte you are receiving would likely be lost as well, which is worse.

There are some steps listed at the end of section 5.4 "I2C Status Register (I2CSTR)" in the F2803x I2C guide for re-discovering the state of the bus and correctly setting the BB bit, assuming the resetting of the I2C peripheral is occurring between transfers. There is a slight correction that needs to be made in a newer revision however, discussed in this prior E2E post: e2e.ti.com/.../696954

http://www.ti.com/lit/sprufz9

All of this being said, it's probably best practice not to reset the I2C module unless necessary. And if you do, it should be performed in-between transfers with the steps in the document implemented.

Best,
Kevin