Hercules RM46 I2C Unexpected Restart

Hi Dave,

  Can you please tell me what speed are you running the I2C and if you have the proper pull up resistor on the I2C bus?

Hi, Charles. Thanks for responding to my question. We're running at 100K baud and have 4.99K pull-ups on SDA and SCL. I'm investigating a possible error in the calculations which makes the baud rate actually be 102K, but I doubt this is a problem. I've tried slower baud rates without the problem going away. One thing I ought to mention is that we're using an LTC4312 as a mux between separate local and interboard I2C buses. I'm currently only using it on the local bus. On the local bus side, there are also 4.99K pull-ups and 100 ohm serial resistors. I'm attaching a piece of the schematic so you can see what I'm talking about.

Thanks again,

Dave

8510.I2CBridge.docx

6283.NoStop.xlsxHi, Charles. The reads and writes are interleaved, but there's about 15 milliseconds between them, so there should be plenty of time for the previous read to finish before the next STT is requested. I'm including a I2C trace (captured using a Beagle from TotalPhase) so you can see the timing and what it looks like. In the trace I'm attaching, each write is actually two bytes, so the readback is from address 0x01 rather than 0x00. I made it a two-byte write so I could see if the restart occurs after just one byte is written or waits until after the two bytes are successfully written. It does appear that the two bytes are written completely before the restart happens with the missing data. See what you think.

As I said, I don't have a lot of  control over the sequence of states in the driver, but I do have the source code, so I can fix it if we figure out what's going on.

Thanks,

Dave

HI Dave,

 I see the trace and I don't know why the start is not ended with the stop in the highlighted command.

 I understand that your driver is checking the BB bit. Is it possible for you to check the ARDY bit before issuing the next command. I want to know if it makes a difference.

 Do you know how much loading is on the bus? Will you be able to try 10k and see if it makes a difference. I reference the below figures. Depending on your loading and if less than 100pf, you may try a higher pullup resistor.

Hi Dave,

 I think we can exclude the pullup as a cause.

 Not sure if you had a chance to reference the flowchart. Below is one example using polling mode. Will you be able to cross check your code against the flowchart and see if it still created the extra restart bit? If you still have problem after following the flowchart then I think you might be hitting some problem that we were not aware of. The I2C module has been out there for a very long time, not just for Hercules but also in many of TI products. We have yet to come across the problem that you reported.

I do suspect that the problem lies in the driver, but I just can't find any bit of code that might be creating that situation. And I haven't been able to catch it doing anything unusual using the debugger and all sorts of breakpoints. The driver for the Sitara processor (which we're also using in the same system) is quite different, even though it also comes from Mentor graphics. I haven't looked at the flowcharts yet, but I'll take a look. Since the problem occurs on the evaluation board as well as our new hardware, I think we can restrict the likely causes to the driver. My hope is that you can tell me what the software would have to do to cause that behavior, or what sort of settings could allow the hardware to do that without processor intervention. I think I said this before, but the problem is intermittent, sometimes very rare, going hundreds of cycles without failing. On other occasions, it happens right away. So there's some timing aspect to it, but I just don't know what it is.

Thanks,
Dave

HI Dave,

  Until now I never asked what other things the CPU is also handling other than  the I2C. It is a good move to check if other activities can affect the I2 operation. The I2C interrupt is mapped to channel 66 on the VIM. It is of lower priority compared to many other peripherals. It may be possible that there are multiple higher priority interrupts become pending around the time when the I2C is having problem. 

Hi, Charles. So it turns out that the serial port activity DOES affect the I2C transfers, though I don't know what the mechanism is. Here's what I did:

1. Turned off the external computer sending commands to the serial port of the Hercules
2. Wrote a test program running on the Hercules to send the write 50 0 command over and over again on the i2c

With the commands running this way, there were no I2C errors whatsoever. I ran it for millions of transactions without problems, but when I started sending carriage returns on the serial port, I started seeing I2C errors. The occasional errors would persist for a few times after I'd stopped hitting carriage return. This was pretty surprising and I haven't been able to figure out why that would happen. But the errors would stop, and it could go millions more transactions without errors.

Some observations about the drivers:

1. The I2C driver disables interrupts during its Interrupt Service Routine
2. The serial driver does not disable interrupts during its ISR

So it looks like the I2C code gets interrupted when the serial interrupt happens (but not during the ISR) and when it returns from the serial ISR, sometimes it doesn't resume properly and creates the repeated start condition. I've tried to duplicate this by adding delays in the I2C code at "strategic spots", but I haven't had any luck causing the repeated start. The Start command doesn't appear to be getting called any extra times. My code checks to see that Starts and Stops are equal and we don't see two Starts in a row, and that never happens.

So maybe it's not really a repeated start, even though it looks like one. I dunno. It's pretty frustrating.

Thanks for your help so far!

Dave

Dave,

IRQs are automatically disabled by the processor upon entry into an IRQ exception.

If you see that the driver for I2C is running with interrupts enabled, that means someone implemented a special sequence of code for nesting.   (or if they didn't then it won't work right!)

What CPU mode is that ISR for the I2C running in?  If it is nesting - it would have had to change modes from IRQ mode to SVC or SYSTEM mode first, before re-enabling interrupts.   If you re-enable interrupts from inside IRQ mode then the IRQ LR would be corrupted when another interrupt interrupts the ISR.  

I could see where if this isn't being handled correctly - it would be the source of an infrequent issue ...

See this appnote SPNA219 by Christian Herget for details on what is required.   

Hi Dave,

  Like Anthony said, the IRQ interrupt (the I bit in the CPU's CPSR) is disabled upon entry into the IRQ vector. So you can check if the  Mentor Graphic's driver is re-enabling the I bit to cause other higher priority interrupt to preempt the I2C ISR from finishing. If this is the case, you want to keep the I bit disabled during I2C ISR. 

  One more thing you can try to is remap the I2C interrupt to a higher priority channel than your serial port interrupt. You can easily do this in the VIM module. There is the VIM Channel Mapping register. Currently the I2C is mapped to VIM channel 66. Try map it to higher than the serial port and see if that makes a difference. 

Hi, Anthony. Thanks for joining the investigation. You're right that IRQs are disabled for both of the ISRs. I checked the CPSR register and bit 7 is already set as it enters each of those ISRs. So when I said that interrupts were disabled in the I2C ISR, that was because they are explicitly disabled at the beginning of that ISR, but they aren't in the Serial ISR. It looks like that doesn't matter, though.

So I don't think any nesting is going on.

Hi Dave,

  Not sure if you are using the VIM API to manage the channel remapping. 

  The I2C is currently mapped to interrupt request 66 which by default also maps to VIM channel 66. The interrupt request connection is hardwired but the channel mapping is programmable. Supposed you want to remap this interrupt request 66 to channel 2 you can do the below. This example assumes that you are not using existing VIM channel 2 which by default is reserved for RTI interrupt. 

  vimChannelMap(66,2, &i2cInterrupt);  // remap request 66 to channel 2. The interrupt vector for I2C is i2cInterrupt

  vimEnableInterrupt(2,SYS_IRQ); // enable channel 2 as a IRQ interrupt

  Perhaps you can try this when you have a chance to get back to this. 

Thanks, Anthony and Charles. I'll definitely use this information when I get back to debugging this.

Dave