This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

DRA712: I2C stuck and controller timed out on DRA712

Part Number: DRA712

Tool/software:

Dear TI Team,

 

We are facing DRA712 I2C bus stuck and I2C controller timed out on our DRA712 based custom board.

The system is running yocto/linux. Kernel version 4.19.73.

 

Linux Kernel log = "omap_i2c 48070000.i2c: controller timed out"

 

 

As shown in the partial block diagram above, the other 2 chips connected on the same I2C bus are TI's DS90UB954 and DS90UB953, which are FPDLink-MIPI_CSI SerDeses.

 

During normal operation, there is some periodic I2C traffic between the two SerDeses every 20ms.

The SoC I2C controller timeout is seen when the SoC tries to access the DS90UB954. Its just a single byte write operation. The issue occurs around 2% of the times SoC accesses the bus.

Using protocol analyzer, we found that during the issue, the I2C bus was stuck for around 1 second.

This stalls the periodic I2C communication between the two SerDeses causing issues in the overall system.

 

  1. Please help us understand what could be causing this issue when the SoC tries to access the I2C bus.
  2. And what can be done to avoid this issue.
  3. Also, in order to reduce the time the bus is stuck, we are considering reducing the value of OMAP_I2C_TIMEOUT (in linux driver i2c-omap.c) which is currently set as 1sec to a few milliseconds.
    1. Do let us know if it's safe to make this change.

Thanks and Regards,

Anand Mistry

  • Hello Anand,

    Does this only happen in the bus that is connected to the FPD-link devices?

    Can this be recreated with a TI evm standalone? 

    Secondly, can you elaborate what the cause for the 2% occurrence is? 

    Ergo, is it a specific register or specific byte? 

    For the third question.. In my opinion, you wouldn't want to change this for the i2c-omap driver because this would affect all i2c transactions in the system and I do not know whether this would cause other issues. Ill keep looking at this issue to see what the best recommendation could be based on the understanding of why this happens.

    -Josue

  • Hi Josue,

     

    Thanks for the reply.

    Please find the answers inline.

     

    Does this only happen in the bus that is connected to the FPD-link devices?

    • Yes

    Can this be recreated with a TI evm standalone? 

    • We haven't tried.

    Secondly, can you elaborate what the cause for the 2% occurrence is? 

    Ergo, is it a specific register or specific byte?

    • It doesn’t look byte/register specific as for the other 98% time also we were doing the same transactions repeatedly.

    Also, for the OMAP_I2C_TIMEOUT reduction approach, need to understand if this is safe to do from the protocol/controller perspective.

    If need arises, how much we can reduce it to?

    Thanks and Regards,

    Anand Mistry

  • Anand, 

    I believe there is ways to reduce the I2C timeout for one device via ioctls, for the driver, I am not sure at what the repercussions are to reduce the timeout from the controller perspective. I will have to look into it and respond by the end of the week.

    -Josue

  • Anand,

    This timeout is usually chosen to be a stable timeout for a myriad of connections/devices. From the controller perspective I believe it is safe but system-wise you have to be aware that  any changes will affect the reliability of other devices connected on I2C. This will require you to do some system validation.

    Have the SDA and SCL lines been inspected using a scope?

    Could you attach a screenshot of the decoded transactions at the failure?

    -Josue

  • Hi Josue, 

    Here is an instance of an I2C timeout (default software, no adjustment to the SOC I2C timeout setting).  I included some of the traffic on either side of the failure to give a sense of the normal traffic between the deserializer (the other I2C master) and serializer on the bus.

    Here is the start of a transaction that times out.  I'd expect a read to I2C address 0x18 reg 0x5D here.  I see the START condition, then 7-bit address 0x18, but then SDA goes low for 1 sec starting at the arrow: 

    Then 1 sec later, the I2C bus unfreezes:

    Thanks
    Lee

  • Lee,

    Thank you for posting the images. This to me makes me suspect the slave is in a state that makes it keep the SDA line low.

    Possible issues:

    • Slave is in faulty state
    • Slave is out of sync with master

    I am assuming the kernel is showing a timeout error? Could you please confirm?

    Also, after the one second timeout, regular operation resumes correct? 

    Have all the voltages and capacitance been checked?

    -Josue

  • Hi Josue,

    Thank you for your reply.

    Please find the answers below.

    > I am assuming the kernel is showing a timeout error? Could you please confirm?

    Yes. Kernel gives the following timeout error, also mentioned in the initial post.

    "omap_i2c 48070000.i2c: controller timed out"

    > Also, after the one second timeout, regular operation resumes correct? 

    Yes

     

    > Possible issues:

    > * Slave is in faulty state

    > * Slave is out of sync with master

    Since both the slaves on the I2C bus are TI FPDLink devices, can you help add/forward this thread to the E2E's Data-Convertors forums?

    Thanks and Regards,

    Anand Mistry

  • Have all the voltages and capacitance been checked?

    Anand,

    Due to the title of this thread, it is better if you use the   button and use the part number that will automatically put the thread in front of the correct team.

    -Josue