RTOS/CC2640R2F: I2C driver causes a crash of system

Hi M-W,

Yes I am using the simplelink_cc2640r2_sdk_3_10_00_15.

As far as I see the simplelink_cc2640r2_sdk_3_10_00_15 and simplelink_cc2640r2_sdk_2_30_00_28 have the same drivers so

what is the difference that can effect the issue I see ?

Secondly I have asked what is the correct procedure to re-initialize the I2C drive ?

BR,

Noam.

Hi Noam,

The two should be largely the same but you should find that the I2CCC26XX_hwiFxn() function is slightly different where it in the new driver ends with:

    else {
        /* Error Handling */
        if ((errStatus & (I2C_MASTER_ERR_ARB_LOST | I2C_MASTER_ERR_ADDR_ACK)) ||
            (object->mode == I2CCC26XX_IDLE_MODE)) {

            if (errStatus & I2C_MASTER_ERR_ADDR_ACK) {

                /*
                 * The CC26XX I2C hardware ignores the NACK condition
                 * for the slave address. Therefore, it sends an additional
                 * byte instead of generating a STOP condition.
                 */
                I2CMasterControl(hwAttrs->baseAddr,
                    I2C_MASTER_CMD_BURST_SEND_ERROR_STOP);
            }

            /* STOP condition already occurred, complete transfer */
            object->mode = I2CCC26XX_ERROR;
            SwiP_post(&(object->swi));
        }
        else {

            /* Error occurred during a transfer, send a STOP condition */
            object->mode = I2CCC26XX_ERROR;
            I2CMasterControl(hwAttrs->baseAddr,
                I2C_MASTER_CMD_BURST_SEND_ERROR_STOP);
        }

        DebugP_log2("I2C:(%p) ISR I2C Bus fault (Status Reg: 0x%x)",
                hwAttrs->baseAddr, errStatus);
    }

while the old one ends with

    else {

        /* Some sort of error happened! */
        object->mode = I2CCC26XX_ERROR;

        if (errStatus & I2C_MASTER_ERR_ARB_LOST) {
            SwiP_post(&(object->swi));
        }
        else {
            /* Try to send a STOP bit to end all I2C communications immediately */
            /*
             * I2C_MASTER_CMD_BURST_SEND_ERROR_STOP -and-
             * I2C_MASTER_CMD_BURST_RECEIVE_ERROR_STOP
             * have the same values
             */
            I2CMasterControl(hwAttrs->baseAddr,
                             I2C_MASTER_CMD_BURST_SEND_ERROR_STOP);

            /* Do not post SWI here to avoid subsequent transfers being stopped.
               Wait till the stop condition interrupt to post SWI */
        }

        DebugP_log2(
                "I2C:(%p) ISR I2C Bus fault (Status Reg: 0x%x)",
                hwAttrs->baseAddr,
                errStatus);
    }
    return;
}

As for the correct procedure to re-initialize the driver, this depends from what state you are returning. There is an issue that if you close the driver while the "busy" flag is set, this will stick around until the I2C hardware has power cycled. This is really nothing the driver can recover from as there is no way to "reset" the I2C hardware. This should only also be an issue if the I2C bus for some reason triggers false start conditions on the bus as this is the reason for the hardware hangup.

Hi M-W,

Thanks for the quick respond, that is very helpful...

I have two BLE projects. One without OAD, based on simple peripheral example. The UART is used from SDI module.
This project uses the simplelink_cc2640r2_sdk_1_50_00_58 SDK... This project re-initializes the I2C driver and it works
like a charm.

I have a new project (same hardware, same board) OAD on chip project, again based on the example but this one uses
simplelink_cc2640r2_sdk_3_10_00_15 SDK. In this project the UART is implemented simply with call-backs and cyclic buffers,
no SDI. This project with the same I2C code crashes and does not properly recover the I2C after a bus busy error.

As I mentioned the hardware is the same.

Do you think that using simplelink_cc2640r2_sdk_2_30_00_28 I2C drivers will solve my issues or do I need to use all of the
simplelink_cc2640r2_sdk_2_30_00_28 SDK ?


BR,
Noam.

Hi Noam,

I think using the older I2C driver might solve your issues. The biggest issue here is the fact that you in some cases can end up corrupting the power constraints when a I2C transaction fails (for example address NACK). What this means that even while the device should for example be active doing UART stuff, this constraint is removed by the I2C driver and you can end up just shutting everything down and going into standby even if you should not.

Without ruling out this bug, it is hard to say if there is any other issues.

Hi M-W,

Well I have copied the I2CCC26XX.c/h files from the 2.3 SDK to my project and now the system does not
crash but at some time after I am running my noisy motor (30 seconds or more) I start to see I2C errors here
and there. I am able to recover for some time (minute or two...) but eventually the I2C does not recover any more.

I have also triad to use the 1.5.. drivers but the result is the same.

With my normal BLE project (no OAD) that is based on the older 1.5 SDK the same I2C code runs and recover ...

The main differences in the two projects are:
1. In older project we used the SDI for UART comunication
2. The I2C transaction is called from within the SimpleBLEPeripheral_performPeriodicTask


1. In the new project with OAD I am using the UART differently, a task is handling that via UART call-back's
2. the I2C transaction is called from its own task.

Maybe the difference is that in the older project I2C and UART are synchronized ?.


BR,
Noam.

Hi Noam,

What kind of I2C errors is it you are seeing, are you also logging the actual I2C bus?

As far as I know, SDI ran it's UART in it's own task but aside from this it is the same driver that is used. A possible reason that comes to mind is that the old system might be going in and out of standby more often than your current system do.

Depending on the kind of I2C errors you are seeing, this could be the reason you are able to recover as this would reset the hardware. In your new project version, are you aware if the standby behavior have changed?

Could you share the code related to I2C and UART?

Hi  M-W,

The BLE is sending periodically a query for battery voltage etc... this query is just two bytes: BMS address and command. The BMS response

is 3 bytes: device address + 2 bytes value. After a successful transaction I see the NAC from the slave just after the last data byte.

If all is OK I can see in my logic analyzer the data. As soon as I start my motor I see that the BLE sends the data correctly but at some time

the BMS starts to send false data back. This data can be two bytes or 3 bytes... in most cases the address is wrong but the most obvious

is that the BMS starts to send FF...

In my older project I can see that once I am doing the recovery the data flow returns to normal flow... once in a while again bad data but then

it is recovering. As the update rate is OK (with errors), we can live with it as is...

In my new project recovery happens several times but after some time it is not recovering anymore.

This is the I2C code:

bool Init_I2C(bool CloseBeforeInit)
{
  bool RetVal = true;
  uint32_t ui32Cmd;
  uint32_t ui32SCLFreq, ui32TPR;



  if(CloseBeforeInit == true)
  {
    DoNotMesureI2C = true;
  
    // if we are here it means that we had errors in communication between the
    // SMBUS and BLE
    //
    // we must have some delay between the failed transaction and the call to
    // I2C_cancel etc... without this delay the I2C is not restoring. The reason is that
    // we call Init_I2C just after the I2C_transfer fails. The I2C driver
    // and maybe the BMS as well need time to end transaction or something...
    //TaskSleepInMs(50);
  
    I2C_cancel(i2c);

    I2C_close(i2c);

    TaskSleepInMs(50);
  }



//  I2C_init();

  // init i2cParams with default settings
 // I2C_Params_init(&i2cParams);     // sets custom to NULL

  pinCfg.pinSDA = WIN5_BMS_SDA0;
  pinCfg.pinSCL = WIN5_BMS_SCL0;

  // update i2cParams with pin configuration
  i2cParams.custom = &pinCfg;

  // set I2C to be blocking mode
  i2cParams.transferMode = I2C_MODE_BLOCKING;
  i2cParams.transferCallbackFxn = NULL;

  // set default bit rate
  i2cParams.bitRate = I2C_100kHz;


  // open the I2C port with the above settings
  i2c = I2C_open(WIN5_I2C_CHANNEL_I2C0, &i2cParams);

  if (!i2c)
  {
    // Error opening I2C
    RetVal = false; //   test = 0;
  }
  else
  {
      // *****************************************************************************
      // Change the I2C clock speed to ?? KHz
      // *****************************************************************************
      // write TPR value
      // SCL clock period
      // This field specifies the period of the SCL clock.
      // SCL_PRD = 2*(1+TPR)*(SCL_LP + SCL_HP)*CLK_PRD
      // where:
      // SCL_PRD is the SCL line period (I2C clock).
      // TPR is the timer period register value (range of 1 to 127)
      // SCL_LP is the SCL low period (fixed at 6).
      // SCL_HP is the SCL high period (fixed at 4).
      // CLK_PRD is the system clock period in ns.
      // Clock_tickPeriod, TI platforms have a default of 1000 us
      // Change the I2C clock speed to 30 KHz  ??
      ui32SCLFreq = 100000; //50000;

      // Compute the clock divider that achieves the fastest speed less than or
      // equal to the desired speed. The numerator is biased to favor a larger
      // clock divider so that the resulting clock is always less than or equal
      // to the desired clock, never greater.
      ui32TPR = (((2 * 10 * ui32SCLFreq) - 1) / (2 * 10 * ui32SCLFreq)) - 1;

      HWREG(I2C0_BASE + I2C_O_MTPR) = ui32TPR;
  }

  DoNotMesureI2C = false;


  return RetVal;
}

//-----------------------------------------------------------------------------

bool ProccessSmbusTransactione(uint8_t *WriteBuff, uint8_t WriteLen, uint8_t *ReadBuff, uint8_t ReadLen)
{
  I2C_Transaction i2cTransaction;
  bool RetVal = false;
 
 
  memset(&i2cTransaction, 0, sizeof(I2C_Transaction));
 
  i2cTransaction.slaveAddress = WriteBuff[0];
  i2cTransaction.writeBuf = &WriteBuff[1];
  i2cTransaction.writeCount = WriteLen;
  i2cTransaction.readBuf = ReadBuff;
  i2cTransaction.readCount = ReadLen;
 
  // i2cTransaction.
  RetVal = I2C_transfer(i2c, &i2cTransaction);
 
  return RetVal;
}

//-----------------------------------------------------------------------------

static bool HandleErrors(void)
{
  static uint8_t CountErrors = 0;
  bool RetVal = true;
 
  // advance error count !
  CountErrors++;
  ToManyErrorsCount ++;
 
  if(CountErrors > MAX_I2C_ERRORS)
  {
    Init_I2C(true);

    CountErrors = 0;
  }
  
  if(ToManyErrorsCount > TO_MANY_ERROR_CONT_LIMIT)
  {
    RetVal = false;
  }
  
  return RetVal;
}

//-----------------------------------------------------------------------------
//-----------------------------------------------------------------------------

void ReadBatteryVoltage(void)
{
  uint8_t WriteBuff[3];
  uint8_t RedBuff[3];
  bool TransactionStatus = false;
  uint16_t NewVoltage = 0;
  
  
  // prepare data to send
  WriteBuff[0] = SMBUS_SLAVE_ADDRESS;
  WriteBuff[1] = SMBUS_READ_VOLTAGE_OP;
  
  TransactionStatus = ProccessSmbusTransactione(WriteBuff, 1, RedBuff, 2);
  
  NewVoltage = ((RedBuff[1] << 8) + RedBuff[0]);
  
  LastValidBmsVoltage = NewVoltage;

  // voltage reading is in mili volts
  if((TransactionStatus == true) && ((NewVoltage > 8500) && (NewVoltage < 12600)))
  {
    LastValidBmsVoltage = NewVoltage;
    
    ToManyErrorsCount = 0;
  }
  else
  {
    if(HandleErrors() == false)
    {
      LastValidBmsVoltage = 0xFFFF;
    }
  }
}

//-----------------------------------------------------------------------------

In the above code I am calling Init_I2C(false); once before entering my periodical task.

The periodical task is as follows:

static void PeriodicHandle_taskFxn(UArg a0, UArg a1)
{
  uint32_t SysClkTick;
  uint32_t timeoutInTicks = 1000 * (1000/Clock_tickPeriod);


//  // initialize the periodic task related
#ifdef I2C
  // init I2C
  I2C_init();

  Init_I2C(false);
#endif


  // Pend (approximately) up to 1 second
  Semaphore_pend(RunPeriodicSemSync, timeoutInTicks);

  //------------------------------------------------

  // periodic Handle loop
  while(1)
  {
    // read system time
    SysClkTick = Clock_getTicks();


#ifdef HART_BIT
    SendHartBit(SysClkTick);
#endif

    // Handle SMBUS periodic transaction
#ifdef I2C
    SMBUS_PeriodicRead(SysClkTick);
#endif

    //-----------------------------------------
#warning "after changing the ST side not to ask for humidity, open this !"
#if 0
    SendSmbusTempAndVoltag(SysClkTick);
#endif

    //-----------------------------------------

    ProcessAdcReading(SysClkTick);

    //-----------------------------------------

    ProcessButtonsReading(SysClkTick);

    //-----------------------------------------

    ProcessMotorFaultReading(SysClkTick);

    //-----------------------------------------

    // put a task delay here that will serve as iteration delay
    TaskSleepInMs(PERIODIC_TASK_INTERVAL_SLEEP_MS);

    Task_yield();
  }
}

//-----------------------------------------------------------------------------

static void SMBUS_PeriodicRead(uint32_t CurrentClkTick)
{
  static uint32_t SavedTick = I2C_READ_INTERVAL_OFFSET_MS;
  static ProcData_t ProcData = eReadBatteryVoltage;


  if(TaskClockDiffMs(CurrentClkTick, SavedTick) > I2C_READ_INTERVAL_MS)
  {
    // reload counter
    SavedTick = CurrentClkTick;

    if(GetDoNotMesureI2cFlagState() == false)
    {
      switch(ProcData)
      {
        case eReadBatteryVoltage:
          ReadBatteryVoltage();
break;
          ProcData = eReadBatteryTemperature;
        break;
        //-----------------------------
        case eReadBatteryTemperature:
          ReadBatteryTemperature();

          ProcData = eReadBatteryCapacity;
        break;
        //-----------------------------
        case eReadBatteryCapacity:
          ReadBatteryCapacity();

          ProcData = eReadBatteryVoltage;
        break;
      }
    }
  }
}

//-----------------------------------------------------------------------------

The periodical task has a priority 3 a while UART has priority 2

The UART code is very simple: I have two cyclic buffers. One for receive one for transmits. when an RX call back is fired the byte is added to the

RX buffer. The data n this buffer is parsed inside the UART task,

The second cyclic buffer (TX) is filed from several places in code. Filling the buffer is done from one and only function protected with a critical section:

void UART_SendMessageToST(uint8_t *Buff)
{
  uint8_t DataLen;
  halIntState_t cs;


  HAL_ENTER_CRITICAL_SECTION(cs);  // Enter critical section.
  {
    // get the total packet size
    DataLen = Buff[API_MSG_SIZE_INDEX];

    // calculate the message checksum before adding the new message to cyclic buffer
    Buff[DataLen - 1] = UART_CheckSum(Buff, (DataLen - 1));

    FillBuffer(&UartTxCyclicHandle, Buff, DataLen);

    CheckIfRestartIsNeeded();
  }
  HAL_EXIT_CRITICAL_SECTION(cs);  // Exit critical section.
}

Every time data is added to the TX buffer we check if write needs to start. The RX call back is triggered after every byte sent. So most of the

UART handling is done in the background (interrupts). A general concept used by many...

If you want access to all of the code or project I can arrange that as well, but here...

Thanks for a super fast response and support.

BR,

Noam

Hi Noam,

Could you maybe share the UART task as well, it would be interested to see how it is setup compared to the periodic task. Further more (I should have asked earlier), have you been able to get any reading on exactly where in the code the exception happens? For example, have you tried to use ROV to try to find the location of the crash?

Hi M-W,

Actually after changing the I2C drivers the software does not crash anymore.. everything seems to work

apart of the I2C that works fine until we start the motor (noise) and start getting errors.

If we could recover the I2C without having issues we would be happy.

I am on the edge of starting a low level bit banging I2C code if we cannot find a solution.

The UART code is part of a messaging software so I need to dig in and do some code sniff's ..

this is how I init the UART driver and task if this is what you were asking:

/*
 * Serial.c
 *
 * Created on Mar 2019, by Noam W.
 *
 *
 *
 */

#include <ti/sysbios/knl/Task.h>
#include <ti/sysbios/knl/Clock.h>
#include <ti/sysbios/knl/Semaphore.h>
#include <hal_mcu.h>

#include <xdc/std.h>
#include <ti/drivers/dpl/HwiP.h>

#include "ApiProcessing.h"
#include "Serial.h"
#include "circ_buff.h"
#include "Win5_HardwareConfig.h"
#include "icall.h"
#include "Main.h"
#include "simple_peripheral_oad_onchip.h"
#include "WIN5_PeriodicTask.h"
#include "win5_i2c.h"


static UART_Handle UartHandle;
static UART_Params UartParams;

static tCircularBuffer UartRxCyclicHandle;
static tCircularBuffer UartTxCyclicHandle;

static uint8_t UART_RxBuff[UART_RX_CYCLIC_BUFFER_SIZE] = {0};
static uint8_t UART_TxBuff[UART_TX_CYCLIC_BUFFER_SIZE] = {0};


// Task setup
static uint8_t UartTaskStack[UART_TASK_STACK_SIZE] = {0};
static Task_Struct UartTask;

static uint8_t GlobalReadByte;

static bool WriteInProgress = false;

Semaphore_Handle RunUartSemSync;
Semaphore_Params RunUartSemSyncParams;


static void UartReadCallBack(UART_Handle UartHandle, void *buf, size_t count);
static void UartWriteCallBack(UART_Handle UartHandle, void *buf, size_t count);

static void UART_InitUart(void);
static void UART_UartWrite(uint8_t *Buff, size_t Size);
static void UART_UartRead(uint8_t *Buff, size_t Size);
static void UartHandle_taskFxn(UArg a0, UArg a1);
static void CheckIfRestartIsNeeded(void);

//-----------------------------------------------------------------------------
//*****************************************************************************
//-----------------------------------------------------------------------------

static void UART_InitUart(void)
{
  // initialize the UART cyclic buffers
  CreateCircularBuffer(&UartRxCyclicHandle, UART_RX_CYCLIC_BUFFER_SIZE, UART_RxBuff);
  CreateCircularBuffer(&UartTxCyclicHandle, UART_TX_CYCLIC_BUFFER_SIZE, UART_TxBuff);


  // init the UART driver
  UART_init();

  // init UART UartParams to default values
  UART_Params_init(&UartParams);

  // set UART UartParams
  UartParams.readMode = UART_MODE_CALLBACK;
  UartParams.writeMode = UART_MODE_CALLBACK;
  UartParams.readTimeout = UART_WAIT_FOREVER;
  UartParams.writeTimeout = UART_WAIT_FOREVER;
  UartParams.readCallback = UartReadCallBack;
  UartParams.writeCallback = UartWriteCallBack;
  UartParams.readReturnMode = UART_RETURN_FULL;
  UartParams.readDataMode = UART_DATA_BINARY;
  UartParams.writeDataMode = UART_DATA_BINARY;
  UartParams.readEcho = UART_ECHO_OFF;
  UartParams.baudRate = BLE_ST_BAUD_RATE;
  UartParams.dataLength = UART_LEN_8;
  UartParams.stopBits = UART_STOP_ONE;
  UartParams.parityType = UART_PAR_NONE;
  UartParams.custom = NULL;

  // we only have one UART in table so the first param to UART_Open is
  // index and it is 0
  UartHandle = UART_open(WIN5_UART0_PORT, &UartParams);

  // check if init OK
  if (UartHandle == NULL)
  {
    // System_printf("UART did not open");
    while(1);
  }


  //
  Semaphore_Params_init(&RunUartSemSyncParams);
  RunUartSemSync = Semaphore_create(0, &RunUartSemSyncParams, NULL); /* Memory allocated in here */

  if (RunUartSemSync == NULL) /* Check if the handle is valid */
  {
    // System_abort("Semaphore could not be created");
    while(1);
  }


  // make a dummy read, actully to initialize the RX interrupt
  UART_UartRead(&GlobalReadByte, 1);
}

//-----------------------------------------------------------------------------

void UART_constractTask(void)
{
  Task_Params taskParams;

  // Configure task
  Task_Params_init(&taskParams);
  taskParams.stack = UartTaskStack;
  taskParams.stackSize = UART_TASK_STACK_SIZE;
  taskParams.priority = UART_TASK_PRIORITY;

  Task_construct(&UartTask, UartHandle_taskFxn, &taskParams, NULL);
}

//-----------------------------------------------------------------------------

static void UartHandle_taskFxn(UArg a0, UArg a1)
{
  uint32_t timeoutInTicks = 1000 * (1000/Clock_tickPeriod);

  // initialize the UART communication channel.
  UART_InitUart();


  // Pend (approximately) up to 1 second
  Semaphore_pend(RunUartSemSync, timeoutInTicks);


  // Wake up the application to flush out any remaining UART data in the queue.
  InitializeParing();


  // run periodic task...
  ReleasePeriodicTaskFromPending();

  //------------------------------------------------

  // UART UartHandle loop
  while(1)
  {
    // UartHandle messages from the ST COM port. If function returns FALSE it means that
    // there is no more bytes in the receive buffer. Go to sleep for 5 ms and then
    // check again
    if((GetDoNotMesureI2cFlagState() == false) && (API_CommunicationReceiveST() == true))
    {
      // add a small delay to avoid starvation, this will put a 1 ms delay
      TaskSleepInMs(1);
    }
    else
    {
      // if we do not have more data to process add a small delay to avoid starvation.
      // this will put a 5 ms delay or less.. need to optimize
      TaskSleepInMs(5);
    }
  }
}

//-----------------------------------------------------------------------------

BR,

Noam

Hi M-W,

I gave up on the I2C driver and I am writing (almost done) a bit banging I2C module.

I have a simple issue that I cannot find an answer. Both my IO's are defined as output, open drain with external pull ups.

How can I read the IO pin state. What ever I do I get "1".. Do I need to also enable input on these pins and read it as inputs ?

BR,

Noam.

Hi Noam,

Based on what I understand from the UART code you posted, it seems that it might be keeping you out of standby all together. In this case, it could be some errors that you would have a hard time recovering from as it sound the bus is getting really impacted by the motor drive.

As for bit-banged I2C drivers goes, have you looked into the Sensor Controller on the device? If you are not using it already, you could use this as the I2C Master. It implements a simpler, bit-banged, I2C API which you can use and it can run independently of the main ARM CPU.