CC1352R: UART2 RX Unexpectedly Disabled due to Lack of Synchronization Between UART2CC26X2_readTimeout and UART2CC26X2_hwiIntFxn

Peter Crowley

Part Number: CC1352R

Hello,

Background

I believe I've uncovered a subtle bug in the UART2CC26X2 driver and I would like to report my findings here so that others don't have to spend so many hours in confusion as I did.

I am developing a TI RTOS application for the CC1352 with SimpleLink SDK version 4.40.04.04. This application includes serial console functionality courtesy of the UART2 driver. In some rare cases, I observed that typing in the serial console would cause the UART interface to stop responding.

Driver Out-of-Sync with Peripheral

I was eventually able to capture this issue in the debugger. Initially I observed the ICall task stuck in NOROM_RFCDoorbellSendTo from rfc.c. A post from the Zigbee & Thread forum led me to investigate the power constraints from the PowerCC26X2 driver. Sure enough, a hardware watchpoint on PowerCC26X2_module.constraintCounts[PowerCC26XX_DISALLOW_STANDBY] showed it being set to 0 while UART receive was active. After this, the CC1352 went into standby, at which point the UART interface was disabled.

The next time the CC1352 woke up, UART2CC26X2_initHw automatically re-enabled the UART peripheral, but did not enable RX or TX. The next call to UART2_readTimeout invoked UART2_rxEnable, which neglected to actually enable reception through UART2Support_enableRx, because state.rxEnabled was still set for the UART2_Object.

Thus, state.rxEnabled is out of sync with the RXE bit in the CTL register, and the UART interface is rendered unable to receive data. Additionally, the fact that the PowerCC26XX_DISALLOW_STANDBY constraint has been released prematurely can cause underflow in PowerCC26X2_module.constraintCounts[PowerCC26XX_DISALLOW_STANDBY] and wreak havoc on the system if the UART interface is intentionally disabled in the future.

But why was the PowerCC26XX_DISALLOW_STANDBY constraint released while UART receive was active?

Investigating Power Constraints

A hardware watchpoint revealed that PowerCC26X2_module.constraintCounts[PowerCC26XX_DISALLOW_STANDBY] was being set to 0 at the call to Power_releaseConstraint within UART2CC26X2_hwiIntFxn:

if (status & (UART_INT_EOT)) {
    /* End of Transmission occurred */
    HWREG(hwAttrs->baseAddr + UART_O_CTL) &= ~UART_CTL_TXE;

    if (object->state.txEnabled) {
        object->state.txEnabled = false;

        if ((object->eventMask & UART2_EVENT_TX_FINISHED) &&
                object->eventCallback) {
            object->eventCallback(handle, UART2_EVENT_TX_FINISHED, 0,
                    object->userArg);
        }

        /* Release constraint because there are no active transactions */
        Power_releaseConstraint(PowerCC26XX_DISALLOW_STANDBY);
        ++UART2CC26X2_hwiIntFxn_release_count;
    }

    UARTIntDisable(hwAttrs->baseAddr, UART_INT_EOT);
}

The corresponding call to Power_setConstraint is in UART2Support_dmaStartTx:

if (object->state.txEnabled == false) {
    /* Set constraints to guarantee transaction */
    Power_setConstraint(PowerCC26XX_DISALLOW_STANDBY);
    object->state.txEnabled = true;
}

A close inspection of this code does not reveal any problems. The state.txEnabled variable tracks whether the PowerCC26XX_DISALLOW_STANDBY has been set. The only writes to state.txEnabled are within these two functions, and UART2Support_dmaStartTx is only called from UART2CC26X2_hwiIntFxn or from UART2_writeTimeout with HWI disabled. From this perspective, it should not be possible for UART2CC26X2_hwiIntFxn to release PowerCC26XX_DISALLOW_STANDBY unless it had already been set by UART2Support_dmaStartTx.

Instrumenting the UART2CC26X2 Driver

As I reached my wits' end, I added some instrumentation code to see for sure whether UART2CC26X2_hwiIntFxn was releasing PowerCC26XX_DISALLOW_STANDBY more times than UART2Support_dmaStartTx set it:

bool UART2CC26X2_backupTxState = false;
uint32_t UART2Support_dmaStartTx_constraint_set_count = 0;
uint32_t UART2CC26X2_hwiIntFxn_release_count = 0;

if (status & (UART_INT_EOT)) {
    /* End of Transmission occurred */
    HWREG(hwAttrs->baseAddr + UART_O_CTL) &= ~UART_CTL_TXE;

    if (object->state.txEnabled) {
        if ((object->eventMask & UART2_EVENT_TX_FINISHED) &&
                object->eventCallback) {
            object->eventCallback(handle, UART2_EVENT_TX_FINISHED, 0,
                    object->userArg);
        }

        /* Release constraint because there are no active transactions */
        ++UART2CC26X2_hwiIntFxn_release_count;
        Power_releaseConstraint(PowerCC26XX_DISALLOW_STANDBY);
        object->state.txEnabled = false;
        UART2CC26X2_backupTxState = false;
    }

    UARTIntDisable(hwAttrs->baseAddr, UART_INT_EOT);
}

if (object->state.txEnabled == false) {
    /* Set constraints to guarantee transaction */
    ++UART2Support_dmaStartTx_constraint_set_count;
    Power_setConstraint(PowerCC26XX_DISALLOW_STANDBY);
    object->state.txEnabled = true;
    UART2CC26X2_backupTxState = true;
}

When PowerCC26X2_module.constraintCounts[PowerCC26XX_DISALLOW_STANDBY] was set to 0 at Power_releaseConstraint in UART2CC26X2_hwiIntFxn, I observed that:

state.txEnabled == true
UART2CC26X2_backupTxState == false
UART2CC26X2_hwiIntFxn_release_count == UART2Support_dmaStartTx_constraint_set_count + 1

This confirmed my suspicion that UART2CC26X2_hwiIntFxn was releasing PowerCC26XX_DISALLOW_STANDBY more times than UART2Support_dmaStartTx set it. Additionally, it proved that state.txEnabled was getting modified unexpectedly.

Bitfield Woes

To continue my investigation, I scoured the code looking for any place that state.txEnabled could be modified unintentionally. Initially I feared some memory access error, but that seemed unlikely, since state.txEnabled is part of a bitfield, of which all the other bits remained in their expected states.

In an attempt to isolate changes to the byte containing state.txEnabled so that I could catch them with a hardware watchpoint, I began eliminating writes to other values in the bitfield. I found that my problem went away when I eliminated the write to state.readTimedOut in UART2CC26X2_readTimeout:

static void UART2CC26X2_readTimeout(uintptr_t arg)
{
    UART2CC26X2_Object *object = ((UART2_Handle)arg)->object;

    object->state.readTimedOut = true;
    SemaphoreP_post(&(object->readSem));
}

But how could setting the value of state.readTimedOut have any affect on state.txEnabled?

The answer lies within the subtlety of bitfield operations and can be observed by looking closely at the assembly instructions for UART2CC26X2_readTimeout:

          UART2CC26X2_readTimeout():
0003a078:   6800                ldr        r0, [r0]
1222          object->state.readTimedOut = true;
0003a07a:   79C1                ldrb       r1, [r0, #7]
0003a07c:   F0410120            orr        r1, r1, #0x20
0003a080:   71C1                strb       r1, [r0, #7]
1223          SemaphoreP_post(&(object->readSem));
0003a082:   3078                adds       r0, #0x78
0003a084:   F000BE26            b.w        SemaphoreP_post
 808          return (-((int_fast16_t)status));

Setting state.readTimedOut is not atomic. In fact, it is a three-step operation:

Load one byte from the address of state.readTimedOut into register R1
OR R1 with the constant value 0x20 and store the result in R1
Store the value of R1 back at the address of state.readTimedOut

Because state is a bitfield, state.readTimedOut is stored in the same byte as other bitfield values, including state.txEnabled. When UART2CC26X2_readTimeout writes to the address of state.readTimedOut in step 3, it uses the value that was retrieved in step 1. If state was modified between steps 1 and 3, the value retrieved in step 1 is out-of-date and writing it back in step 3 causes those modifications to be lost.

Conclusion

I believe this is exactly what is happening:

UART2CC26X2_readTimeout loads from &state.readTimedOut into R1
The UART interrupt fires, causing UART2CC26X2_hwiIntFxn to preempt UART2CC26X2_readTimeout, and modifies state, setting state.txEnabled = false.
UART2CC26X2_readTimeout loads R1 back into &state.readTimedOut, thereby setting state.txEnabled = true, since the contents of R1 were not affected by UART2CC26X2_hwiIntFxn.

Unfortunately, I have not been able to reproduce this behavior in a smaller example application (nor have I spent much time attempting to do so). Regardless, I believe that the UART2CC26X2 driver should be updated to disable HWI within UART2CC26X2_readTimeout (and perhaps UART2CC26X2_writeTimeout), thereby protecting the contents of object->state.

In my own project, I solved the problem by passing timeout = 0 when calling UART2_readTimeout.
(Previously, I had been setting timeout = 1, which likely exacerbated the issue and was entirely unnecessary.)

Thank you for reading! I hope this has been helpful to some; and perhaps the driver can be patched in the future.

over 3 years ago

0 AndresM over 3 years ago

TI__Expert 3886 points

Hi Peter,

I really appreciate the time you took to find the issue and then to explain it in such a comprehensive way. It really helps a lot!

There have been a couple of SDK releases since v4.40, and some work has been done to the UART2 driver, but I don't believe that this has been covered before. Having said this, do you have any tips on how I can reproduce what you are seeing in a simpler example application? This way I can pass the information to R&D and we can take care of it in a coming SDK release.

Again, thank a lot.

BR,
Andres

0 Peter Crowley over 3 years ago in reply to AndresM

Intellectual 282 points

Andres,

I think your best bet for reproducing this reliably is to artificially extend the delay between the operations used to set readTimedOut within UAR2CC26X2_readTimeout. For example:

static void UART2CC26X2_readTimeout(uintptr_t arg)
{
    UART2CC26X2_Object *object = ((UART2_Handle)arg)->object;
    uint8_t readTimedOut = ((uint8_t *)(&object->state))[7];
    uint32_t wait = 100000;
    while(wait--);
    readTimedOut |= 0x20;
    ((uint8_t *)(&object->state))[7] = readTimedOut;
    SemaphoreP_post(&(object->readSem));
}

That change, along with the following modifications to uart2echo.c, allowed me to reproduce the problem using the UART2 Echo example.

void *mainThread(void *arg0)
{
    char         input;
    const char   echoPrompt[] = "Echoing characters:\r\n";
    UART2_Handle uart;
    UART2_Params uartParams;
    size_t       bytesRead;
    size_t       bytesWritten = 0;

    /* Call driver init functions */
    GPIO_init();

    /* Configure the LED pin */
    GPIO_setConfig(CONFIG_GPIO_LED_0, GPIO_CFG_OUT_STD | GPIO_CFG_OUT_LOW);

    /* Create a UART where the default read and write mode is BLOCKING */
    UART2_Params_init(&uartParams);
    uartParams.baudRate = 115200;
    uartParams.readMode = UART2_Mode_BLOCKING;
    uartParams.readReturnMode = UART2_ReadReturnMode_PARTIAL;
    uartParams.writeMode = UART2_Mode_BLOCKING;

    uart = UART2_open(CONFIG_UART2_0, &uartParams);

    if (uart == NULL) {
        /* UART2_open() failed */
        while (1);
    }

    /* Turn on user LED to indicate successful initialization */
    GPIO_write(CONFIG_GPIO_LED_0, CONFIG_GPIO_LED_ON);

    UART2_write(uart, echoPrompt, sizeof(echoPrompt), &bytesWritten);

    /* Loop forever echoing */
    while (1) {
        bytesRead = 0;
        while (bytesRead == 0) {
            UART2_readTimeout(uart, &input, 1, &bytesRead, 1);
        }
        UART2_write(uart, &input, 1, NULL);
    }
}

(Note: I also disabled compiler optimizations during my testing.)

With these changes, I set a watchpoint to trigger when &((PowerCC26X2_module).constraintCounts)[2] was set to 0.
After sending a few characters via UART, the watchpoint was triggered within UART2CC26X2_hwiIntFxn. This indicates that the CC would be allowed to go into standby while UART RX is enabled, which should not happen.

0 AndresM over 3 years ago in reply to Peter Crowley