LAUNCHXL-CC1310: Possible UART driver issue when UART flooded

Mike Kay

Part Number: LAUNCHXL-CC1310
Other Parts Discussed in Thread: CC1310

Hello,

I am running CCS 9.3.0 with cc13x0 SDK v4.10 (also tried 3.20 with same issues) and have been running into a potential issue where the UART driver in RX mode appears to lock up when the UART is flooded with characters. I am working on hardening the UART interface to an application.

I have stripped the application down to the bare minimium and the same programs still exist.

The system is a very basic command/response protocol to communicate with another device over UART. Essentially how this works is that the CC1310 issues a "ready for command" character to the UART and then issues a non blocking read for 1 character. When the uart callback fires it looks at that single character to determine the command and how many more bytes to look for. This is because the full application has commands of different sizes (eg. command 1 might require a payload of 10 bytes). To prevent any issues i refrained from calling another uart_read from inside any callbacks. At the same time a timer is triggered to abort the operation if the read command is not received within so much time. This allows the unit to "reset" itself in the event of a partial transmission. Once the command is complete it issues a confirmation and then goes back to step 0 which issues another "ready for command" and the whole thing starts again.

This code appears to work fine. All you need to test it is a launchXL board, and use the terminal program in CCS. if you interact with it at human speeds (issuing commands via a keyboard) it will work as stated however if you try to issue a blast of data (i.e. flood the UART) by copying and pasting some random stuff (use shift-insert in the terminal window) the unit appears to get into some sort of state where the unit is stuck in step 0 and the uart_read command appears to do nothing for many seconds (perhaps 45-60). When i have tried to debug this I have monitored the uartCC26XXObjects it appears the count in the ringbuffer goes to some number (eg. 15) and then slowly decrements by one every few seconds until it gets to zero and then wraps around back to 32 and starts to decrement again.

As my larger program will have other stuff going on I cannot use blocking mode UART calls.

Any help or suggestions to get this running would be appreciated. At this point I don't feel the UART system/logic is hardened enough for a production application. I have also tried to increase the stack to 0x2048 and heap to 0x1800 in TIRTOS.CMD with no luck.

Any help or suggestions would be greatly appreciated. Thank you!

Main.c:

#include <stdlib.h>
#include <unistd.h>

/* TI Drivers */
#include <ti/drivers/UART.h>
#include <ti/sysbios/knl/Clock.h>
/* Board Header files */
#include "Board.h"


Clock_Struct UART_clk_Struct;
Clock_Handle UART_clk_Handle;
Clock_Struct UART_ka_clk_Struct;
Clock_Handle UART_ka_clk_Handle;
Clock_Struct step_clk_Struct;
Clock_Handle step_clk_Handle;

static uint8_t uart_state = 0;

#define MAX_NUM_RX_BYTES    860   // Maximum RX bytes to receive in one go
uint8_t rxBuf[MAX_NUM_RX_BYTES];   // Receive buffer
char uart_cmd = '.';


const char echoPrompt[] = "BOOT:\r\n";
const char msg_xxx_ok[] = "xxx ok\r\n";
const char msg_xxx_err[] = "xxx err\r\n";
const char msg_err_badcmd[] = "error - bad command\r\n";
const char msg_err_unkown[] = "error - unknown\r\n";
const char msg_err_timeout[] = "error - RX timeout\r\n";
const char msg_req[] = ">";

static bool uart_tx_ready = 1;


Void UART_clk_fxn(UArg arg0)
{
    // if the timer expires and we are in steps 21 set state to 3 (timeout error)
    if (uart_state == 21)
    {
        uart_state = 3;
    }
}

Void UART_ka_clk_fxn(UArg arg0)
{
    // if it's been too long waiting for a command, reset to step 0 to issue another command request (in event system gets out of sync)
    if (uart_state == 1)
    {
        uart_state = 0;
    }
}

// UART Write callback function
static void writeCallback(UART_Handle handle, void *txBuf, size_t size)
{
    uart_tx_ready = 1;
}


// UART Read callback function
// completely re-written as state machine to avoid any read or write commands within callback (due to strange behaviour)
static void readCallback(UART_Handle handle, void *rxBuf, size_t size)
{
    // if we are waiting for a command, set next state based on command
    if (uart_state == 1)
    {
        uart_cmd = ((uint8_t*)rxBuf)[0];
        // sample command
        if (uart_cmd == '=')
        {
            uart_state = 20;
        }

        // bad command!
        else
        {
            uart_state = 2;
        }
    }

    // process message
    else if (uart_state == 21)
    {
            // do stuff here
            uart_state = 22;
    }

    // unknown error
    else
    {
        uart_state = 9;
    }
}


void *mainThread(void *arg0)
{

    // UART STUFF
    UART_Handle handle;
    UART_Params params;
    UART_init();

    UART_Params_init(&params);
    params.baudRate      = 115200;
    params.writeMode     = UART_MODE_CALLBACK;
    params.writeDataMode = UART_DATA_BINARY;
    params.readMode      = UART_MODE_CALLBACK;
    params.readDataMode  = UART_DATA_BINARY;
    params.readCallback  = readCallback;
    params.writeCallback  = writeCallback;

    // Open the UART and initiate the first read
    handle = UART_open(Board_UART0, &params);

    // generate one shot clock with interrupt to UART_clk_fxn - USED FOR message timeout (if message not fully received)
    Clock_Params UART_clk;
    Clock_Params_init(&UART_clk);
    UART_clk.period = 0;
    UART_clk.startFlag = FALSE;
    // Construct a one-shot Clock Instance (100 millisec)
    Clock_construct(&UART_clk_Struct, (Clock_FuncPtr)UART_clk_fxn, 100000/Clock_tickPeriod, &UART_clk);
    UART_clk_Handle = Clock_handle(&UART_clk_Struct);

    // generate one shot clock with interrupt to UART_ka_clk_fxn - USED FOR UART keepalive (in case PI reboots or gets out of sync)
    Clock_Params UART_ka_clk;
    Clock_Params_init(&UART_ka_clk);
    UART_ka_clk.period = 0;
    UART_ka_clk.startFlag = FALSE;
    // Construct a one-shot Clock Instance (5 seconds)
    Clock_construct(&UART_ka_clk_Struct, (Clock_FuncPtr)UART_ka_clk_fxn, 5000000/Clock_tickPeriod, &UART_ka_clk);
    UART_ka_clk_Handle = Clock_handle(&UART_ka_clk_Struct);


    while(1)
    {
        switch(uart_state)
        {
        // ready for a new uart command, write ready message to UART and issue UART read for one byte
        case 0:
            if (uart_tx_ready)
            {
                UART_read(handle, rxBuf, 1);
                UART_write(handle, msg_req, sizeof(msg_req));
                Clock_start(UART_ka_clk_Handle);
                uart_cmd = '\0';
                uart_state = 1;
                uart_tx_ready = 0;
            }
            break;
        // bad command issued
        case 2:
            if (uart_tx_ready)
            {
                UART_write(handle, msg_err_badcmd, sizeof(msg_err_badcmd));
                UART_readCancel(handle);
                uart_state = 0;
                uart_tx_ready = 0;
            }
            break;
        // RX Timeout
        case 3:
            if (uart_tx_ready)
            {
                UART_write(handle, msg_err_timeout, sizeof(msg_err_timeout));
                UART_readCancel(handle);
                uart_state = 0;
                uart_tx_ready = 0;
            }
            break;
        // unknown error
        case 9:
            if (uart_tx_ready)
            {
                UART_write(handle, msg_err_unkown, sizeof(msg_err_unkown));
                uart_state = 0;
                uart_tx_ready = 0;
            }
            break;
        // xxx command received
        case 20:
            Clock_start(UART_clk_Handle);
            UART_read(handle, rxBuf, 848);
            uart_state = 21;
            break;
        // xxxx command ok
        case 22:
            if (uart_tx_ready)
            {
                UART_write(handle, msg_xxx_ok, sizeof(msg_xxx_ok));
                uart_state = 0;
                uart_tx_ready = 0;
            }
            break;
        // xxxx message command error
        case 29:
            if (uart_tx_ready)
            {
                UART_write(handle, msg_xxx_err, sizeof(msg_xxx_err));
                uart_state = 0;
                uart_tx_ready = 0;
            }
            break;
        }
    }
}

main_tirtos.c

/*
 *  ======== main_tirtos.c ========
 */
#include <stdint.h>

/* POSIX Header files */
#include <pthread.h>

/* RTOS header files */
#include <ti/sysbios/BIOS.h>

/* Example/Board Header files */
#include "Board.h"

extern void *mainThread(void *arg0);

/* Stack size in bytes */
#define THREADSTACKSIZE    1024

/*
 *  ======== main ========
 */
int main(void)
{
    pthread_t           thread;
    pthread_attr_t      attrs;
    struct sched_param  priParam;
    int                 retc;
    int                 detachState;

    /* Call driver init functions */
    Board_initGeneral();

    /* Set priority and stack size attributes */
    pthread_attr_init(&attrs);
    priParam.sched_priority = 1;

    detachState = PTHREAD_CREATE_DETACHED;
    retc = pthread_attr_setdetachstate(&attrs, detachState);
    if (retc != 0) {
        /* pthread_attr_setdetachstate() failed */
        while (1);
    }

    pthread_attr_setschedparam(&attrs, &priParam);

    retc |= pthread_attr_setstacksize(&attrs, THREADSTACKSIZE);
    if (retc != 0) {
        /* pthread_attr_setstacksize() failed */
        while (1);
    }

    retc = pthread_create(&thread, &attrs, mainThread, NULL);
    if (retc != 0) {
        /* pthread_create() failed */
        while (1);
    }

    BIOS_start();

    return (0);
}

CC1310_launchxl.cmd

/*
 *  ======== CC1310_LAUNCHXL.cmd ========
 */

//--stack_size=1024   /* C stack is also used for ISR stack */
//--stack_size=4096   /* C stack is also used for ISR stack */
--stack_size=2048   /* C stack is also used for ISR stack */

//HEAPSIZE = 0x1000;  /* Size of heap buffer used by HeapMem */
HEAPSIZE = 0x1800;  /* Size of heap buffer used by HeapMem */

/* Override default entry point.                                             */
--entry_point ResetISR
/* Allow main() to take args                                                 */
--args 0x8
/* Suppress warnings and errors:                                             */
/* - 10063: Warning about entry point not being _c_int00                     */
/* - 16011, 16012: 8-byte alignment errors. Observed when linking in object  */
/*   files compiled using Keil (ARM compiler)                                */
--diag_suppress=10063,16011,16012

/* The starting address of the application.  Normally the interrupt vectors  */
/* must be located at the beginning of the application.                      */
#define FLASH_BASE              0x0
#define FLASH_SIZE              0x20000
#define RAM_BASE                0x20000000
#define RAM_SIZE                0x5000

/* System memory map */

MEMORY
{
    /* Application stored in and executes from internal flash */
    FLASH (RX) : origin = FLASH_BASE, length = FLASH_SIZE
    /* Application uses internal RAM for data */
    SRAM (RWX) : origin = RAM_BASE, length = RAM_SIZE
}

/* Section allocation in memory */

SECTIONS
{
    .text           :   >> FLASH
    .TI.ramfunc     : {} load=FLASH, run=SRAM, table(BINIT)
    .const          :   >> FLASH
    .constdata      :   >> FLASH
    .rodata         :   >> FLASH
    .cinit          :   > FLASH
    .pinit          :   > FLASH
    .init_array     :   > FLASH
    .emb_text       :   >> FLASH
    .ccfg           :   > FLASH (HIGH)

    .data           :   > SRAM
    .bss            :   > SRAM
    .sysmem         :   > SRAM
    .nonretenvar    :   > SRAM

    /* Heap buffer used by HeapMem */
    .priheap   : {
        __primary_heap_start__ = .;
        . += HEAPSIZE;
        __primary_heap_end__ = .;
    } > SRAM align 8

    .stack          :   > SRAM (HIGH)
    .vtable_ram     :   > SRAM
}

over 5 years ago

0 IUsedToBeARealPerson (M-W) over 5 years ago

TI__Guru 58425 points

HI Mike,

There seems like there is a lot of potential for unexpected program execution n case the UART would be "flood" as you put it. Just take case "0" as an example, consider what happens if the read callback fires before you actually set the "uart_state" variable and how that could escalate depending on what the actual payload is.

What you seem to be looking for is a synchronized solution (without using blocking mode) but what you have is something that is something asynchronous that also seems to be deterministic. Also, it is important that you understand that a "read cancel" will invoke the callback which means you once again would get a callback before you actually update the state.

I would suggest you first try to re-arrange your code to update all state variables before you issue any read command.

0 Mike Kay over 5 years ago in reply to IUsedToBeARealPerson (M-W)

Prodigy 45 points

Hi M-w,

Thanks for looking into this for me. I have re-arranged setting some state variables and this has improved the functionality when flooding the system however the UART still seems to lock up after about 1h of heavy serial traffic. I've determined that the XDS110 serial to USB bridge is not the issue here as I have also tested it with a separate bridge with the same results. There is still something funny with the UART driver.

One of the reasons why I had to create this complex state machine for UART handling is due to the fact that it is not possible to issue more than one UART_write within a callback function. Consider the following simple program:

If you run it and interact with it via the terminal, pressing the '=' key will only output "ok" and it will not output "test". the same is if you press any other key (e.g. 'h') - the program will output "not ok" and then still not output "test". Why will the second blocking UART_write not execute?

If i were to change it to a non-blocking UART_write, there is a very high probability the second write would not happen as the UART would be busy but that doesn't really help.

Note that there is no support for polling reads or writes in the CC13XX series.

I am considering using the CC13XXware driverlib UART driver to see if I get better success. Do you have any example code for configuring and opening the UART using the driverlib Uart driver that I can use as a comparison?

#include <stdlib.h>
#include <unistd.h>

/* TI Drivers */
#include <ti/drivers/UART.h>
#include <ti/sysbios/knl/Clock.h>
/* Board Header files */
#include "Board.h"

#define MAX_NUM_RX_BYTES    100   // Maximum RX bytes to receive in one go
uint8_t rxBuf[MAX_NUM_RX_BYTES];   // Receive buffer


const char msg_ok[] = "ok\r\n";
const char msg_nok[] = "not ok\r\n";
const char msg_test[] = "test\r\n";

// UART Write callback function
static void writeCallback(UART_Handle handle, void *txBuf, size_t size)
{
    //
}

// UART Read callback function
// completely re-written as state machine to avoid any read or write commands within callback (due to strange behaviour)
static void readCallback(UART_Handle handle, void *rxBuf, size_t size)
{
    char uart_cmd = '.';
    uart_cmd = ((uint8_t*)rxBuf)[0];

    // sample command
    if (uart_cmd == '=')
    {
        UART_write(handle, msg_ok, sizeof(msg_ok));
    }

    else
    {
        UART_write(handle, msg_nok, sizeof(msg_nok));
    }


    UART_write(handle, msg_test, sizeof(msg_test));
}


void *mainThread(void *arg0)
{
    UART_Handle handle;
    UART_Params params;
    UART_init();

    UART_Params_init(&params);
    params.baudRate      = 115200;
    //params.writeMode     = UART_MODE_CALLBACK;
    params.writeMode     = UART_MODE_BLOCKING;
    params.writeDataMode = UART_DATA_BINARY;
    params.readMode      = UART_MODE_CALLBACK;
    params.readDataMode  = UART_DATA_BINARY;
    params.readCallback  = readCallback;
    //params.writeCallback  = writeCallback;

    // Open the UART and initiate the first read
    handle = UART_open(Board_UART0, &params);

    while(1)
    {
        UART_read(handle, rxBuf, 1);
    }
}

0 IUsedToBeARealPerson (M-W) over 5 years ago in reply to Mike Kay

TI__Guru 58425 points

Hi Mike,

The code example above is not a "OK" use-case in the RTOS context. As you are configuring the write mode to be blocking, you are not allowed to perform any write from inside an ISR context (which is what a callback should be considered to be). Just consider what happens if you try to block and wait for an interrupt inside another interrupt, you would end up blocking many parts of the kernel (often with a deadlock as the ultimate result) as only "high priority" interrupts could possibly get in. Now it sometimes happen to work with a blocking write inside the callback but the behavior is highly unexpected, don't do this.

The example would however be OK if you configured the write mode to be "callback" type as this is a non-blocking operation. This does however mean that you yourself need to keep track of when it is possible to call write.

The UART driver do not buffer any writes which means that in your case (when doing callback writes), you first say "send OK" and the driver will start to work on this. As you right away will say "and also send TEST", the driver will reject this as it would likely not be done sending out the OK.

Using the "DriverLib" would not give you better performance in this regard, it would likely only make things more complicated as you need to interface with the power driver and Hwi module to set something up that works in the TI-RTOS environment.

I would suggest you implement a buffered write logic to allow you to buffer the data you want to write out. For example, you could do something like this:

uartWriteCallback ( ... )
{
    if (dataToSend > 0) {
        size_t len = dataToSend > SEND_BUFFER_SIZE ? SEND_BUFFER_SIZE : dataToSend;
        // This part depends on how you implement your global buffer, I make no assumption but
        // you might want to have some additional logic here depending on if it could wrap the pointer (ring buffer)
        // or not.
        memcpy(sendBuffer, globalBuffer + tail, len);

        tail -= len;
        dataToSend -= len;

        UART_write(handle, sendBuffer, len);
    }
    else
    {
        // No more to send
        writeIsActive = false;
    }
}

writeToUart( string, len) {
    // Write is active?
    if (! writeIsActive) {

        size_t smallLen = len > SEND_BUFFER_SIZE ? SEND_BUFFER_SIZE : len;

        memcpy(sendBuffer, string, smallLen;

        // Do we need to put something in the global buffer?
        if (smallLen < len)
        {
            memcpy(globalBuffer, string + smallLen, len - smallLen);
            dataToSend += len - smallLen;
            tail += len - smallLen;
        }

        UART_write(handle, sendBuffer, smallLen);
        writeIsActive = true;
    }
    else {
        // UART busy, put it in the buffer for it to be processed from the uartWriteCallback
        // CRITICAL SECTION
        key = Hwi_disable();

        // Again, I'm not considering the max length here, you likely want to do this :)

        memcpy(globalBuffer, string + smallLen, len - smallLen);
        dataToSend += len - smallLen;
        tail += len - smallLen;

        Hwi_enable(key);
    }
}

0 Mike Kay over 5 years ago in reply to IUsedToBeARealPerson (M-W)

Prodigy 45 points

Hi M-W,

I rewrote the application to use a callback for the UART write with queue as suggested above. This worked as anticipated however after a longer period of system operation eventually the UART write function would stop running even though the main application and other functions kept on working fine. I tested this by toggling some GPIO right before the write statement so I can be sure it was being executed. This is also similar to what I had noticed before as well. Changing the uart write from callback to blocking (and moving it outside the readcallback as well) seems to be very stable and has been running for many days reliably.

Unfortunately I still think something is strange about the UART driver and it's operation when using uart write with callbacks.

It seems like my issue has also been mentioned in this thread as well:

https://e2e.ti.com/support/wireless-connectivity/sub-1-ghz/f/156/t/765425

I inspected the latest sdk (4.10) uart code and it appears the thread posters code has been added but I still don't feel that it's 100% fixed. Unfortunately it's beyond my skill to try and troubleshoot the UART driver.

I hope you had a nice vacation.

0 IUsedToBeARealPerson (M-W) over 5 years ago in reply to Mike Kay

TI__Guru 58425 points

Hi Mike,

There is actually no difference between operating the driver as blocking or callback mode from a logic point of view. Blocking mode is just the following logic built in:

void writeCallback( ... ) {

SemaphoreP_post(&mySemaphore);

}

void myTask () {

...

// In callback mode
UART_write( ... );

// Wait for callback
SemaphoreP_pend(&mySemaphore, FOREVER);

So basically, when in blocking mode, the callback is an internal function and the write/read API will pend on a semaphore before returning.

What this means is that if you find it to work when using blocking mode and note callback mode it is more likely a logic issue in the implementation and not in the driver (again, as blocking mode is just callback mode with a built in semaphore pend/post). If the write "stops" you should look at the parts responsible for keeping it alive:

* Is writeIsActive true, do we "think" we are actively writing but we are not?

* Did the UART_write actually return "success" when we called the API?

* Do you check all return values and handle errors accordingly?

* What if a UART_write() returns you with "error"?

Sub-1 GHz

Sub-1 GHz forum

LAUNCHXL-CC1310: Possible UART driver issue when UART flooded