This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC2340R5: UART Rx missing bytes when BLE is enabled

Part Number: CC2340R5
Other Parts Discussed in Thread: SYSCONFIG

Tool/software:

My customer is trying to develop a high performance UART to BLE bridge application on CC2340R5. The design goal is to steady work under 460800 baud rate. To improve the performance a ring buffer is used to store the received UART data, BLE application will retrive data from this ring buffer and send through BLE notification to the peer device.

The application is working fine with 115200 baud rate, but when the baud rate increased to 460800, missing bytes are seen at the UART Rx side.

  • The bytes are directly missed from UART Rx buffer. CC2340 UART never received these bytes. This is verified from a count of received bytes in UART callback.
  • If BLE and the ring buffer is not used(only receive UART data but not put it into the ring buffer), the issue does not happen.
  • If ring buffer is used but BLE is not(put received data into ring buffer but does not send through BLE notification), the issue still happens.

The customer has a project to reproduce this issue(please reach out to me offline for the project). To reproduce the issue, please follow the steps:

  1. Run the project on a CC2340R5 LaunchPad.
  2. Connect to a cellphone.
  3. Request an MTU of 247 bytes from the phone.
  4. Enable notification from the phone.
  5. Open a UART terminal(hTerm) using 460800 baud rate.
  6. Send a file(I created a 100kB sample file as attached below) through UART to CC2340.
  7. Watch for uart_received_bytes in CCS watch window.

Repeat step 6 and 7 and watch for the uart_received_bytes variable, each time the UART terminal sends 100,000 bytes to CC2340, and once in a while uart_received_bytes will be one or two bytes less than the sent bytes.

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/538/test_5F00_100kb.bin

It looks the issue is related to the interrupt priority or DMA transfer, but I am having trouble locating the exact location of the issue. I have tried to adjust the UART HWI priority and the UART ring buffer size, but both did not make a difference. Would you please help locate the root cause? Thanks.

Best regards,

Shuyang

  • Hi Shuyang,

    What SimpleLink F3 SDK version is being used?  Are you able to replicate this behavior in a TI Drivers project that does not initialize BLE, or is the existence of the BLE stack/connection necessary to replicate the issue?  What UART Ring Buffer Sizes have you tried?

    Regards,
    Ryan

  • Hi Ryan,

    The SDK version is simplelink_lowpower_f3_sdk_9_10_00_83.

    I did two tests to verify if the BLE stack is essential:

    1. BLE is advertising, but no connection, UART callback puts the received data into the ring buffer, but BLE task does not send it - the issue can still be reproduced. 

    2. BLE is not advertising nor connected, UART callback puts the received data into the ring buffer, BLE task retrives data from the ring buffer but does nothing about it - the issue does not happen.

    It looks BLE activity is essential for the issue to happen.

    The UART ring buffer size is 256(Rx) and 32(Tx).

    What's the next step for debugging? Since the BLE HWI always has the highest priority, is there a way to avoid UART to miss any byte at the Rx side?

    Best regards,

    Shuyang

  • Hi Shuyang,

    This is not the first time that users have noticed bytes missing when using the UART2 TI Driver.

    https://e2e.ti.com/f/1/t/1430261 
    https://e2e.ti.com/f/1/t/1363778 

    Here is what I am currently thinking:

    • The UART peripheral receive FIFO is 8 entries in length
    • The UART2 TI Driver implements DMA is basic mode (i.e. not ping-pong or any other mode)
    • When an entire DMA transfer is completed (UART2LPF3_hwiIntRead) then there is a time for which the DMA RX transaction is stopped (UART2Support_dmaStopRx) to further process the incoming data state before re-starting DMA RX transactions (UART2Support_dmaStartRx)
    • During this time when the DMA RX transaction is paused, there is a race condition involved to make sure that the RX FIFO is not filled
    • Given the increased baud rate and higher priority of the BLE stack advertising, it is possible that the FIFO is overflowing during this time within UART2LPF3_hwiIntRead, resulting in missed bytes.

    If this theory is correct then you would be able to use critical sections within UART2LPF3_hwiIntRead to avoid BLE stack activity during this time sensitive period:

    /*
     *  ======== UART2LPF3_hwiIntRead ========
     *  Function called by Hwi to handle read-related interrupt
     */
    static void UART2LPF3_hwiIntRead(uintptr_t arg, uint32_t status)
    {
        UART2_Handle handle              = (UART2_Handle)arg;
        UART2LPF3_HWAttrs const *hwAttrs = handle->hwAttrs;
        UART2LPF3_Object *object         = handle->object;
        void *readBufCopy;
        unsigned int key;
    
        // Enter critical section.
        key = Hwi_disable();
    
        /* Temporarily stop RX transaction */
        UART2Support_dmaStopRx(handle);
    
    //...
    
        /* Start another RX transaction */
        UART2Support_dmaStartRx(handle);
        
        // Exit critical section.
        Hwi_restore(key);
    }

    What I cannot determine is the effect this will have on the BLE stack as this will be actively blocking it from sending out the scheduled advertisement.  Please try out my suggestion (with as large of a ring buffer as possible to minimize critical section interruptions) but keep in mind the implications this could have for further BLE activity (such as packet transmission/reception during an active connection).

    Regards,
    Ryan

  • Hi Ryan,

    After adding the critical section, the issue still happens. I did several tests and the behavior was pretty the same as before. Is there other things to consider? It would be better if you could help reproduce it at your side.

    Best regards,

    Shuyang

  • Hello Shuyang,

    Please help me with the following questions.

    1. What is the max UART RX buffer size you are using? I would think increasing the buffer size would help to alleviate this in case the issue is some type of overflow. Have you try instrumenting the params.eventCallback = NPITLUART_eventCallBack; to include params.eventMask |= UART2_EVENT_OVERRUN; as well and check if this condition is met?
    2. What is the adv/conn interval, have you tried with longer intervals and see the impact?
    3. Could you share a code-snippet of how the UART driver is being used in the project? Are you using the Display UART or have you defined another one?

    Thanks.

    David.

  • Hi Shuyang,

    Can you set the SysConfig Power policy to "doWFI" so that we can confirm that Power constraints/dependencies are not playing a role?  Is there some way to determine which bytes from the 100 kb file are missing (i.e. beginning, middle, end)?

    Regards,
    Ryan

  • Hi David,

    1. The UART Rx ring buffer size is 256 bytes, and the application ring buffer is 2048 bytes. I did not register the callback but I have checked the overrun bit in CCS debug mode, did not observed overrun. I will add the callback later.

    2. ADV interval is 100ms, conn interval is 30ms. Will try bigger value.

    3. The project is sent offline. I am using UART2 driver, not display driver.

  • Hi Ryan,

    I will try WFI.

    I did not find a very good way to locate the missing byte, I added some code in the Rx callback to send the last byte back through UART. When the code was added I found the issue likely happened around the 2048 byte, that’s when the application ring buffer was full.

    But the I found this Tx code actually increased the chance for the issue to happen, so I am not sure if it introduced a issue itself.

    Without the Tx code there will be 1 to 3 bytes missed in 100,000 bytes, so it is quite difficult to tell the exact position indeed.

    Best regards,

    Shuyang

  • Hi David, Ryan,

    Update the latest debug findings:

    1. Using doWFI - the issue still happens.

    2. Increase UART Rx ring buffer size to 20480 - the issue still happens.

    3. Monitor FIFO overrun - orverrun did happen, but the overrun count is less than the number of missed bytes. Below data was calculated with a 15ms BLE connection interval, to compare different connection intervals the data was not sent through BLE notification, but only moved into and out of the ring buffer.

    I also found the RSR_ECR register is not cleared automatically, so I have to add some code to clear it to let the overrun count able to increment. I'm not sure my implementation introduced an issue, it looks the overrun count is always 1 less than the actually missing bytes.

    Transmitted bytes

    uart_received_bytes

    uart_overrun_count

    100,000

    99,996

    3

    200,000

    199,996

    3

    300,000

    299,993

    5

    400,000

    399,987

    10

    500,000

    499,985

    12

    600,000

    599,985

    12

    700,000

    699,984

    13

    800,000

    799,983

    14

    900,000

    899,980

    17

    1,000,000

    999,980

    17

    4. Increase the connection interval - the connection interval does make a difference. See the table below for count of received bytes under different connection intervals. I sent a 100kB file each time, so the transmitted bytes are incremented by 100,000 each time, the number of missing bytes is smaller with a larger connection interval.

    Transmitted bytes

    15ms

    30ms

    50ms

    100ms

    100,000

    99,997

    99,999

    100,000

    99,999

    200,000

    199,995

    199,998

    199,999

    199,999

    300,000

    299,991

    299,997

    299,999

    299,999

    400,000

    399,987

    399,995

    399,997

    399,999

    500,000

    499,985

    499,995

    499,997

    499,999

    600,000

    599,983

    599,995

    599,996

    599,999

    700,000

    699,982

    699,995

    699,995

    699,999

    800,000

    799,977

    799,994

    799,995

    799,999

    900,000

    899,976

    899,992

    899,995

    899,999

    1,000,000

    999,975

    999,992

    999,995

    999,999

    Best regards,

    Shuyang

  • As mentioned above, I found the RSR_ECR register is not cleared after the OE bit is set to 1. I would like to add more detail about this finding because I am not sure if this is an issue or simply my implementation.

    As in the TRM, the OE bit in RSR_ECR should be cleared to 0 once there is an empty space in the FIFO:

    However, I found it NOT being cleared once is set to 1, even if the RXFE is 1 which means the FIFO is empty:

    This makes the overrun count stays at 1, because the overrun interrupt is not invoked again when RSR_ECR.OE is 1.

    To clear the OE bit, I added some code in UART2LPF3_hwiIntFxn to clear this bit:

    After it is cleared manually, the OE interrupt was able to be hit again.

    I'm not sure if this is an issue on the hardware or simply a misunderstanding of the TRM, would you help confirm?

    Best regards,

    Shuyang

  • 1. Using doWFI - the issue still happens.

    2. Increase UART Rx ring buffer size to 20480 - the issue still happens.

    Okay so not a low-power entry issue (expected) and MAX_DMA_SIZE = UDMA_XFER_SIZE_MAX = 1024 so the UART RX ring buffer would be limited by this value.

    I don't know much about the overflow bit but based on your analysis it would appear that it needs to be cleared manually which would contradict the TRM.

    Regards,
    Ryan

  • Hi Ryan, David,

    I think I have located the root cause of this issue. It is caused by UART overrun.

    First of all, adding critical section to UART2LPF3_hwiIntRead does help improve the performance by eliminating the situation that UART2LPF3_hwiIntRead is preempted by higher priority HWI(although it does not completely eliminate it for the reason I will elaborate below).

    But UART overrun still could happen after adding the critical section, because there is a chance the RF_commandHwi is invoked right before the moment which UART2LPF3_hwiIntFxn should take place. To verify this I added GPIO toggle inside both UART2LPF3_hwiIntFxn and RF_commandHwi. From the logic trace we can see the UART2LPF3_hwiIntFxn is delayed due to being preempted by RF_commandHwi:

    For comparison, a regular interval from UART2_read being called to invoke of UART2LPF3_hwiIntFxn is 5.21ms, and when the issue happens the interval is delayed to 5.34ms. This matches the period of RF_commandHwi execution which is 150us.

    With a baud rate of 460800, this period of 150us equals to the transmission time of 7 bytes, during this time the UART Rx is not able to be enabled. Therefore when the next UART_read is called the UART Rx FIFO is already full, causing the overrun to happen. That also explains why the issue did not happen with lower baud rate.

    Since the RF_commandHwi has to stay at the highest priority, I guess there is no good way to completely eliminate this issue with the current UART DMA implementation. Using DMA Ping-Pong mode seems promissing since it enables a continuous data transfer, do you think it's a good idea?

    ps. For the critical section not eliminating the preemption completely, it is because I found a small issue in current implementation. The UART callback is calling BLEAppUtil_invokeFunctionNoData which calls pvPortMalloc to allocate memory from the heap, and pvPortMalloc calls xTaskResumeAll which uses taskENTER_CRITICAL and taskEXIT_CRITICAL for critical section. Since these APIs are not meant for ISR context, they will accidentally enabled the global interrupt. Although the later calls of HwiP_disable will turn off the global interrupt shortly, this call does increase the possibility for the HWI to be preempted by higher priority HWI.

    Best regards,

    Shuyang

  • Hi Shuyang,

    Thank you for your in-depth analysis to confirm the issue.  The combination of higher UART baud rate, RF operation, and DMA basic mode is creating a corner case which is not avoidable under the current conditions.  With this greater understanding of the limitation involved, the host could insert a brief pause in between sending every 1024 bytes which would still allow for the higher baud rate while only briefly delaying the overall transmission.  Otherwise DMA ping-pong mode is certainly an option which would need to be further developed, you could refer to the ADCBuf TI Driver as a solution which already uses this.

    Regards,
    Ryan

  •  Is there any update about this question?

  • Hi linjie,

    I am not aware of actions required of me for which to provide an update.

    Regards,
    Ryan