AM2432: UART DMA Cause Data Missing when Ethercat Upgrade in Flash

Part Number: AM2432

Tool/software:

Hi TI Experts,

Customer is working on SDK9.2.

They recently meet a strange problem when Ethercat upgrade in flash.

If the UART DMA is enabled (non-cacheable) when Ethercat upgrade, write value 1 to the flash address 0x7e000, the read out value at the same flash address is 0 not expected 1. The same situations happens for writing many other address in the flash also.

If the UART DMA is disabled when Ethercat upgrade, write value 1 to the flash address 0x7f000, the read out value at the same flash address is 1 as expected. And it is working in all other address writing.

This really confuse customer how the UART DMA could somehow affect the ethercat flashing values.

The UART Transfer coding is below.

The UART Receiver coding is below.

Could you help provide some suggestions please?

Thanks,

Kevin

  • How are they flashing ? Using MCU+ SDK McSPI or OSPI driver ? May be something to do with CBASS QoS settings. Can they reproduce without EtherCAT ? It works when UART is used without DMA ?

  • Hi Pratheesh,

    Thanks for your reply.

    Customer is using OSPI driver. If not using EtherCat, then it is hard to reproduce this problem.

    It works normally if UART is used without DMA.

    May I know could you share some guide with CBASS QoS settings please?

    Thanks,

    Kevin

  • Hi Kevin,

    We'll check on this and get back to you by EOD today. Thank you for your patience.

    Regards,
    Aaron 

  • Hi Aaron,

    This seems a system level problem. Customer previously only used R5f0_0 SOP last year. And in that project customer also used Ethercat upgrade in flash, and they could also use UART DMA at the same time. The new project involves additional R5f1_0 core, and additional UART ports & IPC communication is introduced associated with R5f1_0. At this moment, if customer disable DMA in the UART it is still working normally, but will have the above problem if DMA is enabled.

    Could you firstly share some guidance of CBASS QoS settings for customer to have a try?

    Thanks,

    Kevin

  • Hi Kevin,

    Can the customer share the interrupt and task priorities? We suspect if the higher priority task/interrupt is preempting the Flash read task.

    Regards,
    Aaron 

  • Hi Aaron,

    The tasks running apart from Ethercat flashing are IPC communication & UART DMA. We are not quite sure how to look at the priorities, but customer provided the syscfg files below, so maybe you could more conveniently look at it.

                   FIOB -> R50-0

                   FIOB_CHA_Safety -> R51-0

    FIOB.syscfg

    FIOB_CHA_Safety.syscfg

    Thanks,

    Kevin

  • If the UART DMA is enabled (non-cacheable)

    Hello Kevin,

    Do we get a customer application to look at the total application ?

    What is the meaning of UART DMA being non-cacheble , which means the buffer that is used for UART communication is kept then as non-cacheble?

    If UART buffets are non-cacheble, then why user use Cache_wb and Cache_Inv API's in the above attached images ?

    Can you also please confirm whether the source buffer and destination buffers belong to flash memory or which memory is used ?

    Is there any connection between the IPC , Ethercat,  UART and flash, like sharing Ethercat buffets for flash writing or reading or similarly to use for UART DMA?

    Can you please share the task details for UART DMA ? 

    What are the priorities for all the tasks ?

    Regards,

    Anil.

  • Hi Anil,

    If UART buffets are non-cacheable, then why user use Cache_wb and Cache_Inv API's in the above attached images ?

    Customer feedback: We just directly use the default SDK configurations according to the uart dma demo,  we did not intentionally configure them. If you think the default Cache_wb and Cache_Inv API will make UART buffer become cacheable, then we could do an experiment to remove them.

    Is there any connection between the IPC , Ethercat,  UART and flash, like sharing Ethercat buffets for flash writing or reading or similarly to use for UART DMA?

    Customer feedback: No, they are all independent. And only Ethercat task will read/write flash.  And only UART will configure DMA.

    What are the priorities for all the tasks ?

    Customer feedback

    Uart

    IPC

    Ethercat

    Can you also please confirm whether the source buffer and destination buffers belong to flash memory or which memory is used?

    Customer Feedback: 

    UART:  buffer is used

    IPC: buffer is used

    Ethercat read/write flash: buffer is used

    Do we get a customer application to look at the total application?

    Can you please share the task details for UART DMA?

    Customer feedback:  We could only provide UART DMA case details below.

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    #define CRC_START_VALUE (0x00)
    #define CHB_PROXY_TASK_STACK_SIZE (1024 * 4)
    #define CHB_PROXY_TASK_PRIORITY (4U)
    #define CHB_META_PACKAGE_BYTES_NUM (4U)
    #define CHB_FRAME_LENGTH_MAX_BYTES_NUM (255U)
    #define CHA_ACK_NACK_PACKAGE_BYTES_NUM (4U)
    #define CHA_ACK_HEADER (0x51)
    #define CHA_NACK_HEADER (0x52)
    #define CHB_CONTENT_PACKAGE_HEADER (0xA2)
    // MAIN_DEG_UART0 is used in DIOB/EIOB for testing.
    // MAIN_ADCIF_UART4 is used in FIOB for development.
    #define CHB_PROXY_TASK_UART_TRANS_INDEX (MAIN_ADCIF_UART4)
    // #define CHB_PROXY_TASK_UART_DMA_STATUS (true)
    #define CHB_ADC_CONTENT_LENGTH (8U) // In bytes.
    #define CHB_TEMP_CONTENT_LENGTH (1U)
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    Thanks,

    Kevin

  • Hello Kevin,

    Can you please confirm the Ethercat and other tasks are running ?

    I suspect that the UART with DMA task creates an issue, and already you have confirmed that the same code without DMA enablement is functioning.

    So, this might be a problem with the UART DMA task and when this task is running and the Ethercat task is not able to write data on the Flash memory.

    That's the reason we are able to see content on the flash memory is always zero.

    Can you please confirm whether all tasks are running or not ?

    Try another test. There is UART_DMA transfer API comment out and verify the same test case .

    Regards,

    Anil.

  • Hi Anil,

    If customer comment out both UART_DMA transfer & receive API, then the Ethercat upgrade read/write in flash becomes normal.

    So that means the root cause comes from UART_DMA transfer & receive API, may I know what could be the next step to debug?

    Thanks,

    Kevin

  • Hello Kevin,

    Yes, the above two APIs seems are the culprit.

    I am not sure what is going on in the customer application when calling UART DMA TX and UART DMA RX APIs. 

    I need the reproducible code on the EVM to debug further. Otherwise, it is very difficult to debug further.

    After calling the UART DMA TX and UART DMA RX, check if the RTOS timer interrupt is coming or not.

    Can you please confirm the break points in all the tasks and see if tasks are hit or not after the UART DMA TX/RX  API is called?

    This gives information about whether all tasks are suspended or not.

    Regards,

    Anil.

  • Hi Anil,

    As discussed offline, customer could continuously see the Ethercat log when Uart DMA is enabled, which means the Ethercat task is running actually.

    It is not that easy to create task reproducing on EVM for a short time. Customer previously only used R5f0_0 SOP last year. And in that project customer also used Ethercat upgrade in flash, and they could also use UART DMA at the same time. The new project involves additional R5f1_0 core, and additional UART ports & IPC communication is introduced associated with R5f1_0. At this moment, if customer disable DMA in the UART it is still working normally, but will have the above problem if DMA is enabled.

    May I know based on the above UART coding & QoS setting, could we have any other things for customer to have a try?

    Thanks,

    Kevin

  • Hello Kevin,

    I discussed offline with Kevin and Kevin will share all customer project details further to debug the issue. 

    Regards,

    Anil.

  • Hi Anil,

    I listed the details for R5F0_0 and R5f1_0 below.

    R5F0_0

    1: Ethercat upgrade flash, using SDK default OSPI driver (DMA disabled). The Ethercat writes data to on-chip RAM firstly, and then R5F0_0 read from the on-chip ram and write to the flash, not using IPC shared data.

    2: UART (DMA enabled). This UART will print the log for R5F1_0 also, the R5F1_0 will put the log into the shared memory, and then R5F0_0 will read the log and print it out using this UART.

    3: IPC with R5F1_0

    4: Timer

    R5F1_0

    1: UART (DMA disabled)

    2: IPC with R5F0_0

    For the last year project, customer only used R5F0_0, and there is no problem. This year new project introduced the R5F1_0, and the overall system will occur this problem.

    Customer has mentioned that they have tried to avoid the on-chip ram overlap for different tasks by setting different buffer shown below (you could refer to the previous reply which has this information).

    Thanks,

    Kevin

  • Hello Kevin,

    Please try the following test cases:

    Test Case 1: Swap Task Priorities

    In the customer application, the following task priorities are currently configured:
    • UART Task Priority: 4
    • EtherCAT (OSPI Write) Task Priority: 2

    Modify the configuration as below:
    • Set UART Task Priority to 2
    • Set EtherCAT Task Priority to 4

    Test Case 2: Buffer Memory Placement

    In the customer application, both UART and EtherCAT tasks are using MSRAM to write data to OSPI, with DMA handling the data transfers.

    To evaluate the impact of memory access latency and bus contention, try the following:
    • Move one of the buffers from MSRAM to TCM:
    • Option A: Place the UART TX/RX DMA buffer in TCM
    • Option B: Place the OSPI Write/Read buffer in TCM

    Test Case 3: UART DMA Frequency

    Check how frequently UART DMA is being triggered in the application.

    Example:
    • Currently, UART DMA is triggered every 1 ms
    • Modify this to 100 ms and observe the behavior

    Regards,

    Anil.

  • Hi Anil,

    Please see the result of the above 3 test cases, thanks.

    Test Case 1: Swap Task Priorities

    Modification

    Result

    This UART operates as a slave, connected to an external UART master device. The master device continuously sends data every 10ms. However, the priority level of the UART task is set too low, causing the receive buffer to overflow. This results in error reports and triggers a watchdog timeout reset due to unprocessed data.

    Test Case 2: Buffer Memory Placement

    Modification

    Result

    After modifying the UART buffer under TCMA, the data read from it is all incorrect for some unknown reason.

    Test Case 3: UART DMA Frequency

    Modification

    Result

    This UART is configured as a slave and connected to an external UART master device. The master device continuously transmits data every 10ms without interruption. Due to the slow processing speed of the UART task, the receive buffer becomes full, triggering errors and causing a watchdog timeout reboot.

    Any clue you could find in the test results?

    Thanks,

    Kevin

  • Hello Kevin,

    The above results are not as expected.

    I need to do analysis on these test cases.

    Meanwhile, please share the exact steps the customer did, and I need to review the code changes as well.

    Regards,

    Anil.