This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5726: UART Driver: UART_write call never returns (writeMode = UART_MODE_BLOCKING, writeTimeout = 100)

Part Number: AM5726
Other Parts Discussed in Thread: SYSBIOS

Hello,

I have a task running on IPU, among others, who is responsible to communicate with Input/Output modules through UART driver:

a few subsequent Read/Write requests are sent every 125 ms by the program and are answered by the IO modules.

For an unknown reason and after some time, in which the system is working fine, a call to UART_write never returns (seen from the logs).

Thanks to our debugging sessions, we could also see that the UART ISR (function ti-pdk/packages/ti/drv/uart/src/v1/UART_v1.c:UART_v1_hwiIntFxn) is actually in a never-ending call sequence, i.e.,it is called again and again with the same arguments (e.g., intType = 0, object->txDataSent = 0), I brought the ISR code below:

static void UART_v1_hwiIntFxn(uintptr_t arg)
{
    UART_Handle         handle = (UART_Handle)arg;
    UART_V1_Object     *object = (UART_V1_Object*)handle->object;
    UART_HwAttrs const *hwAttrs = (UART_HwAttrs*)handle->hwAttrs;
    uint32_t            intType;

    while (TRUE)
    {
        intType = UARTIntIdentityGet(hwAttrs->baseAddr);
        
        if ((intType & UART_INTID_RX_THRES_REACH) == UART_INTID_RX_THRES_REACH)
        {
            if ((intType & UART_INTID_RX_LINE_STAT_ERROR) ==
                UART_INTID_RX_LINE_STAT_ERROR)
            {
                /* RX line status error */
                UART_procLineStatusErr(handle);
            }
            else
            {
                if ((intType & UART_INTID_CHAR_TIMEOUT) == UART_INTID_CHAR_TIMEOUT)
                {
                    /* rx timeout, log the rx timeout errors */
                    object->rxTimeoutCnt++;
                }

                /* RX FIFO threshold reached */
                if (object->readSize)
                {
                    object->readSize = UART_v1_readData(handle, object->readSize);
                    if ((object->readSize) == 0U)
                    {
                        UARTIntDisable(hwAttrs->baseAddr, UART_INT_RHR_CTI | UART_INT_LINE_STAT);
                        /* Reset the read buffer so we can pass it back */
                        object->readBuf = (uint8_t *)object->readBuf - object->readCount;
                        if (object->readTrans != NULL)
                        {
                            object->readTrans->count = object->readCount;
                            object->readTrans->status = UART_TRANSFER_STATUS_SUCCESS;
                        }

                        /* Call back to application if in callback mode */
                        UART_v1_callback(handle, true);
                        object->readTrans = NULL;
                    }
                }
            }
        }
        else if ((intType & UART_INTID_TX_THRES_REACH) == UART_INTID_TX_THRES_REACH)
        {
            /* TX FIFO threshold reached */
            if (object->writeSize)
            {
                object->writeSize = UART_writeData_v1(handle, object->writeSize);
                if ((object->writeSize) == 0U)
                {
                    UARTIntDisable(hwAttrs->baseAddr, UART_INT_THR);
                    /* Reset the write buffer so we can pass it back */
                    object->writeBuf = (uint8_t *)object->writeBuf - object->writeCount;

                    if (object->writeTrans != NULL)
                    {
                        object->writeTrans->count = object->writeCount;
                        object->writeTrans->status = UART_TRANSFER_STATUS_SUCCESS;
                    }

                    object->txDataSent = TRUE;
                    UARTInt2Enable(hwAttrs->baseAddr, UART_INT2_TX_EMPTY);
                }
            }
        }
        else
        {
            break;
        }
    }

    if (object->txDataSent == TRUE)
    {
        intType = UARTInt2StatusGet(hwAttrs->baseAddr);
        if (intType & UART_INT2_TX_EMPTY)
        {
            UARTInt2Disable(hwAttrs->baseAddr, UART_INT2_TX_EMPTY);

            /* Call back to application if in callback mode */
            UART_v1_callback(handle, false);
            object->writeTrans = NULL;
            object->txDataSent = FALSE;
        }
    }
}

The UART initialization code:

	UART_Params_init(&params);
	params.baudRate = DUART_BAUD_RATE; // -> 115200
	params.readEcho = UART_ECHO_OFF;
	params.readDataMode = UART_DATA_BINARY;
	params.writeDataMode = UART_DATA_BINARY;
	params.writeMode = UART_MODE_BLOCKING;
	params.writeTimeout = DUART_WRITE_TIMEOUT; // -> 100
	params.readMode = UART_MODE_BLOCKING;
	params.readTimeout = DUART_READ_TIMEOUT; // -> 100
	params.readReturnMode = UART_RETURN_FULL;

	UART_socGetInitCfg(UART_INSTANCE, &uart_hwAttrs);
	uart_hwAttrs.baseAddr = SOC_UART5_BASE + IPC_REGISTER_OFFSET;
	uart_hwAttrs.intNum = 48; /* use interrupt #48 */
	uart_hwAttrs.txTrigLvl=UART_TXTRIGLVL_8; /* TX FIFO trigger at lowest level */
	UART_socSetInitCfg(UART_INSTANCE, &uart_hwAttrs);

	/* configure the irq corssbar to route the irq from uart5 to ipu1-core1-irq48 */
	DIRQ_voCustomCSL_xbarIpuIrqConfigure(1, CSL_XBAR_INST_IPU1_IRQ_48, CSL_XBAR_UART5_IRQ);
	UART_init();

	uart_handle = UART_open(UART_INSTANCE, &params);

We already had issues working with the UART_CALLBACK_MODE for write so we opted for the BLOCKING_MODE  (pls see the thread below):

RTOS/AM5726: A lower priority task "interferes" with the job of a higher priority task - Processors forum - Processors - TI E2E support forums

Thanks to our logs we could see that everything goes well (no uart_error, no timeout, all the bytes were written successfully) until the very last call to the UART_Write which never returns.

The main question here is, in which circumstances and why, a call to UART_Write with a given timeout, never returns? 

Let me know if you need more info.

Thanks,

Reza

  • Hi Reza,

    Can you do step by step debug and check what is happening when UART_write is not returning? Is it looping somewhere or is it hanging at some operation?
    Also, can you please share which SDK version are you using?

    Regards,
    Parth

  • Hi Parth,

    We cannot do the step by step debug as the issue happens only after a large random number of calls to UART_write. All the previous calls go well (no error, no timeout, successfully writes all the requested bytes).

    Alternatively, when the issue happens, we set breakpoints at the beginning of every single loop in the driver code: None of them is reached that implies there must have been a hanging operation.

    In this situation, as explained above, the UART_v1_hwiIntFxn function is called again and again, in a never-ending call sequence:

    IntType and object->txDataSent always equals  0.

    The register UART_IER2 remains equal to UART_IER2_EN_TXFIFO_EMPTY_MASK (value 2).

    We tried to compile with CIO_DRV_CONSOLE flag enabled, to see the driver logs. However, we could not see any log, probably because the printfs are not routed. 

    We are using buildroot with the following versions of ti packages:

    xdctools_3_32_01_22

    ti-processor-sdk-rtos-am57xx-evm-04.01.00.06-Linux-x86

    Regards,

    Reza

       

  • Hi Parth,

    From UART_v1.c « v1 « src « uart « drv « ti « packages - processor-sdk/pdk - Processor SDK RTOS PDK drivers.

    I found a more recent version of UART_v1.c.

    The new ISR code has some modifications compared the one we use. In particular, the two else conditions in lines 49 and 79, which are missing in the version we currently use:

    static void UART_v1_hwiIntFxn(uintptr_t arg)
    {
        UART_Handle         handle = (UART_Handle)arg;
        UART_V1_Object     *object = (UART_V1_Object*)handle->object;
        UART_HwAttrs const *hwAttrs = (UART_HwAttrs const *)handle->hwAttrs;
        uint32_t            intType;
        uint8_t             rdData;
    
        while ((bool)true)
        {
            intType = UARTIntIdentityGet(hwAttrs->baseAddr);
    
            if ((intType & UART_INTID_RX_THRES_REACH) == UART_INTID_RX_THRES_REACH)
            {
                if ((intType & UART_INTID_RX_LINE_STAT_ERROR) ==
                    UART_INTID_RX_LINE_STAT_ERROR)
                {
                    /* RX line status error */
                    UART_procLineStatusErr(handle);
                }
                else
                {
                    if ((intType & UART_INTID_CHAR_TIMEOUT) == UART_INTID_CHAR_TIMEOUT)
                    {
                        /* rx timeout, log the rx timeout errors */
                        object->rxTimeoutCnt++;
                    }
    
                    /* RX FIFO threshold reached */
                    if (object->readSize > 0U)
                    {
                        object->readSize = (size_t)UART_v1_readData(handle, (int32_t)(object->readSize));
                        if ((object->readSize) == 0U)
                        {
                            UARTIntDisable(hwAttrs->baseAddr, UART_INT_RHR_CTI | UART_INT_LINE_STAT);
                            /* Reset the read buffer so we can pass it back */
                            object->readBuf = (uint8_t *)object->readBuf - object->readCount;
                            if (object->readTrans != NULL)
                            {
                                object->readTrans->count = (uint32_t)(object->readCount);
                                object->readTrans->status = UART_TRANSFER_STATUS_SUCCESS;
                            }
    
                            /* Call back to application if in callback mode */
                            UART_v1_callback(handle, (bool)true);
                            object->readTrans = NULL;
                        }
                    }
                    else
                    {
                        (void)UARTCharGetNonBlocking2(hwAttrs->baseAddr, &rdData);
                        UARTIntDisable(hwAttrs->baseAddr, UART_INT_RHR_CTI | UART_INT_LINE_STAT);
                    }
                }
            }
            else if ((intType & UART_INTID_TX_THRES_REACH) == UART_INTID_TX_THRES_REACH)
            {
                /* TX FIFO threshold reached */
                if (object->writeSize > 0U)
                {
                    object->writeSize = (size_t)UART_writeData_v1(handle, (int32_t)(object->writeSize));
                    if ((object->
    writeSize) == 0U)
                    {
                        UARTIntDisable(hwAttrs->baseAddr, UART_INT_THR);
                        /* Reset the write buffer so we can pass it back */
                        object->writeBuf = (const uint8_t *)object->writeBuf - object->writeCount;
    
                        if (object->writeTrans != NULL)
                        {
                            object->writeTrans->count = (uint32_t)(object->writeCount);
                            object->writeTrans->status = UART_TRANSFER_STATUS_SUCCESS;
                        }
    
                        object->txDataSent = TRUE;
                        UARTInt2Enable(hwAttrs->baseAddr, UART_INT2_TX_EMPTY);
                    }
                }
                else
                {
                    UARTIntDisable(hwAttrs->baseAddr, UART_INT_THR);
                }
            }
            else
            {
                break;
            }
        }
    
        if (object->txDataSent == TRUE)
        {
            intType = UARTInt2StatusGet(hwAttrs->baseAddr);
            if ((intType & UART_INT2_TX_EMPTY) != 0U)
            {
                UARTInt2Disable(hwAttrs->baseAddr, UART_INT2_TX_EMPTY);
    
                /* Call back to application if in callback mode */
                UART_v1_callback(handle, (bool)false);
                object->writeTrans = NULL;
                object->txDataSent = FALSE;
            }
        }
    }

    It would be a great help, if you could let us know;

    - if these two modifications are related to a bug fix which could be related to our problem?

    - if these changes are included in a more recent official pdk release?

    Please note that, it is a quite big effort for us to update the pdk in our build system and repeat the tests. So, we would like to make sure that UART driver changes are relevant to our problem. beforehand.

    Thanks,

    Reza  

  • Hi Reza,

    if these two modifications are related to a bug fix which could be related to our problem?

    Its difficult to confirm this before we root cause the issue.

    Can you please share what kind of device your UART is communicating with?

    a few subsequent Read/Write requests are sent every 125 ms by the program and are answered by the IO modules.

    Also, can you please share what is the size of data that is sent every 125ms?

    Regards,
    Parth

  • Hi Parth,

    Thank you for your reply.

    Meanwhile we performed many more tests to isolate the issue.

    It seems to be a race condition/conflict between the UART operations and an interrupt received on a GPIO.

    As long as the interrupt is disabled, no matter how many UART read/writes we do, there is no issue.

    If we enable this interrupt, the system will fail after a random time between a few minutes to a few hours.

    I have now three specific questions about the contents of the file ti-pdk/packages/ti/drv/gpio/soc/am572/GPIO_soc.c (the code below):

    1. What is the meaning of the field line1EventId in GPIO_v1_HwAttrs struct?

    * typedef struct GPIO_v1_HwAttrs_s entries:
    * uint32_t baseAddr; GPIO Peripheral base address
    * uint32_t line1IntNum; GPIO Peripheral interrupt vector
    * uint32_t line2IntNum; GPIO Peripheral interrupt vector
    * uint32_t line1EventId; GPIO Peripheral interrupt vector
    * uint32_t line2EventId; GPIO Peripheral interrupt vector

    2. Why is value 0 is used as the line1EventId for GPIO8 in the file, what does that mean? Is it possible to use a value like 62? What would be the difference?

    3. Why is lline1IntNum = 49 is used for GPIO3 to GPIO8 in the file? Sholudn't the lline1IntNum be unique among different GPIOs?

    Thanks

    Reza

    /**
     *  \file   GPIO_soc.c
     *
     *  \brief  AM572x SOC specific GPIO hardware attributes.
     *
     *   This file contains the GPIO hardware attributes like base address and
     *   interrupt ids.
     */
    
    /*
     * Copyright (C) 2014-2015 Texas Instruments Incorporated - http://www.ti.com/
     *
     * Redistribution and use in source and binary forms, with or without
     * modification, are permitted provided that the following conditions
     * are met:
     *
     * Redistributions of source code must retain the above copyright
     * notice, this list of conditions and the following disclaimer.
     *
     * Redistributions in binary form must reproduce the above copyright
     * notice, this list of conditions and the following disclaimer in the
     * documentation and/or other materials provided with the
     * distribution.
     *
     * Neither the name of Texas Instruments Incorporated nor the names of
     * its contributors may be used to endorse or promote products derived
     * from this software without specific prior written permission.
     *
     * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
     * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
     * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
     * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
     * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
     * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
     * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
     * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
     * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
     * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
     * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
     *
     */
    
    #include <ti/csl/csl_utils.h>
    #include <ti/csl/soc/am572x/src/cslr_soc.h>
    #include <ti/csl/csl_types.h>
    #include <ti/drv/gpio/GPIO.h>
    #include <ti/drv/gpio/soc/GPIO_soc.h>
    
    /* Number of GPIO ports */
    #define CSL_GPIO_PER_CNT    8U
    
    /* GPIO Driver hardware attributes */
    GPIO_v1_hwAttrs_list GPIO_v1_hwAttrs = {
        {
    #ifdef _TMS320C6X
           CSL_DSP_GPIO1_REGS,
           OSAL_REGINT_INTVEC_EVENT_COMBINER,
    #elif defined(__ARM_ARCH_7A__)
           CSL_MPU_GPIO1_REGS,
           61,
    #else
           CSL_IPU_GPIO1_REGS,
           51,
    #endif
           0,
           55,
           0
        },
        {
    #ifdef _TMS320C6X
           CSL_DSP_GPIO2_REGS,
           OSAL_REGINT_INTVEC_EVENT_COMBINER,
    #elif defined(__ARM_ARCH_7A__)
           CSL_MPU_GPIO2_REGS,
           62,
    #else
           CSL_IPU_GPIO2_REGS,
           52,
    #endif
           0,
           56,
           0
        },
        {
    #ifdef _TMS320C6X
           CSL_DSP_GPIO3_REGS,
           OSAL_REGINT_INTVEC_EVENT_COMBINER,
    #elif defined(__ARM_ARCH_7A__)
           CSL_MPU_GPIO3_REGS,
           63,
    #else
           CSL_IPU_GPIO3_REGS,
           49,
    #endif
           0,
           57,
           0
        },
        {
    #ifdef _TMS320C6X
           CSL_DSP_GPIO4_REGS,
           OSAL_REGINT_INTVEC_EVENT_COMBINER,
    #elif defined(__ARM_ARCH_7A__)
           CSL_MPU_GPIO4_REGS,
           64,
    #else
           CSL_IPU_GPIO4_REGS,
           49,
    #endif
           0,
           58,
           0
        },
        {
    #ifdef _TMS320C6X
           CSL_DSP_GPIO5_REGS,
           OSAL_REGINT_INTVEC_EVENT_COMBINER,
    #elif defined(__ARM_ARCH_7A__)
           CSL_MPU_GPIO5_REGS,
           65,
    #else
           CSL_IPU_GPIO5_REGS,
           49,
    #endif
           0,
           59,
           0
        },
        {
    #ifdef _TMS320C6X
           CSL_DSP_GPIO6_REGS,
           OSAL_REGINT_INTVEC_EVENT_COMBINER,
    #elif defined(__ARM_ARCH_7A__)
           CSL_MPU_GPIO6_REGS,
           66,
    #else
           CSL_IPU_GPIO6_REGS,
           49,
    #endif
           0,
           60,
           0
        },
        {
    #ifdef _TMS320C6X
           CSL_DSP_GPIO7_REGS,
           OSAL_REGINT_INTVEC_EVENT_COMBINER,
    #elif defined(__ARM_ARCH_7A__)
           CSL_MPU_GPIO7_REGS,
           67,
    #else
           CSL_IPU_GPIO7_REGS,
           49,
    #endif
           0,
           61,
           0
        },
        {
    #ifdef _TMS320C6X
           CSL_DSP_GPIO8_REGS,
           OSAL_REGINT_INTVEC_EVENT_COMBINER,
    #elif defined(__ARM_ARCH_7A__)
           CSL_MPU_GPIO8_REGS,
           153,
    #else
           CSL_IPU_GPIO8_REGS,
           49,
    #endif
           0,
           0,
           0
        },
    };
    
    /* GPIO configuration structure */
    CSL_PUBLIC_CONST GPIOConfigList GPIO_config =
    {
        {
            &GPIO_FxnTable_v1,
            NULL,
            NULL
        },
        /* "pad to full predefined length of array" */
        {
            NULL,
            NULL,
            NULL
        },
        {
            NULL,
            NULL,
            NULL
        }
    };
    

  • Hi Parth,

    To Answer your question, about UART:

    - The size of data is less than 50 bytes.

    - The device is an Advantech ADAM Module 4150.

    The issue, however, does not happen if we run only the task responsible for UART communication.

    As explained above, it seems to be race condition/conflict.

    Best

    Reza

  • Hi Reza,

    line1EventId is used by DSP and should not affect your application as it is running on IPU. Please refer to thread  https://e2e.ti.com/support/processors-group/processors/f/processors-forum/574005/rtos-tmdsevm572x-question-on-structure-gpio_v1_hwattrs_s for details

    Why is lline1IntNum = 49 is used for GPIO3 to GPIO8 in the file? Sholudn't the lline1IntNum be unique among different GPIOs?

    There is no restriction as such, Interrupt number should be configurable, but for some reason driver is using it this way.

    It seems to be a race condition/conflict between the UART operations and an interrupt received on a GPIO.

    Can you please share mode details over here:
    1) What GPIO interrupt are doing on trigger?
    2) How frequently these interrupts are triggered?
    3) How are you configuring these interrupts?

    Regards,
    Parth

  • Hi Parth, 

    Thanks for your reply.

    1. Even with a simplified version where we print two lines of logs in the ipu trace (entered/exited), the issue still happens.

    2. 16 interrupts are coming every 40 milliseconds (the first 8 comes in 12 ms and the second 8 in the rest of the time) , then there is a delay of 60 milliseconds, and another sequence of 16 interrupt comes.

    The answer to your 3rd q: this is how we configure the interrupt from GPIO8:

    A. We overwrite the default hwAttrs found in packages/ti/drv/gpio/soc/am572x/GPIO_soc.c

    (please take a look at the last part of the code below for GPIO8, line1IntNum = 49 , line2EventId = 0)

    /**
     * GPIO Driver hardware attributes 
     * this overrides the default hwAttrs found in packages/ti/drv/gpio/soc/dra72x/GPIO_soc.c
     * we add the IPC_REGISTER_OFFSET and can modify the interrupt id, if needed
     *
     * typedef struct GPIO_v1_HwAttrs_s entries:
     *  uint32_t baseAddr;      GPIO Peripheral base address
     *  uint32_t line1IntNum;   GPIO Peripheral interrupt vector
     *  uint32_t line2IntNum;   GPIO Peripheral interrupt vector
     *  uint32_t line1EventId;  GPIO Peripheral interrupt vector
     *  uint32_t line2EventId;  GPIO Peripheral interrupt vector
     */
    
    GPIO_v1_hwAttrs_list GPIO_v1_hwAttrs = {
        {
           CSL_IPU_GPIO1_REGS+IPC_REGISTER_OFFSET,
           51,
           0,
           55,
           0
        },
        {
           CSL_IPU_GPIO2_REGS+IPC_REGISTER_OFFSET,
           52,
           0,
           56,
           0
        },
        {
           CSL_IPU_GPIO3_REGS+IPC_REGISTER_OFFSET,
           49,
           0,
           57,
           0
        },
        {
           CSL_IPU_GPIO4_REGS+IPC_REGISTER_OFFSET,
           49,
           0,
           58,
           0
        },
        {
           CSL_IPU_GPIO5_REGS+IPC_REGISTER_OFFSET,
           49,
           0,
           59,
           0
        },
        {
           CSL_IPU_GPIO6_REGS+IPC_REGISTER_OFFSET,
           49,
           0,
           60,
           0
        },
        {
           CSL_IPU_GPIO7_REGS+IPC_REGISTER_OFFSET,
           49,
           0,
           61,
           0
        },
        {
           CSL_IPU_GPIO8_REGS+IPC_REGISTER_OFFSET,
           49,
           0,
           0,
           0
        },
    };

    B. We then configure the crossbar to route the GPIO8_IR_1 to to IPU1_IRQ_49

    DIRQ_voCustomCSL_xbarIpuIrqConfigure(1, CSL_XBAR_INST_IPU1_IRQ_49, CSL_XBAR_GPIO8_IRQ_1);

    void DIRQ_voCustomCSL_xbarIpuIrqConfigure(Uint32 ipuInst, Uint32 irqNumIdx, CSL_XbarIrq xbarIrq)
    {
        CSL_control_intr_dmaRegs *ctrlCoreIntrDmaReg =
            (CSL_control_intr_dmaRegs *) CSL_MPU_IRQ_DMARQ_CROSSBAR_REGISTERS_REGS;
        CSL_control_coreRegs *ctrlCoreReg =
            (CSL_control_coreRegs *) CSL_MPU_CTRL_MODULE_CORE_CORE_REGISTERS_REGS;
        Uint32 regIdx = (irqNumIdx - 1U) / 2U;
        Uint32 regLsb = (((irqNumIdx - 1U) % 2U) * 16U);
        Uint32 regMsb = regLsb + 16U;
    
    
        if(1U == ipuInst)
        {
            ctrlCoreReg->MMR_LOCK_2 = MAGIC_UNLOCK;
            CSL_FINSR(ctrlCoreIntrDmaReg->IPU1_IRQ[regIdx],
                regMsb, regLsb, xbarIrq);
            ctrlCoreReg->MMR_LOCK_2 = MAGIC_LOCK;
        }
        if(2U == ipuInst)
        {
            ctrlCoreReg->MMR_LOCK_2 = MAGIC_UNLOCK;
            CSL_FINSR(ctrlCoreIntrDmaReg->IPU2_IRQ[regIdx],
                regMsb, regLsb, xbarIrq);
            ctrlCoreReg->MMR_LOCK_2 = MAGIC_LOCK;
        }
    
    }
    

  • Hi again,

    I also did some investigation regarding the UART communication and checked the uart-lld git repo:

    this is the commit we use from UART_v1.c ,as part of the ti-pdk 04.01.00.06 release:

    https://git.ti.com/cgit/keystone-rtos/uart-lld/commit/src/v1/UART_v1.c?id=835a8ddf52c390ead51e32fe882c3e32c460dc7b

    It could be seen from here, there are several commits (including bug fixes) which are not included in the version we use:

    keystone-rtos/uart-lld - Project to include GIT repos for RTOS components including user mode CSL & NetCP LLDs (CPPI/QMSS/PA/RM), PDK components (NWAL/PKTLIB) and Transport Net Lib components (NetAPI/HPLIB) - UART Driver Support for Processor SDK-RTOS. (ti.com)

    As an example, we had issue with reading the last byte of the write echo in UART_RETURN_FULL mode which is apparently fixed in this commit:

    keystone-rtos/uart-lld - Project to include GIT repos for RTOS components including user mode CSL & NetCP LLDs (CPPI/QMSS/PA/RM), PDK components (NWAL/PKTLIB) and Transport Net Lib components (NetAPI/HPLIB) - UART Driver Support for Processor SDK-RTOS. (ti.com)

    As you have a better overview from the changes, could any of these commits be related to the issue we have (for example the ones below, among others)?

    keystone-rtos/uart-lld - Project to include GIT repos for RTOS components including user mode CSL & NetCP LLDs (CPPI/QMSS/PA/RM), PDK components (NWAL/PKTLIB) and Transport Net Lib components (NetAPI/HPLIB) - UART Driver Support for Processor SDK-RTOS. (ti.com)

    keystone-rtos/uart-lld - Project to include GIT repos for RTOS components including user mode CSL & NetCP LLDs (CPPI/QMSS/PA/RM), PDK components (NWAL/PKTLIB) and Transport Net Lib components (NetAPI/HPLIB) - UART Driver Support for Processor SDK-RTOS. (ti.com)

    keystone-rtos/uart-lld - Project to include GIT repos for RTOS components including user mode CSL & NetCP LLDs (CPPI/QMSS/PA/RM), PDK components (NWAL/PKTLIB) and Transport Net Lib components (NetAPI/HPLIB) - UART Driver Support for Processor SDK-RTOS. (ti.com)

    Thanks,

    Reza

  • Hi Reza,

    As you have a better overview from the changes, could any of these commits be related to the issue we have

    I don't think that would help as this is not a UART issue as such because with the GPIO interrupts disabled, your writes are working fine.

    I am suspecting something is going wrong during interrupt handling.

    Can you please try the following:
    Initialize a global counter and increment it inside the interrupt handler. Remove everything else from the handler and see if your UART writes still hangs.

    Regards,
    Parth

  • Hi Parth,

    I will do a test as you explained and report the outcome here.

    Indeed, the reproducibility of the issue depends on the time we spend in the callback.

    This is the callback code, in the normal operation of the system which fails after a long random time (up to few days):

    void DIFE_voEnIlluCallback(void) {
    
    	// Store timer counter
    
    	uint32_t ulTickTimestamp = TIMERCounterGet(ICTRegisterBaseAddress);
    	DIFE_monitorData[(DIFE_currentMonitorDataIndex + 1) % MONITOR_DATA_ENTRIES].ulaTickTimestamp[DIFE_ulIlluEnaInterruptCounter
    			& 0x000F] = ulTickTimestamp;
    
    	// Increment counter
    	DIFE_ulIlluEnaInterruptCounter++;
    
    	if (GPIO_read(ILLU_ENA) == 1) { // Rising edge
    		// Post new subframe event
    		Event_post(DIFE_eInterruptTrigger, NEW_SUBFRAME_EVENT);
    	} else { // Falling edge
    			 //a local variable to compute the pulse number based on interrupt counter
    		volatile uint32_t ulPulseCnt;
    
    		// Store sub-frame counter modulo 4096(12 bits)
    		ulPulseCnt = (DIFE_ulIlluEnaInterruptCounter >> 1);
    		DIFE_monitorData[(DIFE_currentMonitorDataIndex + 1)
    				% MONITOR_DATA_ENTRIES].usaSubFrameCounter[((ulPulseCnt - 1)
    				& 0x7)] = (ulPulseCnt & 0xFFF);
    
    		// Check end-octet condition
    		if ((DIFE_ulIlluEnaInterruptCounter != 0) && !(ulPulseCnt & 0x7)) {
    			Event_post(DIFE_eInterruptTrigger, OCTET_END_EVENT);
    		}
    		//set the illusafecheckcounter to 0,
    		//so the check starts after the first falling edge received after the acquisition start
    		SLFT_scIlluSafeCheckCounter = 0;
    	}
    }
    

    As soon as I add logs at the entrance and exit of the callback, no matter if I include or exclude the code above, the issue happens within an hour or less:

    sprintf(s, "DIFE_voEnIlluCallback entered!");
    Log_print0(Diags_INFO, s);
    
    ....
    
    sprintf(s, "DIFE_voEnIlluCallback about to Exit!");
    Log_print0(Diags_INFO, s);

    Now, with the version of the callback you proposed, I may need to run the system for a few days before the issue happens. I will run 2 test benches to increase the chance.

    Another idea we had to isolate the issue further, was to disable the interrupt on UART and works with it in polling mode. Will let you know the outcome of this test as well.

    Thanks,

    Reza

     

    Bers,

    Reza


     

  • Hi Parth,

    None of the tests failed after 3 days:

    Neither the one you proposed with only increasing a global counter within the GPIO callback,

    Nor the one where we disabled the UART interrupt and worked with UART in the polling mode.

    It seems now we have at least a workaround as we can use UART in polling mode and rely on GPIO callbacks.

    However, please let us know how we should proceed to root cause the issue when both interrupts are enabled and GPIO callback runs the complete code. 

    We would like to understand the root cause and use the UART in interrupt mode.

    Are you aware of any bug fixed in the pdk/sysbios related to this issue of handling the concurrent interrupts compared to the version we use (ti-processor-sdk-rtos-am57xx-evm-04.01.00.06-Linux-x86)? If not, where else do you think we should look in our configuration/code to root cause the issue?

    Thanks,

    Reza 

  • Hi Reza,

    Neither the one you proposed with only increasing a global counter within the GPIO callback,

    Is this working with UART in interrupt mode as well?

    Also, can you please share what you are doing in DIFE_monitorData?

    Regards,
    Parth

  • Hi Parth,

    Is this working with UART in interrupt mode as well?

    Yes, this version (only incrementing a counter in the GPIO callback) worked with the UART in interrupt mode. However, as I said, according to our tests the reproducibility of the issue depends on the time we spend in the GPIO callback.

     DIFE_monitorData is not a function, but an array. We are modifying its contents in the GPIO callback.

    Nevertheless, the code we execute in the callback could not be guilty. As explained above, the issue happens even if I completely comment this code and keep only two lines of logs in the IPU trace in the GPIO callback (the code below):


    void DIFE_voEnIlluCallback(void) {
    
    sprintf(s, "DIFE_voEnIlluCallback entered!");
    Log_print0(Diags_INFO, s);
    
    
    sprintf(s, "DIFE_voEnIlluCallback about to Exit!");
    Log_print0(Diags_INFO, s);
    
    }

  • Hi Reza,
    Can you try one more thing:

    This is the callback code, in the normal operation of the system which fails after a long random time (up to few days):

    In your call back function, just comment out 

    Event_post(DIFE_eInterruptTrigger, NEW_SUBFRAME_EVENT)

    and 

    Event_post(DIFE_eInterruptTrigger, OCTET_END_EVENT);

    Keeping everything else same, can you try and see if you see the failure with UART in interrupt mode?

    Regards,
    Parth

  • Hi Parth,

    In your call back function, just comment out 

    Event_post(DIFE_eInterruptTrigger, NEW_SUBFRAME_EVENT)

    and 

    Event_post(DIFE_eInterruptTrigger, OCTET_END_EVENT);

    Ok, I will run this test as well and let you know the outcome.

    However, as I said, the issue happens even with a GPIO interrupt containing only two lines of logs (see the code below):

    void DIFE_voEnIlluCallback(void) {
    
    sprintf(s, "DIFE_voEnIlluCallback entered!");
    Log_print0(Diags_INFO, s);
    
    
    sprintf(s, "DIFE_voEnIlluCallback about to Exit!");
    Log_print0(Diags_INFO, s);
    
    }