UART Driver issue - Interrupt lost venerability

Hi Ashish,

We will look into it and come back to you. Thanks for your patience.

Hi Ashish,

The case when both TX and REC occur simultaneously the TX interrupt will not be serviced since the ISR will first attend to receive interrupt and then read the IIR again. Reading IIR clears any THR empty (THRE) interrupts that are pending.

For which part and device you are talking about ?
Will you please validate in which scenario both TX and RX interrupt will be generated simultaneously?
Have you validated the issue by reproducing it device or by doing some experiments ? I doubt TX and RX interrupt might not occur simultaneously.

I have checked the uart driver code. Their is a function called "UartintrHandler" which is called by "uartTxSwiHandler" and "uartRxSwiHandler" inside which we have a if else block(line 1912).

Is my understanding correct ? Please validate

Hi Arvind,

This is TMS320C6748 chip. We are communicating with the WL1835MOD via UART0. I have set-up the uart for INTERRUPT mode and use GIO_read/GIO_write for RX/TX. We have observed that the  UART locks-up intermittently under heavy two way traffic. I have scoped TX, RX ,CTS & RTS lines. The HW flow control seems to be operating OK. Hence investigating the uart driver.

My understanding is GIO_write() calls GIO_submit() synchronously. The call graph looks like-

GIO_submit()()  -> uartMdSubmitChan() -> uartSubmitIoReq()  -> uartSubmitIoReqIntDma() 

uartSubmitIoReqIntDma will assign the passed packet to activeIOP then uartIntrEnable (Uart_Intr_THR // for transmit) and return IOM_PENDING. Now we Sync_wait in GIO_submit() for transfer completion.

When the THR gets empty, The uartIsr gets called at which time the IIR register gets read. The IIR register could well have the RDA bit set as well since there is a possibility of receive data also being available at the same time. The ISR will service read first since it is higher priority and then invoke Swi_post for Receive. (uartRxSwiHandler -> uartIntrHandler is called for receive).

Kindly Note that the uartIntrHandler shall re-enable RHR interrupt only and not THR (since channel mode == INPUT)  . Thus the TX interrupt is lost. (since the IIR register is re-read in uartIsr)

 

regards,

Ashish

Arvind,

In the begining of uartIntrHandler we see,
"EventCombiner_enableEvent(instHandle->deviceInfo.cpuEventNumber);" - line 1898

and then towards the end again,
EventCombiner_enableEvent(instHandle->deviceInfo.cpuEventNumber); - line 2015

where is this event disabled?

Regards,

Ashish

Hi Ashish,

EventCombiner_enableEvent(instHandle->deviceInfo.cpuEventNumber); - line 2015
where is this event disabled?

This event is disabled from inside the uartUnRegisterInterrupt() -line 1806
"EventCombiner_disableEvent(evt)." - line 1812

Hi Arvind,

Thank you for replying.

Yes, The event is disabled in the uartUnRegisterInterrupt(). However I did not see uartUnRegisterInterrup() being called form anywhere else apart from uartMdUnBindDev() and that is done at the time of closing of the instance. 

Request your kind help in investigating the reason for uart driver is lock-up under heavy TX and RX. We continue to see this issue quite frequently and need a resolution at the earliest.

Also, Do you agree with my earlier post regarding the possibility of TX interrupt being lost if both TX and RX interrupts occur simultaneously?

Regards,

Ashish

Hi Ashish,

Apologies for the delayed response.

Yes I do agrees with you, if both TX and RX interrupts occur simultaneously, TX interrupt will lost.
Actually driver is designed and its functionality is in this way only , that it will handle only one interrupt at a time either Rx interrupt or Tx interrupt.

But i will suggest you to control receiver and transmitter in your application by doing some synchronization.
As you were using GIO_read/GIO_write for RX/TX, you can use any locking mechanism and can make sure that both (Tx and Rx) interrupt should not occur simultaneously.

Hi Arvind,

I do not feel it is possible to prevent TX and RX interrupts from occurring simultaneously. From the application, we can control when to send data but have no control when the data is received.

The GIO_read/GIO_write being synchronous calls we will not be able to do synchronization particularly since our application is Bluetooth based where RX depends on TX. (i.e we can receive a RX packet following 20 or more TX packets and vice versa)

BTW i have also tried Polled mode without success.

regards,

Ashish

 

Hi Arvind,

I spoke with Ashish today and wanted to provide the full picture of how he mitigated the occurence of simultaneous interrupts for RX and TX. The customer is handling this in the ISR by taking the checking of THRE interrupt outside of the "else if" into a separate "if" and inside there setting a TX pending flag. Then upon exiting the main while loop, if the TX pending flag is set, the SWI for TX processing with be posted.

In order to see this exactly I am attaching the unmodified ISR and then the modified ISR and a patch diff file (you could e.g. use meld to see side by side).

This mitigation fix of the ISR for simultaneous occuring RX and TX interrupts and deferred TX processing has worked fine with baudrates upto 115200.

The first question would be to comment on the exact modified ISR and whether there is agreement on the defer-tx fix approach.

 

Now the customer needs to increase baudrate on the UART to 750000 for higher throughput to the Bluetooth UART.

The concern now is that TX interrupts may occur even at higher rates. There is a possibility that before TX interrupt detection is happening other TX interrupts may occur and TX may be missed. However since the processor is in control of TX, maybe the rate can be controlled. Given the current ISR with deferred TX, are there any additional suggestions to improve this?

 

Thanks,

--Gunter

uartissue.zip

 

 

Hi Gunter,

The first question would be to comment on the exact modified ISR and whether there is agreement on the defer-tx fix approach.

The defer-TX ISR looks good and i think it will work at higher rates also.

Regarding the possibility that before TX interrupt detection happens and other Tx interrupt may occur and TX may be missed,
It might not be possible as 2nd time TX interrupt will only occur when previously triggered interrupt will be cleared.
and that happens when,
Firstly, IIR register will be read(interrupt identification ) and TX interrupt will be cleared.
Secondly, at the time when data is been loaded into THR Register and TX interrupt will be cleared.

Hi Arvind,

Thank you for looking into the modified UartIsr.

Well, Event after the modifications we still continue to see TX lock-up and hence I continue to investigate the Uart driver.

I would now like to draw your attention to uartIntrHandler [line 1881] in Uart.c.  In the uartIntrHandler we have two function calls.

1. Uart_localCompleteCurrentIO  [line no 1953]

2. Uart_loadPendedIops [line no 1959]

The Uart_localCompleteCurrentIO function powers OFF (line 3094) the uart peripheral and then calls the call back function registered with it.

The following function "Uart_loadPendedIops" checks if there are any pending IO requests and if yes enables the respective interrupts to initiate new IO transfer. However this function does not power back ON the uart peripheral.  This could be an issue if the application calls GIO_write is called from different threads?

Now drawing your attention to uartIsr once again.

The IIR is read into local variable "status" [line 3509]. The contents of "status" will get overwritten @ line 3578. This is before the TX interrupts are serviced?

Eagerly awaiting your response.

regards,

Ashish

 

Hi Arvind,

Ashish and I talked again, and we want to separate several potential problems, Ashish has been looking at:

[1] usage of Uart_localLpscOff()

To summarize, the opposing function Uart_localLpscOn() is used in the GIO_write context, which is typically at the Task level. GIO_write causes the eventually uartMdSubmitChan() which then calls Uart_localLpscOn() at line 1182 in Uart.c. With this multiple tasks could call this GIO_write, which will then cause transfers to be added to a queue, which needs to be serviced.

The function Uart_localLpscOff() is called in the SWI context inside the uartIntrHandler() and as you can see in line 1929, the check for zero bytes remaining TRUE will call Uart_localCompleteCurrentIO() and thereby Uart_localLpscOff(), and put the UART into a power off state. Now if there are additional pending transfers (line 1959) in the queue, the UART is still powered off and won't be able to transfer and send interrupts. These additional pending transfers in queue could come from other tasks with GIO_write e.g.

Our understanding is that only a fresh GIO_write would be able to turn the UART back on. Which is very concerning and should prevent to handle pending transfers.

Could you clarify?

 

[2] usage of variable "status" in the uartIsr()

The variable "status" is used for two different purposes, (1) hold IIR value (2) if BIOS_PWRM_ENABLE is TRUE, for some power dependency. In the customer modified uartIsr() this is not good, since status later is still expected to hold the IIR value, but is potential already overwritten by (2).

So we are separating the variables now and retest.

 

[3] How is the BIOS_PWRM_ENABLE switch enabled/disabled? Is this done via the BIOS config file?

 

Thanks,

--Gunter

Hi Arvind,

Ashish did further testing. He did this:

[1] comment out Uart_localLpscOff() inside uartIntrHandler()
[2] separate the usage of "status" variable so that the first IIR read will not be overwritten by another purpose

However the UART lockups still persist and happen fairly frequently at the higher 750000 baudrate.

Do you have any additional ideas on how to root cause this problem?

Thanks,
--Gunter

Hi Ashish,

Have you tried GIO_Submit call a with callback function.
Checkout this thread, Specially ( Raghavendra's Comments)

e2e.ti.com/.../183633