This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM625: AM625x UART 2Mbps packet loss issue

Part Number: AM625
Other Parts Discussed in Thread: SK-AM62B

AM625x UART 2Mbps packet loss issue

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1504677/am625-run-multiple-serial-ports-simultaneously-and-toggle-the-wired-nic-to-introduce-interference

The above is about the previous issue of packet loss at 2Mbps for AM625x UART. Previously, when no GPU was used, UART at 2Mbps did not experience packet loss, and increasing CPU usage with stress-ng also did not cause packet loss;
Recently, I have noticed that when the GPU usage exceeds 12% (under Qt calling OpenGL for real-time rendering), UART starts to encounter OE errors (as shown by cat /proc/tty/driver/serial)


root@am62xx-evm:~# cat /sys/kernel/debug/pvr/status
Driver Status:   OK
Device ID: 0:128
Firmware Status: OK
Server Errors:   0
HWR Event Count: 0
CRR Event Count: 0
SLR Event Count: 0
WGP Error Count: 0
TRP Error Count: 0
FWF Event Count: 0
APM Event Count: 10
GPU Utilisation: 14%
DM Utilisation:  VM0
           2D:   0%
         GEOM:   0%
           3D:  13%
          CDM:   0%
          
root@am62xx-evm:~# cat /proc/tty/driver/serial
serinfo:1.0 driver revision:
0: uart:unknown port:00000000 irq:0
1: uart:unknown port:00000000 irq:0
2: uart:8250 mmio:0x02800000 irq:240 tx:258355 rx:141 RTS|DTR|DSR
3: uart:8250 mmio:0x02810000 irq:241 tx:0 rx:0 RTS|DTR
4: uart:8250 mmio:0x02820000 irq:242 tx:19305 rx:325769 RTS|DTR|DSR
5: uart:unknown port:00000000 irq:0
6: uart:8250 mmio:0x02840000 irq:243 tx:0 rx:1 brk:1 RTS|DTR|DSR
7: uart:8250 mmio:0x02850000 irq:244 tx:87498 rx:37328920 oe:440 RTS|DTR|DSR
8: uart:8250 mmio:0x02860000 irq:245 tx:18791 rx:195160 RTS|DTR|DSR
9: uart:unknown port:00000000 irq:0
10: uart:unknown port:00000000 irq:0
11: uart:unknown port:00000000 irq:0

ttyS7: Packet loss is very serious

 

SDK:          ti-processor-sdk-linux-am62xx-evm-10.01.10.04-Linux-x86-Install.bin
Linux kernel: 6.6.58+


Only when the GPU usage rate is above 12%, does the UART driver layer start to encounter OE errors;
Previously, TI technical personnel responded that UART does not support DMA; it may be due to high bandwidth interruptions from the GPU, causing UART to operate in non-DMA mode and resulting in packet loss.

  • Hi,

    Sorry I didn't get a chance to look into this today. I will get back to you by this Thursday.

  • Hi Dongliang,

    The oe error in procfs means the UART RX FIFO overflows.

    In the previous e2e thread, I have mentioned irq affinity, do you use it to handle the irq of UART and GPU on the two A53s separately?

  • Check interrupt number
    root@am62xx-evm:~# cat /proc/interrupts
               CPU0       CPU1       
     11:   18315696   20059903     GICv3  30 Level     arch_timer
     14:          0          0     GICv3 130 Level     pinctrl
     15:          0          0     GICv3  23 Level     arm-pmu
     16:      53512          0     GICv3  66 Level     4d000000.mailbox thr_012
     25:          0          0     GICv3 269 Level     2b10000.audio-controller_rx
     26:          0          0     GICv3 270 Level     2b10000.audio-controller_tx
     27:     117305          0     GICv3 193 Level     20000000.i2c
     28:         74          0     GICv3 194 Level     20010000.i2c
     29:      49429          0     GICv3 195 Level     20020000.i2c
     48:     986221          0  MSI-INTA 1714176 Edge      485c0100.dma-controller chan0
     60:      12118          0  MSI-INTA 1714688 Level     485c0100.dma-controller chan0
     78:          0          0  MSI-INTA 1715718 Edge      485c0100.dma-controller chan1
     96:          0          0  MSI-INTA 1716230 Level     485c0100.dma-controller chan1
    118:         30          0  MSI-INTA 1970707 Level     8000000.ethernet-tx0
    126:          0         32  MSI-INTA 1970715 Level     8000000.ethernet-tx1
    134:         26          0  MSI-INTA 1970723 Level     8000000.ethernet-tx2
    142:          0         32  MSI-INTA 1970731 Level     8000000.ethernet-tx3
    150:         45          0  MSI-INTA 1970739 Level     8000000.ethernet-tx4
    158:          0         48  MSI-INTA 1970747 Level     8000000.ethernet-tx5
    166:      15749          0  MSI-INTA 1970755 Level     8000000.ethernet-tx6
    174:          0         29  MSI-INTA 1970763 Level     8000000.ethernet-tx7
    208:     127573          0  MSI-INTA 1971731 Level     8000000.ethernet
    240:     175044          0     GICv3 210 Level     2800000.serial
    241:          0          0     GICv3 211 Level     2810000.serial
    242:     343207          0     GICv3 212 Level     2820000.serial
    243:          0          0     GICv3 214 Level     2840000.serial
    244:   23615107          0     GICv3 215 Level     2850000.serial
    245:     600530          0     GICv3 216 Level     2860000.serial
    246:          0          0     GICv3 134 Level     8000000.ethernet
    256:          0          0     GICv3 220 Level     xhci-hcd:usb1
    257:         34          0     GICv3 258 Level     xhci-hcd:usb2
    258:      59816          0     GICv3 165 Level     mmc0
    259:         52          0     GICv3 114 Level     mmc2
    308:          0          0      GPIO  38 Edge    -davinci_gpio  1-0039
    369:         22          0      GPIO   3 Edge    -davinci_gpio  egalax_i2c
    418:    2575257          0     GICv3 116 Level     tidss
    419:          0          0     GICv3 115 Level     mmc1
    420:    3646414          0     GICv3 118 Level     pvrsrvkm
    IPI0:    296497     716920       Rescheduling interrupts
    IPI1:   7130301   21971156       Function call interrupts
    IPI2:         0          0       CPU stop interrupts
    IPI3:         0          0       CPU stop (for crash dump) interrupts
    IPI4:         0          0       Timer broadcast interrupts
    IPI5:    937695     881413       IRQ work interrupts
    IPI6:         0          0       CPU wake-up interrupts
    Err:          0
    root@am62xx-evm:~#

    UART6 2Mbps interrupt number 244 CPU0
    244: 23615107 0 GICv3 215 Level 2850000.serial

    GPU driver interrupt number 420 CPU0
    420: 3646414 0 GICv3 118 Level pvrsrvkm

    Bind GPU interrupt number 420 to CPU1
    # echo 2 > /proc/irq/420/smp_affinity


    root@am62xx-evm:~# cat /proc/interrupts
               CPU0       CPU1       
    244:   23757254          0     GICv3 215 Level     2850000.serial
    420:    3666378       2380     GICv3 118 Level     pvrsrvkm

    root@am62xx-evm:~# cat /proc/interrupts
               CPU0       CPU1       
    244:   23757716          0     GICv3 215 Level     2850000.serial
    420:    3666378       2454     GICv3 118 Level     pvrsrvkm

    Binding successful
    Check UART packet loss rate command
    watch -n1 -d cat /proc/tty/driver/serial

    Check GPU usage command
    watch -n1 -d cat /sys/kernel/debug/pvr/status

    Check interrupt number growth command
    watch -n1 -d cat /proc/interrupts



    result:
    Every 1.0s: cat /sys/kernel/debug/pvr/status                                                                                                                           am62xx-evm: Tue Jun 30 23:46:28 2026

    Driver Status:   OK

    Device ID: 0:128
    Firmware Status: OK
    Server Errors:   0
    HWR Event Count: 0
    CRR Event Count: 0
    SLR Event Count: 0
    WGP Error Count: 0
    TRP Error Count: 0
    FWF Event Count: 0
    APM Event Count: 37
    GPU Utilisation: 43%
    DM Utilisation:  VM0
               2D:   0%
             GEOM:   1%
               3D:  42%
              CDM:   0%


    Every 1.0s: cat /proc/tty/driver/serial                                                                                                                                am62xx-evm: Tue Jun 30 23:47:09 2026

    serinfo:1.0 driver revision:
    0: uart:unknown port:00000000 irq:0
    1: uart:unknown port:00000000 irq:0
    2: uart:8250 mmio:0x02800000 irq:240 tx:255277 rx:700 RTS|DTR|DSR
    3: uart:8250 mmio:0x02810000 irq:241 tx:0 rx:0 RTS|DTR
    4: uart:8250 mmio:0x02820000 irq:242 tx:5689 rx:218707 RTS|DTR|DSR
    5: uart:unknown port:00000000 irq:0
    6: uart:8250 mmio:0x02840000 irq:243 tx:0 rx:0 DSR
    7: uart:8250 mmio:0x02850000 irq:244 tx:71224 rx:25825006 oe:81 RTS|DTR|DSR
    8: uart:8250 mmio:0x02860000 irq:245 tx:13002 rx:135524 RTS|DTR|DSR
    9: uart:unknown port:00000000 irq:0
    10: uart:unknown port:00000000 irq:0
    11: uart:unknown port:00000000 irq:0


    Experiment summary:
    In the case of GPU driver bound to CPU0 and UART bound to CPU1 (network interrupt bound to CPU0):
    The GPU usage rate reached around 15%, and two OE errors occurred in about 8 minutes (compared to one OE error every 50 seconds before). The problem still persists;
    When the GPU usage rate reaches 45%, the frequency of triggering this error is alleviated, but the problem still persists (with a probability of two OE errors occurring every 20 seconds)

    Bind CPU cores, there are improvements but the problem still persists.

  • Thanks for the test.

    Bind GPU interrupt number 420 to CPU1
    # echo 2 > /proc/irq/420/smp_affinity

    Since CPU0 handles the irq for most of the modules by default. Can you please try to move the UART6 irq to CPU1 to see if this makes difference?

    echo 1 > /proc/irq/244/smp_affinity_list

    By the way, have you tested with the RT kernel to see if the problem is the same?

  • Both CPU0 and CPU1 have been tried in exchange, but the problem still persists;
    I haven't tried the real-time kernel yet. I'll give it a try on my end in the next couple of days.

  • Thanks for the experiments. I will try to replicate the issue on my evm later next week. I am currently out of office for the holidays and will be back mid of next week.

  • I added "maxcpus=2" into the kernel bootargs so that only 2 CPUs are running on SK-AM62B EVM. Then I ran serialcheck loopback on UART5 with baud 2Mbps, along with gpu demo program "glmark2-es2-drm" which makes GPU Utilization almost up to 100%, but I don't see OE error on UART5.

    Can you please run glmark2-es2-drm instead of your Qt application in your test to see if OE error still happens? I want to check if we are on the same page.

  • Test record: UART 2Mbps bound to CPU1, 51KB/s data upload, stress-ng not started, glmark2 running in a loop uboot setenv bootargs "setenv bootargs console=ttyS2,115200n8tyS2,115200n8 root=/dev/mmcblk0p1 ro rootfstype=ext3 rootwait maxcpus=2"

    7.10 21:58 pe:1 oe:75

    7.11 08:44 pe:1 oe:1476

  • Test record: UART 2Mbps bound to CPU1, with 51KB/s data upload, stress-ng not started, glmark2 running in a loop

    Time                Result
    7.10 21:58       pe:1 oe:75
    7.11 08:44       pe:1 oe:1476

    Running glmark2 about twice will result in OE, but sometimes it doesn't appear even after one run

  • I tested glmark2-es2-drm in a while loop and serialcheck on UART5 on SK-AM62B EVM for hours but still don't see the OE error.

    How do you use the 2Mbps UART port? test it in external loopback or just receive data from another UART device?

    Your kernel has the following two patches I provided before, right?

    1. tty: manual revert: tty: Prevent writing chars during tcsetattr TCSADRAIN/FLUSH
    2. 8250_omap: change tx loadsz from 64 to 48 bytes

    What is the DDR size on your board?

  • I have been testing with UART external loopback using serialcheck and didn't see the OE problem, but now I switched to AM62x UART receiving data from an external device, now I see problems, only when running glmark2-es2-drm program. I am not sure yet if this is the same problem that you have, but I am looking into it. I will keep you updated.

  • By the way, you don't need to test with RT kernel if you have not done so. I still see the OE issue with RT kernel.

  • Hi Dongliang,

    Finally I am able to clear all the blockers and get the UART OE issue resolved in my test.

    In summary, the solution is:

    1. Use CPU isolation and IRQ affinity

    2. Bypass kernel crypto entrophy collection from the isolated CPU

    Details:

    1. CPU isolation and IRQ affinity

    We need to isolate CPU1 to only handle the UART IRQ on it. All other process and IRQ, including the UART user space program, are running CPU0.

    1.1. Add "isolcpus=1" to kernel parameters. You can add optargs="isolcpus=1" to U-Boot bootargs.

    1.2. Some ethernet IRQs are on CPU1 by default. Move them to CPU0:

        cd  /proc/
        for i in $(grep ether interrupts | cut -d: -f1); do echo 0 > irq/$i/smp_affinity_list; done

    1.3 Move the UART IRQ to CPU1. Using main_uart1 as an example:

        cd /proc/
        i=$(grep 2810000 interrupts | cut -d: -f1)
        echo 1 > irq/$i/smp_affinity_list

        Note: In non-RT kernel, the UART IRQ is not registered until the UART user space program started. So you either

        a. start the program then find the IRQ number and change its affinity, or

        b. modify the UART kernel driver to hard code the IRQ affinity in the driver

    2. bypass kernel crypto entrophy collection on the isolated CPU by applying the kernel patch:

    diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
    index 9489f93b3db3..90b1175b3bc6 100644
    --- a/kernel/irq/handle.c
    +++ b/kernel/irq/handle.c
    @@ -192,7 +192,11 @@ irqreturn_t handle_irq_event_percpu(struct irq_desc *desc)
     
            retval = __handle_irq_event_percpu(desc);
     
    -       add_interrupt_randomness(desc->irq_data.irq);
    +       /* cpu1 is isolated for critical tasks.
    +        * to reduce scheduling jitter, don't harvest entrophy from it
    +        */
    +       if (raw_smp_processor_id() != 1)
    +               add_interrupt_randomness(desc->irq_data.irq);
     
            if (!irq_settings_no_debug(desc))
                    note_interrupt(desc, retval);

    Most of my debug work was done on the RT kernel, which requires additional tuning. But the UART OE issue can be resolved on non-RT kernel with the procedure above. If you ever need to switch to RT kernel, please let me know.

  • Forgot to mention that I have changed RX_TRIGGER from 48 to 4 in drivers/tty/serial/8250/8250_omap.c to tolerate ISR scheduling jitter.

    I don't have time to test now but I think RX_TRIGGER can be set a little bigger, for example, to 24, to reduce the UART IRQ frequency. But setting it to 48 is too risky to OE issue for high baud rate.

  • Hi Bin:
    After making the modifications as you suggested, the issue did not occur during a 10-hour test, compared to the previous situation where it occurred every minute. Currently, the test is showing a GPU utilization of 70% and CPU utilization of 80%. With one UART at 2Mbps and 50KB/s, there has been no OE error for 10 hours. Thank you for your guidance.

  • Glad to hear the issue is resolved. Thanks for the update. 
    did you adjust RX_TRIGGER setting in your resolution?