PROCESSOR-SDK-AM437X: CPU stall on closing UART port

Yang Hsiung

CPU Stall due to memory access failure when closing UART port. Linux Kernel 4.4.57. The following is the kernel backtrace.

INFO: rcu_preempt self-detected stall on CPU

[16696455.621185] 0-...: (1654942656 ticks this GP) idle=9d3/140000000000002/0 softirq=4514/4514 fqs=0

[16696455.630380] (t=1669616376 jiffies g=790 c=789 q=213)

[16696455.635774] rcu_preempt kthread starved for 1669616376 jiffies! g790 c789 f0x0 s3 ->state=0x1

[16696455.644537] Task dump for CPU 0:

[16696455.648012] launchupgrade.s R running 0 425 1 0x00000007

[16696455.654619] Backtrace:

[16696455.657330] [<c0013470>] (dump_backtrace) from [<c001366c>] (show_stack+0x18/0x1c)

[16696455.665141] r7:00000098 r6:c0872154 r5:00000001 r4:cf438000

[16696455.671074] [<c0013654>] (show_stack) from [<c005a964>] (sched_show_task+0xbc/0x110)

[16696455.679064] [<c005a8a8>] (sched_show_task) from [<c005b954>] (dump_cpu_task+0x2c/0x30)

[16696455.687220] r5:80070193 r4:00000001

[16696455.691060] [<c005b928>] (dump_cpu_task) from [<c007908c>] (rcu_dump_cpu_stacks+0xa8/0xfc)

[16696455.699570] [<c0078fe4>] (rcu_dump_cpu_stacks) from [<c007ca84>] (rcu_check_callbacks+0x488/0x8c0)

[16696455.708765] r10:00000000 r9:c0872154 r8:c087214c r7:c08806ec r6:c08802c0 r5:c08802c0

[16696455.716864] r4:c0880744 r3:cf906000

[16696455.720705] [<c007c5fc>] (rcu_check_callbacks) from [<c00801ec>] (update_process_times+0x40/0x6c)

[16696455.729815] r10:c08812cc r9:cf906000 r8:c0090a34 r7:003b5156 r6:85d08b0d r5:00000000

[16696455.737914] r4:cf438000

[16696455.740710] [<c00801ac>] (update_process_times) from [<c0090a30>] (tick_sched_handle+0x50/0x54)

[16696455.749646] r5:cf907be8 r4:c0881bc0

[16696455.753488] [<c00909e0>] (tick_sched_handle) from [<c0090a98>] (tick_sched_timer+0x64/0xb8)

[16696455.762086] [<c0090a34>] (tick_sched_timer) from [<c0080f24>] (__hrtimer_run_queues.constprop.6+0x138/0x1d0)

[16696455.772148] r7:00000000 r6:c08812c0 r5:c0881bc0 r4:c0881280

[16696455.778081] [<c0080dec>] (__hrtimer_run_queues.constprop.6) from [<c00811ec>] (hrtimer_interrupt+0xb4/0x260)

[16696455.788144] r10:c0881338 r9:cf907af0 r8:ffffffff r7:00000003 r6:cf906000 r5:7fffffff

[16696455.796243] r4:c0881280

[16696455.799037] [<c0081138>] (hrtimer_interrupt) from [<c0015f70>] (twd_handler+0x38/0x48)

[16696455.807193] r10:00000000 r9:cf9520c0 r8:ef6c9140 r7:00000012 r6:ef01bbc0 r5:c089aac8

[16696455.815292] r4:00000001

[16696455.818085] [<c0015f38>] (twd_handler) from [<c0073814>] (handle_percpu_devid_irq+0x68/0x80)

[16696455.826761] r5:c089aac8 r4:ef004d80

[16696455.830600] [<c00737ac>] (handle_percpu_devid_irq) from [<c006f6dc>] (generic_handle_irq+0x2c/0x3c)

[16696455.839883] r9:cf9520c0 r8:ef008000 r7:00000001 r6:cf907d00 r5:00000000 r4:c089a984

[16696455.847901] [<c006f6b0>] (generic_handle_irq) from [<c006f948>] (__handle_domain_irq+0x5c/0xb0)

[16696455.856842] [<c006f8ec>] (__handle_domain_irq) from [<c00094d8>] (gic_handle_irq+0x40/0x6c)

[16696455.865431] r9:cf9520c0 r8:fa241100 r7:fa240100 r6:cf907be8 r5:fa24010c r4:c08724a8

[16696455.873448] [<c0009498>] (gic_handle_irq) from [<c0014114>] (__irq_svc+0x54/0x90)

[16696455.881173] Exception stack(0xcf907be8 to 0xcf907c30)

[16696455.886474] 7be0: c08c2b40 00404404 c08c2b00 00000000 00000202 00000000

[16696455.894896] 7c00: 00000000 00000001 ef008000 cf9520c0 00000000 cf907c94 cf907c98 cf907c38

[16696455.903316] 7c20: c003a108 c0039c54 60070113 ffffffff

[16696455.908612] r9:cf9520c0 r8:ef008000 r7:cf907c1c r6:ffffffff r5:60070113 r4:c0039c54

[16696455.916631] [<c0039bb4>] (__do_softirq) from [<c003a108>] (irq_exit+0xec/0x120)

[16696455.924181] r10:00000000 r9:cf9520c0 r8:ef008000 r7:00000001 r6:00000000 r5:00000000

[16696455.932280] r4:c089a984

[16696455.935072] [<c003a01c>] (irq_exit) from [<c006f94c>] (__handle_domain_irq+0x60/0xb0)

[16696455.943142] r5:00000000 r4:c089a984

[16696455.946980] [<c006f8ec>] (__handle_domain_irq) from [<c00094d8>] (gic_handle_irq+0x40/0x6c)

[16696455.955569] r9:cf9520c0 r8:fa241100 r7:fa240100 r6:cf907d00 r5:fa24010c r4:c08724a8

[16696455.963586] [<c0009498>] (gic_handle_irq) from [<c0014114>] (__irq_svc+0x54/0x90)

[16696455.971310] Exception stack(0xcf907d00 to 0xcf907d48)

[16696455.976613] 7d00: c08f69bc 00000000 00000730 00000000 00000004 00000000 c08f69bc 00000001

[16696455.985035] 7d20: 00000004 cf9520c0 00000000 cf907d5c cf907d50 cf907d50 c001b77c c001b788

[16696455.993453] 7d40: a0070013 ffffffff

[16696455.997188] r9:cf9520c0 r8:00000004 r7:cf907d34 r6:ffffffff r5:a0070013 r4:c001b788

[16696456.005209] [<c001b758>] (arm_heavy_mb) from [<c03a8e48>] (mem_serial_out+0x24/0x34)

[16696456.013199] [<c03a8e24>] (mem_serial_out) from [<c03b00bc>] (omap_8250_shutdown+0x74/0xf4)

[16696456.021702] r7:00000001 r6:cf8b0200 r5:c08f69bc r4:c08f69bc

[16696456.027634] [<c03b0048>] (omap_8250_shutdown) from [<c03a9c74>] (serial8250_shutdown+0x1c/0x28)

[16696456.036570] r5:c08f69bc r4:ef2090b8

[16696456.040411] [<c03a9c58>] (serial8250_shutdown) from [<c03a4274>] (uart_shutdown+0x120/0x174)

[16696456.049095] [<c03a4154>] (uart_shutdown) from [<c03a4e80>] (uart_hangup+0x4c/0x13c)

[16696456.056992] r5:ef209130 r4:ef2090b8

[16696456.060834] [<c03a4e34>] (uart_hangup) from [<c0386c38>] (__tty_hangup+0x2e8/0x508)

[16696456.068730] r7:00000001 r6:cf906000 r5:cf8b0200 r4:ffffffff

[16696456.074664] [<c0386950>] (__tty_hangup) from [<c03874d4>] (disassociate_ctty.part.8+0x3c/0x2f4)

[16696456.083600] r10:cf907ec8 r9:cf89cfc0 r8:cf4daa00 r7:cf8ee000 r6:cf8ee034 r5:cf8b0200

[16696456.091699] r4:00000001

[16696456.094493] [<c0387498>] (disassociate_ctty.part.8) from [<c03877e8>] (disassociate_ctty+0x30/0x34)

[16696456.103776] r7:cf8ee000 r6:cf8ee034 r5:cf8ee000 r4:cf438000

[16696456.109708] [<c03877b8>] (disassociate_ctty) from [<c0038fe8>] (do_exit+0x6dc/0x9e8)

[16696456.117697] [<c003890c>] (do_exit) from [<c0039374>] (do_group_exit+0x44/0xf8)

[16696456.125160] r7:cf907edc

[16696456.127952] [<c0039330>] (do_group_exit) from [<c00437f8>] (get_signal+0x208/0x5fc)

[16696456.135848] r5:00000009 r4:cf906000

[16696456.139690] [<c00435f0>] (get_signal) from [<c0012b9c>] (do_signal+0xd0/0x42c)

[16696456.147153] r10:00000000 r9:cf906000 r8:b6dadabc r7:cf907ec8 r6:00000000 r5:b6dadab8

[16696456.155252] r4:cf907fb0

[16696456.158047] [<c0012acc>] (do_signal) from [<c00130c0>] (do_work_pending+0xa8/0xc0)

[16696456.165857] r10:00000000 r9:cf906000 r8:c000fb44 r7:00000025 r6:cf907fb0 r5:c000fb44

[16696456.173956] r4:cf906010

[16696456.176750] [<c0013018>] (do_work_pending) from [<c000f9d4>] (slow_work_pending+0xc/0x20)

[16696456.185166] r7:00000025 r6:000ed290 r5:000001d7 r4:00000000

over 5 years ago

0 Biser Gatchev-XID over 5 years ago

TI__Guru**** 393215 points

Please provide complete information of your use case.

0 Bin Liu over 5 years ago

TI__Guru**** 170191 points

Hi Yang,

Which uart ports are enabled in your system? How are they used?

Have you tried the v4.19 kernel in the latest Processor SDK release? The TI v4.4.y kernel is no longer maintained.

Have you tried to replicate the issue on TI AM437x EVM?

0 Yang Hsiung over 5 years ago in reply to Bin Liu

Prodigy 30 points

UART 0 and 1 are actively used

There are three RS232 UARTS for interfacing to external devices.

• UART 0 – Intended for communication with Management Switches. Utilizes software flow control and does not require hardware flow control signals.

• UART 1 – Intended for communication with the Data Center Digi device. Utilizes full hardware flow control (RTS, CTS, DSR, DTR).

• UART 2 – Intended for debug and future expansion. Utilizes software flow control and does not require hardware flow control signals.

We're only using v4.4.57.

We don't know how to replicate the issue.

0 Bin Liu over 5 years ago in reply to Yang Hsiung

TI__Guru**** 170191 points

Hi Yang,

Did you enable EDMA on these 3 uart?

Please test with the v4.19 kernel in the Processor SDK v6.1 to see if the problem still happens. There are some serial/uart driver updates in the newer kernels.

0 Yang Hsiung over 5 years ago in reply to Bin Liu

Prodigy 30 points

Hi Bin,

What is EDMA?

How to figure out it is enabled or not?

How to enable it (by kernel driver or by application)? Please give an example.

Thanks.

0 Bin Liu over 5 years ago in reply to Yang Hsiung

TI__Guru**** 170191 points

Hi Yang,

EDMA is a DMA controller on AM437x/AM438x which provides DMA services for any DMA-capable peripherals.

Enabling EDMA for UART RX/TX is done in your board device tree file. So please check your board DTS, when you setting UART0-2 pinmux and enabling UART0-2 if you add property "dmas" and "dma-names" in those UART nodes. AM437x by default doesn't use EDMA for all the UART ports, but here is an example in AM335x which UART has EDMA enabled:

arch/arm/boot/dts/am33xx.dtsi:

 356                 uart0: serial@44e09000 {  
 357                         compatible = "ti,am3352-uart", "ti,omap3-uart";  
 358                         ti,hwmods = "uart1";  
 359                         clock-frequency = <48000000>;  
 360                         reg = <0x44e09000 0x2000>;  
 361                         interrupts = <72>;  
 362                         status = "disabled";  
 363                         dmas = <&edma 26 0>, <&edma 27 0>;  
 364                         dma-names = "tx", "rx"; 
 365                 }; 
 366

If you didn't enable DMA on your UART0-2 ports, just leave it as is so that we can ignore this factor.

Processors

Processors forum

PROCESSOR-SDK-AM437X: CPU stall on closing UART port