Part Number: AM3358
We have a C++ app running on an AM3358, with Linux kernel 4.14.79-rt47.
The app handles serial communications with various external devices, using FTDI USB-to-serial converters (FT4232H).
Depending on certain conditions, the app may close and re-open one of the com ports (e.g. /dev/ttyUSB0) multiple times, over and over, every few seconds, in an attempt to re-establish comms with the external device.
After some time (typically about 6 hours, but has also happened within minutes), all /dev/ttyUSBx com ports will freeze up, and will log the backtrace as shown below.
We have applied all patches and settings changes that we can find, including from the following forums:
Linux: AM335x Quad-USB-to-UART Bridge (CP2108) Communcation Freeze
Linux/PROCESSOR-SDK-AM335X: Kernel issue with USB WiFi dongle
Linux/OMAP-L138: Long delay in cppi41_dma_channel_abort causing application level errors
Disabling the cppi41 autosuspend, as suggested in one of the forum threads, seemed to prolong the time until the lockup, but didn't ultimately solve the problem.
May 19 06:51:09 ABB user.warn kernel: [64462.358211] ------------[ cut here ]------------
May 19 06:51:09 ABB user.warn kernel: [64462.362916] WARNING: CPU: 0 PID: 1431 at drivers/dma/ti/cppi41.c:724 cppi41_stop_chan+0x2f0/0x3d8
May 19 06:51:09 ABB user.warn kernel: [64462.371828] Modules linked in: ti_prueth pru_rproc pm33xx pruss pruss_intc wkup_m3_ipc wkup_m3_rproc remoteproc pruss_soc_bus gpio_pca953x dp83822 [last unloaded: wlcore]
May 19 06:51:09 ABB user.warn kernel: [64462.387155] CPU: 0 PID: 1431 Comm: P: Not tainted 4.14.79-rt47 #1
May 19 06:51:09 ABB user.warn kernel: [64462.397635] Hardware name: Generic AM33XX (Flattened Device Tree)
May 19 06:51:09 ABB user.warn kernel: [64462.403755] Backtrace:
May 19 06:51:09 ABB user.warn kernel: [64462.406241] [<c010b420>] (dump_backtrace) from [<c010b704>] (show_stack+0x18/0x1c)
May 19 06:51:09 ABB user.warn kernel: [64462.413848] r7:00000009 r6:00000000 r5:c0969fd8 r4:00000000
May 19 06:51:09 ABB user.warn kernel: [64462.419546] [<c010b6ec>] (show_stack) from [<c077e3e0>] (dump_stack+0x24/0x28)
May 19 06:51:09 ABB user.warn kernel: [64462.426818] [<c077e3bc>] (dump_stack) from [<c012bfb8>] (__warn+0xe8/0x100)
May 19 06:51:09 ABB user.warn kernel: [64462.433817] [<c012bed0>] (__warn) from [<c012c088>] (warn_slowpath_null+0x28/0x30)
May 19 06:51:09 ABB user.warn kernel: [64462.441427] r9:dd041000 r8:0000003c r7:00000013 r6:d96c38a8 r5:d96c3810 r4:d96ad6a0
May 19 06:51:09 ABB user.warn kernel: [64462.449214] [<c012c060>] (warn_slowpath_null) from [<c03daa2c>] (cppi41_stop_chan+0x2f0/0x3d8)
May 19 06:51:09 ABB user.warn kernel: [64462.457881] [<c03da73c>] (cppi41_stop_chan) from [<c04e45d4>] (cppi41_dma_channel_abort+0x100/0x2b8)
May 19 06:51:09 ABB user.warn kernel: [64462.467060] r9:dc23e010 r8:d9779b00 r7:c0b87b18 r6:00000000 r5:00000020 r4:d97d8120
May 19 06:51:09 ABB user.warn kernel: [64462.474846] [<c04e44d4>] (cppi41_dma_channel_abort) from [<c04dcf2c>] (musb_cleanup_urb+0x60/0x21c)
May 19 06:51:09 ABB user.warn kernel: [64462.483938] r10:cf855de0 r9:dc23e010 r8:d97d8120 r7:00000000 r6:00000080 r5:cfb08080
May 19 06:51:09 ABB user.warn kernel: [64462.491801] r4:dc23e790
May 19 06:51:09 ABB user.warn kernel: [64462.494350] [<c04dcecc>] (musb_cleanup_urb) from [<c04dd738>] (musb_urb_dequeue+0xd4/0x124)
May 19 06:51:09 ABB user.warn kernel: [64462.502745] r10:cf855de0 r9:00000001 r8:c0438380 r7:00000000 r6:cfad2d00 r5:dc23e010
May 19 06:51:09 ABB user.warn kernel: [64462.510610] r4:cfb08080
May 19 06:51:09 ABB user.warn kernel: [64462.513170] [<c04dd664>] (musb_urb_dequeue) from [<c04b69a0>] (unlink1+0x34/0x108)
May 19 06:51:09 ABB user.warn kernel: [64462.520779] r9:00000003 r8:00000000 r7:fffffffe r6:d977c800 r5:cf809c00 r4:cfb08080
May 19 06:51:09 ABB user.warn kernel: [64462.528566] [<c04b696c>] (unlink1) from [<c04b8b94>] (usb_hcd_unlink_urb+0x64/0x80)
May 19 06:51:09 ABB user.warn kernel: [64462.536260] r7:d8871700 r6:fffffffe r5:cf809c00 r4:cfb08080
May 19 06:51:09 ABB user.warn kernel: [64462.541954] [<c04b8b30>] (usb_hcd_unlink_urb) from [<c04b9d18>] (usb_kill_urb.part.4+0x38/0xb0)
May 19 06:51:09 ABB user.warn kernel: [64462.550793] r7:d8871700 r6:cfa7f400 r5:cfb0808c r4:cfb08080
May 19 06:51:09 ABB user.warn kernel: [64462.556504] [<c04b9ce0>] (usb_kill_urb.part.4) from [<c04b9dc0>] (usb_kill_urb+0x30/0x34)
May 19 06:51:09 ABB user.warn kernel: [64462.564726] r6:cfa7f400 r5:cfb0c100 r4:cfb0c000
May 19 06:51:09 ABB user.warn kernel: [64462.569390] [<c04b9d90>] (usb_kill_urb) from [<c04d390c>] (usb_serial_generic_close+0x5c/0x68)
May 19 06:51:09 ABB user.warn kernel: [64462.578054] [<c04d38b0>] (usb_serial_generic_close) from [<c04d1308>] (serial_port_shutdown+0x28/0x2c)
May 19 06:51:09 ABB user.warn kernel: [64462.587404] r5:cfb0c0c8 r4:cfb0c004
May 19 06:51:09 ABB user.warn kernel: [64462.591010] [<c04d12e0>] (serial_port_shutdown) from [<c03f96b4>] (tty_port_shutdown+0x94/0xa0)
May 19 06:51:09 ABB user.warn kernel: [64462.599757] [<c03f9620>] (tty_port_shutdown) from [<c03fa394>] (tty_port_close+0x44/0x78)
May 19 06:51:09 ABB user.warn kernel: [64462.607974] r7:d8871700 r6:00000000 r5:cfb0c004 r4:cfa7f400
May 19 06:51:09 ABB user.warn kernel: [64462.613665] [<c03fa350>] (tty_port_close) from [<c04d1a18>] (serial_close+0x20/0x24)
May 19 06:51:09 ABB user.warn kernel: [64462.621446] r7:d8871700 r6:d9622b10 r5:00000000 r4:cfa7f400
May 19 06:51:09 ABB user.warn kernel: [64462.627139] [<c04d19f8>] (serial_close) from [<c03f0678>] (tty_release+0xf8/0x460)
May 19 06:51:09 ABB user.warn kernel: [64462.634752] [<c03f0580>] (tty_release) from [<c0223758>] (__fput+0x8c/0x234)
May 19 06:51:09 ABB user.warn kernel: [64462.641837] r10:cf855de0 r9:00000008 r8:00000000 r7:cf855de0 r6:d9622b10 r5:dba2a960
May 19 06:51:09 ABB user.warn kernel: [64462.649701] r4:d8871700
May 19 06:51:09 ABB user.warn kernel: [64462.652251] [<c02236cc>] (__fput) from [<c0223958>] (____fput+0x10/0x14)
May 19 06:51:09 ABB user.warn kernel: [64462.658987] r10:00000006 r9:d8871700 r8:c0b47d3c r7:ffffe000 r6:cf928e80 r5:cf92938c
May 19 06:51:09 ABB user.warn kernel: [64462.666853] r4:00000000
May 19 06:51:09 ABB user.warn kernel: [64462.669412] [<c0223948>] (____fput) from [<c0146c10>] (task_work_run+0xc8/0x100)
May 19 06:51:09 ABB user.warn kernel: [64462.676855] [<c0146b48>] (task_work_run) from [<c010af54>] (do_work_pending+0xc4/0xc8)
May 19 06:51:09 ABB user.warn kernel: [64462.684815] r9:cfcde000 r8:c0107b84 r7:00000006 r6:cfcdffb0 r5:c0107b84 r4:ffffe000
May 19 06:51:09 ABB user.warn kernel: [64462.692601] [<c010ae90>] (do_work_pending) from [<c01079f4>] (slow_work_pending+0xc/0x20)
May 19 06:51:09 ABB user.warn kernel: [64462.700819] r7:00000006 r6:00000000 r5:00000050 r4:003def68
May 19 06:51:09 ABB user.warn kernel: [64462.706504] ---[ end trace 0000000000000002 ]---
May 19 06:51:09 ABB user.emerg kernel: [64462.711160] Internal error: Oops - undefined instruction: 0 [#1] PREEMPT ARM
May 19 06:51:09 ABB user.warn kernel: [64462.711164] Modules linked in: ti_prueth pru_rproc pm33xx pruss pruss_intc wkup_m3_ipc wkup_m3_rproc remoteproc pruss_soc_bus gpio_pca953x dp83822 [last unloaded: wlcore]
May 19 06:51:09 ABB user.warn kernel: [64462.711201] CPU: 0 PID: 1431 Comm: P: Tainted: G W 4.14.79-rt47 #1
May 19 06:51:09 ABB user.warn kernel: [64462.711204] Hardware name: Generic AM33XX (Flattened Device Tree)
May 19 06:51:09 ABB user.warn kernel: [64462.711209] task: cf928e80 task.stack: cfcde000
May 19 06:51:09 ABB user.warn kernel: [64462.711221] PC is at cppi41_stop_chan+0x35c/0x3d8
May 19 06:51:09 ABB user.warn kernel: [64462.711230] LR is at add_taint+0x28/0x64
May 19 06:51:09 ABB user.warn kernel: [64462.711235] pc : [<c03daa98>] lr : [<c012bb68>] psr: 600c0013
May 19 06:51:09 ABB user.warn kernel: [64462.711239] sp : cfcdfd00 ip : 00000000 fp : cfcdfd2c
May 19 06:51:09 ABB user.warn kernel: [64462.711243] r10: e0a44c10 r9 : dd041000 r8 : 0000003c
May 19 06:51:09 ABB user.warn kernel: [64462.711248] r7 : 00000013 r6 : d96c38a8 r5 : d96c3810 r4 : d96ad6a0
May 19 06:51:09 ABB user.warn kernel: [64462.711252] r3 : 00000000 r2 : 00000000 r1 : 00000003 r0 : 00000000
May 19 06:51:09 ABB user.warn kernel: [64462.711260] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
May 19 06:51:09 ABB user.warn kernel: [64462.711265] Control: 10c5387d Table: 98890019 DAC: 00000051
May 19 06:51:09 ABB user.emerg kernel: [64462.711270] Process P: (pid: 1431, stack limit = 0xcfcde208)
May 19 06:51:09 ABB user.emerg kernel: [64462.711275] Stack: (0xcfcdfd00 to 0xcfce0000)
May 19 06:51:09 ABB user.emerg kernel: [64462.711288] fd00: 00000003 dc23e010 d97d8120 00000020 00000000 c0b87b18 d9779b00 dc23e010
May 19 06:51:09 ABB user.emerg kernel: [64462.711299] fd20: cfcdfd6c cfcdfd30 c04e45d4 c03da748 cfcdfd2c c0b87afc c0b87b04 e0a44c10
May 19 06:51:09 ABB user.emerg kernel: [64462.711310] fd40: cf809c00 dc23e790 cfb08080 00000080 00000000 d97d8120 dc23e010 cf855de0
May 19 06:51:09 ABB user.emerg kernel: [64462.711321] fd60: cfcdfdac cfcdfd70 c04dcf2c c04e44e0 cf809c00 cfb08300 00000bb8 cfb0c000
May 19 06:51:09 ABB user.emerg kernel: [64462.711332] fd80: ffffe000 cfb08080 dc23e010 cfad2d00 00000000 c0438380 00000001 cf855de0
May 19 06:51:09 ABB user.emerg kernel: [64462.711343] fda0: cfcdfdd4 cfcdfdb0 c04dd738 c04dced8 cfb08080 cf809c00 d977c800 fffffffe
May 19 06:51:09 ABB user.emerg kernel: [64462.711354] fdc0: 00000000 00000003 cfcdfdf4 cfcdfdd8 c04b69a0 c04dd670 cfb08080 cf809c00
May 19 06:51:09 ABB user.emerg kernel: [64462.711365] fde0: fffffffe d8871700 cfcdfe14 cfcdfdf8 c04b8b94 c04b6978 cfb08080 cfb0808c
May 19 06:51:09 ABB user.emerg kernel: [64462.711376] fe00: cfa7f400 d8871700 cfcdfe4c cfcdfe18 c04b9d18 c04b8b3c cfa7f400 cf855de0
May 19 06:51:09 ABB user.emerg kernel: [64462.711387] fe20: cfcdfe3c cfcdfe30 c0796020 c0794500 ffffe000 cfb0c000 cfb0c100 cfa7f400
May 19 06:51:09 ABB user.emerg kernel: [64462.711397] fe40: cfcdfe5c cfcdfe50 c04b9dc0 c04b9cec cfcdfe74 cfcdfe60 c04d390c c04b9d9c
May 19 06:51:09 ABB user.emerg kernel: [64462.711408] fe60: cfb0c004 cfb0c0c8 cfcdfe84 cfcdfe78 c04d1308 c04d38bc cfcdfea4 cfcdfe88
May 19 06:51:09 ABB user.emerg kernel: [64462.711419] fe80: c03f96b4 c04d12ec cfa7f400 cfb0c004 00000000 d8871700 cfcdfec4 cfcdfea8
May 19 06:51:09 ABB user.emerg kernel: [64462.711430] fea0: c03fa394 c03f962c cfa7f400 00000000 d9622b10 d8871700 cfcdfed4 cfcdfec8
May 19 06:51:09 ABB user.emerg kernel: [64462.711441] fec0: c04d1a18 c03fa35c cfcdff14 cfcdfed8 c03f0678 c04d1a04 00000000 00000001
May 19 06:51:09 ABB user.emerg kernel: [64462.711451] fee0: 0000000e 00000000 00000002 d8871700 dba2a960 d9622b10 cf855de0 00000000
May 19 06:51:09 ABB user.emerg kernel: [64462.711462] ff00: 00000008 cf855de0 cfcdff54 cfcdff18 c0223758 c03f058c 00000000 00000000
May 19 06:51:09 ABB user.emerg kernel: [64462.711473] ff20: 00000000 d8871708 d8871700 00000000 cf92938c cf928e80 ffffe000 c0b47d3c
May 19 06:51:09 ABB user.emerg kernel: [64462.711484] ff40: d8871700 00000006 cfcdff64 cfcdff58 c0223958 c02236d8 cfcdff8c cfcdff68
May 19 06:51:09 ABB user.emerg kernel: [64462.711494] ff60: c0146c10 c0223954 ffffe000 c0107b84 cfcdffb0 00000006 c0107b84 cfcde000
May 19 06:51:09 ABB user.emerg kernel: [64462.711505] ff80: cfcdffac cfcdff90 c010af54 c0146b54 003def68 00000050 00000000 00000006
May 19 06:51:09 ABB user.emerg kernel: [64462.711516] ffa0: 00000000 cfcdffb0 c01079f4 c010ae9c 00000000 00000000 00000002 ac6fe4a4
May 19 06:51:09 ABB user.emerg kernel: [64462.711526] ffc0: 003def68 00000050 00000000 00000006 001dda68 00000001 00000000 00000000
May 19 06:51:09 ABB user.emerg kernel: [64462.711537] ffe0: 00000000 ac6fdcd0 b6fa1a4b b6fa1a56 800f0030 00000050 00000000 00000000
May 19 06:51:09 ABB user.warn kernel: [64462.711539] Backtrace:
May 19 06:51:09 ABB user.warn kernel: [64462.711556] [<c03da73c>] (cppi41_stop_chan) from [<c04e45d4>] (cppi41_dma_channel_abort+0x100/0x2b8)
May 19 06:51:09 ABB user.warn kernel: [64462.711567] r9:dc23e010 r8:d9779b00 r7:c0b87b18 r6:00000000 r5:00000020 r4:d97d8120
May 19 06:51:09 ABB user.warn kernel: [64462.711578] [<c04e44d4>] (cppi41_dma_channel_abort) from [<c04dcf2c>] (musb_cleanup_urb+0x60/0x21c)
May 19 06:51:09 ABB user.warn kernel: [64462.711588] r10:cf855de0 r9:dc23e010 r8:d97d8120 r7:00000000 r6:00000080 r5:cfb08080
May 19 06:51:09 ABB user.warn kernel: [64462.711592] r4:dc23e790
May 19 06:51:09 ABB user.warn kernel: [64462.711602] [<c04dcecc>] (musb_cleanup_urb) from [<c04dd738>] (musb_urb_dequeue+0xd4/0x124)
May 19 06:51:09 ABB user.warn kernel: [64462.711612] r10:cf855de0 r9:00000001 r8:c0438380 r7:00000000 r6:cfad2d00 r5:dc23e010
May 19 06:51:09 ABB user.warn kernel: [64462.711615] r4:cfb08080
May 19 06:51:09 ABB user.warn kernel: [64462.711628] [<c04dd664>] (musb_urb_dequeue) from [<c04b69a0>] (unlink1+0x34/0x108)
May 19 06:51:09 ABB user.warn kernel: [64462.711637] r9:00000003 r8:00000000 r7:fffffffe r6:d977c800 r5:cf809c00 r4:cfb08080
May 19 06:51:09 ABB user.warn kernel: [64462.711649] [<c04b696c>] (unlink1) from [<c04b8b94>] (usb_hcd_unlink_urb+0x64/0x80)
May 19 06:51:09 ABB user.warn kernel: [64462.711656] r7:d8871700 r6:fffffffe r5:cf809c00 r4:cfb08080
May 19 06:51:09 ABB user.warn kernel: [64462.711668] [<c04b8b30>] (usb_hcd_unlink_urb) from [<c04b9d18>] (usb_kill_urb.part.4+0x38/0xb0)
May 19 06:51:09 ABB user.warn kernel: [64462.711676] r7:d8871700 r6:cfa7f400 r5:cfb0808c r4:cfb08080
May 19 06:51:09 ABB user.warn kernel: [64462.711688] [<c04b9ce0>] (usb_kill_urb.part.4) from [<c04b9dc0>] (usb_kill_urb+0x30/0x34)
May 19 06:51:09 ABB user.warn kernel: [64462.711694] r6:cfa7f400 r5:cfb0c100 r4:cfb0c000
May 19 06:51:09 ABB user.warn kernel: [64462.711708] [<c04b9d90>] (usb_kill_urb) from [<c04d390c>] (usb_serial_generic_close+0x5c/0x68)
May 19 06:51:09 ABB user.warn kernel: [64462.711720] [<c04d38b0>] (usb_serial_generic_close) from [<c04d1308>] (serial_port_shutdown+0x28/0x2c)
May 19 06:51:09 ABB user.warn kernel: [64462.711725] r5:cfb0c0c8 r4:cfb0c004
May 19 06:51:09 ABB user.warn kernel: [64462.711737] [<c04d12e0>] (serial_port_shutdown) from [<c03f96b4>] (tty_port_shutdown+0x94/0xa0)
May 19 06:51:09 ABB user.warn kernel: [64462.711749] [<c03f9620>] (tty_port_shutdown) from [<c03fa394>] (tty_port_close+0x44/0x78)
May 19 06:51:09 ABB user.warn kernel: [64462.711756] r7:d8871700 r6:00000000 r5:cfb0c004 r4:cfa7f400
May 19 06:51:09 ABB user.warn kernel: [64462.711767] [<c03fa350>] (tty_port_close) from [<c04d1a18>] (serial_close+0x20/0x24)
May 19 06:51:09 ABB user.warn kernel: [64462.711775] r7:d8871700 r6:d9622b10 r5:00000000 r4:cfa7f400
May 19 06:51:09 ABB user.warn kernel: [64462.711784] [<c04d19f8>] (serial_close) from [<c03f0678>] (tty_release+0xf8/0x460)
May 19 06:51:09 ABB user.warn kernel: [64462.711795] [<c03f0580>] (tty_release) from [<c0223758>] (__fput+0x8c/0x234)
May 19 06:51:09 ABB user.warn kernel: [64462.711805] r10:cf855de0 r9:00000008 r8:00000000 r7:cf855de0 r6:d9622b10 r5:dba2a960
May 19 06:51:09 ABB user.warn kernel: [64462.711808] r4:d8871700
May 19 06:51:09 ABB user.warn kernel: [64462.711816] [<c02236cc>] (__fput) from [<c0223958>] (____fput+0x10/0x14)
May 19 06:51:09 ABB user.warn kernel: [64462.711826] r10:00000006 r9:d8871700 r8:c0b47d3c r7:ffffe000 r6:cf928e80 r5:cf92938c
May 19 06:51:09 ABB user.warn kernel: [64462.711830] r4:00000000
May 19 06:51:09 ABB user.warn kernel: [64462.711841] [<c0223948>] (____fput) from [<c0146c10>] (task_work_run+0xc8/0x100)
May 19 06:51:09 ABB user.warn kernel: [64462.711854] [<c0146b48>] (task_work_run) from [<c010af54>] (do_work_pending+0xc4/0xc8)
May 19 06:51:09 ABB user.warn kernel: [64462.711864] r9:cfcde000 r8:c0107b84 r7:00000006 r6:cfcdffb0 r5:c0107b84 r4:ffffe000
May 19 06:51:09 ABB user.warn kernel: [64462.711875] [<c010ae90>] (do_work_pending) from [<c01079f4>] (slow_work_pending+0xc/0x20)
May 19 06:51:09 ABB user.warn kernel: [64462.711882] r7:00000006 r6:00000000 r5:00000050 r4:003def68
May 19 06:51:09 ABB user.emerg kernel: [64462.711895] Code: e5d4308c e3833002 e5c4308c eaffff50 (e7f001f2)
May 19 06:51:09 ABB user.warn kernel: [64463.272649] ---[ end trace 0000000000000003 ]---
