Linux/AM3352: RCU self-detected stall sometimes happens.

user5132856

Part Number: AM3352

Tool/software: Linux

Hi,

We have developed a custom board with AM3352 Processor. Sometimes it causes RCU self-detected stall while communicating with Windows PC board via TCP/IP.
Any helps or advises are appreciated. Thank you in advance.

[Environment]
AM3352 SDK version 3.0.0.4, Linux Kernel 4.4.12 (with edma backported from kernel 4.13.4 to avoid a bug)
Almost all configurations are default settings. The only difference is removal of unnecessary drivers.

[Problem]
TCP/IP connection is infrequently disconnected and terminal of our Linux board shows the following message. And the terminal stops. No response to any keyboard input.

> INFO: rcu_preempt self-detected stall on cpu 0

The problem has been happened several times, and thread name shown in terminal with above message is different every time. Once it was a SWAPPER.

Sometimes the problem recovers automatically in about 3 minutes, and sometimes not.
"Recover" means that every function start working again, including TCP/IP communication.

It seems that scheduler of Linux Kernel went wrong when the problem did not recover. Some active threads seem not be able to get any CPU time.
Login to the board via telnet or ssh is successful while the problem is repdoduced. However, CPU load of any thread or process is 0% according to "top" command.

See the following log to know more details.

[Log 1 - terminal]

_____ _____ _ _
| _ |___ ___ ___ ___ | _ |___ ___ |_|___ ___| |_
| | _| .'| . | . | | __| _| . | | | -_| _| _|
|__|__|_| |__,|_ |___| |__| |_| |___|_| |___|___|_|
|___| |___|

Arago Project http://arago-project.org am335x-evm ttyS0

Arago 2016.05 am335x-evm ttyS0

am335x-evm login: root
root@am335x-evm:~#

root@am335x-evm:~#
root@am335x-evm:~#
root@am335x-evm:~#
root@am335x-evm:~#
root@am335x-evm:~# [291983.288232] INFO: rcu_preempt self-detected stall on CPU
[291983.293710] 0-...: (8411371 ticks this GP) idle=fb7/140000000000001/0 softirq=30168883/30168883 fqs=0
[291983.303234] (t=2100 jiffies g=20049383 c=20049382 q=3)
[291983.308663] rcu_preempt kthread starved for 2100 jiffies! g20049383 c20049382 f0x0 s3 ->state=0x1
[291983.317658] Task dump for CPU 0:
[291983.320982] DBGMON R running 0 833 1 0x00000002
[291983.327462] Backtrace:
[291983.330047] [<c00134e0>] (dump_backtrace) from [<c00136dc>] (show_stack+0x18/0x1c)
[291983.337734] r7:c094c14c r6:80070193 r5:00000001 r4:eef9ba80
[291983.343537] [<c00136c4>] (show_stack) from [<c0061d4c>] (sched_show_task+0xbc/0x110)
[291983.351403] [<c0061c90>] (sched_show_task) from [<c0062d14>] (dump_cpu_task+0x2c/0x30)
[291983.359436] r5:c095c568 r4:00000001
[291983.363137] [<c0062ce8>] (dump_cpu_task) from [<c007a778>] (rcu_dump_cpu_stacks+0xa8/0x100)
[291983.371617] [<c007a6d0>] (rcu_dump_cpu_stacks) from [<c007e184>] (rcu_check_callbacks+0x484/0x8d4)
[291983.380698] r9:c094c144 r8:00000000 r7:c095c994 r6:c095c568 r5:c095c568 r4:c095c9ec
[291983.388595] [<c007dd00>] (rcu_check_callbacks) from [<c0081560>] (update_process_times+0x44/0x6c)
[291983.397588] r10:c095d54c r9:c0090820 r8:00000000 r7:00010a06 r6:00000001 r5:eef9ba80
[291983.405562] r4:ffffe000
[291983.408207] [<c008151c>] (update_process_times) from [<c009081c>] (tick_sched_handle+0x50/0x54)
[291983.417025] r7:00010a06 r6:a7eb2f90 r5:ededbfb0 r4:c095dc40
[291983.422820] [<c00907cc>] (tick_sched_handle) from [<c0090884>] (tick_sched_timer+0x64/0xb8)
[291983.431299] [<c0090820>] (tick_sched_timer) from [<c0082288>] (__hrtimer_run_queues.constprop.6+0x150/0x1e0)
[291983.441250] r7:c095d540 r6:ededa000 r5:c095dc40 r4:c095d500
[291983.447044] [<c0082138>] (__hrtimer_run_queues.constprop.6) from [<c0082548>] (hrtimer_interrupt+0xb4/0x250)
[291983.456996] r10:c095d5b8 r9:ededbe80 r8:ffffffff r7:7fffffff r6:00000003 r5:ededa000
[291983.464972] r4:c095d500
[291983.467611] [<c0082494>] (hrtimer_interrupt) from [<c002d854>] (omap2_gp_timer_interrupt+0x30/0x38)
[291983.476779] r10:c0999fd1 r9:ee804c00 r8:00000010 r7:00000000 r6:00000000 r5:ee804c10
[291983.484754] r4:c09552b4
[291983.487403] [<c002d824>] (omap2_gp_timer_interrupt) from [<c00723b4>] (handle_irq_event_percpu+0xb4/0x16c)
[291983.497187] [<c0072300>] (handle_irq_event_percpu) from [<c00724d0>] (handle_irq_event+0x64/0x90)
[291983.506180] r10:00000001 r9:b12fef90 r8:ee806000 r7:00000000 r6:00000000 r5:ee804c10
[291983.514154] r4:ee804c00
[291983.516791] [<c007246c>] (handle_irq_event) from [<c00752ac>] (handle_level_irq+0xb8/0x168)
[291983.525261] r5:ee804c10 r4:ee804c00
[291983.528950] [<c00751f4>] (handle_level_irq) from [<c0071a7c>] (generic_handle_irq+0x2c/0x3c)
[291983.537506] r5:00000010 r4:c0967f20
[291983.541196] [<c0071a50>] (generic_handle_irq) from [<c0071ce8>] (__handle_domain_irq+0x5c/0xb0)
[291983.550021] [<c0071c8c>] (__handle_domain_irq) from [<c0009468>] (omap_intc_handle_irq+0x3c/0x98)
[291983.559012] r9:b12fef90 r8:10c5387d r7:10c5387d r6:ffffffff r5:60070030 r4:c09cd100
[291983.566907] [<c000942c>] (omap_intc_handle_irq) from [<c00144f4>] (__irq_usr+0x54/0x80)
[291983.575029] Exception stack(0xededbfb0 to 0xededbff8)
[291983.580189] bfa0: 00000000 00000000 00000341 00000000
[291983.588490] bfc0: 012cb910 b12feacc f0000000 00000032 0082d2c4 b12fef90 00000001 b6015b60
[291983.596790] bfe0: 012aa6b8 b12fea58 005f4ef9 b613bf9a 60070030 ffffffff
[291983.603514] r5:60070030 r4:b613bf9a
[292046.318216] INFO: rcu_preempt self-detected stall on CPU
[292046.323648] 0-...: (33198455 ticks this GP) idle=fb7/140000000000001/0 softirq=30168883/30168883 fqs=0
[292046.333251] (t=8403 jiffies g=20049383 c=20049382 q=3)
[292046.338678] rcu_preempt kthread starved for 8403 jiffies! g20049383 c20049382 f0x0 s3 ->state=0x1
[292046.347672] Task dump for CPU 0:
[292046.350996] DBGMON R running 0 833 1 0x00000002
[292046.357473] Backtrace:
[292046.360027] [<c00134e0>] (dump_backtrace) from [<c00136dc>] (show_stack+0x18/0x1c)
[292046.367712] r7:c094c14c r6:80070193 r5:00000001 r4:eef9ba80
[292046.373508] [<c00136c4>] (show_stack) from [<c0061d4c>] (sched_show_task+0xbc/0x110)
[292046.381372] [<c0061c90>] (sched_show_task) from [<c0062d14>] (dump_cpu_task+0x2c/0x30)
[292046.389406] r5:c095c568 r4:00000001
[292046.393096] [<c0062ce8>] (dump_cpu_task) from [<c007a778>] (rcu_dump_cpu_stacks+0xa8/0x100)
[292046.401572] [<c007a6d0>] (rcu_dump_cpu_stacks) from [<c007e184>] (rcu_check_callbacks+0x484/0x8d4)
[292046.410653] r9:c094c144 r8:00000000 r7:c095c994 r6:c095c568 r5:c095c568 r4:c095c9ec
[292046.418548] [<c007dd00>] (rcu_check_callbacks) from [<c0081560>] (update_process_times+0x44/0x6c)
[292046.427541] r10:c095d54c r9:c0090820 r8:00000000 r7:00010a15 r6:00000001 r5:eef9ba80
[292046.435518] r4:ffffe000
[292046.438155] [<c008151c>] (update_process_times) from [<c009081c>] (tick_sched_handle+0x50/0x54)
[292046.446974] r7:00010a15 r6:54ccaa78 r5:ededbfb0 r4:c095dc40
[292046.452767] [<c00907cc>] (tick_sched_handle) from [<c0090884>] (tick_sched_timer+0x64/0xb8)
[292046.461243] [<c0090820>] (tick_sched_timer) from [<c0082288>] (__hrtimer_run_queues.constprop.6+0x150/0x1e0)
[292046.471196] r7:c095d540 r6:ededa000 r5:c095dc40 r4:c095d500
[292046.476990] [<c0082138>] (__hrtimer_run_queues.constprop.6) from [<c0082548>] (hrtimer_interrupt+0xb4/0x250)
[292046.486943] r10:c095d5b8 r9:ededbe80 r8:ffffffff r7:7fffffff r6:00000003 r5:ededa000
[292046.494917] r4:c095d500
[292046.497554] [<c0082494>] (hrtimer_interrupt) from [<c002d854>] (omap2_gp_timer_interrupt+0x30/0x38)
[292046.506721] r10:c0999fd1 r9:ee804c00 r8:00000010 r7:00000000 r6:00000000 r5:ee804c10
[292046.514695] r4:c09552b4
[292046.517332] [<c002d824>] (omap2_gp_timer_interrupt) from [<c00723b4>] (handle_irq_event_percpu+0xb4/0x16c)
[292046.527117] [<c0072300>] (handle_irq_event_percpu) from [<c00724d0>] (handle_irq_event+0x64/0x90)
[292046.536110] r10:00000001 r9:b12fef90 r8:ee806000 r7:00000000 r6:00000000 r5:ee804c10
[292046.544086] r4:ee804c00
[292046.546722] [<c007246c>] (handle_irq_event) from [<c00752ac>] (handle_level_irq+0xb8/0x168)
[292046.555192] r5:ee804c10 r4:ee804c00
[292046.558880] [<c00751f4>] (handle_level_irq) from [<c0071a7c>] (generic_handle_irq+0x2c/0x3c)
[292046.567436] r5:00000010 r4:c0967f20
[292046.571126] [<c0071a50>] (generic_handle_irq) from [<c0071ce8>] (__handle_domain_irq+0x5c/0xb0)
[292046.579949] [<c0071c8c>] (__handle_domain_irq) from [<c0009468>] (omap_intc_handle_irq+0x3c/0x98)
[292046.588942] r9:b12fef90 r8:10c5387d r7:10c5387d r6:ffffffff r5:60070030 r4:c09cd100
[292046.596835] [<c000942c>] (omap_intc_handle_irq) from [<c00144f4>] (__irq_usr+0x54/0x80)
[292046.604957] Exception stack(0xededbfb0 to 0xededbff8)
[292046.610115] bfa0: 00000000 00000000 00000341 00000000
[292046.618416] bfc0: 012cb910 b12feacc f0000000 00000032 0082d2c4 b12fef90 00000001 b6015b60
[292046.626715] bfe0: 012aa6b8 b12fea58 005f4ef9 b613bf9a 60070030 ffffffff
[292046.633440] r5:60070030 r4:b613bf9a
[292048.506706] INFO: rcu_preempt self-detected stall on CPU
[292048.512135] 0-...: (33940546 ticks this GP) idle=fb7/140000000000001/0 softirq=30168883/30168883 fqs=0
[292048.521737] (t=25799 jiffies g=20049383 c=20049382 q=3)
[292048.527251] rcu_preempt kthread starved for 25799 jiffies! g20049383 c20049382 f0x0 s3 ->state=0x1
[292048.536331] Task dump for CPU 0:
[292048.539655] DBGMON R running 0 833 1 0x00000002
[292048.546131] Backtrace:
[292048.548683] [<c00134e0>] (dump_backtrace) from [<c00136dc>] (show_stack+0x18/0x1c)
[292048.556369] r7:c094c14c r6:80070193 r5:00000001 r4:eef9ba80
[292048.562163] [<c00136c4>] (show_stack) from [<c0061d4c>] (sched_show_task+0xbc/0x110)
[292048.570028] [<c0061c90>] (sched_show_task) from [<c0062d14>] (dump_cpu_task+0x2c/0x30)
[292048.578062] r5:c095c568 r4:00000001
[292048.581751] [<c0062ce8>] (dump_cpu_task) from [<c007a778>] (rcu_dump_cpu_stacks+0xa8/0x100)
[292048.590228] [<c007a6d0>] (rcu_dump_cpu_stacks) from [<c007e184>] (rcu_check_callbacks+0x484/0x8d4)
[292048.599307] r9:c094c144 r8:00000000 r7:c095c994 r6:c095c568 r5:c095c568 r4:c095c9ec
[292048.607201] [<c007dd00>] (rcu_check_callbacks) from [<c0081560>] (update_process_times+0x44/0x6c)
[292048.616194] r10:c095d54c r9:c0090820 r8:00000000 r7:00010a3d r6:00000001 r5:eef9ba80
[292048.624171] r4:ffffe000
[292048.626807] [<c008151c>] (update_process_times) from [<c009081c>] (tick_sched_handle+0x50/0x54)
[292048.635626] r7:00010a3d r6:d5fe5858 r5:ededbfb0 r4:c095dc40
[292048.641419] [<c00907cc>] (tick_sched_handle) from [<c0090884>] (tick_sched_timer+0x64/0xb8)
[292048.649895] [<c0090820>] (tick_sched_timer) from [<c0082288>] (__hrtimer_run_queues.constprop.6+0x150/0x1e0)
[292048.659847] r7:c095d540 r6:ededa000 r5:c095dc40 r4:c095d500
[292048.665642] [<c0082138>] (__hrtimer_run_queues.constprop.6) from [<c0082548>] (hrtimer_interrupt+0xb4/0x250)
[292048.675595] r10:c095d5b8 r9:ededbe80 r8:ffffffff r7:7fffffff r6:00000003 r5:ededa000
[292048.683572] r4:c095d500
[292048.686208] [<c0082494>] (hrtimer_interrupt) from [<c002d854>] (omap2_gp_timer_interrupt+0x30/0x38)
[292048.695375] r10:c0999fd1 r9:ee804c00 r8:00000010 r7:00000000 r6:00000000 r5:ee804c10
[292048.703350] r4:c09552b4
[292048.705987] [<c002d824>] (omap2_gp_timer_interrupt) from [<c00723b4>] (handle_irq_event_percpu+0xb4/0x16c)
[292048.715771] [<c0072300>] (handle_irq_event_percpu) from [<c00724d0>] (handle_irq_event+0x64/0x90)
[292048.724763] r10:00000001 r9:b12fef90 r8:ee806000 r7:00000000 r6:00000000 r5:ee804c10
[292048.732738] r4:ee804c00
[292048.735374] [<c007246c>] (handle_irq_event) from [<c00752ac>] (handle_level_irq+0xb8/0x168)
[292048.743843] r5:ee804c10 r4:ee804c00
[292048.747533] [<c00751f4>] (handle_level_irq) from [<c0071a7c>] (generic_handle_irq+0x2c/0x3c)
[292048.756090] r5:00000010 r4:c0967f20
[292048.759780] [<c0071a50>] (generic_handle_irq) from [<c0071ce8>] (__handle_domain_irq+0x5c/0xb0)
[292048.768604] [<c0071c8c>] (__handle_domain_irq) from [<c0009468>] (omap_intc_handle_irq+0x3c/0x98)
[292048.777596] r9:b12fef90 r8:10c5387d r7:10c5387d r6:ffffffff r5:60070030 r4:c09cd100
[292048.785490] [<c000942c>] (omap_intc_handle_irq) from [<c00144f4>] (__irq_usr+0x54/0x80)
[292048.793612] Exception stack(0xededbfb0 to 0xededbff8)
[292048.798772] bfa0: 00000000 00000000 00000341 00000000
[292048.807073] bfc0: 012cb910 b12feacc f0000000 00000032 0082d2c4 b12fef90 00000001 b6015b60
[292048.815372] bfe0: 012aa6b8 b12fea58 005f4ef9 b613bf9a 60070030 ffffffff
[292048.822096] r5:60070030 r4:b613bf9a
[352488.903487] cpts: unable to obtain a time stamp
[385923.120440] cpts: unable to obtain a time stamp

[Log 2 - telnet]

_____ _____ _ _
| _ |___ ___ ___ ___ | _ |___ ___ |_|___ ___| |_
| | _| .'| . | . | | __| _| . | | | -_| _| _|
|__|__|_| |__,|_ |___| |__| |_| |___|_| |___|___|_|
|___| |___|

Arago Project http://arago-project.org am335x-evm

Arago 2016.05 am335x-evm

am335x-evm login: root
Last login: Sun Sep 24 10:13:54 UTC 2017 on pts/0
root@am335x-evm:~# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Sep19 ? 00:51:30 /sbin/init
root 2 0 0 Sep19 ? 00:00:00 [kthreadd]
root 3 2 0 Sep19 ? 00:03:46 [ksoftirqd/0]
root 5 2 0 Sep19 ? 00:00:00 [kworker/0:0H]
root 7 2 0 Sep19 ? 00:13:05 [rcu_preempt]
root 8 2 0 Sep19 ? 00:00:07 [rcu_sched]
root 9 2 0 Sep19 ? 00:00:00 [rcu_bh]
root 10 2 0 Sep19 ? 00:00:00 [kdevtmpfs]
root 11 2 0 Sep19 ? 00:00:00 [perf]
root 13 2 0 Sep19 ? 00:00:00 [writeback]
root 14 2 0 Sep19 ? 00:00:00 [crypto]
root 15 2 0 Sep19 ? 00:00:00 [bioset]
root 16 2 0 Sep19 ? 00:00:00 [kblockd]
root 17 2 0 Sep19 ? 00:00:00 [edac-poller]
root 18 2 0 Sep19 ? 00:00:00 [rpciod]
root 36 2 0 Sep19 ? 00:00:00 [kswapd0]
root 37 2 0 Sep19 ? 00:00:00 [fsnotify_mark]
root 38 2 0 Sep19 ? 00:00:00 [nfsiod]
root 67 2 0 Sep19 ? 00:00:00 [dma_wq]
root 68 2 0 Sep19 ? 00:00:00 [bioset]
root 69 2 0 Sep19 ? 00:00:00 [bioset]
root 70 2 0 Sep19 ? 00:00:00 [bioset]
root 71 2 0 Sep19 ? 00:00:00 [bioset]
root 72 2 0 Sep19 ? 00:00:00 [bioset]
root 73 2 0 Sep19 ? 00:00:00 [bioset]
root 74 2 0 Sep19 ? 00:00:00 [bioset]
root 75 2 0 Sep19 ? 00:00:00 [bioset]
root 76 2 0 Sep19 ? 00:00:11 [kworker/u2:1]
root 79 2 0 Sep19 ? 00:00:00 [kpsmoused]
root 93 2 0 Sep19 ? 00:00:00 [ipv6_addrconf]
root 94 2 0 Sep19 ? 00:00:00 [deferwq]
root 97 2 0 Sep19 ? 00:00:00 [bioset]
root 99 2 0 Sep19 ? 00:01:10 [HscMac0]
root 101 2 0 Sep19 ? 00:00:00 [HscMac1]
root 103 2 0 Sep19 ? 00:00:00 [HscMac2]
root 105 2 0 Sep19 ? 00:00:00 [HscMac3]
root 106 2 0 Sep19 ? 00:00:00 [irq/160-44e0b00]
root 107 2 0 Sep19 ? 00:00:00 [irq/161-4802a00]
root 108 2 0 Sep19 ? 00:00:00 [irq/162-4819c00]
root 110 2 0 Sep19 ? 00:00:00 [bioset]
root 111 2 0 Sep19 ? 00:00:00 [mmcqd/0]
root 112 2 0 Sep19 ? 00:00:00 [bioset]
root 113 2 0 Sep19 ? 00:00:00 [mmcqd/0boot0]
root 114 2 0 Sep19 ? 00:00:00 [bioset]
root 115 2 0 Sep19 ? 00:00:00 [mmcqd/0boot1]
root 117 2 0 Sep19 ? 00:00:00 [kworker/0:1H]
root 118 2 0 Sep19 ? 00:00:00 [jbd2/mmcblk0p2-]
root 119 2 0 Sep19 ? 00:00:00 [ext4-rsv-conver]
root 120 2 0 Sep19 ? 00:00:00 [scsi_eh_0]
root 121 2 0 Sep19 ? 00:00:00 [scsi_tmf_0]
root 122 2 0 Sep19 ? 00:00:29 [usb-storage]
root 133 2 0 Sep19 ? 00:00:00 [bioset]
root 136 1 0 Sep19 ? 00:06:24 /lib/systemd/systemd-journald
root 204 1 0 Sep19 ? 00:00:01 /lib/systemd/systemd-udevd
root 285 2 0 Sep19 ? 00:00:00 [jbd2/sda1-8]
root 288 2 0 Sep19 ? 00:00:00 [ext4-rsv-conver]
root 519 2 0 Sep19 ? 00:00:00 [jbd2/mmcblk0p4-]
root 520 2 0 Sep19 ? 00:00:00 [ext4-rsv-conver]
root 537 2 0 Sep19 ? 00:00:00 [jbd2/mmcblk0p3-]
root 538 2 0 Sep19 ? 00:00:00 [ext4-rsv-conver]
message+ 589 1 0 Sep19 ? 00:03:57 /usr/bin/dbus-daemon --system --address=sy
root 592 1 0 Sep19 ? 00:01:02 /lib/systemd/systemd-logind
root 595 1 0 Sep19 ? 00:00:00 /sbin/klogd -n
systemd+ 605 1 0 Sep19 ? 00:00:01 /lib/systemd/systemd-networkd
root 609 1 0 Sep19 ? 00:00:24 /sbin/syslogd -n -C
root 612 1 0 Sep19 ? 00:00:00 /usr/sbin/telnetd
root 637 2 0 Sep19 ? 00:00:06 [kworker/u2:3]
root 673 1 0 Sep19 ttyS0 00:00:00 /bin/login --
root 687 1 0 Sep19 tty1 00:00:00 /sbin/agetty --noclear tty1 linux
root 696 1 0 Sep19 ? 00:01:12 /usr/sbin/rngd -r /dev/urandom
root 780 1 0 Sep19 ? 00:00:00 /sbin/dhcpcd -q -t 0 -f /yamaha/settings/d
root 798 1 0 Sep19 ? 00:00:00 /sbin/dhcpcd -q -t 0 -f /yamaha/settings/d
root 810 1 8 Sep19 ? 09:31:58 /yamaha/app/engineunit_maincpu_prg
root 819 1 0 Sep19 ? 00:00:04 /sbin/dhcpcd -q -t 0 -f /yamaha/settings/d
root 951 1 0 Sep19 ? 00:00:00 /sbin/dhcpcd -q -t 0 -f /yamaha/settings/d
root 962 1 0 Sep19 ? 00:00:01 /sbin/dhcpcd -q -t 0 -f /yamaha/settings/d
root 21559 1 0 Sep20 ? 00:00:00 /lib/systemd/systemd --user
root 21568 21559 0 Sep20 ? 00:00:00 (sd-pam)
root 21571 673 0 Sep20 ttyS0 00:00:00 -sh
root 27046 2 0 09:49 ? 00:00:00 [kworker/0:3]
root 27404 2 0 10:04 ? 00:00:00 [kworker/0:0]
root 27435 2 0 10:05 ? 00:00:00 [kworker/0:1]
root 27572 2 0 10:11 ? 00:00:00 [kworker/0:2]
root 27699 1 0 10:16 ? 00:00:00 /sbin/agetty -8 -L ttyS3 115200 xterm
root 27702 612 0 10:16 pts/0 00:00:00 /bin/login
root 27704 27702 0 10:13 pts/0 00:00:00 -sh
root 27707 612 0 10:16 pts/1 00:00:00 /bin/login
root 27709 27707 0 10:16 pts/1 00:00:00 -sh
root 27711 27709 0 10:16 pts/1 00:00:00 ps -ef
root@am335x-evm:~#
root@am335x-evm:~# top H
top - 10:16:38 up 4 days, 12:55, 4 users, load average: 0.00, 0.00, 0.00
Threads: 132 total, 2 running, 130 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1028568 total, 795224 free, 54772 used, 178572 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 839780 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 5296 4228 2752 S 0.0 0.4 51:30.11 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.14 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 3:46.04 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root 20 0 0 0 0 S 0.0 0.0 13:05.31 rcu_preempt
8 root 20 0 0 0 0 S 0.0 0.0 0:07.24 rcu_sched
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
10 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kdevtmpfs
11 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 perf
13 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
14 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd
17 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 edac-poller
18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 rpciod
36 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0
37 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_mark
38 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 nfsiod
67 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 dma_wq
68 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
69 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
70 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
71 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
72 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
73 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
74 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
75 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
76 root 20 0 0 0 0 S 0.0 0.0 0:11.27 kworker/u2:1
79 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
93 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ipv6_addrconf

top - 10:16:39 up 4 days, 12:55, 4 users, load average: 0.00, 0.00, 0.00
Threads: 132 total, 2 running, 130 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1028568 total, 795224 free, 54772 used, 178572 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 839780 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 5296 4228 2752 S 0.0 0.4 51:30.11 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.14 kthreadd
3 root 20 0 0 0 0 R 0.0 0.0 3:46.04 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root 20 0 0 0 0 S 0.0 0.0 13:05.31 rcu_preempt
8 root 20 0 0 0 0 S 0.0 0.0 0:07.24 rcu_sched
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
10 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kdevtmpfs
11 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 perf
13 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
14 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset top - 10:14:49 up 4 days, 12:53, 4 users, load average: 0.00, 0.00, 0.00
Threads: 132 total, 1 running, 131 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1028568 total, 779104 free, 54904 used, 194560 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 823672 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
825 root rt -2 481812 59012 19728 S 0.0 5.7 0:00.00 mix-PWDWN
826 root -99 -2 481812 59012 19728 S 0.0 5.7 0:00.00 mix-POFF
827 root -11 -2 481812 59012 19728 S 0.0 5.7 21:51.74 mix-TMU
832 root 17 -3 481812 59012 19728 S 0.0 5.7 91:22.94 TIMKRNL
833 root 21 1 481812 59012 19728 S 0.0 5.7 26:13.35 DBGMON
834 root 21 1 481812 59012 19728 S 0.0 5.7 0:00.02 STORAGE
835 root -51 -2 481812 59012 19728 S 0.0 5.7 0:00.00 mix-EINT0
836 root -51 -2 481812 59012 19728 S 0.0 5.7 0:00.00 mix-EINT1
837 root -51 -2 481812 59012 19728 S 0.0 5.7 0:00.00 mix-EINT2
838 root 19 -1 481812 59012 19728 S 0.0 5.7 100:23.79 VsockWrap
839 root 19 -1 481812 59012 19728 S 0.0 5.7 7:31.75 YSDPSVC
840 root 20 0 481812 59012 19728 S 0.0 5.7 1:16.61 DEVCTRL
841 root 20 0 481812 59012 19728 S 0.0 5.7 20:44.99 COMSRV
842 root 20 0 481812 59012 19728 S 0.0 5.7 0:08.55 TOPOLGY
843 root 20 0 481812 59012 19728 S 0.0 5.7 8:15.62 SNKCLI
844 root 20 0 481812 59012 19728 S 0.0 5.7 2:43.21 TLCTRL
845 root 20 0 481812 59012 19728 S 0.0 5.7 39:40.50 MIXDIO
846 root 20 0 481812 59012 19728 S 0.0 5.7 9:27.87 MIXDSP
847 root 20 0 481812 59012 19728 S 0.0 5.7 0:00.88 PERHST
848 root 20 0 481812 59012 19728 S 0.0 5.7 83:42.63 PERDSP
849 root 20 0 481812 59012 19728 S 0.0 5.7 11:11.20 MTRRLY
850 root 20 0 481812 59012 19728 S 0.0 5.7 1:55.61 DIAG
851 root 20 0 481812 59012 19728 S 0.0 5.7 2:00.84 FRMWAR
852 root 20 0 481812 59012 19728 S 0.0 5.7 0:00.55 PRMTR
853 root 20 0 481812 59012 19728 S 0.0 5.7 21:59.53 PNLCTRL
854 root 20 0 481812 59012 19728 S 0.0 5.7 1:34.16 COMJK
855 root 20 0 481812 59012 19728 S 0.0 5.7 0:11.87 EVLOG
856 root 20 0 481812 59012 19728 S 0.0 5.7 2:24.38 EDETECT
857 root 20 0 481812 59012 19728 S 0.0 5.7 0:00.01 PREFTCH
858 root 20 0 481812 59012 19728 S 0.0 5.7 42:41.84 FADCOM
859 root 20 0 481812 59012 19728 S 0.0 5.7 11:41.63 BCACOM
860 root 20 0 481812 59012 19728 S 0.0 5.7 10:42.33 BCAMAIN
top - 10:14:11 up 4 days, 12:53, 4 users, load average: 0.00, 0.00, 0.00
Threads: 132 total, 2 running, 130 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1028568 total, 778004 free, 54900 used, 195664 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 822588 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND WCHAN Flags
825 root rt -2 481812 59012 19728 S 0.0 5.7 0:00.00 mix-PWDWN futex_wai+ 4.4.4.4.
826 root -99 -2 481812 59012 19728 S 0.0 5.7 0:00.00 mix-POFF poll_sche+ ..4.4.4.
827 root -11 -2 481812 59012 19728 S 0.0 5.7 21:51.74 mix-TMU timerfd_r+ ..4.4.4.
832 root 17 -3 481812 59012 19728 S 0.0 5.7 91:22.94 TIMKRNL poll_sche+ ..4.414.
833 root 21 1 481812 59012 19728 S 0.0 5.7 26:13.35 DBGMON poll_sche+ ..4.4.4.
834 root 21 1 481812 59012 19728 S 0.0 5.7 0:00.02 STORAGE poll_sche+ ..4.4.4.
835 root -51 -2 481812 59012 19728 S 0.0 5.7 0:00.00 mix-EINT0 poll_sche+ ..4.4.4.
836 root -51 -2 481812 59012 19728 S 0.0 5.7 0:00.00 mix-EINT1 poll_sche+ ..4.4.4.
837 root -51 -2 481812 59012 19728 S 0.0 5.7 0:00.00 mix-EINT2 poll_sche+ ..4.4.4.
838 root 19 -1 481812 59012 19728 S 0.0 5.7 100:23.79 VsockWrap futex_wai+ 4.4.4.4.
839 root 19 -1 481812 59012 19728 S 0.0 5.7 7:31.75 YSDPSVC futex_wai+ 4.4.4.4.
840 root 20 0 481812 59012 19728 S 0.0 5.7 1:16.61 DEVCTRL poll_sche+ ..4.4.4.
841 root 20 0 481812 59012 19728 S 0.0 5.7 20:44.99 COMSRV futex_wai+ 4.4.4.4.
842 root 20 0 481812 59012 19728 S 0.0 5.7 0:08.55 TOPOLGY poll_sche+ ..4.4.4.
843 root 20 0 481812 59012 19728 S 0.0 5.7 8:15.62 SNKCLI futex_wai+ 4.4.4.4.
844 root 20 0 481812 59012 19728 S 0.0 5.7 2:43.21 TLCTRL poll_sche+ ..4.4.4.
845 root 20 0 481812 59012 19728 S 0.0 5.7 39:40.50 MIXDIO poll_sche+ ..4.4.4.
846 root 20 0 481812 59012 19728 S 0.0 5.7 9:27.87 MIXDSP poll_sche+ ..4.4.4.
847 root 20 0 481812 59012 19728 S 0.0 5.7 0:00.88 PERHST poll_sche+ ..4.4.4.
848 root 20 0 481812 59012 19728 S 0.0 5.7 83:42.63 PERDSP poll_sche+ ..4.4.4.
849 root 20 0 481812 59012 19728 S 0.0 5.7 11:11.20 MTRRLY poll_sche+ ..4.4.4.
850 root 20 0 481812 59012 19728 S 0.0 5.7 1:55.61 DIAG poll_sche+ ..4.4.4.
851 root 20 0 481812 59012 19728 S 0.0 5.7 2:00.84 FRMWAR poll_sche+ ..4.4.4.
852 root 20 0 481812 59012 19728 S 0.0 5.7 0:00.55 PRMTR poll_sche+ ..4.4.4.
853 root 20 0 481812 59012 19728 S 0.0 5.7 21:59.53 PNLCTRL poll_sche+ ..4.4.4.
854 root 20 0 481812 59012 19728 S 0.0 5.7 1:34.16 COMJK poll_sche+ ..4.4.4.
855 root 20 0 481812 59012 19728 S 0.0 5.7 0:11.87 EVLOG poll_sche+ ..4.4.4.
856 root 20 0 481812 59012 19728 S 0.0 5.7 2:24.38 EDETECT futex_wai+ 4.4.4.4.
857 root 20 0 481812 59012 19728 S 0.0 5.7 0:00.01 PREFTCH poll_sche+ ..4.4.4.
858 root 20 0 481812 59012 19728 S 0.0 5.7 42:41.84 FADCOM poll_sche+ ..4.4.4.
859 root 20 0 481812 59012 19728 S 0.0 5.7 11:41.63 BCACOM poll_sche+ ..4.4.4.
860 root 20 0 481812 59012 19728 S 0.0 5.7 10:42.33 BCAMAIN poll_sche+ ..4.4.4.

over 8 years ago

0 Cvetolin Shulev-XID over 8 years ago

TI__Guru 65405 points

Hi user5132856,

Could you check whether the issue appears with the last Processor SDK v04.01 available at:
www.ti.com/.../processor-sdk-am335x
because there is many changes and patches.

BR
Tsvetolin Shulev

Processors

Processors forum

Linux/AM3352: RCU self-detected stall sometimes happens.