This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/AM3352: CPU stall message

Part Number: AM3352

Tool/software: Linux

Hello

my SDK is SDK04.00.00.04, linux version is 4.9.28.

my system sometimes got this message as following:

[ 1900.380254] INFO: rcu_preempt self-detected stall on CPU
[ 1900.385887] 0-...: (2 ticks this GP) idle=2db/1/0 softirq=58034/58034 fqs=0
[ 1900.393370] (t=17896 jiffies g=40285 c=40284 q=1)
[ 1900.398499] rcu_preempt kthread starved for 17896 jiffies! g40285 c40284 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
[ 1900.408802] rcu_preempt S 0 7 2 0x00000000
[ 1900.414561] Backtrace:
[ 1900.417163] [<c07c4a64>] (__schedule) from [<c07c508c>] (schedule+0x54/0xb8)
[ 1900.424564] r10:c0c13d80 r9:c0c12c80 r8:00000000 r7:c0c13d80 r6:cf06fed8 r5:00027126
[ 1900.432770] r4:cf06e000
[ 1900.435439] [<c07c5038>] (schedule) from [<c07c82e8>] (schedule_timeout+0x17c/0x224)
[ 1900.443559] r5:00027126 r4:c0c12c80
[ 1900.447318] [<c07c816c>] (schedule_timeout) from [<c0175bd4>] (rcu_gp_kthread+0x690/0x99c)
[ 1900.455988] r9:00000001 r8:c0c125a8 r7:c0c124d0 r6:00000003 r5:cf06e000 r4:c0c124d0
[ 1900.464125] [<c0175544>] (rcu_gp_kthread) from [<c0147e18>] (kthread+0xf8/0x110)
[ 1900.471878] r7:c0c124d0
[ 1900.474551] [<c0147d20>] (kthread) from [<c0107d38>] (ret_from_fork+0x14/0x3c)
[ 1900.482126] r8:00000000 r7:00000000 r6:00000000 r5:c0147d20 r4:cf01f7c0
[ 1900.489155] Task dump for CPU 0:
[ 1900.492536] swapper R running task 0 0 0 0x00000002
[ 1900.499934] Backtrace:
[ 1900.502517] [<c010b790>] (dump_backtrace) from [<c010ba4c>] (show_stack+0x18/0x1c)
[ 1900.510457] r7:c0c02900 r6:00000000 r5:00000000 r4:c0c05580
[ 1900.516400] [<c010ba34>] (show_stack) from [<c015133c>] (sched_show_task+0xb8/0x108)
[ 1900.524524] [<c0151284>] (sched_show_task) from [<c015235c>] (dump_cpu_task+0x2c/0x30)
[ 1900.532823] r5:c0c00000 r4:c0c124d0
[ 1900.536590] [<c0152330>] (dump_cpu_task) from [<c01b6620>] (rcu_dump_cpu_stacks+0xb4/0xd8)
[ 1900.545261] [<c01b656c>] (rcu_dump_cpu_stacks) from [<c01767b4>] (rcu_check_callbacks+0x4fc/0x970)
[ 1900.554659] r10:c0c02900 r9:c0c028f8 r8:c0c13d80 r7:00000000 r6:c0c12968 r5:c0c124d0
[ 1900.562865] r4:c0c129b4 r3:c0c00000
[ 1900.566623] [<c01762b8>] (rcu_check_callbacks) from [<c017a068>] (update_process_times+0x3c/0x64)
[ 1900.575928] r10:c0c13e4c r9:c018a284 r8:00000000 r7:000001e4 r6:00000000 r5:c0c05580
[ 1900.584135] r4:ffffe000
[ 1900.586809] [<c017a02c>] (update_process_times) from [<c018a280>] (tick_sched_handle+0x50/0x54)
[ 1900.595931] r7:000001e4 r6:2224823f r5:c0c01ed0 r4:c0c144c0
[ 1900.601870] [<c018a230>] (tick_sched_handle) from [<c018a2e8>] (tick_sched_timer+0x64/0xb8)
[ 1900.610633] [<c018a284>] (tick_sched_timer) from [<c017ada8>] (__hrtimer_run_queues.constprop.4+0x150/0x1e0)
[ 1900.620938] r7:c0c13e40 r6:c0c00000 r5:c0c144c0 r4:c0c13e00
[ 1900.626878] [<c017ac58>] (__hrtimer_run_queues.constprop.4) from [<c017b068>] (hrtimer_interrupt+0xb4/0x250)
[ 1900.637184] r10:c0c13eb8 r9:c0c01d80 r8:ffffffff r7:7fffffff r6:00000002 r5:c0c00000
[ 1900.645388] r4:c0c13e00
[ 1900.648064] [<c017afb4>] (hrtimer_interrupt) from [<c0119ab8>] (omap2_gp_timer_interrupt+0x30/0x38)
[ 1900.657551] r10:c0c15c48 r9:cf019000 r8:00000010 r7:c0c01e34 r6:00000000 r5:cf019000
[ 1900.665755] r4:c0c07674
[ 1900.668431] [<c0119a88>] (omap2_gp_timer_interrupt) from [<c0168770>] (__handle_irq_event_percpu+0xb4/0x144)
[ 1900.678743] [<c01686bc>] (__handle_irq_event_percpu) from [<c0168824>] (handle_irq_event_percpu+0x24/0x60)
[ 1900.688871] r10:00000000 r9:c0c00000 r8:cf006000 r7:00000000 r6:00000000 r5:cf019000
[ 1900.697076] r4:cf019000
[ 1900.699739] [<c0168800>] (handle_irq_event_percpu) from [<c01688c4>] (handle_irq_event+0x64/0x90)
[ 1900.709037] r5:cf019010 r4:cf019000
[ 1900.712793] [<c0168860>] (handle_irq_event) from [<c016ba04>] (handle_level_irq+0xb8/0x164)
[ 1900.721544] r5:cf019010 r4:cf019000
[ 1900.725296] [<c016b94c>] (handle_level_irq) from [<c0167958>] (generic_handle_irq+0x2c/0x3c)
[ 1900.734138] r5:00000010 r4:c0c1d7ec
[ 1900.737893] [<c016792c>] (generic_handle_irq) from [<c0168014>] (__handle_domain_irq+0x5c/0xb0)
[ 1900.747017] [<c0167fb8>] (__handle_domain_irq) from [<c0101484>] (omap_intc_handle_irq+0x3c/0x98)
[ 1900.756323] r9:c0c00000 r8:0097cd54 r7:c0c01f04 r6:ffffffff r5:20000013 r4:c0c745f0
[ 1900.764443] [<c0101448>] (omap_intc_handle_irq) from [<c010c54c>] (__irq_svc+0x6c/0xa8)
[ 1900.772832] Exception stack(0xc0c01ed0 to 0xc0c01f18)
[ 1900.778130] 1ec0: 00000000 000001ba f779933d c0c0f2d8
[ 1900.786710] 1ee0: 76e3d342 000001ba 00000000 cf4d6000 0097cd54 000001ba 00000000 c0c01f54
[ 1900.795284] 1f00: c0c01f20 c0c01f20 c060f390 c060f398 20000013 ffffffff
[ 1900.802217] r5:20000013 r4:c060f398
[ 1900.805977] [<c060f20c>] (cpuidle_enter_state) from [<c060f510>] (cpuidle_enter+0x1c/0x20)
[ 1900.814646] r10:c0c15c45 r9:c0c028d0 r8:c0c36358 r7:c0c028c8 r6:c0c36798 r5:c0c00000
[ 1900.822854] r4:cf4d6000
[ 1900.825526] [<c060f4f4>] (cpuidle_enter) from [<c0159400>] (call_cpuidle+0x28/0x40)
[ 1900.833561] [<c01593d8>] (call_cpuidle) from [<c0159628>] (cpu_startup_entry+0xe8/0x198)
[ 1900.842050] [<c0159540>] (cpu_startup_entry) from [<c07c403c>] (rest_init+0x8c/0x90)
[ 1900.850166] r7:c0c02840
[ 1900.852836] [<c07c3fb0>] (rest_init) from [<c0b00d44>] (start_kernel+0x3a8/0x3b4)
[ 1900.860679] r5:c0c42140 r4:c0c4218c
[ 1900.864435] [<c0b0099c>] (start_kernel) from [<80008078>] (0x80008078)

there is a simple thread:

but I can't understand the solution, could anyone tell me how to deal this issue? Is there any patch can use?

  • Hi Longfei Li,

    Could you give more details about your environment? Are you using custom board and if it is true what clock frequency you are using (32KHz or 25KHz or other)?
    Did you done some changes to the Linux kernel or default board config file?
    Only for test could you check whether the issue appears with the last Processor SDK v4.03?

    BR
    Tsvetolin Shulev
  • hello Tsvetolin
    my board is made by ourself, I don't know what clock frequency on my board, but I know that my board uses 24Mhz as Crystal Frequency which need sysboot[15:14]=01.
    for kernel config, I use tisdk_am335x-evm_defconfig as base, and I change to use OMAP serial port support instead 8250/16550 and compatible serial support.
    Use newer SDK to test need to spend some time to transplant the system to my board, but SDK6.00.00.00 which linux version is 3.2 can run well in my board without this issue.
  • Longfei Li,



    You probably have a real time application that is consuming all cpu (some bad implementation) and because of its real time scheduling priority the system doesn't have enough resources available for other tasks.
    I suggests that you remove real time priority from your applications and check which one is consuming a lot of CPU and, after correcting the problem, puts it back to real time priority
    You can found more suggestions at:
    stackoverflow.com/.../rcu-preempt-self-detected-stall-on-cpu-0
    www.kernel.org/.../stallwarn.txt

    BR
    Tsvetolin Shulev