This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/AM3352: LCPD-14446 and LCPD-14448 : 8250_omap: kernel dump happens when using multiple uart ports

Part Number: AM3352

Tool/software: Linux

Hi ,

  I used 6 uarts of the am3352(SDK 05.01.00.11) , five uarts for data transfer(3M baudrate)  and the other one(uart2 115200) for debug. 

Sometimes the kernel dump and the system works very slowly ,and the debug console can't input anything. 

  I don't know if this is the problem LCPD-14446  describes. It's a very important question to decide whether to choose this chip or not.

  You may have other solutions, please help me.

  • Hi,

    As I explained in thread e2e.ti.com/.../762161, this issue is caused by a kernel driver bug, it will be fix in the next Processor SDK release, at the moment please use the SDK v5.0.0.15 which doesn't have this driver bug.
  • I have test the SDK v5.0.0.15, it also have the problem.
  • Hi,
    Do you know when the next SDK version will be released?
  • The next SDK release v6.0 which will have the fix is scheduled to release in June. But if you still see the problem with SDK v5.0.0.15, it means it is a different issue from the one referred in LCPD-14446 ticket. Please describe your test case how to reproduce the problem and attach the full kernel dump log along with kernel boot log. I will try to replicate it and debug it.

    Please don't copy & paste the log in the post, attach it as a file instead.
  • [ 1703.874087] INFO: rcu_preempt self-detected stall on CPU
    [ 1703.879435]  0-...: (108435 ticks this GP) idle=7b6/140000000000002/0 softirq=20675/20675 fqs=0 
    [ 1703.888255]   (t=109251 jiffies g=14782 c=14781 q=2)
    [ 1703.893253] rcu_preempt kthread starved for 109251 jiffies! g14782 c14781 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0
    [ 1703.903907] rcu_preempt     R  running task        0     8      2 0x00000000
    [ 1703.910992] Backtrace: 
    [ 1703.913465] [<c085d3a0>] (__schedule) from [<c085db18>] (schedule+0x58/0xc4)
    [ 1703.920551]  r10:c0d14d50 r9:00000003 r8:c0d15400 r7:c0d15440 r6:cf05bee8 r5:c0d15440
    [ 1703.928415]  r4:ffffe000
    [ 1703.930968] [<c085dac0>] (schedule) from [<c086139c>] (schedule_timeout+0x184/0x264)
    [ 1703.938746]  r5:c0d15440 r4:000079a1
    [ 1703.942345] [<c0861218>] (schedule_timeout) from [<c0179540>] (rcu_gp_kthread+0x688/0xaa4)
    [ 1703.950650]  r8:00000001 r7:c0d14e0c r6:c0d15400 r5:ffffe000 r4:00000001
    [ 1703.957388] [<c0178eb8>] (rcu_gp_kthread) from [<c0145d4c>] (kthread+0x15c/0x164)
    [ 1703.964903]  r7:cf05a000
    [ 1703.967455] [<c0145bf0>] (kthread) from [<c0107e08>] (ret_from_fork+0x14/0x2c)
    [ 1703.974716]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0145bf0
    [ 1703.982579]  r4:cf030740
    [ 1703.985123] Task dump for CPU 0:
    [ 1703.988363] uart split      R  running task        0   219    110 0x00000002
    [ 1703.995448] Backtrace: 
    [ 1703.997919] [<c010ba48>] (dump_backtrace) from [<c010bd2c>] (show_stack+0x18/0x1c)
    [ 1704.005525]  r7:c0d14d50 r6:c0d03104 r5:0000006e r4:ce112400
    [ 1704.011220] [<c010bd14>] (show_stack) from [<c014b7a4>] (sched_show_task.part.9+0xc4/0xdc)
    [ 1704.019529] [<c014b6e0>] (sched_show_task.part.9) from [<c014fff0>] (dump_cpu_task+0x38/0x3c)
    [ 1704.028092]  r5:00000000 r4:c0d14d50
    [ 1704.031690] [<c014ffb8>] (dump_cpu_task) from [<c017bd20>] (rcu_dump_cpu_stacks+0x90/0xd0)
    [ 1704.039999] [<c017bc90>] (rcu_dump_cpu_stacks) from [<c017abdc>] (rcu_check_callbacks+0x5c4/0x8f0)
    [ 1704.049004]  r10:c0d030f8 r9:c0d03104 r8:c0d15400 r7:00000000 r6:c0d14d50 r5:c0d14d50
    [ 1704.056869]  r4:c0d15198 r3:ce0d8000
    [ 1704.060469] [<c017a618>] (rcu_check_callbacks) from [<c017f548>] (update_process_times+0x3c/0x64)
    [ 1704.069387]  r10:c018ecb4 r9:00000000 r8:00000001 r7:0000018c r6:00000000 r5:ce112400
    [ 1704.077251]  r4:ffffe000
    [ 1704.079803] [<c017f50c>] (update_process_times) from [<c018ead4>] (tick_sched_handle+0x5c/0x60)
    [ 1704.088543]  r7:0000018c r6:b6914099 r5:ce0d9eb0 r4:c0d16c80
    [ 1704.094233] [<c018ea78>] (tick_sched_handle) from [<c018ed18>] (tick_sched_timer+0x64/0xd0)
    [ 1704.102629] [<c018ecb4>] (tick_sched_timer) from [<c0180148>] (__hrtimer_run_queues.constprop.5+0x150/0x1e8)
    [ 1704.112504]  r7:ffffe000 r6:c0d165c0 r5:c0d16c80 r4:c0d16580
    [ 1704.118199] [<c017fff8>] (__hrtimer_run_queues.constprop.5) from [<c01804b8>] (hrtimer_interrupt+0x9c/0x248)
    [ 1704.128077]  r10:c0d16638 r9:c0d16580 r8:ffffffff r7:7fffffff r6:00000003 r5:ffffe000
    [ 1704.135941]  r4:c0d16580
    [ 1704.138495] [<c018041c>] (hrtimer_interrupt) from [<c011913c>] (omap2_gp_timer_interrupt+0x30/0x38)
    [ 1704.147589]  r10:c0d4fcc0 r9:cf006200 r8:00000010 r7:ce0d9e14 r6:00000000 r5:cf006200
    [ 1704.155452]  r4:c0d07f80
    [ 1704.158006] [<c011910c>] (omap2_gp_timer_interrupt) from [<c016a49c>] (__handle_irq_event_percpu+0x6c/0x140)
    [ 1704.167887] [<c016a430>] (__handle_irq_event_percpu) from [<c016a594>] (handle_irq_event_percpu+0x24/0x60)
    [ 1704.177591]  r10:bea93854 r9:ce0d8000 r8:cf008000 r7:ce0d9fb0 r6:00000000 r5:cf006200
    [ 1704.185454]  r4:cf006200
    [ 1704.188008] [<c016a570>] (handle_irq_event_percpu) from [<c016a634>] (handle_irq_event+0x64/0x90)
    [ 1704.196920]  r5:00000010 r4:cf006200
    [ 1704.200519] [<c016a5d0>] (handle_irq_event) from [<c016da5c>] (handle_level_irq+0xb4/0x158)
    [ 1704.208908]  r5:00000010 r4:cf006200
    [ 1704.212506] [<c016d9a8>] (handle_level_irq) from [<c01696ac>] (generic_handle_irq+0x2c/0x3c)
    [ 1704.220981]  r5:00000010 r4:c0d4e71c
    [ 1704.224581] [<c0169680>] (generic_handle_irq) from [<c0169d64>] (__handle_domain_irq+0x5c/0xb0)
    [ 1704.233326] [<c0169d08>] (__handle_domain_irq) from [<c0101464>] (omap_intc_handle_irq+0x3c/0x98)
    [ 1704.242244]  r9:ce0d8000 r8:cf008000 r7:ce0d9ee4 r6:ffffffff r5:60000113 r4:c0d834b0
    [ 1704.250028] [<c0101428>] (omap_intc_handle_irq) from [<c010c8cc>] (__irq_svc+0x6c/0xa8)
    [ 1704.258066] Exception stack(0xce0d9eb0 to 0xce0d9ef8)
    [ 1704.263145] 9ea0:                                     c0d51c40 00404040 c0d51c00 00000000
    [ 1704.271367] 9ec0: 00000202 00000010 00000000 ffffe000 cf008000 c0d51c40 bea93854 ce0d9f5c
    [ 1704.279586] 9ee0: ce0d9f60 ce0d9f00 c012c4e8 c0101554 60000113 ffffffff
    [ 1704.286230]  r5:60000113 r4:c0101554
    [ 1704.289829] [<c01014c0>] (__do_softirq) from [<c012c4e8>] (irq_exit+0x120/0x154)
    [ 1704.297263]  r10:bea93854 r9:b624ffa0 r8:cf008000 r7:00000000 r6:00000000 r5:00000010
    [ 1704.305127]  r4:c0d4e71c
    [ 1704.307680] [<c012c3c8>] (irq_exit) from [<c0169d68>] (__handle_domain_irq+0x60/0xb0)
    [ 1704.315553] [<c0169d08>] (__handle_domain_irq) from [<c0101464>] (omap_intc_handle_irq+0x3c/0x98)
    [ 1704.324472]  r9:b624ffa0 r8:10c53c7d r7:10c5387d r6:ffffffff r5:40000010 r4:c0d834b0
    [ 1704.332255] [<c0101428>] (omap_intc_handle_irq) from [<c010ccd4>] (__irq_usr+0x54/0x80)
    [ 1704.340294] Exception stack(0xce0d9fb0 to 0xce0d9ff8)
    [ 1704.345372] 9fa0:                                     00000000 00000000 b62504e4 00000000
    [ 1704.353592] 9fc0: 00000076 b6f03980 00000000 00000000 00000000 b624ffa0 bea93854 b624fdbc
    [ 1704.361812] 9fe0: 00000000 b624fd5c b6b37dec b6a66bc0 40000010 ffffffff
    [ 1704.368455]  r5:40000010 r4:b6a66bc0
    [ 1766.904085] INFO: rcu_preempt self-detected stall on CPU
    [ 1766.909433]  0-...: (114690 ticks this GP) idle=7b6/140000000000002/0 softirq=20675/20675 fqs=0 
    [ 1766.918254]   (t=115554 jiffies g=14782 c=14781 q=2)
    [ 1766.923251] rcu_preempt kthread starved for 115554 jiffies! g14782 c14781 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0
    [ 1766.933905] rcu_preempt     R  running task        0     8      2 0x00000000
    [ 1766.940991] Backtrace: 
    [ 1766.943464] [<c085d3a0>] (__schedule) from [<c085db18>] (schedule+0x58/0xc4)
    [ 1766.950550]  r10:c0d14d50 r9:00000003 r8:c0d15400 r7:c0d15440 r6:cf05bee8 r5:c0d15440
    [ 1766.958414]  r4:ffffe000
    [ 1766.960966] [<c085dac0>] (schedule) from [<c086139c>] (schedule_timeout+0x184/0x264)
    [ 1766.968744]  r5:c0d15440 r4:000079a1
    [ 1766.972343] [<c0861218>] (schedule_timeout) from [<c0179540>] (rcu_gp_kthread+0x688/0xaa4)
    [ 1766.980650]  r8:00000001 r7:c0d14e0c r6:c0d15400 r5:ffffe000 r4:00000001
    [ 1766.987388] [<c0178eb8>] (rcu_gp_kthread) from [<c0145d4c>] (kthread+0x15c/0x164)
    [ 1766.994903]  r7:cf05a000
    [ 1766.997455] [<c0145bf0>] (kthread) from [<c0107e08>] (ret_from_fork+0x14/0x2c)
    [ 1767.004716]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0145bf0
    [ 1767.012580]  r4:cf030740
    [ 1767.015123] Task dump for CPU 0:
    [ 1767.018363] uart split      R  running task        0   219    110 0x00000002
    [ 1767.025449] Backtrace: 
    [ 1767.027920] [<c010ba48>] (dump_backtrace) from [<c010bd2c>] (show_stack+0x18/0x1c)
    [ 1767.035527]  r7:c0d14d50 r6:c0d03104 r5:0000006e r4:ce112400
    [ 1767.041221] [<c010bd14>] (show_stack) from [<c014b7a4>] (sched_show_task.part.9+0xc4/0xdc)
    [ 1767.049529] [<c014b6e0>] (sched_show_task.part.9) from [<c014fff0>] (dump_cpu_task+0x38/0x3c)
    [ 1767.058093]  r5:00000000 r4:c0d14d50
    [ 1767.061691] [<c014ffb8>] (dump_cpu_task) from [<c017bd20>] (rcu_dump_cpu_stacks+0x90/0xd0)
    [ 1767.070001] [<c017bc90>] (rcu_dump_cpu_stacks) from [<c017abdc>] (rcu_check_callbacks+0x5c4/0x8f0)
    [ 1767.079005]  r10:c0d030f8 r9:c0d03104 r8:c0d15400 r7:00000000 r6:c0d14d50 r5:c0d14d50
    [ 1767.086871]  r4:c0d15198 r3:ce0d8000
    [ 1767.090472] [<c017a618>] (rcu_check_callbacks) from [<c017f548>] (update_process_times+0x3c/0x64)
    [ 1767.099390]  r10:c018ecb4 r9:00000000 r8:00000001 r7:0000019b r6:00000000 r5:ce112400
    [ 1767.107253]  r4:ffffe000
    [ 1767.109805] [<c017f50c>] (update_process_times) from [<c018ead4>] (tick_sched_handle+0x5c/0x60)
    [ 1767.118546]  r7:0000019b r6:6372b2cd r5:ce0d9eb0 r4:c0d16c80
    [ 1767.124236] [<c018ea78>] (tick_sched_handle) from [<c018ed18>] (tick_sched_timer+0x64/0xd0)
    [ 1767.132633] [<c018ecb4>] (tick_sched_timer) from [<c0180148>] (__hrtimer_run_queues.constprop.5+0x150/0x1e8)
    [ 1767.142508]  r7:ffffe000 r6:c0d165c0 r5:c0d16c80 r4:c0d16580
    [ 1767.148202] [<c017fff8>] (__hrtimer_run_queues.constprop.5) from [<c01804b8>] (hrtimer_interrupt+0x9c/0x248)
    [ 1767.158079]  r10:c0d16638 r9:c0d16580 r8:ffffffff r7:7fffffff r6:00000003 r5:ffffe000
    [ 1767.165943]  r4:c0d16580
    [ 1767.168497] [<c018041c>] (hrtimer_interrupt) from [<c011913c>] (omap2_gp_timer_interrupt+0x30/0x38)
    [ 1767.177590]  r10:c0d4fcc0 r9:cf006200 r8:00000010 r7:ce0d9e14 r6:00000000 r5:cf006200
    [ 1767.185453]  r4:c0d07f80
    [ 1767.188009] [<c011910c>] (omap2_gp_timer_interrupt) from [<c016a49c>] (__handle_irq_event_percpu+0x6c/0x140)
    [ 1767.197891] [<c016a430>] (__handle_irq_event_percpu) from [<c016a594>] (handle_irq_event_percpu+0x24/0x60)
    [ 1767.207594]  r10:bea93854 r9:ce0d8000 r8:cf008000 r7:ce0d9fb0 r6:00000000 r5:cf006200
    [ 1767.215457]  r4:cf006200
    [ 1767.218012] [<c016a570>] (handle_irq_event_percpu) from [<c016a634>] (handle_irq_event+0x64/0x90)
    [ 1767.226924]  r5:00000010 r4:cf006200
    [ 1767.230525] [<c016a5d0>] (handle_irq_event) from [<c016da5c>] (handle_level_irq+0xb4/0x158)
    [ 1767.238914]  r5:00000010 r4:cf006200
    [ 1767.242512] [<c016d9a8>] (handle_level_irq) from [<c01696ac>] (generic_handle_irq+0x2c/0x3c)
    [ 1767.250989]  r5:00000010 r4:c0d4e71c
    [ 1767.254588] [<c0169680>] (generic_handle_irq) from [<c0169d64>] (__handle_domain_irq+0x5c/0xb0)
    [ 1767.263333] [<c0169d08>] (__handle_domain_irq) from [<c0101464>] (omap_intc_handle_irq+0x3c/0x98)
    [ 1767.272251]  r9:ce0d8000 r8:cf008000 r7:ce0d9ee4 r6:ffffffff r5:60000113 r4:c0d834b0
    [ 1767.280035] [<c0101428>] (omap_intc_handle_irq) from [<c010c8cc>] (__irq_svc+0x6c/0xa8)
    [ 1767.288073] Exception stack(0xce0d9eb0 to 0xce0d9ef8)
    [ 1767.293151] 9ea0:                                     c0d51c40 00404040 c0d51c00 00000000
    [ 1767.301373] 9ec0: 00000202 00000010 00000000 ffffe000 cf008000 c0d51c40 bea93854 ce0d9f5c
    [ 1767.309592] 9ee0: ce0d9f60 ce0d9f00 c012c4e8 c0101554 60000113 ffffffff
    [ 1767.316236]  r5:60000113 r4:c0101554
    [ 1767.319834] [<c01014c0>] (__do_softirq) from [<c012c4e8>] (irq_exit+0x120/0x154)
    [ 1767.327269]  r10:bea93854 r9:b624ffa0 r8:cf008000 r7:00000000 r6:00000000 r5:00000010
    [ 1767.335133]  r4:c0d4e71c
    [ 1767.337686] [<c012c3c8>] (irq_exit) from [<c0169d68>] (__handle_domain_irq+0x60/0xb0)
    [ 1767.345559] [<c0169d08>] (__handle_domain_irq) from [<c0101464>] (omap_intc_handle_irq+0x3c/0x98)
    [ 1767.354477]  r9:b624ffa0 r8:10c53c7d r7:10c5387d r6:ffffffff r5:40000010 r4:c0d834b0
    [ 1767.362259] [<c0101428>] (omap_intc_handle_irq) from [<c010ccd4>] (__irq_usr+0x54/0x80)
    [ 1767.370299] Exception stack(0xce0d9fb0 to 0xce0d9ff8)
    [ 1767.375376] 9fa0:                                     00000000 00000000 b62504e4 00000000
    [ 1767.383598] 9fc0: 00000076 b6f03980 00000000 00000000 00000000 b624ffa0 bea93854 b624fdbc
    [ 1767.391817] 9fe0: 00000000 b624fd5c b6b37dec b6a66bc0 40000010 ffffffff
    [ 1767.398460]  r5:40000010 r4:b6a66bc0
    [ 1829.934083] INFO: rcu_preempt self-detected stall on CPU
    [ 1829.939434]  0-...: (120945 ticks this GP) idle=7b6/140000000000002/0 softirq=20675/20675 fqs=0 
    [ 1829.948254]   (t=121857 jiffies g=14782 c=14781 q=2)
    [ 1829.953252] rcu_preempt kthread starved for 121857 jiffies! g14782 c14781 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=0
    [ 1829.963906] rcu_preempt     R  running task        0     8      2 0x00000000
    [ 1829.970992] Backtrace: 
    [ 1829.973463] [<c085d3a0>] (__schedule) from [<c085db18>] (schedule+0x58/0xc4)
    [ 1829.980551]  r10:c0d14d50 r9:00000003 r8:c0d15400 r7:c0d15440 r6:cf05bee8 r5:c0d15440
    [ 1829.988415]  r4:ffffe000
    [ 1829.990968] [<c085dac0>] (schedule) from [<c086139c>] (schedule_timeout+0x184/0x264)
    [ 1829.998745]  r5:c0d15440 r4:000079a1
    [ 1830.002345] [<c0861218>] (schedule_timeout) from [<c0179540>] (rcu_gp_kthread+0x688/0xaa4)
    [ 1830.010650]  r8:00000001 r7:c0d14e0c r6:c0d15400 r5:ffffe000 r4:00000001
    [ 1830.017389] [<c0178eb8>] (rcu_gp_kthread) from [<c0145d4c>] (kthread+0x15c/0x164)
    [ 1830.024903]  r7:cf05a000
    [ 1830.027457] [<c0145bf0>] (kthread) from [<c0107e08>] (ret_from_fork+0x14/0x2c)
    [ 1830.034718]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0145bf0
    [ 1830.042581]  r4:cf030740
    [ 1830.045125] Task dump for CPU 0:
    [ 1830.048364] uart split      R  running task        0   219    110 0x00000002
    [ 1830.055450] Backtrace: 
    [ 1830.057920] [<c010ba48>] (dump_backtrace) from [<c010bd2c>] (show_stack+0x18/0x1c)
    [ 1830.065527]  r7:c0d14d50 r6:c0d03104 r5:0000006e r4:ce112400
    [ 1830.071222] [<c010bd14>] (show_stack) from [<c014b7a4>] (sched_show_task.part.9+0xc4/0xdc)
    [ 1830.079530] [<c014b6e0>] (sched_show_task.part.9) from [<c014fff0>] (dump_cpu_task+0x38/0x3c)
    [ 1830.088094]  r5:00000000 r4:c0d14d50
    [ 1830.091692] [<c014ffb8>] (dump_cpu_task) from [<c017bd20>] (rcu_dump_cpu_stacks+0x90/0xd0)
    [ 1830.100000] [<c017bc90>] (rcu_dump_cpu_stacks) from [<c017abdc>] (rcu_check_callbacks+0x5c4/0x8f0)
    [ 1830.109006]  r10:c0d030f8 r9:c0d03104 r8:c0d15400 r7:00000000 r6:c0d14d50 r5:c0d14d50
    [ 1830.116871]  r4:c0d15198 r3:ce0d8000
    [ 1830.120472] [<c017a618>] (rcu_check_callbacks) from [<c017f548>] (update_process_times+0x3c/0x64)
    [ 1830.129390]  r10:c018ecb4 r9:00000000 r8:00000001 r7:000001aa r6:00000000 r5:ce112400
    [ 1830.137254]  r4:ffffe000
    [ 1830.139805] [<c017f50c>] (update_process_times) from [<c018ead4>] (tick_sched_handle+0x5c/0x60)
    [ 1830.148546]  r7:000001aa r6:1054279c r5:ce0d9eb0 r4:c0d16c80
    [ 1830.154236] [<c018ea78>] (tick_sched_handle) from [<c018ed18>] (tick_sched_timer+0x64/0xd0)
    [ 1830.162633] [<c018ecb4>] (tick_sched_timer) from [<c0180148>] (__hrtimer_run_queues.constprop.5+0x150/0x1e8)
    [ 1830.172508]  r7:ffffe000 r6:c0d165c0 r5:c0d16c80 r4:c0d16580
    [ 1830.178204] [<c017fff8>] (__hrtimer_run_queues.constprop.5) from [<c01804b8>] (hrtimer_interrupt+0x9c/0x248)
    [ 1830.188081]  r10:c0d16638 r9:c0d16580 r8:ffffffff r7:7fffffff r6:00000003 r5:ffffe000
    [ 1830.195945]  r4:c0d16580
    [ 1830.198501] [<c018041c>] (hrtimer_interrupt) from [<c011913c>] (omap2_gp_timer_interrupt+0x30/0x38)
    [ 1830.207593]  r10:c0d4fcc0 r9:cf006200 r8:00000010 r7:ce0d9e14 r6:00000000 r5:cf006200
    [ 1830.215456]  r4:c0d07f80
    [ 1830.218012] [<c011910c>] (omap2_gp_timer_interrupt) from [<c016a49c>] (__handle_irq_event_percpu+0x6c/0x140)
    [ 1830.227893] [<c016a430>] (__handle_irq_event_percpu) from [<c016a594>] (handle_irq_event_percpu+0x24/0x60)
    [ 1830.237597]  r10:bea93854 r9:ce0d8000 r8:cf008000 r7:ce0d9fb0 r6:00000000 r5:cf006200
    [ 1830.245461]  r4:cf006200
    [ 1830.248015] [<c016a570>] (handle_irq_event_percpu) from [<c016a634>] (handle_irq_event+0x64/0x90)
    [ 1830.256927]  r5:00000010 r4:cf006200
    [ 1830.260525] [<c016a5d0>] (handle_irq_event) from [<c016da5c>] (handle_level_irq+0xb4/0x158)
    [ 1830.268915]  r5:00000010 r4:cf006200
    [ 1830.272514] [<c016d9a8>] (handle_level_irq) from [<c01696ac>] (generic_handle_irq+0x2c/0x3c)
    [ 1830.280989]  r5:00000010 r4:c0d4e71c
    [ 1830.284589] [<c0169680>] (generic_handle_irq) from [<c0169d64>] (__handle_domain_irq+0x5c/0xb0)
    [ 1830.293333] [<c0169d08>] (__handle_domain_irq) from [<c0101464>] (omap_intc_handle_irq+0x3c/0x98)
    [ 1830.302251]  r9:ce0d8000 r8:cf008000 r7:ce0d9ee4 r6:ffffffff r5:60000113 r4:c0d834b0
    [ 1830.310035] [<c0101428>] (omap_intc_handle_irq) from [<c010c8cc>] (__irq_svc+0x6c/0xa8)
    [ 1830.318074] Exception stack(0xce0d9eb0 to 0xce0d9ef8)
    [ 1830.323152] 9ea0:                                     c0d51c40 00404040 c0d51c00 00000000
    [ 1830.331375] 9ec0: 00000202 00000010 00000000 ffffe000 cf008000 c0d51c40 bea93854 ce0d9f5c
    [ 1830.339593] 9ee0: ce0d9f60 ce0d9f00 c012c4e8 c0101554 60000113 ffffffff
    [ 1830.346238]  r5:60000113 r4:c0101554
    [ 1830.349836] [<c01014c0>] (__do_softirq) from [<c012c4e8>] (irq_exit+0x120/0x154)
    [ 1830.357271]  r10:bea93854 r9:b624ffa0 r8:cf008000 r7:00000000 r6:00000000 r5:00000010
    [ 1830.365134]  r4:c0d4e71c
    [ 1830.367688] [<c012c3c8>] (irq_exit) from [<c0169d68>] (__handle_domain_irq+0x60/0xb0)
    [ 1830.375560] [<c0169d08>] (__handle_domain_irq) from [<c0101464>] (omap_intc_handle_irq+0x3c/0x98)
    [ 1830.384478]  r9:b624ffa0 r8:10c53c7d r7:10c5387d r6:ffffffff r5:40000010 r4:c0d834b0
    [ 1830.392260] [<c0101428>] (omap_intc_handle_irq) from [<c010ccd4>] (__irq_usr+0x54/0x80)
    [ 1830.400299] Exception stack(0xce0d9fb0 to 0xce0d9ff8)
    [ 1830.405378] 9fa0:                                     00000000 00000000 b62504e4 00000000
    [ 1830.413599] 9fc0: 00000076 b6f03980 00000000 00000000 00000000 b624ffa0 bea93854 b624fdbc
    [ 1830.421819] 9fe0: 00000000 b624fd5c b6b37dec b6a66bc0 40000010 ffffffff
    [ 1830.428462]  r5:40000010 r4:b6a66bc0
    
    The reproduce case:

    Uart 2 for console(115200) and uart0 uart1 uart3 uart4 for data transfer(3M) , and don't use DMA.

    The application sends(write) 64 bytes from uarts to the slave board every 200ms and  separate thread per uart. The slave board recv the command and return  some data randomly. One uart corresponds to a slave board. And the application use select and read to recv the return data. The slave board always responds to some data,  even if no data is sent from the uart.

    When I debug the application , I will  use ./app to run and ctrl+c to end the app. After repeating the operation a few times, the kernel dumps.

    Because of using the uart so frequently, I never  close the uart fd.   But if I don't connect the uart to the slave board, the kernel dump will not happen.

    If used SDK v05.00.00.15, there is no kenerl dump prints on the console.

    If used SDK v05.01.00.11, the kernel dump in the attach file.

  • Hi Paul,

    Paul.Wu said:
    If used SDK v05.00.00.15, there is no kenerl dump prints on the console.

    I am confused. In your earlier post, you mentioned you also have such issue with SDK v5.0.0.15. Can you please clarify?

  • I only catch the 5.0.0.15 kernel dump one time , please see the attach file.But more often than not,the console prints nothing.

    Aftern so many times debugging,  I think the system may hung-up if   multiple uart ports is opened  when a lot of data coming in from the uarts.

    I reproduce the problem and the probability is 2 percent.

    [  448.453994] INFO: rcu_preempt self-detected stall on CPU
    [  448.459389]  0-...: (2100 ticks this GP) idle=7ea/140000000000001/0 softirq=31421/31421 fqs=0 
    [  448.468038]   (t=2100 jiffies g=14732 c=14731 q=652)
    [  448.473039] rcu_preempt kthread starved for 2100 jiffies! g14732 c14731 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=0
    [  448.483694] rcu_preempt     I    0     8      2 0x00000000
    [  448.489210] Backtrace: 
    [  448.491702] [<c08458e0>] (__schedule) from [<c0846058>] (schedule+0x58/0xc4)
    [  448.498792]  r10:c0d14d60 r9:00000003 r8:c0d15400 r7:c0d15440 r6:cf05bee8 r5:c0d15440
    [  448.506655]  r4:ffffe000
    [  448.509214] [<c0846000>] (schedule) from [<c08498dc>] (schedule_timeout+0x184/0x264)
    [  448.516992]  r5:c0d15440 r4:000031ca
    [  448.520599] [<c0849758>] (schedule_timeout) from [<c017947c>] (rcu_gp_kthread+0x688/0xaa4)
    [  448.528906]  r8:00000001 r7:c0d14e1c r6:c0d15400 r5:ffffe000 r4:00000001
    [  448.535655] [<c0178df4>] (rcu_gp_kthread) from [<c0145bb8>] (kthread+0x15c/0x164)
    [  448.543170]  r7:cf05a000
    [  448.545732] [<c0145a5c>] (kthread) from [<c0107e08>] (ret_from_fork+0x14/0x2c)
    [  448.552993]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0145a5c
    [  448.560858]  r4:cf030680
    [  448.563411] Task dump for CPU 0:
    [  448.566654] kworker/u2:0    R  running task        0     5      2 0x00000002
    [  448.573761] Workqueue: events_unbound flush_to_ldisc
    [  448.578748] Backtrace: 
    [  448.581221] [<c010ba48>] (dump_backtrace) from [<c010bd2c>] (show_stack+0x18/0x1c)
    [  448.588828]  r7:c0d14d60 r6:c0d03100 r5:00000002 r4:cf041800
    [  448.594531] [<c010bd14>] (show_stack) from [<c014b614>] (sched_show_task.part.9+0xc4/0xdc)
    [  448.602844] [<c014b550>] (sched_show_task.part.9) from [<c014fe58>] (dump_cpu_task+0x38/0x3c)
    [  448.611408]  r5:00000000 r4:c0d14d60
    [  448.615008] [<c014fe20>] (dump_cpu_task) from [<c017bc5c>] (rcu_dump_cpu_stacks+0x90/0xd0)
    [  448.623318] [<c017bbcc>] (rcu_dump_cpu_stacks) from [<c017ab18>] (rcu_check_callbacks+0x5c4/0x8f0)
    [  448.632324]  r10:c0d030f4 r9:c0d03100 r8:c0d15400 r7:00000000 r6:c0d14d60 r5:c0d14d60
    [  448.640188]  r4:c0d151a8 r3:cf050000
    [  448.643793] [<c017a554>] (rcu_check_callbacks) from [<c017f488>] (update_process_times+0x3c/0x64)
    [  448.652712]  r10:c018ebf4 r9:00000000 r8:00000007 r7:00000068 r6:00000000 r5:cf041800
    [  448.660575]  r4:ffffe000
    [  448.663129] [<c017f44c>] (update_process_times) from [<c018ea14>] (tick_sched_handle+0x5c/0x60)
    [  448.671870]  r7:00000068 r6:69b40c76 r5:cf051eb0 r4:c0d16c80
    [  448.677559] [<c018e9b8>] (tick_sched_handle) from [<c018ec58>] (tick_sched_timer+0x64/0xd0)
    [  448.685958] [<c018ebf4>] (tick_sched_timer) from [<c0180088>] (__hrtimer_run_queues.constprop.5+0x150/0x1e8)
    [  448.695833]  r7:ffffe000 r6:c0d165c0 r5:c0d16c80 r4:c0d16580
    [  448.701528] [<c017ff38>] (__hrtimer_run_queues.constprop.5) from [<c01803f8>] (hrtimer_interrupt+0x9c/0x248)
    [  448.711405]  r10:c0d16638 r9:c0d16580 r8:ffffffff r7:7fffffff r6:00000003 r5:ffffe000
    [  448.719267]  r4:c0d16580
    [  448.721832] [<c018035c>] (hrtimer_interrupt) from [<c01190fc>] (omap2_gp_timer_interrupt+0x30/0x38)
    [  448.730925]  r10:c0d4ec40 r9:cf006200 r8:00000010 r7:cf051e14 r6:00000000 r5:cf006200
    [  448.738789]  r4:c0d07fb4
    [  448.741355] [<c01190cc>] (omap2_gp_timer_interrupt) from [<c016a3e0>] (__handle_irq_event_percpu+0x6c/0x140)
    [  448.751236] [<c016a374>] (__handle_irq_event_percpu) from [<c016a4d8>] (handle_irq_event_percpu+0x24/0x60)
    [  448.760939]  r10:c0d15400 r9:cf050000 r8:cf008000 r7:00000000 r6:00000000 r5:cf006200
    [  448.768802]  r4:cf006200
    [  448.771356] [<c016a4b4>] (handle_irq_event_percpu) from [<c016a578>] (handle_irq_event+0x64/0x90)
    [  448.780269]  r5:00000010 r4:cf006200
    [  448.783871] [<c016a514>] (handle_irq_event) from [<c016d960>] (handle_level_irq+0xb4/0x158)
    [  448.792259]  r5:00000010 r4:cf006200
    [  448.795858] [<c016d8ac>] (handle_level_irq) from [<c01695f0>] (generic_handle_irq+0x2c/0x3c)
    [  448.804335]  r5:00000010 r4:c0d4d6d0
    [  448.807934] [<c01695c4>] (generic_handle_irq) from [<c0169ca8>] (__handle_domain_irq+0x5c/0xb0)
    [  448.816679] [<c0169c4c>] (__handle_domain_irq) from [<c0101464>] (omap_intc_handle_irq+0x3c/0x98)
    [  448.825598]  r9:cf050000 r8:cf005000 r7:cf051ee4 r6:ffffffff r5:20070013 r4:c0d823f0
    [  448.833382] [<c0101428>] (omap_intc_handle_irq) from [<c010c8cc>] (__irq_svc+0x6c/0xa8)
    [  448.841422] Exception stack(0xcf051eb0 to 0xcf051ef8)
    [  448.846502] 1ea0:                                     cf02a274 00000002 cf005014 cf051f00
    [  448.854722] 1ec0: cf02a274 cf031200 cf004100 00000000 cf005000 00000000 c0d15400 cf051f34
    [  448.862942] 1ee0: cf051f00 cf051f00 c013fb44 c013fb4c 20070013 ffffffff
    [  448.869586]  r5:20070013 r4:c013fb4c
    [  448.873186] [<c013f97c>] (process_one_work) from [<c013fe44>] (worker_thread+0x84/0x680)
    [  448.881319]  r10:c0d15400 r9:00000088 r8:cf005014 r7:ffffe000 r6:cf031218 r5:cf005000
    [  448.889183]  r4:cf031200
    [  448.891734] [<c013fdc0>] (worker_thread) from [<c0145bb8>] (kthread+0x15c/0x164)
    [  448.899170]  r10:cf03fe94 r9:c013fdc0 r8:cf031200 r7:cf050000 r6:00000000 r5:cf030500
    [  448.907033]  r4:cf0305c0
    [  448.909585] [<c0145a5c>] (kthread) from [<c0107e08>] (ret_from_fork+0x14/0x2c)
    [  448.916846]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0145a5c
    [  448.924709]  r4:cf030500
    
     _____                    _____           _         _   
     |  _ [  494.633997] INFO: rcu_preempt self-detected stall on CPU
     [  494.639388]  0-...: (2100 ticks this GP) idle=7ea/140000000000001/0 softirq=31423/31423 fqs=0 
     [  494.648036]   (t=2100 jiffies g=14734 c=14733 q=1599)
     [  494.653124] rcu_preempt kthread starved for 2100 jiffies! g14734 c14733 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=0
     [  494.663779] rcu_preempt     I    0     8      2 0x00000000
     [  494.669295] Backtrace: 
     [  494.671785] [<c08458e0>] (__schedule) from [<c0846058>] (schedule+0x58/0xc4)
     [  494.678873]  r10:c0d14d60 r9:00000003 r8:c0d15400 r7:c0d15440 r6:cf05bee8 r5:c0d15440
     [  494.686737]  r4:ffffe000
     [  494.689295] [<c0846000>] (schedule) from [<c08498dc>] (schedule_timeout+0x184/0x264)
     [  494.697073]  r5:c0d15440 r4:000043d4
     [  494.700680] [<c0849758>] (schedule_timeout) from [<c017947c>] (rcu_gp_kthread+0x688/0xaa4)
     [  494.708987]  r8:00000001 r7:c0d14e1c r6:c0d15400 r5:ffffe000 r4:00000001
     [  494.715735] [<c0178df4>] (rcu_gp_kthread) from [<c0145bb8>] (kthread+0x15c/0x164)
     [  494.723251]  r7:cf05a000
     [  494.725812] [<c0145a5c>] (kthread) from [<c0107e08>] (ret_from_fork+0x14/0x2c)
     [  494.733073]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0145a5c
     [  494.740937]  r4:cf030680
     [  494.743491] Task dump for CPU 0:
     [  494.746733] uart split      R  running task        0  1163    173 0x00000002
     [  494.753820] Backtrace: 
     [  494.756296] [<c010ba48>] (dump_backtrace) from [<c010bd2c>] (show_stack+0x18/0x1c)
     [  494.763904]  r7:c0d14d60 r6:c0d03100 r5:000000ad r4:ce398c00
     [  494.769608] [<c010bd14>] (show_stack) from [<c014b614>] (sched_show_task.part.9+0xc4/0xdc)
     [  494.777919] [<c014b550>] (sched_show_task.part.9) from [<c014fe58>] (dump_cpu_task+0x38/0x3c)
     [  494.786482]  r5:00000000 r4:c0d14d60
     [  494.790081] [<c014fe20>] (dump_cpu_task) from [<c017bc5c>] (rcu_dump_cpu_stacks+0x90/0xd0)
     [  494.798390] [<c017bbcc>] (rcu_dump_cpu_stacks) from [<c017ab18>] (rcu_check_callbacks+0x5c4/0x8f0)
     [  494.807396]  r10:c0d030f4 r9:c0d03100 r8:c0d15400 r7:00000000 r6:c0d14d60 r5:c0d14d60
     [  494.815261]  r4:c0d151a8 r3:ce460000
     [  494.818865] [<c017a554>] (rcu_check_callbacks) from [<c017f488>] (update_process_times+0x3c/0x64)
     [  494.827783]  r10:c018ebf4 r9:00000000 r8:00000007 r7:00000073 r6:00000000 r5:ce398c00
     [  494.835647]  r4:ffffe000
     [  494.838202] [<c017f44c>] (update_process_times) from [<c018ea14>] (tick_sched_handle+0x5c/0x60)
     [  494.846944]  r7:00000073 r6:2a3eff63 r5:ce461dc8 r4:c0d16c80
     [  494.852633] [<c018e9b8>] (tick_sched_handle) from [<c018ec58>] (tick_sched_timer+0x64/0xd0)
     [  494.861031] [<c018ebf4>] (tick_sched_timer) from [<c0180088>] (__hrtimer_run_queues.constprop.5+0x150/0x1e8)
     [  494.870907]  r7:ffffe000 r6:c0d165c0 r5:c0d16c80 r4:c0d16580
     [  494.876601] [<c017ff38>] (__hrtimer_run_queues.constprop.5) from [<c01803f8>] (hrtimer_interrupt+0x9c/0x248)
     [  494.886479]  r10:c0d16638 r9:c0d16580 r8:ffffffff r7:7fffffff r6:00000003 r5:ffffe000
     [  494.894342]  r4:c0d16580
     [  494.896907] [<c018035c>] (hrtimer_interrupt) from [<c01190fc>] (omap2_gp_timer_interrupt+0x30/0x38)
     [  494.905999]  r10:c0d4ec40 r9:cf006200 r8:00000010 r7:ce461d2c r6:00000000 r5:cf006200
     [  494.913863]  r4:c0d07fb4
     [  494.916430] [<c01190cc>] (omap2_gp_timer_interrupt) from [<c016a3e0>] (__handle_irq_event_percpu+0x6c/0x140)
     [  494.926312] [<c016a374>] (__handle_irq_event_percpu) from [<c016a4d8>] (handle_irq_event_percpu+0x24/0x60)
     [  494.936016]  r10:cf4a855d r9:ce460000 r8:cf008000 r7:00000000 r6:00000000 r5:cf006200
     [  494.943879]  r4:cf006200
     [  494.946433] [<c016a4b4>] (handle_irq_event_percpu) from [<c016a578>] (handle_irq_event+0x64/0x90)
     [  494.955346]  r5:00000010 r4:cf006200
     [  494.958949] [<c016a514>] (handle_irq_event) from [<c016d960>] (handle_level_irq+0xb4/0x158)
     [  494.967338]  r5:00000010 r4:cf006200
     [  494.970937] [<c016d8ac>] (handle_level_irq) from [<c01695f0>] (generic_handle_irq+0x2c/0x3c)
     [  494.979413]  r5:00000010 r4:c0d4d6d0
     [  494.983012] [<c01695c4>] (generic_handle_irq) from [<c0169ca8>] (__handle_domain_irq+0x5c/0xb0)
     [  494.991758] [<c0169c4c>] (__handle_domain_irq) from [<c0101464>] (omap_intc_handle_irq+0x3c/0x98)
     [  495.000677]  r9:ce460000 r8:c0845b38 r7:ce461dfc r6:ffffffff r5:600b0113 r4:c0d823f0
     [  495.008462] [<c0101428>] (omap_intc_handle_irq) from [<c010c8cc>] (__irq_svc+0x6c/0xa8)
     [  495.016501] Exception stack(0xce461dc8 to 0xce461e10)
     [  495.021582] 1dc0:                   ce285200 00000002 00000000 ce461e18 00000000 00000000
     [  495.029803] 1de0: ce285200 ce398c00 c0845b38 c0846058 cf4a855d ce461e4c ce461e50 ce461e18
     [  495.038020] 1e00: c0845b38 c014bd10 600b0113 ffffffff
     [  495.043094]  r5:600b0113 r4:c014bd10
     [  495.046695] [<c014bcb0>] (finish_task_switch) from [<c0845b38>] (__schedule+0x258/0x720)
     [  495.054826]  r8:cf7c3e00 r7:ce20ea80 r6:ce285200 r5:ce398c00 r4:c0d0f970
     [  495.061563] [<c08458e0>] (__schedule) from [<c0846058>] (schedule+0x58/0xc4)
     [  495.068650]  r10:00000000 r9:00000000 r8:c0d76914 r7:00000001 r6:0000c350 r5:ffffe000
     [  495.076514]  r4:ffffe000
     [  495.079067] [<c0846000>] (schedule) from [<c0849b14>] (do_nanosleep+0xa8/0x1cc)
     [  495.086409]  r5:ffffe000 r4:ce461f18
     [  495.090009] [<c0849a6c>] (do_nanosleep) from [<c01807b8>] (hrtimer_nanosleep+0xe4/0x174)
     [  495.098142]  r9:00000000 r8:0000c350 r7:00000001 r6:0000c350 r5:00000000 r4:00100590
     [  495.105929] [<c01806d4>] (hrtimer_nanosleep) from [<c01808c8>] (SyS_nanosleep+0x80/0x98)
     [  495.114062]  r10:00000000 r9:ce460000 r8:c0107f04 r7:000000a2 r6:000003e8 r5:00000106
     [  495.121926]  r4:00000000
     [  495.124479] [<c0180848>] (SyS_nanosleep) from [<c0107d20>] (ret_fast_syscall+0x0/0x54)
    

  • Hi,

    This sounds like a different issue from LCPD-14446, does the issue happen If you decrease the baud rate?
    Is the hw flow control enabled on the 4 ports? I don't think you can avoid data loss if all the 4 ports in 3Mbps baud rate without hw flow control and DMA.
  • Maybe it's another issue.
    I must use the baudrate >= 3M and the hw flow control is disable.
  • Can I download the new kernel from git://git.ti.com/ti-linux-kernel/ti-linux-kernel.git what is fixed the bug LCPD-14446 and LCPD-14448 ?
  • The new kernel on git.ti.com very likely doesn't solve the issue you have, because we don't know its root cause.
    LCPD-14446 is irrelevant, since it is only happens when DMA is enabled on UART.
    And LCPD-14448 is irrelevant either, because it is about byte loss and only happens in rs-485.
  • Paul.Wu said:
    Maybe it's another issue.
    I must use the baudrate >= 3M and the hw flow control is disable.

    Again, does the issue happen If you decrease the baud rate?

    With hw flow control, you would have data loss on multiple uart ports with 3M baud, is it not an issue in your use case?

  • It's difficult to change the slave board's baud rate. So I don't known if the issue happen when decrease the baud rate.
    With 3M baud rate, the probability of data loss is very low and we don't care, at present.
  • OK, thanks for getting back to us. We'll go ahead and close this issue for now and you can always create a new post or reply back if needed.