• State
  • Replies 9 replies
  • Subscribers 100 subscribers
  • Views 184 views
  • Users 0 members are here
Support feedback
Options
Options
Related

AM57-LINUX-RT-SDK: AM5716 prueth driver cause linux system crash.

jiannan zhang
Prodigy 30 points
Part Number: AM57L-LINUX-RT-SDK
Other Parts Discussed in Thread: AM5716, AM62L

Hi,TI

  We are currently using the prueth driver's Ethernet port to send TCP or UDP protocol data. However, during communication, plugging or unplugging the Ethernet cable causes the system to crash.

The Linux kernel only responds to break signals. This issue occurs only in SDK version 06.03.00.106; the same hardware and application run without any problems in SDK version 08.02.01.00.

Since SDK version 06.03.00.106 has already been deployed in production, could this issue be fixed in this version instead of requiring an upgrade to a newer SDK version? 

Below is the output printed after sending the "w" command following a break signal when the system becomes unresponsive:

[  222.783511] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  222.783516] rcu:     (detected by 0, t=8407 jiffies, g=51273, q=29372)
[  222.783526] rcu: All QSes seen, last rcu_preempt kthread activity 8407 (-7722--16129), jiffies_till_next_fqs=1, root ->qsmask 0x0
[  222.783530] ktimersoftd/0   R  running task        0    10      2 0x00001000
[  222.783538] Backtrace:
[  222.783555] [<c020d164>] (dump_backtrace) from [<c020d4cc>] (show_stack+0x20/0x24)
[  222.783561]  r7:c0e1f594 r6:00000002 r5:c0e06888 r4:cb524000
[  222.783573] [<c020d4ac>] (show_stack) from [<c025730c>] (sched_show_task.part.2+0xec/0x108)
[  222.783581] [<c0257220>] (sched_show_task.part.2) from [<c0252044>] (sched_show_task+0x24/0x28)
[  222.783586]  r6:c0e1f1c4 r5:c0e1f838 r4:c0e1f1c4
[  222.783596] [<c0252020>] (sched_show_task) from [<c0287a0c>] (rcu_check_callbacks+0x934/0xa88)
[  222.783607] [<c02870d8>] (rcu_check_callbacks) from [<c028f1b0>] (update_process_times+0x44/0x74)
[  222.783612]  r10:c02a0e04 r9:ffffffff r8:c0e6a520 r7:00000033 r6:00000000 r5:cb524000
[  222.783614]  r4:ffffe000
[  222.783625] [<c028f16c>] (update_process_times) from [<c02a0af0>] (tick_sched_handle+0x3c/0x78)
[  222.783629]  r7:00000033 r6:deb90dd4 r5:cb529e40 r4:c0e22540
[  222.783638] [<c02a0ab4>] (tick_sched_handle) from [<c02a0e5c>] (tick_sched_timer+0x58/0xb4)
[  222.783641]  r5:cb529e40 r4:c0e22540
[  222.783650] [<c02a0e04>] (tick_sched_timer) from [<c0290138>] (__hrtimer_run_queues.constprop.3+0x118/0x410)
[  222.783654]  r7:ffffe000 r6:c0e21800 r5:c0e22540 r4:c0e21850
[  222.783662] [<c0290020>] (__hrtimer_run_queues.constprop.3) from [<c0290c74>] (hrtimer_interrupt+0x128/0x340)
[  222.783668]  r10:ffffe000 r9:ffffffff r8:7fffffff r7:00000003 r6:20070193 r5:ffffe000
[  222.783670]  r4:c0e21800
[  222.783679] [<c0290b4c>] (hrtimer_interrupt) from [<c06c748c>] (arch_timer_handler_phys+0x38/0x40)
[  222.783684]  r10:ffffe000 r9:fa213000 r8:cb420000 r7:c0e070a4 r6:00000013 r5:cb4ef200
[  222.783687]  r4:cb41f600
[  222.783696] [<c06c7454>] (arch_timer_handler_phys) from [<c027994c>] (handle_percpu_devid_irq+0x88/0x244)
[  222.783705] [<c02798c4>] (handle_percpu_devid_irq) from [<c0273c3c>] (generic_handle_irq+0x34/0x44)
[  222.783710]  r10:ffffe000 r9:fa213000 r8:cb420000 r7:00000000 r6:00000001 r5:00000000
[  222.783713]  r4:c0e65eac r3:c02798c4
[  222.783720] [<c0273c08>] (generic_handle_irq) from [<c0274544>] (__handle_domain_irq+0x64/0xb8)
[  222.783727] [<c02744e0>] (__handle_domain_irq) from [<c02021fc>] (gic_handle_irq+0x5c/0x90)
[  222.783732]  r9:fa213000 r8:cb529e40 r7:fa212000 r6:fa21200c r5:c0e43304 r4:c0e070a4
[  222.783737] [<c02021a0>] (gic_handle_irq) from [<c02019b8>] (__irq_svc+0x58/0xa0)
[  222.783741] Exception stack(0xcb529e40 to 0xcb529e88)
[  222.783746] 9e40: ca03f290 00000000 cb524000 00000003 ca03f200 00000000 00000000 ca03f290
[  222.783751] 9e60: 00000002 00000108 ffffe000 cb529e9c cb529e90 cb529e90 c080de1c c0285528
[  222.783754] 9e80: 20070013 ffffffff
[  222.783759]  r9:cb528000 r8:00000002 r7:cb529e74 r6:ffffffff r5:20070013 r4:c0285528
[  222.783772] [<c080ddb4>] (rt_spin_unlock) from [<c0717b34>] (net_tx_action+0x1bc/0x2c8)
[  222.783782] [<c0717978>] (net_tx_action) from [<c022e438>] (do_current_softirqs+0x1f0/0x3d0)
[  222.783788]  r10:ffffe000 r9:00000108 r8:00000002 r7:00000040 r6:04208140 r5:c0e16378
[  222.783790]  r4:00000008
[  222.783798] [<c022e248>] (do_current_softirqs) from [<c022e654>] (run_ksoftirqd+0x3c/0x5c)
[  222.783803]  r10:cb503de0 r9:00000000 r8:c0e06888 r7:c0e16024 r6:00000001 r5:ffffe000
[  222.783805]  r4:ffffe000
[  222.783815] [<c022e618>] (run_ksoftirqd) from [<c024ee50>] (smpboot_thread_fn+0x2c8/0x2f4)
[  222.783818]  r5:ffffe000 r4:cb4eff80
[  222.783825] [<c024eb88>] (smpboot_thread_fn) from [<c024a54c>] (kthread+0x164/0x16c)
[  222.783830]  r9:c024eb88 r8:cb4eff80 r7:cb528000 r6:00000000 r5:cb50a000 r4:cb50a040
[  222.783836] [<c024a3e8>] (kthread) from [<c02010b4>] (ret_from_fork+0x14/0x20)
[  222.783838] Exception stack(0xcb529fb0 to 0xcb529ff8)
[  222.783842] 9fa0:                                     00000000 00000000 00000000 00000000
[  222.783846] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  222.783850] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  222.783855]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c024a3e8
[  222.783857]  r4:cb50a000
[  222.783864] rcu: rcu_preempt kthread starved for 8407 jiffies! g51273 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
[  222.783866] rcu: RCU grace-period kthread stack dump:
[  222.783868] rcu_preempt     R  running task        0    11      2 0x00001000
[  222.783875] Backtrace:
[  222.783884] [<c0809048>] (__schedule) from [<c0809770>] (schedule+0x70/0x134)
[  222.783890]  r10:c0e206c0 r9:c0e1fa80 r8:00000000 r7:c0e06888 r6:cb52becc r5:ffffe000
[  222.783892]  r4:cb524700
[  222.783900] [<c0809700>] (schedule) from [<c080ca64>] (schedule_timeout+0x204/0x464)
[  222.783903]  r5:c0e206c0 r4:ffffc100
[  222.783912] [<c080c860>] (schedule_timeout) from [<c0286450>] (rcu_gp_kthread+0x624/0xcec)
[  222.783916]  r10:c0e69dab r9:c0e1f304 r8:00000001 r7:c0e1fa80 r6:cb52a000 r5:c0e1f1c4
[  222.783918]  r4:00000001
[  222.783925] [<c0285e2c>] (rcu_gp_kthread) from [<c024a54c>] (kthread+0x164/0x16c)
[  222.783927]  r7:cb52a000
[  222.783932] [<c024a3e8>] (kthread) from [<c02010b4>] (ret_from_fork+0x14/0x20)
[  222.783935] Exception stack(0xcb52bfb0 to 0xcb52bff8)
[  222.783938] bfa0:                                     00000000 00000000 00000000 00000000
[  222.783942] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  222.783946] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  222.783951]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c024a3e8
[  222.783953]  r4:cb50a000
[  224.665707] sysrq: SysRq : Show Blocked State
[  225.190516]   task                PC stack   pid father

  • 0
    TI__Guru** 103140 points

    Hello,

    Please clarify for me what processor you are using. AM62L is one processor that does not have a PRU subsystem. AM5716 is a different processor that does have PRU Ethernet.

    I am reassigning your thread as if this question is about AM57x.

    Regards,

    Nick

  • 0 in reply to Nick Saulnier
    Prodigy 30 points

    yes, it's AM5716, not AM62L

  • 0 in reply to Nick Saulnier
    Prodigy 30 points

    yes, is AM5716,Here is another system crash, no action on uart and ethernet. only sysreq+h output from uart


    14:10:55.796# root@am571x-evm:~# [ 160.244357] prueth pruss2_eth eth1: Link is Up - 100Mbps/Full - flow control off
    14:10:55.895# 1999-04-07T12:52:00.101Z: Cmp=CmpRouter, Class=1, Error=0, Info=5, pszInfo= Network interface <interface>ether 2</interface> unregistered
    14:11:26.735# [ 185.413513] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
    14:11:26.740# [ 185.413518] rcu: (detected by 0, t=2102 jiffies, g=61721, q=8214)
    14:11:26.752# [ 185.413530] rcu: All QSes seen, last rcu_preempt kthread activity 2102 (-11459--13561), jiffies_till_next_fqs=1, root ->qsmask 0x0
    14:11:26.760# [ 185.413534] ktimersoftd/0 R running task 0 10 2 0x00001000
    14:11:26.762# [ 185.413542] Backtrace:
    14:11:26.770# [ 185.413560] [<c020d164>] (dump_backtrace) from [<c020d4cc>] (show_stack+0x20/0x24)
    14:11:26.777# [ 185.413566] r7:c0e1f594 r6:00000002 r5:c0e06888 r4:cb524000
    14:11:26.784# [ 185.413581] [<c020d4ac>] (show_stack) from [<c025730c>] (sched_show_task.part.2+0xec/0x108)
    14:11:26.792# [ 185.413589] [<c0257220>] (sched_show_task.part.2) from [<c0252044>] (sched_show_task+0x24/0x28)
    14:11:26.798# [ 185.413594] r6:c0e1f1c4 r5:c0e1f838 r4:c0e1f1c4
    14:11:26.807# [ 185.413607] [<c0252020>] (sched_show_task) from [<c0287a0c>] (rcu_check_callbacks+0x934/0xa88)
    14:11:26.831# [ 185.413618] [<c02870d8>] (rcu_check_callbacks) from [<c028f1b0>] (update_process_times+0x44/0x74)
    14:11:26.831# [ 185.413625] r10:c02a0e04 r9:ffffffff r8:c0e6a520 r7:0000002b r6:00000000 r5:cb524000
    14:11:26.831# [ 185.413627] r4:ffffe000
    14:11:26.835# [ 185.413638] [<c028f16c>] (update_process_times) from [<c02a0af0>] (tick_sched_handle+0x3c/0x78)
    14:11:26.841# [ 185.413643] r7:0000002b r6:2b4c1cbc r5:cb529e00 r4:c0e22540
    14:11:26.849# [ 185.413651] [<c02a0ab4>] (tick_sched_handle) from [<c02a0e5c>] (tick_sched_timer+0x58/0xb4)
    14:11:26.853# [ 185.413655] r5:cb529e00 r4:c0e22540
    14:11:26.863# [ 185.413664] [<c02a0e04>] (tick_sched_timer) from [<c0290138>] (__hrtimer_run_queues.constprop.3+0x118/0x410)
    14:11:26.868# [ 185.413669] r7:ffffe000 r6:c0e21800 r5:c0e22540 r4:c0e21850
    14:11:26.877# [ 185.413678] [<c0290020>] (__hrtimer_run_queues.constprop.3) from [<c0290c74>] (hrtimer_interrupt+0x128/0x340)
    14:11:26.886# [ 185.413683] r10:ffffe000 r9:ffffffff r8:7fffffff r7:00000003 r6:200d0193 r5:ffffe000
    14:11:26.889# [ 185.413685] r4:c0e21800
    14:11:26.897# [ 185.413696] [<c0290b4c>] (hrtimer_interrupt) from [<c06c748c>] (arch_timer_handler_phys+0x38/0x40)
    14:11:26.905# [ 185.413702] r10:ffffe000 r9:fa213000 r8:cb420000 r7:c0e070a4 r6:00000013 r5:cb4ef200
    14:11:26.908# [ 185.413704] r4:cb41f600
    14:11:26.918# [ 185.413713] [<c06c7454>] (arch_timer_handler_phys) from [<c027994c>] (handle_percpu_devid_irq+0x88/0x244)
    14:11:26.926# [ 185.413722] [<c02798c4>] (handle_percpu_devid_irq) from [<c0273c3c>] (generic_handle_irq+0x34/0x44)
    14:11:26.934# [ 185.413728] r10:ffffe000 r9:fa213000 r8:cb420000 r7:00000000 r6:00000001 r5:00000000
    14:11:26.939# [ 185.413730] r4:c0e65eac r3:c02798c4
    14:11:26.947# [ 185.413738] [<c0273c08>] (generic_handle_irq) from [<c0274544>] (__handle_domain_irq+0x64/0xb8)
    14:11:26.957# [ 185.413745] [<c02744e0>] (__handle_domain_irq) from [<c02021fc>] (gic_handle_irq+0x5c/0x90)
    14:11:26.964# [ 185.413750] r9:fa213000 r8:cb529e00 r7:fa212000 r6:fa21200c r5:c0e43304 r4:c0e070a4
    14:11:26.971# [ 185.413756] [<c02021a0>] (gic_handle_irq) from [<c02019b8>] (__irq_svc+0x58/0xa0)
    14:11:26.976# [ 185.413760] Exception stack(0xcb529e00 to 0xcb529e48)
    14:11:26.985# [ 185.413766] 9e00: ca1d006c 00000000 cb524000 00000000 ca1d0000 cacda3b0 ca1d006c caf20d80
    14:11:26.991# [ 185.413771] 9e20: 00000002 00000108 ffffe000 cb529e9c 00000000 cb529e50 c080dc24 c074fe20
    14:11:26.995# [ 185.413774] 9e40: 600d0013 ffffffff
    14:11:27.003# [ 185.413780] r9:cb528000 r8:00000002 r7:cb529e34 r6:ffffffff r5:600d0013 r4:c074fe20
    14:11:27.011# [ 185.413791] [<c074f83c>] (__qdisc_run) from [<c0717b20>] (net_tx_action+0x1a8/0x2c8)
    14:11:27.018# [ 185.413797] r10:ffffe000 r9:00000108 r8:00000002 r7:ca1d0090 r6:00000000 r5:00000000
    14:11:27.020# [ 185.413799] r4:ca1d0000
    14:11:27.033# [ 185.413810] [<c0717978>] (net_tx_action) from [<c022e438>] (do_current_softirqs+0x1f0/0x3d0)
    14:11:27.038# [ 185.413816] r10:ffffe000 r9:00000108 r8:00000002 r7:00000040 r6:04208140 r5:c0e16378
    14:11:27.041# [ 185.413818] r4:00000008
    14:11:27.049# [ 185.413827] [<c022e248>] (do_current_softirqs) from [<c022e654>] (run_ksoftirqd+0x3c/0x5c)
    14:11:27.056# [ 185.413832] r10:cb503de0 r9:00000000 r8:c0e06888 r7:c0e16024 r6:00000001 r5:ffffe000
    14:11:27.059# [ 185.413834] r4:ffffe000
    14:11:27.067# [ 185.413844] [<c022e618>] (run_ksoftirqd) from [<c024ee50>] (smpboot_thread_fn+0x2c8/0x2f4)
    14:11:27.070# [ 185.413847] r5:ffffe000 r4:cb4eff80
    14:11:27.078# [ 185.413854] [<c024eb88>] (smpboot_thread_fn) from [<c024a54c>] (kthread+0x164/0x16c)
    14:11:27.086# [ 185.413860] r9:c024eb88 r8:cb4eff80 r7:cb528000 r6:00000000 r5:cb50a000 r4:cb50a040
    14:11:27.094# [ 185.413866] [<c024a3e8>] (kthread) from [<c02010b4>] (ret_from_fork+0x14/0x20)
    14:11:27.100# [ 185.413868] Exception stack(0xcb529fb0 to 0xcb529ff8)
    14:11:27.107# [ 185.413872] 9fa0: 00000000 00000000 00000000 00000000
    14:11:27.115# [ 185.413877] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    14:11:27.121# [ 185.413881] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
    14:11:27.129# [ 185.413886] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c024a3e8
    14:11:27.133# [ 185.413888] r4:cb50a000
    14:11:27.143# [ 185.413895] rcu: rcu_preempt kthread starved for 2102 jiffies! g61721 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
    14:11:27.148# [ 185.413897] rcu: RCU grace-period kthread stack dump:
    14:11:27.154# [ 185.413899] rcu_preempt R running task 0 11 2 0x00001000
    14:11:27.158# [ 185.413906] Backtrace:
    14:11:27.164# [ 185.413917] [<c0809048>] (__schedule) from [<c0809770>] (schedule+0x70/0x134)
    14:11:27.172# [ 185.413924] r10:c0e206c0 r9:c0e1fa80 r8:00000000 r7:c0e06888 r6:cb52becc r5:ffffe000
    14:11:27.176# [ 185.413926] r4:cb524700
    14:11:27.185# [ 185.413934] [<c0809700>] (schedule) from [<c080ca64>] (schedule_timeout+0x204/0x464)
    14:11:27.187# [ 185.413937] r5:c0e206c0 r4:ffffcb08
    14:11:27.196# [ 185.413947] [<c080c860>] (schedule_timeout) from [<c0286450>] (rcu_gp_kthread+0x624/0xcec)
    14:11:27.203# [ 185.413952] r10:c0e69dab r9:c0e1f304 r8:00000001 r7:c0e1fa80 r6:cb52a000 r5:c0e1f1c4
    14:11:27.205# [ 185.413954] r4:00000001
    14:11:27.213# [ 185.413960] [<c0285e2c>] (rcu_gp_kthread) from [<c024a54c>] (kthread+0x164/0x16c)
    14:11:27.215# [ 185.413963] r7:cb52a000
    14:11:27.223# [ 185.413968] [<c024a3e8>] (kthread) from [<c02010b4>] (ret_from_fork+0x14/0x20)
    14:11:27.227# [ 185.413971] Exception stack(0xcb52bfb0 to 0xcb52bff8)
    14:11:27.236# [ 185.413975] bfa0: 00000000 00000000 00000000 00000000
    14:11:27.244# [ 185.413979] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    14:11:27.250# [ 185.413983] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
    14:11:27.259# [ 185.413988] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c024a3e8
    14:11:27.261# [ 185.413990] r4:cb50a000
    14:11:27.269# [ 188.003513] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ktimersoftd/0:10]
    14:11:27.276# [ 188.003515] Modules linked in: xhci_plat_hcd xhci_hcd dwc3 udc_core dwc3_omap
    14:11:27.283# [ 188.003536] CPU: 0 PID: 10 Comm: ktimersoftd/0 Not tainted 4.19.94-rt39-ga242ccf3f1 #111
    14:11:27.290# [ 188.003538] Hardware name: Generic DRA72X (Flattened Device Tree)
    14:11:27.294# [ 188.003548] PC is at sch_direct_xmit+0x208/0x2c0
    14:11:27.299# [ 188.003558] LR is at rt_spin_lock+0x40/0x80
    14:11:27.306# [ 188.003562] pc : [<c074f784>] lr : [<c080dc24>] psr: a0070013
    14:11:27.311# [ 188.003565] sp : cb529e10 ip : 00000000 fp : cb529e4c
    14:11:27.316# [ 188.003567] r10: 00000000 r9 : caa38000 r8 : caf20d80
    14:11:27.322# [ 188.003570] r7 : ca1d006c r6 : c0e06888 r5 : ca1d0000 r4 : cacda3b0
    14:11:27.329# [ 188.003573] r3 : ca1d0068 r2 : cb524000 r1 : 00000000 r0 : ca1d0040
    14:11:27.336# [ 188.003578] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
    14:11:27.342# [ 188.003581] Control: 30c53c7d Table: 886df380 DAC: fffffffd
    14:11:27.351# [ 188.003586] CPU: 0 PID: 10 Comm: ktimersoftd/0 Not tainted 4.19.94-rt39-ga242ccf3f1 #111
    14:11:27.356# [ 188.003589] Hardware name: Generic DRA72X (Flattened Device Tree)
    14:11:27.359# [ 188.003590] Backtrace:
    14:11:27.365# [ 188.003601] [<c020d164>] (dump_backtrace) from [<c020d4cc>] (show_stack+0x20/0x24)
    14:11:27.372# [ 188.003607] r7:00000000 r6:cb528000 r5:c0e069f0 r4:c0e25368
    14:11:27.380# [ 188.003614] [<c020d4ac>] (show_stack) from [<c07f2fe8>] (dump_stack+0x24/0x28)
    14:11:27.386# [ 188.003623] [<c07f2fc4>] (dump_stack) from [<c020967c>] (show_regs+0x1c/0x20)
    14:11:27.397# [ 188.003636] [<c0209660>] (show_regs) from [<c02bfe58>] (watchdog_timer_fn+0x1e8/0x220)
    14:11:27.403# [ 188.003646] [<c02bfc70>] (watchdog_timer_fn) from [<c0290138>] (__hrtimer_run_queues.constprop.3+0x118/0x410)
    14:11:27.411# [ 188.003652] r10:c02bfc70 r9:ffffffff r8:c0e6a520 r7:ffffe000 r6:c0e21800 r5:c0e253a0
    14:11:27.415# [ 188.003655] r4:c0e21850
    14:11:27.425# [ 188.003663] [<c0290020>] (__hrtimer_run_queues.constprop.3) from [<c0290c74>] (hrtimer_interrupt+0x128/0x340)
    14:11:27.433# [ 188.003669] r10:00000000 r9:ffffffff r8:7fffffff r7:00000003 r6:20070193 r5:ffffe000
    14:11:27.436# [ 188.003671] r4:c0e21800
    14:11:27.443# [ 188.003680] [<c0290b4c>] (hrtimer_interrupt) from [<c06c748c>] (arch_timer_handler_phys+0x38/0x40)
    14:11:27.452# [ 188.003686] r10:00000000 r9:fa213000 r8:cb420000 r7:c0e070a4 r6:00000013 r5:cb4ef200
    14:11:27.454# [ 188.003688] r4:cb41f600
    14:11:27.464# [ 188.003698] [<c06c7454>] (arch_timer_handler_phys) from [<c027994c>] (handle_percpu_devid_irq+0x88/0x244)
    14:11:27.475# [ 188.003706] [<c02798c4>] (handle_percpu_devid_irq) from [<c0273c3c>] (generic_handle_irq+0x34/0x44)
    14:11:27.481# [ 188.003711] r10:00000000 r9:fa213000 r8:cb420000 r7:00000000 r6:00000001 r5:00000000
    14:11:27.486# [ 188.003714] r4:c0e65eac r3:c02798c4
    14:11:27.493# [ 188.003721] [<c0273c08>] (generic_handle_irq) from [<c0274544>] (__handle_domain_irq+0x64/0xb8)
    14:11:27.502# [ 188.003727] [<c02744e0>] (__handle_domain_irq) from [<c02021fc>] (gic_handle_irq+0x5c/0x90)
    14:11:27.510# [ 188.003732] r9:fa213000 r8:cb529dc0 r7:fa212000 r6:fa21200c r5:c0e43304 r4:c0e070a4
    14:11:27.519# [ 188.003738] [<c02021a0>] (gic_handle_irq) from [<c02019b8>] (__irq_svc+0x58/0xa0)
    14:11:27.521# [ 188.003741] Exception stack(0xcb529dc0 to 0xcb529e08)
    14:11:27.622# [ 188.003746] 9dc0: ca1d0040 00000000 cb524000 ca1d0068 cacda3b0 ca1d0000 c0e06888 ca1d006c
    14:11:27.622# [ 188.003751] 9de0: caf20d80 caa38000 00000000 cb529e4c 00000000 cb529e10 c080dc24 c074f784
    14:11:27.622# [ 188.003753] 9e00: a0070013 ffffffff
    14:11:27.622# [ 188.003759] r9:cb528000 r8:caf20d80 r7:cb529df4 r6:ffffffff r5:a0070013 r4:c074f784
    14:11:27.622# [ 188.003766] [<c074f57c>] (sch_direct_xmit) from [<c074f9a0>] (__qdisc_run+0x164/0x5ec)
    14:11:27.622# [ 188.003771] r10:ffffe000 r9:00000108 r8:00000000 r7:caf20d80 r6:00000001 r5:caf20d80
    14:11:27.622# [ 188.003773] r4:ca1d0000
    14:11:27.622# [ 188.003783] [<c074f83c>] (__qdisc_run) from [<c0717b20>] (net_tx_action+0x1a8/0x2c8)
    14:11:27.622# [ 188.003788] r10:ffffe000 r9:00000108 r8:00000002 r7:ca1d0090 r6:00000000 r5:00000000
    14:11:27.623# [ 188.003790] r4:ca1d0000
    14:11:27.623# [ 188.003800] [<c0717978>] (net_tx_action) from [<c022e438>] (do_current_softirqs+0x1f0/0x3d0)
    14:11:27.623# [ 188.003806] r10:ffffe000 r9:00000108 r8:00000002 r7:00000040 r6:04208140 r5:c0e16378
    14:11:27.623# [ 188.003807] r4:00000008
    14:11:27.623# [ 188.003816] [<c022e248>] (do_current_softirqs) from [<c022e654>] (run_ksoftirqd+0x3c/0x5c)
    14:11:27.623# [ 188.003821] r10:cb503de0 r9:00000000 r8:c0e06888 r7:c0e16024 r6:00000001 r5:ffffe000
    14:11:27.625# [ 188.003823] r4:ffffe000
    14:11:27.633# [ 188.003835] [<c022e618>] (run_ksoftirqd) from [<c024ee50>] (smpboot_thread_fn+0x2c8/0x2f4)
    14:11:27.638# [ 188.003838] r5:ffffe000 r4:cb4eff80
    14:11:27.645# [ 188.003845] [<c024eb88>] (smpboot_thread_fn) from [<c024a54c>] (kthread+0x164/0x16c)
    14:11:27.653# [ 188.003850] r9:c024eb88 r8:cb4eff80 r7:cb528000 r6:00000000 r5:cb50a000 r4:cb50a040
    14:11:27.659# [ 188.003855] [<c024a3e8>] (kthread) from [<c02010b4>] (ret_from_fork+0x14/0x20)
    14:11:27.665# [ 188.003858] Exception stack(0xcb529fb0 to 0xcb529ff8)
    14:11:27.672# [ 188.003861] 9fa0: 00000000 00000000 00000000 00000000
    14:11:27.680# [ 188.003866] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    14:11:27.688# [ 188.003870] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
    14:11:27.695# [ 188.003874] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c024a3e8
    14:11:27.697# [ 188.003876] r4:cb50a000
    14:11:27.715# [ 191.185775] sysrq: SysRq : HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) dump-ftrace-buffer(z)

  • 0 in reply to Nick Saulnier
    Prodigy 30 points

    The SDK is PROCESSOR-SDK-LINUX-RT-AM57X 06.03.00.106

  • 0 in reply to xie wengang
    TI__Mastermind 33115 points

    Hello Xie,

    This SDK is no longer supported. Our official recommendation would be to move to one of the newer SDKs. I can refer you to one of our third party partners to assess your issue on the SDK you have in production.

    This release was very stable and I do not believe there was such a bug like this that caused a crash when a ethernet port was disconnected.

    Is your design a custom board? 

    Was there any changes done on top of the default SDK?

    -Josue

  • 0 in reply to Josue Zamitiz-Ayala
    Prodigy 30 points

    This is easily reproducible. Run iperf as client, iper3 -c 192.168.xx.xx -i 1 -u -t 10000 connect to ipef server use UDP,  and keep plugging and unplugging the network cable, it will appear.

  • 0 in reply to xie wengang
    TI__Mastermind 33115 points

    Xie,

    I believe you. Unfortunately this SDK has been out of scope of support for many years now. Our official suggestion is to migrate to the SDK that has the issue fixed. 

    You could also back port this yourself or I can help you get in contact with one of our third parties to do the backport for you.

    Best,

    Josue

  • 0 in reply to Josue Zamitiz-Ayala
    TI__Mastermind 29362 points

    Hi Josue,

    Customer tried to upgrade to SDK9.3, but the new SDK real time performance is fairly poor compare to SDK6.03, it reduced their product overall performance. 

    Is the RT Linux real time performance degradation was noticed by BU? Is it improved in SDK 11. 

    BTW, I can reproduce similar issue with SDK6.03 on IDK:

    manually plug/unplug eth2 several times, the port hang doesn't detect cable connection any more.

    root@am57xx-evm:/home#
root@am57xx-evm:/home# [18089.764226] prueth pruss2_eth eth2: Link is Down
[18093.924448] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
[18127.204240] prueth pruss2_eth eth2: Link is Down
[18130.324452] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
[18134.484212] prueth pruss2_eth eth2: Link is Down
[18136.564447] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
[18143.844238] prueth pruss2_eth eth2: Link is Down
[18146.964434] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
[18152.164240] prueth pruss2_eth eth2: Link is Down
[18155.284446] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
[18158.404222] prueth pruss2_eth eth2: Link is Down
[18160.484457] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
[18166.724225] prueth pruss2_eth eth2: Link is Down
[18169.844450] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
[18172.964245] prueth pruss2_eth eth2: Link is Down
[18176.084461] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
[18179.204212] prueth pruss2_eth eth2: Link is Down
[18181.284441] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
[18188.564227] prueth pruss2_eth eth2: Link is Down
[18191.684448] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
[18193.764214] prueth pruss2_eth eth2: Link is Down
[18194.804437] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
[18198.964227] prueth pruss2_eth eth2: Link is Down
[18203.124438] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off
[18209.044253] prueth pruss2_eth eth2: Link is Down
[18212.164481] prueth pruss2_eth eth2: Link is Up - 100Mbps/Full - flow control off

root@am57xx-evm:/home#
root@am57xx-evm:/home#
root@am57xx-evm:/home# ifconfig
eth0      Link encap:Ethernet  HWaddr 94:E3:6D:36:07:66
          inet addr:10.85.130.191  Bcast:10.85.130.255  Mask:255.255.255.0
          inet6 addr: fe80::96e3:6dff:fe36:766/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:41375771 errors:0 dropped:0 overruns:0 frame:0
          TX packets:358993247 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2900690851 (2.7 GiB)  TX bytes:3981657355 (3.7 GiB)
          Interrupt:86

eth1      Link encap:Ethernet  HWaddr 94:E3:6D:36:07:67
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth2      Link encap:Ethernet  HWaddr 70:FF:76:1C:2C:13
          inet6 addr: fe80::72ff:76ff:fe1c:2c13/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:12685 errors:0 dropped:843 overruns:0 frame:0
          TX packets:41441 errors:0 dropped:49 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1957655 (1.8 MiB)  TX bytes:6691990 (6.3 MiB)

eth3      Link encap:Ethernet  HWaddr 70:FF:76:1C:2C:14
          inet6 addr: fe80::72ff:76ff:fe1c:2c14/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:24 errors:0 dropped:0 overruns:0 frame:0
          TX packets:45 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2870 (2.8 KiB)  TX bytes:8316 (8.1 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:577256 errors:0 dropped:0 overruns:0 frame:0
          TX packets:577256 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:598533684 (570.8 MiB)  TX bytes:598533684 (570.8 MiB)

root@am57xx-evm:/home# ifconfig eth2 down

^C

    log with putty console to check cpu loading which is 100% and can't kill the thread.

    root@am57xx-evm:/home# top
top - 07:31:23 up  5:15,  3 users,  load average: 17.88, 15.30, 10.38
Tasks: 188 total,  10 running, 178 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us, 10.9 sy,  0.0 ni, 49.4 id,  0.0 wa,  0.0 hi, 39.5 si,  0.0 st
MiB Mem :   1470.8 total,   1038.7 free,    142.2 used,    289.9 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   1265.1 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   23 root      -2   0       0      0      0 R 100.0   0.0  16:19.97 ktimersoftd/1
 9244 root      20   0    3020   1976   1504 R   1.0   0.1   0:02.35 top
   94 root     -51   0       0      0      0 S   0.3   0.0  13:31.86 irq/86-48484000
 9046 root      20   0       0      0      0 I   0.3   0.0   0:00.27 kworker/u4:2-flush-+
    1 root      20   0   32684   5556   3900 S   0.0   0.4   0:12.42 systemd

    up/down eth2 with script can't trigger error/crash

    #!/bin/bash
a=1000
 
until [ $a -le 0 ]
do
ifconfig eth0 down
sleep 1
ifconfig eth0 up
sleep 6
echo "a = " $a
((a--))
done

  • 0 in reply to Tony Tang
    TI__Mastermind 33115 points

    Hello Tony,

    Unfortunately since this is outside of our support period, the only support we can offer at this moment is the one mentioned prior.

    I am unaware of any particular RT Linux performance degradation and I would have to interface with our Dev team to confirm.

    I currently do not have bandwidth to test anything outside of our standard support.

    Best,

    Josue