AM625: rcu_preempt self-detected stall on CPU error during runtime

Veysel Oguz

Part Number: AM625

Tool/software:

Hi,

We are using AM625 with Processor SDK 09.01.00.08. We are observing hang & reset during runtime multiple times caused by rcu stall on CPU

rcu: INFO: rcu_preempt self-detected stall on CPU
[2024-09-19 09:32:58.595] [30531.965487] rcu: 	0-....: (115351 ticks this GP) idle=ce0c/1/0x4000000000000000 softirq=4495200/4495202 fqs=43318
[2024-09-19 09:32:58.595] [30531.975732] 	(t=115571 jiffies g=9036749 q=19802 ncpus=2)
[2024-09-19 09:32:58.595] [30531.981122] CPU: 0 PID: 62393 Comm: python3 Tainted: G        W  O       6.1.46-g247b2535b2 #1
[2024-09-19 09:32:58.595] [30531.989716] Hardware name: Texas Instruments AM625 SK (DT)
[2024-09-19 09:32:58.595] [30531.995187] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[2024-09-19 09:32:58.595] [30532.002133] pc : _stext+0xa4/0x2a4
[2024-09-19 09:32:58.595] [30532.005533] lr : _stext+0x6c/0x2a4

Below are the logs that are collected on different devices, each have rcu-stall issue. Please check.

Ä±Â¡1Ä±[ 3364.131838] omap_i2c 20010000.i2c: controller timed out
[ 3364.263807] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 3364.269928] rcu:     0-....: (8 ticks this GP) idle=99e4/0/0x3 softirq=152687/152688 fqs=34110
[ 3364.278268]  (detected by 1, t=68272 jiffies, g=297909, q=9923 ncpus=2)
[ 3364.284870] Task dump for CPU 0:
[ 3364.288089] task:swapper/0       state:R  running task     stack:0     pid:0     ppid:0      flags:0x0000000a
[ 3364.297990] Call trace:
[ 3364.300426]  __switch_to+0xd4/0x130
[ 3364.303920]  linux_banner+0x1c0390/0x257e08
[ 3365.667858] omap_i2c 20010000.i2c: controller timed out
[ 3367.203844] omap_i2c 20010000.i2c: controller timed out
[ 3368.739838] omap_i2c 20010000.i2c: controller timed out
[ 3370.275837] omap_i2c 20010000.i2c: controller timed out
[ 3371.811840] omap_i2c 20010000.i2c: controller timed out
[ 3373.347844] omap_i2c 20010000.i2c: controller timed out
[ 3374.883852] omap_i2c 20010000.i2c: controller timed out
[ 3376.419844] omap_i2c 20010000.i2c: controller timed out
[ 3377.475835] omap_i2c 20010000.i2c: controller timed out
[ 3378.531837] omap_i2c 20010000.i2c: controller timed out
[ 3380.067836] omap_i2c 20010000.i2c: controller timed out
[ 3381.603836] omap_i2c 20010000.i2c: controller timed out
[ 3383.139840] omap_i2c 20010000.i2c: controller timed out
[ 3384.675836] omap_i2c 20010000.i2c: controller timed out
[ 3386.211837] omap_i2c 20010000.i2c: controller timed out
[ 3387.747837] omap_i2c 20010000.i2c: controller timed out
[ 3389.283837] omap_i2c 20010000.i2c: controller timed out
[ 3390.819835] omap_i2c 20010000.i2c: controller timed out
[ 3392.355838] omap_i2c 20010000.i2c: controller timed out
[ 3393.891836] omap_i2c 20010000.i2c: controller timed out
[ 3395.427840] omap_i2c 20010000.i2c: controller timed out
[ 3396.483836] omap_i2c 20010000.i2c: controller timed out
[ 3397.539832] omap_i2c 20010000.i2c: controller timed out
[ 3399.075836] omap_i2c 20010000.i2c: controller timed out
[ 3400.611832] omap_i2c 20010000.i2c: controller timed out
[ 3402.147838] omap_i2c 20010000.i2c: controller timed out
[ 3403.683839] omap_i2c 20010000.i2c: controller timed out
[ 3405.219838] omap_i2c 20010000.i2c: controller timed out
[ 3406.755830] omap_i2c 20010000.i2c: controller timed out
[ 3408.291835] omap_i2c 20010000.i2c: controller timed out
[ 3409.827831] omap_i2c 20010000.i2c: controller timed out
[ 3411.363835] omap_i2c 20010000.i2c: controller timed out
[ 3412.899843] omap_i2c 20010000.i2c: controller timed out

[2024-09-19 09:32:58.584] [30526.951751] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:350704 dql_avail:-244 free_desc:510
[2024-09-19 09:32:58.595] [30529.283759] omap_i2c 20010000.i2c: controller timed out
[2024-09-19 09:32:58.595] [30531.815748] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:355568 dql_avail:-244 free_desc:510
[2024-09-19 09:32:58.595] [30531.959722] rcu: INFO: rcu_preempt self-detected stall on CPU
[2024-09-19 09:32:58.595] [30531.965487] rcu: 	0-....: (115351 ticks this GP) idle=ce0c/1/0x4000000000000000 softirq=4495200/4495202 fqs=43318
[2024-09-19 09:32:58.595] [30531.975732] 	(t=115571 jiffies g=9036749 q=19802 ncpus=2)
[2024-09-19 09:32:58.595] [30531.981122] CPU: 0 PID: 62393 Comm: python3 Tainted: G        W  O       6.1.46-g247b2535b2 #1
[2024-09-19 09:32:58.595] [30531.989716] Hardware name: Texas Instruments AM625 SK (DT)
[2024-09-19 09:32:58.595] [30531.995187] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[2024-09-19 09:32:58.595] [30532.002133] pc : _stext+0xa4/0x2a4
[2024-09-19 09:32:58.595] [30532.005533] lr : _stext+0x6c/0x2a4
[2024-09-19 09:32:58.595] [30532.008924] sp : ffff800008003f60
[2024-09-19 09:32:58.595] [30532.012225] x29: ffff800008003f60 x28: ffff000006b863c0 x27: 0000000000000200
[2024-09-19 09:32:58.595] [30532.019357] x26: 0000000000000000 x25: 0000000000000000 x24: ffff80000914d000
[2024-09-19 09:32:58.595] [30532.026479] x23: 0000000020000000 x22: ffff800009153000 x21: ffff80000914d9c8
[2024-09-19 09:32:58.610] [30532.033604] x20: ffff8000092b60c0 x19: ffff000006b863c0 x18: 0000000000000000
[2024-09-19 09:32:58.610] [30532.040730] x17: ffff80002ec4b000 x16: ffff800008000000 x15: 0000a9e5807298e4
[2024-09-19 09:32:58.610] [30532.047857] x14: 01fa4fb86496916c x13: 0000000000000001 x12: 0000000000000004
[2024-09-19 09:32:58.610] [30532.054983] x11: 0000000000000001 x10: 00000000000001ab x9 : ffff8000092b9d38
[2024-09-19 09:32:58.610] [30532.062107] x8 : 00001b59186f8100 x7 : 7fffffffffffffff x6 : 000004cd02f58c8a
[2024-09-19 09:32:58.610] [30532.069230] x5 : 03ffffffffffffff x4 : 0000000000000015 x3 : ffff000006b863c0
[2024-09-19 09:32:58.610] [30532.076356] x2 : ffff80002ec4b000 x1 : 00000000000000e0 x0 : ffff800009153b00
[2024-09-19 09:32:58.610] [30532.083485] Call trace:
[2024-09-19 09:32:58.610] [30532.085921]  _stext+0xa4/0x2a4
[2024-09-19 09:32:58.610] [30532.088965]  ____do_softirq+0x10/0x20
[2024-09-19 09:32:58.610] [30532.092618]  call_on_irq_stack+0x24/0x4c
[2024-09-19 09:32:58.610] [30532.096532]  do_softirq_own_stack+0x1c/0x30
[2024-09-19 09:32:58.610] [30532.100704]  __irq_exit_rcu+0xcc/0xf4
[2024-09-19 09:32:58.610] [30532.104360]  irq_exit_rcu+0x10/0x20
[2024-09-19 09:32:58.610] [30532.107841]  el0_interrupt+0x50/0xdc
[2024-09-19 09:32:58.610] [30532.111410]  __el0_irq_handler_common+0x18/0x24
[2024-09-19 09:32:58.610] [30532.115929]  el0t_64_irq_handler+0x10/0x20
[2024-09-19 09:32:58.610] [30532.120014]  el0t_64_irq+0x18c/0x190
[2024-09-19 09:33:01.019] [30536.935743] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:360688 dql_avail:-244 free_desc:510
[2024-09-19 09:33:06.152] [30542.055743] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:365808 dql_avail:-244 free_desc:510
[2024-09-19 09:33:09.630] [30545.543755] omap_i2c 20010000.i2c: controller timed out
[2024-09-19 09:33:12.041] [30547.943755] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:371696 dql_avail:-244 free_desc:510
[2024-09-19 09:33:17.154] [30553.063743] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:376816 dql_avail:-244 free_desc:510
[2024-09-19 09:33:23.033] [30558.947756] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:382700 dql_avail:-244 free_desc:510
[2024-09-19 09:33:25.887] [30561.799750] omap_i2c 20010000.i2c: controller timed out
[2024-09-19 09:33:27.911] [30563.815748] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:387568 dql_avail:-244 free_desc:510
[2024-09-19 09:33:33.028] [30568.931750] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:392684 dql_avail:-244 free_desc:510
[2024-09-19 09:33:38.145] [30574.055751] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:397808 dql_avail:-244 free_desc:510
[2024-09-19 09:33:42.152] [30578.051764] omap_i2c 20010000.i2c: controller timed out
[2024-09-19 09:33:44.028] [30579.939757] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:403692 dql_avail:-244 free_desc:510
[2024-09-19 09:33:49.155] [30585.063750] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:408816 dql_avail:-244 free_desc:510
[2024-09-19 09:33:55.041] [30590.947744] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:414700 dql_avail:-244 free_desc:510
[2024-09-19 09:33:58.400] [30594.311754] omap_i2c 20010000.i2c: controller timed out
[2024-09-19 09:33:59.091] [30594.995729] rcu: INFO: rcu_preempt self-detected stall on CPU
[2024-09-19 09:33:59.091] [30595.001491] rcu: 	0-....: (131071 ticks this GP) idle=ce0c/1/0x4000000000000000 softirq=4495200/4495202 fqs=49243
[2024-09-19 09:33:59.106] [30595.011738] 	(t=131330 jiffies g=9036749 q=20082 ncpus=2)
[2024-09-19 09:33:59.107] [30595.017130] CPU: 0 PID: 62393 Comm: python3 Tainted: G        W  O       6.1.46-g247b2535b2 #1
[2024-09-19 09:33:59.122] [30595.025724] Hardware name: Texas Instruments AM625 SK (DT)
[2024-09-19 09:33:59.124] [30595.031194] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[2024-09-19 09:33:59.130] [30595.038140] pc : _stext+0xa4/0x2a4
[2024-09-19 09:33:59.139] [30595.041539] lr : _stext+0x6c/0x2a4
[2024-09-19 09:33:59.139] [30595.044934] sp : ffff800008003f60
[2024-09-19 09:33:59.141] [30595.048235] x29: ffff800008003f60 x28: ffff000006b863c0 x27: 0000000000000200
[2024-09-19 09:33:59.154] [30595.055361] x26: 0000000000000000 x25: 0000000000000000 x24: ffff80000914d000
[2024-09-19 09:33:59.157] [30595.062484] x23: 0000000020000000 x22: ffff800009153000 x21: ffff80000914d9c8
[2024-09-19 09:33:59.162] [30595.069607] x20: ffff8000092b60c0 x19: ffff000006b863c0 x18: 0000000000000000
[2024-09-19 09:33:59.170] [30595.076729] x17: ffff80002ec4b000 x16: ffff800008000000 x15: 0000a9e5807298e4
[2024-09-19 09:33:59.174] [30595.083852] x14: 01fa4fb86496916c x13: 0000000000000001 x12: 0000000000000004
[2024-09-19 09:33:59.189] [30595.090975] x11: 0000000000000001 x10: 00000000000001ab x9 : ffff8000092b9d38
[2024-09-19 09:33:59.190] [30595.098098] x8 : 00001b59186f8100 x7 : 7fffffffffffffff x6 : 000004cd02f58c8a
[2024-09-19 09:33:59.203] [30595.105221] x5 : 03ffffffffffffff x4 : 0000000000000015 x3 : ffff000006b863c0
[2024-09-19 09:33:59.205] [30595.112348] x2 : ffff80002ec4b000 x1 : 00000000000000e0 x0 : ffff800009153b00
[2024-09-19 09:33:59.217] [30595.119478] Call trace:
[2024-09-19 09:33:59.217] [30595.121914]  _stext+0xa4/0x2a4
[2024-09-19 09:33:59.219] [30595.124958]  ____do_softirq+0x10/0x20
[2024-09-19 09:33:59.219] [30595.128612]  call_on_irq_stack+0x24/0x4c
[2024-09-19 09:33:59.224] [30595.132525]  do_softirq_own_stack+0x1c/0x30
[2024-09-19 09:33:59.236] [30595.136696]  __irq_exit_rcu+0xcc/0xf4
[2024-09-19 09:33:59.239] [30595.140352]  irq_exit_rcu+0x10/0x20
[2024-09-19 09:33:59.239] [30595.143834]  el0_interrupt+0x50/0xdc
[2024-09-19 09:33:59.241] [30595.147402]  __el0_irq_handler_common+0x18/0x24
[2024-09-19 09:33:59.250] [30595.151921]  el0t_64_irq_handler+0x10/0x20
[2024-09-19 09:33:59.250] [30595.156006]  el0t_64_irq+0x18c/0x190
[2024-09-19 09:33:59.910] [30595.815751] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:419568 dql_avail:-244 free_desc:510
[2024-09-19 09:34:05.019] [30600.931751] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:424684 dql_avail:-244 free_desc:510
[2024-09-19 09:34:10.152] [30606.055744] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:429808 dql_avail:-244 free_desc:510
[2024-09-19 09:34:14.663] [30610.567751] omap_i2c 20010000.i2c: controller timed out
[2024-09-19 09:34:16.031] [30611.943744] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:435696 dql_avail:-244 free_desc:510

[2024-09-17 15:26:49.388] [ 8010.051980] omap_i2c 20010000.i2c: controller timed out
[2024-09-17 15:27:06.218] [ 8026.883959] omap_i2c 20010000.i2c: controller timed out
[2024-09-17 15:27:22.238] [ 8042.903824] rcu: INFO: rcu_preempt self-detected stall on CPU
[2024-09-17 15:27:22.248] [ 8042.909666] rcu: 	0-....: (711802 ticks this GP) idle=1864/1/0x4000000000000000 softirq=265238/265238 fqs=355903
[2024-09-17 15:27:22.248] [ 8042.919900] 	(t=714415 jiffies g=531109 q=64865 ncpus=2)
[2024-09-17 15:27:22.258] [ 8042.925253] CPU: 0 PID: 5320 Comm: python3 Tainted: G           O       6.1.46-g247b2535b2 #1
[2024-09-17 15:27:22.268] [ 8042.933812] Hardware name: Texas Instruments AM625 SK (DT)
[2024-09-17 15:27:22.278] [ 8042.939313] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[2024-09-17 15:27:22.278] [ 8042.946309] pc : _raw_spin_unlock_irq+0x10/0x50
[2024-09-17 15:27:22.288] [ 8042.950899] lr : __run_timers.part.0+0x128/0x234
[2024-09-17 15:27:22.288] [ 8042.955559] sp : ffff800008003ea0
[2024-09-17 15:27:22.288] [ 8042.958888] x29: ffff800008003ef0 x28: 0000000000000005 x27: 0000000000000020
[2024-09-17 15:27:22.298] [ 8042.966092] x26: ffff80000806a450 x25: ffff8000092b79c0 x24: ffff8000092b60c8
[2024-09-17 15:27:22.308] [ 8042.973289] x23: 0000000000000100 x22: 0000000000000008 x21: ffff80000914d9c8
[2024-09-17 15:27:22.318] [ 8042.980485] x20: ffff000037d99180 x19: ffff000002c3c208 x18: 0000000000000000
[2024-09-17 15:27:22.318] [ 8042.987680] x17: ffff80002ec4b000 x16: ffff800008000000 x15: 0000000000000000
[2024-09-17 15:27:22.328] [ 8042.994876] x14: 00000000000002ca x13: 0000000000000001 x12: 0000000000000000
[2024-09-17 15:27:22.338] [ 8043.002070] x11: 0000000000000001 x10: 00000000000002ca x9 : 00000000000003fd
[2024-09-17 15:27:22.338] [ 8043.009263] x8 : 00000000005b8d80 x7 : ffff000037db21c0 x6 : ffff000003870500
[2024-09-17 15:27:22.348] [ 8043.016458] x5 : 0000000000000000 x4 : 0000000000000000 x3 : ffff000037d99180
[2024-09-17 15:27:22.358] [ 8043.023648] x2 : 0000000000000001 x1 : 0000000000000000 x0 : 00000000000000e0
[2024-09-17 15:27:22.368] [ 8043.030844] Call trace:
[2024-09-17 15:27:22.368] [ 8043.033309]  _raw_spin_unlock_irq+0x10/0x50
[2024-09-17 15:27:22.368] [ 8043.037528]  run_timer_softirq+0x3c/0x7c
[2024-09-17 15:27:22.378] [ 8043.041488]  _stext+0x124/0x2a4
[2024-09-17 15:27:22.378] [ 8043.044670]  ____do_softirq+0x10/0x20
[2024-09-17 15:27:22.378] [ 8043.048372]  call_on_irq_stack+0x24/0x4c
[2024-09-17 15:27:22.388] [ 8043.052335]  do_softirq_own_stack+0x1c/0x30
[2024-09-17 15:27:22.398] [ 8043.056555]  __irq_exit_rcu+0xcc/0xf4
[2024-09-17 15:27:22.398] [ 8043.060257]  irq_exit_rcu+0x10/0x20
[2024-09-17 15:27:22.398] [ 8043.063781]  el1_interrupt+0x38/0x70
[2024-09-17 15:27:22.408] [ 8043.067393]  el1h_64_irq_handler+0x18/0x2c
[2024-09-17 15:27:22.408] [ 8043.071524]  el1h_64_irq+0x64/0x68
[2024-09-17 15:27:22.408] [ 8043.074957]  __kmem_cache_alloc_node+0x30/0x3d0
[2024-09-17 15:27:22.418] [ 8043.079529]  __kmalloc_node_track_caller+0x44/0x70
[2024-09-17 15:27:22.418] [ 8043.084364]  kmalloc_reserve+0x5c/0xd0
[2024-09-17 15:27:22.418] [ 8043.088156]  __alloc_skb+0xa4/0x194
[2024-09-17 15:27:22.428] [ 8043.091688]  alloc_skb_with_frags+0x50/0x1fc
[2024-09-17 15:27:22.438] [ 8043.096006]  sock_alloc_send_pskb+0x2a4/0x2bc
[2024-09-17 15:27:22.438] [ 8043.100397]  unix_stream_sendmsg+0x12c/0x6cc
[2024-09-17 15:27:22.438] [ 8043.104713]  sock_write_iter+0xa8/0x11c
[2024-09-17 15:27:22.448] [ 8043.108593]  vfs_write+0x25c/0x2b4
[2024-09-17 15:27:22.448] [ 8043.112037]  ksys_write+0xe4/0x100
[2024-09-17 15:27:22.448] [ 8043.115473]  __arm64_sys_write+0x1c/0x30
[2024-09-17 15:27:22.458] [ 8043.119432]  invoke_syscall+0x48/0x114
[2024-09-17 15:27:22.458] [ 8043.123225]  el0_svc_common.constprop.0+0xd4/0xfc
[2024-09-17 15:27:22.458] [ 8043.127972]  do_el0_svc+0x20/0x30
[2024-09-17 15:27:22.468] [ 8043.131328]  el0_svc+0x28/0xa0
[2024-09-17 15:27:22.478] [ 8043.134415]  el0t_64_sync_handler+0xbc/0x140
[2024-09-17 15:27:22.478] [ 8043.138721]  el0t_64_sync+0x18c/0x190
[2024-09-17 15:27:23.058] [ 8043.715967] omap_i2c 20010000.i2c: controller timed out
[2024-09-17 15:27:39.888] [ 8060.548050] omap_i2c 20010000.i2c: controller timed out
[2024-09-17 15:27:56.718] [ 8077.379950] omap_i2c 20010000.i2c: controller timed out

[2024-09-20 08:11:15.202] [17108.515876] omap_i2c 20010000.i2c: controller timed out
[2024-09-20 08:11:15.231] [17113.059872] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16659904 dql_avail:-314 free_desc:503
[2024-09-20 08:11:15.231] [17118.947868] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16665792 dql_avail:-314 free_desc:503
[2024-09-20 08:11:15.231] [17123.811875] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16670656 dql_avail:-314 free_desc:503
[2024-09-20 08:11:15.231] [17125.379877] omap_i2c 20010000.i2c: controller timed out
[2024-09-20 08:11:15.231] [17126.307845] rcu: INFO: rcu_preempt self-detected stall on CPU
[2024-09-20 08:11:15.231] [17126.313619] rcu: 	0-....: (4184416 ticks this GP) idle=df54/1/0x4000000000000000 softirq=57916/57917 fqs=2089643
[2024-09-20 08:11:15.231] [17126.323778] 	(t=4197560 jiffies g=99709 q=111344 ncpus=2)
[2024-09-20 08:11:15.231] [17126.329169] CPU: 0 PID: 756 Comm: Thread-3 Tainted: G        W  O       6.1.46-g247b2535b2 #1
[2024-09-20 08:11:15.231] [17126.337676] Hardware name: Texas Instruments AM625 SK (DT)
[2024-09-20 08:11:15.231] [17126.343146] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[2024-09-20 08:11:15.231] [17126.350092] pc : _stext+0xa4/0x2a4
[2024-09-20 08:11:15.231] [17126.353493] lr : _stext+0x6c/0x2a4
[2024-09-20 08:11:15.231] [17126.356884] sp : ffff800008003f60
[2024-09-20 08:11:15.231] [17126.360185] x29: ffff800008003f60 x28: ffff000029dd0000 x27: 0000000000000080
[2024-09-20 08:11:15.267] [17126.367315] x26: 0000000000000000 x25: ffff000000066528 x24: ffff80000914d000
[2024-09-20 08:11:15.267] [17126.374439] x23: 0000000080000005 x22: ffff800009153000 x21: ffff80000914d9c8
[2024-09-20 08:11:15.267] [17126.381565] x20: ffff8000092b60c0 x19: ffff000029dd0000 x18: 0000000000000000
[2024-09-20 08:11:15.267] [17126.388688] x17: ffff80002ec4b000 x16: ffff800008000000 x15: 00001125dfd0181c
[2024-09-20 08:11:15.267] [17126.395813] x14: 0037a11d041e67c8 x13: 0000000000000001 x12: 0000000000000001
[2024-09-20 08:11:15.267] [17126.402937] x11: 0000000000000004 x10: 0000000000000018 x9 : ffff8000092b9d38
[2024-09-20 08:11:15.267] [17126.410062] x8 : 0000004dda1fd100 x7 : 7fffffffffffffff x6 : 00000815f57f9d5c
[2024-09-20 08:11:15.267] [17126.417186] x5 : 03ffffffffffffff x4 : 0000000000000015 x3 : ffff000029dd0000
[2024-09-20 08:11:15.267] [17126.424310] x2 : ffff80002ec4b000 x1 : 00000000000000e0 x0 : ffff800009153b00
[2024-09-20 08:11:15.267] [17126.431442] Call trace:
[2024-09-20 08:11:15.267] [17126.433877]  _stext+0xa4/0x2a4
[2024-09-20 08:11:15.267] [17126.436923]  ____do_softirq+0x10/0x20
[2024-09-20 08:11:15.267] [17126.440578]  call_on_irq_stack+0x24/0x4c
[2024-09-20 08:11:15.267] [17126.444491]  do_softirq_own_stack+0x1c/0x30
[2024-09-20 08:11:15.267] [17126.448664]  __irq_exit_rcu+0xcc/0xf4
[2024-09-20 08:11:15.287] [17126.452324]  irq_exit_rcu+0x10/0x20
[2024-09-20 08:11:15.287] [17126.455803]  el1_interrupt+0x38/0x70
[2024-09-20 08:11:15.287] [17126.459370]  el1h_64_irq_handler+0x18/0x2c
[2024-09-20 08:11:15.287] [17126.463457]  el1h_64_irq+0x64/0x68
[2024-09-20 08:11:15.287] [17126.466848]  _raw_spin_unlock_irqrestore+0xc/0x50
[2024-09-20 08:11:15.287] [17126.471540]  n_tty_write+0x318/0x490
[2024-09-20 08:11:15.287] [17126.475109]  file_tty_write.constprop.0+0x118/0x2b0
[2024-09-20 08:11:15.287] [17126.479976]  tty_write+0x14/0x20
[2024-09-20 08:11:15.287] [17126.483196]  vfs_write+0x228/0x2b4
[2024-09-20 08:11:15.287] [17126.486589]  ksys_write+0x6c/0x100
[2024-09-20 08:11:15.287] [17126.489981]  __arm64_sys_write+0x1c/0x30
[2024-09-20 08:11:15.287] [17126.493893]  invoke_syscall+0x48/0x114
[2024-09-20 08:11:15.287] [17126.497635]  el0_svc_common.constprop.0+0xd4/0xfc
[2024-09-20 08:11:15.287] [17126.502329]  do_el0_svc+0x20/0x30
[2024-09-20 08:11:15.287] [17126.505635]  el0_svc+0x28/0xa0
[2024-09-20 08:11:15.287] [17126.508680]  el0t_64_sync_handler+0xbc/0x140
[2024-09-20 08:11:15.287] [17126.512938]  el0t_64_sync+0x18c/0x190
[2024-09-20 08:11:17.302] [17128.931875] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16675776 dql_avail:-314 free_desc:503
[2024-09-20 08:11:22.421] [17134.051866] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16680896 dql_avail:-314 free_desc:503
[2024-09-20 08:11:28.312] [17139.939876] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16686784 dql_avail:-314 free_desc:503
[2024-09-20 08:11:30.616] [17142.243879] omap_i2c 20010000.i2c: controller timed out
[2024-09-20 08:11:33.431] [17145.059876] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16691904 dql_avail:-314 free_desc:503
[2024-09-20 08:11:39.317] [17150.947875] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16697792 dql_avail:-314 free_desc:503
[2024-09-20 08:11:44.182] [17155.811866] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16702656 dql_avail:-314 free_desc:503
[2024-09-20 08:11:47.517] [17159.139873] omap_i2c 20010000.i2c: controller timed out
[2024-09-20 08:11:49.304] [17160.931875] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16707776 dql_avail:-314 free_desc:503
[2024-09-20 08:11:54.423] [17166.051870] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16712896 dql_avail:-314 free_desc:503
[2024-09-20 08:12:00.310] [17171.939866] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16718784 dql_avail:-314 free_desc:503
[2024-09-20 08:12:04.350] [17175.971880] omap_i2c 20010000.i2c: controller timed out
[2024-09-20 08:12:05.431] [17177.059873] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16723904 dql_avail:-314 free_desc:503
[2024-09-20 08:12:11.320] [17182.947874] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16729792 dql_avail:-314 free_desc:503
[2024-09-20 08:12:16.184] [17187.811871] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16734656 dql_avail:-314 free_desc:503
[2024-09-20 08:12:17.724] [17189.347844] rcu: INFO: rcu_preempt self-detected stall on CPU
[2024-09-20 08:12:17.730] [17189.353617] rcu: 	0-....: (4200125 ticks this GP) idle=df54/1/0x4000000000000000 softirq=57916/57917 fqs=2097488
[2024-09-20 08:12:17.737] [17189.363775] 	(t=4213319 jiffies g=99709 q=111344 ncpus=2)
[2024-09-20 08:12:17.747] [17189.369165] CPU: 0 PID: 756 Comm: Thread-3 Tainted: G        W  O       6.1.46-g247b2535b2 #1
[2024-09-20 08:12:17.757] [17189.377673] Hardware name: Texas Instruments AM625 SK (DT)
[2024-09-20 08:12:17.761] [17189.383144] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[2024-09-20 08:12:17.761] [17189.390090] pc : _stext+0xa4/0x2a4
[2024-09-20 08:12:17.771] [17189.393491] lr : _stext+0x6c/0x2a4
[2024-09-20 08:12:17.771] [17189.396882] sp : ffff800008003f60
[2024-09-20 08:12:17.774] [17189.400184] x29: ffff800008003f60 x28: ffff000029dd0000 x27: 0000000000000080
[2024-09-20 08:12:17.780] [17189.407315] x26: 0000000000000000 x25: ffff000000066528 x24: ffff80000914d000
[2024-09-20 08:12:17.787] [17189.414438] x23: 0000000080000005 x22: ffff800009153000 x21: ffff80000914d9c8
[2024-09-20 08:12:17.796] [17189.421563] x20: ffff8000092b60c0 x19: ffff000029dd0000 x18: 0000000000000000
[2024-09-20 08:12:17.800] [17189.428686] x17: ffff80002ec4b000 x16: ffff800008000000 x15: 00001125dfd0181c
[2024-09-20 08:12:17.811] [17189.435810] x14: 0037a11d041e67c8 x13: 0000000000000001 x12: 0000000000000001
[2024-09-20 08:12:17.818] [17189.442935] x11: 0000000000000004 x10: 0000000000000018 x9 : ffff8000092b9d38
[2024-09-20 08:12:17.822] [17189.450058] x8 : 0000004dda1fd100 x7 : 7fffffffffffffff x6 : 00000815f57f9d5c
[2024-09-20 08:12:17.832] [17189.457183] x5 : 03ffffffffffffff x4 : 0000000000000015 x3 : ffff000029dd0000
[2024-09-20 08:12:17.836] [17189.464311] x2 : ffff80002ec4b000 x1 : 00000000000000e0 x0 : ffff800009153b00
[2024-09-20 08:12:17.856] [17189.471441] Call trace:
[2024-09-20 08:12:17.856] [17189.473879]  _stext+0xa4/0x2a4
[2024-09-20 08:12:17.856] [17189.476926]  ____do_softirq+0x10/0x20
[2024-09-20 08:12:17.856] [17189.480581]  call_on_irq_stack+0x24/0x4c
[2024-09-20 08:12:17.861] [17189.484495]  do_softirq_own_stack+0x1c/0x30
[2024-09-20 08:12:17.871] [17189.488668]  __irq_exit_rcu+0xcc/0xf4
[2024-09-20 08:12:17.871] [17189.492326]  irq_exit_rcu+0x10/0x20
[2024-09-20 08:12:17.871] [17189.495805]  el1_interrupt+0x38/0x70
[2024-09-20 08:12:17.875] [17189.499372]  el1h_64_irq_handler+0x18/0x2c
[2024-09-20 08:12:17.875] [17189.503458]  el1h_64_irq+0x64/0x68
[2024-09-20 08:12:17.880] [17189.506849]  _raw_spin_unlock_irqrestore+0xc/0x50
[2024-09-20 08:12:17.886] [17189.511541]  n_tty_write+0x318/0x490
[2024-09-20 08:12:17.895] [17189.515110]  file_tty_write.constprop.0+0x118/0x2b0
[2024-09-20 08:12:17.895] [17189.519976]  tty_write+0x14/0x20
[2024-09-20 08:12:17.898] [17189.523195]  vfs_write+0x228/0x2b4
[2024-09-20 08:12:17.902] [17189.526589]  ksys_write+0x6c/0x100
[2024-09-20 08:12:17.902] [17189.529980]  __arm64_sys_write+0x1c/0x30
[2024-09-20 08:12:17.912] [17189.533892]  invoke_syscall+0x48/0x114
[2024-09-20 08:12:17.917] [17189.537633]  el0_svc_common.constprop.0+0xd4/0xfc
[2024-09-20 08:12:17.917] [17189.542328]  do_el0_svc+0x20/0x30
[2024-09-20 08:12:17.917] [17189.545635]  el0_svc+0x28/0xa0
[2024-09-20 08:12:17.923] [17189.548680]  el0t_64_sync_handler+0xbc/0x140
[2024-09-20 08:12:17.923] [17189.552938]  el0t_64_sync+0x18c/0x190
[2024-09-20 08:12:21.246] [17192.867877] omap_i2c 20010000.i2c: controller timed out
[2024-09-20 08:12:21.307] [17192.931863] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16739776 dql_avail:-314 free_desc:503
[2024-09-20 08:12:26.427] [17198.051876] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16744896 dql_avail:-314 free_desc:503
[2024-09-20 08:12:32.314] [17203.939874] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16750784 dql_avail:-314 free_desc:503
[2024-09-20 08:12:37.435] [17209.059865] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16755904 dql_avail:-314 free_desc:503
[2024-09-20 08:12:38.111] [17209.731874] omap_i2c 20010000.i2c: controller timed out
[2024-09-20 08:12:43.323] [17214.947867] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16761792 dql_avail:-314 free_desc:503
[2024-09-20 08:12:48.187] [17219.811877] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16766656 dql_avail:-314 free_desc:503
[2024-09-20 08:12:53.307] [17224.931867] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16771776 dql_avail:-314 free_desc:503
[2024-09-20 08:12:55.011] [17226.627880] omap_i2c 20010000.i2c: controller timed out
[2024-09-20 08:12:58.428] [17230.051875] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16776896 dql_avail:-314 free_desc:503
[2024-09-20 08:13:04.315] [17235.939876] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16782784 dql_avail:-314 free_desc:503
[2024-09-20 08:13:09.436] [17241.059876] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16787904 dql_avail:-314 free_desc:503
[2024-09-20 08:13:11.874] [17243.491879] omap_i2c 20010000.i2c: controller timed out
[2024-09-20 08:13:15.325] [17246.947871] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16793792 dql_avail:-314 free_desc:503
[2024-09-20 08:13:20.190] [17251.811874] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:16798656 dql_avail:-314 free_desc:503
[2024-09-20 08:13:20.766] [17252.387843] rcu: INFO: rcu_preempt self-detected stall on CPU
[2024-09-20 08:13:20.774] [17252.393616] rcu: 	0-....: (4215834 ticks this GP) idle=df54/1/0x4000000000000000 softirq=57916/57917 fqs=2105333
[2024-09-20 08:13:20.787] [17252.403775] 	(t=4229079 jiffies g=99709 q=111349 ncpus=2)
[2024-09-20 08:13:20.796] [17252.409166] CPU: 0 PID: 756 Comm: Thread-3 Tainted: G        W  O       6.1.46-g247b2535b2 #1
[2024-09-20 08:13:20.810] [17252.417673] Hardware name: Texas Instruments AM625 SK (DT)
[2024-09-20 08:13:20.810] [17252.423145] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[2024-09-20 08:13:20.822] [17252.430091] pc : _stext+0xa4/0x2a4
[2024-09-20 08:13:20.822] [17252.433490] lr : _stext+0x6c/0x2a4
[2024-09-20 08:13:20.822] [17252.436882] sp : ffff800008003f60
[2024-09-20 08:13:20.828] [17252.440185] x29: ffff800008003f60 x28: ffff000029dd0000 x27: 0000000000000080
[2024-09-20 08:13:20.828] [17252.447315] x26: 0000000000000000 x25: ffff000000066528 x24: ffff80000914d000
[2024-09-20 08:13:20.828] [17252.454438] x23: 0000000080000005 x22: ffff800009153000 x21: ffff80000914d9c8
[2024-09-20 08:13:20.840] [17252.461562] x20: ffff8000092b60c0 x19: ffff000029dd0000 x18: 0000000000000000
[2024-09-20 08:13:20.848] [17252.468688] x17: ffff80002ec4b000 x16: ffff800008000000 x15: 00001125dfd0181c
[2024-09-20 08:13:20.854] [17252.475812] x14: 0037a11d041e67c8 x13: 0000000000000001 x12: 0000000000000001
[2024-09-20 08:13:20.866] [17252.482936] x11: 0000000000000004 x10: 0000000000000018 x9 : ffff8000092b9d38
[2024-09-20 08:13:20.866] [17252.490062] x8 : 0000004dda1fd100 x7 : 7fffffffffffffff x6 : 00000815f57f9d5c
[2024-09-20 08:13:20.872] [17252.497185] x5 : 03ffffffffffffff x4 : 0000000000000015 x3 : ffff000029dd0000
[2024-09-20 08:13:20.884] [17252.504309] x2 : ffff80002ec4b000 x1 : 00000000000000e0 x0 : ffff800009153b00
[2024-09-20 08:13:20.890] [17252.511441] Call trace:
[2024-09-20 08:13:20.890] [17252.513877]  _stext+0xa4/0x2a4
[2024-09-20 08:13:20.896] [17252.516922]  ____do_softirq+0x10/0x20
[2024-09-20 08:13:20.896] [17252.520576]  call_on_irq_stack+0x24/0x4c
[2024-09-20 08:13:20.905] [17252.524489]  do_softirq_own_stack+0x1c/0x30
[2024-09-20 08:13:20.912] [17252.528662]  __irq_exit_rcu+0xcc/0xf4
[2024-09-20 08:13:20.912] [17252.532320]  irq_exit_rcu+0x10/0x20
[2024-09-20 08:13:20.912] [17252.535799]  el1_interrupt+0x38/0x70
[2024-09-20 08:13:20.922] [17252.539366]  el1h_64_irq_handler+0x18/0x2c
[2024-09-20 08:13:20.922] [17252.543453]  el1h_64_irq+0x64/0x68
[2024-09-20 08:13:20.929] [17252.546846]  _raw_spin_unlock_irqrestore+0xc/0x50
[2024-09-20 08:13:20.936] [17252.551538]  n_tty_write+0x318/0x490
[2024-09-20 08:13:20.936] [17252.555109]  file_tty_write.constprop.0+0x118/0x2b0
[2024-09-20 08:13:20.936] [17252.559978]  tty_write+0x14/0x20
[2024-09-20 08:13:20.946] [17252.563197]  vfs_write+0x228/0x2b4
[2024-09-20 08:13:20.952] [17252.566590]  ksys_write+0x6c/0x100
[2024-09-20 08:13:20.952] [17252.569982]  __arm64_sys_write+0x1c/0x30
[2024-09-20 08:13:20.952] [17252.573893]  invoke_syscall+0x48/0x114
[2024-09-20 08:13:20.962] [17252.577637]  el0_svc_common.constprop.0+0xd4/0xfc
[2024-09-20 08:13:20.962] [17252.582330]  do_el0_svc+0x20/0x30
[2024-09-20 08:13:20.968] [17252.585636]  el0_svc+0x28/0xa0
[2024-09-20 08:13:20.968] [17252.588682]  el0t_64_sync_handler+0xbc/0x140
[2024-09-20 08:13:20.968] [17252.592940]  el0t_64_sync+0x18c/0x190

Is it a common or known issue in your side ? What is the root cause of this problem and Is there any debug method to find out root cause of this issue ?

Or if you have a fix related with it, can you please share with us ?

Thanks in advance,
Veysel

11 months ago

0 Bin Liu 11 months ago

TI__Guru**** 168261 points

Hi Veysel,

Please apply the following kernel patch and let me know if the issue still happens.

Fullscreen 8420.kernel-61-dss-irq-workaround.diff Download

diff --git a/drivers/gpu/drm/tidss/tidss_dispc.c b/drivers/gpu/drm/tidss/tidss_dispc.c
index 5bcc9153a977..2cd18955d174 100644
--- a/drivers/gpu/drm/tidss/tidss_dispc.c
+++ b/drivers/gpu/drm/tidss/tidss_dispc.c
@@ -939,6 +954,7 @@ void dispc_k3_clear_irqstatus(struct dispc_device *dispc, dispc_irq_t clearmask)
 {
 	unsigned int i;
 	u32 top_clear = 0;
+	u32 val;
 
 	for (i = 0; i < dispc->feat->num_vps; ++i) {
 		if (clearmask & DSS_IRQ_VP_MASK(i)) {
@@ -955,10 +971,18 @@ void dispc_k3_clear_irqstatus(struct dispc_device *dispc, dispc_irq_t clearmask)
 	if (dispc->feat->subrev == DISPC_K2G)
 		return;
 
+	val = dispc_read(dispc, DISPC_IRQSTATUS);
+	if (val > top_clear) {
+		dev_warn_ratelimited(dispc->dev,
+			"lvl1 irq 0x%x, lvl2 irqmask 0x%x\n", val, top_clear);
+		top_clear = val;
+	}
 	dispc_write(dispc, DISPC_IRQSTATUS, top_clear);
 
 	/* Flush posted writes */
-	dispc_read(dispc, DISPC_IRQSTATUS);
+	val = dispc_read(dispc, DISPC_IRQSTATUS);
+	if (val)
+		dev_warn_ratelimited(dispc->dev, "irq 0x%x not cleared\n", val);
 }
 
 static

0 Veysel Oguz 11 months ago in reply to Bin Liu

Prodigy 210 points

Hi Bin,

I applied the patch and tested again, unfortunately issue happened again. Please find the logs below.

[2024-10-08 08:20:09.521] û!¡!±±¡%ñ!¡!±±¡¡ş¡1ı!¡!±®¡5%ñ!¡!±!
[2024-10-08 08:20:09.521] ±1ô[42769.607863] rcu: INFO: rcu_preempt self-detected stall on CPU
[2024-10-08 08:20:22.340] [42769.613622] rcu: 	0-....: (10605087 ticks this GP) idle=229c/1/0x4000000000000000 softirq=17350/17359 fqs=3748339
[2024-10-08 08:20:22.340] [42769.623870] 	(t=10642898 jiffies g=23769 q=132403 ncpus=2)
[2024-10-08 08:20:22.340] [42769.629347] CPU: 0 PID: 194 Comm: systemd-journal Tainted: G        W  O       6.1.46-g247b2535b2 #1
[2024-10-08 08:20:22.355] [42769.638463] Hardware name: Texas Instruments AM625 SK (DT)
[2024-10-08 08:20:22.371] [42769.643934] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[2024-10-08 08:20:22.371] [42769.650882] pc : _stext+0xa4/0x2a4
[2024-10-08 08:20:22.377] [42769.654282] lr : _stext+0x6c/0x2a4
[2024-10-08 08:20:22.377] [42769.657674] sp : ffff800008003f60
[2024-10-08 08:20:22.377] [42769.660975] x29: ffff800008003f60 x28: ffff000003498000 x27: 0000000000000200
[2024-10-08 08:20:22.393] [42769.668108] x26: ffff000002f90478 x25: 00000000000000c0 x24: ffff80000914d000
[2024-10-08 08:20:22.393] [42769.675233] x23: 0000000040000005 x22: ffff800009153000 x21: ffff80000914d9c8
[2024-10-08 08:20:22.409] [42769.682357] x20: ffff8000092b60c0 x19: ffff000003498000 x18: 0000000000000000
[2024-10-08 08:20:22.409] [42769.689482] x17: ffff80002ec4b000 x16: ffff800008000000 x15: 00003d0900000000
[2024-10-08 08:20:22.409] [42769.696607] x14: 00000000000c3500 x13: 00008c05b022dbec x12: 023054e4bb162a18
[2024-10-08 08:20:22.424] [42769.703732] x11: 000000000000ba7e x10: 0000000000000001 x9 : ffff8000092b9d38
[2024-10-08 08:20:22.424] [42769.710857] x8 : 0000002e1ae96900 x7 : 7fffffffffffffff x6 : 0000022b61020ad7
[2024-10-08 08:20:22.440] [42769.717981] x5 : 03ffffffffffffff x4 : 0000000000000015 x3 : ffff000003498000
[2024-10-08 08:20:22.440] [42769.725106] x2 : ffff80002ec4b000 x1 : 00000000000000e0 x0 : ffff800009153b00
[2024-10-08 08:20:22.456] [42769.732234] Call trace:
[2024-10-08 08:20:22.456] [42769.734670]  _stext+0xa4/0x2a4
[2024-10-08 08:20:22.456] [42769.737721]  ____do_softirq+0x10/0x20
[2024-10-08 08:20:22.471] [42769.741376]  call_on_irq_stack+0x24/0x4c
[2024-10-08 08:20:22.471] [42769.745290]  do_softirq_own_stack+0x1c/0x30
[2024-10-08 08:20:22.471] [42769.749463]  __irq_exit_rcu+0xcc/0xf4
[2024-10-08 08:20:22.478] [42769.753118]  irq_exit_rcu+0x10/0x20
[2024-10-08 08:20:22.478] [42769.756597]  el1_interrupt+0x38/0x70
[2024-10-08 08:20:22.478] [42769.760165]  el1h_64_irq_handler+0x18/0x2c
[2024-10-08 08:20:22.493] [42769.764251]  el1h_64_irq+0x64/0x68
[2024-10-08 08:20:22.493] [42769.767642]  __rcu_read_lock+0x4/0x20
[2024-10-08 08:20:22.493] [42769.771299]  memcg_list_lru_alloc+0xa0/0x36c
[2024-10-08 08:20:22.493] [42769.775563]  kmem_cache_alloc_lru+0xa0/0x510
[2024-10-08 08:20:22.493] [42769.779824]  __d_alloc+0x34/0x204
[2024-10-08 08:20:22.509] [42769.783144]  d_alloc_parallel+0x70/0x47c
[2024-10-08 08:20:22.509] [42769.787057]  __lookup_slow+0x60/0x140
[2024-10-08 08:20:22.509] [42769.790710]  walk_component+0xd8/0x190
[2024-10-08 08:20:22.509] [42769.794451]  path_lookupat+0x78/0x19c
[2024-10-08 08:20:22.525] [42769.798104]  filename_lookup+0x98/0x194
[2024-10-08 08:20:22.525] [42769.801931]  user_path_at_empty+0x4c/0x74
[2024-10-08 08:20:22.525] [42769.805932]  do_faccessat+0x90/0x2cc
[2024-10-08 08:20:22.525] [42769.809501]  __arm64_sys_faccessat+0x24/0x30
[2024-10-08 08:20:22.540] [42769.813765]  invoke_syscall+0x48/0x114
[2024-10-08 08:20:22.540] [42769.817509]  el0_svc_common.constprop.0+0xd4/0xfc
[2024-10-08 08:20:22.540] [42769.822204]  do_el0_svc+0x20/0x30
[2024-10-08 08:20:22.556] [42769.825511]  el0_svc+0x28/0xa0
[2024-10-08 08:20:22.556] [42769.828557]  el0t_64_sync_handler+0xbc/0x140
[2024-10-08 08:20:22.556] [42769.832816]  el0t_64_sync+0x18c/0x190
[2024-10-08 08:20:24.663] [42771.943888] am65-cpsw-nuss 8000000.ethernet eth1: txq:0 DRV_XOFF:0 tmo:42559012 dql_avail:-52 free_desc:511
[2024-10-08 08:20:24.678] [42771.953667] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:42451340 dql_avail:-314 free_desc:511
[2024-10-08 08:20:29.783] [42777.063877] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:42456452 dql_avail:-314 free_desc:511
[2024-10-08 08:20:29.799] [42777.073740] am65-cpsw-nuss 8000000.ethernet eth1: txq:0 DRV_XOFF:0 tmo:42564140 dql_avail:-52 free_desc:511
[2024-10-08 08:20:34.745] [42782.023890] omap_i2c 20010000.i2c: controller timed out
[2024-10-08 08:20:35.677] [42782.947889] am65-cpsw-nuss 8000000.ethernet eth1: txq:0 DRV_XOFF:0 tmo:42570016 dql_avail:-52 free_desc:511
[2024-10-08 08:20:35.677] [42782.957667] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:42462344 dql_avail:-314 free_desc:511
[2024-10-08 08:20:40.538] [42787.811884] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:42467200 dql_avail:-314 free_desc:511
[2024-10-08 08:20:40.538] [42787.821752] am65-cpsw-nuss 8000000.ethernet eth1: txq:0 DRV_XOFF:0 tmo:42574888 dql_avail:-52 free_desc:511
[2024-10-08 08:20:45.662] [42792.935878] am65-cpsw-nuss 8000000.ethernet eth1: txq:0 DRV_XOFF:0 tmo:42580004 dql_avail:-52 free_desc:511
[2024-10-08 08:20:45.662] [42792.945668] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:42472332 dql_avail:-314 free_desc:511
[2024-10-08 08:20:50.785] [42798.055887] am65-cpsw-nuss 8000000.ethernet eth0: txq:0 DRV_XOFF:0 tmo:42477444 dql_avail:-314 free_desc:511
[2024-10-08 08:20:50.785] [42798.065756] am65-cpsw-nuss 8000000.ethernet eth1: txq:0 DRV_XOFF:0 tmo:42585132 dql_avail:-52 free_desc:511
[2024-10-08 08:20:51.572] [42798.851886] omap_i2c 20010000.i2c: controller timed out

We need your help to debug & fix this rcu stall issue. Please check and feedback.

Thanks in advance,

Veysel

0 Veysel Oguz 11 months ago in reply to Veysel Oguz

Prodigy 210 points

Hi Bin,

There are additional log files starting from boot, below. Can you please check ?

Oct  7 10:22:19 kernel: [    0.000000] Machine model: Texas Instruments AM625 SK 0.000000] efi: UEFI not found. 0.000000] Reserved memory: created CMA memory pool at 0x00000000b8000000, size 128 MiB 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool 0.000000] Reserved memory: created DMA memory pool at 0x000000009c800000, size 3 MiB 0.000000] OF: reserved mem: initialized node ipc-memories@9c800000, compatible id shared-dma-pool 0.000000] Reserved memory: created DMA memory pool at 0x000000009cb00000, size 1 MiB 0.000000] OF: reserved mem: initialized node m4f-dma-memory@9cb00000, compatible id shared-dma-pool 0.000000] Reserved memory: created DMA memory pool at 0x000000009cc00000, size 14 MiB 0.000000] OF: reserved mem: initialized node m4f-memory@9cc00000, compatible id shared-dma-pool 0.000000] Reserved memory: created DMA memory pool at 0x000000009da00000, size 1 MiB 0.000000] OF: reserved mem: initialized node r5f-dma-memory@9da00000, compatible id shared-dma-pool 0.000000] Reserved memory: created DMA memory pool at 0x000000009db00000, size 12 MiB 0.000000] OF: reserved mem: initialized node r5f-memory@9db00000, compatible id shared-dma-pool 0.000000] Zone ranges: 0.000000]   DMA      [mem 0x0000000080000000-0x00000000bfffffff] 0.000000]   DMA32    empty 0.000000]   Normal   empty 0.000000] Movable zone start for each node 0.000000] Early memory node ranges 0.000000]   node   0: [mem 0x0000000080000000-0x000000009c7fffff] 0.000000]   node   0: [mem 0x000000009c800000-0x000000009e6fffff] 0.000000]   node   0: [mem 0x000000009e700000-0x000000009e77ffff] 0.000000]   node   0: [mem 0x000000009e780000-0x000000009fffffff] 0.000000]   node   0: [mem 0x00000000a0000000-0x00000000bfffffff] 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000bfffffff] 0.000000] psci: probing for conduit method from DT. 0.000000] psci: PSCIv1.1 detected in firmware. 0.000000] psci: Using standard PSCI v0.2 function IDs 0.000000] psci: Trusted OS migration not required 0.000000] psci: SMC Calling Convention v1.4 0.000000] percpu: Embedded 19 pages/cpu s37992 r8192 d31640 u77824 0.000000] pcpu-alloc: s37992 r8192 d31640 u77824 alloc=19*4096 0.000000] pcpu-alloc: [0] 0 [0] 1 0.000000] Detected VIPT I-cache on CPU0 0.000000] CPU features: detected: GIC system register CPU interface 0.000000] CPU features: detected: ARM erratum 845719 0.000000] alternatives: applying boot alternatives 0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 258048 0.000000] Kernel command line: ostree=/ostree/boot.0/arago/2b5e4456444f16ac259c42646a91976b1a788f80a2e97186a33e7bd69f4dadc9/0 console=ttyS2,115200n8 ostree_root=/dev/mmcblk0p1 0.000000] Unknown kernel command line parameters "ostree=/ostree/boot.0/arago/2b5e4456444f16ac259c42646a91976b1a788f80a2e97186a33e7bd69f4dadc9/0 ostree_root=/dev/mmcblk0p1", will be passed to user space. 0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) 0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear) 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off 0.000000] Memory: 815480K/1048576K available (12288K kernel code, 1262K rwdata, 4024K rodata, 2688K init, 434K bss, 102024K reserved, 131072K cma-reserved) 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 0.000000] rcu: Preemptible hierarchical RCU implementation. 0.000000] rcu: 	RCU event tracing is enabled. 0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2. 0.000000] 	Trampoline variant of Tasks RCU enabled. 0.000000] 	Tracing variant of Tasks RCU enabled. 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 0.000000] GICv3: GIC: Using split EOI/Deactivate mode 0.000000] GICv3: 256 SPIs implemented 0.000000] GICv3: 0 Extended SPIs implemented 0.000000] Root IRQ handler: gic_handle_irq 0.000000] GICv3: GICv3 features: 16 PPIs 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001880000 0.000000] ITS [mem 0x01820000-0x0182ffff] 0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS 0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19 0.000000] ITS@0x0000000001820000: allocated 524288 Devices @80800000 (flat, esz 8, psz 64K, shr 0) 0.000000] ITS: using cache flushing for cmd queue 0.000000] GICv3: using LPI property table @0x0000000080030000 0.000000] GIC: using cache flushing for LPI property table 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000080040000 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. 0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys). 0.000000] clocksource: arch_sys_counter: mask: 0x3ffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns 0.000001] sched_clock: 58 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns 0.000350] Console: colour dummy device 80x25 0.000404] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS (lpj=800000) 0.000421] pid_max: default: 32768 minimum: 301 0.000469] LSM: Security Framework initializing 0.000601] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear) 0.000617] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear) 0.002476] cblist_init_generic: Setting adjustable number of callback queues. 0.002492] cblist_init_generic: Setting shift to 1 and lim to 1. 0.002562] cblist_init_generic: Setting adjustable number of callback queues. 0.002566] cblist_init_generic: Setting shift to 1 and lim to 1. 0.002730] rcu: Hierarchical SRCU implementation. 0.002733] rcu: 	Max phase no-delay instances is 1000. 0.003019] Platform MSI: msi-controller@1820000 domain created 0.003282] PCI/MSI: /bus@f0000/interrupt-controller@1800000/msi-controller@1820000 domain created 0.003532] EFI services will not be available. 0.003753] smp: Bringing up secondary CPUs ... 0.012544] Detected VIPT I-cache on CPU1 0.012662] GICv3: CPU1: found redistributor 1 region 0:0x00000000018a0000 0.012684] GICv3: CPU1: using allocated LPI pending table @0x0000000080050000 0.012746] CPU1: Booted secondary processor 0x0000000001 [0x410fd034] 0.012868] smp: Brought up 1 node, 2 CPUs 0.012874] SMP: Total of 2 processors activated. 0.012880] CPU features: detected: 32-bit EL0 Support 0.012885] CPU features: detected: CRC32 instructions 0.012954] CPU: All CPU(s) started at EL2 0.012957] alternatives: applying system-wide alternatives 0.014529] devtmpfs: initialized 0.023395] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns 0.023427] futex hash table entries: 512 (order: 3, 32768 bytes, linear) 0.028440] pinctrl core: initialized pinctrl subsystem 0.029109] DMI not present or invalid. 0.029799] NET: Registered PF_NETLINK/PF_ROUTE protocol family 0.030959] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations 0.031062] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations 0.031147] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations 0.031236] audit: initializing netlink subsys (disabled) 0.031420] audit: type=2000 audit(0.028:1): state=initialized audit_enabled=0 res=1 0.031924] thermal_sys: Registered thermal governor 'step_wise' 0.031929] thermal_sys: Registered thermal governor 'power_allocator' 0.031975] cpuidle: using governor menu 0.032138] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers. 0.032220] ASID allocator initialised with 65536 entries 0.047792] platform 30200000.dss: Fixed dependency cycle(s) with /display 0.048287] KASLR disabled due to lack of seed 0.055453] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages 0.055467] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page 0.055472] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages 0.055479] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page 0.055483] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages 0.055487] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page 0.055491] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages 0.055494] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page 0.057096] k3-chipinfo 43000014.chipid: Family:AM62X rev:SR1.0 JTAGID[0x0bb7e02f] Detected 0.057608] reg-fixed-voltage regulator-0: GPIO lookup for consumer (null) 0.057615] reg-fixed-voltage regulator-0: using device tree for GPIO lookup 0.057632] of_get_named_gpiod_flags: can't parse 'gpios' property of node '/regulator-0[0]' 0.057648] of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator-0[0]' 0.057660] reg-fixed-voltage regulator-0: using lookup tables for GPIO lookup 0.057666] reg-fixed-voltage regulator-0: No GPIO consumer (null) found 0.057938] reg-fixed-voltage regulator-1: GPIO lookup for consumer (null) 0.057944] reg-fixed-voltage regulator-1: using device tree for GPIO lookup 0.057952] of_get_named_gpiod_flags: can't parse 'gpios' property of node '/regulator-1[0]' 0.057964] of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator-1[0]' 0.057974] reg-fixed-voltage regulator-1: using lookup tables for GPIO lookup 0.057978] reg-fixed-voltage regulator-1: No GPIO consumer (null) found 0.058192] reg-fixed-voltage regulator-2: GPIO lookup for consumer (null) 0.058198] reg-fixed-voltage regulator-2: using device tree for GPIO lookup 0.058207] of_get_named_gpiod_flags: can't parse 'gpios' property of node '/regulator-2[0]' 0.058219] of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator-2[0]' 0.058228] reg-fixed-voltage regulator-2: using lookup tables for GPIO lookup 0.058233] reg-fixed-voltage regulator-2: No GPIO consumer (null) found 0.058443] reg-fixed-voltage regulator-3: GPIO lookup for consumer (null) 0.058448] reg-fixed-voltage regulator-3: using device tree for GPIO lookup 0.058457] of_get_named_gpiod_flags: can't parse 'gpios' property of node '/regulator-3[0]' 0.058469] of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator-3[0]' 0.058479] reg-fixed-voltage regulator-3: using lookup tables for GPIO lookup 0.058483] reg-fixed-voltage regulator-3: No GPIO consumer (null) found 0.058681] reg-fixed-voltage regulator-5: GPIO lookup for consumer (null) 0.058686] reg-fixed-voltage regulator-5: using device tree for GPIO lookup 0.058695] of_get_named_gpiod_flags: can't parse 'gpios' property of node '/regulator-5[0]' 0.058707] of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator-5[0]' 0.058717] reg-fixed-voltage regulator-5: using lookup tables for GPIO lookup 0.058721] reg-fixed-voltage regulator-5: No GPIO consumer (null) found 0.058908] reg-fixed-voltage regulator-6: GPIO lookup for consumer (null) 0.058914] reg-fixed-voltage regulator-6: using device tree for GPIO lookup 0.058926] of_get_named_gpiod_flags: can't parse 'gpios' property of node '/regulator-6[0]' 0.058938] of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator-6[0]' 0.058948] reg-fixed-voltage regulator-6: using lookup tables for GPIO lookup 0.058952] reg-fixed-voltage regulator-6: No GPIO consumer (null) found 0.059157] reg-fixed-voltage regulator-8: GPIO lookup for consumer (null) 0.059163] reg-fixed-voltage regulator-8: using device tree for GPIO lookup 0.059171] of_get_named_gpiod_flags: can't parse 'gpios' property of node '/regulator-8[0]' 0.059183] of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator-8[0]' 0.059193] reg-fixed-voltage regulator-8: using lookup tables for GPIO lookup 0.059197] reg-fixed-voltage regulator-8: No GPIO consumer (null) found 0.059917] iommu: Default domain type: Translated 0.059925] iommu: DMA domain TLB invalidation policy: strict mode 0.060224] SCSI subsystem initialized 0.060384] libata version 3.00 loaded. 0.060673] usbcore: registered new interface driver usbfs 0.060707] usbcore: registered new interface driver hub 0.060732] usbcore: registered new device driver usb 0.061264] pps_core: LinuxPPS API ver. 1 registered 0.061269] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> 0.061287] PTP clock support registered 0.061428] EDAC MC: Ver: 3.0.0 0.062444] omap-mailbox 29000000.mailbox: omap mailbox rev 0x66fc9100 0.062843] FPGA manager framework 0.062938] Advanced Linux Sound Architecture Driver Initialized. 0.063958] clocksource: Switched to clocksource arch_sys_counter 0.064212] VFS: Disk quotas dquot_6.6.0 0.064251] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) 0.070488] NET: Registered PF_INET protocol family 0.070883] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear) 0.072128] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear) 0.072167] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) 0.072180] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear) 0.072238] TCP bind hash table entries: 8192 (order: 6, 262144 bytes, linear) 0.072516] TCP: Hash tables configured (established 8192 bind 8192) 0.072685] UDP hash table entries: 512 (order: 2, 16384 bytes, linear) 0.072726] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear) 0.072903] NET: Registered PF_UNIX/PF_LOCAL protocol family 0.073506] RPC: Registered named UNIX socket transport module. 0.073515] RPC: Registered udp transport module. 0.073518] RPC: Registered tcp transport module. 0.073520] RPC: Registered tcp NFSv4.1 backchannel transport module. 0.073528] NET: Registered PF_XDP protocol family 0.073551] PCI: CLS 0 bytes, default 64 0.073840] Unpacking initramfs... 0.084088] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available 0.086215] Initialise system trusted keyrings 0.086631] workingset: timestamp_bits=46 max_order=18 bucket_order=0 0.091369] squashfs: version 4.0 (2009/01/31) Phillip Lougher 0.092290] NFS: Registering the id_resolver key type 0.092362] Key type id_resolver registered 0.092366] Key type id_legacy registered 0.092436] nfs4filelayout_init: NFSv4 File Layout Driver Registering... 0.092443] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering... 0.126739] Key type asymmetric registered 0.126764] Asymmetric key parser 'x509' registered 0.126899] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245) 0.127257] io scheduler mq-deadline registered 0.127265] io scheduler kyber registered 0.130970] pinctrl-single 4084000.pinctrl: 34 pins, size 136 0.131626] pinctrl-single f4000.pinctrl: 171 pins, size 684 0.132778] pinctrl-single a40000.pinctrl: 512 pins, size 2048 0.140709] Serial: 8250/16550 driver, 6 ports, IRQ sharing enabled 0.150870] loop: module loaded 0.152425] megasas: 07.719.03.00-rc1 0.154833] mdio_bus fixed-0: GPIO lookup for consumer reset 0.154856] mdio_bus fixed-0: using lookup tables for GPIO lookup 0.154862] mdio_bus fixed-0: No GPIO consumer reset found 0.156274] tun: Universal TUN/TAP device driver, 1.6 0.157291] thunder_xcv, ver 1.0 0.157343] thunder_bgx, ver 1.0 0.157377] nicpf, ver 1.0 0.157604] e1000: Intel(R) PRO/1000 Network Driver 0.157608] e1000: Copyright (c) 1999-2006 Intel Corporation. 0.157654] e1000e: Intel(R) PRO/1000 Network Driver 0.157657] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. 0.157690] igb: Intel(R) Gigabit Ethernet Network Driver 0.157694] igb: Copyright (c) 2007-2014 Intel Corporation. 0.157730] igbvf: Intel(R) Gigabit Virtual Function Network Driver 0.157734] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. 0.157865] sky2: driver version 1.30 0.158764] VFIO - User Level meta-driver version: 0.3 0.159848] usbcore: registered new interface driver usb-storage 0.160646] [egalax_i2c]:  Driver init done! 0.161031] i2c_dev: i2c /dev entries driver 0.162957] sdhci: Secure Digital Host Controller Interface driver 0.162972] sdhci: Copyright(c) Pierre Ossman 0.163430] sdhci-pltfm: SDHCI platform and OF driver helper 0.164591] ledtrig-cpu: registered to indicate activity on CPUs 0.164850] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping .... 0.165307] usbcore: registered new interface driver usbhid 0.165314] usbhid: USB HID core driver 0.166456] optee: probing for conduit method. 0.166504] optee: revision 4.0 (2a5b1d12) 0.183370] optee: dynamic shared memory is enabled 0.183822] optee: initialized driver 0.186304] Initializing XFRM netlink socket 0.186356] NET: Registered PF_PACKET protocol family 13.934513] gpio gpiochip0: Persistence not supported for GPIO 12 13.945813] gpio-361 (sysfs): gpiod_request: status -16 13.945851] export_store: status -16 13.956483] gpio-432 (sysfs): gpiod_request: status -16 13.956521] export_store: status -16 14.076897] [drm] Initialized tidss 1.0.0 20180215 for 30200000.dss on minor 0 14.088372] Console: switching to colour dummy device 80x25 14.107616] Console: switching to colour frame buffer device 240x67 14.114113] tidss 30200000.dss: [drm] fb0: tidssdrmfb frame buffer device 14.260273] PVR_K:  219: Device: fd00000.gpu 14.322884] remoteproc remoteproc1: attaching to 78000000.r5f 14.339248] PVR_K:  219: Read BVNC 33.15.11.3 from HW device registers 14.433065] platform 78000000.r5f: R5F core initialized in IPC-only mode 14.459547] PVR_K:  219: RGX Device registered with BVNC 33.15.11.3 14.472679] [drm] Initialized pvr 23.2.6460340 20170530 for fd00000.gpu on minor 1 14.538517] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@9da00000 14.632278] virtio_rpmsg_bus virtio0: rpmsg host is online 14.638094] virtio_rpmsg_bus virtio0: creating channel ti.ipc4.ping-pong addr 0xd 14.646062] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0xe 14.734728] gpio gpiochip0: Persistence not supported for GPIO 35 14.750775] gpio gpiochip0: Persistence not supported for GPIO 40 14.833337] rproc-virtio rproc-virtio.1.auto: registered virtio0 (type 7) 14.925422] remoteproc remoteproc1: remote processor 78000000.r5f is now attached 15.072086] asoc-simple-card sound: GPIO lookup for consumer simple-audio-card,hp-det 15.072126] asoc-simple-card sound: using device tree for GPIO lookup 15.072152] of_get_named_gpiod_flags: can't parse 'simple-audio-card,hp-det-gpios' property of node '/sound[0]' 15.072182] of_get_named_gpiod_flags: can't parse 'simple-audio-card,hp-det-gpio' property of node '/sound[0]' 15.072201] asoc-simple-card sound: using lookup tables for GPIO lookup 15.072207] asoc-simple-card sound: No GPIO consumer simple-audio-card,hp-det found 15.072214] asoc-simple-card sound: GPIO lookup for consumer simple-audio-card,mic-det 15.072219] asoc-simple-card sound: using device tree for GPIO lookup 15.072231] of_get_named_gpiod_flags: can't parse 'simple-audio-card,mic-det-gpios' property of node '/sound[0]' 15.072244] of_get_named_gpiod_flags: can't parse 'simple-audio-card,mic-det-gpio' property of node '/sound[0]' 15.072254] asoc-simple-card sound: using lookup tables for GPIO lookup 15.072259] asoc-simple-card sound: No GPIO consumer simple-audio-card,mic-det found 16.917836] audit: type=1334 audit(1728296489.516:6): prog-id=9 op=LOAD 16.932307] audit: type=1334 audit(1728296489.532:7): prog-id=10 op=LOAD 16.944255] audit: type=1334 audit(1728296489.536:8): prog-id=11 op=LOAD 16.951093] audit: type=1334 audit(1728296489.536:9): prog-id=12 op=LOAD 17.008943] audit: type=1334 audit(1728296489.608:10): prog-id=13 op=LOAD 17.018452] audit: type=1334 audit(1728296489.612:11): prog-id=14 op=LOAD 18.809226] audit: type=1334 audit(1728296491.408:12): prog-id=8 op=UNLOAD 18.816291] audit: type=1334 audit(1728296491.408:13): prog-id=7 op=UNLOAD 18.832157] audit: type=1334 audit(1728296491.424:14): prog-id=6 op=UNLOAD 18.844191] audit: type=1334 audit(1728296491.424:15): prog-id=5 op=UNLOAD 23.065233] gpio-455 (sysfs): gpiod_request: status -16 23.065281] export_store: status -16 23.065400] gpio-455 (sysfs): gpiod_request: status -16 23.065409] export_store: status -16 23.066776] gpio-460 (sysfs): gpiod_request: status -16 23.066808] export_store: status -16 23.066888] gpio-460 (sysfs): gpiod_request: status -16 23.066896] export_store: status -16 24.839693] remoteproc remoteproc2: 30074000.pru is available 24.849494] remoteproc remoteproc3: 30078000.pru is available 25.342827] cdns-csi2rx: probe of 30101000.csi-bridge failed with error -22 26.183063] xhci-hcd xhci-hcd.2.auto: xHCI Host Controller 26.232216] xhci-hcd xhci-hcd.2.auto: new USB bus registered, assigned bus number 1 26.243723] xhci-hcd xhci-hcd.2.auto: USB3 root hub has no ports 26.260139] xhci-hcd xhci-hcd.2.auto: hcc params 0x0258fe6d hci version 0x110 quirks 0x0000000000010010 26.309122] xhci-hcd xhci-hcd.2.auto: irq 442, io mem 0x31000000 26.330272] hub 1-0:1.0: USB hub found 26.335787] hub 1-0:1.0: 1 port detected 26.346104] xhci-hcd xhci-hcd.3.auto: xHCI Host Controller 26.353545] xhci-hcd xhci-hcd.3.auto: new USB bus registered, assigned bus number 2 26.363426] xhci-hcd xhci-hcd.3.auto: USB3 root hub has no ports 26.369601] xhci-hcd xhci-hcd.3.auto: hcc params 0x0258fe6d hci version 0x110 quirks 0x0000000000010010 26.380822] xhci-hcd xhci-hcd.3.auto: irq 443, io mem 0x31100000 26.398532] hub 2-0:1.0: USB hub found 26.411040] hub 2-0:1.0: 1 port detected 26.676224] usb 2-1: new high-speed USB device number 2 using xhci-hcd 26.900622] hub 2-1:1.0: USB hub found 26.904955] hub 2-1:1.0: 3 ports detected 27.256175] usb 2-1.3: new high-speed USB device number 3 using xhci-hcd 27.627431] usbcore: registered new interface driver cdc_wdm 27.684461] usbcore: registered new interface driver option 27.700941] usbserial: USB Serial support registered for GSM modem (1-port) 27.711249] qmi_wwan 2-1.3:1.4: cdc-wdm0: USB WDM device 27.735008] qmi_wwan 2-1.3:1.4 wwan0: register 'qmi_wwan' at usb-xhci-hcd.3.auto-1.3, WWAN/QMI device, 9a:c8:13:be:c3:25 27.755768] usbcore: registered new interface driver qmi_wwan_q 27.755768] option 2-1.3:1.0: GSM modem (1-port) converter detected 27.768605] usb 2-1.3: GSM modem (1-port) converter now attached to ttyUSB0 27.781966] usbcore: registered new interface driver qmi_wwan 27.802495] option 2-1.3:1.1: GSM modem (1-port) converter detected 27.813783] usb 2-1.3: GSM modem (1-port) converter now attached to ttyUSB1 27.823015] option 2-1.3:1.2: GSM modem (1-port) converter detected 27.841341] usb 2-1.3: GSM modem (1-port) converter now attached to ttyUSB2 27.850586] option 2-1.3:1.3: GSM modem (1-port) converter detected 27.865178] usb 2-1.3: GSM modem (1-port) converter now attached to ttyUSB3 34.425573] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Quota mode: none. 39.107420] remoteproc remoteproc2: powering up 30074000.pru 39.115902] remoteproc remoteproc2: Booting fw image am62x-pru0-fw, size 63988 39.127786] virtio_rpmsg_bus virtio1: rpmsg host is online 39.135180] rproc-virtio rproc-virtio.4.auto: registered virtio1 (type 7) 39.142278] remoteproc remoteproc2: remote processor 30074000.pru is now up 39.267796] remoteproc remoteproc3: powering up 30078000.pru 39.280570] remoteproc remoteproc3: Booting fw image am62x-pru1-fw, size 63924 39.304511] virtio_rpmsg_bus virtio2: rpmsg host is online 39.311855] rproc-virtio rproc-virtio.5.auto: registered virtio2 (type 7) 39.322338] remoteproc remoteproc3: remote processor 30078000.pru is now up 39.992613] kauditd_printk_skb: 2 callbacks suppressed 39.992642] audit: type=1334 audit(1728296512.592:18): prog-id=15 op=LOAD 40.004845] audit: type=1334 audit(1728296512.596:19): prog-id=16 op=LOAD 40.460694] audit: type=1006 audit(1728296513.060:20): pid=640 uid=0 old-auid=4294967295 auid=1000 tty=(none) old-ses=4294967295 ses=1 res=1 40.475168] audit: type=1300 audit(1728296513.060:20): arch=c00000b7 syscall=64 success=yes exit=4 a0=8 a1=ffffdcb3d2b8 a2=4 a3=ffff95aef020 items=0 ppid=1 pid=640 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="(systemd)" exe="/usr/rootdirs/lib/systemd/systemd" key=(null) 40.503536] audit: type=1327 audit(1728296513.060:20): proctitle="(systemd)" 41.074444] audit: type=1006 audit(1728296513.672:21): pid=633 uid=0 old-auid=4294967295 auid=1000 tty=tty7 old-ses=4294967295 ses=2 res=1 41.097698] audit: type=1300 audit(1728296513.672:21): arch=c00000b7 syscall=64 success=yes exit=4 a0=8 a1=ffffdcb3d2c8 a2=4 a3=ffff95aef020 items=0 ppid=1 pid=633 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=tty7 ses=2 comm="(test)" exe="/usr/rootdirs/lib/systemd/systemd" key=(null) 41.137259] audit: type=1327 audit(1728296513.672:21): proctitle="(test)" 41.234580] audit: type=1006 audit(1728296513.832:22): pid=647 uid=0 old-auid=4294967295 auid=1000 tty=tty7 old-ses=4294967295 ses=3 res=1 41.248674] audit: type=1300 audit(1728296513.832:22): arch=c00000b7 syscall=64 success=yes exit=4 a0=8 a1=ffffdcb3d2c8 a2=4 a3=ffff95aef020 items=0 ppid=1 pid=647 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=tty7 ses=3 comm="(weston)" exe="/usr/rootdirs/lib/systemd/systemd" key=(null) 41.664374] PVR_K:  647: RGX Firmware image 'rgx.fw.33.15.11.3' loaded 41.685649] PVR_K:  647: Shader binary image 'rgx.sh.33.15.11.3' loaded 65.487953] kauditd_printk_skb: 1 callbacks suppressed 65.494304] audit: type=1334 audit(1728296538.084:23): prog-id=17 op=LOAD 65.506994] audit: type=1334 audit(1728296538.100:24): prog-id=18 op=LOAD 0.186454] Key type dns_resolver registered 0.186993] registered taskstats version 1 0.187030] Loading compiled-in X.509 certificates 0.198916] ti-sci 44043000.system-controller: ABI: 3.1 (firmware rev 0x0009 '9.1.8--v09.01.08 (Kool Koala)') 0.242705] Freeing initrd memory: 2932K 0.256293] i2c i2c-0: GPIO lookup for consumer scl 0.256313] i2c i2c-0: using device tree for GPIO lookup 0.256330] of_get_named_gpiod_flags: can't parse 'scl-gpios' property of node '/bus@f0000/i2c@20000000[0]' 0.256349] of_get_named_gpiod_flags: can't parse 'scl-gpio' property of node '/bus@f0000/i2c@20000000[0]' 0.256365] i2c i2c-0: using lookup tables for GPIO lookup 0.256370] i2c i2c-0: No GPIO consumer scl found 0.256388] i2c i2c-0: GPIO lookup for consumer sda 0.256392] i2c i2c-0: using device tree for GPIO lookup 0.256400] of_get_named_gpiod_flags: can't parse 'sda-gpios' property of node '/bus@f0000/i2c@20000000[0]' 0.256413] of_get_named_gpiod_flags: can't parse 'sda-gpio' property of node '/bus@f0000/i2c@20000000[0]' 0.256425] i2c i2c-0: using lookup tables for GPIO lookup 0.256429] i2c i2c-0: No GPIO consumer sda found 0.256832] omap_i2c 20000000.i2c: bus 0 rev0.12 at 400 kHz 0.257891] i2c i2c-1: GPIO lookup for consumer scl 0.257905] i2c i2c-1: using device tree for GPIO lookup 0.257916] of_get_named_gpiod_flags: can't parse 'scl-gpios' property of node '/bus@f0000/i2c@20010000[0]' 0.257931] of_get_named_gpiod_flags: can't parse 'scl-gpio' property of node '/bus@f0000/i2c@20010000[0]' 0.257945] i2c i2c-1: using lookup tables for GPIO lookup 0.257950] i2c i2c-1: No GPIO consumer scl found 0.257970] i2c i2c-1: GPIO lookup for consumer sda 0.257974] i2c i2c-1: using device tree for GPIO lookup 0.257982] of_get_named_gpiod_flags: can't parse 'sda-gpios' property of node '/bus@f0000/i2c@20010000[0]' 0.257995] of_get_named_gpiod_flags: can't parse 'sda-gpio' property of node '/bus@f0000/i2c@20010000[0]' 0.258006] i2c i2c-1: using lookup tables for GPIO lookup 0.258010] i2c i2c-1: No GPIO consumer sda found 0.258802] omap_i2c 20010000.i2c: bus 1 rev0.12 at 400 kHz 0.259315] ti-sci-intr bus@f0000:interrupt-controller@a00000: Interrupt Router 3 domain created 0.259648] ti-sci-inta 48000000.interrupt-controller: Interrupt Aggregator domain 28 created 0.260526] simple-framebuffer ff700000.framebuffer: framebuffer at 0xff700000, 0x8ca000 bytes 0.260540] simple-framebuffer ff700000.framebuffer: format=x8r8g8b8, mode=1024x600x32, linelength=4096 0.264642] Console: switching to colour frame buffer device 128x37 0.272097] simple-framebuffer ff700000.framebuffer: fb0: simplefb registered! 0.273654] ti-udma 485c0100.dma-controller: Number of rings: 82 0.275738] ti-udma 485c0100.dma-controller: Channels: 48 (bchan: 18, tchan: 12, rchan: 18) 0.278418] ti-udma 485c0000.dma-controller: Number of rings: 150 0.282354] ti-udma 485c0000.dma-controller: Channels: 35 (tchan: 20, rchan: 15) 0.285103] printk: console [ttyS2] disabled 0.285133] omap8250 2800000.serial: GPIO lookup for consumer rs485-term 0.285140] omap8250 2800000.serial: using device tree for GPIO lookup 0.285152] of_get_named_gpiod_flags: can't parse 'rs485-term-gpios' property of node '/bus@f0000/serial@2800000[0]' 0.285175] of_get_named_gpiod_flags: can't parse 'rs485-term-gpio' property of node '/bus@f0000/serial@2800000[0]' 0.285191] omap8250 2800000.serial: using lookup tables for GPIO lookup 0.285196] omap8250 2800000.serial: No GPIO consumer rs485-term found 0.285230] 2800000.serial: ttyS2 at MMIO 0x2800000 (irq = 238, base_baud = 3000000) is a 8250 1.656896] printk: console [ttyS2] enabled 1.662418] omap8250 2820000.serial: GPIO lookup for consumer rs485-term 1.662436] omap8250 2820000.serial: using device tree for GPIO lookup 1.662449] of_get_named_gpiod_flags: can't parse 'rs485-term-gpios' property of node '/bus@f0000/serial@2820000[0]' 1.662466] of_get_named_gpiod_flags: can't parse 'rs485-term-gpio' property of node '/bus@f0000/serial@2820000[0]' 1.662483] omap8250 2820000.serial: using lookup tables for GPIO lookup 1.662489] omap8250 2820000.serial: No GPIO consumer rs485-term found 1.662520] 2820000.serial: ttyS0 at MMIO 0x2820000 (irq = 240, base_baud = 3000000) is a 8250 1.672194] omap8250 2840000.serial: GPIO lookup for consumer rs485-term 1.672210] omap8250 2840000.serial: using device tree for GPIO lookup 1.672224] of_get_named_gpiod_flags: can't parse 'rs485-term-gpios' property of node '/bus@f0000/serial@2840000[0]' 1.672241] of_get_named_gpiod_flags: can't parse 'rs485-term-gpio' property of node '/bus@f0000/serial@2840000[0]' 1.672254] omap8250 2840000.serial: using lookup tables for GPIO lookup 1.672260] omap8250 2840000.serial: No GPIO consumer rs485-term found 1.672286] 2840000.serial: ttyS4 at MMIO 0x2840000 (irq = 241, base_baud = 3000000) is a 8250 1.681965] omap8250 2860000.serial: GPIO lookup for consumer rs485-term 1.681982] omap8250 2860000.serial: using device tree for GPIO lookup 1.681998] of_get_named_gpiod_flags: can't parse 'rs485-term-gpios' property of node '/bus@f0000/serial@2860000[0]' 1.682014] of_get_named_gpiod_flags: can't parse 'rs485-term-gpio' property of node '/bus@f0000/serial@2860000[0]' 1.682026] omap8250 2860000.serial: using lookup tables for GPIO lookup 1.682032] omap8250 2860000.serial: No GPIO consumer rs485-term found 1.682056] 2860000.serial: ttyS1 at MMIO 0x2860000 (irq = 242, base_baud = 3000000) is a 8250 1.691930] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode 1.698804] mdio_bus 8000f00.mdio: GPIO lookup for consumer reset 1.698813] mdio_bus 8000f00.mdio: using device tree for GPIO lookup 1.698826] of_get_named_gpiod_flags: can't parse 'reset-gpios' property of node '/bus@f0000/ethernet@8000000/mdio@f00[0]' 1.698843] of_get_named_gpiod_flags: can't parse 'reset-gpio' property of node '/bus@f0000/ethernet@8000000/mdio@f00[0]' 1.698859] mdio_bus 8000f00.mdio: using lookup tables for GPIO lookup 1.698870] mdio_bus 8000f00.mdio: No GPIO consumer reset found 1.735963] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000 1.747983] mdio_bus 8000f00.mdio:01: GPIO lookup for consumer reset 1.748024] mdio_bus 8000f00.mdio:01: using device tree for GPIO lookup 1.748042] of_get_named_gpiod_flags: can't parse 'reset-gpios' property of node '/bus@f0000/ethernet@8000000/mdio@f00/ethernet-phy@1[0]' 1.748074] of_get_named_gpiod_flags: can't parse 'reset-gpio' property of node '/bus@f0000/ethernet@8000000/mdio@f00/ethernet-phy@1[0]' 1.748094] mdio_bus 8000f00.mdio:01: using lookup tables for GPIO lookup 1.748100] mdio_bus 8000f00.mdio:01: No GPIO consumer reset found 1.751861] mdio_bus 8000f00.mdio:03: GPIO lookup for consumer reset 1.751890] mdio_bus 8000f00.mdio:03: using device tree for GPIO lookup 1.751906] of_get_named_gpiod_flags: can't parse 'reset-gpios' property of node '/bus@f0000/ethernet@8000000/mdio@f00/ethernet-phy@3[0]' 1.751936] of_get_named_gpiod_flags: can't parse 'reset-gpio' property of node '/bus@f0000/ethernet@8000000/mdio@f00/ethernet-phy@3[0]' 1.751997] mdio_bus 8000f00.mdio:03: using lookup tables for GPIO lookup 1.752004] mdio_bus 8000f00.mdio:03: No GPIO consumer reset found 1.752376] davinci_mdio 8000f00.mdio: phy[1]: device 8000f00.mdio:01, driver ICPlus IP101G 1.760729] davinci_mdio 8000f00.mdio: phy[3]: device 8000f00.mdio:03, driver TI DP83826E 1.768944] am65-cpsw-nuss 8000000.ethernet: initializing am65 cpsw nuss version 0x6BA01103, cpsw version 0x6BA81103 Ports: 3 quirks:00000006 1.781847] Attempting to get eFuse MAC ID 1.785947] Retrieved MAC address: c0:d6:0a:e2:5f:c7 1.791003] Attempting to get eFuse MAC ID 1.795094] Retrieved MAC address: 00:00:00:00:00:00 1.800055] Invalid MAC address, incrementing 1.804408] New MAC address: c0:d6:0a:e2:5f:c8 1.808845] am65-cpsw-nuss 8000000.ethernet: Use incremented MAC address 1.815540] am65-cpsw-nuss 8000000.ethernet: initialized cpsw ale version 1.5 1.822668] am65-cpsw-nuss 8000000.ethernet: ALE Table size 512 1.829482] am65-cpsw-nuss 8000000.ethernet: CPTS ver 0x4e8a010c, freq:500000000, add_val:1 pps:0 1.840213] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19 1.851553] sdhci-am654 fa00000.mmc: GPIO lookup for consumer cd 1.851580] sdhci-am654 fa00000.mmc: using device tree for GPIO lookup 1.851600] of_get_named_gpiod_flags: can't parse 'cd-gpios' property of node '/bus@f0000/mmc@fa00000[0]' 1.851629] of_get_named_gpiod_flags: can't parse 'cd-gpio' property of node '/bus@f0000/mmc@fa00000[0]' 1.851644] sdhci-am654 fa00000.mmc: using lookup tables for GPIO lookup 1.851650] sdhci-am654 fa00000.mmc: No GPIO consumer cd found 1.851657] sdhci-am654 fa00000.mmc: GPIO lookup for consumer wp 1.851662] sdhci-am654 fa00000.mmc: using device tree for GPIO lookup 1.851671] of_get_named_gpiod_flags: can't parse 'wp-gpios' property of node '/bus@f0000/mmc@fa00000[0]' 1.851687] of_get_named_gpiod_flags: can't parse 'wp-gpio' property of node '/bus@f0000/mmc@fa00000[0]' 1.851699] sdhci-am654 fa00000.mmc: using lookup tables for GPIO lookup 1.851704] sdhci-am654 fa00000.mmc: No GPIO consumer wp found 1.852114] mmc1: CQHCI version 5.10 1.852254] sdhci-am654 fa10000.mmc: GPIO lookup for consumer cd 1.855706] sdhci-am654 fa10000.mmc: using device tree for GPIO lookup 1.855718] of_get_named_gpiod_flags: can't parse 'cd-gpios' property of node '/bus@f0000/mmc@fa10000[0]' 1.855740] of_get_named_gpiod_flags: can't parse 'cd-gpio' property of node '/bus@f0000/mmc@fa10000[0]' 1.855753] sdhci-am654 fa10000.mmc: using lookup tables for GPIO lookup 1.855763] sdhci-am654 fa10000.mmc: No GPIO consumer cd found 1.855770] sdhci-am654 fa10000.mmc: GPIO lookup for consumer wp 1.855775] sdhci-am654 fa10000.mmc: using device tree for GPIO lookup 1.855784] of_get_named_gpiod_flags: can't parse 'wp-gpios' property of node '/bus@f0000/mmc@fa10000[0]' 1.855798] of_get_named_gpiod_flags: can't parse 'wp-gpio' property of node '/bus@f0000/mmc@fa10000[0]' 1.855813] sdhci-am654 fa10000.mmc: using lookup tables for GPIO lookup 1.855818] sdhci-am654 fa10000.mmc: No GPIO consumer wp found 1.856257] mmc0: CQHCI version 5.10 1.860453] gpiochip_find_base: found new base at 420 1.860698] gpio gpiochip0: (600000.gpio): added GPIO chardev (254:0) 1.860765] gpio gpiochip0: registered GPIOs 420 to 511 on 600000.gpio 1.863490] gpiochip_find_base: found new base at 368 1.863705] gpio gpiochip1: (601000.gpio): added GPIO chardev (254:1) 1.863765] gpio gpiochip1: registered GPIOs 368 to 419 on 601000.gpio 1.865277] reg-fixed-voltage regulator-7: GPIO lookup for consumer (null) 1.865293] reg-fixed-voltage regulator-7: using device tree for GPIO lookup 1.865316] of_get_named_gpiod_flags: parsed 'gpios' property of node '/regulator-7[0]' - status (0) 1.865340] gpio gpiochip0: Persistence not supported for GPIO 71 1.865731] gpio-regulator regulator-4: GPIO lookup for consumer (null) 1.865740] gpio-regulator regulator-4: using device tree for GPIO lookup 1.865754] of_get_named_gpiod_flags: parsed 'gpios' property of node '/regulator-4[0]' - status (0) 1.865768] gpio gpiochip0: Persistence not supported for GPIO 31 1.865779] gpio-regulator regulator-4: GPIO lookup for consumer enable 1.865785] gpio-regulator regulator-4: using device tree for GPIO lookup 1.865795] of_get_named_gpiod_flags: can't parse 'enable-gpios' property of node '/regulator-4[0]' 1.865808] of_get_named_gpiod_flags: can't parse 'enable-gpio' property of node '/regulator-4[0]' 1.865822] gpio-regulator regulator-4: using lookup tables for GPIO lookup 1.865827] gpio-regulator regulator-4: No GPIO consumer enable found 1.888250] bq32k 0-0068: invalid resistor value (3000) 1.894027] bq32k 0-0068: registered as rtc0 1.898322] mmc0: SDHCI controller on fa10000.mmc [fa10000.mmc] using ADMA 64-bit 1.898597] bq32k 0-0068: setting system clock to 2024-10-07T10:21:14 UTC (1728296474) 1.913924] pca953x 0-0020: GPIO lookup for consumer reset 1.913931] pca953x 0-0020: using device tree for GPIO lookup 1.913942] of_get_named_gpiod_flags: can't parse 'reset-gpios' property of node '/bus@f0000/i2c@20000000/gpio@20[0]' 1.913959] of_get_named_gpiod_flags: can't parse 'reset-gpio' property of node '/bus@f0000/i2c@20000000/gpio@20[0]' 1.913975] pca953x 0-0020: using lookup tables for GPIO lookup 1.913983] pca953x 0-0020: No GPIO consumer reset found 1.914007] pca953x 0-0020: supply vcc not found, using dummy regulator 1.920826] pca953x 0-0020: using no AI 1.925803] gpiochip_find_base: found new base at 352 1.926180] gpio gpiochip2: (0-0020): added GPIO chardev (254:2) 1.926291] gpio gpiochip2: registered GPIOs 352 to 367 on 0-0020 1.928917] [egalax_i2c]:  Start probe 1.928967] of_get_named_gpiod_flags: parsed 'int-gpios' property of node '/bus@f0000/i2c@20010000/touchscreen@2a[0]' - status (0) 1.929312] input: eGalax_Touch_Screen as /devices/virtual/input/input0 1.936488] [egalax_i2c]:  Register input device done 1.936594] input: eGalax_Touch_Pen as /devices/virtual/input/input1 1.943107] [egalax_i2c]:  Register input device pen done 1.943353] [egalax_i2c]:  Request irq(273) gpio(439) with result:0 1.943371] [egalax_i2c]:  I2C probe done 1.943625] pca953x 1-0021: GPIO lookup for consumer reset 1.943632] pca953x 1-0021: using device tree for GPIO lookup 1.943643] of_get_named_gpiod_flags: can't parse 'reset-gpios' property of node '/bus@f0000/i2c@20010000/gpio@21[0]' 1.943662] of_get_named_gpiod_flags: can't parse 'reset-gpio' property of node '/bus@f0000/i2c@20010000/gpio@21[0]' 1.943677] pca953x 1-0021: using lookup tables for GPIO lookup 1.943683] pca953x 1-0021: No GPIO consumer reset found 1.943814] input: PCA955x Button as /devices/platform/bus@f0000/20010000.i2c/i2c-1/1-0021/input/input2 1.953484] pca953x 1-0021: supply vcc not found, using dummy regulator 1.960238] pca953x 1-0021: using no AI 1.989077] gpiochip_find_base: found new base at 336 1.989133] gpio gpiochip3: Persistence not supported for GPIO 0 1.989251] gpio-336 (Bos1): hogged as input 1.993556] gpio gpiochip3: Persistence not supported for GPIO 1 1.993669] gpio-337 (Bos2): hogged as input 1.998003] gpio gpiochip3: Persistence not supported for GPIO 2 1.998121] gpio-338 (Bos3): hogged as input 2.002425] gpio gpiochip3: Persistence not supported for GPIO 3 2.002532] gpio-339 (SW1): hogged as input 2.006727] gpio gpiochip3: Persistence not supported for GPIO 4 2.006834] gpio-340 (SW2): hogged as input 2.011034] gpio gpiochip3: Persistence not supported for GPIO 5 2.011142] gpio-341 (SW3): hogged as input 2.015331] gpio gpiochip3: Persistence not supported for GPIO 6 2.015440] gpio-342 (SW4): hogged as input 2.019657] gpio gpiochip3: Persistence not supported for GPIO 7 2.019765] gpio-343 (Bos4): hogged as input 2.024095] gpio gpiochip3: Persistence not supported for GPIO 8 2.024203] gpio-344 (LED1_P1): hogged as input 2.028755] gpio gpiochip3: Persistence not supported for GPIO 9 2.028874] gpio-345 (LED1_P2): hogged as input 2.033683] gpio gpiochip3: Persistence not supported for GPIO 10 2.033791] gpio-346 (LED2_P1): hogged as input 2.038336] gpio gpiochip3: Persistence not supported for GPIO 11 2.038450] gpio-347 (LED2_P2): hogged as input 2.043261] gpio gpiochip3: Persistence not supported for GPIO 12 2.043374] gpio-348 (LED3_P1): hogged as input 2.047926] gpio gpiochip3: Persistence not supported for GPIO 13 2.048064] gpio-349 (LED3_P2): hogged as input 2.048079] mmc0: Command Queue Engine enabled 2.057047] mmc0: new high speed MMC card at address 0001 2.062598] gpio gpiochip3: Persistence not supported for GPIO 14 2.062706] gpio-350 (LED4_P1): hogged as input 2.063278] mmcblk0: mmc0:0001 8GTF4R 7.28 GiB 2.067271] gpio gpiochip3: Persistence not supported for GPIO 15 2.071933] gpio-351 (LED4_P2): hogged as input 2.076598]  mmcblk0: p1 p2 2.079844] gpio gpiochip3: (1-0021): added GPIO chardev (254:3) 2.079916] gpio gpiochip3: registered GPIOs 336 to 351 on 1-0021 2.080182] mmcblk0boot0: mmc0:0001 8GTF4R 4.00 MiB 2.082792] omap8250 2810000.serial: GPIO lookup for consumer rs485-term 2.085261] omap8250 2810000.serial: using device tree for GPIO lookup 2.085293] of_get_named_gpiod_flags: can't parse 'rs485-term-gpios' property of node '/bus@f0000/serial@2810000[0]' 2.085322] of_get_named_gpiod_flags: can't parse 'rs485-term-gpio' property of node '/bus@f0000/serial@2810000[0]' 2.085346] omap8250 2810000.serial: using lookup tables for GPIO lookup 2.085352] omap8250 2810000.serial: No GPIO consumer rs485-term found 2.085371] omap8250 2810000.serial: GPIO lookup for consumer rts 2.085376] omap8250 2810000.serial: using device tree for GPIO lookup 2.085398] of_get_named_gpiod_flags: parsed 'rts-gpios' property of node '/bus@f0000/serial@2810000[0]' - status (0) 2.085420] gpio gpiochip1: Persistence not supported for GPIO 8 2.085540] 2810000.serial: ttyS3 at MMIO 0x2810000 (irq = 402, base_baud = 3000000) is a 8250 2.086311] mmcblk0boot1: mmc0:0001 8GTF4R 4.00 MiB 2.095520] omap8250 2850000.serial: GPIO lookup for consumer rs485-term 2.099174] omap8250 2850000.serial: using device tree for GPIO lookup 2.099187] of_get_named_gpiod_flags: can't parse 'rs485-term-gpios' property of node '/bus@f0000/serial@2850000[0]' 2.099204] of_get_named_gpiod_flags: can't parse 'rs485-term-gpio' property of node '/bus@f0000/serial@2850000[0]' 2.099236] omap8250 2850000.serial: using lookup tables for GPIO lookup 2.099246] omap8250 2850000.serial: No GPIO consumer rs485-term found 2.099272] omap8250 2850000.serial: GPIO lookup for consumer rts 2.099281] omap8250 2850000.serial: using device tree for GPIO lookup 2.099324] of_get_named_gpiod_flags: parsed 'rts-gpios' property of node '/bus@f0000/serial@2850000[0]' - status (0) 2.099345] gpio gpiochip0: Persistence not supported for GPIO 1 2.099372] 2850000.serial: ttyS5 at MMIO 0x2850000 (irq = 403, base_baud = 3000000) is a 8250 2.100210] mmcblk0rpmb: mmc0:0001 8GTF4R 512 KiB, chardev (240:0) 2.108900] of_get_named_gpiod_flags: parsed 'gpios' property of node '/leds/led-0[0]' - status (0) 2.114207] gpio gpiochip1: Persistence not supported for GPIO 49 2.114213] gpio-417 (?): no flags found for gpios 2.115521] debugfs: Directory 'pd:182' with parent 'pm_genpd' already present! 2.123437] sdhci-am654 fa20000.mmc: GPIO lookup for consumer wp 2.123463] sdhci-am654 fa20000.mmc: using device tree for GPIO lookup 2.123475] of_get_named_gpiod_flags: can't parse 'wp-gpios' property of node '/bus@f0000/mmc@fa20000[0]' 2.123500] of_get_named_gpiod_flags: can't parse 'wp-gpio' property of node '/bus@f0000/mmc@fa20000[0]' 2.123525] sdhci-am654 fa20000.mmc: using lookup tables for GPIO lookup 2.123531] sdhci-am654 fa20000.mmc: No GPIO consumer wp found 2.124006] mmc2: CQHCI version 5.10 2.125238] debugfs: Directory 'pd:186' with parent 'pm_genpd' already present! 2.146697] ALSA device list: 2.149694]   No soundcards found. 2.163556] mmc2: SDHCI controller on fa20000.mmc [fa20000.mmc] using ADMA 64-bit 3.369847] sdhci-am654 fa00000.mmc: Power on failed 3.405888] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit 3.416841] Freeing unused kernel memory: 2688K 3.421703] Run /init as init process 3.425428]   with arguments: 3.425437]     /init 3.425446]   with environment: 3.425453]     HOME=/ 3.425461]     TERM=linux 3.425469]     ostree=/ostree/boot.0/arago/2b5e4456444f16ac259c42646a91976b1a788f80a2e97186a33e7bd69f4dadc9/0 3.425481]     ostree_root=/dev/mmcblk0p1 3.617361] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Quota mode: none. 4.170916] NET: Registered PF_INET6 protocol family 4.177621] Segment Routing with IPv6 4.181387] In-situ OAM (IOAM) with IPv6 6.610129] fuse: init (API version 7.37) 6.738949] cryptodev: loading out-of-tree module taints kernel. 6.747274] cryptodev: driver 1.12 loaded. 6.835744] EXT4-fs (mmcblk0p1): re-mounted. Quota mode: none. 7.498198] audit: type=1334 audit(1728296480.096:2): prog-id=5 op=LOAD 7.508221] audit: type=1334 audit(1728296480.104:3): prog-id=6 op=LOAD 8.319343] random: crng init done 8.454308] pwm-backlight backlight: GPIO lookup for consumer enable 8.454342] pwm-backlight backlight: using device tree for GPIO lookup 8.454382] of_get_named_gpiod_flags: parsed 'enable-gpios' property of node '/backlight[0]' - status (0) 8.454416] gpio gpiochip0: Persistence not supported for GPIO 0 8.454424] gpio-420 (enable): no flags found for enable 8.454458] pwm-backlight backlight: supply power not found, using dummy regulator 8.628310] of_get_named_gpiod_flags: can't parse 'reset-gpios' property of node '/bus@f0000/i2c@20010000/audio-codec@1b[0]' 8.628369] of_get_named_gpiod_flags: can't parse 'gpio-reset' property of node '/bus@f0000/i2c@20010000/audio-codec@1b[0]' 8.633683] st_lsm6dsx_i2c 1-006a: supply vdd not found, using dummy regulator 8.678014] st_lsm6dsx_i2c 1-006a: supply vddio not found, using dummy regulator 8.769773] mc: Linux media interface: v0.10 8.864678] videodev: Linux video capture interface: v2.00 8.893828] audit: type=1334 audit(1728296481.492:4): prog-id=7 op=LOAD 8.908111] audit: type=1334 audit(1728296481.504:5): prog-id=8 op=LOAD 9.386092] st_lsm6dsx_i2c 1-006a: mounting matrix not found: using identity... 10.064130] EXT4-fs (mmcblk0p1): re-mounted. Quota mode: none. 10.301673] rtc-ti-k3 2b1f0000.rtc: registered as rtc1 10.315481] k3-m4-rproc 5000000.m4fss: assigned reserved memory node m4f-dma-memory@9cb00000 10.340726] rtc-ti-k3 2b1f0000.rtc: GPIO lookup for consumer wp 10.340762] rtc-ti-k3 2b1f0000.rtc: using device tree for GPIO lookup 10.340789] of_get_named_gpiod_flags: can't parse 'wp-gpios' property of node '/bus@f0000/bus@b00000/rtc@2b1f0000[0]' 10.340826] of_get_named_gpiod_flags: can't parse 'wp-gpio' property of node '/bus@f0000/bus@b00000/rtc@2b1f0000[0]' 10.340844] rtc-ti-k3 2b1f0000.rtc: using lookup tables for GPIO lookup 10.340851] rtc-ti-k3 2b1f0000.rtc: No GPIO consumer wp found 10.493095] cfg80211: Loading compiled-in X.509 certificates for regulatory database 10.525442] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' 10.749928] k3-m4-rproc 5000000.m4fss: configured M4 for remoteproc mode 10.821176] k3-m4-rproc 5000000.m4fss: local reset is deasserted for device 10.957410] am65-cpsw-nuss 8000000.ethernet eth1: PHY [8000f00.mdio:03] driver [TI DP83826E] (irq=POLL) 10.957446] am65-cpsw-nuss 8000000.ethernet eth1: configuring for phy/rmii link mode 11.023376] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:01] driver [ICPlus IP101G] (irq=POLL) 11.023421] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rmii link mode 11.408282] remoteproc remoteproc0: 5000000.m4fss is available 11.705140] remoteproc remoteproc0: Direct firmware load for am62-mcu-m4f0_0-fw failed with error -2 11.718937] remoteproc remoteproc0: powering up 5000000.m4fss 11.725065] remoteproc remoteproc0: Direct firmware load for am62-mcu-m4f0_0-fw failed with error -2 11.735737] remoteproc remoteproc0: request_firmware failed: -2 13.008822] panel-simple display: supply power not found, using dummy regulator 13.029644] am65-cpsw-nuss 8000000.ethernet eth1: Link is Up - 100Mbps/Full - flow control rx/tx 13.038681] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready 13.060575] am65-cpsw-nuss 8000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx 13.128189] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 13.308263] platform 78000000.r5f: R5F core may have been powered on by a different host, programmed state (0) != actual state (1) 13.645275] gpio gpiochip2: Persistence not supported for GPIO 0 13.660876] gpio gpiochip2: Persistence not supported for GPIO 1 13.669513] gpio gpiochip2: Persistence not supported for GPIO 2 13.685333] gpio gpiochip2: Persistence not supported for GPIO 3 13.695410] gpio gpiochip2: Persistence not supported for GPIO 4 13.712358] gpio gpiochip2: Persistence not supported for GPIO 5 13.713742] gpio gpiochip2: Persistence not supported for GPIO 6 13.724512] gpio gpiochip2: Persistence not supported for GPIO 7 13.737151] gpio gpiochip2: Persistence not supported for GPIO 8 13.770045] gpio gpiochip2: Persistence not supported for GPIO 9 13.786804] gpio gpiochip2: Persistence not supported for GPIO 10 13.810579] gpio gpiochip2: Persistence not supported for GPIO 11 13.811757] gpio gpiochip2: Persistence not supported for GPIO 12 13.814220] gpio gpiochip2: Persistence not supported for GPIO 13 13.816397] panel-simple display: GPIO lookup for consumer enable 13.816436] panel-simple display: using device tree for GPIO lookup 13.816477] of_get_named_gpiod_flags: parsed 'enable-gpios' property of node '/display[0]' - status (0) 13.816516] gpio gpiochip0: Persistence not supported for GPIO 4 13.835052] platform 78000000.r5f: configured R5F for IPC-only mode 13.852219] platform 78000000.r5f: assigned reserved memory node r5f-dma-memory@9da00000 13.865742] gpio gpiochip2: Persistence not supported for GPIO 14 13.905907] gpio gpiochip2: Persistence not supported for GPIO 15 13.933494] remoteproc remoteproc1: 78000000.r5f is available 66.977677] gpio gpiochip0: Persistence not supported for GPIO 41 96.241274] audit: type=1334 audit(1728296568.836:25): prog-id=18 op=UNLOAD 96.252616] audit: type=1334 audit(1728296568.836:26): prog-id=17 op=UNLOAD 337.800050] omap_i2c 20010000.i2c: controller timed out 339.332034] omap_i2c 20010000.i2c: controller timed out 340.868071] omap_i2c 20010000.i2c: controller timed out 342.404078] omap_i2c 20010000.i2c: controller timed out 343.940055] omap_i2c 20010000.i2c: controller timed out 345.476431] omap_i2c 20010000.i2c: controller timed out 348.584054] omap_i2c 20010000.i2c: controller timed out 351.720088] omap_i2c 20010000.i2c: controller timed out 354.856070] omap_i2c 20010000.i2c: controller timed out 357.924044] omap_i2c 20010000.i2c: controller timed out 360.996060] omap_i2c 20010000.i2c: controller timed out 364.104026] omap_i2c 20010000.i2c: controller timed out 367.204062] omap_i2c 20010000.i2c: controller timed out 370.276274] omap_i2c 20010000.i2c: controller timed out 373.380040] omap_i2c 20010000.i2c: controller timed out 376.452064] omap_i2c 20010000.i2c: controller timed out 379.524069] omap_i2c 20010000.i2c: controller timed out 347.044066] omap_i2c 20010000.i2c: controller timed out 350.152068] omap_i2c 20010000.i2c: controller timed out 353.288055] omap_i2c 20010000.i2c: controller timed out 356.388151] omap_i2c 20010000.i2c: controller timed out 357.727960] rcu: INFO: rcu_preempt self-detected stall on CPU 357.733753] rcu: 	0-....: (5264 ticks this GP) idle=b304/1/0x4000000000000000 softirq=23369/23371 fqs=2083 357.743422] 	(t=5250 jiffies g=33901 q=5361 ncpus=2) 357.748400] CPU: 0 PID: 194 Comm: systemd-journal Tainted: G           O       6.1.46-g247b2535b2 #1 357.757542] Hardware name: Texas Instruments AM625 SK (DT) 357.763039] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) 357.770005] pc : _stext+0xa4/0x2a4 357.773440] lr : _stext+0x6c/0x2a4 357.776851] sp : ffff800008003f60 357.780164] x29: ffff800008003f60 x28: ffff0000034b0000 x27: 0000000000000200 357.787323] x26: 0000000000000000 x25: 0000000000000000 x24: ffff80000914d000 357.794474] x23: 0000000020000000 x22: ffff800009153000 x21: ffff80000914d9c8 357.801627] x20: ffff8000092b60c0 x19: ffff0000034b0000 x18: 0000000000000000 357.808779] x17: ffff80002ec4b000 x16: ffff800008000000 x15: 00003d0900000000 357.815933] x14: 00000000000002b2 x13: 0000000000000001 x12: 0000000000000000 357.823083] x11: 0000000000000002 x10: 00000000000002b2 x9 : ffff8000092b9d38 357.830234] x8 : 0000004e65996200 x7 : 7fffffffffffffff x6 : 000001ef33d7c3e3 357.837385] x5 : 03ffffffffffffff x4 : 0000000000000015 x3 : ffff0000034b0000 357.844536] x2 : ffff80002ec4b000 x1 : 00000000000000e0 x0 : ffff800009153b00 357.851697] Call trace: 357.854147]  _stext+0xa4/0x2a4 357.857213]  ____do_softirq+0x10/0x20 357.860887]  call_on_irq_stack+0x24/0x4c 357.864821]  do_softirq_own_stack+0x1c/0x30 357.869015]  __irq_exit_rcu+0xcc/0xf4 357.872688]  irq_exit_rcu+0x10/0x20 357.876185]  el0_interrupt+0x50/0xdc 357.879773]  __el0_irq_handler_common+0x18/0x24 357.884312]  el0t_64_irq_handler+0x10/0x20 357.888417]  el0t_64_irq+0x18c/0x190 359.460202] omap_i2c 20010000.i2c: controller timed out 362.536056] omap_i2c 20010000.i2c: controller timed out 365.640050] omap_i2c 20010000.i2c: controller timed out 368.740096] omap_i2c 20010000.i2c: controller timed out 371.816034] omap_i2c 20010000.i2c: controller timed out 374.916058] omap_i2c 20010000.i2c: controller timed out 377.988026] omap_i2c 20010000.i2c: controller timed out 381.060051] omap_i2c 20010000.i2c: controller timed out 383.559995] omap_i2c 20010000.i2c: controller timed out 385.128007] omap_i2c 20010000.i2c: controller timed out 386.691994] omap_i2c 20010000.i2c: controller timed out 388.264003] omap_i2c 20010000.i2c: controller timed out

Additionally, there are logs from patch file you shared, as seen below:

Oct 7 06:11:23 kernel: [ 3126.499786] tidss 30200000.dss: lvl1 irq 0x1, lvl2 irqmask 0x0
Oct 7 06:16:48 kernel: [ 3451.875812] tidss 30200000.dss: lvl1 irq 0x1, lvl2 irqmask 0x0

Rcu-stall issue occured later on if you check the log timestamps.

Oct 7 10:27:10 kernel: [ 357.727960] rcu: INFO: rcu_preempt self-detected stall on CPU
Oct 7 10:27:10 kernel: [ 357.733753] rcu: 0-....: (5264 ticks this GP) idle=b304/1/0x4000000000000000 softirq=23369/23371 fqs=2083

Do you think your patch resolved the issue related with tidss ? There may be another module/part that causes rcu stall additionally ? Can you please check and feedback ?

Thanks in advance,
Veysel

0 Bin Liu 11 months ago in reply to Veysel Oguz

TI__Guru**** 168261 points

Hi Veysel,

Currently there is an known issue in tidss which could trigger rcu stall, but other customers reported my first patch above resolves the issue.

Since this patch doesn't fix in your case, it is likely the issue is caused by something else. To confirm this, can you please apply the following kernel patch and capture the kernel dmesg log until the rcu stall happens. The debug log should tell if the issue is caused by tidss or not.

Fullscreen 1884.kernel-61-dss-irq-dump.diff Download

diff --git a/drivers/gpu/drm/tidss/tidss_dispc.c b/drivers/gpu/drm/tidss/tidss_dispc.c
index 5bcc9153a977..92ceff0a049f 100644
--- a/drivers/gpu/drm/tidss/tidss_dispc.c
+++ b/drivers/gpu/drm/tidss/tidss_dispc.c
@@ -870,6 +870,21 @@ void dispc_k2g_set_irqenable(struct dispc_device *dispc, dispc_irq_t mask)
 	dispc_k2g_read_irqenable(dispc);
 }
 
+void dispc_dump_irq_regs(struct dispc_device *dispc, int pos)
+{
+	u32 reg[5];
+	char *prefix = pos ? "<<" : "__";
+
+	reg[0] = dispc_read(dispc, DISPC_IRQSTATUS);
+	reg[1] = dispc_read(dispc, DISPC_VID_IRQSTATUS(0));
+	reg[2] = dispc_read(dispc, DISPC_VID_IRQSTATUS(1));
+	reg[3] = dispc_read(dispc, DISPC_VP_IRQSTATUS(0));
+	reg[4] = dispc_read(dispc, DISPC_VP_IRQSTATUS(1));
+
+	pr_err("%s: irq 0x%x, vid0 0x%x, vid1 0x%x, vp0 0x%x, vp1 0x%x\n",
+			prefix, reg[0], reg[1], reg[2], reg[3], reg[4]);
+}
+
 static dispc_irq_t dispc_k3_vp_read_irqstatus(struct dispc_device *dispc,
 					      u32 vp_idx)
 {
diff --git a/drivers/gpu/drm/tidss/tidss_dispc.h b/drivers/gpu/drm/tidss/tidss_dispc.h
index dee647145d51..12e68dae8d01 100644
--- a/drivers/gpu/drm/tidss/tidss_dispc.h
+++ b/drivers/gpu/drm/tidss/tidss_dispc.h
@@ -154,5 +154,6 @@ int dispc_init(struct tidss_device *tidss);
 void dispc_remove(struct tidss_device *tidss);
 
 void dispc_splash_fini(struct dispc_device *dispc);
+void dispc_dump_irq_regs(struct dispc_device *dispc, int pos);
 
 #endif
diff --git a/drivers/gpu/drm/tidss/tidss_irq.c b/drivers/gpu/drm/tidss/tidss_irq.c
index a4f2757cb196..c2ec62652b5c 100644
--- a/drivers/gpu/drm/tidss/tidss_irq.c
+++ b/drivers/gpu/drm/tidss/tidss_irq.c
@@ -60,6 +60,8 @@ static irqreturn_t tidss_irq_handler(int irq, void *arg)
 	unsigned int id;
 	dispc_irq_t irqstatus;
 
+	dispc_dump_irq_regs(tidss->dispc, 0);
+
 	irqstatus = dispc_read_and_clear_irqstatus(tidss->dispc);
 
 	for (id = 0; id < tidss->num_crtcs; id++) {
@@ -81,6 +83,7 @@ static irqreturn_t tidss_irq_handler(int irq, void *arg)
 	if (irqstatus & DSS_IRQ_DEVICE_OCP_ERR)
 		dev_err_ratelimited(tidss->dev, "OCP error\n");
 
+	dispc_dump_irq_regs(tidss->dispc, 1);
 	return IRQ_HANDLED;
 }

0 Pawan S 11 months ago in reply to Bin Liu

Prodigy 180 points

Hi Bin,

Sorry to interrupt.
Facing similar issue.

Bin Liu said:
first patch above resolves the issue.

Is the issue solved in latest ti linux kernel.
if not where to find such major stalling issue's patches .

Thanks & regards,

Pawan

0 Veysel Oguz 11 months ago in reply to Bin Liu

Prodigy 210 points

Hi Bin,

We've implemented your latest patch for debugging. However there are very intense logs coming, not able to use terminal. Please check below as snapshot

[   52.621617] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.627436] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.638317] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.644168] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.655019] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.660856] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.671712] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.677557] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.688410] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.694207] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.705110] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.710910] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.721808] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.727609] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.738508] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.744309] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.755205] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.761003] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.771908] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.777702] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.788605] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.794405] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.805324] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.811125] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.822001] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.827796] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.838701] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.844499] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.855402] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.861192] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.872099] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.877897] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.888796] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.894592] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.905496] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.911290] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.922193] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.927993] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.938892] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.944696] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.955592] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.961393] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.972291] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.978092] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   52.988989] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   52.994791] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.005692] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.011474] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.022390] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.028191] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.034253] audit: type=1334 audit(1728483399.092:11): prog-id=11 op=LOAD
[   53.041169] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.041222] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.055784] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.061569] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.067764] audit: type=1334 audit(1728483399.108:12): prog-id=12 op=LOAD
[   53.074652] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.074691] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.086586] audit: type=1334 audit(1728483399.108:13): prog-id=3 op=UNLOAD
[   53.093486] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.093524] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.105880] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.111659] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.120252] audit: type=1334 audit(1728483399.108:14): prog-id=4 op=UNLOAD
[   53.127330] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.127380] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.139275] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.145055] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.155978] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.161762] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.172599] audit: type=1334 audit(1728483399.112:15): prog-id=13 op=LOAD
[   53.172672] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.185226] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.191093] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.196884] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.202818] audit: type=1334 audit(1728483399.120:16): prog-id=14 op=LOAD
[   53.209707] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.209753] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.221461] audit: type=1334 audit(1728483399.120:17): prog-id=5 op=UNLOAD
[   53.228409] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.228453] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.240056] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.240089] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.256171] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.261952] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.269721] audit: type=1334 audit(1728483399.120:18): prog-id=6 op=UNLOAD
[   53.276641] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.276687] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.289570] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.295354] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.305851] audit: type=1334 audit(1728483399.144:19): prog-id=15 op=LOAD
[   53.312674] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.312717] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.324352] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.324392] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.339664] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.345450] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.356367] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.362153] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.368120] audit: type=1334 audit(1728483399.144:20): prog-id=16 op=LOAD
[   53.375105] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.375155] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.389760] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.395560] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.406460] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.412244] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.423158] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.428959] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.439862] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.445666] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.456555] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.462356] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
***************************************************************[   53.473255] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.483966] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0

**************************************************************[   53.489951] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.501185] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
*
NOTICE: This file system contains the following GPL-3.0 packa[   53.507058] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.518319] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
ges:
[   53.524158] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.530414] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.540050] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.545850] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.556748] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.562552] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.573446] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.579246] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.590147] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.595947] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.606845] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.612644] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.623543] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.629345] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.640242] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.646046] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.656940] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.662745] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.673641] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.679440] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.690344] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.696147] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.707036] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.712834] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.723735] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.729616] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.740434] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.746228] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.757138] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.762940] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.773835] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.779635] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.790531] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.796331] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.807228] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.813034] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.823929] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.829731] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.840627] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.846430] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.857328] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.863131] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.874026] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.879829] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.890731] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.896532] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.907425] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.913231] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.924124] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.929925] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.940821] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.946625] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.957521] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.963323] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.974218] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.980011] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   53.990916] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   53.996716] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.007614] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.013406] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.024317] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.030119] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.041012] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.046813] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.057713] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.063516] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.074410] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.080208] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.091108] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.096910] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.107808] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.113611] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.124512] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.130312] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.141208] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.147011] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.157905] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.163707] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.174604] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.180408] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.191303] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.197106] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.208002] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.213807] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.224699] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.230500] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.241398] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.247201] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.258097] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.263888] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.274799] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.280603] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.291496] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.297299] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.308196] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.313998] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.324894] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.330694] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.341592] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.347395] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.358290] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.364093] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.374991] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.380792] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.391689] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.397490] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.408388] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.414190] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.425084] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.430876] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.441786] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.447588] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.458482] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.464283] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.475181] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.480982] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.491885] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.497687] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.508579] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.514372] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.525278] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.531080] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.541981] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.547782] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.558679] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.564480] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.575375] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.581175] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.592073] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.597873] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.608773] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.614577] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.625477] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.631278] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.642169] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.647967] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.658873] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.664674] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.675568] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.681367] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.692265] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.698067] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.708964] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.714765] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.725666] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.731466] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.742364] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.748165] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.759062] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.764863] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.775759] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.781560] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.792459] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.798259] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.809156] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.814961] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.825861] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.831665] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.842556] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.848356] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.859254] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.865052] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.875952] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.881748] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.892651] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.898442] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.909349] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.915141] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.926049] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.931849] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.942747] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.948543] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.959447] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.965251] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.976144] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.981945] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   54.992848] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   54.998648] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.009543] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.015346] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.026242] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.032043] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.042941] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.048746] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.059641] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.065442] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.076338] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.082138] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.093037] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.098837] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.109738] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.115539] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.126435] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.132238] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.143133] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.148937] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.159840] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.165646] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.176533] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.182334] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.193229] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.199061] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.209929] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.217210] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0

[   55.226627] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.232430] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.243327] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.249131] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.260024] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.265825] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.276724] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.282526] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.293421] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.299210] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.310122] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.315921] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.326820] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.332622] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.343519] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.349336] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[  OK  ] Started EVCX UI Service.
[   55.360220] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.366487] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.376917] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.384005] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0

[   55.393622] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.399422] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.410315] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.416119] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.427017] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.432823] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.443712] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.449517] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.460410] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.466193] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.477110] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.482911] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.493808] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.499600] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.510511] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.516295] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.527203] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.532990] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.543908] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.549691] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.560604] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.566387] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.577320] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.583122] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.593999] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.599779] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.610696] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.616474] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.627394] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.633172] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.644092] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.649869] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.660793] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.666568] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.677490] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.683267] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.694189] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.699964] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.710887] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.716660] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.727588] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.733365] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.744286] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.750066] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.760985] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.766761] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.777684] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.783460] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.794383] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.800155] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.811081] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.816853] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.827782] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.833552] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.844480] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.850256] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.861181] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.866953] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.877875] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.883652] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.894577] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.900353] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.911274] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.917048] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.927975] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.933753] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.944671] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.950448] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.961370] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.967145] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.978069] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   55.983843] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   55.994767] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.000540] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.011466] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.017239] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.028165] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.033937] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.044864] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.050640] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.061562] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.067333] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.078263] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.084039] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.094962] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.100739] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.111667] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.117435] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.128359] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.134135] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.145056] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.150832] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.161756] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.167531] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.178457] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.184234] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.195153] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.200931] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.211853] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.217630] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.228551] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.234327] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.245252] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.251026] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.261950] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.267724] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.278649] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.284436] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.295352] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.301141] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.312048] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.317831] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.328744] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.334525] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.345443] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.351219] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.362142] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.367920] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.378842] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.384619] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.395541] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.401319] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.412240] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.418018] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.428937] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.434713] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.445636] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.451413] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.462334] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.468106] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.479033] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.484809] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.495730] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.501499] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.512431] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.518205] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.529136] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.534912] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.545832] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.551633] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.562532] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.568333] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.579230] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.585019] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.595927] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.601705] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.612623] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.618400] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.629326] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.635100] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.646020] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.651792] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.662717] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.668486] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.679420] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.685198] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.696119] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.701899] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.712818] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.718594] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.729518] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.735289] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.746213] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.751989] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.762912] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.768684] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.779612] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.785387] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.796312] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.802087] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.813010] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.818786] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.829708] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.835482] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.846408] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.852185] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.863111] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.868884] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.879804] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.885580] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.896502] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.902276] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.913201] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.918976] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.929901] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.935675] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.946598] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.952370] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.963300] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.969077] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.979998] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   56.985775] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   56.996696] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.002470] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.013395] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.019172] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.030093] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.035869] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.046793] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.052569] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.063490] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.069265] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.080192] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.085968] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.096889] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.102666] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.113589] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.119365] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.130286] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.136061] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.146985] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.152753] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.163687] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.169459] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.180384] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.186155] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.197081] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.202851] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.213779] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.219554] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.230478] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.236252] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.247178] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.252954] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.263877] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.269652] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.280577] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.286358] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.297277] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.303062] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.313973] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.319750] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.330672] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.336448] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.347370] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.353142] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.364069] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.369845] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.380769] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.386546] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.397470] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.403245] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.414168] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.419942] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.430865] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.436641] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.447564] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.453342] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.464263] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.470038] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.480963] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.486738] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.497665] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.503436] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.514358] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.520133] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.531059] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.536839] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.547760] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.553532] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.564455] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.570232] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.581154] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.586933] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.597853] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.603630] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.614553] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.620328] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.631253] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.637025] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.647950] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.653723] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.664647] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.670417] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.681346] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.687121] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0
[   57.698049] __: irq 0x1, vid0 0x0, vid1 0x0, vp0 0x2, vp1 0x0
[   57.703823] <<: irq 0x0, vid0 0x0, vid1 0x0, vp0 0x0, vp1 0x0

Is it an expected behavior in terms of logging with your patch ? Please feedback ASAP since this issue is very critical in our side, blocking lots of tests.

Thanks in advance,

Veysel

0 Bin Liu 11 months ago in reply to Veysel Oguz

TI__Guru**** 168261 points

Hi Veysel,

Yes, the log is expected. If the rcu stall issue was related to tidss, the timestamps intervals will be much shorter after (or close to) the rcu stall happens.

0 Veysel Oguz 11 months ago in reply to Bin Liu

Prodigy 210 points

Hi Bin,

Ok, got it. So we will continue to test and collect logs with this patch

0 Bin Liu 11 months ago in reply to Veysel Oguz

TI__Guru**** 168261 points

Currently in the log, the timestamp delta between '<<:' line and '__:' line is about 10 milliseconds. After the rcu stall happened and it was caused by tidss, the timestamp delta would be a few microseconds.

0 Bin Liu 11 months ago in reply to Veysel Oguz

TI__Guru**** 168261 points

Hi Veysel,

Veysel Oguz said:
Below are the logs that are collected on different devices, each have rcu-stall issue. Please check.

In your first post above, you provided multiple rcu stall logs on multiple boards. Is each rcu stall message the very first occurrence in the test? Typically when the rcu stall problem happens, the kernel log will have multiple rcu stall messages, but only the very first would be helpful in debugging.

0 Veysel Oguz 11 months ago in reply to Bin Liu

Prodigy 210 points

Hi Bin,

It must be first occurrences. Do you have any opinion regarding the root cause of the stalls up until now ? If there are additional debug methods please provide.

Thanks in advance,

Veysel

0 Bin Liu 11 months ago in reply to Veysel Oguz

TI__Guru**** 168261 points

Hi Veysel,

With current information we have, I cannot tell what would cause the issue. Let's wait for the current test result, then we will think how to debug it next.

0 Bin Liu 9 months ago in reply to Bin Liu

TI__Guru**** 168261 points

Hi Veysel,

Is this rcu stall issue still open?

0 Veysel Oguz 9 months ago in reply to Bin Liu

Prodigy 210 points

Hi Bin,

I applied the first patch you suggested in this topic, and did not see rcu_stall issue up until now. Can you please verify that patch resolves the issue or if you have any newer and verified patchset, please share with us.

Thanks in advance,

Veysel

+1 Bin Liu 9 months ago in reply to Veysel Oguz

TI__Guru**** 168261 points

Hi Veysel,

Thanks for confirming.

The similar patch attached below has been merged to TI kernel v6.6.y branch.

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/0001_2D00_drm_2D00_tidss_2D00_Fix_2D00_issue_2D00_in_2D00_irq_2D00_handling_2D00_causing_2D00_irq_2D00_floo.patch

0 Veysel Oguz 9 months ago in reply to Bin Liu

Prodigy 210 points

Hi Bin,

One more question, I checked the TI kernel v6.6 y branch and there are more patches related in tidss:

drm/tidss: Fix issue in irq handling causing irq-flood issue -> this is the one that we already applied.

How about below 6 patchset ? Do you think we should also apply the below patchset ? Additionally we are using SDK 09.01.00.08 which has TI kernel 6.1

drm/tidss: Remove unused OCP error flag

drm/tidss: Remove extra K2G check

drm/tidss: Add printing of underflows

drm/tidss: Clear the interrupt status for interrupts being disabled

drm/tidss: Fix race condition while handling interrupt registers

drm/tidss: Rename 'wait_lock' to 'irq_lock'

Please let me know

Thanks in advance,

Veysel

0 Bin Liu 9 months ago in reply to Veysel Oguz

TI__Guru**** 168261 points

Hi Veysel,

I am not a DSS expert but after reviewed these patches, I believe patch 4 & 5 that you listed were discovered during debugging this rcu stall issue and recommend to apply both too. Other patches in the list do not really fix any problem, just code cleanup or improvement. It is okay to not apply them.

BTY, if possible, it is highly recommended to move to SDK10.0 in your project due to a PLL fix. But if it is already too late in your development cycle, you should consider to at least migrate to SDK10.0 versions of the SYSFW binaries. Please see the details in the link below.

https://software-dl.ti.com/processor-sdk-linux/esd/AM62X/10_00_07_04/exports/docs/devices/AM62X/linux/Release_Specific_Migration_Guide.html#pll-programing-sequence-update-to-avoid-pll-instability

Processors

Processors forum

AM625: rcu_preempt self-detected stall on CPU error during runtime