AM623: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks

Part Number: AM623

Tool/software:

Hi, TI expert! 

I have a development environment for am6234, with our own evaluation board.

The software SDK version is PROCESSOR-SDK-LINUX-RT-AM62X-08.06.00.42.

Recently, we found that resetting our 4G module can cause the system to freeze.

Today, when I hung up and didn't plug in the 4G module, I was letting it run on its own and found that it was running dead.

The scheduler automatically crashed,as shown in the following log:

[ 8942.705521] task:network_daemon  state:D stack:    0 pid:  475 ppid:   436 flags:0x00000004
[ 8942.705547] Call trace:
[ 8942.705553]  __switch_to+0x104/0x178
[ 8942.705584]  __schedule+0x48c/0xc30
[ 8942.705614]  preempt_schedule_lock+0x44/0x98
[ 8942.705646]  rt_spin_lock_slowlock_locked+0x1a4/0x410
[ 8942.705671]  rt_spin_lock_slowlock+0xac/0xf0
[ 8942.705693]  rt_spin_lock+0x8c/0xa8
[ 8942.705718]  lru_cache_add+0xbc/0x380
[ 8942.705746]  shmem_getpage_gfp.isra.0+0x4d0/0x978
[ 8942.705773]  shmem_write_begin+0x80/0xc8
[ 8942.705797]  generic_perform_write+0x144/0x2a8
[ 8942.705835]  __generic_file_write_iter+0x1b0/0x260
[ 8942.705860]  generic_file_write_iter+0x8c/0x168
[ 8942.705884]  new_sync_write+0x1cc/0x2d0
[ 8942.705913]  vfs_write+0x474/0x640
[ 8942.705938]  ksys_write+0xc8/0x180
[ 8942.705965]  __arm64_sys_write+0x44/0x58
[ 8942.705993]  el0_svc_common.constprop.0+0xac/0x260
[ 8942.706031]  do_el0_svc+0x38/0xb0
[ 8942.706060]  el0_svc+0x1c/0x28
[ 8942.706085]  el0_sync_handler+0xb0/0xb8
[ 8942.706118]  el0_sync+0x180/0x1c0
[ 8942.706154] INFO: task network_daemon:962 blocked for more than 120 seconds.
[ 8942.706168]       Not tainted 5.10.168-rt83-00019-g1e7cc1bccf5e #1
[ 8942.706182] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

The complete test records are as follows:

7043.test_log.txt
sub connect success
pub connect success
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
Device "wwan0" does not exist.
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
can send error: No buffer space available
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
chown: /media/data/lighttpd: No such file or directory
/
yi ti ji
sub connect success
pub connect success
QStandardPaths: wrong permissions on runtime directory /usr/local/apps/lcd/, 0775 instead of 0700
ts_setup() failed (No such file or directory)
Cannot open mouse input device /dev/input/event1 (No such file or directory)
evdevmouse: Failed to open mouse device /dev/input/event1
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
Device "wwan0" does not exist.
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
can send error: No buffer space available
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
Device "wwan0" does not exist.
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
can send error: No buffer space available
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
chown: /media/data/lighttpd: No such file or directory
/
yi ti ji
sub connect success
pub connect success
QStandardPaths: wrong permissions on runtime directory /usr/local/apps/lcd/, 0775 instead of 0700
ts_setup() failed (No such file or directory)
Cannot open mouse input device /dev/input/event1 (No such file or directory)
evdevmouse: Failed to open mouse device /dev/input/event1
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
Device "wwan0" does not exist.
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
can send error: No buffer space available
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
Device "wwan0" does not exist.
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
can send error: No buffer space available
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
/
[INFO ] Init zlog finish
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
chown: /media/data/lighttpd: No such file or directory
/
yi ti ji
sub connect success
pub connect success
QStandardPaths: wrong permissions on runtime directory /usr/local/apps/lcd/, 0775 instead of 0700
ts_setup() failed (No such file or directory)
Cannot open mouse input device /dev/input/event1 (No such file or directory)
evdevmouse: Failed to open mouse device /dev/input/event1
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
Device "wwan0" does not exist.
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
can send error: No buffer space available
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
Device "wwan0" does not exist.
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
can send error: No buffer space available
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
chown: /media/data/lighttpd: No such file or directory
/
yi ti ji
sub connect success
pub connect success
QStandardPaths: wrong permissions on runtime directory /usr/local/apps/lcd/, 0775 instead of 0700
ts_setup() failed (No such file or directory)
Cannot open mouse input device /dev/input/event1 (No such file or directory)
evdevmouse: Failed to open mouse device /dev/input/event1
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
Device "wwan0" does not exist.
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
can send error: No buffer space available
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
can send error: No buffer space available
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
     Zmq_Uninit
yi ti ji
sub connect success
pub connect success
chown: /media/data/lighttpd: No such file or directory
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
chown: /media/data/lighttpd: No such file or directory
/
yi ti ji
sub connect success
pub connect success
QStandardPaths: wrong permissions on runtime directory /usr/local/apps/lcd/, 0775 instead of 0700
ts_setup() failed (No such file or directory)
Cannot open mouse input device /dev/input/event1 (No such file or directory)
evdevmouse: Failed to open mouse device /dev/input/event1
/var/jenkins_node/workspace/ZYX_DC_A53_Baseline/apps/network/network_daemon/dfx/dfx.cpp:     DfxHexNoticeLog:  99:
Device "wwan0" does not exist.
chown: /media/data/lighttpd: No such file or directory
can send error: No buffer space available
[ 8771.426579] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 8771.426671] rcu:     Tasks blocked on level-0 rcu_node (CPUs 0-1): P30049/1:b..l
[ 8771.426813]  (detected by 0, t=5252 jiffies, g=2404689, q=4495)
[ 8771.426884] task:exe             state:D stack:    0 pid:30049 ppid: 30045 flags:0x0000000c
[ 8771.426983] Call trace:
[ 8771.427009]  __switch_to+0x104/0x178
[ 8771.427150]  __schedule+0x48c/0xc30
[ 8771.427264]  preempt_schedule_lock+0x44/0x98
[ 8771.427378]  rt_spin_lock_slowlock_locked+0x1a4/0x410
[ 8771.427467]  rt_spin_lock_slowlock+0xac/0xf0
[ 8771.427549]  rt_spin_lock+0x8c/0xa8
[ 8771.427631]  lru_cache_add+0xbc/0x380
[ 8771.427735]  lru_cache_add_inactive_or_unevictable+0x40/0x198
[ 8771.427854]  handle_mm_fault+0xb24/0x12a0
[ 8771.427963]  __get_user_pages+0x34c/0x5a0
[ 8771.428052]  __get_user_pages_remote+0xf4/0x330
[ 8771.428148]  get_user_pages_remote+0x28/0x40
[ 8771.428241]  get_arg_page+0xa8/0x158
[ 8771.428324]  copy_string_kernel+0xb8/0x1e8
[ 8771.428409]  do_execveat_common.isra.0+0x170/0x308
[ 8771.428505]  __arm64_sys_execve+0x60/0x78
[ 8771.428593]  el0_svc_common.constprop.0+0xac/0x260
[ 8771.428717]  do_el0_svc+0x38/0xb0
[ 8771.428817]  el0_svc+0x1c/0x28
[ 8771.428906]  el0_sync_handler+0xb0/0xb8
[ 8771.429006]  el0_sync+0x180/0x1c0
     Zmq_Uninit
[ 8834.446553] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 8834.446645] rcu:     Tasks blocked on level-0 rcu_node (CPUs 0-1): P30049/1:b..l
[ 8834.446779]  (detected by 1, t=21007 jiffies, g=2404689, q=8500)
[ 8834.446845] task:exe             state:D stack:    0 pid:30049 ppid: 30045 flags:0x0000000c
[ 8834.446941] Call trace:
[ 8834.446969]  __switch_to+0x104/0x178
[ 8834.447101]  __schedule+0x48c/0xc30
[ 8834.447216]  preempt_schedule_lock+0x44/0x98
[ 8834.447329]  rt_spin_lock_slowlock_locked+0x1a4/0x410
[ 8834.447417]  rt_spin_lock_slowlock+0xac/0xf0
[ 8834.447494]  rt_spin_lock+0x8c/0xa8
[ 8834.447574]  lru_cache_add+0xbc/0x380
[ 8834.447676]  lru_cache_add_inactive_or_unevictable+0x40/0x198
[ 8834.447798]  handle_mm_fault+0xb24/0x12a0
[ 8834.447903]  __get_user_pages+0x34c/0x5a0
[ 8834.447990]  __get_user_pages_remote+0xf4/0x330
[ 8834.448086]  get_user_pages_remote+0x28/0x40
[ 8834.448179]  get_arg_page+0xa8/0x158
[ 8834.448262]  copy_string_kernel+0xb8/0x1e8
[ 8834.448348]  do_execveat_common.isra.0+0x170/0x308
[ 8834.448445]  __arm64_sys_execve+0x60/0x78
[ 8834.448533]  el0_svc_common.constprop.0+0xac/0x260
[ 8834.448658]  do_el0_svc+0x38/0xb0
[ 8834.448758]  el0_svc+0x1c/0x28
[ 8834.448848]  el0_sync_handler+0xb0/0xb8
[ 8834.448948]  el0_sync+0x180/0x1c0
[ 8897.466538] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 8897.466585] rcu:     Tasks blocked on level-0 rcu_node (CPUs 0-1): P30049/1:b..l
[ 8897.466645]  (detected by 0, t=36762 jiffies, g=2404689, q=12452)
[ 8897.466667] task:exe             state:D stack:    0 pid:30049 ppid: 30045 flags:0x0000000c
[ 8897.466697] Call trace:
[ 8897.466709]  __switch_to+0x104/0x178
[ 8897.466762]  __schedule+0x48c/0xc30
[ 8897.466799]  preempt_schedule_lock+0x44/0x98
[ 8897.466832]  rt_spin_lock_slowlock_locked+0x1a4/0x410
[ 8897.466864]  rt_spin_lock_slowlock+0xac/0xf0
[ 8897.466886]  rt_spin_lock+0x8c/0xa8
[ 8897.466910]  lru_cache_add+0xbc/0x380
[ 8897.466944]  lru_cache_add_inactive_or_unevictable+0x40/0x198
[ 8897.466978]  handle_mm_fault+0xb24/0x12a0
[ 8897.467015]  __get_user_pages+0x34c/0x5a0
[ 8897.467042]  __get_user_pages_remote+0xf4/0x330
[ 8897.467070]  get_user_pages_remote+0x28/0x40
[ 8897.467097]  get_arg_page+0xa8/0x158
[ 8897.467130]  copy_string_kernel+0xb8/0x1e8
[ 8897.467154]  do_execveat_common.isra.0+0x170/0x308
[ 8897.467182]  __arm64_sys_execve+0x60/0x78
[ 8897.467209]  el0_svc_common.constprop.0+0xac/0x260
[ 8897.467246]  do_el0_svc+0x38/0xb0
[ 8897.467275]  el0_svc+0x1c/0x28
[ 8897.467301]  el0_sync_handler+0xb0/0xb8
[ 8897.467330]  el0_sync+0x180/0x1c0
[ 8942.698961] INFO: task rcub/0:13 blocked for more than 120 seconds.
[ 8942.699080]       Not tainted 5.10.168-rt83-00019-g1e7cc1bccf5e #1
[ 8942.699128] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8942.699164] task:rcub/0          state:D stack:    0 pid:   13 ppid:     2 flags:0x00000028
[ 8942.699267] Call trace:
[ 8942.699294]  __switch_to+0x104/0x178
[ 8942.699431]  __schedule+0x48c/0xc30
[ 8942.699545]  schedule+0x138/0x1f8
[ 8942.699645]  __rt_mutex_slowlock+0x110/0x1b0
[ 8942.699724]  rt_mutex_slowlock_locked+0xb8/0x2c0
[ 8942.699807]  rt_mutex_slowlock+0xc8/0x138
[ 8942.699879]  rt_mutex_lock+0x94/0xa8
[ 8942.699950]  rcu_boost_kthread+0x10c/0x1a8
[ 8942.700078]  kthread+0x274/0x280
[ 8942.700170]  ret_from_fork+0x10/0x30
[ 8942.700630] INFO: task jbd2/mmcblk0p6-:211 blocked for more than 120 seconds.
[ 8942.700689]       Not tainted 5.10.168-rt83-00019-g1e7cc1bccf5e #1
[ 8942.700735] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8942.700774] task:jbd2/mmcblk0p6- state:D stack:    0 pid:  211 ppid:     2 flags:0x00000028
[ 8942.700865] Call trace:
[ 8942.700890]  __switch_to+0x104/0x178
[ 8942.700998]  __schedule+0x48c/0xc30
[ 8942.701104]  preempt_schedule_lock+0x44/0x98
[ 8942.701217]  rt_spin_lock_slowlock_locked+0x1a4/0x410
[ 8942.701305]  rt_spin_lock_slowlock+0xac/0xf0
[ 8942.701382]  rt_spin_lock+0x8c/0xa8
[ 8942.701465]  lru_cache_add+0xbc/0x380
[ 8942.701569]  add_to_page_cache_lru+0x10c/0x1a0
[ 8942.701652]  pagecache_get_page+0x1dc/0x428
[ 8942.701728]  __getblk_gfp+0x13c/0x430
[ 8942.701822]  jbd2_journal_get_descriptor_buffer+0xc4/0x280
[ 8942.701921]  journal_submit_commit_record.part.0+0x8c/0x418
[ 8942.702051]  jbd2_journal_commit_transaction+0x1f40/0x2518
[ 8942.702170]  kjournald2+0x144/0x3f0
[ 8942.702278]  kthread+0x274/0x280
[ 8942.702369]  ret_from_fork+0x10/0x30
[ 8942.702482] INFO: task udevd:243 blocked for more than 120 seconds.
[ 8942.704706]       Not tainted 5.10.168-rt83-00019-g1e7cc1bccf5e #1
[ 8942.704736] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8942.704749] task:udevd           state:D stack:    0 pid:  243 ppid:     1 flags:0x00000004
[ 8942.704785] Call trace:
[ 8942.704793]  __switch_to+0x104/0x178
[ 8942.704840]  __schedule+0x48c/0xc30
[ 8942.704881]  preempt_schedule_lock+0x44/0x98
[ 8942.704913]  rt_spin_lock_slowlock_locked+0x1a4/0x410
[ 8942.704939]  rt_spin_lock_slowlock+0xac/0xf0
[ 8942.704962]  rt_spin_lock+0x8c/0xa8
[ 8942.704985]  lru_cache_add+0xbc/0x380
[ 8942.705017]  lru_cache_add_inactive_or_unevictable+0x40/0x198
[ 8942.705051]  wp_page_copy+0x2e8/0xcd8
[ 8942.705083]  do_wp_page+0x104/0x878
[ 8942.705189]  handle_mm_fault+0xb9c/0x12a0
[ 8942.705224]  do_page_fault+0x190/0x4b8
[ 8942.705254]  do_mem_abort+0x5c/0xf0
[ 8942.705281]  el0_da+0x2c/0x38
[ 8942.705308]  el0_sync_handler+0x88/0xb8
[ 8942.705337]  el0_sync+0x180/0x1c0
[ 8942.705477] INFO: task network_daemon:475 blocked for more than 120 seconds.
[ 8942.705492]       Not tainted 5.10.168-rt83-00019-g1e7cc1bccf5e #1
[ 8942.705506] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8942.705521] task:network_daemon  state:D stack:    0 pid:  475 ppid:   436 flags:0x00000004
[ 8942.705547] Call trace:
[ 8942.705553]  __switch_to+0x104/0x178
[ 8942.705584]  __schedule+0x48c/0xc30
[ 8942.705614]  preempt_schedule_lock+0x44/0x98
[ 8942.705646]  rt_spin_lock_slowlock_locked+0x1a4/0x410
[ 8942.705671]  rt_spin_lock_slowlock+0xac/0xf0
[ 8942.705693]  rt_spin_lock+0x8c/0xa8
[ 8942.705718]  lru_cache_add+0xbc/0x380
[ 8942.705746]  shmem_getpage_gfp.isra.0+0x4d0/0x978
[ 8942.705773]  shmem_write_begin+0x80/0xc8
[ 8942.705797]  generic_perform_write+0x144/0x2a8
[ 8942.705835]  __generic_file_write_iter+0x1b0/0x260
[ 8942.705860]  generic_file_write_iter+0x8c/0x168
[ 8942.705884]  new_sync_write+0x1cc/0x2d0
[ 8942.705913]  vfs_write+0x474/0x640
[ 8942.705938]  ksys_write+0xc8/0x180
[ 8942.705965]  __arm64_sys_write+0x44/0x58
[ 8942.705993]  el0_svc_common.constprop.0+0xac/0x260
[ 8942.706031]  do_el0_svc+0x38/0xb0
[ 8942.706060]  el0_svc+0x1c/0x28
[ 8942.706085]  el0_sync_handler+0xb0/0xb8
[ 8942.706118]  el0_sync+0x180/0x1c0
[ 8942.706154] INFO: task network_daemon:962 blocked for more than 120 seconds.
[ 8942.706168]       Not tainted 5.10.168-rt83-00019-g1e7cc1bccf5e #1
[ 8942.706182] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8942.706194] task:network_daemon  state:D stack:    0 pid:  962 ppid:   436 flags:0x00000004
[ 8942.706219] Call trace:
[ 8942.706226]  __switch_to+0x104/0x178
[ 8942.706255]  __schedule+0x48c/0xc30
[ 8942.706284]  schedule+0x138/0x1f8
[ 8942.706313]  schedule_timeout+0x334/0x3e8
[ 8942.706338]  wait_for_completion+0xf0/0x1b0
[ 8942.706373]  __wait_rcu_gp+0x1e4/0x1f0
[ 8942.706404]  synchronize_rcu+0xec/0xf8
[ 8942.706437]  synchronize_net+0x28/0x38
[ 8942.706465]  __unregister_prot_hook+0x17c/0x1b8
[ 8942.706499]  packet_do_bind+0x370/0x550
[ 8942.707798]  packet_bind+0x74/0xa0
[ 8942.707838]  __sys_bind+0x140/0x158
[ 8942.707864]  __arm64_sys_bind+0x44/0x60
[ 8942.707886]  el0_svc_common.constprop.0+0xac/0x260
[ 8942.707928]  do_el0_svc+0x38/0xb0
[ 8942.707956]  el0_svc+0x1c/0x28
[ 8942.707985]  el0_sync_handler+0xb0/0xb8
[ 8942.708014]  el0_sync+0x180/0x1c0
[ 8942.708039] INFO: task network_daemon:963 blocked for more than 120 seconds.
[ 8942.708054]       Not tainted 5.10.168-rt83-00019-g1e7cc1bccf5e #1
[ 8942.708068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8942.708079] task:network_daemon  state:D stack:    0 pid:  963 ppid:   436 flags:0x00000004
[ 8942.708105] Call trace:
[ 8942.708112]  __switch_to+0x104/0x178
[ 8942.708143]  __schedule+0x48c/0xc30
[ 8942.708176]  schedule+0x138/0x1f8
[ 8942.708206]  __rt_mutex_slowlock+0x110/0x1b0
[ 8942.708228]  rt_mutex_slowlock_locked+0xb8/0x2c0
[ 8942.708253]  rt_mutex_slowlock+0xc8/0x138
[ 8942.708274]  __rt_mutex_lock_state+0x70/0xc8
[ 8942.708299]  __down_write_common+0x74/0x238
[ 8942.708321]  __down_write+0x14/0x20
[ 8942.708344]  down_write+0x10/0x20
[ 8942.708375]  generic_file_write_iter+0x48/0x168
[ 8942.708401]  new_sync_write+0x1cc/0x2d0
[ 8942.708430]  vfs_write+0x474/0x640
[ 8942.708456]  ksys_write+0xc8/0x180
[ 8942.708481]  __arm64_sys_write+0x44/0x58
[ 8942.708508]  el0_svc_common.constprop.0+0xac/0x260
[ 8942.708541]  do_el0_svc+0x38/0xb0
[ 8942.708570]  el0_svc+0x1c/0x28
[ 8942.708596]  el0_sync_handler+0xb0/0xb8
[ 8942.708628]  el0_sync+0x180/0x1c0
[ 8942.708658] INFO: task network_daemon:964 blocked for more than 120 seconds.
[ 8942.708673]       Not tainted 5.10.168-rt83-00019-g1e7cc1bccf5e #1
[ 8942.708686] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8942.708698] task:network_daemon  state:D stack:    0 pid:  964 ppid:   436 flags:0x00000004
[ 8942.708723] Call trace:
[ 8942.708730]  __switch_to+0x104/0x178
[ 8942.708759]  __schedule+0x48c/0xc30
[ 8942.708790]  schedule+0x138/0x1f8
[ 8942.708819]  __rt_mutex_slowlock+0x110/0x1b0
[ 8942.708844]  rt_mutex_slowlock_locked+0xb8/0x2c0
[ 8942.708867]  rt_mutex_slowlock+0xc8/0x138
[ 8942.708893]  __rt_mutex_lock_state+0x70/0xc8
[ 8942.708915]  __down_write_common+0x74/0x238
[ 8942.708939]  __down_write+0x14/0x20
[ 8942.708959]  down_write+0x10/0x20
[ 8942.708990]  generic_file_write_iter+0x48/0x168
[ 8942.709014]  new_sync_write+0x1cc/0x2d0
[ 8942.709040]  vfs_write+0x474/0x640
[ 8942.709065]  ksys_write+0xc8/0x180
[ 8942.709090]  __arm64_sys_write+0x44/0x58
[ 8942.709118]  el0_svc_common.constprop.0+0xac/0x260
[ 8942.709152]  do_el0_svc+0x38/0xb0
[ 8942.709181]  el0_svc+0x1c/0x28
[ 8942.709208]  el0_sync_handler+0xb0/0xb8
[ 8942.709237]  el0_sync+0x180/0x1c0
[ 8942.709290] INFO: task ocpp:553 blocked for more than 120 seconds.
[ 8942.709304]       Not tainted 5.10.168-rt83-00019-g1e7cc1bccf5e #1
[ 8942.709317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8942.709330] task:ocpp            state:D stack:    0 pid:  553 ppid:   436 flags:0x00000004
[ 8942.709355] Call trace:
[ 8942.709361]  __switch_to+0x104/0x178
[ 8942.709792]  __schedule+0x48c/0xc30
[ 8942.709828]  schedule+0x138/0x1f8
[ 8942.709858]  __rt_mutex_slowlock+0x110/0x1b0
[ 8942.709880]  rt_mutex_slowlock_locked+0xb8/0x2c0
[ 8942.709903]  rt_mutex_slowlock+0xc8/0x138
[ 8942.709924]  __rt_mutex_lock_state+0x70/0xc8
[ 8942.709947]  __down_write_common+0x74/0x238
[ 8942.709969]  __down_write+0x14/0x20
[ 8942.709990]  down_write+0x10/0x20
[ 8942.710102]  generic_file_write_iter+0x48/0x168
[ 8942.710128]  new_sync_write+0x1cc/0x2d0
[ 8942.710155]  vfs_write+0x474/0x640
[ 8942.710181]  ksys_write+0xc8/0x180
[ 8942.710207]  __arm64_sys_write+0x44/0x58
[ 8942.710236]  el0_svc_common.constprop.0+0xac/0x260
[ 8942.710271]  do_el0_svc+0x38/0xb0
[ 8942.710299]  el0_svc+0x1c/0x28
[ 8942.710325]  el0_sync_handler+0xb0/0xb8
[ 8942.710354]  el0_sync+0x180/0x1c0
[ 8942.710383] INFO: task ocpp:554 blocked for more than 120 seconds.
[ 8942.710401]       Not tainted 5.10.168-rt83-00019-g1e7cc1bccf5e #1
[ 8942.710414] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8942.710425] task:ocpp            state:D stack:    0 pid:  554 ppid:   436 flags:0x00000004
[ 8942.710451] Call trace:
[ 8942.710459]  __switch_to+0x104/0x178
[ 8942.710491]  __schedule+0x48c/0xc30
[ 8942.712278]  schedule+0x138/0x1f8
[ 8942.712324]  __rt_mutex_slowlock+0x110/0x1b0
[ 8942.712347]  rt_mutex_slowlock_locked+0xb8/0x2c0
[ 8942.712371]  rt_mutex_slowlock+0xc8/0x138
[ 8942.712392]  __rt_mutex_lock_state+0x70/0xc8
[ 8942.712416]  __down_write_common+0x74/0x238
[ 8942.712439]  __down_write+0x14/0x20
[ 8942.712461]  down_write+0x10/0x20
[ 8942.712494]  generic_file_write_iter+0x48/0x168
[ 8942.712519]  new_sync_write+0x1cc/0x2d0
[ 8942.712548]  vfs_write+0x474/0x640
[ 8942.712573]  ksys_write+0xc8/0x180
[ 8942.712599]  __arm64_sys_write+0x44/0x58
[ 8942.712628]  el0_svc_common.constprop.0+0xac/0x260
[ 8942.712663]  do_el0_svc+0x38/0xb0
[ 8942.712691]  el0_svc+0x1c/0x28
[ 8942.712717]  el0_sync_handler+0xb0/0xb8
[ 8942.712746]  el0_sync+0x180/0x1c0
[ 8942.712778] INFO: task ocpp:620 blocked for more than 120 seconds.
[ 8942.712793]       Not tainted 5.10.168-rt83-00019-g1e7cc1bccf5e #1
[ 8942.712806] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8942.712817] task:ocpp            state:D stack:    0 pid:  620 ppid:   436 flags:0x00000004
[ 8942.712844] Call trace:
[ 8942.712853]  __switch_to+0x104/0x178
[ 8942.712882]  __schedule+0x48c/0xc30
[ 8942.712912]  schedule+0x138/0x1f8
[ 8942.712944]  __rt_mutex_slowlock+0x110/0x1b0
[ 8942.712966]  rt_mutex_slowlock_locked+0xb8/0x2c0
[ 8942.712989]  rt_mutex_slowlock+0xc8/0x138
[ 8942.713010]  __rt_mutex_lock_state+0x70/0xc8
[ 8942.713033]  __down_write_common+0x74/0x238
[ 8942.713056]  __down_write+0x14/0x20
[ 8942.713078]  down_write+0x10/0x20
[ 8942.713108]  generic_file_write_iter+0x48/0x168
[ 8942.713133]  new_sync_write+0x1cc/0x2d0
[ 8942.713159]  vfs_write+0x474/0x640
[ 8942.713184]  ksys_write+0xc8/0x180
[ 8942.713210]  __arm64_sys_write+0x44/0x58
[ 8942.713237]  el0_svc_common.constprop.0+0xac/0x260
[ 8942.713272]  do_el0_svc+0x38/0xb0
[ 8942.713302]  el0_svc+0x1c/0x28
[ 8942.713328]  el0_sync_handler+0xb0/0xb8
[ 8942.713358]  el0_sync+0x180/0x1c0
[ 8942.713753] Kernel panic - not syncing:
[ 8942.713764] hung_task: blocked tasks
[ 8942.713773] CPU: 0 PID: 29 Comm: khungtaskd Not tainted 5.10.168-rt83-00019-g1e7cc1bccf5e #1
[ 8942.713795] Hardware name: Texas Instruments AM625 SK (DT)
[ 8942.713807] Call trace:
[ 8942.713814]  dump_backtrace+0x0/0x2c0
[ 8942.713847]  show_stack+0x18/0x28
[ 8942.713876]  dump_stack+0x114/0x160
[ 8942.713913]  panic+0x13c/0x438
[ 8942.713938]  proc_dohung_task_timeout_secs+0x0/0x60
[ 8942.713971]  kthread+0x274/0x280
[ 8942.713999]  ret_from_fork+0x10/0x30
[ 8943.902788] SMP: stopping secondary CPUs
[ 8943.906805] Kernel Offset: disabled
[ 8943.910303] CPU features: 0x00040002,2000200c
[ 8943.914684] Memory Limit: none
[ 8944.021798] Rebooting in 30 seconds..

U-Boot SPL 2021.01-00001-g23084e57 (Dec 26 2024 - 16:22:36 +0800)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.4--v08.06.04 (Chill Capybar')
SPL initial stack usage: 13424 bytes
WDT:   Not found!
Trying to boot from SPI
Loading Environment from SPIFlash... OK

load_os [A]
Booting A(0x80000) OS...

Authentication passed
Authentication passed
Authentication passed
Authentication passed
Authentication passed
Loading Environment from SPIFlash... OK
Success read efuse otp mmr_val:0x0
Success read efuse otp mmr_val:0x0
Starting ATF on ARM64 core...

NOTICE:  BL31: v2.10.0  (release):v2.0.0_beta-1-gd4742c5
NOTICE:  BL31: Built : 16:21:56, Dec 26 2024
NOTICE:  Reset reason (0x0)
NOTICE:    Cold reset by MCU reset pin.
NOTICE:    Write Reset reason (0x0) to register RTC_RTC_SCRATCH0_2(0x2b1f0038).
NOTICE:    Reset reason register(CTRLMMR_MCU_RST_SRC) cleared.
ERROR:   Failed to get response (-19)
ERROR:   Transfer send failed (-19)
ERROR:   Unable to query firmware capabilities (-19)
I/TC:
I/TC: OP-TEE version: v2.0.0_beta (gcc version 9.2.1 20191025 (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10))) #1 Thu Dec 26 08:22:08 UTC 2024 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.4--v08.06.04 (Chill Capybar')
I/TC: HUK Initialized
I/TC: Activated SA2UL device
I/TC: Enabled firewalls for SA2UL TRNG device
I/TC: SA2UL TRNG initialized
I/TC: SA2UL Drivers initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2021.01-00001-g23084e57 (Dec 26 2024 - 16:22:16 +0800)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.4--v08.06.04 (Chill Capybar')
WDT:   Not found!
Trying to boot from SPI
Loading Environment from SPIFlash... OK

load_os [A]
Booting A(0x380000) OS...

Authentication passed
Authentication passed


U-Boot 2021.01-00001-g23084e57 (Dec 26 2024 - 16:22:16 +0800), Build: jenkins-ZYX_DC_A53_Baseline-7

SoC:   AM62X SR1.0 HS-FS
Model: Texas Instruments AM625 SK
EEPROM not available at 0x50, trying to read at 0x51
Reading on-board EEPROM at 0x51 failed 1
DRAM:  1 GiB
MMC:   mmc@fa10000: 0, mmc@fa00000: 1, mmc@fa20000: 2
Loading Environment from SPIFlash... SF: Detected is25lp064 with page size 256 Bytes, erase size 64 KiB, total 8 MiB
OK
In:    serial@2800000
Out:   serial@2800000
Err:   serial@2800000
EEPROM not available at 0x50, trying to read at 0x51
Reading on-board EEPROM at 0x51 failed 1
Net:   eth0: ethernet@8000000port@1

How can we solve this problem?

Regards,

Li

  • On a quick look it seams the device driver for the 4G module is the issue probably the way it uses splinlocks , based to me on device wwan0 is not working.

    Is the driver already upstream? Has it been successfully run with some Linux version and PREMPT_RT and some host processor?

      Pekka