TDA4VM: TDA4VH RCU CPU stall warning.

li quan

Part Number: TDA4VM
Other Parts Discussed in Thread: TDA4VH

Hi, expers

When enabling CONFIG_PREEMPT and running the `rcu-torture` torture tests on TDA4VH platform. Sometimes we are observing the following RCU CPU stall warning.

[15:13:07]]root@j7-evm:/run/media/mmcblk1p1# insmod torture.ko

[15:13:12]]root@j7-evm:/run/media/mmcblk1p1# insmod rcutorture.ko

[08:06:42]][60830.687903] rcu-torture: rtc: 00000000f24553fd ver: 614614 tfle: 0 rta: 614614 rtaf: 0 rtf: 614605 rtmbe: 0 rtbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 22200203 onoff: 0/0:0/0 -1,0:-1,0 0:0 (HZ=250) barrier: 0/0:0 read-exits: 57409
[08:06:42]][60830.708581] rcu-torture: Reader Pipe: 133426404760 2118331 0 0 0 0 0 0 0 0 0
[08:06:42]][60830.715735] rcu-torture: Reader Batch: 133422733238 5792891 0 0 0 0 0 0 0 0 0
[08:06:42]][60830.722981] rcu-torture: Free-Block Circulation: 614613 614613 614612 614611 614610 614609 614608 614607 614606 614605 0
[08:06:42]][60830.733964] ??? Writer stall state RTWS_STUTTER(8) g4639925 f0x0 ->state 0x1 cpu 4
[08:06:42]][60830.742578] rcu: rcu_preempt: wait state: RCU_GP_WAIT_FQS(5) ->state: 0x402 delta ->gp_activity 2 ->gp_req_activity 3642057 ->gp_wake_time 13151 ->gp_wake_seq 4639925 ->gp_seq 4639925 ->gp_seq_needed 4639932 ->gp_flags 0x0
[08:06:42]][60830.742584] rcu: rcu_node 0:7 ->gp_seq 4639925 ->gp_seq_needed 4639932
[08:06:42]][60830.742590] rcu: cpu 0 ->gp_seq_needed 4639928
[08:06:42]][60830.742596] rcu: cpu 1 ->gp_seq_needed 4639928
[08:06:42]][60830.742602] rcu: cpu 2 ->gp_seq_needed 4639928
[08:06:42]][60830.742607] rcu: cpu 3 ->gp_seq_needed 4639928
[08:06:42]][60830.742612] rcu: cpu 5 ->gp_seq_needed 4639928
[08:06:42]][60830.742618] rcu: cpu 6 ->gp_seq_needed 4639928
[08:06:42]][60830.742624] rcu: cpu 7 ->gp_seq_needed 4639932
[08:06:42]][60830.742631] rcu: RCU callbacks invoked since boot: 223104966
[08:06:42]][60830.742638] rcu_tasks: RTGS_WAIT_CBS(11) since 15207685 g:0 i:0/0 k.
[08:06:42]][60830.742645] rcu_tasks_rude: RTGS_WAIT_CBS(11) since 15207685 g:0 i:0/0 k.
[08:06:42]][60830.742653] rcu_tasks_trace: RTGS_WAIT_CBS(11) since 15207685 g:0 i:0/0 k. N0 h:0/0/0
[08:06:42]][60841.139879] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[08:06:42]][60841.145967] rcu: 4-...0: (19 ticks this GP) idle=33a/1/0x4000000000000000 softirq=3058565/3058567 fqs=1347361
[08:06:42]][60841.156032] (detected by 3, t=3644661 jiffies, g=4639925, q=4273405)
[08:06:42]][60841.156035] Task dump for CPU 4:
[08:06:42]][60841.156039] task:rcu_torture_rea state:S stack: 0 pid: 619 ppid: 2 flags:0x0000002a
[08:06:42]][60841.156047] Call trace:
[08:06:42]][60841.156058] __switch_to+0xc0/0x118
[08:06:42]][60841.156062] 0xffff000b7e46a080

Thanks,

quanli

over 3 years ago

0 Keerthy J over 3 years ago

TI__Guru**** 170880 points

Hi,

The CONFIG_PREEMPT is already enabled in arch/arm64/configs/tisdk_j784s4-evm_defconfig.

Which SDK are you using?

- Keerthy

0 li quan over 3 years ago in reply to Keerthy J

Intellectual 955 points

HI,

Keerthy J said:
Which SDK are you using?

TDA4VH SDK8.2.2

Thanks

quanli

0 li quan over 3 years ago in reply to Keerthy J

Intellectual 955 points

Hi,

the kernel is compiled with following RCU configurations:

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_EXPERT=y
CONFIG_SRCU=y
CONFIG_TREE_SRCU=y
CONFIG_TASKS_RCU_GENERIC=y
CONFIG_TASKS_RCU=y
CONFIG_TASKS_RUDE_RCU=y
CONFIG_TASKS_TRACE_RCU=y
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_NEED_SEGCBLIST=y
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
# CONFIG_RCU_FAST_NO_HZ is not set
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_DELAY=500
# CONFIG_RCU_NOCB_CPU is not set
# CONFIG_TASKS_TRACE_RCU_READ_MB is not set
# end of RCU Subsystem

#
# RCU Debugging
#
CONFIG_PROVE_RCU=y
# CONFIG_PROVE_RCU_LIST is not set
CONFIG_TORTURE_TEST=m
CONFIG_RCU_SCALE_TEST=m
CONFIG_RCU_TORTURE_TEST=m
# CONFIG_RCU_REF_SCALE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=21
CONFIG_RCU_TRACE=y
CONFIG_RCU_EQS_DEBUG=y
# CONFIG_RCU_STRICT_GRACE_PERIOD is not set
# end of RCU Debugging

#
# Lock Debugging (spinlocks, mutexes, etc...)
#
CONFIG_LOCK_DEBUGGING_SUPPORT=y
CONFIG_PROVE_LOCKING=y
# CONFIG_PROVE_RAW_LOCK_NESTING is not set
# CONFIG_LOCK_STAT is not set
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_WW_MUTEX_SLOWPATH=y
CONFIG_DEBUG_RWSEMS=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_LOCKDEP=y
# CONFIG_DEBUG_LOCKDEP is not set
# CONFIG_DEBUG_ATOMIC_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_LOCK_TORTURE_TEST is not set
# CONFIG_WW_MUTEX_SELFTEST is not set
# CONFIG_SCF_TORTURE_TEST is not set
# CONFIG_CSD_LOCK_WAIT_DEBUG is not set
# end of Lock Debugging (spinlocks, mutexes, etc...)

Thanks

quanli

0 Keerthy J over 3 years ago in reply to li quan

TI__Guru**** 170880 points

Hi Quanli,

Thanks for the details. Lastt couple of questions.

Is this a regression that you are reporting or you ran for the first time on 8.2 SDK & this failed?

li quan said:
15:13:07]]root@j7-evm:/run/media/mmcblk1p1# insmod torture.ko

[15:13:12]]root@j7-evm:/run/media/mmcblk1p1# insmod rcutorture.ko

One more suggestion is can you run memtester continuously & check if there are any issues?

Best Regards,
Keerthy

0 li quan over 3 years ago in reply to Keerthy J

Intellectual 955 points

Hi, Keerthy

Keerthy J said:
The above 2 modules insertion is good enough to reproduce the issue or you need to run any other test case in parallel?

Yes, the above 2 modules insertion is good enough to reproduce the issue. The issue will occur after insmod 2 modules （about 5-10hours）every time。

Keerthy J said:
One more suggestion is can you run memtester continuously & check if there are any issues?

Thanks for your suggestion , I'm running memtester. I will reply to you if there is a problem.

Thanks

quanli

0 li quan over 3 years ago in reply to Keerthy J

Intellectual 955 points

Hi, Keerthy

There is no problem with the memtester test。

Thanks

quanli

0 Keerthy J over 3 years ago in reply to li quan

TI__Guru**** 170880 points

Hi,

Thanks for confirming so there is no issue with DDR. Is this impacting any of your user case?

Regards,

Keerthy

0 li quan over 3 years ago in reply to Keerthy J

Intellectual 955 points

Hi, Keerthy

The rcu cpu stall warnning occur while we run Design Validation Test (system Full load) and the system stuck.

[CPU7] time:thread 2.724192s Feq:1970 Mhz real:2.797871s Bandwidth:40.815072 GB/s

[CPU4] time:thread 2.702266s Feq:1986 Mhz real:2.822112s[17755.387587] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:

[17755.393728] rcu: 5-...0: (1 ticks this GP) idle=51e/1/0x4000000000000002 softirq=202913/202914 fqs=1250644

[17818.407590] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:

[17818.413727] rcu: 5-...0: (1 ticks this GP) idle=51e/1/0x4000000000000002 softirq=202913/202914 fqs=1257695

[17881.427584] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:

[17881.433730] rcu: 5-...0: (1 ticks this GP) idle=51e/1/0x4000000000000002 softirq=202913/202914 fqs=1264853

[17944.447602] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:

[17944.453721] rcu: 5-...0: (1 ticks this GP) idle=51e/1/0x4000000000000002 softirq=202913/202914 fqs=1271935

[18007.467583] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:

Thanks

quanli

0 Keerthy J over 3 years ago in reply to li quan

TI__Guru**** 170880 points

Hi,

Thanks for the logs. We have reproduced similar failures with TDA4VH while running stress tests. Investaigations are on going. We will keep you posted on this thread.

Best Regards,

Keerthy

0 li quan over 3 years ago in reply to Keerthy J

Intellectual 955 points

Hi, Keerthy

Any progress on RCU issues？ Now, we can avoid rcu problems by modifying the number of Linux system processors (dual core A72 run Linux and six core A72 run freertos).

Thanks

quanli

0 Keelung Yang over 3 years ago in reply to Keerthy J

Expert 1796 points

Keerthy J, I'm meeting the similar issue while trying to support more then 8 cameras on TDAVH, PSDKR v8.2.2

On my side, this issue can be reoccurred by enable CSIRx2 J7_Capture.h in tiovx and single came app

0 li quan over 3 years ago in reply to li quan

Intellectual 955 points

Hi, Keerthy
we try run ti-linux-kernel-08.05.00.004(5.10.153) on the TDA4VH EVM board to perform the RCU stress test(8-core A72 running linux), the problem is also reproduced.
log see attachment.

Thanks

Fullscreen linux5.10.153-rcu-oops.txt Download

[  720.323809] rcu-torture: rcu_torture_read_exit: End of episode
[  733.713673] rcu-torture: rcu_torture_read_exit: Start of episode
[  733.724358] rcu-torture: rcu_torture_read_exit: End of episode
[  747.089598] rcu-torture: rcu_torture_read_exit: Start of episode
[  747.104170] rcu-torture: rcu_torture_read_exit: End of episode
[  760.593578] rcu-torture: rcu_torture_read_exit: Start of episode
[  760.601676] rcu-torture: rcu_torture_read_exit: End of episode
[  766.321565] rcu_torture_fwd_prog_nr: Duration 3372 cver 92 gps 133
[  766.705662] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[  766.714443] Mem abort info:
[  766.717226]   ESR = 0x86000006
[  766.720299]   EC = 0x21: IABT (current EL), IL = 32 bits
[  766.725620]   SET = 0, FnV = 0
[  766.728664]   EA = 0, S1PTW = 0
[  766.731821] user pgtable: 4k pages, 48-bit VAs, pgdp=00000008874e6000
[  766.738267] [0000000000000000] pgd=0000000882b95003, p4d=0000000882b95003, pud=00000008838f9003, pmd=0000000000000000
[  766.748884] Internal error: Oops: 86000006 [#1] PREEMPT SMP
[  766.754442] Modules linked in: rcutorture torture
[  766.759143] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.10.153 #1
[  766.765219] Hardware name: Texas Instruments J784S4 EVM (DT)
[  766.770865] pstate: 00000005 (nzcv daif -PAN -UAO -TCO BTYPE=--)
[  766.776857] pc : 0x0
[  766.779041] lr : rcu_core+0x304/0x5a8
[  766.782690] sp : ffff80001124be50
[  766.785994] x29: ffff80001124be50 x28: ffff0008002a8e40 
[  766.791294] x27: 000000b28364704a x26: ffff8000100d073c 
[  766.796595] x25: 0000000000002710 x24: ffff80001124bec0 
[  766.801899] x23: ffff8000111216c0 x22: ffff000f7bfa7e70 
[  766.807197] x21: ffff0008002a8e40 x20: ffff0008002a8e40 
[  766.812495] x19: ffff000f7bfa7e00 x18: 0000000000000000 
[  766.817794] x17: 0000000000000000 x16: 0000000000000000 
[  766.823093] x15: 0000000000000afa x14: 0000000000000bbc 
[  766.828390] x13: 0000000000000003 x12: 0000000000000003 
[  766.833688] x11: 0000000000000004 x10: 0000000000000002 
[  766.838986] x9 : ffff000800355e00 x8 : 0000000000000004 
[  766.844283] x7 : 0000000000072003 x6 : 000000004ddd3aae 
[  766.849581] x5 : 000000000000009f x4 : 000000000000000a 
[  766.854880] x3 : 0000000000000026 x2 : ffff000807d3be00 
[  766.860177] x1 : 0000000000000000 x0 : ffff000807d3bd80 
[  766.865476] Call trace:
[  766.867915]  0x0
[  766.869750]  rcu_core_si+0x10/0x20
[  766.873143]  efi_header_end+0x120/0x268
[  766.876969]  irq_exit+0xc0/0xe0
[  766.880102]  __handle_domain_irq+0x68/0xc0
[  766.884190]  gic_handle_irq+0x58/0x128
[  766.887929]  el1_irq+0xcc/0x180
[  766.891064]  arch_cpu_idle+0x18/0x28
[  766.894633]  default_idle_call+0x20/0x68
[  766.898546]  do_idle+0xc0/0x128
[  766.901679]  cpu_startup_entry+0x24/0x60
[  766.905593]  secondary_start_kernel+0x14c/0x178
[  766.910116] Code: bad PC value
[  766.913172] ---[ end trace f04df93615417ece ]---
[  766.917778] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[  766.924642] SMP: stopping secondary CPUs
[  766.928558] Kernel Offset: disabled
[  766.932034] CPU features: 0x28040022,20006008
[  766.936375] Memory Limit: none
[  766.939424] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

0 Keerthy J over 3 years ago in reply to li quan

TI__Guru**** 170880 points

Hi,

Thanks a lot for trying on the latest SDK! We have been doing multiple experiments in root causing this issue.

Increasing core/MPU voltage but we still see the rcu crash.
Tried combinations of enabling/disabling L3 cache but still the issue occurred.

We have involved Hardware experts for looking into this. We will keep you posted on updates.

Best Regards,
Keerthy

0 Keerthy J over 3 years ago in reply to Keerthy J

TI__Guru**** 170880 points

Hi quan,

On your testing observations, how long does it take to crash? Is it consistently crashing within 15 minutes? I see that gro your log it takes about 766 seconds. Is that consistent?

Best Regards,

Keerthy

0 li quan over 3 years ago in reply to Keerthy J

Intellectual 955 points

Hi, Keerthy

The actual time of occurrence ranges from ten minutes to more than ten hours，we have another log that runs for ten hours with an exception.

Thanks

Fullscreen linux5.10.153-rcu-oops-2.txt Download

[38771.127170] rcu_torture_fwd_prog_nr: Duration 4872 cver 123 gps 188
[38771.456204] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[38771.464986] Mem abort info:
[38771.467780]   ESR = 0x86000006
[38771.470823]   EC = 0x21: IABT (current EL), IL = 32 bits
[38771.476130]   SET = 0, FnV = 0
[38771.479184]   EA = 0, S1PTW = 0
[38771.482317] user pgtable: 4k pages, 48-bit VAs, pgdp=00000008e0ef2000
[38771.488751] [0000000000000000] pgd=00000008e0ee8003, p4d=00000008e0ee8003, pud=00000008e0ee7003, pmd=0000000000000000
[38771.499357] Internal error: Oops: 86000006 [#1] PREEMPT SMP
[38771.504915] Modules linked in: rcutorture torture
[38771.509614] CPU: 7 PID: 48 Comm: ksoftirqd/7 Not tainted 5.10.153 #1
[38771.515951] Hardware name: Texas Instruments J784S4 EVM (DT)
[38771.521596] pstate: 00000005 (nzcv daif -PAN -UAO -TCO BTYPE=--)
[38771.527589] pc : 0x0
[38771.529772] lr : rcu_core+0x2f4/0x588
[38771.533421] sp : ffff800011643ce0
[38771.536725] x29: ffff800011643ce0 x28: ffff0008602e5580 
[38771.542024] x27: 000023432f06a465 x26: ffff800011643d40 
[38771.547323] x25: ffff000b7e4acdf0 x24: ffff80001111ffc0 
[38771.552623] x23: ffff8000100cffd8 x22: ffff0008602e5580 
[38771.557920] x21: 0000000000002710 x20: ffff0008602e5580 
[38771.563219] x19: ffff000b7e4acd80 x18: 0000000000000002 
[38771.568519] x17: 0000000000000000 x16: 00000000e0e62fdb 
[38771.573818] x15: 00000000f5257d14 x14: 00000000000002d9 
[38771.579118] x13: 0000000000000000 x12: 0000000000000cd8 
[38771.584416] x11: 000000000000016a x10: 00000000000009d0 
[38771.589715] x9 : ffff800011643d50 x8 : ffff0008602e5fb0 
[38771.595015] x7 : ffff000b7e4ac180 x6 : 00000666b8ecc110 
[38771.600313] x5 : 000000000000009f x4 : 0000000000000008 
[38771.605612] x3 : 000000000000001e x2 : ffff000865fc3400 
[38771.610911] x1 : 0000000000000000 x0 : ffff000865fc3380 
[38771.616209] Call trace:
[38771.618647]  0x0
[38771.620482]  rcu_core_si+0x10/0x20
[38771.623876]  efi_header_end+0x120/0x268
[38771.627702]  run_ksoftirqd+0x4c/0x60
[38771.631268]  smpboot_thread_fn+0x1c4/0x280
[38771.635353]  kthread+0x140/0x160
[38771.638569]  ret_from_fork+0x10/0x34
[38771.642139] Code: bad PC value
[38771.645193] ---[ end trace 0cbd287974ba8c9c ]---
[38771.649797] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[38771.656658] SMP: stopping secondary CPUs
[38771.660575] Kernel Offset: disabled
[38771.664052] CPU features: 0x28040022,20006008
[38771.668392] Memory Limit: none
[38771.671442] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

0 Keerthy J over 3 years ago in reply to li quan

TI__Guru**** 170880 points

Hi,

Do you have the modules enabled? Do you have all the remote cores enabled? Can you try if this is seen with
all the modules disabled? We are trying out experiments in parallel.

- Keerthy

0 li quan over 3 years ago in reply to Keerthy J

Intellectual 955 points

Hi, Keerthy

The modules & the remote cores are enabled， we will do experiments with all the modules disabled.

Thanks

Processors

Processors forum

TDA4VM: TDA4VH RCU CPU stall warning.