Hi,
We are doing thermal testing for our custom boards at -20 degree Celsius temperature. When the chamber reaches the -20 we start getting the below rcu stall errors. at normal room temperature we have never observed this logs.
[ 3370.144258] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 3370.150371] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=2602
[ 3370.158623] (detected by 2, t=5252 jiffies, g=778213, q=503)
[ 3370.164356] Task dump for CPU 3:
[ 3370.167576] swapper/3 R running task 0 0 1 0x0000002a
[ 3370.174616] Call trace:
[ 3370.177069] __switch_to+0x100/0x1c0
[ 3370.180639] 0xe80008d5a40f53
[ 3433.164258] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 3433.170370] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=10475
[ 3433.178711] (detected by 2, t=21007 jiffies, g=778213, q=1604)
[ 3433.184620] Task dump for CPU 3:
[ 3433.187844] swapper/3 R running task 0 0 1 0x0000002a
[ 3433.194884] Call trace:
[ 3433.197338] __switch_to+0x100/0x1c0
[ 3433.200908] 0xe80008d5a40f53
[ 3496.184254] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 3496.190364] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=18349
[ 3496.198702] (detected by 1, t=36764 jiffies, g=778213, q=2667)
[ 3496.204609] Task dump for CPU 3:
[ 3496.207832] swapper/3 R running task 0 0 1 0x0000002a
[ 3496.214870] Call trace:
[ 3496.217323] __switch_to+0x100/0x1c0
[ 3496.220894] 0xe80008d5a40f53
[ 3559.204255] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 3559.210367] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=25838
[ 3559.218706] (detected by 1, t=52520 jiffies, g=778213, q=3734)
[ 3559.224615] Task dump for CPU 3:
[ 3559.227836] swapper/3 R running task 0 0 1 0x0000002a
[ 3559.234878] Call trace:
[ 3559.237335] __switch_to+0x100/0x1c0
[ 3559.240906] 0xe80008d5a40f53
[ 3622.224256] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 3622.230363] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=32559
[ 3622.238700] (detected by 0, t=68275 jiffies, g=778213, q=4814)
[ 3622.244608] Task dump for CPU 3:
[ 3622.247832] swapper/3 R running task 0 0 1 0x0000002a
[ 3622.254872] Call trace:
[ 3622.257326] __switch_to+0x100/0x1c0
[ 3622.260897] 0xe80008d5a40f53
[ 3685.244256] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 3685.250367] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=39289
[ 3685.258705] (detected by 2, t=84027 jiffies, g=778213, q=5580)
[ 3685.264610] Task dump for CPU 3:
[ 3685.267830] swapper/3 R running task 0 0 1 0x0000002a
[ 3685.274871] Call trace:
[ 3685.277324] __switch_to+0x100/0x1c0
[ 3685.280894] 0xe80008d5a40f53
[ 3748.264254] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 3748.270364] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=45995
[ 3748.278702] (detected by 1, t=99785 jiffies, g=778213, q=5963)
[ 3748.284609] Task dump for CPU 3:
[ 3748.287830] swapper/3 R running task 0 0 1 0x0000002a
[ 3748.294870] Call trace:
[ 3748.297324] __switch_to+0x100/0x1c0
[ 3748.300894] 0xe80008d5a40f53
[ 3811.284254] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 3811.290362] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=52992
[ 3811.298701] (detected by 1, t=115540 jiffies, g=778213, q=6356)
[ 3811.304695] Task dump for CPU 3:
[ 3811.307916] swapper/3 R running task 0 0 1 0x0000002a
[ 3811.314955] Call trace:
[ 3811.317408] __switch_to+0x100/0x1c0
[ 3811.320979] 0xe80008d5a40f53
[ 3874.304255] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 3874.310364] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=59999
[ 3874.318701] (detected by 1, t=131295 jiffies, g=778213, q=6743)
[ 3874.324695] Task dump for CPU 3:
[ 3874.327918] swapper/3 R running task 0 0 1 0x0000002a
[ 3874.334958] Call trace:
[ 3874.337412] __switch_to+0x100/0x1c0
[ 3874.340985] 0xe80008d5a40f53
[ 3937.324255] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 3937.330364] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=66992
[ 3937.338702] (detected by 0, t=147050 jiffies, g=778213, q=7145)
[ 3937.344695] Task dump for CPU 3:
[ 3937.347917] swapper/3 R running task 0 0 1 0x0000002a
[ 3937.354957] Call trace:
[ 3937.357410] __switch_to+0x100/0x1c0
[ 3937.360980] 0xe80008d5a40f53
[ 4000.344255] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4000.350363] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=74003
[ 4000.358702] (detected by 1, t=162805 jiffies, g=778213, q=7556)
[ 4000.364697] Task dump for CPU 3:
[ 4000.367918] swapper/3 R running task 0 0 1 0x0000002a
[ 4000.374957] Call trace:
[ 4000.377410] __switch_to+0x100/0x1c0
[ 4000.380981] 0xe80008d5a40f53
[ 4063.364254] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4063.370363] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=81023
[ 4063.378701] (detected by 0, t=178560 jiffies, g=778213, q=7972)
[ 4063.384695] Task dump for CPU 3:
[ 4063.387916] swapper/3 R running task 0 0 1 0x0000002a
[ 4063.394956] Call trace:
[ 4063.397408] __switch_to+0x100/0x1c0
[ 4063.400978] 0xe80008d5a40f53
[ 4126.384255] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4126.390363] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=88038
[ 4126.398701] (detected by 0, t=194315 jiffies, g=778213, q=8363)
[ 4126.404695] Task dump for CPU 3:
[ 4126.407916] swapper/3 R running task 0 0 1 0x0000002a
[ 4126.414955] Call trace:
[ 4126.417407] __switch_to+0x100/0x1c0
[ 4126.420977] 0xe80008d5a40f53
[ 4189.404254] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4189.410362] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=95046
[ 4189.418698] (detected by 0, t=210070 jiffies, g=778213, q=8751)
[ 4189.424692] Task dump for CPU 3:
[ 4189.427913] swapper/3 R running task 0 0 1 0x0000002a
[ 4189.434952] Call trace:
[ 4189.437405] __switch_to+0x100/0x1c0
[ 4189.440976] 0xe80008d5a40f53
[ 4252.424257] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4252.430368] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=102011
[ 4252.438794] (detected by 1, t=225825 jiffies, g=778213, q=9162)
[ 4252.444788] Task dump for CPU 3:
[ 4252.448010] swapper/3 R running task 0 0 1 0x0000002a
[ 4252.455050] Call trace:
[ 4252.457503] __switch_to+0x100/0x1c0
[ 4252.461075] 0xe80008d5a40f53
[ 4315.444256] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4315.450363] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=108986
[ 4315.458787] (detected by 0, t=241580 jiffies, g=778213, q=9569)
[ 4315.464780] Task dump for CPU 3:
[ 4315.468001] swapper/3 R running task 0 0 1 0x0000002a
[ 4315.475041] Call trace:
[ 4315.477494] __switch_to+0x100/0x1c0
[ 4315.481064] 0xe80008d5a40f53
[ 4378.464258] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4378.470369] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=115930
[ 4378.478796] (detected by 1, t=257335 jiffies, g=778213, q=9962)
[ 4378.484791] Task dump for CPU 3:
[ 4378.488013] swapper/3 R running task 0 0 1 0x0000002a
[ 4378.495054] Call trace:
[ 4378.497510] __switch_to+0x100/0x1c0
[ 4378.501081] 0xe80008d5a40f53
[ 4441.484254] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4441.490362] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=122860
[ 4441.498786] (detected by 0, t=273090 jiffies, g=778213, q=10353)
[ 4441.504867] Task dump for CPU 3:
[ 4441.508089] swapper/3 R running task 0 0 1 0x0000002a
[ 4441.515127] Call trace:
[ 4441.517580] __switch_to+0x100/0x1c0
[ 4441.521150] 0xe80008d5a40f53
[ 4504.504256] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4504.510365] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=129820
[ 4504.518790] (detected by 0, t=288845 jiffies, g=778213, q=10761)
[ 4504.524871] Task dump for CPU 3:
[ 4504.528093] swapper/3 R running task 0 0 1 0x0000002a
[ 4504.535132] Call trace:
[ 4504.537586] __switch_to+0x100/0x1c0
[ 4504.541157] 0xe80008d5a40f53
[ 4567.524255] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4567.530363] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=136779
[ 4567.538787] (detected by 0, t=304600 jiffies, g=778213, q=11167)
[ 4567.544869] Task dump for CPU 3:
[ 4567.548090] swapper/3 R running task 0 0 1 0x0000002a
[ 4567.555130] Call trace:
[ 4567.557583] __switch_to+0x100/0x1c0
[ 4567.561153] 0xe80008d5a40f53
[ 4630.544256] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4630.550367] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=143721
[ 4630.558793] (detected by 1, t=320355 jiffies, g=778213, q=11565)
[ 4630.564875] Task dump for CPU 3:
[ 4630.568096] swapper/3 R running task 0 0 1 0x0000002a
[ 4630.575137] Call trace:
[ 4630.577591] __switch_to+0x100/0x1c0
[ 4630.581161] 0xe80008d5a40f53
[ 4693.564255] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4693.570365] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=150658
[ 4693.578792] (detected by 1, t=336110 jiffies, g=778213, q=11951)
[ 4693.584872] Task dump for CPU 3:
[ 4693.588093] swapper/3 R running task 0 0 1 0x0000002a
[ 4693.595132] Call trace:
[ 4693.597584] __switch_to+0x100/0x1c0
[ 4693.601155] 0xe80008d5a40f53
[ 4756.584253] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4756.590364] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=157614
[ 4756.598788] (detected by 1, t=351865 jiffies, g=778213, q=12351)
[ 4756.604869] Task dump for CPU 3:
[ 4756.608091] swapper/3 R running task 0 0 1 0x0000002a
[ 4756.615132] Call trace:
[ 4756.617586] __switch_to+0x100/0x1c0
[ 4756.621158] 0xe80008d5a40f53
[ 4819.604254] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4819.610362] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=164596
[ 4819.618787] (detected by 0, t=367620 jiffies, g=778213, q=12746)
[ 4819.624867] Task dump for CPU 3:
[ 4819.628090] swapper/3 R running task 0 0 1 0x0000002a
[ 4819.635128] Call trace:
[ 4819.637582] __switch_to+0x100/0x1c0
[ 4819.641152] 0xe80008d5a40f53
[ 4882.624256] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4882.630365] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=171529
[ 4882.638789] (detected by 0, t=383375 jiffies, g=778213, q=13163)
[ 4882.644870] Task dump for CPU 3:
[ 4882.648090] swapper/3 R running task 0 0 1 0x0000002a
[ 4882.655127] Call trace:
[ 4882.657579] __switch_to+0x100/0x1c0
[ 4882.661150] 0xe80008d5a40f53
[ 4945.644254] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 4945.650362] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=178482
[ 4945.658787] (detected by 1, t=399130 jiffies, g=778213, q=13569)
[ 4945.664868] Task dump for CPU 3:
[ 4945.668089] swapper/3 R running task 0 0 1 0x0000002a
[ 4945.675128] Call trace:
[ 4945.677579] __switch_to+0x100/0x1c0
[ 4945.681151] 0xe80008d5a40f53
[ 5008.664258] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 5008.670369] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=185406
[ 5008.678794] (detected by 1, t=414885 jiffies, g=778213, q=13970)
[ 5008.684875] Task dump for CPU 3:
[ 5008.688095] swapper/3 R running task 0 0 1 0x0000002a
[ 5008.695135] Call trace:
[ 5008.697587] __switch_to+0x100/0x1c0
[ 5008.701157] 0xe80008d5a40f53
[ 5071.684255] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 5071.690365] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=192332
[ 5071.698788] (detected by 0, t=430640 jiffies, g=778213, q=14353)
[ 5071.704868] Task dump for CPU 3:
[ 5071.708089] swapper/3 R running task 0 0 1 0x0000002a
[ 5071.715129] Call trace:
[ 5071.717582] __switch_to+0x100/0x1c0
[ 5071.721153] 0xe80008d5a40f53
[ 5134.704256] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 5134.710366] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=199259
[ 5134.718789] (detected by 0, t=446395 jiffies, g=778213, q=14744)
[ 5134.724870] Task dump for CPU 3:
[ 5134.728090] swapper/3 R running task 0 0 1 0x0000002a
[ 5134.735129] Call trace:
[ 5134.737582] __switch_to+0x100/0x1c0
[ 5134.741153] 0xe80008d5a40f53
[ 5197.724254] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 5197.730362] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=206202
[ 5197.738787] (detected by 0, t=462150 jiffies, g=778213, q=15155)
[ 5197.744868] Task dump for CPU 3:
[ 5197.748091] swapper/3 R running task 0 0 1 0x0000002a
[ 5197.755130] Call trace:
[ 5197.757584] __switch_to+0x100/0x1c0
[ 5197.761155] 0xe80008d5a40f53
[ 5260.744260] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 5260.750370] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=213126
[ 5260.758796] (detected by 0, t=477905 jiffies, g=778213, q=15554)
[ 5260.764875] Task dump for CPU 3:
[ 5260.768097] swapper/3 R running task 0 0 1 0x0000002a
[ 5260.775137] Call trace:
[ 5260.777590] __switch_to+0x100/0x1c0
[ 5260.781160] 0xe80008d5a40f53
[ 5323.764254] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 5323.770362] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=220058
[ 5323.778786] (detected by 0, t=493660 jiffies, g=778213, q=15947)
[ 5323.784868] Task dump for CPU 3:
[ 5323.788090] swapper/3 R running task 0 0 1 0x0000002a
[ 5323.795130] Call trace:
[ 5323.797584] __switch_to+0x100/0x1c0
[ 5323.801155] 0xe80008d5a40f53
[ 5386.784256] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 5386.790367] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=226992
[ 5386.798793] (detected by 0, t=509415 jiffies, g=778213, q=16354)
[ 5386.804874] Task dump for CPU 3:
[ 5386.808095] swapper/3 R running task 0 0 1 0x0000002a
[ 5386.815136] Call trace:
[ 5386.817589] __switch_to+0x100/0x1c0
[ 5386.821159] 0xe80008d5a40f53
[ 5449.804255] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 5449.810362] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=233912
[ 5449.818787] (detected by 1, t=525170 jiffies, g=778213, q=16741)
[ 5449.824867] Task dump for CPU 3:
[ 5449.828089] swapper/3 R running task 0 0 1 0x0000002a
[ 5449.835131] Call trace:
[ 5449.837586] __switch_to+0x100/0x1c0
[ 5449.841158] 0xe80008d5a40f53
[ 5512.824255] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 5512.830364] rcu: 3-...0: (1 ticks this GP) idle=61e/0/0x1 softirq=269120/269120 fqs=240851
[ 5512.838788] (detected by 0, t=540925 jiffies, g=778213, q=17147)
[ 5512.844867] Task dump for CPU 3:
[ 5512.848088] swapper/3 R running task 0 0 1 0x0000002a
[ 5512.855128] Call trace:
[ 5512.857581] __switch_to+0x100/0x1c0
[ 5512.861152] 0xe80008d5a40f53
We are using SDK 7.1 for testing. Anything we are missing here? Issue is only observed at -20 temp.
