This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5726: thermal thermal_zone5: failed to read out thermal zone - kernel stall

Part Number: AM5726

During some overnight testing at room temperature with the TI EVM the system stopped functioning and the following was logged in the serial port console, the kernel stalled it appears as well.

The EVM had to be reset to recover.   

[582255.976697] omap_i2c 48070000.i2c: controller timed out 2

[582256.006722] thermal thermal_zone5: failed to read out thermal zone (-110) 3

[582258.616712] omap_i2c 48070000.i2c: controller timed out 4

[582258.646747] thermal thermal_zone5: failed to read out thermal zone (-110) 5

[582261.256765] omap_i2c 48070000.i2c: controller timed out 6

[582261.286764] thermal thermal_zone5: failed to read out thermal zone (-110)

...

Additionnaly

[582310.057069] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[582310.063119] rcu: 0-....: (16658690 ticks this GP) idle=442/1/0x40000004 softirq=3593341/3593427 fqs=8313108
[582310.073164] rcu: (detected by 1, t=16798622 jiffies, g=276861, q=433)
[582310.079817] Sending NMI from CPU 1 to CPUs 0:
[582310.084561] NMI backtrace for cpu 0
[582310.084564] CPU: 0 PID: 32308 Comm: python3 Tainted: G W O 4.19.94-g5a23bc00e0 #1
[582310.084566] Hardware name: Generic DRA74X (Flattened Device Tree)
[582310.084568] PC is at __do_softirq+0xa8/0x28c
[582310.084570] LR is at 0x404040
[582310.084572] pc : [<c02020e0>] lr : [<00404040>] psr: 60000113
[582310.084574] sp : d3089b20 ip : 00404040 fp : d3089b7c
[582310.084576] r10: 00000001 r9 : d3088000 r8 : ee80c000
[582310.084578] r7 : 00000001 r6 : 00000000 r5 : 00000000 r4 : 00000282
[582310.084580] r3 : c1067d40 r2 : 2dce7000 r1 : 00000000 r0 : 2dce7000
[582310.084582] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[582310.084584] Control: 30c5387d Table: 9307ccc0 DAC: fffffffd
[582310.084587] CPU: 0 PID: 32308 Comm: python3 Tainted: G W O 4.19.94-g5a23bc00e0 #1
[582310.084589] Hardware name: Generic DRA74X (Flattened Device Tree)
[582310.084590] Backtrace:
[582310.084594] [<c020ca34>] (dump_backtrace) from [<c020cd6c>] (show_stack+0x18/0x1c)
[582310.084596] r7:d3089ad0 r6:60000193 r5:00000000 r4:c12502f0
[582310.084598] [<c020cd54>] (show_stack) from [<c09fcac4>] (dump_stack+0x9c/0xb0)
[582310.084601] [<c09fca28>] (dump_stack) from [<c020912c>] (show_regs+0x14/0x18)
[582310.084602] r7:d3089ad0 r6:00000000 r5:d3089ad0 r4:00000000
[582310.084605] [<c0209118>] (show_regs) from [<c0a035fc>] (nmi_cpu_backtrace+0xc0/0xc4)
[582310.084607] [<c0a0353c>] (nmi_cpu_backtrace) from [<c020f5e4>] (handle_IPI+0xdc/0x1dc)
[582310.084609] r5:d3089be0 r4:c1067cfc
[582310.084611] [<c020f508>] (handle_IPI) from [<c0553550>] (gic_handle_irq+0x7c/0x80)
[582310.084613] r6:fa212000 r5:fa21200c r4:c1205100
[582310.084615] [<c05534d4>] (gic_handle_irq) from [<c02019f8>] (__irq_svc+0x58/0x8c)
[582310.084617] Exception stack(0xd3089ad0 to 0xd3089b18)
[582310.084619] 9ac0: 2dce7000 00000000 2dce7000 c1067d40
[582310.084622] 9ae0: 00000282 00000000 00000000 00000001 ee80c000 d3088000 00000001 d3089b7c
[582310.084624] 9b00: 00404040 d3089b20 00404040 c02020e0 60000113 ffffffff
[582310.084626] r9:d3088000 r8:ee80c000 r7:d3089b04 r6:ffffffff r5:60000113 r4:c02020e0
[582310.084628] [<c0202038>] (__do_softirq) from [<c0232e68>] (irq_exit+0xd4/0x110)
[582310.084630] r10:00000001 r9:d3088000 r8:ee80c000 r7:00000001 r6:00000000 r5:00000000
[582310.084632] r4:c1067cfc
[582310.084634] [<c0232d94>] (irq_exit) from [<c0288900>] (__handle_domain_irq+0x68/0xbc)
[582310.084637] [<c0288898>] (__handle_domain_irq) from [<c0553518>] (gic_handle_irq+0x44/0x80)
[582310.084639] r9:d3088000 r8:fa213000 r7:d3089be0 r6:fa212000 r5:fa21200c r4:c1205100
[582310.084641] [<c05534d4>] (gic_handle_irq) from [<c02019f8>] (__irq_svc+0x58/0x8c)
[582310.084643] Exception stack(0xd3089be0 to 0xd3089c28)
[582310.084645] 9be0: c128c9f0 00000017 0000000b ae2d8724 ee2c3240 c1204c48 ee2c32b8 c128c9f0
[582310.084647] 9c00: df903000 ed20d200 00000001 d3089c94 d3089c98 d3089c30 c061f6c8 c061ec80
[582310.084649] 9c20: 20000013 ffffffff
[582310.084651] r9:d3088000 r8:df903000 r7:d3089c14 r6:ffffffff r5:20000013 r4:c061ec80
[582310.084653] [<c061eb40>] (omap_8250_rx_dma) from [<c061f6c8>] (omap_8250_startup+0x158/0x1f4)
[582310.084655] r10:c0c443c0 r9:ed20d200 r8:df903000 r7:00000000 r6:00000000 r5:ee2c3240
[582310.084657] r4:c128c9f0
[582310.084659] [<c061f570>] (omap_8250_startup) from [<c0619748>] (serial8250_startup+0x1c/0x28)
[582310.084661] r6:ed20d200 r5:c128c9f0 r4:eebba8b8
[582310.084663] [<c061972c>] (serial8250_startup) from [<c0614240>] (uart_startup.part.4+0x1a0/0x264)
[582310.084665] [<c06140a0>] (uart_startup.part.4) from [<c061435c>] (uart_port_activate+0x58/0x7c)
[582310.084667] r9:ed20d200 r8:ed2f33c0 r7:eebba94c r6:eebba91c r5:00000000 r4:eebba8b8
[582310.084670] [<c0614304>] (uart_port_activate) from [<c05fcafc>] (tty_port_open+0x84/0xc8)
[582310.084671] r5:ed20d200 r4:eebba8b8
[582310.084673] [<c05fca78>] (tty_port_open) from [<c0612bb0>] (uart_open+0x20/0x28)
[582310.084675] r9:ed20d200 r8:d3088000 r7:c0e653fc r6:00400049 r5:ed20d200 r4:ed2f33c0
[582310.084678] [<c0612b90>] (uart_open) from [<c05f5a08>] (tty_open+0xf0/0x458)
[582310.084680] [<c05f5918>] (tty_open) from [<c0359ccc>] (chrdev_open+0xb4/0x19c)
[582310.084682] r10:00000006 r9:c0359c18 r8:ed2f33c0 r7:ee2e2c00 r6:ee2c6bc0 r5:c1204c48
[582310.084684] r4:c0c44440
[582310.084686] [<c0359c18>] (chrdev_open) from [<c0351488>] (do_dentry_open+0x11c/0x3c0)
[582310.084688] r9:c0359c18 r8:ed2f33c8 r7:00020902 r6:00000000 r5:ee2e2c00 r4:ed2f33c0
[582310.084690] [<c035136c>] (do_dentry_open) from [<c035299c>] (vfs_open+0x34/0x38)
[582310.084692] r9:ed2f33c0 r8:ee4023b8 r7:00020902 r6:00000000 r5:00000000 r4:d3089e90
[582310.084695] [<c0352968>] (vfs_open) from [<c0364308>] (path_openat+0x2b8/0x1114)
[582310.084697] [<c0364050>] (path_openat) from [<c03661ec>] (do_filp_open+0x7c/0xe4)
[582310.084699] r10:00000142 r9:d3088000 r8:00000001 r7:d3089f50 r6:d3089e90 r5:c1204c48
[582310.084701] r4:c1204c48
[582310.084702] [<c0366170>] (do_filp_open) from [<c0352c98>] (do_sys_open+0x170/0x1f4)
[582310.084705] r8:fffff000 r7:ee8e6000 r6:ffffff9c r5:c1204c48 r4:00000006
[582310.084707] [<c0352b28>] (do_sys_open) from [<c0352d54>] (sys_openat+0x14/0x18)
[582310.084709] r10:00000142 r9:d3088000 r8:c0201204 r7:00000142 r6:000001ff r5:b6e69000
[582310.084711] r4:b5f32390
[582310.084713] [<c0352d40>] (sys_openat) from [<c0201000>] (ret_fast_syscall+0x0/0x4c)
[582310.084715] Exception stack(0xd3089fa8 to 0xd3089ff0)
[582310.084717] 9fa0: b5f32390 b6e69000 ffffff9c b5f32390 000a0902 00000000
[582310.084719] 9fc0: b5f32390 b6e69000 000001ff 00000142 00080902 b5e13d64 0028d650 b5e13d70
[582310.084721] 9fe0: 00000000 b5e13cd8 b5e16920 b6c78e28
[582310.087069] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[582310.605242] rcu: 0-....: (1 GPs behind) idle=442/1/0x40000004 softirq=3593426/3593427 fqs=8318175
[582310.614416] rcu: (detected by 1, t=16810088 jiffies, g=5190205, q=1403330)
[582310.621503] Sending NMI from CPU 1 to CPUs 0:
[582310.626239] NMI backtrace for cpu 0
[582310.626242] CPU: 0 PID: 32308 Comm: python3 Tainted: G W O 4.19.94-g5a23bc00e0 #1
[582310.626245] Hardware name: Generic DRA74X (Flattened Device Tree)
[582310.626246] PC is at __do_softirq+0xa8/0x28c
[582310.626248] LR is at 0x404040
[582310.626250] pc : [<c02020e0>] lr : [<00404040>] psr: 60000113
[582310.626252] sp : d3089b20 ip : 00404040 fp : d3089b7c
[582310.626254] r10: 00000001 r9 : d3088000 r8 : ee80c000
[582310.626256] r7 : 00000001 r6 : 00000000 r5 : 00000000 r4 : 00000282
[582310.626258] r3 : c1067d40 r2 : 2dce7000 r1 : 00000000 r0 : 2dce7000
[582310.626260] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[582310.626262] Control: 30c5387d Table: 9307ccc0 DAC: fffffffd
[582310.626264] CPU: 0 PID: 32308 Comm: python3 Tainted: G W O 4.19.94-g5a23bc00e0 #1
[582310.626266] Hardware name: Generic DRA74X (Flattened Device Tree)
[582310.626268] Backtrace:
[582310.626272] [<c020ca34>] (dump_backtrace) from [<c020cd6c>] (show_stack+0x18/0x1c)
[582310.626274] r7:d3089ad0 r6:60000193 r5:00000000 r4:c12502f0
[582310.626276] [<c020cd54>] (show_stack) from [<c09fcac4>] (dump_stack+0x9c/0xb0)
[582310.626278] [<c09fca28>] (dump_stack) from [<c020912c>] (show_regs+0x14/0x18)
[582310.626280] r7:d3089ad0 r6:00000000 r5:d3089ad0 r4:00000000
[582310.626282] [<c0209118>] (show_regs) from [<c0a035fc>] (nmi_cpu_backtrace+0xc0/0xc4)
[582310.626284] [<c0a0353c>] (nmi_cpu_backtrace) from [<c020f5e4>] (handle_IPI+0xdc/0x1dc)
[582310.626286] r5:d3089be0 r4:c1067cfc
[582310.626288] [<c020f508>] (handle_IPI) from [<c0553550>] (gic_handle_irq+0x7c/0x80)
[582310.626290] r6:fa212000 r5:fa21200c r4:c1205100
[582310.626293] [<c05534d4>] (gic_handle_irq) from [<c02019f8>] (__irq_svc+0x58/0x8c)
[582310.626294] Exception stack(0xd3089ad0 to 0xd3089b18)
[582310.626296] 9ac0: 2dce7000 00000000 2dce7000 c1067d40
[582310.626299] 9ae0: 00000282 00000000 00000000 00000001 ee80c000 d3088000 00000001 d3089b7c
[582310.626301] 9b00: 00404040 d3089b20 00404040 c02020e0 60000113 ffffffff
[582310.626303] r9:d3088000 r8:ee80c000 r7:d3089b04 r6:ffffffff r5:60000113 r4:c02020e0
[582310.626305] [<c0202038>] (__do_softirq) from [<c0232e68>] (irq_exit+0xd4/0x110)
[582310.626307] r10:00000001 r9:d3088000 r8:ee80c000 r7:00000001 r6:00000000 r5:00000000
[582310.626309] r4:c1067cfc
[582310.626311] [<c0232d94>] (irq_exit) from [<c0288900>] (__handle_domain_irq+0x68/0xbc)
[582310.626313] [<c0288898>] (__handle_domain_irq) from [<c0553518>] (gic_handle_irq+0x44/0x80)
[582310.626315] r9:d3088000 r8:fa213000 r7:d3089be0 r6:fa212000 r5:fa21200c r4:c1205100
[582310.626317] [<c05534d4>] (gic_handle_irq) from [<c02019f8>] (__irq_svc+0x58/0x8c)
[582310.626319] Exception stack(0xd3089be0 to 0xd3089c28)
[582310.626321] 9be0: c128c9f0 00000017 0000000b ae2d8724 ee2c3240 c1204c48 ee2c32b8 c128c9f0
[582310.626324] 9c00: df903000 ed20d200 00000001 d3089c94 d3089c98 d3089c30 c061f6c8 c061ec80
[582310.626325] 9c20: 20000013 ffffffff
[582310.626327] r9:d3088000 r8:df903000 r7:d3089c14 r6:ffffffff r5:20000013 r4:c061ec80
[582310.626330] [<c061eb40>] (omap_8250_rx_dma) from [<c061f6c8>] (omap_8250_startup+0x158/0x1f4)
[582310.626332] r10:c0c443c0 r9:ed20d200 r8:df903000 r7:00000000 r6:00000000 r5:ee2c3240
[582310.626333] r4:c128c9f0
[582310.626335] [<c061f570>] (omap_8250_startup) from [<c0619748>] (serial8250_startup+0x1c/0x28)
[582310.626337] r6:ed20d200 r5:c128c9f0 r4:eebba8b8
[582310.626340] [<c061972c>] (serial8250_startup) from [<c0614240>] (uart_startup.part.4+0x1a0/0x264)
[582310.626342] [<c06140a0>] (uart_startup.part.4) from [<c061435c>] (uart_port_activate+0x58/0x7c)
[582310.626344] r9:ed20d200 r8:ed2f33c0 r7:eebba94c r6:eebba91c r5:00000000 r4:eebba8b8
[582310.626346] [<c0614304>] (uart_port_activate) from [<c05fcafc>] (tty_port_open+0x84/0xc8)
[582310.626348] r5:ed20d200 r4:eebba8b8
[582310.626350] [<c05fca78>] (tty_port_open) from [<c0612bb0>] (uart_open+0x20/0x28)
[582310.626352] r9:ed20d200 r8:d3088000 r7:c0e653fc r6:00400049 r5:ed20d200 r4:ed2f33c0
[582310.626354] [<c0612b90>] (uart_open) from [<c05f5a08>] (tty_open+0xf0/0x458)
[582310.626356] [<c05f5918>] (tty_open) from [<c0359ccc>] (chrdev_open+0xb4/0x19c)
[582310.626359] r10:00000006 r9:c0359c18 r8:ed2f33c0 r7:ee2e2c00 r6:ee2c6bc0 r5:c1204c48
[582310.626360] r4:c0c44440
[582310.626362] [<c0359c18>] (chrdev_open) from [<c0351488>] (do_dentry_open+0x11c/0x3c0)
[582310.626364] r9:c0359c18 r8:ed2f33c8 r7:00020902 r6:00000000 r5:ee2e2c00 r4:ed2f33c0
[582310.626366] [<c035136c>] (do_dentry_open) from [<c035299c>] (vfs_open+0x34/0x38)
[582310.626369] r9:ed2f33c0 r8:ee4023b8 r7:00020902 r6:00000000 r5:00000000 r4:d3089e90
[582310.626371] [<c0352968>] (vfs_open) from [<c0364308>] (path_openat+0x2b8/0x1114)
[582310.626373] [<c0364050>] (path_openat) from [<c03661ec>] (do_filp_open+0x7c/0xe4)
[582310.626375] r10:00000142 r9:d3088000 r8:00000001 r7:d3089f50 r6:d3089e90 r5:c1204c48
[582310.626376] r4:c1204c48
[582310.626379] [<c0366170>] (do_filp_open) from [<c0352c98>] (do_sys_open+0x170/0x1f4)
[582310.626381] r8:fffff000 r7:ee8e6000 r6:ffffff9c r5:c1204c48 r4:00000006
[582310.626383] [<c0352b28>] (do_sys_open) from [<c0352d54>] (sys_openat+0x14/0x18)
[582310.626385] r10:00000142 r9:d3088000 r8:c0201204 r7:00000142 r6:000001ff r5:b6e69000
[582310.626386] r4:b5f32390
[582310.626389] [<c0352d40>] (sys_openat) from [<c0201000>] (ret_fast_syscall+0x0/0x4c)
[582310.626390] Exception stack(0xd3089fa8 to 0xd3089ff0)
[582310.626393] 9fa0: b5f32390 b6e69000 ffffff9c b5f32390 000a0902 00000000
[582310.626395] 9fc0: b5f32390 b6e69000 000001ff 00000142 00080902 b5e13d64 0028d650 b5e13d70

Yocto Build

ID="arago"
NAME="Arago"
VERSION="2019.11"
VERSION_ID="2019.11"
PRETTY_NAME="Arago 2019.11"

  • Hi,

    I am using the latest 6.3 SDK: software-dl.ti.com/.../index_FDS.html

    The below ouput.

    root@am57xx-evm:~# cat /sys/class/thermal/thermal_zone*/type
    cpu_thermal
    gpu_thermal
    core_thermal
    dspeve_thermal
    iva_thermal

    Which SDK version are you using? I do not the thermal_zone5 on AM574x-IDK. Which board are you using? I believe since it needs I2C it should be an on board temperature
    sensor. Can you answer the above questions?

    - Keerthy