Other Parts Discussed in Thread: AM4378,
Tool/software: Linux
We have a product that is running embedded linux on an AM4378 and uses a WL1837MOD for wifi support. We have been seeing an issue where, across dozens of test machines, around once a week one of the machines will completely freeze up and will not even respond using a serial console. We have finally managed to capture the serial console output immediately prior to the occurrence of this error, and there appears to be a kernel panic occurring while executing code within the kernel wifi stack:
[147700.320248] wlcore: down [147760.720488] wlcore: PHY firmware version: Rev 8.2.0.0.236 [147761.008867] wlcore: firmware booted (Rev 8.9.0.0.69) [147880.316402] wlcore: down [147940.734170] wlcore: PHY firmware version: Rev 8.2.0.0.236 [147941.020464] wlcore: firmware booted (Rev 8.9.0.0.69) [148060.335751] wlcore: down [148120.738197] wlcore: PHY firmware version: Rev 8.2.0.0.236 [148121.018737] wlcore: firmware booted (Rev 8.9.0.0.69) [148240.340744] wlcore: down [148300.721253] wlcore: PHY firmware version: Rev 8.2.0.0.236 [148301.009512] wlcore: firmware booted (Rev 8.9.0.0.69) [148420.349807] wlcore: down [148480.725376] wlcore: PHY firmware version: Rev 8.2.0.0.236 [148481.009504] wlcore: firmware booted (Rev 8.9.0.0.69) [148600.358891] wlcore: down [148660.746666] wlcore: PHY firmware version: Rev 8.2.0.0.236 [148661.029751] wlcore: firmware booted (Rev 8.9.0.0.69) [148763.319215] Unable to handle kernel NULL pointer dereference at virtual address 0000001c [148763.328535] pgd = c0004000 [148763.331453] [0000001c] *pgd=00000000 [148763.335504] Internal error: Oops: 17 [#1] ARM [148763.339969] Modules linked in: libcomposite bufferclass_ti(O) configfs omaplfb(O) wl18xx wlcore mac80211 rpmsg_pru virtio_rpmsg_bus cfg80211 pru_rproc pm33xx pvrsrvkm(O) uvcvideo videobuf2_vmalloc videobuf2_memops pruss videobuf2_core v4l2_common videodev media edt_ft5x06 wlcore_sdio omap_mailbox wkup_m3_ipc wkup_m3_rproc remoteproc virtio virtio_ring [148763.374262] CPU: 0 PID: 3388 Comm: kworker/u2:0 Tainted: G W O 4.1.18-g727f88a #1 [148763.382823] Hardware name: Generic AM43 (Flattened Device Tree) [148763.388981] Workqueue: wl12xx_wq wl1271_netstack_work [wlcore] [148763.394935] task: dddcdb80 ti: d2d50000 task.ti: d2d50000 [148763.400614] PC is at ieee80211_scan_rx+0x150/0x160 [mac80211] [148763.406484] LR is at 0x0 [148763.409120] pc : [<bf16b494>] lr : [<00000000>] psr: 20010013 sp : d2d51d70 ip : 0000d5e5 fp : d2d51e4c [148763.420824] r10: ddc02c00 r9 : 00000000 r8 : 00000000 [148763.426159] r7 : 00000000 r6 : d37af440 r5 : dca9a3a0 r4 : d37aeb80 [148763.432801] r3 : 00001500 r2 : 00000000 r1 : 0000015a r0 : 826de394 [148763.439446] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel [148763.446875] Control: 10c5387d Table: 9cb98059 DAC: 00000015 [148763.452734] Process kworker/u2:0 (pid: 3388, stack limit = 0xd2d50210) [148763.459378] Stack: (0xd2d51d70 to 0xd2d52000) [148763.463845] 1d60: d2d51e14 d2d51e14 d2d51dfc 00000080 [148763.472151] 1d80: ddd55488 dc201880 d37af440 ddd55460 00000000 ddd55460 d2d51e94 d2d51da8 [148763.480457] 1da0: bf196304 c0567394 00000000 00000080 dca9a3a0 000fac01 0a002000 c080fa50 [148763.488763] 1dc0: d37af4d7 d37af4f3 d37af515 d37af523 d2d51e1c d2d51de0 dddcdbb0 ddc4b170 [148763.497068] 1de0: 00005ed8 5e5164a9 00002ea4 5e383bb9 d2d51e14 d2d51e00 c0054f78 c0243564 [148763.505373] 1e00: c080f000 ddc4b170 c080f000 00000000 d2d51e5c d2d51e20 dddcdbb0 dd5b2d30 [148763.513678] 1e20: 00004f38 5e5f4821 dca9a3a0 00000050 d37aeb80 d37af440 00000000 00000000 [148763.521983] 1e40: d2d51eb4 d2d51e50 bf1824f0 bf16b350 c080f000 40010013 dcb1e980 00000000 [148763.530288] 1e60: d37aeb80 dca9a3a0 00000000 00000000 00000000 00000000 00000010 00000010 [148763.538592] 1e80: 00000000 00000000 ddf63300 dca9ab80 ffffe000 dca9ad30 bf1cb2a8 ddc02c00 [148763.546897] 1ea0: 00000000 ddc02c00 d2d51ed4 d2d51eb8 bf1cb2c4 bf181de0 dca9adf4 dca9ab80 [148763.555202] 1ec0: 00000000 ddc2a100 d2d51eec d2d51ed8 bf1cb354 bf1cb290 dca9adf4 d2e58680 [148763.563507] 1ee0: d2d51f2c d2d51ef0 c004788c bf1cb344 dddcdb80 d2d50000 ddc02c00 ddc02c10 [148763.571812] 1f00: d2d50000 ddc02c00 d2e58698 ddc02c10 d2d50000 00000088 d2e58680 ddc02c00 [148763.580117] 1f20: d2d51f64 d2d51f30 c0047bc0 c004779c ddc02d60 c080fa50 c0047b60 00000000 [148763.588421] 1f40: d2d9d440 d2e58680 c0047b60 00000000 00000000 00000000 d2d51fac d2d51f68 [148763.596726] 1f60: c004cc54 c0047b6c c00521a4 00000000 60010113 d2e58680 00000000 d2d51f7c [148763.605031] 1f80: d2d51f7c 00000000 d2d51f88 d2d51f88 d2d9d440 c004cb78 00000000 00000000 [148763.613335] 1fa0: 00000000 d2d51fb0 c000f688 c004cb84 00000000 00000000 00000000 00000000 [148763.621639] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [148763.629944] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 007d7673 00000000 [148763.638239] Backtrace: [148763.640960] [<bf16b344>] (ieee80211_scan_rx [mac80211]) from [<bf1824f0>] (ieee80211_rx+0x71c/0x7a4 [mac80211]) [148763.651179] r9:00000000 r8:00000000 r7:d37af440 r6:d37aeb80 r5:00000050 r4:dca9a3a0 [148763.659233] [<bf181dd4>] (ieee80211_rx [mac80211]) from [<bf1cb2c4>] (wl1271_flush_deferred_work+0x40/0xb4 [wlcore]) [148763.669888] r10:ddc02c00 r9:00000000 r8:ddc02c00 r7:bf1cb2a8 r6:dca9ad30 r5:ffffe000 [148763.677883] r4:dca9ab80 [148763.680592] [<bf1cb284>] (wl1271_flush_deferred_work [wlcore]) from [<bf1cb354>] (wl1271_netstack_work+0x1c/0x2c [wlcore]) [148763.691771] r7:ddc2a100 r6:00000000 r5:dca9ab80 r4:dca9adf4 [148763.697631] [<bf1cb338>] (wl1271_netstack_work [wlcore]) from [<c004788c>] (process_one_work+0xfc/0x3d0) [148763.707238] r5:d2e58680 r4:dca9adf4 [148763.710940] [<c0047790>] (process_one_work) from [<c0047bc0>] (worker_thread+0x60/0x528) [148763.719151] r10:ddc02c00 r9:d2e58680 r8:00000088 r7:d2d50000 r6:ddc02c10 r5:d2e58698 [148763.727142] r4:ddc02c00 [148763.729794] [<c0047b60>] (worker_thread) from [<c004cc54>] (kthread+0xdc/0xf4) [148763.737133] r10:00000000 r9:00000000 r8:00000000 r7:c0047b60 r6:d2e58680 r5:d2d9d440 [148763.745123] r4:00000000 [148763.747776] [<c004cb78>] (kthread) from [<c000f688>] (ret_from_fork+0x14/0x2c) [148763.755114] r7:00000000 r6:00000000 r5:c004cb78 r4:d2d9d440 [148763.760920] Code: e023300c e0222000 e1923003 0affffc3 (e597301c) [148763.768936] ---[ end trace 6e1debf7f75c18ff ]--- [148763.774060] Kernel panic - not syncing: Fatal exception in interrupt [148763.780537] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
The kernel is from the 02.00.02 SDK release, which I believe is the latest bugfix in the 02 SDK series.
What can be done to prevent this from happening? (Please let me know if additional information is necessary to find the cause of this crash.)
Chris
