Other Parts Discussed in Thread: WL1833
Tool/software: Linux
On one of our Linux platforms, we are using WL1833 module.
We are seeing _random_ kernel lockups on our platform that suggests that the IO_RW_EXTENDED (CMD53) request from the host is not getting completed.
This call chain locks wl->mutex and then waits from the above request to complete thereby causing a deadlock between related tasks in the kernel. This can be seen from the tasks that go into deep sleep state:
>>>>
….
[ 8178.341454] irq/107-wl18xx D c04e5fb4 0 1026 2 0x00000000
[ 8178.347841] [<c04e5fb4>] (__schedule) from [<c04e6468>] (schedule+0x4c/0xa4)
[ 8178.354903] [<c04e6468>] (schedule) from [<c04e9298>] (schedule_timeout+0x1cc/0x20c)
[ 8178.362659] [<c04e9298>] (schedule_timeout) from [<c04e7168>] (wait_for_common+0x9c/0x154)
[ 8178.370935] [<c04e7168>] (wait_for_common) from [<c04e7240>] (wait_for_completion+0x20/0x24)
[ 8178.379392] [<c04e7240>] (wait_for_completion) from [<c0389aac>] (mmc_wait_for_req_done+0x5c/0x14c)
[ 8178.388456] [<c0389aac>] (mmc_wait_for_req_done) from [<c0389bc8>] (mmc_wait_for_req+0x2c/0x30)
[ 8178.397173] [<c0389bc8>] (mmc_wait_for_req) from [<c0394c10>] (mmc_io_rw_extended+0x254/0x2fc)
[ 8178.405802] [<c0394c10>] (mmc_io_rw_extended) from [<c039622c>] (sdio_io_rw_ext_helper+0xc8/0x1ac)
[ 8178.414778] [<c039622c>] (sdio_io_rw_ext_helper) from [<c0396558>] (sdio_readsb+0x2c/0x34)
[ 8178.423068] [<c0396558>] (sdio_readsb) from [<bf849490>] (wl12xx_sdio_raw_read+0xbc/0x1a8 [wlcore_sdio])
[ 8178.432615] [<bf849490>] (wl12xx_sdio_raw_read [wlcore_sdio]) from [<bfa313a4>] (wlcore_rx+0x1dc/0x764 [wlcore])
[ 8178.442847] [<bfa313a4>] (wlcore_rx [wlcore]) from [<bfa248d4>] (wlcore_irq_locked+0xf0/0x344 [wlcore])
[ 8178.452286] [<bfa248d4>] (wlcore_irq_locked [wlcore]) from [<bfa25890>] (wlcore_irq+0xdc/0x19c [wlcore])
[ 8178.461801] [<bfa25890>] (wlcore_irq [wlcore]) from [<c006f57c>] (irq_thread_fn+0x2c/0x50)
[ 8178.470083] [<c006f57c>] (irq_thread_fn) from [<c006f8d8>] (irq_thread+0x13c/0x188)
[ 8178.477756] [<c006f8d8>] (irq_thread) from [<c0044c18>] (kthread+0xec/0x104)
[ 8178.484822] [<c0044c18>] (kthread) from [<c00107e8>] (ret_from_fork+0x14/0x2c)
….
[ 8177.737949] wpa_supplicant D c04e5fb4 0 282 1 0x00000000
[ 8177.744335] [<c04e5fb4>] (__schedule) from [<c04e6468>] (schedule+0x4c/0xa4)
[ 8177.751396] [<c04e6468>] (schedule) from [<c04e686c>] (schedule_preempt_disabled+0x20/0x2c)
[ 8177.759760] [<c04e686c>] (schedule_preempt_disabled) from [<c04e8054>] (__mutex_lock_slowpath+0xa0/0x180)
[ 8177.769339] [<c04e8054>] (__mutex_lock_slowpath) from [<c04e8190>] (mutex_lock+0x5c/0x60)
[ 8177.777564] [<c04e8190>] (mutex_lock) from [<bfa22560>] (wl1271_op_hw_scan+0x5c/0x104 [wlcore])
[ 8177.786357] [<bfa22560>] (wl1271_op_hw_scan [wlcore]) from [<bf98d37c>] (__ieee80211_start_scan+0x220/0x7b8 [mac80211])
[ 8177.797224] [<bf98d37c>] (__ieee80211_start_scan [mac80211]) from [<bf98e390>] (ieee80211_request_scan+0x34/0x4c [mac80211])
[ 8177.808534] [<bf98e390>] (ieee80211_request_scan [mac80211]) from [<bf9a130c>] (ieee80211_scan+0x94/0xd0 [mac80211])
[ 8177.819218] [<bf9a130c>] (ieee80211_scan [mac80211]) from [<bf91be8c>] (nl80211_trigger_scan+0x5f8/0x784 [cfg80211])
[ 8177.829815] [<bf91be8c>] (nl80211_trigger_scan [cfg80211]) from [<c040e4f4>] (genl_rcv_msg+0x26c/0x3ec)
[ 8177.839229] [<c040e4f4>] (genl_rcv_msg) from [<c040d70c>] (netlink_rcv_skb+0xb0/0xcc)
[ 8177.847074] [<c040d70c>] (netlink_rcv_skb) from [<c040e278>] (genl_rcv+0x34/0x44)
[ 8177.854572] [<c040e278>] (genl_rcv) from [<c040d02c>] (netlink_unicast+0x168/0x254)
[ 8177.862244] [<c040d02c>] (netlink_unicast) from [<c040d4ec>] (netlink_sendmsg+0x2fc/0x360)
[ 8177.870524] [<c040d4ec>] (netlink_sendmsg) from [<c03cd704>] (sock_sendmsg+0x24/0x34)
[ 8177.878368] [<c03cd704>] (sock_sendmsg) from [<c03ce1b8>] (___sys_sendmsg+0x1dc/0x1e4)
[ 8177.886298] [<c03ce1b8>] (___sys_sendmsg) from [<c03cef80>] (__sys_sendmsg+0x4c/0x78)
[ 8177.894142] [<c03cef80>] (__sys_sendmsg) from [<c03cefc4>] (SyS_sendmsg+0x18/0x1c)
[ 8177.901727] [<c03cefc4>] (SyS_sendmsg) from [<c0010740>] (ret_fast_syscall+0x0/0x3c)
….
<<<<
This is the sequence that I think is leading to the above hang:
- wilink driver holds wl->mutex and waits for IO_RW_EXTENDED request to complete.
- wpa_supplicant holds rtnl_mutex and waits for wl->mutex.
- Tasks waiting for rtnl_mutex get blocked.
We are using Kernel 4.1.13 and I can see the last commit in drivers/net/wireless/ti as (5c73cc4b6c83e88863a5de869cc5df3b913aef4a).
Is this a known issue?
-Thanks,
Sarang.