• State Resolved
  • Locked Locked
  • Replies 43 replies
  • Subscribers 41 subscribers
  • Views 2087 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

WL1837MOD: WoWlan wake randomly stops working

Trond Sandvik
Prodigy 30 points

Part Number: WL1837MOD
Other Parts Discussed in Thread: WL1837, WL1271

Hello support team,

we have released a product using a WL1387 connected to an iMX7D.

The first release does not include WoWlan; now we want to make a new release with WoWlan.

For that purpose we are upgrading to Linux 5.4.3 from Linux 4.9.67. The versions for the WL1387 firmware:

wlcore: PHY firmware version: Rev 8.2.0.0.245

wlcore: firmware booted (Rev 8.9.0.0.85).

We are doing a lot of testing on multiple systems doing thousands of state transitions to s2ram standby that should be waked over WLAN.

We are setting up for wake on any:
 # iw phy0 wowlan show
WoWLAN is enabled:
* wake up on special any trigger

We have an issue that is a show stopper for production quality - the wake over WLAN is not always working.

I have a setup with the target connected as an AP to a Mac, using Wireshark I can see there will be TCP retransmissions without ACK from target in 60 seconds when our application is giving up.

Please let me know how to proceed to solve this issue.

Regards,

Trond

  • 0
    TI__Mastermind 36360 points

    Hello Trond,

    Could you please update to the latest firmware (Rev 8.9.0.0.88) and replicate the issue? There was a fix for unstable connections.  

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hello Sabeeh,

    Where do I find this revision?

    Thanks,

    Trond

  • 0 in reply to Trond Sandvik
    Prodigy 30 points

    Hello Sabeeh,

    I see the same issue with the latest firmware (Rev 8.9.0.0.88) .

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Hi Trond,

    Are you able to wake up the device from any method other than WoWLAN? For example, sending a command over terminal. If not, then you may be missing something in your DTS. 
    I also assume everything else you have set correctly in DTS, such as WLAN_IRQ. 

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hi Sabeeh,

    Yes i can wake up the system by our power button driver.

    This issue is happening randomly; sometimes after just a few suspends, other times after several hundreds suspends.

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Hi Trond,

    Have you limited the power of the WL18? Could there possibly be any hardware restrictions that wouldn't allow the device to ACK?

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hi Sabeeh,

    We should have plenty of power supplying the WL18. I will connect power supply and SDIO clock line to a scope tomorrow for investigation.

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Hi Trond, 

    Could you please share your DTS for wlcore? if the parent is also a GPIO, could you share that too? 

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hi Sabeeh,

    I measured the power supply today. It is perfectly stable, also when seeing the issue.

    Another thing I noticed is that the IRQ line is asynchronous relative the SDIO clock; that is running at 50 MHz. 

    Here is the DTS entry for wlcore.

    /* SDIO interface for Wi-Fi TI WL18XX module */
    &usdhc1 {
    pinctrl-names = "default", "state_100mhz", "state_200mhz";
    pinctrl-0 = <&pinctrl_usdhc1>;
    pinctrl-1 = <&pinctrl_usdhc1_100mhz>;
    pinctrl-2 = <&pinctrl_usdhc1_200mhz>;
    wakeup-source;
    keep-power-in-suspend;
    cd-post;
    pm-ignore-notify;
    wifi-host;

    no-1-8-v;
    bus-width = <4>;
    non-removable;
    status = "okay";
    vmmc-supply = <&w18xx_wifi_reg>;

    cap-power-off-card;
    #address-cells = <1>;
    #size-cells = <0>;
    wlcore: wlcore@0 {
    compatible = "ti,wl1837";
    reg = <2>;

    interrupt-parent = <&gpio6>;
    interrupts = <12 IRQ_TYPE_EDGE_RISING>;
    };
    };

    The "pinctrl-x" entries are selecting the SDIO function and  IO parameters for the pin; drive strength, slew rate, hysteresis, pull up/down.

    I do not understand what you mean with "if the parent is also a GPIO" is is "interrupt-parent"?. The interrupt line is at GPIO6 bit 12.

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    By parent, I was referring to the definition for, in your case, gpio6. Making sure that everything is configured correctly. 

    Do you happen to have the wireshark log? It seems that the device is getting disconnected from the AP. 

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    On the Mac I see that the device is not disconnected - actually the device is the AP in this setup with the Mac. I see the WiFi is connected all the time on the Mac.
    I have a Power MCU that will wake the system on a WDOG that times out on a timer set higher (34 seconds) than the application WDOG that is running at 30 seconds. This resolves the hanged issue and the application continues the communication. If I disable the WDOG in the MCU, the system will hang with no TCP ACK from the device.

    To me this points to an issue with the WLAN firmware since the system is suspended and the WLAN firmware should wake caused by the incoming TCP transmission.

    Trond

  • 0 in reply to Trond Sandvik
    Prodigy 30 points

    I have captured the wlan_irq line, sdio clock and one data bit just before entering the suspend state (right cursor in logic analyzer waveform).

    I have noticed that the wlan_irq line is going high 33ms before entering the suspend state, and staying high during the hanged situation. This is observed twice now.

    wlan-suspend-entry-that-will-hang.pdf

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Thanks for this post Trond. I am using a beaglebone black as STA and have experienced the hanging. I will report this issue internally. 

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Nice that you could reproduce.

    I saved a logic analyzer waveform showing the hang resolution by a button press wake.

    wlan-suspend-hang-resolved-by-button-wake.pdf

    This was after hanging through 6 hours over the night - the communication resumed by the button press wake.

    I will be available for testing / debugging if that can speed up finding a fix for this issue.

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Hi Trond,

    Could you try adjusting the irq_timeout parameter in the wlconf?

    Default value is posted below, shortening this value should cause the timeout to occur earlier: 

    core.rx.irq_timeout = 0x0258


  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hi Sabeeh,

    I am not sure about which timeout the irq_timeout parameter is controlling with respect to the logic analyzer waveform in the hanging situation. I have several different occurrences of  this hanging issue  with the wlan_irg stuck high until wake from an alternate source (button). All are indicating a race condition between incoming network packets and entering the suspend state; the wlan_irq line is activated just some milliseconds before the suspend state is entered and staying stuck afterwards. 
    I have also tried different IRQ_TYPE for the wlan_irq; IRQ_TYPE_EDGE_RISING, IRQ_TYPE_LEVEL_HIGH and IRQ_TYPE_LEVEL_LOW. I will get the issue with all these IRQ_TYPE settings.

    Did you try to adjust the irq_timeout parameter such as the hanging you experienced went away?

    Will there be more TI internal experts working on making a fix for this issue in the WL1837 firmware and/or the wlcore kernel driver? 

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Hi Trond,

    I am going through the same debug steps as you, and found that the issue persists. 

    I am also getting support internally and will try my best to get an update to you soon. 

  • 0 in reply to Sabeeh Khan1
    TI__Mastermind 36360 points

    Hi Trond,

    I wanted to inform you that this problem is still being worked on. 

  • 0 in reply to Sabeeh Khan1
    TI__Mastermind 36360 points

    Hi Trond,

    Is using WoWLAN as an AP a hard requirement for you? It's not recommended as it may slow down your connected stations. 

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hi Sabeeh,

    We are using WoWlan both as AP and STA dependent of the connected device; if it is an IOS or Android device.

    We are seeing the reported hang issue in both roles.

    Trond

  • 0 in reply to Trond Sandvik
    Prodigy 30 points

    Hi Sabeeh,

    I will also like to add that the issue will show up both with and without WARN_ON messages from the wlcore driver.

    In the case there is a WARN_ON message I have managed to make a workaround with a callback to our power button kernel driver; that will make a wake as soon as suspend is entered. Here is my modification in wlcore/main.c:

    #ifdef CONFIG_GEHC_MELODY
    extern void powermcu_wake_on_suspend(void);
    #endif
    void wl12xx_queue_recovery_work(struct wl1271 *wl)
    {
      /* Avoid a recursive recovery */
      if (wl->state == WLCORE_STATE_ON) {
    #ifdef CONFIG_GEHC_MELODY
        if (!test_bit(WL1271_FLAG_INTENDED_FW_RECOVERY,&wl->flags)) {
          powermcu_wake_on_suspend();
        }
    #endif
        WARN_ON(!test_bit(WL1271_FLAG_INTENDED_FW_RECOVERY,
                           &wl->flags));

        wl->state = WLCORE_STATE_RESTARTING;
        set_bit(WL1271_FLAG_RECOVERY_IN_PROGRESS, &wl->flags);
        ieee80211_queue_work(wl->hw, &wl->recovery_work);
      }
    }

    So what is left is the cases where there are no WARN_ON messages in the wlcore code. If we can find them I believe my workaround with callback into our power button driver will work for those cases as well.

    Trond 

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Hi Trond, 

    For your application, is there any method of either reducing the number of IRQ or reducing the amount of suspends? It seems that if you know when some communication has completed, then you can go into suspend without expecting a race condition. 

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hi Sabeeh,

    I see no way to avoid race condition, especially in AP mode when multiple applications in the connected device can do network activity towards the AP.

    It is better to:

    1. Detect the race condition
    2. Make a workaround for it

    By the way - I mentioned that we have a workaround with a callback function to our power MCU button kernel driver when the race condition is detected by a WARN_ON message in the wlcore code. Until now I have been testing without using the BT part of the WL1387 module. We just got the BT part up running. I detected today that when the BT part is active, the above mentioned workaround does not always work. The connection in the connected device will fall down and I am seeing this endless logging from the wlcore driver:

    [ 249.604377] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 249.619239] wlcore: ERROR reg domain conf error
    [ 250.034388] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 250.049194] wlcore: ERROR reg domain conf error
    [ 250.464406] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 250.479050] wlcore: ERROR reg domain conf error
    [ 250.483733] wlcore: ERROR firmware boot failed despite 3 retries
    [ 256.374402] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 256.389136] wlcore: ERROR reg domain conf error
    [ 256.804196] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 256.819086] wlcore: ERROR reg domain conf error
    [ 257.233969] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 257.248582] wlcore: ERROR reg domain conf error
    [ 257.253303] wlcore: ERROR firmware boot failed despite 3 retries
    .
    .
    .

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Hi Trond,

    I have noticed that the PM module seems to be blocking the driver. Note these logs in the image attached. My secondary wakeup source is UART. The driver kicks in and handles the interrupt. Can you confirm something similar on your end? 

    You can enable DEBUG_IRQ via:

    $ echo 8 > /proc/sys/kernel/printk

    $ echo 1 > /sys/module/wlcore/parameters/debug_level 

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hi Sabeeh,

    I do not understand what is wrong with the kernel logging you are showing - it looks like it should be from my perspective.

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Hi Trond,

    I have consulted with the Linux SDK team and the conclusion is that this is a linux kernel issue, not an issue with the Wilink driver. When linux is transitioning to suspend, wakeup events are not monitored. Therefore, we have to either reduce the time it takes to enter suspend or reduce the number of total WL_IRQs, but the issue will not be completely resolved. 

     Here is a command to shrink the time it takes for linux to suspend:

    $ cat /sys/power/wakeup_count > /sys/power/wakeup_count
$ echo mem > /sys/power/state

    As for the IRQs, you could use the wowlan filter set for a specific IP/port or type of traffic. You can read more about this in the Advanced demo guide: 

    https://www.ti.com/lit/swru576  

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hi Sabeeh,

    What you are saying is that there are no solution to the problem using Linux, hence the WoWLAN is not working for the WL1837 module using Linux.

    Why are you then promoting this functionality.

    About setting filter for WoWLAN - I have tried that and it is not functioning at all in any of AP or STA roles - hence I am not using it (wake on any).

    Luckily I think I have a fix for the wake issue using our own button kernel driver - when receiving suspend callback from PM just look at the IRQ line from the WL1837 WLAN module; if active I will send a command to a power MCU that will be active during suspend and tell it to wake  some milliseconds after suspend.

    Why cannot TI do the same with your firmware in the WL1837?

    As mentioned earlier: I am now facing an even more serious error after enabling the BT part of the WL1837:

    The connection in the connected device will fall down and I am seeing this endless logging from the wlcore driver:

    [ 249.604377] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 249.619239] wlcore: ERROR reg domain conf error
    [ 250.034388] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 250.049194] wlcore: ERROR reg domain conf error
    [ 250.464406] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 250.479050] wlcore: ERROR reg domain conf error
    [ 250.483733] wlcore: ERROR firmware boot failed despite 3 retries
    [ 256.374402] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 256.389136] wlcore: ERROR reg domain conf error
    [ 256.804196] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 256.819086] wlcore: ERROR reg domain conf error
    [ 257.233969] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 257.248582] wlcore: ERROR reg domain conf error
    [ 257.253303] wlcore: ERROR firmware boot failed despite 3 retries
    .
    .
    .

    How to resolve this? As of now I have to reboot the system to get the WLAN module back on its feet.

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Hi Trond, 

    Yes I completely understand your frustration. We are actively investigating a solution on the firmware side. As for the filter, could you please give me an example of what filter you are setting and how it is not working? 

    And for the BT, is there a certain point that it fails at? Have you verified that the BT_EN is set up correctly from the DTS?

  • 0 in reply to Sabeeh Khan1
    Intellectual 615 points

    Hi all,

    I must say I am also kind of frustrated as I reported this issue more than one year ago without a solution.

    Its clearly an issue of firmware as it should pull down IRQ line when it goes to wowlan state or wilink driver to clean up IRQ line when suspend callback is called.

    As Trond wrote, if this feature is promoted as functional it should be functional. Right now its really blocking our solution.

    Petr

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hi Sabeeh,

    We are just not using any packet filter in the WLAN module; found out it is better to do the filtering in our user application, not relying on the WLAN firmware.

    I think it is a coincident that WLAN went down when testing with BT enabled. It is very difficult to reproduce - I just had an incident were I noticed the WLAN_EN pin is low. 

    Trond 

  • 0 in reply to Trond Sandvik
    Prodigy 30 points

    Hi Sabeeh,

    I have done some more testing today and now I get a lot of these disconnect errors. It starts in a resume and the only way to resume is to reboot the system:

    [ 190.011650] *** powermcu_send_command: will send command 0x13 to MCU ***
    [ 190.022279] mmc0: error -110 during resume (card was removed?)
    [ 190.024987] fec_nm 30bf0000.ethernet rgmii: Link is Up - 1Gbps/Full - flow control off
    [ 190.036170] ------------[ cut here ]------------
    [ 190.040822] WARNING: CPU: 0 PID: 999 at /mnt/yocto-melody/build-imx7-melody-dev-rev05-linux-rev5.4/tmp/work-shared/gm400-rev05-mx7/kernel-source/drivers/net/wireless/ti/wlcore/sdio.c:131 wl12xx_sdio_raw_write+0xec/0x1c8 [wlcore_sdio]
    [ 190.061564] Modules linked in: usb_f_mass_storage usb_f_acm u_serial libcomposite configfs wl18xx wlcore mac80211 cfg80211 fec_nm(O) netmap(O) wlcore_sdio
    [ 190.075468] CPU: 0 PID: 999 Comm: kworker/u4:5 Tainted: G W O 5.4.3-fslc+g397e75b #1
    [ 190.084344] Hardware name: Freescale i.MX7 Dual (Device Tree)
    [ 190.090108] Workqueue: events_unbound async_run_entry_fn
    [ 190.095446] [<8010fef4>] (unwind_backtrace) from [<8010c6b8>] (show_stack+0x10/0x14)
    [ 190.103204] [<8010c6b8>] (show_stack) from [<808db7ec>] (dump_stack+0x94/0xa8)
    [ 190.110442] [<808db7ec>] (dump_stack) from [<8012fee4>] (__warn+0xc0/0xd8)
    [ 190.117327] [<8012fee4>] (__warn) from [<8012ffac>] (warn_slowpath_fmt+0xb0/0xb8)
    [ 190.124827] [<8012ffac>] (warn_slowpath_fmt) from [<7f00022c>] (wl12xx_sdio_raw_write+0xec/0x1c8 [wlcore_sdio])
    [ 190.135113] [<7f00022c>] (wl12xx_sdio_raw_write [wlcore_sdio]) from [<7f137364>] (wlcore_runtime_resume+0xe0/0x22c [wlcore])
    [ 190.146433] [<7f137364>] (wlcore_runtime_resume [wlcore]) from [<805606ec>] (__rpm_callback+0xb4/0x11c)
    [ 190.155838] [<805606ec>] (__rpm_callback) from [<805607a8>] (rpm_callback+0x54/0x80)
    [ 190.163590] [<805607a8>] (rpm_callback) from [<805602ec>] (rpm_resume+0x380/0x5ac)
    [ 190.171170] [<805602ec>] (rpm_resume) from [<80560564>] (__pm_runtime_resume+0x4c/0x64)
    [ 190.179271] [<80560564>] (__pm_runtime_resume) from [<7f13eb14>] (wl1271_op_resume+0xbc/0x344 [wlcore])
    [ 190.188988] [<7f13eb14>] (wl1271_op_resume [wlcore]) from [<7f100338>] (ieee80211_reconfig+0x5c/0xe10 [mac80211])
    [ 190.199592] [<7f100338>] (ieee80211_reconfig [mac80211]) from [<7f065efc>] (wiphy_resume+0xb4/0xe0 [cfg80211])
    [ 190.209721] [<7f065efc>] (wiphy_resume [cfg80211]) from [<805621b0>] (dpm_run_callback+0x30/0x60)
    [ 190.218606] [<805621b0>] (dpm_run_callback) from [<805626a4>] (device_resume+0xcc/0x25c)
    [ 190.226707] [<805626a4>] (device_resume) from [<80562850>] (async_resume+0x1c/0x44)
    [ 190.234376] [<80562850>] (async_resume) from [<801511e8>] (async_run_entry_fn+0x48/0x1b4)
    [ 190.242568] [<801511e8>] (async_run_entry_fn) from [<801486d8>] (process_one_work+0x204/0x45c)
    [ 190.251190] [<801486d8>] (process_one_work) from [<80148974>] (worker_thread+0x44/0x598)
    [ 190.259290] [<80148974>] (worker_thread) from [<8014eb88>] (kthread+0x14c/0x150)
    [ 190.266697] [<8014eb88>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
    [ 190.273923] Exception stack(0x8cb5bfb0 to 0x8cb5bff8)
    [ 190.278983] bfa0: 00000000 00000000 00000000 00000000
    [ 190.287169] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [ 190.295353] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
    [ 190.302215] ---[ end trace 15fbc21f7450083b ]---
    [ 190.306892] wl1271_sdio mmc0:0001:2: sdio write failed (-110)
    [ 190.312692] wlcore: WARNING Enable for recovery failed
    [ 190.318626] OOM killer enabled.
    [ 190.321776] Restarting tasks ...
    [ 190.321891] wlcore: down
    [ 190.325288] done.
    [ 190.331269] PM: suspend exit
    [ 190.332157] wlcore: down
    [ 190.337794] wlcore: down
    [ 190.346146] ieee80211 phy0: Hardware restart was requestedroot
     [ 190.871006] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 190.885523] wlcore: ERROR reg domain conf error
    [ 191.300525] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 191.314957] wlcore: ERROR reg domain conf error
    [ 191.736121] PM: suspend entry (deep)
    [ 191.743221] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 191.757509] wlcore: ERROR reg domain conf error
    [ 191.762130] wlcore: ERROR firmware boot failed despite 3 retries
    [ 191.768189] ------------[ cut here ]------------
    [ 191.773239] WARNING: CPU: 0 PID: 20 at /mnt/yocto-melody/build-imx7-melody-dev-rev05-linux-rev5.4/tmp/work-shared/gm400-rev05-mx7/kernel-source/net/mac80211/util.c:2235 ieee80211_reconfig+0x1a8/0xe10 [mac80211]
    [ 191.791961] Modules linked in: usb_f_mass_storage usb_f_acm u_serial libcomposite configfs wl18xx wlcore mac80211 cfg80211 fec_nm(O) netmap(O) wlcore_sdio
    [ 191.805997] CPU: 0 PID: 20 Comm: kworker/0:1 Tainted: G W O 5.4.3-fslc+g397e75b #1
    [ 191.814701] Hardware name: Freescale i.MX7 Dual (Device Tree)
    [ 191.820588] Workqueue: events_freezable ieee80211_restart_work [mac80211]
    [ 191.827409] [<8010fef4>] (unwind_backtrace) from [<8010c6b8>] (show_stack+0x10/0x14)
    [ 191.835167] [<8010c6b8>] (show_stack) from [<808db7ec>] (dump_stack+0x94/0xa8)
    [ 191.842404] [<808db7ec>] (dump_stack) from [<8012fee4>] (__warn+0xc0/0xd8)
    [ 191.849290] [<8012fee4>] (__warn) from [<8012ffac>] (warn_slowpath_fmt+0xb0/0xb8)
    [ 191.856899] [<8012ffac>] (warn_slowpath_fmt) from [<7f100484>] (ieee80211_reconfig+0x1a8/0xe10 [mac80211])
    [ 191.866789] [<7f100484>] (ieee80211_reconfig [mac80211]) from [<7f0cc8b8>] (ieee80211_restart_work+0xac/0xe4 [mac80211])
    [ 191.877785] [<7f0cc8b8>] (ieee80211_restart_work [mac80211]) from [<801486d8>] (process_one_work+0x204/0x45c)
    [ 191.887712] [<801486d8>] (process_one_work) from [<80148974>] (worker_thread+0x44/0x598)
    [ 191.895813] [<80148974>] (worker_thread) from [<8014eb88>] (kthread+0x14c/0x150)
    [ 191.903219] [<8014eb88>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
    [ 191.910446] Exception stack(0x8c127fb0 to 0x8c127ff8)
    [ 191.915505] 7fa0: 00000000 00000000 00000000 00000000
    [ 191.923692] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [ 191.931876] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
    [ 191.939542] ---[ end trace 15fbc21f7450083c ]---
    [ 191.944414] ------------[ cut here ]------------
    [ 191.949287] WARNING: CPU: 0 PID: 20 at /mnt/yocto-melody/build-imx7-melody-dev-rev05-linux-rev5.4/tmp/work-shared/gm400-rev05-mx7/kernel-source/net/mac80211/driver-ops.h:19 drv_remove_interface+0x6c/0x74 [mac80211]
    [ 191.968410] p2p-dev-wlan0: Failed check-sdata-in-driver check, flags: 0x0
    [ 191.975579] Modules linked in: usb_f_mass_storage usb_f_acm u_serial libcomposite configfs wl18xx wlcore mac80211 cfg80211 fec_nm(O) netmap(O) wlcore_sdio
    [ 191.989657] CPU: 0 PID: 20 Comm: kworker/0:1 Tainted: G W O 5.4.3-fslc+g397e75b #1
    [ 191.998359] Hardware name: Freescale i.MX7 Dual (Device Tree)
    [ 192.004250] Workqueue: events_freezable ieee80211_restart_work [mac80211]
    [ 192.011064] [<8010fef4>] (unwind_backtrace) from [<8010c6b8>] (show_stack+0x10/0x14)
    [ 192.018821] [<8010c6b8>] (show_stack) from [<808db7ec>] (dump_stack+0x94/0xa8)
    [ 192.026057] [<808db7ec>] (dump_stack) from [<8012fee4>] (__warn+0xc0/0xd8)
    [ 192.032943] [<8012fee4>] (__warn) from [<8012ff70>] (warn_slowpath_fmt+0x74/0xb8)
    [ 192.040550] [<8012ff70>] (warn_slowpath_fmt) from [<7f0cf968>] (drv_remove_interface+0x6c/0x74 [mac80211])
    [ 192.050439] [<7f0cf968>] (drv_remove_interface [mac80211]) from [<7f0e2ca4>] (ieee80211_do_stop+0x618/0x884 [mac80211])
    [ 192.061561] [<7f0e2ca4>] (ieee80211_do_stop [mac80211]) from [<7f064e24>] (cfg80211_stop_p2p_device+0x58/0x138 [cfg80211])
    [ 192.072844] [<7f064e24>] (cfg80211_stop_p2p_device [cfg80211]) from [<7f065078>] (cfg80211_shutdown_all_interfaces+0x98/0xe0 [cfg80211])
    [ 192.085351] [<7f065078>] (cfg80211_shutdown_all_interfaces [cfg80211]) from [<7f100740>] (ieee80211_reconfig+0x464/0xe10 [mac80211])
    [ 192.097499] [<7f100740>] (ieee80211_reconfig [mac80211]) from [<7f0cc8b8>] (ieee80211_restart_work+0xac/0xe4 [mac80211])
    [ 192.108498] [<7f0cc8b8>] (ieee80211_restart_work [mac80211]) from [<801486d8>] (process_one_work+0x204/0x45c)
    [ 192.118425] [<801486d8>] (process_one_work) from [<80148974>] (worker_thread+0x44/0x598)
    [ 192.126526] [<80148974>] (worker_thread) from [<8014eb88>] (kthread+0x14c/0x150)
    [ 192.133932] [<8014eb88>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
    [ 192.141157] Exception stack(0x8c127fb0 to 0x8c127ff8)
    [ 192.146216] 7fa0: 00000000 00000000 00000000 00000000
    [ 192.154402] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [ 192.162588] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
    [ 192.170346] ---[ end trace 15fbc21f7450083d ]---
    [ 192.175325] ------------[ cut here ]------------
    [ 192.180206] WARNING: CPU: 0 PID: 20 at /mnt/yocto-melody/build-imx7-melody-dev-rev05-linux-rev5.4/tmp/work-shared/gm400-rev05-mx7/kernel-source/net/mac80211/driver-ops.h:19 drv_remove_interface+0x6c/0x74 [mac80211]
    [ 192.199310] wlan0: Failed check-sdata-in-driver check, flags: 0x0
    [ 192.205559] Modules linked in: usb_f_mass_storage usb_f_acm u_serial libcomposite configfs wl18xx wlcore mac80211 cfg80211 fec_nm(O) netmap(O) wlcore_sdio
    [ 192.219480] CPU: 0 PID: 20 Comm: kworker/0:1 Tainted: G W O 5.4.3-fslc+g397e75b #1
    [ 192.228184] Hardware name: Freescale i.MX7 Dual (Device Tree)
    [ 192.234078] Workqueue: events_freezable ieee80211_restart_work [mac80211]
    [ 192.240895] [<8010fef4>] (unwind_backtrace) from [<8010c6b8>] (show_stack+0x10/0x14)
    [ 192.248653] [<8010c6b8>] (show_stack) from [<808db7ec>] (dump_stack+0x94/0xa8)
    [ 192.255889] [<808db7ec>] (dump_stack) from [<8012fee4>] (__warn+0xc0/0xd8)
    [ 192.262775] [<8012fee4>] (__warn) from [<8012ff70>] (warn_slowpath_fmt+0x74/0xb8)
    [ 192.270381] [<8012ff70>] (warn_slowpath_fmt) from [<7f0cf968>] (drv_remove_interface+0x6c/0x74 [mac80211])
    [ 192.280273] [<7f0cf968>] (drv_remove_interface [mac80211]) from [<7f0e2ca4>] (ieee80211_do_stop+0x618/0x884 [mac80211])
    [ 192.291288] [<7f0e2ca4>] (ieee80211_do_stop [mac80211]) from [<7f0e2f20>] (ieee80211_stop+0x10/0x18 [mac80211])
    [ 192.301501] [<7f0e2f20>] (ieee80211_stop [mac80211]) from [<80731168>] (__dev_close_many+0xa0/0x118)
    [ 192.310646] [<80731168>] (__dev_close_many) from [<8073125c>] (dev_close_many+0x7c/0x110)
    [ 192.318836] [<8073125c>] (dev_close_many) from [<807330b8>] (dev_close+0x48/0x70)
    [ 192.326505] [<807330b8>] (dev_close) from [<7f065054>] (cfg80211_shutdown_all_interfaces+0x74/0xe0 [cfg80211])
    [ 192.336746] [<7f065054>] (cfg80211_shutdown_all_interfaces [cfg80211]) from [<7f100740>] (ieee80211_reconfig+0x464/0xe10 [mac80211])
    [ 192.348892] [<7f100740>] (ieee80211_reconfig [mac80211]) from [<7f0cc8b8>] (ieee80211_restart_work+0xac/0xe4 [mac80211])
    [ 192.359887] [<7f0cc8b8>] (ieee80211_restart_work [mac80211]) from [<801486d8>] (process_one_work+0x204/0x45c)
    [ 192.369814] [<801486d8>] (process_one_work) from [<80148974>] (worker_thread+0x44/0x598)
    [ 192.377914] [<80148974>] (worker_thread) from [<8014eb88>] (kthread+0x14c/0x150)
    [ 192.385321] [<8014eb88>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
    [ 192.392547] Exception stack(0x8c127fb0 to 0x8c127ff8)
    [ 192.397606] 7fa0: 00000000 00000000 00000000 00000000
    [ 192.405793] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [ 192.413978] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
    [ 192.420734] ---[ end trace 15fbc21f7450083e ]---
    [ 192.848213] Filesystems sync: 1.108 seconds
    [ 192.852928] Freezing user space processes ... (elapsed 0.001 seconds) done.
    [ 192.861459] OOM killer disabled.
    [ 192.864754] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
    [ 192.874422] fec_nm 30bf0000.ethernet rgmii: Link is Down
    [ 192.967199] *** powermcu_send_command: will send command 0x02 to MCU ***
    [ 192.977477] Disabling non-boot CPUs ...
    [ 192.982757] Enabling non-boot CPUs ...
    [ 192.987386] CPU1 is up
    [ 192.991627] *** powermcu_send_command: will send command 0x13 to MCU ***
    [ 193.004371] fec_nm 30bf0000.ethernet rgmii: Link is Up - 1Gbps/Full - flow control off
    [ 193.050054] OOM killer enabled.
    [ 193.053242] Restarting tasks ... done.
    [ 193.059242] PM: suspend exit

    This is now the blocking factor for us - please advice how to resolve this.

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Hi Trond,

    Are you using the bluetopia stack to control the BT? 

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hi Sabeeh,

    No we are not using the bluetopia stack to control the BT.

    Actually I just found out that the blocking issue above also is happening when BT module is disabled.

    Another observation is that the workaround I have made will only work for incoming packets. We have this function that the user can wake from pushing the button on the device - this packet will be delayed until a new packet is received. That will take the WDOG time that is 30s - of course not acceptable.

    Trond

  • 0 in reply to Trond Sandvik
    Prodigy 30 points

    Hi Sabeeh,

    I did some more testing and was surprised how easy it was to breakdown the WLAN module when WoWlan is enabled.

    I just ensured the WLAN module was idle, meaning no net connections were established.
    I entered s2ram manually; by: #echo mem > /sys/power/state.
    Then I did a wake with the power button.

    This was enough to get the WLAN module down, only way to resolve this is to reboot the system.

    Here is the logic analyzer output (the kernel logging is as before pasted):

    wlan-idle-manual-s2ram-breakdown.pdf

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Hi Trond,

    I would ask you to please revisit the two commands I posted earlier. Using these two commands every time to put the machine into suspend has been quite successful in my tests. I have been running a loop of these two commands, while using another machine to ping the wilink and trigger wakeup. At the time of this writing, I have been running without issue for a couple of hours. 

    $ cat /sys/power/wakeup_count > /sys/power/wakeup_count
$ echo mem > /sys/power/state

    Using both of these commands when putting the machine into suspend "triggers a mechanism whereby any subsequent wakeup events will cause an attempted suspend operation to abort." [1]. The issue here throughout does not seem to be due to Wilink firmware nor the driver. It is simply that wakeup events were missed when linux was transitioning to suspend. What reading from and writing back to the wakeup_count allows the suspend operation to abort and handle the wakeup event. 

    Read more about the wakeup_count here:

    [1] https://lwn.net/Articles/393314/ 

    [2] https://titanwolf.org/Network/Articles/Article?AID=bc3c54c1-929b-4409-904e-af2773cec9d5 

    [3] https://linux-kernel.vger.kernel.narkive.com/LhQv7XEm/rfc-patch-pm-avoid-losing-wakeup-events-during-suspend 

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hi Sabeeh,

    I tried your suggestion - I had to use a file to store the count otherwise it did not work:

    # cat /sys/power/wakeup_count > /sys/power/wakeup_count
    cat: /sys/power/wakeup_count: input file is output file

    But it did not do any difference - the WLAN module is still easily crashing (reboot needed for resolution):

    root@gm400-rev05-mx7:~# cat /sys/power/wakeup_count > wakeup_count ; cat wakeup_count > /sys/power/wakeup_count
    root@gm400-rev05-mx7:~# /bin/echo mem > /sys/power/state
    [ 229.461790] PM: suspend entry (deep)
    [ 229.479502] Filesystems sync: 0.014 seconds
    [ 229.492862] Freezing user space processes ... (elapsed 0.001 seconds) done.
    [ 229.501213] OOM killer disabled.
    [ 229.504444] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
    [ 229.526454] ------------[ cut here ]------------
    [ 229.531317] WARNING: CPU: 1 PID: 411 at /mnt/yocto-melody/build-imx7-melody-dev-rev05-linux-rev5.4/tmp/work-shared/gm400-rev05-mx7/kernel-source/drivers/net/wireless/ti/wlcore/main.c:804 wl12xx_queue_recovery_work.part.14+0x70/0x74 [wlcore]
    [ 229.552638] Modules linked in: usb_f_mass_storage usb_f_acm u_serial libcomposite configfs wl18xx wlcore mac80211 cfg80211 fec_nm(O) netmap(O) wlcore_sdio
    [ 229.557404] fec_nm 30bf0000.ethernet rgmii: Link is Down
    [ 229.566516] CPU: 1 PID: 411 Comm: irq/207-wl18xx Tainted: G O 5.4.3-fslc+gdbe2bb1 #1
    [ 229.580849] Hardware name: Freescale i.MX7 Dual (Device Tree)
    [ 229.586625] [<8010fef4>] (unwind_backtrace) from [<8010c6b8>] (show_stack+0x10/0x14)
    [ 229.594384] [<8010c6b8>] (show_stack) from [<808db82c>] (dump_stack+0x94/0xa8)
    [ 229.601619] [<808db82c>] (dump_stack) from [<8012ff28>] (__warn+0xc0/0xd8)
    [ 229.608505] [<8012ff28>] (__warn) from [<8012fff0>] (warn_slowpath_fmt+0xb0/0xb8)
    [ 229.616108] [<8012fff0>] (warn_slowpath_fmt) from [<7f14ccb8>] (wl12xx_queue_recovery_work.part.14+0x70/0x74 [wlcore])
    [ 229.626990] [<7f14ccb8>] (wl12xx_queue_recovery_work.part.14 [wlcore]) from [<7f1523d0>] (wlcore_irq+0x160/0x194 [wlcore])
    [ 229.638133] [<7f1523d0>] (wlcore_irq [wlcore]) from [<801786a4>] (irq_thread_fn+0x1c/0x78)
    [ 229.646410] [<801786a4>] (irq_thread_fn) from [<801789c4>] (irq_thread+0x140/0x21c)
    [ 229.654077] [<801789c4>] (irq_thread) from [<8014ebc8>] (kthread+0x14c/0x150)
    [ 229.661222] [<8014ebc8>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
    [ 229.668447] Exception stack(0x8cc53fb0 to 0x8cc53ff8)
    [ 229.673506] 3fa0: 00000000 00000000 00000000 00000000
    [ 229.681692] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [ 229.689876] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
    [ 229.696552] ---[ end trace a7084b3108e9a7aa ]---
    [ 229.701207] queueing ieee80211 work while going to suspend
    [ 229.703115] *** powermcu_send_command: will send command 0x21 to MCU ***
    [ 229.714919] *** powermcu_send_command: will send command 0x02 to MCU ***
    [ 229.725978] Disabling non-boot CPUs ...
    [ 229.731555] Enabling non-boot CPUs ...
    [ 229.736167] CPU1 is up
    [ 229.740905] *** powermcu_send_command: will send command 0x13 to MCU ***
    [ 229.750735] Bluetooth: hci0: Frame reassembly failed (-84)
    [ 229.757917] wlcore: WARNING queuing forgotten recovery on resume
    [ 229.758863] fec_nm 30bf0000.ethernet rgmii: Link is Up - 1Gbps/Full - flow control off
    [ 229.772006] wlcore: Hardware recovery in progress. FW ver: Rev 8.9.0.0.88
    [ 229.772824] OOM killer enabled.
    [ 229.782095] Restarting tasks ... done.
    [ 229.789181] wlcore: pc: 0x0, hint_sts: 0x00000020 count: 1
    [ 229.789184] PM: suspend exit
    root@gm400-rev05-mx7:~# [ 229.798502] wlcore: down
    [ 229.802502] ieee80211 phy0: Hardware restart was requested
    [ 230.176502] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 230.291500] wlcore: firmware booted (Rev 8.9.0.0.88)
    root@gm400-rev05-mx7:~# cat /sys/power/wakeup_count > wakeup_count ; cat wakeup_count > /sys/power/wakeup_count
    root@gm400-rev05-mx7:~# /bin/echo mem > /sys/power/state
    [ 245.151950] PM: suspend entry (deep)
    [ 245.174604] Filesystems sync: 0.019 seconds
    [ 245.181294] Freezing user space processes ... (elapsed 0.001 seconds) done.
    [ 245.189722] OOM killer disabled.
    [ 245.192955] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
    [ 245.215404] fec_nm 30bf0000.ethernet rgmii: Link is Down
    [ 245.312822] *** powermcu_send_command: will send command 0x02 to MCU ***
    [ 245.323851] Disabling non-boot CPUs ...
    [ 245.329132] Enabling non-boot CPUs ...
    [ 245.333760] CPU1 is up
    [ 245.338284] *** powermcu_send_command: will send command 0x13 to MCU ***
    [ 245.348928] mmc0: error -110 during resume (card was removed?)
    [ 245.357087] fec_nm 30bf0000.ethernet rgmii: Link is Up - 1Gbps/Full - flow control off
    [ 245.357248] ------------[ cut here ]------------
    [ 245.369695] WARNING: CPU: 0 PID: 1341 at /mnt/yocto-melody/build-imx7-melody-dev-rev05-linux-rev5.4/tmp/work-shared/gm400-rev05-mx7/kernel-source/drivers/net/wireless/ti/wlcore/sdio.c:147 wl12xx_sdio_raw_write+0x100/0x1dc [wlcore_sdio]
    [ 245.390650] Modules linked in: usb_f_mass_storage usb_f_acm u_serial libcomposite configfs wl18xx wlcore mac80211 cfg80211 fec_nm(O) netmap(O) wlcore_sdio
    [ 245.404551] CPU: 0 PID: 1341 Comm: kworker/u4:3 Tainted: G W O 5.4.3-fslc+gdbe2bb1 #1
    [ 245.413513] Hardware name: Freescale i.MX7 Dual (Device Tree)
    [ 245.419278] Workqueue: events_unbound async_run_entry_fn
    [ 245.424616] [<8010fef4>] (unwind_backtrace) from [<8010c6b8>] (show_stack+0x10/0x14)
    [ 245.432375] [<8010c6b8>] (show_stack) from [<808db82c>] (dump_stack+0x94/0xa8)
    [ 245.439611] [<808db82c>] (dump_stack) from [<8012ff28>] (__warn+0xc0/0xd8)
    [ 245.446496] [<8012ff28>] (__warn) from [<8012fff0>] (warn_slowpath_fmt+0xb0/0xb8)
    [ 245.453997] [<8012fff0>] (warn_slowpath_fmt) from [<7f000240>] (wl12xx_sdio_raw_write+0x100/0x1dc [wlcore_sdio])
    [ 245.464365] [<7f000240>] (wl12xx_sdio_raw_write [wlcore_sdio]) from [<7f14e384>] (wlcore_runtime_resume+0xe0/0x22c [wlcore])
    [ 245.475686] [<7f14e384>] (wlcore_runtime_resume [wlcore]) from [<8056072c>] (__rpm_callback+0xb4/0x11c)
    [ 245.485091] [<8056072c>] (__rpm_callback) from [<805607e8>] (rpm_callback+0x54/0x80)
    [ 245.492843] [<805607e8>] (rpm_callback) from [<8056032c>] (rpm_resume+0x380/0x5ac)
    [ 245.500423] [<8056032c>] (rpm_resume) from [<805605a4>] (__pm_runtime_resume+0x4c/0x64)
    [ 245.508521] [<805605a4>] (__pm_runtime_resume) from [<7f155b38>] (wl1271_op_resume+0xbc/0x344 [wlcore])
    [ 245.518235] [<7f155b38>] (wl1271_op_resume [wlcore]) from [<7f100338>] (ieee80211_reconfig+0x5c/0xe10 [mac80211])
    [ 245.528841] [<7f100338>] (ieee80211_reconfig [mac80211]) from [<7f065efc>] (wiphy_resume+0xb4/0xe0 [cfg80211])
    [ 245.538971] [<7f065efc>] (wiphy_resume [cfg80211]) from [<805621f0>] (dpm_run_callback+0x30/0x60)
    [ 245.547856] [<805621f0>] (dpm_run_callback) from [<805626e4>] (device_resume+0xcc/0x25c)
    [ 245.555958] [<805626e4>] (device_resume) from [<80562890>] (async_resume+0x1c/0x44)
    [ 245.563628] [<80562890>] (async_resume) from [<80151228>] (async_run_entry_fn+0x48/0x1b4)
    [ 245.571819] [<80151228>] (async_run_entry_fn) from [<80148718>] (process_one_work+0x204/0x45c)
    [ 245.580442] [<80148718>] (process_one_work) from [<801489b4>] (worker_thread+0x44/0x598)
    [ 245.588541] [<801489b4>] (worker_thread) from [<8014ebc8>] (kthread+0x14c/0x150)
    [ 245.595947] [<8014ebc8>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
    [ 245.603173] Exception stack(0x8ce67fb0 to 0x8ce67ff8)
    [ 245.608233] 7fa0: 00000000 00000000 00000000 00000000
    [ 245.616421] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [ 245.624605] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
    [ 245.631400] ---[ end trace a7084b3108e9a7ab ]---
    [ 245.636035] wl1271_sdio mmc0:0001:2: sdio write failed (-110)
    [ 245.641832] wlcore: WARNING Enable for recovery failed
    [ 245.647047] wlcore: down
    [ 245.650446] OOM killer enabled.
    [ 245.653598] Restarting tasks ...
    [ 245.653732] ieee80211 phy0: Hardware restart was requested
    [ 245.659587] done.
    [ 245.664727] PM: suspend exit
    root@gm400-rev05-mx7:~# [ 246.074946] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 246.089271] wlcore: ERROR reg domain conf error
    [ 246.504858] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 246.519160] wlcore: ERROR reg domain conf error
    [ 246.934912] wlcore: PHY firmware version: Rev 8.2.0.0.245
    [ 246.949233] wlcore: ERROR reg domain conf error
    [ 246.953858] wlcore: ERROR firmware boot failed despite 3 retries
    [ 246.959911] ------------[ cut here ]------------
    [ 246.964780] WARNING: CPU: 0 PID: 20 at /mnt/yocto-melody/build-imx7-melody-dev-rev05-linux-rev5.4/tmp/work-shared/gm400-rev05-mx7/kernel-source/net/mac80211/util.c:2235 ieee80211_reconfig+0x1a8/0xe10 [mac80211]
    [ 246.983512] Modules linked in: usb_f_mass_storage usb_f_acm u_serial libcomposite configfs wl18xx wlcore mac80211 cfg80211 fec_nm(O) netmap(O) wlcore_sdio
    [ 246.997409] CPU: 0 PID: 20 Comm: kworker/0:1 Tainted: G W O 5.4.3-fslc+gdbe2bb1 #1
    [ 247.006112] Hardware name: Freescale i.MX7 Dual (Device Tree)
    [ 247.011992] Workqueue: events_freezable ieee80211_restart_work [mac80211]
    [ 247.018813] [<8010fef4>] (unwind_backtrace) from [<8010c6b8>] (show_stack+0x10/0x14)
    [ 247.026572] [<8010c6b8>] (show_stack) from [<808db82c>] (dump_stack+0x94/0xa8)
    [ 247.033809] [<808db82c>] (dump_stack) from [<8012ff28>] (__warn+0xc0/0xd8)
    [ 247.040695] [<8012ff28>] (__warn) from [<8012fff0>] (warn_slowpath_fmt+0xb0/0xb8)
    [ 247.048303] [<8012fff0>] (warn_slowpath_fmt) from [<7f100484>] (ieee80211_reconfig+0x1a8/0xe10 [mac80211])
    [ 247.058195] [<7f100484>] (ieee80211_reconfig [mac80211]) from [<7f0cc8b8>] (ieee80211_restart_work+0xac/0xe4 [mac80211])
    [ 247.069188] [<7f0cc8b8>] (ieee80211_restart_work [mac80211]) from [<80148718>] (process_one_work+0x204/0x45c)
    [ 247.079115] [<80148718>] (process_one_work) from [<801489b4>] (worker_thread+0x44/0x598)
    [ 247.087216] [<801489b4>] (worker_thread) from [<8014ebc8>] (kthread+0x14c/0x150)
    [ 247.094622] [<8014ebc8>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
    [ 247.101848] Exception stack(0x8c127fb0 to 0x8c127ff8)
    [ 247.106908] 7fa0: 00000000 00000000 00000000 00000000
    [ 247.115095] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [ 247.123281] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
    [ 247.129962] ---[ end trace a7084b3108e9a7ac ]---
    [ 247.134837] ------------[ cut here ]------------
    [ 247.139696] WARNING: CPU: 0 PID: 20 at /mnt/yocto-melody/build-imx7-melody-dev-rev05-linux-rev5.4/tmp/work-shared/gm400-rev05-mx7/kernel-source/net/mac80211/driver-ops.h:19 drv_remove_interface+0x6c/0x74 [mac80211]
    [ 247.158750] wlan0: Failed check-sdata-in-driver check, flags: 0x0
    [ 247.164932] Modules linked in: usb_f_mass_storage usb_f_acm u_serial libcomposite configfs wl18xx wlcore mac80211 cfg80211 fec_nm(O) netmap(O) wlcore_sdio
    [ 247.178802] CPU: 0 PID: 20 Comm: kworker/0:1 Tainted: G W O 5.4.3-fslc+gdbe2bb1 #1
    [ 247.187512] Hardware name: Freescale i.MX7 Dual (Device Tree)
    [ 247.193388] Workqueue: events_freezable ieee80211_restart_work [mac80211]
    [ 247.200201] [<8010fef4>] (unwind_backtrace) from [<8010c6b8>] (show_stack+0x10/0x14)
    [ 247.207959] [<8010c6b8>] (show_stack) from [<808db82c>] (dump_stack+0x94/0xa8)
    [ 247.215194] [<808db82c>] (dump_stack) from [<8012ff28>] (__warn+0xc0/0xd8)
    [ 247.222080] [<8012ff28>] (__warn) from [<8012ffb4>] (warn_slowpath_fmt+0x74/0xb8)
    [ 247.229690] [<8012ffb4>] (warn_slowpath_fmt) from [<7f0cf968>] (drv_remove_interface+0x6c/0x74 [mac80211])
    [ 247.239580] [<7f0cf968>] (drv_remove_interface [mac80211]) from [<7f0e2ca4>] (ieee80211_do_stop+0x618/0x884 [mac80211])
    [ 247.250592] [<7f0e2ca4>] (ieee80211_do_stop [mac80211]) from [<7f0e2f20>] (ieee80211_stop+0x10/0x18 [mac80211])
    [ 247.260807] [<7f0e2f20>] (ieee80211_stop [mac80211]) from [<807311a8>] (__dev_close_many+0xa0/0x118)
    [ 247.269953] [<807311a8>] (__dev_close_many) from [<8073129c>] (dev_close_many+0x7c/0x110)
    [ 247.278142] [<8073129c>] (dev_close_many) from [<807330f8>] (dev_close+0x48/0x70)
    [ 247.285855] [<807330f8>] (dev_close) from [<7f065054>] (cfg80211_shutdown_all_interfaces+0x74/0xe0 [cfg80211])
    [ 247.296103] [<7f065054>] (cfg80211_shutdown_all_interfaces [cfg80211]) from [<7f100740>] (ieee80211_reconfig+0x464/0xe10 [mac80211])
    [ 247.308253] [<7f100740>] (ieee80211_reconfig [mac80211]) from [<7f0cc8b8>] (ieee80211_restart_work+0xac/0xe4 [mac80211])
    [ 247.319248] [<7f0cc8b8>] (ieee80211_restart_work [mac80211]) from [<80148718>] (process_one_work+0x204/0x45c)
    [ 247.329175] [<80148718>] (process_one_work) from [<801489b4>] (worker_thread+0x44/0x598)
    [ 247.337275] [<801489b4>] (worker_thread) from [<8014ebc8>] (kthread+0x14c/0x150)
    [ 247.344680] [<8014ebc8>] (kthread) from [<801010e8>] (ret_from_fork+0x14/0x2c)
    [ 247.351906] Exception stack(0x8c127fb0 to 0x8c127ff8)
    [ 247.356965] 7fa0: 00000000 00000000 00000000 00000000
    [ 247.365153] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [ 247.373337] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
    [ 247.380035] ---[ end trace a7084b3108e9a7ad ]---

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    If I understand correctly, the crash only occurs if BT is enabled? If you were to remove your workarounds and start fresh, does the WoWLAN atleast function correctly with the added command to suspend? How are you enabling BT? BT and wlcore are separated and should not be causing a crash, unless you have mapped GPIO incorrectly.

    As for the cat wakeup_count > wakeup_count command, it is critical that the same value read from the wakeup_count is the same and is being written back to the wakeup_count.

  • 0 in reply to Trond Sandvik
    Prodigy 30 points

    Hi Sabbeh,

    We are running thousands of state transitions to and from suspend automatically on multiple systems.

    Lately I saw a 3rd issue with the WLAN module:

    Nov 16 04:21:02 gm400-rev05-mx7 user.info kernel: [ 6511.462774] Filesystems sync: 0.589 seconds
    Nov 16 04:21:02 gm400-rev05-mx7 user.info kernel: [ 6511.467434] Freezing user space processes ... (elapsed 0.001 seconds) done.
    Nov 16 04:21:02 gm400-rev05-mx7 user.info kernel: [ 6511.476067] OOM killer disabled.
    Nov 16 04:21:02 gm400-rev05-mx7 user.info kernel: [ 6511.479359] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
    Nov 16 04:21:02 gm400-rev05-mx7 user.info kernel: [ 6511.487935] printk: Suspending console(s) (use no_console_suspend to debug)
    Nov 16 04:21:02 gm400-rev05-mx7 user.warn kernel: [ 6512.008893] wlcore: WARNING ELP wakeup timeout!
    Nov 16 04:21:02 gm400-rev05-mx7 user.warn kernel: [ 6512.008901] queueing ieee80211 work while going to suspend
    Nov 16 04:21:02 gm400-rev05-mx7 user.warn kernel: [ 6512.008913] wlcore: WARNING CONFIGURE command NOK
    Nov 16 04:21:02 gm400-rev05-mx7 user.warn kernel: [ 6512.008920] wlcore: WARNING failed to set beacon filter opt: -5
    Nov 16 04:21:02 gm400-rev05-mx7 user.warn kernel: [ 6512.008924] wlcore: WARNING couldn't prepare device to suspend
    Nov 16 04:21:02 gm400-rev05-mx7 user.err kernel: [ 6512.009196] PM: dpm_run_callback(): wiphy_suspend+0x0/0x190 [cfg80211] returns -5
    Nov 16 04:21:02 gm400-rev05-mx7 user.err kernel: [ 6512.009211] PM: Device phy0 failed to suspend async: error -5
    Nov 16 04:21:02 gm400-rev05-mx7 user.err kernel: [ 6512.009233] PM: Some devices failed to suspend, or early wake event detected
    Nov 16 04:21:02 gm400-rev05-mx7 user.info kernel: [ 6512.057095] OOM killer enabled.
    Nov 16 04:21:02 gm400-rev05-mx7 user.info kernel: [ 6512.060254] Restarting tasks ... done.
    Nov 16 04:21:02 gm400-rev05-mx7 user.info kernel: [ 6512.065203] PM: suspend exit
    Nov 16 04:21:14 gm400-rev05-mx7 user.err kernel: [ 6524.600947] p2p-wlan0-0: failed to remove key (0, 18:65:90:d4:53:25) from hardware (-11)
    Nov 16 04:21:14 gm400-rev05-mx7 user.info kernel: [ 6524.611922] p2p-wlan0-0: HW problem - can not stop rx aggregation for 18:65:90:d4:53:25 tid 1
    Nov 16 04:21:14 gm400-rev05-mx7 user.err kernel: [ 6524.621198] p2p-wlan0-0: failed to remove key (1, ff:ff:ff:ff:ff:ff) from hardware (-11)
    Nov 16 04:21:15 gm400-rev05-mx7 user.warn kernel: [ 6525.148873] wlcore: WARNING Unable to flush all TX buffers, timed out (timeout 500 ms
    Nov 16 04:21:15 gm400-rev05-mx7 user.info kernel: [ 6525.231261] wlcore: down
    Nov 16 04:21:15 gm400-rev05-mx7 user.warn kernel: [ 6525.233844] wlcore: WARNING CONFIGURE command NOK
    Nov 16 04:21:15 gm400-rev05-mx7 user.warn kernel: [ 6525.238646] wlcore: WARNING failed to set acx_event_mbox_mask: -5
    Nov 16 04:21:15 gm400-rev05-mx7 user.warn kernel: [ 6525.245177] wlcore: WARNING CONFIGURE command NOK
    Nov 16 04:21:15 gm400-rev05-mx7 user.err kernel: [ 6525.250100] wlcore: ERROR could not configure sleep_auth to 2: -5
    Nov 16 04:21:15 gm400-rev05-mx7 user.info kernel: [ 6525.409237] wlcore: down

    This also requires a complete reboot to be resolved.

    Trond

  • 0 in reply to Trond Sandvik
    TI__Mastermind 36360 points

    Please help me understand how all of these errors are occuring. Is the device in AP or STA? Are you using 2.4 or 5Ghz? Explanations of what is going on would be helpful as I'm no longer able to reproduce any of these scenarios after my previous suggestion. 

  • 0 in reply to Sabeeh Khan1
    Prodigy 30 points

    Hi Sabeeh,

    The above error is happening regardless of BT enabled or not. I just let the WLAN mode in idle, meaning no connection is established - the poff!

    Trond

  • 0 in reply to Trond Sandvik
    Prodigy 30 points

    Hi Sabeeh,

    I have been deep-diving into the wlcore kernel driver code lately and luckily found a resolution of the issue after doing some debugging.

    This simple patch was all it took:

    diff --git a/drivers/net/wireless/ti/wlcore/main.c b/drivers/net/wireless/ti/wlcore/main.c
    index 547ad538d8b6..6ecd61eebf77 100644
    --- a/drivers/net/wireless/ti/wlcore/main.c
    +++ b/drivers/net/wireless/ti/wlcore/main.c
    @@ -679,9 +679,9 @@ static irqreturn_t wlcore_irq(int irq, void *cookie)
    mutex_lock(&wl->mutex);

    ret = wlcore_irq_locked(wl);
    - if (ret)
    + if (ret && (ret != -EACCES)) {
    wl12xx_queue_recovery_work(wl);
    -
    + }
    spin_lock_irqsave(&wl->wl_lock, flags);
    /* In case TX was not handled here, queue TX work */
    clear_bit(WL1271_FLAG_TX_PENDING, &wl->flags);
    @@ -1766,6 +1766,9 @@ static int __maybe_unused wl1271_op_suspend(struct ieee80211_hw *hw,

    out_sleep:
    pm_runtime_put_noidle(wl->dev);
    + if (wlcore_irq_locked(wl)) {
    + wl1271_info("%s: wlcore_irq_locked returns %d", __func__, ret);
    + }
    mutex_unlock(&wl->mutex);

    if (ret < 0) {

    Trond