WL1831: AP mode: New client unable to connect and all connected devices show failed in neighbor table

Hi Chander,

Is it possible that you have more than 10 devices connecting to the AP? There is limit on the number of devices (10) so I am curious if this limit is being reached. 

Which kernel version are you on? I see that your system is quiet outdated. We have FW, driver, and hostapd/supplicant updates. To update your system, I would recommend following this guide: https://www.ti.com/lit/ug/swru561a/swru561a.pdf 

Hi Sabeeh,

Here is the response from the customer:

Is it possible that you have more than 10 devices connecting to the AP? There is limit on the number of devices (10) so I am curious if this limit is being reached. 

>> We can see this observation less than on 4-5 devices. It is not towards AP limit.

Which kernel version are you on? I see that your system is quiet outdated. We have FW, driver, and hostapd/supplicant updates. To update your system, I would recommend following this guide: https://www.ti.com/lit/ug/swru561a/swru561a.pdf 

>> Observation is bit surprised for us because this is very basic feature & should work reliably with our version as well. We would like your support to find out root cause of issue in our version. We will be able to take upgrade or patching decision once we know root cause.

We can support you in providing log’s or run any test stuff to analyse issue faster. Please do the needful. Pls see version details below:

kernel-4.1.15

--------------------------------------------------------------------------

wl1271-nvs cortexa7hf-vfp-neon R8.6_SP1+git0+d39cb9d352-r1

wl18xx-calibrator cortexa7hf-vfp-neon 8.7.3-r0

wl18xx-firmware cortexa7hf-vfp-neon 8.9.0-r0

wlconf cortexa7hf-vfp-neon R8.6_SP1+git0+d39cb9d352-r1

wpa-supplicant cortexa7hf-vfp-neon 2.7-r0

wpa-supplicant-cli cortexa7hf-vfp-neon 2.7-r0

wpa-supplicant-passphrase cortexa7hf-vfp-neon 2.7-r0

--------------------------------------------------------------------------

Firmware revision :

wlcore: PHY firmware version: Rev 8.2.0.0.242
wlcore: firmware booted (Rev 8.9.0.0.79)

--------------------------------------------------------------------------

Hi Chander,

I strongly recommend that customer update the wl18xx-fw.bin to a 8.9.0.0.90 and re-test: https://git.ti.com/cgit/wilink8-wlan/wl18xx_fw/commit/?id=d2588c16809ecca8e0dc7ea011fc6180c7b08a92 

This may not resolve their particular issue, but we need to understand where this bug is coming from; hostapd, wl18x driver, or FW. 

After updating FW, please collect wi-fi sniffer logs with wireshark. I need more details in order to better support. 

Hello Sabeeh,

we have updated wl18xx-fw.bin to an 8.9.0.0.90 version and this update did not solve our issue i also collected Wireshark log. i have attached log file with this message

 wireshark log.zip

Hello Sabeeh,

we have updated wl18xx-fw.bin to an 8.9.0.0.90 version and this update did not solve our issue i also collected Wireshark log. i have attached log file with this message

 wireshark log.zip

We observed that some of client connection failed & some client can connect OK to Wi-Fi AP.

Hi Lokesh,

The wireshark log is not helpful. It seems you have collected ethernet sniffer. The wireshark capture should show the MAC address of the WL18x device. Can you please recollect? 

The log should show association and authentication frames. If there is an issue with the clients connecting, then the wireshark log will show this. Here is an example: https://documentation.meraki.com/General_Administration/Tools_and_Troubleshooting/Analyzing_Wireless_Packet_Captures 

Hello Sabeeh,

our system does not support monitor mode in wireshark, so can we collect log without monitor mode or we need to use external Wi-Fi Adapter that support monitor mode.

Hi Lokesh, 

I would encourage you to acquire a Wi-Fi adapter that supports monitor mode. 

Can you try to disable ELP mode and see if this rectifies the issue? ELP can be disabled with these commands:

iw wlan0 set power_save off

echo 0 > /sys/kernel/debug/ieee80211/phy0/wlcore/sleep_auth

Hello Sabeeh,

We have collected Wireshark log that attached in below zip file there are two types of log file in this one is for when all clients are connected properly and another one when this issue created.

and here MAC address of Gateways and clients for your better Understanding

Gateway Mac Address (DUT) : 34:15:13:3e:34:0c

And Mac Addresses of clients : 

laptop mac address : 74:04:f1:9a:d3:0a

Tablet mac address :  84:b8:b8:98:a5:f3

mobile mac address : 1a:19:4e:61:56:58

Wireshark_log.zip

Hello Sabeeh,

we tried using disable ELP mode by given command but it didn't rectifies our issue.

And yesterday i have send Wireshark log to you please analysis this log APAS.

Hi Lokesh,

Thank you for providing the logs. Can you please describe what that log shows?

Here is what I see. It seems that the gateway, laptop, and tablet are communicating without issues. Only the mobile device attempts to connect but sends multiple deauthentication frames. Is this what you experience?

We will probably also need to collect logs from the WL18x itself. Here is a guide on how to do this: https://www.ti.com/lit/ug/swru435a/swru435a.pdf 

Hello Sabeeh,

              As per your response mobile is sending multiple deauthentication frames. As per our Analysis other clients also sending multiple deauthentication frames. It may be happened because when this issue created client not able to connect to Wi-fi. So I off the client Wi-fi and I wait some time client tried to connect to Gateway (DUT) but it not able to connect then I repeat this process multiple time may be due to this reason we can see multiple deauthentication frames.

And according to you gateway, laptop, and tablet are communicating without issues, but I saw when this issue happened, in Wireshark log nothing special we can see as per my observation and understanding.

if we are unable to find issue according to Wireshark log then what's next Action we need to do so we can collect more information regarding this issue and do analysis further.

Here I have attached some more Wireshark log files in this time issue happened, but in this file many logs are generated when system is ok so I am unable to filter it Please see these logs if you found any informative logs so It will be helpful for us.

2308.log.zip

As you suggested to collect log from WL18x itself. but for this we need to arrange HW so this process may take times so please suggest any other method so we can help you, methods related to something changes in Wireshark configuration and other any method we can do quickly so our issue get resolve ASAP.

We observed one more issue it happens when client connectivity accrued.
generally my Gateway always connected with 3 clients (laptop,mobile and tablet)

i observed that when my issue accured i off the wifi of 2 clients then we saw that firmware of Wi-Fi module got crashed and wi-fi module start recover itself in few time then again client got connected automatically. generally when all clients connected on that time also this firmware also got crashed but it takes log time (almost 1 hour and more).

may be this observation will help you in resolve the issue.

Firmware crash log is shown below:

Sep 6 06:09:34 hostapd: wlanap1: STA 1a:19:4e:61:56:58 WPA: group key handshake completed (RSN)
Sep 6 06:09:35 hostapd: wlanap1: STA 84:b8:b8:98:a5:f3 IEEE 802.11: authenticated
Sep 6 06:09:35 hostapd: wlanap1: STA 84:b8:b8:98:a5:f3 IEEE 802.11: associated (aid 1)
Sep 6 06:09:35 hostapd: wlanap1: STA 84:b8:b8:98:a5:f3 WPA: pairwise key handshake completed (RSN)
Sep 6 06:09:53 hostapd: wlanap1: STA 84:b8:b8:98:a5:f3 WPA: group key handshake failed (RSN) after 4 tries
Sep 6 06:09:55 hostapd: wlanap1: STA 84:b8:b8:98:a5:f3 IEEE 802.11: authenticated
Sep 6 06:09:55 hostapd: wlanap1: STA 84:b8:b8:98:a5:f3 IEEE 802.11: associated (aid 1)
Sep 6 06:09:55 hostapd: wlanap1: STA 84:b8:b8:98:a5:f3 WPA: pairwise key handshake completed (RSN)     //at this point i off the wifi of tablet and laptop
Sep 6 06:10:32 hostapd: wlanap1: STA 1a:19:4e:61:56:58 IEEE 802.11: authenticated
Sep 6 06:10:32 hostapd: wlanap1: STA 1a:19:4e:61:56:58 IEEE 802.11: associated (aid 1)
Sep 6 06:10:33 hostapd: wlanap1: STA 1a:19:4e:61:56:58 WPA: pairwise key handshake completed (RSN)
Sep 6 06:10:54 hostapd: wlanap1: STA 1a:19:4e:61:56:58 IEEE 802.11: authenticated
Sep 6 06:10:54 hostapd: wlanap1: STA 1a:19:4e:61:56:58 IEEE 802.11: associated (aid 1)
Sep 6 06:10:55 hostapd: wlanap1: STA 1a:19:4e:61:56:58 WPA: pairwise key handshake completed (RSN)
Sep 6 06:11:14 hostapd: wlanap1: STA 1a:19:4e:61:56:58 IEEE 802.11: authenticated
Sep 6 06:11:14 hostapd: wlanap1: STA 1a:19:4e:61:56:58 IEEE 802.11: associated (aid 1)
Sep 6 06:11:14 hostapd: wlanap1: STA 1a:19:4e:61:56:58 WPA: pairwise key handshake completed (RSN)
Sep 6 06:11:37 kernel: wlcore: ERROR Tx stuck (in FW) for 5000 ms. Starting recovery
Sep 6 06:11:37 kernel: ------------[ cut here ]------------
Sep 6 06:11:37 kernel: WARNING: CPU: 0 PID: 1813 at /home/build/distro/master/skyline410/sky2.0/build/tmp/work-shared/imx6ul-skyline/kernel-source/drivers/net/wireless/ti/wlcore/main.c:796 wl12xx_queue_recovery_work.part.9+0x58/0x5c()
Sep 6 06:11:37 kernel: Modules linked in: cdc_acm evbug
Sep 6 06:11:37 kernel: CPU: 0 PID: 1813 Comm: kworker/u2:1 Not tainted 4.1.15-SML-20200120+g77f6154 #3
Sep 6 06:11:37 kernel: Hardware name: Freescale i.MX6 Ultralite (Device Tree)
Sep 6 06:11:37 kernel: Workqueue: phy0 wl12xx_tx_watchdog_work
Sep 6 06:11:37 kernel: [<80015aa0>] (unwind_backtrace) from [<80012664>] (show_stack+0x10/0x14)
Sep 6 06:11:37 kernel: [<80012664>] (show_stack) from [<805ed99c>] (dump_stack+0x90/0xd0)
Sep 6 06:11:37 kernel: [<805ed99c>] (dump_stack) from [<8002b95c>] (warn_slowpath_common+0x80/0xb0)
Sep 6 06:11:37 kernel: [<8002b95c>] (warn_slowpath_common) from [<8002ba28>] (warn_slowpath_null+0x1c/0x24)
Sep 6 06:11:37 kernel: [<8002ba28>] (warn_slowpath_null) from [<80362f54>] (wl12xx_queue_recovery_work.part.9+0x58/0x5c)
Sep 6 06:11:37 kernel: [<80362f54>] (wl12xx_queue_recovery_work.part.9) from [<80363074>] (wl12xx_tx_watchdog_work+0x11c/0x120)
Sep 6 06:11:37 kernel: [<80363074>] (wl12xx_tx_watchdog_work) from [<8003ebec>] (process_one_work+0x118/0x3e4)
Sep 6 06:11:37 kernel: [<8003ebec>] (process_one_work) from [<8003ef04>] (worker_thread+0x4c/0x4f4)
Sep 6 06:11:37 kernel: [<8003ef04>] (worker_thread) from [<80043e44>] (kthread+0xdc/0xf4)
Sep 6 06:11:37 kernel: [<80043e44>] (kthread) from [<8000f468>] (ret_from_fork+0x14/0x2c)
Sep 6 06:11:37 kernel: ---[ end trace 5b6ecaa2359b7107 ]---
Sep 6 06:11:37 kernel: wlcore: Hardware recovery in progress. FW ver: Rev 8.9.0.0.79
Sep 6 06:11:37 kernel: wlcore: pc: 0x0, hint_sts: 0x00000020 count: 1
Sep 6 06:11:37 kernel: wlcore: down
Sep 6 06:11:37 kernel: ieee80211 phy0: Hardware restart was requested
Sep 6 06:11:38 kernel: wlcore: PHY firmware version: Rev 8.2.0.0.242
Sep 6 06:11:38 kernel: wlcore: firmware booted (Rev 8.9.0.0.79)
Sep 6 06:12:01 CROND[2065]: (root) CMD (/etc/beanbag/syswatcher.sh 10240)
Sep 6 06:12:31 CROND[2064]: (root) MAIL (mailed 14 bytes of output but got status 0x004e )
Sep 6 06:14:31 hostapd: wlanap1: STA 1a:19:4e:61:56:58 IEEE 802.11: authenticated
Sep 6 06:14:31 hostapd: wlanap1: STA 1a:19:4e:61:56:58 IEEE 802.11: associated (aid 1)
Sep 6 06:14:31 hostapd: wlanap1: STA 1a:19:4e:61:56:58 WPA: pairwise key handshake completed (RSN)
Sep 6 06:15:36 hostapd: wlanap1: STA 84:b8:b8:98:a5:f3 IEEE 802.11: authenticated
Sep 6 06:15:36 hostapd: wlanap1: STA 84:b8:b8:98:a5:f3 IEEE 802.11: associated (aid 2)
Sep 6 06:15:36 hostapd: wlanap1: STA 84:b8:b8:98:a5:f3 WPA: pairwise key handshake completed (RSN)
Sep 6 06:18:01 CROND[2134]: (root) CMD (/etc/beanbag/syswatcher.sh 10240)
Sep 6 06:18:31 CROND[2133]: (root) MAIL (mailed 14 bytes of output but got status 0x004e )
Sep 6 06:19:18 hostapd: wlanap1: STA 84:b8:b8:98:a5:f3 WPA: group key handshake completed (RSN)
Sep 6 06:19:48 hostapd: wlanap1: STA 1a:19:4e:61:56:58 IEEE 802.11: authenticated
Sep 6 06:19:48 hostapd: wlanap1: STA 1a:19:4e:61:56:58 IEEE 802.11: associated (aid 1)
Sep 6 06:19:48 hostapd: wlanap1: STA 1a:19:4e:61:56:58 WPA: pairwise key handshake completed (RSN)
Sep 6 06:24:01 CROND[2190]: (root) CMD (/etc/beanbag/syswatcher.sh 10240)

Hi Lokesh,

From the logs I see that the AP's frequency is changing. What channel have you set the AP to?

Also, can you describe the Motorola device? Is it a new product?

Hi Sabeeh,

Currently our setup run on Auto channel.as per my observations maximum time it run on channel 1. if we need to change configure on particular channel, we can do please suggest we will change.

Motorola Device is my tablet you can check its mac address it's same as i have provided to you of mac address of tablet 84:b8:b8:98:a5:f3 

Hi,

As I mentioned on the call:

1. You can disable QoS by setting 'wmm_enabled=0' in hostapd.conf.

2. To restart the WL18x chip, you can execute on linux command line 'echo 1 > /sys/kernel/debug/ieee80211/phy0/wlcore/start_recovery'.

Hi Amrit,

It is difficult to detect from the AP in software. You can probably check the status of the driver or check the number of connected stations. If any stations drop, then you have detected the failure. 

In software, you would have to check if one of the clients dropped, so that number would ideally decrement. 

Hello Sabeeh,

we have again face the issue and i have collected the log of Wireshark and Wl18x itself. and i hope we have collected correct Wl18x log if we need to change something please suggest.  

in Wireshark log some log are generated when client is perfectly connected.

log file is attached here. 8304.log.zip

here are the Mac addresses of client and DUT

Gateway (DUT) Mac Address = d4:36:39:88:78:e8

Client Mac Address

b2:4f:d3:9d:c3:0a

74:04:f1:9a:d3:0a

5a:d1:1d:3e:dc:09

84:b8:b8:98:a5:f3

ba:9a:7b:1e:11:c0

Hi Lokesh, 

Thank you for providing the logs, but it seems it was only taken for 2 minutes. Is it possible to collect logs for when the client disconnection occurs? 

Also, Are these logs taken with the 8.9.0.0.90 FW? 

Hello Sabeeh,

Sorry for the late reply due to technical issue i am not able to provide you a log for WL18 itself.

Now i have collected log of WL18x itself sometime this issue occurs for some time that's why duration of this log is short. i have tried to collect as much as possible Wl18 log for long duration and also i have updated firmware to 8.9.0.0.90. logs are attached here 2210.log.zip

And i have some query to you

1.  is there any method where I can find total number of clients or if any client got disconnected it shows.

2. we trying to detect the issue by "ip neigh show" command by ping all available clients when all available clients not pingable then we restart the Wi-Fi module and hostapd. but we thought that "ip neigh show" list will clear after restart the Wi-Fi module but it not happening so how can we flush all ip from ip neigh show list. 

Hi Lokesh,

Does 'ip neigh show' show the same information as 'iw wlan1 station dump'? Since each client sends deauth message the AP should subtract this from connected list. 

You might want to try periodically restarting the AP and maybe the clients will automatically reconnect on restart. 

I will also further review your logs.

Hi Sabeeh,

'ip neigh show' shows ip of particular clients with their state. and 'iw wlan1 station dump' show information of all clients like their mac address, connected time, signal strength etc. it does not show ip address.

in 'ip neigh show' we observed that many time when our client's wi-fi is off on that time it also shows client is reachable and after restart the Wi-fi module and hostapd Deamon it shows previous assign ip with failure or incomplete state we want if this list got clear than only, we restart hoastapd then our issue might be solve.

and if we restart AP periodically than all clients will take time to connect. and in what time we restart AP it will difficulty for us because some time this issue got occur after reboot the system (sometime later).

Hi Lokesh,

Other than commands you can use in userspace, we don't have the ability to determine if a particular device is losing communciation with AP. 

Hello Sabeeh,

As you suggested on Sept 14 for WL18 log for longer duration, we provided required log to you with upgraded FW on Sept 15.

1.So please share what finding you got from the logs.

2.is there any further action to find out root cause (if still you not find the root cause) we need to do let us know.

3.When we can expect any solution from your side?

regards,

Lokesh

Hi Lokesh,

The wl18x logs you shared were not parsed correctly as compared to those shared earlier and thus do not show anything useful. All files show "Bad Packet(s) Received". The earlier ones 

As a debug step, it would be good to get a AM335x EVM or Beaglebone Black with WL18x and use our latest software. You can find the guide to our latest software here: https://www.ti.com/lit/ug/swru561a/swru561a.pdf 

If the issue is not reproducible on our latest, then it will help us begin to understand what the issue is. As a reminder, your current software builds is outdated. Let's work backwards to solve this. 

Hi Sabeeh,

Today I have also collected some more log of Wl18 itself. I have highlighted time stamp when issue is created and also I have highlighted in red color when module FW got crashed, I hope this will help to you to find root cause of this connectivity issue.

logs are attached here.glogger_log.zip

And I have one query to you in your latest FW (8.9.0.0.90) is there any logic in which if our clients not able to connect to AP then module FW got crashed.

Hi Lokesh,

Thanks for these logs, I am actively reviewing this. 

I'm not sure what you mean by your question. In this scenario, the FW is not crashing. It seems to be an error with the hostapd or wl18xx linux driver. This is why I am suggesting to try the latest linux software so that we can confirm that the latest does not have the same issue. 

What I see in this log is that a client attempts to connect, and then the WL18x immediately disconnects. I'm still looking into why this could be

Have you tried .configuring hostapd using an open network? Does the issue continue?

Hello Sabeeh,

As suggested, we will update results with you when hostapd using an open network testing done.

Regards,

Amrit

Hello Sabeeh,

As suggested by you, we set hostapd.conf for open network by setting wpa=0

We observed same issue persist, When we restart hostapd & udhcpd Linux process, but issue persist.
When we restart Wi-Fi module firmware by "echo 1 > /sys/kernel/debug/ieee80211/phy0/wlcore/start_recovery" command, issue get resolved.

Regards,

Amrit

Hello Sabeeh,

As per discussed I have collected Wireshark and Wilink itself logs. logs are attached here. 2746.logs.zip

in ZIP file both (wireshark and wilink) logs are saved of same duration

this issue occurred from IST 14:35 - 18:00 on 3rd Oct 2023

in the end i manually fire command fire for recover the Wi-Fi module 

in AP channel is set to 1 

mac of AP  34:15:13:3e:32:6c

and the MAC address of clients are

74:04:f1:9a:d3:0a
00:13:e1:12:19:da
ba:43:23:c1:48:91
84:b8:b8:98:a5:f3

Hi Lokesh,

These logs are not that much different compared to the last. The sniffer is not capturing enough data to better debug.

Please see this screenshot: 

There are multiple time jumps in the log and they jump multiple seconds!! The AP is sending out beacon frames every 100ms, so there are thousands of messages that are not captured. 

Have you had a chance to get a different sniffer? If not, I would recommend testing in a RF chamber. Or you can poweroff all of the WiFi devices (mobile devices included) in the area and hopefully the sniffer can capture more data. 

Hello Sabeeh,

Now we are using WLAN sniffer TP-LINK (model : TL-WN722N) 

please suggest if we need to configure some settings in order to increase buffer size and then able to log more data.

I am not familiar of that part exactly. I would encourage you to search around the web and understand what the issue might be. Or you can turn off other WiFi devices in the area. 

Hello Sabeeh,

As per suggested by you we have create issue in RF chamber also and i have attached log (Wireshark and wilink both) here New folder.zip
in this log issue is occurred at 10-10-2023 10:50 (IST) and I recover this issue by manually at 10-10-2023 11:19 (IST)

Hi Lokesh,

I need to review further, but unfortunately these logs don't tell me much. See this image below of a DEAUTH message. There are many ACKs and then the client drops out. There is no way of telling if WL8 or your client device is the issue. 

Your sniffer also suggests that the AP is on many different channels, so I think there is an issue with the sniffer itself. 

Hi Amrit,

We need to understand why clients are sending DEAUTH or disconnect messages. It is not just that beacons are missing, it is that there are many messages missing from the sniffer, so it is impossible to understand why the client or the AP is sending DEAUTH. For example, if WL18x sends a message to client, and client is expecting it but does not receive, then likely it will send DEAUTH. This entire transaction should be seen in sniffer log. 

The log you have only has ACKs from both devices, there is no data in between. No dhcp request or similar. Not much use to look at. 

Hi Amrit, 

It should perform much better than the current monitoring hardware, and note that it is most commonly used with Ubuntu machines. 

If you are looking for ready made sniffer, then please read through this guide: https://wiki.wireshark.org/CaptureSetup/WLAN There you can find descriptions of various adapters and how they work with WireShark. 

Hi Amrit, 

I'm not aware of any. As I mentioned, if you're looking for a ready made product, please read through the guide above.