WL1807MOD: mpath dump indicates very high PATH_CHANGE

Hi,

Hard to tell what could cause this high counter number. Usually, it may increase in case of heavy loaded environment, drops, bad RSSI, etc.

However, 3 times a second sounds a lot.

Does it matter how many nodes are connected?

can you start in low number of mesh devices and increase gradually to see whether you can see a point where it starts exploding?

is the counter reset to 0 before starting?

What are the values set in your code for mesh?

Shlomi

Yes, the number of nodes matters. For example, a network with 12 nodes works fine, while 32 nodes is typically only stable for a few hours, then various "offline storms" occurs. It is a bit difficult to say what is the limit since the issues appears gradually more often. We will do additional tests, though. 

Mesh gateway uses:

mesh_hwmp_rootmode=4
mesh_gate_announcements=1

Other nodes have zero for these parameters. Additionally, in wpa_supplicant:

max_peer_links=8
mesh_max_inactivity=10
beacon_int=500 # only on gateway
mesh_basic_rates=60 120 240

We have also started evaluating the use of:

mesh_rssi_threshold=-80

Did not solve the problem on the 32 node mesh, however. 

As we understand it, it is preferred to not use CCK at all. However, we have not found an effective way of entirely disabling CCK, since "basic rates" AFAIK only applies to broadcast etc. Even with the settings above, we can see that there are established stations which uses 1.0 MBit/s. Do you have any recommends for disabling CCK? And what settings to use in general for a 32 node mesh?

In general, I think the system is a bit too aggressive in creating new "PLINKs". Many nodes are creating 8 PLINKs; the maximum. This does not make sense to me, but perhaps there is a reason? We cannot lower max_peer_links since that means that new nodes will not be able to connect. 

Hi,

Interesting. If you want to disable CCK, maybe you can try to work in 5GHz where the lowest rate would be OFDM 6Mbps.

I never tested with so many devices in mesh so cannot comment further.

I can try to ask around internally.

Regards,

Shlomi

We cannot use 5 GHz since we only have 2.4 GHz antenna etc. 

I am surprised to hear that you have never tested 32 nodes, since https://www.ti.com/lit/ug/swru423a/swru423a.pdf says:

Maximum number of nodes in the entire network: 32

Are you saying that 32 nodes is not supported?

I am not saying it is not supported, just that I have not tried it or tested it.

Will touch base with the test team and ask if they did.

Shlomi

Hi again, have you talked to test team yet?

Not yet unfortunately since the main expert is out on vacation.

once he is back, I will close the loop.

Sorry for the delay.

Shlomi

Hi again, it's been a month now, any update on this topic? If 32 nodes is not tested and supported, what is the actual limit? 

Hi,

from the testing team I understand that 32 nodes were not tested.

The bench we have is composed of 10 units.

Regards,

Shlomi

We are surprised to hear this. Stability issues in general tend to appear when you stress the system. If you do not test many nodes, high load, corner cases etc, you will not be able to find critical bugs. 

Perhaps this all explains why we are seing for example firmware regressions while you are not :WL1807MOD: WL1807 firmware Rev 8.9.1.0.2 regressions .

We have now switched to as much as upstream software as possible, instead of using WiLink components and patches:

* upstream kernel 6.6 (with patches from 6.11)

* upstream wpa_supplicant 2.11 

Now it works better, but we are still seeing kernel tracebacks/warnings, and nodes sometimes goes offline. However, we cannot debug the firmware since it is closed source. I think you should consider either doing more testing, or provide the source code for the firmware so that bugs and stability issues can be fixed by customers and the Open Source community instead. 

Hi,

This is understood Peter.

Firmware is a closed binary and cannot be shared. Mesh related implementation is in driver/supplicant anyway.

Not sure what you get with the uplink version, but it may be missing many patches. I believe there are patches that are mesh related on both, driver and supplicant.

Shlomi

I am wondering, if we donate 32 Beaglebone Green Wireless, can you start doing regression tests with 32 nodes?