CC3200: EAP / Radius request timeouts

Hi Michael,

this is not what we're experiencing with our customer but we might be talking about 2 separate things here. The customer is seeing that duplicate Radius / EAP access requests are sent in fast succession when their Radius server is currently slow to respond. Again, our theory here is that the CC3200 internally times out any sent EAP request if it does not receive an ACK or reply within a certain amount of time. This is different from a Radius authentication retry (interval) as our devices don't necessarily disconnect in that process but just take e.g. a longer than expected time to authenticate. As far as I know, we get no feedback or errors during this process that would allow us to throttle.

Can you please provide us with information about Radius / EAP request timeouts in the CC3200? Ideally we need a way to configure these to a higher value to (optionally) accomodate slower network architecture.

Thanks,

Lutz

Hi Lutz,

Perhaps the test I was doing on my end does not fully replicate what you are running into on your side. I have setup an EAP server that is deliberately misconfigured so that the eap authentication will fail during the initial connect phase. This does cause an explicit disconnect error event to be returned to the host.

I'll see if I can replicate the case you are seeing, where the EAP server simply doesn't respond at all to the EAP request.

While I do that and look at the NWP supplicant for the details you are requesting, are you also able to collect some NWP logs from the EAP connect attempt? That would help me identify what precisely happens and perhaps devise a method to detect a throttle condition.

Regards,
Michael

Hi Michael,

thanks for your efforts to replicate this scenario on your side. This is likely the best path forward here as collecting NWP logs is currently not possible for us. Our devices don't expose or store that information. We might be able to build special hardware for this purpose and send that over to the customer site. But that'll take us some time.

Your first test scenario is definitely not the correct one for this case. Our devices are generally able to authenticate in the customer network. The problem appears when too many devices try to authenticate at the same time and thereby induce too much load on the server. We have seen 3 scenarios:

1. Connect a small number of devices. They authenticate OK and don't send unexpected EAP requests (i.e. retries).

2. Connect a medium number of devices. They send a large number of (duplicate) EAP requests in a short amount of time but do eventually authenticate. The assumption here is that the larger number of concurrently authenticating devices cause a delay in the server's responses. This causes the devices themselves to retry their requests - which further slows down the server. The server however manages to catch up and authenticate one device after another after some time.

3. Connect a large number of devices. The number of initial EAP requests from these devices is too large for the server to handle. It slows down again. Our devices keep retrying EAP requests indefinitely because the server is no longer able to catch up. None of the devices manage to authenticate as the server has become effectively unresponsive. 

In cases 2 and 3 it's likely that our devices did at some point receive EAP packets from the server that were sent in response to an old / stale EAP request of the device. Do you know how such a case is handled? Does the CC3200 e.g. disregard answers to previously timed out requests?

Thanks,

Lutz

Hi Lutz,

Thanks for explaining your test scenarios. I'm still working through trying to get my EAP server to exhibit the same delay behavior that triggers the retry storm. I would think that the CC3200 will discard answers to requests that have already timed out, but that is something that I will need to confirm by investigating the ROM code of the device. 

Regardless of the retry interval and the behavior of the device after it gets a timed out response, it will most likely not be feasible to change those characteristics of the CC3200 due to it being in the NWP ROM code. You will probably need to perform a workaround on the application side similar to what I described in an earlier post. However, I'll keep working on replicating what you are seeing, since there might be some insight in terms of how to most effectively perform a backoff on the host MCU side.

Regards,

Michael

Hi Michael,

if the described retry behaviour is indeed reproducable and it's part of the NWP ROM code then it might not be possible to implement a backoff on the host MCU side that works in such a scenario. The problem is that the host does not receive any feedback in this case as the authentication attempt has neither failed nor succeeded while these retries are done on the NWP side. Please confirm if this is the case or if there's a possible path to resolve with the current NWP code.

If not, then we most likely require updates to the NWP code in form of a service pack drop. I can think of things like:

- WLAN events that provide the host with additional details about EAP authentication issues (e.g. high retry rates)
- APIs / configuration settings that allow the host to (optionally) control the EAP request retry interval , max allowed retries etc.
- Hitting the max retries could then trigger an auth failure event that gives the host the ability to implement a backoff / retry scheme

Please let us know your thoughts on the above and also provide an ETA for your test results.

Thank you,

Lutz

Hi Lutz,

There are three approaches that I am still working on:

1. Identifying a feedback mechanism that is currently available within the NWP code to act as a signal to the host MCU that the EAP authentication is timing out (but not failing outright). This is however looking unlikely at this point.
2. Identifying patchable code to add the feedback events that you mentioned in your previous post including a max retry limit leading to a disconnect event.
3. Identifying patchable code to adjust the EAP retry interval so as to reduce the potential of an EAP request storm.

As I am still not able to replicate the delay case with my EAP setup here, it might take a couple days to work out which of the three approaches above would work best. At this point, I am thinking the patch to adjust the EAP retry interval might be the simplest to implement, but I still need to investigate further. 

Please be aware that if a NWP patch is required, then we will need to discuss off E2E the timeline for that.

Regards,

Michael

Hi Michael,

thaks a lot for your feedback. We would very much welcome a patch that ideally includes options #2 and #3. As that would provide most flexibility in dealing with this scenario on the host side.

Since you are not able to reproduce the issue in your own lab at this point we are happy to test the mentioned patches in form of an engineering service pack. We can run such a test both in the actual customer site and their network provider's lab environment - which has an exact replica of the problematic network. This would show us immediately if the changes made lead to an improvement. It would also allow us to show some progress on our path to resolve the issue. After all, this is a very time critical issue with our customer.

In terms of discussing a timeline we're happy to continue this discussion outside of E2E. We have an ongoing email thread that Wolfgang Wendler can include you in  for example.

Thanks,

Lutz

Hi Lutz,

Going back to the first idea of having the host MCU backoff the wlanconnect if it detects that there might be an EAP request storm, there might be something that you could try.

What if the host MCU started a timer upon the wlanconnect, and if a connection is not established within say, 30 seconds, the AP/EAP infrastructure is assumed to be busy? At this point, you could perform an sl_Stop() to halt the NWP, wait for a random amount of time, then restart the process.

In our network terminal example, the wlanconnect command does actually implement this timeout idea. If you take a look at cmdWlanConnectCallback() of wlan_cmd.c, after the issuing the sl_WlanConnect() the code will pend on a semaphore. This semaphore is posted by the async event handler upon a successful wi-fi connection. If the semaphore pend times out, the wlanconnect command will assume that the sl_WlanConnect() will never complete successfully, and inform the rest of the application that the connection failed. 

While the network terminal doesn't do much with this timeout mechanism beyond informing the user that the connection failed and they should check their AP details, you could take that concept and instead perform the sl_Stop() backoff.

What do you think of that idea and its feasibility in your system?

Regards,
Michael

Hi Michael,

we do already have timeouts in place in our host code and will cancel a Wifi connection / restart the NWP in case we exceed the timeouts and / or fail to connect multiple times in a row. We are definitely going to test randomized connection retry backoff intervals as well. We are currently waiting for access to a proper network environment to test such changes.

These adjustments can help improve the situation by reducing the amount of excess request retries over time. But any connection attempt in a slow server scenario will still cause these retries as our changes can't alter the internal behaviour of the NWP at this point. And our customer expects us to (ideally) remove these retries altogether.

I believe that due to this we still require an NWP code update that provides visibility into such failure cases (by providing feedback to the host) and/or provides configurability to reduce or even terminate the issue by increasing timeouts, setting max retries etc.

As mentioned before we are happy to help you in developing and testing such changes since we have access to the affected environments.

Please let us know the feasibility, implementation details and possible timeline for the mentioned updates.

Thank you,

Lutz

Hi Lutz,

I think that the best path forward for now will be to use the randomized backoffs. The randomization should help greatly, since otherwise all of your devices will simply retry at the exactly the same time, resulting in the radius server being loaded just like before.

I'm not entirely sure that a NWP code update would result in better behavior compared to the random backoff alone. I investigated the NWP code some more and have the following observations on the NWP patch options:

1. Providing a feedback event to the host MCU when radius authentication timeouts occur:
   • This would the most technically challenging to implement, since it will touch many modules of the NWP code, and would likely require changes to the state machines and event handlers not only for the EAP authentication, but also the WPA supplicant. Furthermore, it wouldn't really prevent the need for the backoff mechanism. Thus, the work on the randomized backoffs would still need to be done.
2. Increasing the time between retries
   • Currently the retry interval is 20 seconds. While this could be increased further in a patch, this may not be the most useful to your application. As part of your Wi-Fi connect code, if you were to set your wi-fi connect timeout to a period less than 20 seconds, then there won't be a retried request. Allowing the wlanconnect code to run for 15-20 seconds is a generous period of time as-is. Thus, if your code wakes up and performs a reconnect attempt for only 15 seconds or so at a time before doing the randomized backoff, adjusting the retry time would not be useful.
3. Setting a max retry limit

• • This would be somewhat difficult to implement, since currently the EAP state machine doesn't have an obvious mechanism to keep track of how many times it's reconnected. Some modification to the wpa supplicant would also be needed to ensure that it handles this max retry limit gracefully. As with the retry interval, if you were to only let the connect sequence run for 15 seconds or so, then a max retry setting would not have an effect.

Something to note with any possible NWP code change is that the development time of a patch is only one component of the time it would take to deliver this patch. The testing for the patch would be time consuming, especially since you mentioned that this change would be deployed devices that are already running in a production setting. Running an untested patch on deployed devices would be a risk that we are not willing to take.

As such, I suggest that you implement a randomized backoff that exponentially increases upon a connection timeout, regardless of cause. That would likely adequately alleviate the loading on the radius server, and also be the quickest way to resolve this issue.

Regards,

Michael

Hi Michael,

first of all, the timeout / retry pattern that we experience is not related to the 20 seconds that the NWP applies after an authentication failure. Again, we are referring to the timeout of individual EAP requests / packets. Which is most likely rather a millisecond value. As our customer sees our devices sending multiple of these packets per second in a slow-server scenario.

The randomized backoff on the host MCU side can help prevent such a scenario by speading the load caused by authentication attempts over time. We can't however always guarantee that our devices are the only ones in the network and load can thereby be caused by other factors as well. If we assume that the server can be under load at any given time, then the randomized backoff alone will not be enough. In such a situation we would (at least) need to be able to configure more tolerant EAP request timeouts to handle the situation gracefully and not contribute to the load.

That being said we should be able to run a lab test on the randomized backoffs this week. I will get back to you with results as soon as possible. During these tests we should also be able to collect NWP logs for evaluation on your side.

Regards,

Lutz

Hi Lutz,

The frequent (quicker than 20s) retries sounds like the CC3200 is retrying not just the authentication with the radius server but also the association with the AP. 

If you could also collect air sniffer logs during your randomized backoff testing that would be useful for debug as well.

Regards,

Michael

Hi Michael,

we have been able to reproduce the issue in our customer's lab environment. I have attached the resulting NWP log file produced by a device that both participated in the EAP request storm pattern described and also ran into the issues described in another E2E ticket that we opened recently (see link below).

Please take a look at these logs and get back to us with results as soon as you can. Thanks!

nwpLog0000.bin.zip

Hi Lutz,

As Vince mentioned in his reply, having the full picture of the EAP + radius server setup would be of great value in letting us replicate what you're seeing. Currently I am running a Cisco AP with a hostapd instance running on my Linux PC running as the radius server. This setup works for basic EAP configurations, but as far as I can tell I can't perform the needed config changes to delay the auth sequence or change the reauth interval.

If you could provide me the following information in a private message it would be greatly appreciated:

1. Your AP configuration settings
2. Your radius server setup - whether it is internal to the AP, on an external host, and its config settings
3. The method which you are using the purposefully delay EAP authentication responses.

Thanks,

Michael

Hi Michael,

I don't know the exact AP / Radius server settings. We'll have to retrieve those from the network provider of our customer and get back to you.

The environment where we were able to collect our logs in is using the exact same Radius / AAA server as the customer's installation site. That server is hosted by another provider at a remote location (lab: Texas, server: California). That distance alone can of course be a source of delays in Radius server responses. The method that we use to induce further delays is to simply overload the server with too many concurrent requests. We do that by connecting a medium number of devices (~20) at the same time by e.g. powering them up simultaneously. The point here is that your Radius server should be setup with insufficient resources to handle the number of incoming requests in a timely fashion.

Please let us know if there was any useful information in the logs that we provided. Most importantly: Are you able to retrace the described issue in these logs? If so, then shouldn't this be enough information to provide you with possible paths to a solution?

Thank you,

Lutz

HI Lutz,

Is the customer setup we are looking at in this thread the same as the setup in the other thread where Vince is helping you? https://e2e.ti.com/support/wireless-connectivity/wifi/f/968/t/855614

In that thread, you mention that using an off-the-shelf freeradius install was enough to trigger the issue there. For the issue described in this thread, are you only able to reproduce it with the customer's offsite radius server?

Your logs do have some useful information regarding the EAP connection, as it does show the state transitions of the EAP engine during the Wlan connect. There seems to be some corruption in the log so I can't be 100% sure that all of the data is there, but what I can decode is informative enough for me to investigate a few possibilities.

In addition to looking at the code, I can work on indirectly reproducing the overloaded EAP server. It seems like on linux there are methods of delaying packets on an interface as well as purposefully inducing packet loss.

Do you know what is the latency between the devices and the radius server? Does it seem like the EAP packets from the server are lost entirely, or merely delayed for too long? If there is packet loss, do you know about what rate that is occurring at?

Regards,

Michael

Hi Michael,

the issue described in the other thread is visible all across my local setup, the customer site and the network provider's lab environment. So that is more of a general problem.

The EAP request retries that we are discussing here is so far only showing in the networks that use our customer's off-site Radius server. I do however assume that it would show up in any environment with similar characteristics. So you should be able to reproduce if you can induce large enough delays.

I think these delays are the main problem. They cause our devices to do request retries and discard server responses because they are received too late. I don't think that packet loss is involved.

In terms of latency I don't have exact measurements but assume that Radius server responses take at least 1 second when using a medium sample size (~20 devices). Probably somewhere in the 1-5 second range. This latency also increases as more load is induced on the Radius server. So it's somewhat variable and depends on the server's resources and the amount of concurrently connecting / authenticating devices. I assume that most of this latency emerges from server load. The physical distance due to the offsite Radius server location might contribute overall but likely to a smaller degree.

Can you please elaborate on the timeline and possibilities that you see to resolve this issue? In my opinion a good solution could be an exponential backoff like you suggested for our connection retries - but on the EAP request level. If the NWP notices that it's in a retry loop it should gradually increase its timeout interval. Or allow the host to set his own timeout.

Regards,

Lutz

Hi Lutz,

I setup a freeradius EAP server with fairly extreme delay and packet loss and got my CC3200 to not authenticate successfully with the EAP server. I setup my server to have 2000ms of delay and 50% packet loss.

However, what also seems to be happening on my end is that the CC3200 gets a async disconnect event from the NWP, with error -109 in the general event handler, as the AP disconnects the CC3200. My EAP server will send Access-reject packets if it EAP request times out on its end, which results in the AP disconnecting the CC3200.

I would suggest that you check for async disconnect errors as one of your triggers to start the backoff process in your application code. On the NWP side, there doesn't seem to be any feasible changes that can be done to change the frequency of retries. At this point in time we are unable to provide any NWP-level adjustment for what you're seeing, and you will need to move forward with an app-side solution.

Regards,
Michael