SIMPLELINK-CONNECT-SW-MOBILE-APP: list of advertising devices does not appear in the Available devices screen

Dear Jan,

apologies for this week-late update, but the reason behind the delay is not procrastination or, even worse, carelessness, but, rather, a very big effort in researching what could have actually gone wrong behind the (many and multi-layered) scenes. Rarely I have devoted this many resources in a non-strictly-strategic issue such as this one, without getting to a clear and definitive conclusion, even though (spoiler), the problem has been solved, albeit without a clear explanation. I must warn you, this is a long post, reflecting the many branches that this investigation led to. I, also, decided to gain access to another device, specifically a Samsung J3 running Android 5, which is the lowest version requirement imposed by the app. In the meantime, I have lost access to the Samsung A22 device (mentioned in my original post and running Android 13) and it's unlikely that I will be able to test it again in the near future. My findings, actions and conclusions, however, are not affected by this last factor, also because the original E2E post I related to and that I linked earlier, already gave a very precise hint about some sort of disconnection between the specific Android versions, also given that 13 is basically still the most current one, except on Google phones and on high-end flagships that have only recently been RTMed.

I will split this post in four separate sections, in order for you (and anyone else affected or interested) to better understand my inquiry, regarding the SimpleLink Connect (SLCA from now on) BLE scanning issue on Android.  

INTRODUCTION

In the original post by user Masaki, it was already clear that she/he noticed a strange behavior according to different app versions (releases) in multiple Android versions and, already, there was something a bit suspicious and not very linear happening, considering that Android API changes seemed not specifically matched by code updates in the SLCA. This is why I began my journey by exploring two paths in parallel: looking into other possible related bugs, especially linked to the frameworks and libraries - I remind you that SLCA is built on the Expo framework, which stacks on top of the React Native Framework (both are JavaScript and Node.js based tools), that then further translates to the respective native Android (and iOS) libraries - used and possibly matching them to the app source code. The toolchain and stack layers here are very complex and you must add to them the huge dependency and versioning nightmare that is everything npm/npx based, plus all the JS quirks that are especially unfamiliar to a C-minded FW developer like me. Finally, one must consider the evolution and changes in the now yearly Android SDK/API releases (and the whole mess that they bring with in manifest-time and runtime permission requirements. It turns out there is another layer, even more obscure and potentially problematic, that I had initially ignored, but that could actually be the main one to blame, which is vendor/device/HW module-specific implementation (i.e. at the Android OS driver level), which is a real factor and probably the reason why vendors, when in good faith and not because it doesn't make sense commercially, claim they cannot update/upgrade a certain phone to the latest Android release. The other, more ordinary, path was to inspect, tweak and rebuild the code - even though I have almost zero experience in app development and I dislike JS, which I have only used in very limited browser and Node.js demos, like, for example, some fancy web-based front-end/dashboard, setting up a basic MQTT server on the cloud and some database-related code for data logging. While parsing the code, I also tried to examine version-specific commits, because, apart from what emerged from the Masaki post, I had briefly tried a previous, much earlier version of the SLCA working and scanning properly on that very same device it was now malfunctioning on. Even though I'm not skilled at it, I was encouraged by the fact that the issue actually took place at the very earliest stage (i.e. BLE device scanning), so I could limit myself in chasing the "bug", on a limited code surface. Here are the results.    
   
CHAPTER A: THE UNEXPECTED BEHAVIOR/BUG (BLE SCANNING NOT WORKING)

1) PRELIMINARY RESEARCH, LINE OF THOUGHT, FINDINGS AND HYPOTHESES

To be honest, my suspicion from the very beginning was that the issue could be related to Android permissions, which are a major source of problems, especially between versions. I had checked, however, both in the SLCA code and within the device settings, that these permissions were both announced in the manifest and requested at runtime (as required) correctly in the code, with some minor uncertainty in some version-specific property settings that, potentially, could have created some problems. I played as much as I could with them (by deleting them or changing the Android versions they applied to) when rebuilding the SLCA (see Chapter B), but this didn't lead me anywhere. I also wanted to remind you that this was the first time ever that a BLE app gave me any problem at the scanning phase, no matter how complicated, device specific or "core" they were. That's when I started investigating every possible obscure and undocumented React Native related issue on GitHub and Stack Overflow, where I found some issues that I could hypothetically relate to. Here is just a small selection of a few of the relevant threads I considered, with their potential root cause listed in the title:

DIFFERENT (GENERIC) ISSUES:
github.com/.../1019

ANDROID 13 SPECIFIC - CHANGE IN DEFAULT BLE STACK:
github.com/.../943
www.xda-developers.com/.../

BLE STACK ISSUES / RESET THE BLE PERIPHERAL:
stackoverflow.com/.../android-10-bluetooth-le-scan-issues

DISABLE / RE-ENABLE LOCATION PERMISSIONS:
github.com/.../695

The last two causes, as you will read later on, are probably the ones to blame, with the last one being the most likely.

So, after spending countless hours searching and reading and (in parallel) in the building activities described in Chapter B, I finally decided to look for a device (a spared Samsung J3 sourced from a friend) running the earliest supported Android version (5), with the intention of climbing up/down the version ladder to see if the v8 I was initially unsuccessfully testing, despite being declared compatible, was at the lower end of the "supported, in theory, but untested" subset of devices. Quite surprisingly, it worked right away. This was useful for three reasons: I could rule out the low-end version theory; I got some great insight with version-related permissions, which are dealt with very differently in every Android release; finally, the fact that it was a Samsung - like the original A22 Android 13 phone, which also had the issue, although from, probably another "Samsung age", since it is at least 7-8 years old - having encountered certain specific vendor-related bug reports, probably with more "Samsung density", given the fact that they are the Android market leaders and likely sell more devices than all the others combined.

2) MITIGATION ATTEMPTS, RESULTS, BUG REPRODUCTION AND FINAL CONSIDERATIONS

Along with the results in parallel from Chapter B, I ultimately singled out, by subtraction, the issue to the vendor's (and the device model's) implementation of the Android OS and/or of parts of it. One more factor - apart from the already mentioned driver layer - to consider is that, while the underlying core OS modules should be (reasonably) the same for all phones, most vendors don't implement "vanilla flavor" Android, but slap their own UI (something I didn't consider too material to my specific issue), on top of the mandatory layers and actually disable and/or tweak/simplify certain features as they like. My last consideration was that, in the past, the only pain I had suffered when using other "core" BLE apps (i.e. BLE Scanner, LightBlue, nRF Connect etc.) was when trying to flush the BLE cache on my phone. The recommended method of deleting the BLE core service cache has never worked for me, something that has led to various headaches and that I'm aware of. So, basically, I suspected that the specific BLE HCI (or, more likely, some other layer) implementation was neither "vanilla" or 100% super compliant (or at least it didn't completely follow the mainstream behavior). The other lead was the ubiquitous and infamous location permission that Google is forcing whenever you're using BLE, which was also a major suspect in the above mentioned bug-related threads. This mandatory requirement, in my view, is not problematic just in principle, but also in practice. I always thought that its real purpose was to feed Google's massive Wi-Fi SSID/MAC (and, possibly even BLE MAC) location database, which they probably keep to enhance their main advertising data offering and revenue stream, but, apparently, they want to track you down with GPS too. My phone, like all other ones, doesn't have a dedicated GPS switch, but the more general "Position" icon, which is always enabled on my phone by default. So, following certain comments in the threads, I decided to focus, as much as I could, given they are user-inaccessible features that are buried in the phone, on trying to change the state of both the BLE and the GPS modules, essentially by playing with the cache, resetting the phone, cross-disabling/enabling them multiple times both in the Apps >> Permissions settings and toggling the BT and position icons and at different stages of the SLCA boot-up and operations. Given that I couldn't plan any meaningful course of action, but only a kind of "CTRL-ALT-DEL" approach, I have no specific and definitive clue about what exactly fixed the issue, only that it happened after quite a few random attempts (not right away). The one other, very minor (as it turns out), factor was that the storage permission was requested at boot time on the Android 5 phone, but was only announced on the Android 8 one (i.e. not requested at runtime and disabled in Settings >> Permissions, something that I enabled manually from there. After the scanning started working, however, I disabled it manually, rebooted the SLCA and it kept on scanning, so it's unlikely that the storage permission was material to the issue.

So, ultimately, the good news is that the SLCA started working after this chaotic intervention, the bad news is that I can't actually pinpoint the single (if it's only one) factor, because I wasn't able to reproduce the "bug" anymore, at least at this early stage. This might not be the end of the story, however, because it could actually stop working again and, in that case, I will try to be more methodic, knowing that the issue (likely) is resolved by what I just described. All this said, my best educated guess is that:

- the reason why the earlier SLCA releases worked could have been related not only to the app itself, but to the specific state (as in "state machine") of my phone (and, specifically, of its BLE and GPS modules). I suspect that, the low-level driver layer exchanges some state information with the Android OS, which might relay some false or misleading states to the permission and (especially) to the peripheral state event flags, with some buggy or unexpected error handling. It should be noted, anyway, that there are countless device and vendor-specific complaints (even related to very popular phones) in the Google Play store reviews, even for many very popular commercial apps, so this is a very well-known Android problem (unlike in the Apple market);

- the SLCA uses a very simple approach to app-building (makes perfect sense, given its purpose and scope) and relies on its multiple multi-platform framework modules, whose error handling might be affected by unknown or incorrect states relayed from the OS, especially when it comes to corner/edge cases such as mine. As opposed to the more robust native approach - because of better module resetting calls and no translation inconsistencies - perhaps used in those other "core apps" which I mentioned earlier (that BTW are also long-term projects and not "collateral" tools for specific embedded device training) this other development strategy might translate into a less robust cross-device behavior;

- SLCA device compatibility testing is probably more limited than average and, again, it makes perfect sense, since it's just basically a "playground", where firmware developers can test their device implementations and study the algorithms/code and proper steps to pass on to phone app developers. We all want TI to focus on devices, SDKS, documentation, great E2E support and not on apps.


CHAPTER B: ISSUES WITH BUILDING THE APP ON THE CLOUD AND LOCALLY

During my investigation, As I have already mentioned, I tried to find out if there was something slightly suspicious in the SLCA code, but first, I had to figure out how to rebuild it. I had realized that it was developed with React Native and built (and, possibly deployed) with Expo, although I discovered it didn't just use Expo-specific modules, but also at least one React Native-native (confusing, I know) one, the document picker. This meant (at least as I have understood), that it cannot be built, debugged and viewed on-the-fly with the handy Expo Go app, but it has to be built separately. This last step can be carried out either in the cloud, by signing up to and using the Expo EAS services, or locally on your machine.  


1) BUILDING ON THE EAS CLOUD

On the EAS cloud, the vanilla code compiled and sideloaded OK (at least up to the enable scan stage), but my tentative experiments and tweaks (which I began on 1.3.3 and then continued on 1.3.4 after it was released) either didn't correct the issue, or else they prevented the app from booting or crashed it. The only problem I had when building v1.3.4 (not with v1.3.3) was that it aborts reporting a syntax error in the app.json file (SCLA root directory) with an extra "," after "com.ti.connectivity.simplelinkconnect", that you might want to look into. Perhaps the developer is using certain more relaxed building/compiler flags/options:

Error parsing JSON: {
  "expo": {
    "name": "SimpleLinkTM Connect",
    "slug": "SimplelinkConnect",
    "version": "1.3.4",
    "orientation": "portrait",
    "icon": "./assets/images/icon.png",
    "scheme": "myapp",
    "userInterfaceStyle": "automatic",
    "splash": {
      "image": "./assets/images/splash.png",
      "resizeMode": "fill",
      "backgroundColor": "#cc0000"
    },
    "updates": {
      "fallbackToCacheTimeout": 0
    },
    "assetBundlePatterns": [
      "**/*"
    ],
    "ios": {
      "supportsTablet": true,
      "bundleIdentifier": "com.ti.connectivity.simplelinkconnect"
    },
    "android": {
      "package": "com.ti.connectivity.simplelinkconnect",
    },
    "web": {
      "favicon": "./assets/images/favicon.png"
    }
  }
}
└─ Cause: SyntaxError: Expected double-quoted property name in JSON at position 672 (line 27 column 5)
  25 |     "android": {
  26 |       "package": "com.ti.connectivity.simplelinkconnect",
> 27 |     },
     |     ^
  28 |     "web": {
  29 |       "favicon": "./assets/images/favicon.png"
  30 |     }


Once I removed the "," I ran:

eas init

(result:)

(node:22799) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
 Existing project found: @disegni/SimplelinkConnect (ID: e1dc81ef-ff2c-4686-ab56-43b735fb7f09). Link this project? … yes
 Project successfully linked (ID: e1dc81ef-ff2c-4686-ab56-43b735fb7f09) (modified app.json)

then I added an eas.json build profile file in the SLCA root directory, in order to build a standalone apk that I could sideload in my phone, which contained the following:

{
  "build": {
    "preview": {
      "android": {
        "buildType": "apk"
      }
    }
  }
}

I then ran:

eas build -p android --profile preview

and everything went fine (with both versions), with the resulting SLCA sideloaded and working on my phone like the one downloaded from the Google Play store. Initially it wouldn't scan, but after the efforts described in Chapter A, once reinstalled after deleting the now working official build from the Google Play store, it would scan. BTW, the only visible difference when built using this EAS cloud method vs. the official version is the much larger file size 38.6MB vs. 15.37MB. This may be due to some optional modules being baked in, or some debugging code that I haven't bothered to flag out.

This (only partially explained) success, however, didn't fully end my investigation. I wanted to check back the earlier 1.3.3 version, in order to see whether the actions described above would allow scanning on that one too. Since I could not choose to download the older version from the Google Play store, I rebuilt it from source on the cloud and sideloaded it, after uninstalling v1.3.4. No matter what I tried, it would not scan, so another finding is that the latest version does indeed bring some earlier Android compatibility benefits. By the way, and I might be wrong, one main behavioral difference that I have noticed before the scanning stage between SLCA v1.3.3 and v1.3.4 is that the latter apparently reboots right after requesting and setting permissions and this could be one of the mitigation factors, since this latest release notes actually did mention earlier Android version fixes.

2) BUILDING LOCALLY

In this case, I found setting up the toolchain properly very complicated, to the point of it being almost undefined, because of the dependency and versioning requirements spread over multiple layers: host OS, Node.js/npm, deprecated Expo version - there are very big code incompatibilities between releases and it's mandatory that you stick to that same version - different OpenJDK versions according to which Expo/RN, Watchman dependencies and absence of pre-built packages, Android toolchain build components, SDK and NDK components and probably even more. The local build process logs, despite trying to follow the (confusing and spread everywhere) instructions, are an endless flow of warnings, deprecations, options, version exceptions etc, which, for me, always led to frustrating abort statements, some at very late stages. It must be said that the recommended instructions at the SLCA GitHub page seem to suggest building directly via React Native (i.e. skipping the Expo stage, as I understand it, perhaps because all Expo modules and configurations are already set up/pre-built, configured or whatever else, in the distributed code). The instructions, however, are very short and they refer to more detailed ones on the React Native website, but that's where I get lost with my Linux machine, due to the versioning problems. It even suggests building and installing Watchman, a very specific file monitor tool, which BTW is apparently in itself one of the abort reasons, from source code and then running it, but this I would say is out of the scope of my journey. This is why I have, at least temporarly, abandoned this local build option and only relied on the successful EAS cloud one.

CONCLUSIONS

I realize that this post is very long, especially given that it is related to my individual and specific issue, and I apologize for the time taken before and after it. There are two reasons why, however, I put all this effort, both in researching the problem and in reporting my actions and findings in such great detail. The latter is so that anyone who encounters another episodic issue with the SLCA, can refer to them, without wasting all the time it took me to investigate and (almost) sort out mine. Perhaps, the random and unclear mitigation steps mentioned earlier will work for them too. The former is that, given the exceptional, personal and free(!) top-tier engineering support that we get here on this amazing E2E forum, I think that it is our moral obligation, once we raise a question or an issue and grab the related attention from TIers (and other users too), that we do our technical very best, as a follow-up and as a possible solution to share with TI and other fellow E2Ers.

Thank you for your kind and personal help and for the new, working, app release. I am now ready to start fully implementing OAD in my CC2340 application framework and flagship device, strictly according to TI's guidelines and best practices.

Best regards,

Stefano

PS feel free to mark the issue as resolved, unless you need more clarifications.

Hi Jan,

thank you for your kind words. Just one more comment and a request to the SW team:

1) I have had a second thought on those permission issues, could it have something to do with permission request timeouts / not waiting for a call's result before actually starting to scan (and it locks for that reason)? I'm a bit shocked every time something in JavaScript is translated into more procedural languages like the native ones. When I approach a language, I'm not so interested in the syntax and the libraries/methods etc. but more about how it (meaning the compiler, which for me IS the language) maps to the actual hardware and OS (perhaps this is due to my FW/C/ASM background. That JS works like it has an in-built scheduler/supervisor and everything is running all over the place has always puzzled me as the exact opposite of being deterministic. Plus, honestly, it seemed a little bizarre that this v1.3.4 looks like it's rebooting after getting permissions, I've never seen this behavior anywhere else. Also, it's possibly the only BLE app I've ever encountered that doesn't check whether BLE is actually enabled. Anyway just a rookie's thoughts...

2) in your next release, could it be possible to only use native Expo modules, I think the only one that is Reactive Native-native is the document picker, but perhaps there are more? The switch shouldn't be too hard, I suspect it only involves choosing the FW bins and perhaps just a few other options. In some comments on the web, I've also read that this approach has solved some compatibility issues. I understand that there could be a reason for this not being possible, but the outcome would be that we could play with the code live and have the results shown on the fly in Expo Go. It would be awesome, as a starting point for app (and FW feature and demo) development.

Thank you and the SW team once more for your great support and again, feel free to tag this issue as "TI thinks it's solved" (and I'll follow up confirming), unless you want to elaborate more or it becomes an open canvas for SimpleLink Connect issues or discussions on its improvement.

Kindest regards

Stefano