CC1312R: mac security

Update:

I have discovered that my Beacons are now sending variable length data. I'm not sure what this is. Before, the beacons only sent one byte which I manually programmed thus:

    ApiMac_mlmeSetReqUint8(ApiMac_attribute_beaconPayloadLength, 1); //set the beacon payload length
    ApiMac_mlmeSetReqArray( ApiMac_attribute_beaconPayload, &current_path_cost); // set the beacon payload

Any ideas why this has changed?

Also, How do I turn off pending addresses? I don't think I've seen that before and I don't need that data to be transmitted in a beacon.

Thanks.

Hi Andrew, 

Please can you elaborate, what you mean by "devices recovering after a few hours"? Do these devices start receiving your data? 

Regards,
Siddanth
 

Hi,

Thanks for your reply. I'm still trying to work out what's going on and am discovering things as I go. If I power up the coordinator then a few devices will rejoin. Then the network doesn't recover, but over a period of several hours, it seems devices will eventually rejoin.

Please be aware my network is not as per your example applications. I have a router based network.

Regardless, I have identified an issue with beacons. All my beacons should be 1 byte, but I am getting beacons with either 17 bytes or 25 bytes. I'm not sure where this is coming from. I've not seen this issue till I updated to SDK 5.30 (from 4.30).

Can you tell me if anything has changed regarding the beacons? As you can see here, the payload is 17 bytes and I have set it to 1:

update:

I've changed to number of bytes in the payload from 1 to 10. Now I get 10 bytes every time. So there's a problem in the stack if the payload is set to 1?

I've also tried changing to 4 bytes, and that works. On top of this, I've tried scattering the beacon payload config around my code but it doesn't make any difference. My coordinator always starts with the correct beacon payload, but then after about 17s it changes to some other value. After some time, it reverts back to the correct payload and length, for a while. I don't seem to be getting this problem with my routers. So is this problem only on the PAN coordinator? Otherwise, the two applications are set up the same, as far as I know.

Hi Andrew,

In the change logs I do not see anything that would affect/cause this behavior. I will look into this.

But please can you clarify the main issue here. What I have understood is this: The beacon payload is not set to the expected length only if it is set to 1 byte but gets set to the correct length when you set it to other lengths(10 bytes, 4 bytes etc)? 

Also the observation where you see the change in the beacon payload after 17s, is this the behavior consistent with any beacon payload length setting?

Regards,
Siddanth 

Hi,

I have just discovered that I see the same issue with 4 bytes payload. I didn't capture it before, but it's there. So it seems it's not related to 1 byte payload specifically.

I am setting the payload and length in the cllc.c init() and not again in the application.

May be I am doing something that is affecting the memory, but I don't know what. The 17s varies. It can easily be 40s. It's useful to know nothing has changed with the beacons. I will see if I can identify at what point the beacon payload changes.

Thanks,

Andy

I cannot get a handle on this at all. I am currently testing the mac_cfg.c settings for mac queue size as that seemed to make a difference. I had them set to a higher value, if I returned them to default sizes the network beacon issue seemed to go away. But I am still trying to prove that.

Please keep us updated regarding your findings and let us know any specific erratic behavior that you see, that we can investigate.

Hi,

Yes I will. I need to confirm if it's my code or the stack.

Many thanks,

Andy

/* ------------------------------------------------------------------------------------------------
 *                                           Constants
 * ------------------------------------------------------------------------------------------------
 */

/* maximum number of data frames in transmit queue */
#ifndef MAC_CFG_TX_DATA_MAX
#define MAC_CFG_TX_DATA_MAX         2
#endif

/* maximum number of frames of all types in transmit queue */
#ifndef MAC_CFG_TX_MAX
#define MAC_CFG_TX_MAX              5
#endif

/* maximum number of frames in receive queue */
#ifndef MAC_CFG_RX_MAX
#define MAC_CFG_RX_MAX              2
#endif

It seems that if I change these values from the default, then I get rogue beacon payloads. I suggest that the memory used by the mac is being overwritten when these values are larger than default. I can only think that the stack has some pre-compiled libraries that are stuck using the default values, regardless of what these configuration values in mac_cfg.c

I also tried putting the defines in collector.opts but it made no difference, still lots of beacon payload issues:

-DMAC_CFG_TX_DATA_MAX=24
-DMAC_CFG_TX_MAX=36
-DMAC_CFG_RX_MAX=24

Here is the wireshark view. I have set a filter to capture any beacons with a data length > 1. Although I don't think it's a beacon problem, just that this is how the issue manifests itself.

This is how I would expect one of my beacons to look:

This was never a problem in SDK 4.30.

Hi Andrew, Thank you for the sniffer logs, I will look into this and see if there are any other changes in the SDK source code that might be causing this.

Most likely this is a non issue, but please can you confirm that you are using the same tool chain and optimization settings as well.

Hi, (I've carried on adding to this response to try and give as much information as possible, sorry it's quite long now).

Many thanks. I need to solve this as soon as possible as my client is now having to hold off deployment. If there is anything else you can think of that is worth checking then please let me know.

I will try to explain the changes to my project below. Please comment if you think any of this may be the cause of the error.

I will continue investigating my code. I did see that if I didn't send any messages out from the collector by not calling ApiMac_mcpsDataReq(), then the beacons from the collector seemed to be okay. For a while at least, I did not test this thoroughly but my thoughts are that it is a mac issue.

I started with the collector example from v5.30 SDK and converted this project to my own so that settings remain the same. I also upgraded the IAR compiler to version 9.20 (previously 8.50.9 when on v4.3 SDK). I have tried the default full optimisation and no optimisation and the problem persists.

My project is different in that I have Heap Tracking enabled. I have removed ti_utils_build_linker.cmd.genlibs in Sysconfig and have a local copy instead with a line commented out. See below.

There are also some other files that have been included manually as shown below, such as mac_util.c, osal_port.c. This is following advice from my previous discussion on heap track:

https://e2e.ti.com/support/wireless-connectivity/sub-1-ghz-group/sub-1-ghz/f/sub-1-ghz-forum/984795/cc1312r-heap-track

This is the line commented out in the ti_utils_build_linker.cmd.genlibs:

//"ti/ti154stack/lib/iar/m4f/maclib_osal_tirtos_cc13x2_26x2.a"

You can see I have also included api_mac.h as I make a small change in that file by adding an extra association response enumeration:

/*! Associate Response status types */
typedef enum
{
    /*! Success, join allowed */
    ApiMac_assocStatus_success = 0,
    /*! PAN at capacity */
    ApiMac_assocStatus_panAtCapacity = 1,
    /*! PAN access denied */
    ApiMac_assocStatus_panAccessDenied = 2,
    /*! Parent at capacity */ 
    ApiMac_assocStatus_parentAtCapacity = 3
} ApiMac_assocStatus_t;

I had to do quite a lot of work to get this migration from SDK 4.3 to work as there was a problem with configPkg being built in the stack as opposed to locally. I had not had this issue before and may be it is linked to this problem. Here is my previous post, please read the last message I posted:

https://e2e.ti.com/support/wireless-connectivity/sub-1-ghz-group/sub-1-ghz/f/sub-1-ghz-forum/1054278/cc1312r-idle-function-and-possible-sysconfig-issue

The migration from SDK 4.3 CC13X2 to the merged SDK 5.3 CC13XX has been difficult with various build problems. I suspect something has gone wrong along the way. I never got an answer to why configPkg was being left in the stack to be built. 

I look forward to your response.

Andy

I have created a brand new project from the examples in SDK v5.3. I can confirm that configPkg is built in the SDK and not local to my project. Can you confirm this is correct? I know it caused issues for me when migrating. You can see here the build time for configPkg is today:

I have had a look at the collector_CC1312R1_LAUNCHXL_tirtos_iar.ipcf file for SDK v5.30 and SDK v4.3 and they are different with respect to the location of configPkg:

V4.3

v5.3

and a freshly downloaded V5.20:

Hi,

Any thoughts on this?

Thanks,

Andy

Hi Andrew, 

I am sorry, I have no updates regarding this. I am looking into it.

Regards,
Siddanth

OK thanks all the same. I tried starting over again with my migration and minimised the changes to the set up as much as I could, so no heap tracking and only one file imported manually. It made no difference, I still had the same issues. I'm certain that the changed and/or incorrect location of configPkg is strongly linked to the problem. All my cllc and collector files are local to the project as they are heavily modified from the examples.

Andy

Hi,

It also seems that 'software_stack' is not local to the project either. This is probably where my issue resides.

Andy

Hi,

I can simplify my issues to two parts:

1) I have tested for the beacon payload issue on my previous versions of SDK (back to SDK V4.3) and the problem is still there. This means the beacon payload issue is not unique to SDK v5.30 as I thought it was. The only way I can eliminate the beacon payload issue is to set the MAC_CFG_TX_DATA_MAX, MAC_CFG_TX_MAX and MAC_CFG_RX_MAX to their default values.

2) SDK v5.30 is still not created properly from the IAR example workspace. configPkg and software_stack are left in the stack and not built locally.

I hope this helps,

Andy

Hi Andrew,

Thanks a lot for your patience, I appreciate it. This issue has been tricky to resolve. I apologize for the delay.
I have consulted a colleague regarding the issue with IAR project management and the build process. 
I am on vacation from tomorrow, so he will follow up with you on Monday. 

Regards,

Siddanth

Hi Andrew,

Just a short update on this.

As you correctly pointed out, there have been some significant changes in the way that TI 15.4 example projects are being handled in the latest release of the SDK. The location of configPkg is one example of this. But this isn't the only thing that I see different.

For instance, in previous versions of the SDK such as SDK 5.20, most of the example files where copied from the SDK install files, rather than being referenced. By doing this, the customer could simply open the projects by going to the SDK example folder, opening the template file (e.g., collector_CC1312R1_LAUNCHXL_tirtos_iar.template.eww), and using IAR to select a directory where the workspace for the project would be set.

Currently, what I'm seeing is that the import process that works for previous SDK versions would not work for SDK 5.30, because the paths for the various files wouldn't be correct and the project wouldn't build. The only way to have a project that builds, is by adding (not importing) the project directly with IAR (that is, in IAR, go to the "Project" tab and click on "Add Existing Project").

Now, this process works in the sense that the project builds, but it is problematic in the sense that local SDK files are being modified, which is something that is not recommended. This is what you are seeing with configPkg, etc.

I need to ask RnD about this because perhaps there's a logic behind the changes between the different SDK releases, or perhaps it was something that they didn't considered.

Regarding your original question about the beacon payload. Give me some time to see how things work, and I'll get back to you.

BR,
Andres 

Hi,

Thank you for your response to this.

Now that I have separated the beacon payload issue from the project configuration in V5.30 (I initially thought they were linked), I think I have overcome the project configuration problems by manually editing the IAR project. All my files are local to my project now, including configPkg and no files are being changed in the SDK. 

This leaves the original beacon payload issue being most time-critical for solving. As previously mentioned, I can confirm this issue is present in all SDK versions from at least v4.30. I have done a fair amount of testing and the only way I can stop the beacon payload issue is by not calling ApiMac_mcpsDataReq(). My Collector will call this function every time a Sleepy Sensor tries to rejoin and the more rejoining Sleepy Sensors the quicker the problem appears. I'm testing with about 60 sensors.

Setting the mac queue sizes to their default solves the beacon payload issue, but my network stops working (in that it slows down on rejoins to the point it's unusable) as it depends on having a mac queue size of, say, 16 or so as all Sensors are sleepy.

Regards,

Andy 

*update: I have also tried keeping the mac queue sizes larger than the default, but limiting the number of times I call ApiMac_mcpsDataReq() so the number of mac messages queued stays within the default value of 2 (just for testing, this won't work for my network). This seems to also stop the beacon payload error. From this I deduce that the problem seems to be with the Tx mac queue rather than the Rx mac queue as I am still receiving a large number of messages.

My original question asking about changing these values:

e2e.ti.com/.../cc1312r-ti-15-4-maximum-number-of-mac-rx-and-tx-buffers

Hi Andrew,

First, I apologize if your thread has been left unnoticed for a while. Since I wasn’t the person originally assigned to it, I didn’t receive any reply notifications. But this shouldn’t happen, so I apologize.

Second, I want to reproduce exactly what you are seeing using the basic sensor/collector examples as a base point, so I need to make sure that I only modify the examples enough to observe the issue (the thread is a bit long now and I don’t want to forget about anything). Please correct me if I’m wrong or I’ve forgotten to add something.

Main issue:  Even if the beacon payload length is set to 1 byte (or any other value), the collector sends beacons with variable payload length.

Necessary modifications to reproduce issue:

1. The beacon length is originally set in Cllc_init() by making a call to ApiMac_mlmeSetReqUint8(ApiMac_attribute_beaconPayloadLength, 1)

Note: The default value for the payload is 0 bytes and it is defined in mac_cfg.c, and the maximum beacon payload length is 52.

2. The constants for the maximum number of data frames in queues (MAC_CFG_TX_DATA_MAX, MAC_CFG_TX_MAX and MAC_CFG_RX_MAX) have been modified. When returning these constants to the default values, the issue disappears.

I'm aware that you have done additional modifications to make the network router-based, but I'm guessing that these modifications are the ones that I need to consider when attempting to reproduce what you are seeing. Am I correct?

BR,
Andres

Hi,

Yes, you are correct. The beacon payload length mostly seems to go to length==17 for some reason. It's not really that variable. It's either 1 (correct) or 17.

I have tried to recreate the issue using standard examples on the TI dev boards. I haven't been able to do this. I think this is  because the problem only presents itself when the Collector is being asked to send lots of messages. I have 60 custom boards with my own end device (Sensor) application running. The beacon issue happens when I turn off the Collector. All sleepy child devices go into a rejoin state. That is, they send out beacon requests, and upon receiving a valid beacon, they send an unsecured rejoin request. (This is functionality I have added and works similar to ZigBee). The Collector receives the rejoin request, adds the device to its tables and transmits back a rejoin response using ApiMac_mcpsDataReq().

It is this ApiMac_mcpsDataReq() that is being ran many times as 60 nodes are trying to rejoin at the same time. And it is the same ApiMac_mcpsDataReq() that if I remove, is the only thing that corrects the beacon payload issue besides leaving  (MAC_CFG_TX_DATA_MAX, MAC_CFG_TX_MAX and MAC_CFG_RX_MAX) as default. Interestingly, I did not see the error when ApiMac_mcpsDataReq() was commented-out even though the receive queue was still functional with a non-default size.

I wanted to get more confidence that the problem isn't my code. While I'm not certain of this, I spent some time trying to comment-out as much of my code as possible. I removed the flash writes upon rejoin and many other things. As mentioned previously, nothing made a difference apart from the removal of the ApiMac_mcpsDataReq() upon rejoin. 

My next move would be to reprogram my 60 sensors with the standard sensor example, but I'm not sure how to replicate the rejoin process without modifying the example code a lot. It's a while since I looked at the standard example code in detail so I'm not sure how easy this is to do, but may be it's what's needed.

In summary, I think the problem is due to calling ApiMac_mcpsDataReq() many times in a short period of time, and I suspect the MAC Tx queue size is hard coded in the library files with the default queue size (MAC_CFG_TX_DATA_MAX), but when running, allows the changed MAC_CFG_TX_DATA_MAX value to add to the queue.

Thanks for your help,

Andy

Hi Andrew,

Andrew Bakin said:

Yes, you are correct. The beacon payload length mostly seems to go to length==17 for some reason. It's not really that variable. It's either 1 (correct) or 17.

Understood.

It would be odd if the number of devices was the culprit for this issue, because in that case, I would expect that making a call to ApiMac_mcpsDataReq() would return an error status (like ApiMac_status_noResources or something else). Is it possible for you to run the same test that you have been running but with fewer devices? This is so that we can be sure that the number of devices is playing a role in the problem.

Andrew Bakin said:

It is this ApiMac_mcpsDataReq() that is being ran many times as 60 nodes are trying to rejoin at the same time. And it is the same ApiMac_mcpsDataReq() that if I remove, is the only thing that corrects the beacon payload issue besides leaving  (MAC_CFG_TX_DATA_MAX, MAC_CFG_TX_MAX and MAC_CFG_RX_MAX) as default. Interestingly, I did not see the error when ApiMac_mcpsDataReq() was commented-out even though the receive queue was still functional with a non-default size.

I'll check the source code to see if somehow ApiMac_mcpsDataReq() can interact with the beacon and affect its payload length. 

Andrew Bakin said:

In summary, I think the problem is due to calling ApiMac_mcpsDataReq() many times in a short period of time, and I suspect the MAC Tx queue size is hard coded in the library files with the default queue size (MAC_CFG_TX_DATA_MAX), but when running, allows the changed MAC_CFG_TX_DATA_MAX value to add to the queue.

I'll also check the source code to see if this is happening, although it would be rather odd because then the issue would manifest itself without the need to make many calls to ApiMac_mcpsDataReq().

If you have any other ideas on how I can reproduce the issue, let me know, and I'll be happy to test.

BR,
Andres

Hi,

Thanks. I will test with fewer devices and try to get a number for the threshold where the error occurs. 

I will also look to see if I can reproduce the problem using the standard examples.

In the meantime, it would be great if you can confirm that the source is correct. 

Regards,

Andy

Hi Andy,

Let me know if you can reproduce the problem.

I've been checking the source code, and I haven't seen anything that would explain what you are seeing. Although I still need to check a few things.

ApiMac_mcpsDataReq() basically just ends up sending the application data to the MAC for transmission in a MAC data frame. But I don't see how it could affect the payload length of something like the Beacon.

I still need to see how things are handled when it comes to sending the beacon, so I'll keep you posted on that one.

BR,
Andres

Hi Andy,

Any updates from your side?

I checked the source code to see how the buffer containing the beacon frame is built and didn't find anything that would explain what you are seeing. It basically calculates the length of the beacon frame using a base length and the beacon payload length defined in the MAC PIB (i.e., the one you set using ApiMac_attribute_beaconPayloadLength).

I've also tried to play with the examples to see if I can reproduce the issue, but so far I haven't had much success, so if you have any ideas, I'll be happy to help.

BR,
Andres

Hi Andres,

Many thanks for your help, I really appreciate it. I am currently programming up the example sensor application on my custom hardware. Hopefully I'll have something to report back by end of Monday.

My best guess is that it's nothing to do with how the beacon payload is constructed, but rather I am seeing a memory leak causing data to get corrupted. This corrupt data just so happens to show itself as an incorrect beacon payload, but could easily be any other process that gets affected, the worse case, a process that I have no visibility on. 

Kind regards,

Andy

Hi Andres,

I am currently configuring the example software to run on my hardware to try and reproduce the issue. One thing that stands out, is the use of 'Orphan Notification'. I do not use it at all. Instead, more like ZigBee, I issue a 'Beacon Request', then a 'Rejoin Request'. So for my device to rejoin the network, the potential parent has to issue a 'Beacon'. 

Are all Ti's examples based upon the orphan notification? If so, the parent (collector) would not be sending lots of beacons like my network.

I'd be interested to know your thoughts.

Kind regards,

Andy

Hi Andres,

Thank you for you reply. Just to be clear, my application does not use orphan scan or coordinator realignment in any way. I have largely rewritten the link controllers to rejoin through ApiMac_mcpsDataReq(). I am changing the examples now to emulate rejoining without orphan functionality to see if I can replicate the issue. If I can I will send you the project code.

Kind regards,

Andy

Hi Andres,

I have now seen the beacon payload error when using the Ti examples.

Because all Ti example code uses orphan notification, rather than performing an active scan, you will never be asking the parent device to transmit beacons when an end device is performing a rejoin. If you are using the same stack for your sub-1GHz Zigbee, then I expect this to be a problem for you as ZigBee does not use orphan notification either. Instead, uses beacons like I am doing.

How shall I send the files with changes I have made to you for analysis?

Kind regards,

Andy

Hi Andres,

Thanks for the update. Someone told me about the ZigBee solution which must have been incorrect information. Thanks for clearing that up.

I think that it is an issue with the stack, or possibly the way I am using it which was not intended by the Ti developers. However, I believe it should work if it is to adhere to the 802.15.4 standard.

I can send you the files using Friend Request. For now, here's an outline of what I've changed:

1) I have disabled orphan scans in jdllc.c by changing handleMaxDataFail(), and instead call an active scan with my own 'network joining' state. This scan does a beacon request which the collector, if online, will respond to with a beacon.

    else
    {
        /* start orphan scan */
        //switchState(Jdllc_deviceStates_scanOrphan);
        //updateState(Jdllc_states_orphan);
      
        switchState(Jdllc_deviceStates_scanActive);
        updateState(Jdllc_states_network_joining);
      
    }

2) Then when the end device scan is complete it sends a message using ApiMac_mcpsDataReq() to the collector. 

3) The collector responds to this message again with a ApiMac_mcpsDataReq() which the end device receives and then it carries on as normal. The messages don't really contain anything useful, as the end device doesn't delete any of its network data, it doesn't need anything from its parent to carry on working. I have put the message transaction in to simulate how my own network rejoins a parent.

4) Data and polls happen every 60s:

5) The mac queue length is larger than default:

/* maximum number of data frames in transmit queue */
#ifndef MAC_CFG_TX_DATA_MAX
#define MAC_CFG_TX_DATA_MAX         16
#endif

/* maximum number of frames of all types in transmit queue */
#ifndef MAC_CFG_TX_MAX
#define MAC_CFG_TX_MAX              24
#endif

/* maximum number of frames in receive queue */
#ifndef MAC_CFG_RX_MAX
#define MAC_CFG_RX_MAX              16
#endif

6) I have increased the mac persistence time in the Collector:

/* MAC Indirect Persistent Timeout */
//#define INDIRECT_PERSISTENT_TIME (MAX((5 * 1000 * CONFIG_POLLING_INTERVAL / 2), MIN_PERSISTENCE_TIME_USEC)/ \
//                                  (BASE_SUPER_FRAME_DURATION * \
//                                    SYMBOL_DURATION_LRM))

#define INDIRECT_PERSISTENT_TIME ((1000 * CONFIG_POLLING_INTERVAL)/ \
                                  (BASE_SUPER_FRAME_DURATION * \
                                    SYMBOL_DURATION_LRM))
                                    
/* adds 30s onto INDIRECT_PERSISTENT_TIME for margin                    */
/* calculated as:                                                       */
/* SYMBOL_DURATION_LRM * 960 (BASE_SUPER_FRAME_DURATION) = 48ms         */
/* 30s / 48ms => 625                                                    */
#define INDIRECT_PERSISTENT_TIME_MARGIN 625
                                    
                                    

ApiMac_mlmeSetReqUint16(ApiMac_attribute_transactionPersistenceTime,
                            INDIRECT_PERSISTENT_TIME + INDIRECT_PERSISTENT_TIME_MARGIN);

7) And add a payload to the beacon in collector.c init():

    uint8_t value = 5;
    ApiMac_mlmeSetReqUint8(ApiMac_attribute_beaconPayloadLength, 1); //set the beacon payload length
    ApiMac_mlmeSetReqArray( ApiMac_attribute_beaconPayload, &value); // set the beacon payload

I get the beacon payload problem when I turn off the collector having had >25 end devices running. All end devices after two minutes go into beacon request mode. The collector, once powered up again, starts sending out beacons. It then receives messages from rejoining end devices which it responds to. Crucially, because the end devices are sleepy, the collector must hold onto any outbound messages till the end devices does a data request. This causes the mac queue to get full easily as its processing lots of messages. The mac should return a fail when ApiMac_mcpsDataReq() is called but the mac queue is full. I think something is going wrong here and mac messages are spilling out into other memory. This memory just so happens to be affecting the beacon payloads. 

I'll send you a friend request and get the files to you.

Thanks so much for your help,

Andy

Hi Andy,

I have accepted the friend request. Once you send the project files I'll take a look at them.

Andrew Bakin said:

This causes the mac queue to get full easily as its processing lots of messages. The mac should return a fail when ApiMac_mcpsDataReq() is called but the mac queue is full.

This scenario is actually covered by the stack as there is a specific status for that case when the MAC queue is full (I have tested this). Nevertheless, it's worth taking a look at it to see if there is something that might explain what you are describing.

BR,
Andres

Hi,

I have sent a link to the files via friend messaging.

I agree that the mac queue full is probably working as intended and you have proved this. However, I think that the problem is that the maximum size for the mac queue is not being globally changed when I select larger values in mac_cfg.c. Instead, the mac is allowing a larger number of messages to be queued as per my modified mac_cfg.c values, but the memory allocated is staying at the default '2'.

Kr,

Andy