CC2652R: Thread DMM integration questions

I have located the source in radio.c that's causing the flooding, which has provided a lot of insight into the issue.

static void rfCoreRxCallback(RF_Handle aRfHandle,
                             RF_CmdHandle aRfCmdHandle,
                             RF_EventMask aRfEventMask)
{
...
        /* The RF driver has preempted the RX command. Check if the RX is
         * running and re-submit the command.
         *
         * Re-submission of the RX command is done in the RX callback to try
         * and avoid the TX command being submitted without a backgrounded RX
         * command.
         */
        if (sReceiveCmd.status != PENDING
                && sReceiveCmd.status != ACTIVE
                && sReceiveCmd.status != IEEE_SUSPENDED)
        {
            sReceiveCmdHandle = rfCoreSendReceiveCmd(sRfHandle);
        }
...
}

As seen here, RX is getting preempted by the radio because DMM is not giving exclusive, indefinite radio control to Thread. The problem is this causes a call to rfCoreSendReceiveCmd, which schedules RX again, which has the callback of this function, rfCoreRXCallback. Rinse and repeat at lightning speed. This is why RX isn't working (preemption) and why the mq_send assert fails (essentially spinning until it overflows the queue).

How can I fix this problem?

Hi,

Thanks for sharing detailed description of the issue.

I'd first recommend considering balanced mode ().

Can you share your .syscfg file for this project?

Another thing to consider is using DMMPOLICY_TIME_NONE_CRITICAL, so that the IEEE RX command can simply be scheduled after an active, higher priority RF command finishes.

Regards,
Toby

I have tried using balanced mode, but the problem still persists. Attached is my current sysconfig, although I have tried some other options, such as using the custom1 and 154sensor roles.

Pausing the BLE stack with the following patch is what enables each stack to work separately.

 dmm.stackRoles                                     = ["blePeripheral","zigbeeRouter"];
 dmm.policyArray.create(2);
 dmm.policyArray[0].$name                           = "ti_dmm_policy_dmm_policy0";
-dmm.policyArray[0].balancedMode                    = "timeBm1";
 dmm.policyArray[0].blePeripheral.$name             = "ti_dmm_policy_stack_dmm_stack_ble0";
 dmm.policyArray[0].blePeripheral.applicationStates = ["applicationState2","applicationState3","applicationState4","applicationState5","applicationState6"];
+dmm.policyArray[0].blePeripheral.pause             = "DMMPOLICY_PAUSED";
 dmm.policyArray[0].zigbeeRouter.$name              = "ti_dmm_policy_stack_dmm_stack_zstack0";
 dmm.policyArray[0].zigbeeRouter.applicationStates  = ["applicationState8","applicationState9"];

Where will I use the DMMPOLICY_TIME_NONE_CRITICAL macro? I don't see documentation on this, just that it has the following definition in my dmm_policy.h:

#define DMMPOLICY_PRIORITY_LOW     0    ///< Low priority used when using application states instead of GPT
#define DMMPOLICY_PRIORITY_HIGH    1    ///< High priority used when using application states instead of GPT

#define DMMPOLICY_TIME_NONE_CRITICAL    0   ///< Stack commands can be delayed when using application states instead of GPT
#define DMMPOLICY_TIME_CRITICAL         1   ///< Stack commands cannot be delayed when using application states instead of GPT
#define DMMPOLICY_TIME_RESERVED         1   ///< Stack commands allow delay settings are not overwritten.

SysConfig:

/**
 * These arguments were used when this file was generated. They will be automatically applied on subsequent loads
 * via the GUI or CLI. Run CLI with '--help' for additional information on how to override these arguments.
 * @cliArgs --device "CC2652R1FRGZ" --package "RGZ" --part "Default" --product "simplelink_cc13x2_26x2_sdk@4.30.00.54"
 * @versions {"data":"2020090214","timestamp":"2020090214","tool":"1.6.0+1543","templates":"2020090214"}
 */

/**
 * Import the modules used in this configuration.
 */
const ble         = scripting.addModule("/ti/ble5stack/ble");
const CCFG        = scripting.addModule("/ti/devices/CCFG");
const rfdesign    = scripting.addModule("/ti/devices/radioconfig/rfdesign");
const dmm         = scripting.addModule("/ti/dmm/dmm");
const AESCCM      = scripting.addModule("/ti/drivers/AESCCM");
const AESCCM1     = AESCCM.addInstance();
const AESCTR      = scripting.addModule("/ti/drivers/AESCTR", {}, false);
const AESCTR1     = AESCTR.addInstance();
const AESCTRDRBG  = scripting.addModule("/ti/drivers/AESCTRDRBG", {}, false);
const AESCTRDRBG1 = AESCTRDRBG.addInstance();
const AESECB      = scripting.addModule("/ti/drivers/AESECB");
const AESECB1     = AESECB.addInstance();
const AESECB2     = AESECB.addInstance();
const DMA         = scripting.addModule("/ti/drivers/DMA");
const ECDH        = scripting.addModule("/ti/drivers/ECDH");
const ECDH1       = ECDH.addInstance();
const ECDSA       = scripting.addModule("/ti/drivers/ECDSA");
const ECDSA1      = ECDSA.addInstance();
const ECJPAKE     = scripting.addModule("/ti/drivers/ECJPAKE");
const ECJPAKE1    = ECJPAKE.addInstance();
const NVS         = scripting.addModule("/ti/drivers/NVS");
const NVS1        = NVS.addInstance();
const Power       = scripting.addModule("/ti/drivers/Power");
const RF          = scripting.addModule("/ti/drivers/RF");
const RTOS        = scripting.addModule("/ti/drivers/RTOS");
const SHA2        = scripting.addModule("/ti/drivers/SHA2");
const SHA21       = SHA2.addInstance();
const TRNG        = scripting.addModule("/ti/drivers/TRNG");
const TRNG1       = TRNG.addInstance();
const TRNG2       = TRNG.addInstance();
const Temperature = scripting.addModule("/ti/drivers/Temperature");
const Watchdog    = scripting.addModule("/ti/drivers/Watchdog");
const Watchdog1   = Watchdog.addInstance();
const thread      = scripting.addModule("/ti/thread/thread");

/**
 * Write custom configuration values to the imported modules.
 */
ble.disableDisplayModule                            = true;
ble.tbmActiveItemsOnly                              = false;
ble.deviceName                                      = "Sussuro";
ble.numOfAdvSets                                    = 1;
ble.defaultTxPower                                  = "HCI_EXT_TX_POWER_5_DBM";
ble.radioConfig.codeExportConfig.$name              = "ti_devices_radioconfig_code_export_param1";
ble.connUpdateParamsPeripheral.$name                = "ti_ble5stack_general_ble_conn_update_params0";
ble.advSet1.$name                                   = "ti_ble5stack_broadcaster_advertisement_set0";
ble.advSet1.advParam1.$name                         = "ti_ble5stack_broadcaster_advertisement_params0";
ble.advSet1.advParam1.txPowerValue                  = 5;
ble.advSet1.advParam1.primIntMin                    = 1000;
ble.advSet1.advParam1.primIntMax                    = 1000;
ble.advSet1.advData1.$name                          = "ti_ble5stack_broadcaster_advertisement_data0";
ble.advSet1.advData1.completeLocalName              = "Sussuro";
ble.advSet1.advData1.GAP_ADTYPE_LOCAL_NAME_COMPLETE = true;
ble.advSet1.advData1.advertisingFlags               = ["GAP_ADTYPE_FLAGS_BREDR_NOT_SUPPORTED","GAP_ADTYPE_FLAGS_GENERAL"];
ble.advSet1.scanRes1.$name                          = "ti_ble5stack_broadcaster_advertisement_data1";
ble.advSet1.scanRes1.completeLocalName              = "Sussuro";
ble.advSet1.scanRes1.GAP_ADTYPE_16BIT_COMPLETE      = true;
ble.advSet1.scanRes1.numOfUUIDs16Complete           = 1;
ble.advSet1.scanRes1.UUID016Complete                = 0x2A1C;

CCFG.levelBootloaderBackdoor = "Active low";
CCFG.dioBootloaderBackdoor   = 1;
CCFG.ccfgTemplate.$name      = "ti_devices_CCFGTemplate0";

dmm.project                                        = "dmm_ble_thread";
dmm.numApplicationStates                           = 10;
dmm.applicationState0                              = "ANY";
dmm.applicationState1                              = "DMMPOLICY_BLE_IDLE";
dmm.applicationState2                              = "DMMPOLICY_BLE_ADV";
dmm.applicationState3                              = "DMMPOLICY_BLE_CONNECTING";
dmm.applicationState4                              = "DMMPOLICY_BLE_HIGH_BANDWIDTH";
dmm.applicationState5                              = "DMMPOLICY_BLE_CONNECTED";
dmm.applicationState6                              = "DMMPOLICY_BLE_OAD";
dmm.applicationState7                              = "DMMPOLICY_THREAD_UNINIT";
dmm.applicationState8                              = "DMMPOLICY_THREAD_PROVISIONING";
dmm.applicationState9                              = "DMMPOLICY_THREAD_CONNECTED";
dmm.stackRoles                                     = ["blePeripheral","zigbeeRouter"];
dmm.policyArray.create(2);
dmm.policyArray[0].$name                           = "ti_dmm_policy_dmm_policy0";
dmm.policyArray[0].balancedMode                    = "timeBm1";
dmm.policyArray[0].blePeripheral.$name             = "ti_dmm_policy_stack_dmm_stack_ble0";
dmm.policyArray[0].blePeripheral.applicationStates = ["applicationState2","applicationState3","applicationState4","applicationState5","applicationState6"];
dmm.policyArray[0].zigbeeRouter.$name              = "ti_dmm_policy_stack_dmm_stack_zstack0";
dmm.policyArray[0].zigbeeRouter.applicationStates  = ["applicationState8","applicationState9"];
dmm.policyArray[0].zigbeeRouter.weight             = 255;
dmm.policyArray[1].$name                           = "ti_dmm_policy_dmm_policy4";
dmm.policyArray[1].balancedMode                    = "timeBm1";
dmm.policyArray[1].blePeripheral.$name             = "ti_dmm_policy_stack_dmm_stack_ble4";
dmm.policyArray[1].blePeripheral.appliedActivity   = ["DMMPOLICY_APPLIED_ACTIVITY_ALL"];
dmm.policyArray[1].blePeripheral.pause             = "DMMPOLICY_PAUSED";
dmm.policyArray[1].zigbeeRouter.$name              = "ti_dmm_policy_stack_dmm_stack_zstack4";
dmm.policyArray[1].zigbeeRouter.weight             = 1;

AESCCM1.$name = "CONFIG_AESCCM_0";

AESCTR1.$name = "CONFIG_AESCTR_0";

AESCTRDRBG1.$name              = "CONFIG_AESCTRDRBG_0";
AESCTRDRBG1.aesctrObject.$name = "CONFIG_AESCTR_1";

AESECB1.$name = "CONFIG_AESECB_0";

AESECB2.$name = "CONFIG_AESECB_1";

ECDH1.$name = "CONFIG_ECDH_0";

ECDSA1.$name = "CONFIG_ECDSA_0";

ECJPAKE1.$name = "CONFIG_ECJPAKE_0";

NVS1.$name                    = "CONFIG_NVSINTERNAL";
NVS1.internalFlash.$name      = "ti_drivers_nvs_NVSCC26XX0";
NVS1.internalFlash.regionBase = 0x52000;
NVS1.internalFlash.regionSize = 0x4000;

Power.calibrateRCOSC_LF = false;
Power.calibrateRCOSC_HF = false;

SHA21.$name = "CONFIG_SHA2_0";

TRNG1.$name = "CONFIG_TRNG_0";

TRNG2.$name = "CONFIG_TRNG_1";

Watchdog1.$name            = "CONFIG_WATCHDOG0";
Watchdog1.watchdog.$assign = "WDT0";

thread.deviceTypeReadOnly                    = true;
thread.pm.$name                              = "ti_thread_pm_thread_pm0";
thread.rf.$name                              = "ti_thread_rf_thread_rf0";
thread.rf.txpower                            = 5;
thread.rf.radioConfig.$name                  = "ti_devices_radioconfig_settings_ieee_15_40";
thread.rf.radioConfig.codeExportConfig.$name = "ti_devices_radioconfig_code_export_param0";
thread.network.$name                         = "ti_thread_network_thread_network0";
thread.network.setNetwork                    = false;
thread.security.$name                        = "ti_thread_security_thread_security0";

Ok, I found it in ti_dmm_application_policy.c (had to exclude SysConfig from building, there's a great tutorial provided here). Unfortunately, no dice. I tried playing around with the .timingConstraint in every permutation possible since these macros are 0 or 1 but still no luck.

DMMPolicy_Policy DMMPolicy_ApplicationPolicy[] = {
    // DMM Policy 0
    {
        // BLE Peripheral Policy
        .appState[BLE_STACK_POLICY_IDX] = {
            .state = (DMMPOLICY_BLE_ADV | DMMPOLICY_BLE_CONNECTING | DMMPOLICY_BLE_HIGH_BANDWIDTH | DMMPOLICY_BLE_CONNECTED | DMMPOLICY_BLE_OAD),
            .weight = 0,
            .timingConstraint = DMMPOLICY_TIME_RESERVED,
            .pause = DMMPOLICY_NOT_PAUSED,
            .appliedActivity =  DMMPOLICY_APPLIED_ACTIVITY_NONE,
        },
        // Zigbee Router Policy
        .appState[ZIGBEE_STACK_POLICY_IDX] = {
            .state = (DMMPOLICY_THREAD_PROVISIONING | DMMPOLICY_THREAD_CONNECTED),
            .weight = 255,
            .timingConstraint = DMMPOLICY_TIME_NONE_CRITICAL,
            .pause = DMMPOLICY_NOT_PAUSED,
            .appliedActivity =  DMMPOLICY_APPLIED_ACTIVITY_NONE,
        },
        //Balanced Mode Policy
        .balancedMode = DMMPOLICY_BALANCED_TIME_BM_1,
    },
    // DMM Policy 1
    {
        // BLE Peripheral Policy
        .appState[BLE_STACK_POLICY_IDX] = {
            .state = ANY,
            .weight = 0,
            .timingConstraint = DMMPOLICY_TIME_RESERVED,
            .pause = DMMPOLICY_PAUSED,
            .appliedActivity =  DMMPOLICY_APPLIED_ACTIVITY_ALL,
        },
        // Zigbee Router Policy
        .appState[ZIGBEE_STACK_POLICY_IDX] = {
            .state = ANY,
            .weight = 1,
            .timingConstraint = DMMPOLICY_TIME_NONE_CRITICAL,
            .pause = DMMPOLICY_NOT_PAUSED,
            .appliedActivity =  DMMPOLICY_APPLIED_ACTIVITY_NONE,
        },
        //Balanced Mode Policy
        .balancedMode = DMMPOLICY_BALANCED_TIME_BM_1,
    },
};

In case this helps or someone needs an extremely lossy solution, you can ifdef out the afflicting code. It will prevent the stack from throwing an abort, but it'll keep going. Since a ton of RX transactions will fail it's super lossy and slow, but Thread will eventually maybe get there after retrying a lot. BLE will run, but it's spotty as well because the TX transmissions appear to get preempted.

#ifndef USE_DMM
        /* The RF driver has preempted the RX command. Check if the RX is
         * running and re-submit the command.
         *
         * Re-submission of the RX command is done in the RX callback to try
         * and avoid the TX command being submitted without a backgrounded RX
         * command.
         */
        if (sReceiveCmd.status != PENDING
                && sReceiveCmd.status != ACTIVE
                && sReceiveCmd.status != IEEE_SUSPENDED)
        {
            System_printf("sReceiveCmd.status: 0x%08x\n", sReceiveCmd.status);
            sReceiveCmdHandle = rfCoreSendReceiveCmd(sRfHandle);
        }
#endif

Hi Alex,

For starters, the DMM and Thread team is still looking into the Thread DMM combinations which means there could be changes required in the typical Thread application to fit the use-case. At some point there should be a official Thread DMM example available but for now, such a combination would likely require some additional effort like in your case.

Now to your issue, it seems the flooding is sort of expected based on the assumption that the RX is preempted vicious "loop". Could you share which SDK version you currently use as well as how your "Global Priority Table" is setup? Ideally, if you could share all the DMM files used (those generated by SysConfig and those available in the project) as well as the file where you configure your DMM setup (I assume main.c), that would be really helpful.

SimpleLink SDK version 4.30.00.54

CCS version 10.1.1.00004

Debugging with IC CC2652R1 on Launchpad platform (roadmap to include CC2652RB and CC2652P1)

Attached is an entire workspace dump. You should be able to import the workspace and hit build. Git history included if you'd like to see history for the current solution and what I've tried.

ccs-workspace.zip

Hi Alex,

As there is yet any DMM thread example, I can't tell you for sure to what degree the work on making the Thread stack fully "DMM compliant" in terms if feature set. This also means it is yet clear which stack activities that might be used by the stack and when. This means that it can be hard to setup a working "Global Priority Table" and that you would need to fully depend on the use "Policy Table" solution instead. 

As that is the case, I would recommend you to change all "DMMPOLICY_APPLIED_ACTIVITY_NONE" to "DMMPOLICY_APPLIED_ACTIVITY_ALL" and "zero out" all the weights in the Global Priority Table (dmm_priority_ble_thread.c). This will move the full decision weight to the Policy Table instead of a combination of the two. This is likely easier for you to control until there is an official Thread DMM example which would mean the Thread stack would be "DMMified" and have properly defined stack activities. 

I will attempt to use the Policy Table standalone and zero out the GPT. If this is a WIP for TI, I can delay some parts of implementation while waiting for a working Thread DMM example. Is there any ETA for when this will become available? Will OpenThread also get updated to a newer stable release?

As for the applied activity, I was led to believe this was an unused variable from the training video provided here at 11:46.

The last entry is the applied activity one. This is currently not used, but will be used in the future together with the index table found in the policy table.

Is there any way to view/log these priority changes or the DMM scheduling? It is difficult to know if what I'm doing is working when the Radio and DMM seem to be a black box.

So bypassing the GPT in favor of the local Policy Table did not work, same problems. Another component I wanted to point out was this - 

//! \brief Structure used to define a DMM Policy
typedef struct
{
    uint32_t     state;                         ///< application state of a policy
    uint8_t      weight;                        ///< amount of adjusted priority for stack activities specified in .appliedActivity
    uint16_t     timingConstraint;              ///< reserved for future usage
    uint32_t     appliedActivity;               ///< Specified the activities of which priority is/are adjusted by .weight
                                                ///< final Priority = GPT (Stack level priority) +  weight (Application Level)
    uint16_t     pause;                         ///< whether or not the specified stack's application is paused during the state:0 not paused, 0x0001 paused
} DMMPolicy_State;

This indicates that .timingConstraint is RFU. Are both the timingConstraint and appliedActivity unused?

Further, what affect does the GPT even have? I can't find anywhere that uses the GPT (other than where I set it in main) or anything that uses the StackActivity tables I set up. How does DMM know what StackActivity is active? How do I know these enums are correct? Is that deeper in the SDK or buried in the radio?

Hi Alex, 

As for the video, it is soon to celebrate its 2 year day and it is becoming rather outdated. When it was made, the DMM was rather "new" and there has been many changes since then, the Global Priority Table being one of them.

I would recommend looking over the SimpleLink Academy trainings  and the User's Guide  (these are always kept up to date with the SDK they ship with) to better understand how it all pieces together but I will try to give you a quick run down here as well:

https://dev.ti.com/tirex/explore/node?node=AEKDw4UXFWWpjzogXjgVAQ__pTTHBmu__LATEST

https://dev.ti.com/tirex/explore/node?node=AIS9yiev7fig2l.r-sVAPQ__pTTHBmu__LATEST

The easiest way to explain it is that the weight/priority of a given command is derived as the sum of "applied policy" + "GPT activity weight". The policy part is what you control from "user context" and what you see spread around the code in the shape of state changes. 

GPT is more of a "stack context" and it is based on information from whoever is issuing the radio command (be that the BLE stack or Thread stack). Take BLE as an example, when the stack executes the command, it passes additional information along with the command that tells the RF Driver (and DMM as it is the scheduler in this case) which BLE activity it is performing and the assigned priority. The DMM scheduler then use this additional information to look up the corresponding weight in the GPT for the given command.  

So in short, the DMM knows which activity a command is because the stack in question communicates this information via the radio driver. This means that you as a user can't impact which activities exists and what the enum values are, this is up to the stack developers to define. You are however free to dictate what the weight applied to each combination is which means you decide if you consider a "urgent BLE connection event" important or not.

This is also why I wanted you to zero this out as there is yet not official "DMM support" in the Thread stack which means it could yet to be communicating this information to the radio driver which means the GPT part would not work as expected. The Policy Table part should however work but there could still be other "inner workings" in the public release of the Thread stack that might not be aligned with the DMM requirements. For example, there is some header file requirements etc. that the stack need to consider and if they don't, the DMM would not really have an affect but your system would be more or less as "two stacks free for all". 

I will see if I can get and ETA on when we potentially would release a Thread DMM example (which would mean that the Thread stack would have been "DMMified") and get back to you on that. Until that point you might need to assume that there is not full DMM support on the Thread side which would cause you issues. In that case I would recommend you try to evaluate the "Block Mode" feature in the DMM to simply but a "mute" on the BLE stack whenever you need it to be lower priority. This should cause the DMM to reject the BLE radio access, no matter what.