CC2652R: TX operation completed without TxDone event

An update. I see now that the pOp->status of the rfc_CMD_IEEE_TX_t command is set to 0x2802, i.e. IEEE_ERROR_NO_FS.

How could this be? Any suggestions on how to handle this error?

Hi Toby,

Thanks for your reply! Is this behaviour documented somewhere?

Where would be the correct place for placing the workaround, in my radio_callback function or in pErrCb?

And regarding pErrCb. In the comment for RF_Params it says that pErrCb is deprecated. Is there any documentation/guidance in how to handle errors reported from the RFC?

Br Henrik

Hi Henrik,

Henrik Smeds said:

Thanks for your reply! Is this behaviour documented somewhere?

There is some detail about it here: https://dev.ti.com/tirex/content/simplelink_cc13xx_cc26xx_sdk_7_41_00_17/docs/proprietary-rf/proprietary-rf-users-guide/rf-core/radio-operation-commands.html?highlight=terminationreason#error-cases

Henrik Smeds said:

Where would be the correct place for placing the workaround, in my radio_callback function or in pErrCb?

To handle it in the radio_callback, you may have to subscribe to that error code (similar to the other statuses for which you expect the callback to be executed).

The pErrCb specifically is called by RF_fsmActiveState (in file <CC13x2/CC26x2 SDK>/source/ti/drivers/rf/RFCC26X2_multiMode.c).

Either may be used to accomplish the same.

Henrik Smeds said:

And regarding pErrCb. In the comment for RF_Params it says that pErrCb is deprecated. Is there any documentation/guidance in how to handle errors reported from the RFC?

Which SDK are you using? At least for the one I have installed, simplelink_cc13xx_cc26xx_sdk_7_40_00_77, the RF Driver still includes it.

I would refer to the same RFCC26X2_multiMode.c for how to handle RF errors. Are there other specific RF errors you are concerned about? I can try to help address those.

Thanks,
Toby

Hi Toby,

At the moment I am using SDK 7.10.01.24, pErrCb exists but is "marked" as deprecated in the comment. But if it doesn't matter then I will handle it in the radio_callback function.

Currently my approach is: The Op command status is checked on all events in the radio_callback, if the status is IEEE_ERROR_NO_FS then a message will be posted to my radio thread (I run TI-RTOS). The radio thread will do this when handling the message:

        // Cancel any RF commands in the queue
        uint8_t stopMode = 0; //1: Stop gracefully, 0: abort abruptly
        RF_cancelCmd(rfHandle, RF_CMDHANDLE_FLUSH_ALL, stopMode);

        // Post a new CMD_FS
        RF_postCmd(rfHandle, (RF_Op *)&RF_cmdFs_ieee154_0, RF_PriorityNormal, 
                    radio_callback, RF_EventLastCmdDone);

Henrik Smeds said:

it seems like the Rx operation that I try to schedule after CMD_FS fails (not sure how yet, just that I don't get any CmdDone or RxEntry events after I posted the Rx command).

Were you able to check the status of that command in the debugger? (CMD_IEEE_RX.status)

It may be that the RX command is active, but doesn't receive a packet... There's a dependency here on the whichever device is expected to transmit a packet. Since there is time consumed for the FS to finish, the RX command may have been started after the expected TX already happened.

Can you try continuously TX'ing the expected packet at regular intervals (independent of system timings)?

Note for the RF_cancelCmd, you may need to have stopMode be bitwise OR'ed with RF_ABORT_FLUSH_ALL (to match the RF_CMDHANDLE_FLUSH_ALL).

Henrik Smeds said:

NO_FS is handled by RF_yield in the link you posted, should I do that instead?

The RF_yield would power the radio off (but maintain pointers, e.g. the FS command to run during the next power up). This is a viable option.

Henrik Smeds said:

Also, this is super tedious to work on since the occurrence is so rare, do you know if it is possible to somehow provoke the behaviour to occur more frequent?

The FS is PLL based. The issue with trying to provoke more FS errors, is that even if the right handling (in the radio application) is implemented, the conditions which artificially provoked the FS/PLL errors would then occur even after the right handling (e.g. after the first FS error and conditions persist, correct error handling, then the following FS cmd would also error).

Usually for these types of issues, we run a test for X amount of time (where X is enough to cause the issue to occur, in this case it seems to be hours). And during this time, gather as many logs as possible while minimally impacting system performance.

Toby said:

Were you able to check the status of that command in the debugger? (CMD_IEEE_RX.status)

No, unfortunately not. I have three devices concurrent devices, and this error might occur on one of them pretty randomly. So in that case I could have to debug all three at the same time (not a bad idea though, I think I will do that and see what I can learn).

Toby said:

t may be that the RX command is active, but doesn't receive a packet... There's a dependency here on the whichever device is expected to transmit a packet. Since there is time consumed for the FS to finish, the RX command may have been started after the expected TX already happened.

Can you try continuously TX'ing the expected packet at regular intervals (independent of system timings)?

Forgot to mention that the RX command have an endTime relative to startTime and endTrigger set to TRIG_REL_START, i.e. I expect to get a RF_EventLastFGCmdDone event even if the TX.

Toby said:

Note for the RF_cancelCmd, you may need to have stopMode be bitwise OR'ed with RF_ABORT_FLUSH_ALL (to match the RF_CMDHANDLE_FLUSH_ALL).

I will look into that!

And as you say, logging seems to be the way to go, that's the approach Ive had so far as well.

Thank you for your helps far! 

Henrik Smeds said:

No, unfortunately not. I have three devices concurrent devices, and this error might occur on one of them pretty randomly. So in that case I could have to debug all three at the same time (not a bad idea though, I think I will do that and see what I can learn).

It is possible to let the device free run (no debugger connected), then connect to running target once the issue occurs on a device. More info on this project-less debug: https://software-dl.ti.com/ccs/esd/documents/users_guide/ccs_debug-main.html#manual-launch

You would want to avoid a reset when connecting, so this one could help: https://e2e.ti.com/support/tools/code-composer-studio-group/ccs/f/code-composer-studio-forum/432770/cc2640-connecting-to-a-running-target

Henrik Smeds said:

Forgot to mention that the RX command have an endTime relative to startTime and endTrigger set to TRIG_REL_START, i.e. I expect to get a RF_EventLastFGCmdDone event even if the TX.

Ah I see, in that case I'd agree that this RX command should've ended at least based on time.

Hi again Toby.

Unfortunately this issue still troubles us. I moved over to just doing an RF_yield in the callback (as suggested in the docs) and posted a message to my rf task that posts a new RX to the RFC and tries to get everything back on track. And it almost works as I hoped, but I still get some strange issues.

I log all calls to the radio_callback, here are the events that occour around the NO_FS status:

 IDX:  CORE_MASK   DRIVER_MASK    TIMESTAMP   pOp                    pOp Status             RadioState   Delta time (us)   MASK_BITS                     
...
 18:   00800000     00000000    538832149   rfCmdRxSample          ACTIVE                   00000005      23624.0   RF_EventRxEntryDone,          
 19:   00000000     40000000    538833897   rfCmdRxSample          IEEE_DONE_STOPPED        00000005        437.0   , RF_EventCmdStopped          
 20:   0000001C     00000000    538881850   rfCmdTx1Sample         IEEE_DONE_OK             00000005     11988.25   RF_EventFGCmdDone | RF_EventLastFGCmdDone | RF_EventTxDone, 
 21:   00800000     00000000    539029399   rfCmdRxSample          ACTIVE                   00000005     36887.25   RF_EventRxEntryDone,          
 22:   00000000     40000000    539031400   rfCmdRxSample          IEEE_DONE_STOPPED        00000005       500.25   , RF_EventCmdStopped          
 23:   0000001C     00000000    539080117   rfCmdTx1Sample         IEEE_DONE_OK             00000005     12179.25   RF_EventFGCmdDone | RF_EventLastFGCmdDone | RF_EventTxDone, 
 24:   0000001C     00000000    539134275   rfCmdTx2Sample         IEEE_DONE_OK             00000005      13539.5   RF_EventFGCmdDone | RF_EventLastFGCmdDone | RF_EventTxDone, 
 25:   00000003     00000000    539260137   rfCmdRxSample          IEEE_DONE_OK             00000005      31465.5   RF_EventCmdDone | RF_EventLastCmdDone, 
 26:   0000001C     00000000    539282307   rfCmdTx1Sample         IEEE_DONE_OK             00000005       5542.5   RF_EventFGCmdDone | RF_EventLastFGCmdDone | RF_EventTxDone, 
 27:   0000001C     00000000    539332451   rfCmdTx2Sample         IEEE_DONE_OK             00000005      12536.0   RF_EventFGCmdDone | RF_EventLastFGCmdDone | RF_EventTxDone, 
 28:   00000003     00000000    539423007   rfCmdRxSample          IEEE_ERROR_NO_FS         00000005      22639.0   RF_EventCmdDone | RF_EventLastCmdDone, 
 29:   00800000     00000000    539822442   rfCmdRxSample          ACTIVE                   00000005     99858.75   RF_EventRxEntryDone,          
 30:   00000000     10000000    539824053   rfCmdTx1Sample         IEEE_DONE_OK             00000005       402.75   , RF_EventCmdCancelled        
 31:   00000003     00000000    539855180   rfCmdRxSample          IEEE_DONE_OK             00000005      7781.75   RF_EventCmdDone | RF_EventLastCmdDone, 

A description of what we see:

On line 18 we get receive a beacon message. When we get that packet we cancel the RX, schedules data that should be sent into a slot and transmits it and after that we will get a TxDone event (line 20). The same goes for the next mainframe, line 21-24.

On line 25 we failed to receive a beacon messge (the RX operation finished without receiving a RxEntryDone event) but we will still assume we should schedule the data in specific slots (using postCmd with the Tx set to trigger at specific times, we do not use chaining so we do a new post for each Tx), so we do that and transmits the data (26 and 27). This sometimes happends and is sortof a normal error handling case in our application. I.e. this can occur and delt with gracefully without a NO_FS occuring.

When the next RX operation completes it has the status ERROR_NO_FS (line 28). Here we do a RF_yield and posts a message to our RF Task. The RF task sleeps for a while (2ms) and then posts a new RX to occur when we expect a beacon message.

At line 29 we receive the beacon message and the code does a RF_CancelCmd on the Rx command (the same as on line 18 and 21) but for some strange reason we get a CmdCancelled event on a Tx command (that has not been posted yet) instead of the expected CmdStopped event on the Rx command.

Any suggestions on how this can be? Does this have to be an issue on my side or could this be an issue in the driver/rfc?

BR Henrik

ps. I was apparently to eager to mark this issue as resolved, should we "unresolve" it?

Hi Henrik,

Henrik Smeds said:

And it almost works as I hoped, but I still get some strange issues.

Does the device/system recover from those strange issues in reasonable time?

Henrik Smeds said:

At line 29 we receive the beacon message and the code does a RF_CancelCmd on the Rx command (the same as on line 18 and 21) but for some strange reason we get a CmdCancelled event on a Tx command (that has not been posted yet) instead of the expected CmdStopped event on the Rx command.

Any suggestions on how this can be? Does this have to be an issue on my side or could this be an issue in the driver/rfc?

RF_EventCmdCancelled means the command was cancelled before it was started (RFCC26X2.h). And if the TX cmd is passed to RF_cancelCmd (vs RX expected), then this status makes sense.

Could it be possible that the wrong RF_CmdHandle (TX) was passed to the RF_cancelCmd?

Henrik Smeds said:

ps. I was apparently to eager to mark this issue as resolved, should we "unresolve" it?

Sure, that may reflect more accurately the status of this post.

Thanks,
Toby