CC1354P10: RF transmit Hangs RF_runScheduleCmd() in CC1354P10

Hi Prasad,

You can make it so a command times out after a given time, as shown here: https://e2e.ti.com/support/wireless-connectivity/sub-1-ghz-group/sub-1-ghz/f/sub-1-ghz-forum/1201184/cc1352p-radio-command-chain-for-tdma/4530336?tisearch=e2e-sitesearch&keymatch=endTrigger#4530336

Regards,

Arthur

Hi Arthur,  

The IEEE 802.15.4 Transmit structure/command (struct __RFC_STRUCT rfc_CMD_IEEE_TX_s)do not have end time. or end trigger.  

How to achieve the timeout for transmit EEE 802.15.4 command?

Regards

Prasad

Hi Prasad,

In that case, I would recommend you to abort the command using RF_cancelCmd, as seen here: https://e2e.ti.com/support/wireless-connectivity/sub-1-ghz-group/sub-1-ghz/f/sub-1-ghz-forum/1236705/launchxl-cc1310-how-to-reset-cc1310-rf-core-my-program-seems-let-rf-stuck-and-can-t-use-rf_close-to-restart-it/4679445#4679445

Regards,

Arthur

Hi Arthur,  

The problem to implement  RF_cancelCmd() is, my program is a single main loop, if the code struck here, kind of main loop is struck forever.  I only have 1-msec timer running apart from main loop which provides timer events to main loop and main loops handles simple scheduler.  If i have to do this cancel cmd now, i have to do this on timer event which is 1-msec timer and  i am really hesitant to execute such a command on 1-msec timer. 

I am trying to figure out why this is struck first of all forever?  Is there a way a timeout can be provided to to transmit to come out if it can transmit with in some milliseconds?

I will try implementing  RF_cancelCmd() in my 1-msec timer interrupt to see if it works but i don't want to take that as solution to this problem. A timeout based option is best for me for Tx command. 

Regards

Prasad 

Hi Prasad,

I wonder if you have already come accross that thread: https://e2e.ti.com/support/wireless-connectivity/zigbee-thread-group/zigbee-and-thread/f/zigbee-thread-forum/980850/launchxl-cc26x2r1-rfpackettx-example-hangs-after-reconfiguration-to-ieee-802-15-4-phy/3645768#3645768

Regards,

Arthur

Hi Arthur,

I have started the original code from this sample code only.  I have tested this code for Tx and Rx packets in IEEE 802.15.4 mode for over night and then only i stated using it in my product.  But now i am struck when i implemented this in my product firmware.  I have tried multiple approaches to resolve this issue but with out any success. 

1.  i can not use RF_cancelCmd() to cancel the hanging Transmit command as i described in my original post.  RF_cancelCmd() needs RF_CmdHandle to cancel that particular command..  but the RF_runScheduleCmd() used to transmit the 802.15.4 packet do not return the RF_CmdHandle.  The RF_CmdHandle() is only sent in in the callback function which happens only after transmit command is successful completed, Since the code is hanging in sempaphore pend i don't know the RF_CmdHandle  to call the    RF_runScheduleCmd()

2.  So i decided to call RF_flushCmd() in the timer interrupt when this problem occured.  As you know, my mail loop is struck in transmit routine of RF  semaphore_pend() when the below routine is executed. I have done RF_flushCmd()  in the timer interrupt, which is the only option for me.   See the screenshot, the flush command is also hanged since the previous command is not completed and waiting for it to complete.  This is a chain., 

RF_EventMask terminationReason = RF_runScheduleCmd(
rfHandle,
(RF_Op*)&RF_cmdIeeeTx_ieee154_0,
&rfSchedParams,
rfTxCallback,
0
);

3.  Next i thought i will do RF_Close() and then reopen the RF using RF_Open() even then RF_Close(0 is also hanged since it is also waiting for the previous TX to complete.

It is mentioned in the comment of the RF_close(0 function, the below statement.  What is the mean of second statement, who has access to RF_RequestAccess() API?.  

* RF_close pends on all commands in the command queue before closing the connection.
* If a client has access to the radio by using RF_RequestAccess API, and the same client calls RF_close  then the connection to the RF driver is closed immediately without waiting for the access duration to be over.

4.  The last thing i tried is by doing following way the transmit,  used 100msec endtime for the transmit to see if the command exits after 100msec in worst-case if it hangs.  But no success with this either. 

rfSchedParams.endType = RF_EndAbs;
rfSchedParams.endTime = RF_getCurrentTime() + RF_convertMsToRatTicks(100);

RF_EventMask terminationReason = RF_runScheduleCmd(
rfHandle,
(RF_Op*)&RF_cmdIeeeTx_ieee154_0,
&rfSchedParams,
rfTxCallback,
0
);

rfSchedParams.endType = RF_EndNotSpecified;
rfSchedParams.endTime = 0;

In all this, the problem occurs only after roughly 2000 packets transfer in 30min or so. 

I badly need a way to fix this issue.  I don't think i had this problem when i ran that sample example of ieee code you shared.  I ran overnight with that code and i don't remember i saw this problem.  I used ot send 1-packet every second in that example.  I used exact same code here.but with my application which don not have any effect on the radio functionality. 

Regadrs

prasad 

Hi Prasad,

You can use RF_scheduleCmd instead of RF_runScheduleCmd.

That way, you do not have to wait for the command to finish before receiving the command handle. You can then trigger a timer by yourself to check whether the command took to long to execute, and then call RF_cancelCmd, which, depending on the arguments given, will flush all the commands.

Regards,

Arthur

Hi Arthur,

Using RF_scheduleCmd() improved but not solved the problem fully yet, let me explain step by step. 

1. My main single loop, sending transmit packet once every 50msec, see below screenshot, before transmit, i am cancelling the RX command which is waiting to receive RF packets (i am not pending for the cancel to complete here). Then transmit the packet, waiting for the transmit command to finish using RF_pendCmd() using issued RF_CmdHandle.  A new packet transmit will not occur before completing this transmit packet since it is a single loop but the radio is already in receive mode and may be receiving a packet when we are about to issue transmit. That's why i am cancelling the receive command before placing radio into Tx.  One improvement i will do later is to check if radio is in the middle of packet reception (if that is possible using your library), then i will wait for radio complete reception and then proceed with transmit, i will do this part later.  For now, it is simply cancelling the Rx command as you see in screenshot and then initiate transmit. 

2.  My timer callback is checking if the transmit is pending for more than 100msec then cancel the transmit, but it gets struck here to cancel the cmd handle we got with RF_scheduleCmd().  It is simply looping in the RFDoorbellSend() library function. The transmit handle is correct.    The program run for more than 10000 packets transmit around 500 seconds (one packet for every 50msec) before it got into this problem.  10000th packet is the one delayed by more than 100msec to transmit and i tried to cancel and it got struck here.  9999 packets got transmitted with in 100msec, so i didn't find a reason to cancel them.                                                                                                                   

Regards

prasad       

Hi Prasad,

Are you saying that the 10000th packet always fails to transmit in time? Is it showing the same behaviour if you transmit every 25 ms?

Regards,

Arthur

Hi Arthur,

No,  i just gave an example. it is not always 10000th packet.  The transmit packet hangs intermittently in the sense, some times 4000th packet some times 9922 packet.  it is random.  What i can say is it fairly works for few thousand packets before it goes to hang state. If you look at the first picture of IAR worbench in my last email right side of that picture, you see transmit count is (RFTransmitCnt) 6523.  So that is the packet number the RF_ScheduleCmd() is hanged for more than a second and then i will try to cancel that command on Timer interrupt and the RF_cancelCmd() also hangs.  I think RF_CancelCmd(0 won't execute until the previous pending RF_ScheduleCmd() is not completed.  I don't know the internal behavior of the driver code, but what i a saying is if RF_ScheduleCmd() is hanging for whatever reason, RF_cancelCmd() doesn't cancel it.  Not sure what do to here?  If you want i can create a sample code creating this issue so that you can reproduce your self. 

Regards

prasad 

Hi Prasad,

It could help with sample code for sure.

In the meantime, I remembered about that thread: https://e2e.ti.com/support/wireless-connectivity/sub-1-ghz-group/sub-1-ghz/f/sub-1-ghz-forum/1136341/cc1312r-after-aborting-rf_cmdrx-to-send-rf_cmdtx-the-rf_cmdrx-cannot-re-enabled/4449092#4449092

Can you try the actions from my latest comment on that thread? That is:

• Enable pastTrig on the tx command
• Output the PA/LNA pins (TX/RX) on a GPIO so that you can run an oscilloscope/logic capture of the failing condition.

Regards,

Arthur

Hi Arthur,

First I have enabled PA/LNA pins on GPIO's here are the results.  First my code screenshot where it got struck.  This is the same code not enabled pastTrig but just enabled TX/RX on GPIO's.  You can see number of packets transmitted and received (37594 and 46146 respectively before I hit this problem.  you see from screenshot where it got struck. scheduled transmit and waiting for the transmit to complete.

My timer callback kicks in because the tx is waiting for more than 100msec.  My timer callback tries to cancel the comamd.  I tried cancel command first and then flush all commands. the results are same.  You can see commented code below which was also tried before. 

2. When the code is struck in above state, here are the logic analyzer output. First line is LNA output and second line is PA output

I transmit 3 packets, every 250msec.  and receive 3 packets in the same 250msec.  There is one more device in my testing, which receives every packet transmitted by this node and retransmits back. 

 Let me expand 1 Tx/Rx instance, As you can see, it took 192usec approximately to switch from Rx to Tx

Let us look at the end what happened when the transmit was completely struck.  It switched from Rx to Tx in 80uSec and then struck forever. 

I checked the few instances in the waveform timing for RX to Tx switching and all the times it was taking 192usec except the last instance where we got struck in tx forever it took 80us to switch from RX to Tx which is very strange and that may explain why Tx got struck forever. 

I will send you the capture by email.  You can open the Logic analyzer capture in the Logic Software from Salae, which you can download for free from Salae website. 

Coming back to pastTrig for Tx command, we have only pastTrig for starting the Tx, not for endTrigger. But I am going to try following thing and see if that helps.  For sure it delays Tx for sometime once Rx is blocked.  Let us see if that helps. . 

1..  Use Nop command after Rx aborted to start the chain command, Nop --> CS --> Transmit to see if that works. 

 2. How to use pastTrig of startTrigger, what does it do really?

// CMD_IEEE_TX
// IEEE 802.15.4 Transmit Command
rfc_CMD_IEEE_TX_t RF_cmdIeeeTx_ieee154_0 =
{
.commandNo = 0x2C01,
.status = 0x0000,
.pNextOp = 0,
.startTime = 0x00000000,
.startTrigger.triggerType = 0x0,
.startTrigger.bEnaCmd = 0x0,
.startTrigger.triggerNo = 0x0,
.startTrigger.pastTrig = 0x0,
.condition.rule = 0x1,
.condition.nSkip = 0x0,
.txOpt.bIncludePhyHdr = 0x0,
.txOpt.bIncludeCrc = 0x0,
.txOpt.payloadLenMsb = 0x0,
.payloadLen = 0x1E,
.pPayload = 0,
.timeStamp = 0x00000000
};

Regards

Prasad 

Hi Prasad,

Thank you for the data, I will look at it today.

As of startTrigger.pastTrig, here is what is said in the Technical Reference Manual:

which is why it would be interesting to try that option to see if we see the blockage.

There is an extra point of data we can look at, and this is the RF_cmdIeeeTx_ieee154_0.status variable when a blockage occurs. What is the value there?

Regards,

Arthur

Hi Arthur,

I implemented for pastTrig for the startTrigger problem still persists

1. From the below screenshot you can see, the code i selected for the pastrigger, added 200usec before Tx to start and this functionality works from my logic analyzer screenshots

2.  Earlier between GPI0 to GPO3 transition was 200usec and now with this pastTrg addition it has gone to 500usec

3. There are also times in the logic analyzer shoss 80usec between Rx to Tx transition 

4.  Again 30usec difference came between RX to Tx transition at the end where the Tx got struct 

5. The status of the Tx command when this happened is 2. As you can see in the watch window on right. 

Regards

Prasad 

Hi Arthur one more observation from me. 

1.  The is the last successful packet screenshot before problem occurred   GPIO3 is the TX Start and GPIO is the PA enable, the PA enable is activating 6usec after Tx start is activated.  This is the case for all successful transmit. 

2. See the screenshot when the problem occurred  The Tx start is activated but the PA never got activated.  is this specific problem to CC1354 series?

Regards

Prasad 

Hi Prasad, 

Sorry for the delays on this thread. 

One observation I had was that it seems that your cancel command uses the abort graceful setting, so if there is indeed an active command that is stuck, this would simply wait for command to end. 

Please can you call the RF_cancelCmd() with the last parameter 0 instead of RF_ABORT_GRACEFULLY? 

This should allow you to cancel the command immediately.

I do have a couple of questions:

1. In your debug session, please can you check the register view, when you enter into this TX stuck case. It would be interesting to see the value of the RFCPEIFG register. This would provide the exact state of the Command and Packet Engine of the RF Core. Please note that when you are running the code, the register might not be readable, but when you hit a breakpoint, CCS/IAR should display the value. 

2. In your logic analyzer plot from before, it seems that the PA was stuck at high? In this response, the logic analyzer plots seem to indicate that the PA signal is high.

Prasad Samudrala said:

I checked the few instances in the waveform timing for RX to Tx switching and all the times it was taking 192usec except the last instance where we got struck in tx forever it took 80us to switch from RX to Tx which is very strange and that may explain why Tx got struck forever. 

But in your latest plot, you mention have an extra GPIO as TX start, and say that PA never goes high. Please can you explain what TX start in this case means?

3. I also want to make sure that you are using RF_open only once, and that the same RF_Handle is used throughout the code. Please could you confirm this.

4. If cancelCmd() does not work even without the graceful setting,  please can you try using postCmd() instead of ScheduleCmd() and see if this behavior is reproduced. postCmd() would not preempt any command if there is one running before.

Regards,

Sid

Hi Sid,

1.  If you look at all myscreenshots I shared, I have treid RF_cancelCmd() with 0 and with F_ABORT_GRACEFULLY option multiple times to confirm it.  In both cases RF_cancelCmd() never cancels the pending RF_scheduleCmd() used packet Transmit.  I would have tried both options at least 30 times in last 2 weeks. In any combination of RF_cancelCmd()  with 0 or GRACEFULLY. and also  RF_flushCmd(rfHandle, RF_CMDHANDLE_FLUSH_ALL, 0); with 0 or RF_ABORT_GRACEFULLY for all pending Cmd handles just simply hangs for ever.  Never comes out.  The only way to come out is simply watchdog to reset the process.  Two many times i did this exercise with multiple combinations, multiple delays etc with no success. 

Even I tired following few things like delayed Tx transmit by using starttrigger, but no luck

// RF_cmdIeeeTx_ieee154_0.startTrigger.triggerType = TRIG_ABSTIME;
// RF_cmdIeeeTx_ieee154_0.startTime = RF_getCurrentTime() + RF_convertUsToRatTicks(200);

I will capture the registers and send you.

2. When this issue occurs, the PA line (GPO1) is never going high, it stays low but the Transmit Start line (GPO3) goes high as expected. In my last reply, I showed, the first screenshot was successful transmit where PA line (GPO1) was high and transmit worked successfully.   In the second screen shot when this problem occurred, GPO1 (PA line) was low and never went high but transmit start line (GPO3) shows high.  The radio circuit seems to start the transmit but could not switch on PA and may be struck forever there.  Not sure exactly what happens internally in radio circuit. 

3. Look at the names of the first 3lines in my plot (Line 1- GPO0, Line 2 - GPO3 and Line 3 - GPO1),.  Ignore Line 4, which is my debug line.  Look at the meaning of those lines as per your datasheet (screenshot shown below).  GPO0 - LNA enable, GPO1 - PA Enable, GPO3 - Transmit start).  The 3 pin cpature represent them.

4.  Yes, RF_open done only once, and that the same RF_Handle was used and RF_Handle was intact during the entire test.  I made sure it never opened again and RF-Handle was not corrupted or changed. 

5.  I will try postCmd() instead of ScheduleCmd() and let you know. 

I have two actions to take from this. 

• Send you Register dump after problem occured.
• postCmd() instead of ScheduleCmd()

Will send them on Monday.

Regards

Prasad 

Hi Sid,

Here is the sample code. 

HyperBank20_1354.eww present in \CoreThermostat20\HyperBank20_1354 path.  I used IAR workbench 9.30.1 and TI SDK  simplelink_cc13xx_cc26xx_sdk_7_10_01_24 

CoreThermostat20.zip

Run this in two CC1354P10 nodes. they both would be transmitting and receiving so fast on channel 25, IEEE 802.15.4 mode..  If possible keep both of them in debugger if not at least one of them in debugger.  The LED red is blinked when packet is transmitted, and Green is blinked when received. 

In DRIV_RF_CC1354.c file, in the function void *mainRFThread(void *arg0), there are two line, enable one for one board another for another board.  Doesn't matter you keep the same but you can sniff them channel sniffer in channel 25.  See my debugger board hung in the same the place i have been telling you. 

//unsigned char data[] = "ffffeeeeddddccccbbbbaaaa9999888877776666555544443333222211110000";
unsigned char data[] = "0000999988887777666655554444333322221111aaaabbbbccccddddeeeeffff";

Sniffer output. 

Note:-  The problem occurs only if both the boards are running.  if the board you are not debugging is hung, power restart him, so that you maysee the problem in the board you are debugging.   There is not time frame when this can happen but most likely in a hour. 

Another thing i want to tell you is, if line 391 - 393 are enabled in the code file DRIV_RF_CC1354.c file, then the occurrence of the problem is very rare.  May be once in a day it may happen.   the probability of courante drops by 90% if these 3 lines are enabled. 

/*RF_cmdNop_ieee154_0.startTime = RF_getCurrentTime() + RF_convertUsToRatTicks(50);
RF_CmdHandle nopCmdHandle = RF_postCmd(rfHandle, (RF_Op*)&RF_cmdNop_ieee154_0, RF_PriorityHigh, NULL, 0);
RF_pendCmd(rfHandle, nopCmdHandle, 0);*/

Please give me some solution at the earlies, it has been a very long wait.

Regards

prasad 

Hi Prasad,

I have it set up and running. I see the green and Red LEDs flashing continuously. There was once the non debugger launchpad got sruck. But haven't managed to get the debugger one to fail, this happened once in about an hour.

1. But I see that all cancel commands are abort gracefully. I assume you have tried switching all cancels to abrupt cancels.

2. Have you tried using postCmd to post the TX command instead of scheduleCmd?

Regards,

Sid

H Sid,

The hang issue not a time basis, I got this occurred within 15min as well as in 2 hours. Make sure both devices are blinking red and green leds, if non-debugging one is struck (both LEDS are stopped blinking) restart him.  If it happens to debugging one, it is in hang state. 

1. Yes, i tried cancel() with abrupt cancels (0) too but as you see in my earlier posts, the problem still exists. abrupt or graceful cancel having the same behavior.

2.  I didn't use RF_postCmd() till now,  but today after i saw your essage, I started using postCmd() instead of RF_scheduleCmd() in this same example for transmitting packet and it seems to be going fine for last 6 hours without any hang.  It used to hang itwo or three times in both boards with RF_scheduleCmd() but with RF_postCmd() seems to be wroking well so far.  I will run it overnight and confirm you again .

txCmdhandle = RF_postCmd (rfHandle, (RF_Op*)&RF_cmdIeeeTx_ieee154_0, RF_PriorityHigh,0, RF_EventLastFGCmdDone );

I used above line instead of RF_scheduleCmd() in rfTransmit().

Even if this resolves the issue, I would still like to know why original RF_scheduleCmd() was hanging. 

Regards

prasad

Hi Prasad,

Did you see the crash again with postCmd()?

The main reason I asked you to test it with the postCmd() is that the Radio commands posted with the postCmd, do not preempt the running RF core commands, but waits for the current command to finish executing. 

You might have had a race condition which got avoided by never preempting a running RF core command. This is not an exact description of the situation, but a possible explanation of why you did not see a stuck application with postCmd().

Regards,

Sid

Hi Sid,

I ran the code with postCmd for 24 hours, i have not seen the crash. 

Regards

prasad

Hi Sid,

I am having trouble in enabling csma transmit command.  It doesn't transmit packet at all. the transmit call back reports ERROR_WRONG_BG ("FG level command not compatible with running BG level command"). I modified the code to enable csma transmit instead of regular transmit i was using earlier.  Can you please take a look at this code and suggest me why csma transmit gives error and doesn't transmit.  I referenced this code from your source files in the SDK.  It is a same old code with small change to enable CSMA transmit.  I need a CSMA based transmit to avoid packet loss. 

Are you able to reproduce the crash with RF_scheduleCmd()?  Any findings to fix that issue.  The postCmd() seems to be working ok for me now. Didn't crash so far. 

Regards

prasad

  CoreThermostat20-csma.zip

Appreciate any update on these issues Sid.   I am waiting for these issues to get resolved quickly

Hi Prasad,

I am sorry for the delay. There have been quite a few things going on. 

FG level command not compatible with running BG level command. This makes me believe that there is an active BG command running. And CSMA command scheduling is failing because the, this cannot run in concurrent to the currently running BG command. 

Looking at your code, it seems that your put the radio in receive and possibly there is an active IEEE RX command running in the BG when you are trying to schedule the csma command. An easy way to find this is check the LNA signal and also toggle an LED before you submit the csma command. We could then see if it is true that the radio is indeed in RX or not. 

Please make this test. If it is indeed the root cause of the issue, I will check with RnD what can be done. Most likely you would need to abort RX and run the csma command and then schedule the RX again.

Regards,
Sid

Hi Sid,

if you look at the sample csma code files I shared, that portioned I copied below, I made sure active RF command is stopped, waited for it to complete. and then only I put the csma transmit command.  Yes, I made sure LNA pin is low then only i started CSMA TX start is through the pin capture.  This is exactly the way i did for non-csma packet transmit as well.  But the CSMA transmit put the radio back into listen mode to detect the energy before initiating transmit. This command itslelf has to put radi oto RX --> Measure Energy--> If free transmit --> backoff and then back to Rx --> and so on.  

Active RX and Tx is the part of the CSMA command itself which is TI library is taking care. My guess is it is an issue in the library code side. Eample code is there for you to review

Regards

prasad

if ((rxCmdhandle != RF_ALLOC_ERROR) &&

Hi Prasad, 

I did speak to our radio expert on IEEE commands. I apologize for my previous answer. The issue here is that when you run the CSMA-CA command, there is no active Rx command.

The technical reference manual does provide a combination of foreground and background commands that can be run concurrently. CSMA-CA in the foreground, with no active background command causes the ERROR_WRONG_BG. This means that, you can run a CSMA command if you have an active RX command running. If you do not, then you need to run an RX command or energy detect and then the CSMA command.

https://dev.ti.com/tirex/explore/node?node=A__AO0VNcTkm37Mb0AVH5vFDQ__cc13x0_devices__coGQ502__LATEST

Please check section 25.5.4 on this document

Regards,
Sid

Hi Sid,

I enabled ED command and then issue CSMA transmit.  Then it started working. When i put the interference source, the command transmit stops as expected.   Thanks for the inputs. 

But what i observed i,  the CSMA command works only with RF_scheduleCmd().  Executing CSMA command with RF_postCmd(() then calling RF_pendCmd() to wait for the command handler returned with RF_postCmd(), simply makes RF_pendCmd() to wait for ever.  This may be due to Energy detect command is already running. Multiple commands running with RF_postCmd() doesn't work?

My worry of not using RF_scheduleCmd() is it hangs once in a while as you know, we started this thread from this issue only. Have you ever found the root cause of the original issue i started this thread for? The CSMA command with RF_scheduleCmd() puts us back into the original hang problem.  Hoping to find the root cause of this issue of RF_scheduleCmd() with test code i had provided you long back.  . 

Regards

Prasad 

Hi Prasad,

You can post multiple commands. But If you take a look at the postCmd source code, you see that it simply appends to the pending command queue, while the scheduleCmd handles FG and BG commands. I assume this is because postCmd does not preempt the running command while ScheduleCmd can preempt commands and run the FG command if needed. 

I will set it up again and run a debug session on my end to check why ScheduleCmd actually caused crashes occasionally.  

Do you see the hang even when you are scheduling the CSMA command or did it happen only with the TX command? 

Regards,
Sid

Hi Sid,

Surprisingly i didn't see Hang issue so far with CSMA command scheduling using RF_scheduleCmd().  Only thing is we have ED command running when this CSMA command is scheduled compared to earlier case where no other command is running when we issued Trasnmit command using RF_scheduleCmd().

Not sure running ED command really has a dependency.

Regards

Prasad