This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

RTOS/CC1310: CC1310 stops transmitting after 3-4 days

Part Number: CC1310

Tool/software: TI-RTOS

Hello, 

          In my system, one CC1310 is initiating transmission periodically every 3 minutes and it waits for 3 seconds in Rx task for receiving the data from another receiver who is always in Rx. I am using Listen before talk in Tx as my system will have many cc1310 Transmitter talking with one receiver. The problem I observed in prolonged testing is transmitter works fine for 3-4 days and it stops sending the data.  Here is my code for Tx task in transmitter :

#define CS_RETRIES_WHEN_BUSY    10      /* Number of times the CS command should run in the case where the channel is BUSY */
#define RSSI_THRESHOLD_DBM      -100     /* The channel is reported BUSY is the RSSI is above this threshold */
#define IDLE_TIME_US            500000   /* idle time between retries is 500 ms */

#define PACKET_INTERVAL_US      200000


static void txTaskFunction(UArg arg0, UArg arg1)
{
    RF_Params rfParams;

    RF_Params_init(&rfParams);

    rfParams.nInactivityTimeout = 100;

    RF_cmdPropTx.pktLen = PAYLOAD_LENGTH;
    RF_cmdPropTx.pPkt = event;
    //RF_cmdPropTx.startTrigger.triggerType = TRIG_NOW;
    //RF_cmdPropTx.startTrigger.pastTrig = 1;
    RF_cmdNop.startTrigger.triggerType = TRIG_NOW;
    RF_cmdNop.startTrigger.pastTrig = 1;

    RF_cmdNop.pNextOp = (rfc_radioOp_t*)&RF_cmdPropCs;
    RF_cmdPropCs.pNextOp = (rfc_radioOp_t*)&RF_cmdCountBranch;
    RF_cmdCountBranch.pNextOp = (rfc_radioOp_t*)&RF_cmdPropTx;
    RF_cmdCountBranch.pNextOpIfOk = (rfc_radioOp_t*)&RF_cmdPropCs;

    /* Customize the API commands with application specific defines */
    RF_cmdPropCs.rssiThr = RSSI_THRESHOLD_DBM;
    RF_cmdPropCs.csEndTime =  IDLE_TIME_US;//((rand() % 500000) + 150) * 4;                 // random waiting time to avoid sensors trying on same pattern                               //(IDLE_TIME_US + 150) * 4; /* Add some margin */
    RF_cmdCountBranch.counter = CS_RETRIES_WHEN_BUSY;

    /* Request access to the radio */
    rfHandle = RF_open(&rfObject, &RF_prop, (RF_RadioSetup*)&RF_cmdPropRadioDivSetup, &rfParams);

    /* Set the frequency */
     RF_postCmd(rfHandle, (RF_Op*)&RF_cmdFs, RF_PriorityNormal, NULL, 0);

    /* Get current time */
     time = RF_getCurrentTime();
    while(1)
    {
        Semaphore_pend(Tx, BIOS_WAIT_FOREVER);
        printf("In the tx task\r\n");

        ccm_encrypt_data(event);

        switch(retries)
        {
            case 0:
                RF_cmdPropRadioDivSetup.txPower = TX_POWER_10;                   // set power 10 dbm
                break;
            case 2:
                RF_cmdPropRadioDivSetup.txPower = TX_POWER_11;                   // set power 11 dbm
                break;
            case 3:
                RF_cmdPropRadioDivSetup.txPower = TX_POWER_12;                    //set power 12 dbm
                break;
            case 4:
                RF_cmdPropRadioDivSetup.txPower = TX_POWER_12_5;                     //set power 12.5 dbm
                break;
            case 5:
                RF_cmdPropRadioDivSetup.txPower = TX_POWER_14;                     // set power 14 dbm
                break;
            default:
                RF_cmdPropRadioDivSetup.txPower = TX_POWER_14;                     // set power 14 dbm
                break;
        }

        time += (PACKET_INTERVAL_US * 4);
        RF_cmdNop.startTime = time;

        /* Send packet */
        RF_runCmd(rfHandle, (RF_Op*)&RF_cmdNop, RF_PriorityNormal, NULL, 0);
        //RF_runCmd(rfHandle, (RF_Op*)&RF_cmdNop, RF_PriorityNormal, txcallback, IRQ_TX_DONE);
        RF_cmdNop.status = IDLE;
        RF_cmdPropCs.status = IDLE;
        RF_cmdCountBranch.status = IDLE;
        RF_cmdPropTx.status = IDLE;
        RF_cmdCountBranch.counter = CS_RETRIES_WHEN_BUSY;
        //PIN_setOutputValue(pinHandle, Board_LED0,!PIN_getOutputValue(Board_LED0));*/

        Semaphore_post(Tx);                                        //keep 2 seconds window on for receiving
        RF_yield(rfHandle);
}
}

Kindly let me know if more information is required. Is there any issue with the parameters I set in Tx task ?

Regards,

Omkar

  • Seems to me like you are experiencing some drifting in the timing setup, which eventually leads to packets not sent.
    As a first debugging step, could you try to use TRIG_ABSTIME instead of TRIG_NOW?
    With TRIG_NOW the command is executed immediately. With TRIG_ABSTIME, the main core either stays in IDLE mode and the RF core is kept on with a RAT running, waiting for the trigger time, or the RF core is powered down and powered up again at the trigger time.

  • Hello FredG,
    Thank you for your prompt reply. As you suggested, I tried using TRIG_ABSTIME, which creates problem of not sending data after first periodic interval i.e. it sends only once after 3 minutes. I am using clock object of 1 second period to trigger tx task by positng semaphore every 3 minutes. What can be the reason?

    Regards,
    Omkar
  • Ok, then this 'debugging' step didn't give us any answer to your issue (which occurs after 3-4 days...).

    You have a lot going on here, RxTask, LTB, RxTask. You should,as the next step, try to test the tasks separately (Tx, LTB) over a prolonged time to check if 1) it works? and 2)if not, identify the issue.
    I will in the meantime do a sanity check on your code here.
  • Hi, there are a few strange things I notice.

    For instance, you have the

    time = RF_getCurrentTime();

    before the semaphore_pend

    and

    time += (PACKET_INTERVAL_US * 4);

    after the semaphore pend which doesn't really make any sense. This time is not guaranteed = RF_GetCurrentTime  as you don't know when the semaphore is posted.

    It should be:

    time = RF_getCurrentTIme + (PACKET_INTERVAL_US * 4); 

    ...on every cycle.

  • Hii FredG,
    I will try this solution and will let you know about my results. Thank you for suggestion.

    Regards,
    Omkar
  • Hello Fred,

                 I have tried with the solution provided by you and observed the system but still, the result was same as transmitter stops sending data after 3-4 days. Kindly have a look if I have understood the solution well and let me know any mistakes made. Followings  are  the code snippets I wrote :

    #define CS_RETRIES_WHEN_BUSY    10      /* Number of times the CS command should run in the case where the channel is BUSY */
    #define RSSI_THRESHOLD_DBM      -100     /* The channel is reported BUSY is the RSSI is above this threshold */
    #define IDLE_TIME_US            500000   /* idle time between retries is 500 ms */
    #define PACKET_INTERVAL_US      200000

    Tx task:

    static void txTaskFunction(UArg arg0, UArg arg1)
    {
        RF_Params rfParams;
    
        RF_Params_init(&rfParams);
    
        rfParams.nInactivityTimeout = 100;
    
        RF_cmdPropTx.pktLen = PAYLOAD_LENGTH;
        RF_cmdPropTx.pPkt = event;
        //RF_cmdPropTx.startTrigger.triggerType = TRIG_NOW;
        //RF_cmdPropTx.startTrigger.pastTrig = 1;
        RF_cmdNop.startTrigger.triggerType = TRIG_NOW;
        RF_cmdNop.startTrigger.pastTrig = 1;
    
        RF_cmdNop.pNextOp = (rfc_radioOp_t*)&RF_cmdPropCs;
        RF_cmdPropCs.pNextOp = (rfc_radioOp_t*)&RF_cmdCountBranch;
        RF_cmdCountBranch.pNextOp = (rfc_radioOp_t*)&RF_cmdPropTx;
        RF_cmdCountBranch.pNextOpIfOk = (rfc_radioOp_t*)&RF_cmdPropCs;
    
        /* Customize the API commands with application specific defines */
        RF_cmdPropCs.rssiThr = RSSI_THRESHOLD_DBM;
        RF_cmdPropCs.csEndTime =  IDLE_TIME_US;//((rand() % 500000) + 150) * 4;                 // random waiting time to avoid sensors trying on same pattern                               //(IDLE_TIME_US + 150) * 4; /* Add some margin */
        RF_cmdCountBranch.counter = CS_RETRIES_WHEN_BUSY;
    
        /* Request access to the radio */
        rfHandle = RF_open(&rfObject, &RF_prop, (RF_RadioSetup*)&RF_cmdPropRadioDivSetup, &rfParams);
    
        /* Set the frequency */
         RF_postCmd(rfHandle, (RF_Op*)&RF_cmdFs, RF_PriorityNormal, NULL, 0);
    
        /* Get current time */
         //time = RF_getCurrentTime();
        while(1)
        {
            Semaphore_pend(Tx, BIOS_WAIT_FOREVER);
            printf("In the tx task\r\n");
    
            ccm_encrypt_data(event);
    
            switch(retries)
            {
                case 0:
                    RF_cmdPropRadioDivSetup.txPower = TX_POWER_8;                   // set power 10 dbm
                    break;
                case 2:
                    RF_cmdPropRadioDivSetup.txPower = TX_POWER_7;                   // set power 11 dbm
                    break;
                case 3:
                    RF_cmdPropRadioDivSetup.txPower = TX_POWER_8;                    //set power 12 dbm
                    break;
                case 4:
                    RF_cmdPropRadioDivSetup.txPower = TX_POWER_9;                     //set power 12.5 dbm
                    break;
                case 5:
                    RF_cmdPropRadioDivSetup.txPower = TX_POWER_10;                     // set power 14 dbm
                    break;
                default:
                    RF_cmdPropRadioDivSetup.txPower = TX_POWER_10;                     // set power 14 dbm
                    break;
            }
    
            time = RF_getCurrentTime() +(PACKET_INTERVAL_US * 4);
            RF_cmdNop.startTime = time;
    
            /* Send packet */
            RF_runCmd(rfHandle, (RF_Op*)&RF_cmdNop, RF_PriorityNormal, NULL, 0);
            //RF_runCmd(rfHandle, (RF_Op*)&RF_cmdNop, RF_PriorityNormal, txcallback, IRQ_TX_DONE);
            RF_cmdNop.status = IDLE;
            RF_cmdPropCs.status = IDLE;
            RF_cmdCountBranch.status = IDLE;
            RF_cmdPropTx.status = IDLE;
            RF_cmdCountBranch.counter = CS_RETRIES_WHEN_BUSY;
            //PIN_setOutputValue(pinHandle, Board_LED0,!PIN_getOutputValue(Board_LED0));*/
    
            Semaphore_post(Tx);                                        //keep 2 seconds window on for receiving
            RF_yield(rfHandle);
        }
    }

    Rx Task:

    static void rxTaskFunction(UArg arg0, UArg arg1)
     {
         RF_Params rfParams;
    
         RF_Params_init(&rfParams);
    
         rfParams.nInactivityTimeout = 100;                         // inactivity timeout to shut RF_core after all commands are done to let cc1310 go in low power mode
    
         if( RFQueue_defineQueue(&dataQueue,
                                 rxDataEntryBuffer,
                                 sizeof(rxDataEntryBuffer),
                                 NUM_DATA_ENTRIES,
                                 MAX_LENGTH + NUM_APPENDED_BYTES))
         {
             /* Failed to allocate space for all data entries */
             while(1);
         }
    
         /*- Modify CMD_PROP_RX command for application needs */
         RF_cmdPropRx.pQueue = &dataQueue;           /* Set the Data Entity queue for received data */
         RF_cmdPropRx.rxConf.bAutoFlushIgnored = 1;  /* Discard ignored packets from Rx queue */
         RF_cmdPropRx.rxConf.bAutoFlushCrcErr = 1;   /* Discard packets with CRC error from Rx queue */
         RF_cmdPropRx.maxPktLen = MAX_LENGTH;        /* Implement packet length filtering to avoid PROP_ERROR_RXBUF */
         RF_cmdPropRx.pktConf.bRepeatOk = 0;
         RF_cmdPropRx.pktConf.bRepeatNok = 0;
         RF_cmdPropRx.endTrigger.triggerType = TRIG_REL_START;
         RF_cmdPropRx.endTrigger.bEnaCmd = 0x0;
         RF_cmdPropRx.endTrigger.triggerNo = 0x0;
         RF_cmdPropRx.endTrigger.pastTrig = 0x0;
         RF_cmdPropRx.endTime =  RF_convertMsToRatTicks(3000);         // wait for 3 sec in rx task
    
         if (!rfHandle) {
             /* Request access to the radio */
             rfHandle = RF_open(&rfObject, &RF_prop, (RF_RadioSetup*)&RF_cmdPropRadioDivSetup, &rfParams);
    
             /* Set the frequency */
             RF_postCmd(rfHandle, (RF_Op*)&RF_cmdFs, RF_PriorityNormal, NULL, 0);
         }
    
         while (1) {
             // Enter RX mode and stay forever in RX
             Semaphore_pend(Tx, BIOS_WAIT_FOREVER);
             RF_runCmd(rfHandle, (RF_Op*)&RF_cmdPropRx, RF_PriorityNormal, &callback, IRQ_RX_ENTRY_DONE);
         }
    }
    void callback(RF_Handle h, RF_CmdHandle ch, RF_EventMask e)
     {
         if (e & RF_EventRxEntryDone)
         {
             /* Toggle pin to indicate RX */
             //PIN_setOutputValue(pinHandle, Board_LED1,!PIN_getOutputValue(Board_LED1));
             printf("In the rx task\r\n");
    
             /* Get current unhandled data entry */
             currentDataEntry = RFQueue_getDataEntry();
    
             /* Handle the packet data, located at &currentDataEntry->data:
              * - Length is the first byte with the current configuration
              * - Data starts from the second byte */
             packetLength      = *(uint8_t*)(&currentDataEntry->data);
             packetDataPointer = (uint8_t*)(&currentDataEntry->data + 1);
    
             /* Copy the payload + the status byte to the packet variable */
             memcpy(rx_packet, packetDataPointer, (packetLength + 1));
    
             ccm_decrypt_data(rx_packet);
                                        //
             RFQueue_nextEntry();
    
             validate_rx_data(rx_packet);
    
             RF_yield(rfHandle);
         }
         else
         {
             if(ack_flag == 0)
             {
                 retries++;
                 if (retries > RETRY_COUNT)
                 {
                     retries = 0;
                     onesec_flag = 0;
                 }else
                 {
                     switch(retries)
                     {
                         case 1:
                             retrytime= 60;                    // retry after 1 min
                             break;
                         case 2:
                             retrytime= 30;                    // retry after 30 secs
                             break;
                         case 3:
                             retrytime= 15;                     // retry after 15 secs
                             break;
                         case 4:
                             retrytime= 7;                     // retry after 7.5 secs
                             break;
                         case 5:
                             retrytime= 5;                     // retry after 5 secs
                             break;
                         default:
                             retrytime= 5;                     // retry after  5 secs
                             break;
                     }
                     onesec_flag = HEALTH_TIMEOUT - retrytime;
                 }
             }
         }
    }

    Regards,

    Omkar

  • Hello,

             Kindly help me about this issue as I am still stuck with this problem and it is critical for my product.

    Regards,

    Omkar

  • There is a chance that the calibration of the synth fails as described in the errata note. Have you taken this into consideration?
    Please see the CC1310 Errata note (http://www.ti.com/lit/swrz062) and the Error Handling section here:
    processors.wiki.ti.com/.../CC1310_rev_B_PCN_information

    BR
    Siri
  • Hello Siri,

              thank you for your answer. I will try this error handling and will let you know about the results. Just one query to ask- is there any possibility that CC1310 will get forever stuck in that do while loop Synthesizer Calibration Fail? In that case is the soft reset of CC1310 advisable?  Also, I would like to know more about the causes of calibration synth fails if any document available.

     

    Regards,

    Omkar

  • Hi Omkar

    Unfortunately I do not have any other info available regarding the synth bug other than what is described in the errata note. As far as I know, the calibration will finally succeed and we have never seen this bug preventing the synth from being calibrated if retries have been implemented. If you have a chip where you are not able to get the synth in lock, I would assume that the chip has been damaged due to ESD etc. and a reset will not fix that.

     

    BR

    Siri

     

  • Hii Siri,
    I understood this well . I will let you know about the error handling fix.
    Thank you!

    Regards,
    Omkar
  • Hello Siri,

               I applied error handling in the code as you suggested. Still, after 4 days, the device stops transmitting. Following is the code for error handling which I used: 

    RF_EventMask result= RF_runCmd(rfHandle, (RF_Op*)&RF_cmdNop, RF_PriorityNormal, NULL, 0);
            switch (((volatile RF_Op*)&RF_cmdPropTx)->status)
            {
               default:
               case PROP_DONE_OK:
                   // Success. Nothing to do
    
                   break;
               case PROP_ERROR_NO_FS:
                   // Due to CMD_FS failing during power-up, CMD_PROP_TX could not succeed.
                   // Repeat CMD_FS until it succeeds.
    
                   PIN_setOutputValue(ledPinHandle, Board_LED1,GPIO_PIN_SET);
                   //SysCtrlSystemReset();
                   do {
                       RF_runCmd(rfHandle, (RF_Op*)&RF_cmdFs, RF_PriorityNormal, NULL, 0);
                   } while (((volatile RF_Op*)&RF_cmdFs)->status == ERROR_SYNTH_PROG);
    
                   // Depending on the application, we might re-run CMD_TX here immediately
                   // or wait for the next possible slot.
                   break;
    
               case ERROR_PAST_START:
                  PIN_setOutputValue(ledPinHandle, Board_LED0,GPIO_PIN_SET);
    
               //Other cases:
            }
    
    

    I have also added error callback to take care of any error occurred :

    RF_Params_init(&rfParams);
    rfParams.pErrCb = errorCallback;
    
    void errorCallback(RF_Handle h, RF_CmdHandle ch, RF_EventMask e) {
    if ((int32_t)ch == RF_ERROR_CMDFS_SYNTH_PROG)
    {
    
    PIN_setOutputValue(ledPinHandle, Board_LED0,GPIO_PIN_SET);
    
    do {
    RF_runCmd(rfHandle, (RF_Op*)&RF_cmdFs, RF_PriorityNormal, NULL, 0);
    } while (((volatile RF_Op*)&RF_cmdFs)->status == ERROR_SYNTH_PROG);
    
    // Depending on the application, we might re-run CMD_TX here immediately
    // or wait for the next possible slot.
    
    // If CMD_FS is executed standalone, re-start CMD_FS in this function if there is time. If not, handle as in previous example
    // When CMD_FS is chained with RX/TX commands, the RX/TX is executed in parallel with this callback by the radio and will subsequently fail. Error can be handled as in previous example.
    }
    
    //SysCtrlSystemReset();
    
    
    }

    Kindly confirm the sanity of the code and let me know if I am missing something.

    To tell you details about communication process between Transmitter and Receiver, My Transmitter sends data every 3 minutes and/or any event generated on GPIO interrupt in between. Clock module of period 1 second is used to post semaphore for Tx task and the same semaphore is released for Rx task which waits for 3 seconds to receive acknowledgement of data from Receiver and exits. My receiver device is always in receiver mode and Transmitter is in low power mode when it is not sending. Kindly let me know if more details are required. 

    I am using long-range mode for communication and listen before talk to avoid interference by other devices.

    Thanks ,

    Omkar

  • Hi Omkar Inamdar:
    Have you ever been solved your problem? Now I am stucked in the problem that cc1310 stop transmitting after running for a long time..... I have no ideal now.
    Can anyone give me an advice?
  • Hello Summery,
    No. I have still not solved problem convincingly. However you can try the soultion provided by siri in her above reply. It may be related to CMD_FS fail as told by Siri. You can also have look at this post : e2e.ti.com/.../2408632

    Let us know if it helps.

    regards,
    Omkar
  • Hi Omkar

    You should not do a RF_runCmd in the callback. Please see the latest post by Richard in this thread:

    https://e2e.ti.com/support/wireless_connectivity/proprietary_sub_1_ghz_simpliciti/f/156/t/652757?tisearch=e2e-sitesearch&keymatch=%20user%3A311342#pi319532=2

    It should not be necessary to do anything in the callback as long as you handle the switch statement (unless you are not running on our SDKs but on some old TI-RTOs version).

    Are you able to get the device to fail when running the debugger so that you can figure out where it is stuck? If that is not possible, can you add some output signals that you can toggle different places in your code and then monitor them using a logic analyzer to figure out where it fails. At this time we are not even sure that the problem is the CMD_FS.

    To narrow down the problem, maybe you can make a simple application where you transmit packets continuously using your LBT algorithm. Do not wait implement any RX code. If the problem is the CMD_FS you should be able to trigger the problems after a while, and then you can verify if the re-calibration you have implemented is OK. If this never fails, It might be that the problem is somewhere else in your SW.

    BR Siri

  • Hii Siri,
    I have seen this post yesterday and applied the necessary changes as suggested by Richard that is to replace runcmd with postcmd and cancelcmd to flush any previous command in queue as said by Richard.
    I have put the LED ON indication in the error callback with RF_ERROR_CMDFS_SYNTH_PROG mask and then kept many devices sending every 10 seconds.
    I could observe the LED indication ON in one device so that may indicate that it does indeed go into error callback of CMD_FS fail.
    So, now I am monitoring with the solution provided by Richard .
    Just to ask one query, I have implemented both switch case statement as well as error callback. Will both cause any issue or one of themm will be redundant ?

    Thank you!
    Regards,
    Omkar
  • Hi Omkar
    I think that the best thing is to only handle this outside the error callback. If you are running on and old TI-RTOS version where you risk being stuck in a while(1) in the default error callback (see: processors.wiki.ti.com/.../CC1310_rev_B_PCN_information, you can simply add an empty callback that will replace the default one, and still do all the error handling outside this function.
    Please note that you are doing LBT by using a chain of commands. It is also possible that the CMD_PROP_CS fails if the CMD_FS failed so you should also check for this in the switch statement.
    BR
    Siri
  • Hii,
    I am using TI-RTOS version 2.21.0.06 and I have added error callback in Tx task but forgot to add it in Rx task. As the link suggests for my version default while(1) will be executed if no callback is registered. So do I have to register it for rx task also as we are also running CMD_FS in rx task. Wha possible actions should I take in case CMD_PROP_CS fails? I am sorry as my queries may sound naive but this problem is crucial for my product.

    Thanks & Regards,
    Omkar
  • Hello Siri,
    I had inserted LED indications in the error callbacks for CMD_FS fail but, one of my devices stopped sending data after 3 days without giving any LED indications. So, it is certain that it is getting stuck somewhere else also. I am suspecting LBT algorithms I inserted. In your previous reply you said that it may fail CMD_PROP_CS .Kindly let me know how to handle CMD_PROP_CS fail and just to mkae sure can you verify LBT parameter I set in my code given above.

    Thanks,
    Omkar
  • Hi

    The CMD_PROP_CS can end with the status codes described in table 23-152. If it returns PROP_ERROR_NO_FS, the results will be ABORT, and this is something you can check for in your application.

    Siri

  • Hello Siri,

                    I will implement this in my code and let you know. Thank you for providing support.

    Regards,

    Omkar

  • Hii Siri,
    I was digging down more in CC1310 Technical Reference Manual to find out any more reasons my device can be hanging. i find out this one in CMD_FS section -
    "The synthesizer is set up in RX mode or TX mode, depending on synthConf.bTxMode. This mode may be changed by radio operation commands when setting up RX or TX. " Default value of synthConf.bTxMode = 0 , which suggests that synthesizer starts in Rx mode but, here my device always wakes up with Tx task . I want to ask that do I have to set it up synthConf.bTxMode = 1 so that it will function well and CMD_NO_FS error can be avoided ?

    Thanks & regards,
    Omkar
  • Hello,
    Can anyone help me to resolve the issue. Still my device hangs after 2-3 days when there are multiple devices communicating with one master device.
  • Hello,
    to update this thread, I have found the reason behind hanging of my sensor. It was observed that CC1310 goes into hardware exceptions handler after 2-3 days, so I inserted my own hardware exception handler where I soft reset CC1310 and now it works fine.

    Regards,
    Omkar