CC3220SF-LAUNCHXL: CC3220SF-LAUNCHXL

Hi Snehal,

This is definitely a weird situation and I will need more information to begin resolving this. 

* Have you seen what function it hangs on?

* Are you following the watchdog example found in the sdk under drivers? If so is the watchdog handle maybe inside an interrupt and the timeout value being reset?

* Have you checked the heap information when the device gets stuck? (Run it connected to the debugger) You did say it can take days so this one might be harder to check.

Kind Regards,

Rogelio

Hi Rogelio,

        Currently we are in process on capturing the logs to monitor the stack and heap usage. There are 2 cases which are observed mostly when the device gets hanged. 

Case 1:- We have created one function to reboot the CC3220SF. We have observed that sometimes the device goes inside this function but does not reboot it, It gets stuck .The function is as follows:-

int32_t system_reboot() {

    log_info("Rebooting system");
    sl_Stop(100);
    sleep(1);
    MAP_PRCMHibernateIntervalSet(330);
    MAP_PRCMHibernateWakeupSourceEnable(PRCM_HIB_SLOW_CLK_CTR);
    MAP_PRCMHibernateEnter();
    return 0;
}

Case 2:- When we try to send MQTT data. The function is as follows:-

void mqtt_send(MqttEvent_t evt, uint16_t size, char *fmt, ...) {


    if(!g_MqttEvtQueue) {
        log_error("Mqtt Send failed - Event Queue does not exist!");
        return;
    }

    log_info("%s %d malloc %d", __FILE__, __LINE__, size);



    char *payload = (char *)malloc(size);
    add_mem_alloc(__FILE__, __LINE__, size, (void *)payload);
    if (payload == NULL) {
        log_error("OUT OF MEMORY. Could not malloc payload with size: %d", size);
        return;
    }
    MqttEventPayload_t mqttEvt;

    memset(payload, 0x00, size);

    va_list va;
    va_start (va, fmt);
    vsprintf (payload, fmt, va);
    va_end (va);

    mqttEvt.event = evt;
    mqttEvt.msgPtr = payload;
    mqttEvt.msgLen = strlen(payload);
    event_queue_emit_mqtt(&mqttEvt);
}





void * Mqtt_clientRun(void *pvParameters)
{
    MqttEventPayload_t queueElemRecv;
    struct mq_attr mqstat;
    long lRetVal;

    log_info("%d %s Starting Mqtt_clientRun", __LINE__, __func__);

    /*handling the signals from various callbacks including the push button  */
    /*prompting the client to publish a msg on PUB_TOPIC OR msg received by  */
    /*the server on enrolled topic(for which the on-board client ha enrolled)*/
    /*from a local client(will be published to the remote broker by the      */
    /*client) OR msg received by the client from the remote broker (need to  */
    /*be sent to the server to see if any local client has subscribed on the */
    /*same topic).                                                           */
    for(;; )
    {
        /*waiting for signals                                                */
        mq_receive(g_MqttEvtQueue, (char*) &queueElemRecv, sizeof(MqttEventPayload_t),
                   NULL);

        log_debug("%d Processing Mqtt Event Queue, Event = %d  %s", __LINE__, queueElemRecv.event, get_mqtt_event_str(queueElemRecv.event));


        if(mq_getattr(g_AppEvtQueue, &mqstat) == 0)
        {
            log_error("%d Queue left g_AppEvtQueue, Msgs: %d  Max: %d", __LINE__, mqstat.mq_curmsgs, mqstat.mq_maxmsg);
        }

        switch(queueElemRecv.event)
        {
        case PUBLISH_CMD_DATA: {
            log_info("PUSH Command Data %s", PUBLISH_CMD_TOPIC);
            log_info("PAYLOAD(%d): %s", queueElemRecv.msgLen, queueElemRecv.msgPtr);
            lRetVal = MQTTClient_publish(gMqttClient, (char*) PUBLISH_CMD_TOPIC, strlen((char*)PUBLISH_CMD_TOPIC),
                                         queueElemRecv.msgPtr, queueElemRecv.msgLen, MQTT_QOS_2 |
                                         ((RETAIN_ENABLE) ? MQTT_PUBLISH_RETAIN : 0));
            if(lRetVal < 0) {
                log_error("Could not PUSH to %s - Error: %d", PUBLISH_CMD_TOPIC, lRetVal);
                mqtt_handle_error(lRetVal);
            }

            free_mem_alloc(__FILE__, __LINE__, (void *)queueElemRecv.msgPtr);
            free(queueElemRecv.msgPtr);
            break;
        }
        case PUBLISH_PINGPONG_PONG: {
            log_info("PUSH Pong %s", PUBLISH_PONG_TOPIC);
            log_info("PAYLOAD(%d): %s", queueElemRecv.msgLen, queueElemRecv.msgPtr);
            lRetVal = MQTTClient_publish(gMqttClient, (char*) PUBLISH_PONG_TOPIC, strlen((char*)PUBLISH_PONG_TOPIC),
                                         queueElemRecv.msgPtr, queueElemRecv.msgLen, MQTT_QOS_2 |
                                         ((RETAIN_ENABLE) ? MQTT_PUBLISH_RETAIN : 0));
            if(lRetVal < 0) {
                log_info("Could not PUSH to %s - Error: %d", PUBLISH_PONG_TOPIC, lRetVal);
                mqtt_handle_error(lRetVal);
            }

            free_mem_alloc(__FILE__, __LINE__, (void *)queueElemRecv.msgPtr);
            free(queueElemRecv.msgPtr);
            break;
        }

        /*On-board client disconnected from remote broker, only      */
        /*local MQTT network will work                               */
        case CLIENT_CONNECT_REQ:
        {
            log_info("Processing CLIENT_CONNECT_REQ");

            int32_t res = MqttClient_start();
            if(res < 0)
            {
                log_info("%s  reconnections - %d", __func__, mqtt_init_fail_count);

                mqtt_init_fail_count++;
                sleep(1);
                if(mqtt_init_fail_count >= MAX_CONNECTION_ATTEMPT)
                {
                    log_info("Reconnection attempts exceeded");
                    radmod_tx_reset_issues();
                    system_reboot();
                }
                log_error("MQTT Client initialization failed");
                MqttEventPayload_t evt;
                evt.event = CLIENT_CONNECT_REQ;
                event_queue_emit_mqtt(&evt);
            }
            else
            {
                log_info("Processing CLIENT_CONNECT_REQ - Error %d", res);

                sys_mqtt_connection_created(); //new-change
                mqtt_init_fail_count = 0;
            }

            break;
        }
        case CLIENT_DISCONNECTED: {
            log_info("Send mqtt client connect request");
	    sys_mqtt_disconnected();	//new-change

            MqttEventPayload_t evt;
            evt.event = CLIENT_CONNECT_REQ;
            event_queue_emit_mqtt(&evt);

            break;
        }
        case THREAD_TERMINATE_REQ:
            pthread_exit(0);
            return(NULL);

        default: {
            sleep(1);
            break;
        }
        }
    }
}

Is anything wrong with the code attached here? 

Hi,

Just a quick comment.

Your restart function may not be reliable in case of stack overflow. Especially sleep(1) is very dangerous at this point. Calling sl_Stop(100) may to be problematic as well. In case simplelink driver crashed due to stack overflow.

Jan

Hi,

 Then what is the best way to reboot the device?? Can you please help me with this?

Thanks & Regards,

Snehal

Hi Snehal,

Code with hibernation is perfectly fine. NWP stop should to be used in case you expect reset from normal condition. But if you are expecting reset from fault state (like stack overflow) you should remove all unnecessary code from reset routine.

Jan

Hi Jan,

  So if I change function to the following,

int32_t system_reboot() {

    log_info("Rebooting system");
    
    MAP_PRCMHibernateIntervalSet(330);
    MAP_PRCMHibernateWakeupSourceEnable(PRCM_HIB_SLOW_CLK_CTR);
    MAP_PRCMHibernateEnter();
    return 0;
}

Is that OK?

Regards,
Snehal

Hi Snehal,

This looks OK, but even code  log_info() may to be dangerous depending how looks code inside this function.

Jan

Hi Jan,

        Okay got it. I will remove log_info as well. I have also noticed device hanging when I try to send mqtt through mqtt_send function. Is my function causing stack overflow?? 

Thanks & Regards,
Snehal

Hi Snehal,

I think it is unlikely that function log_info() can cause stuck, but for more confidence it can be reasonable remove it.

Who knows what is going on at your case. For deeper debugging of  RTOS you can use ROV. Here is nice video about debugging common application issues with TI-RTOS.

Jan