This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC3200MOD: Debugging device failing entertry into sleep

Part Number: CC3200MOD
Other Parts Discussed in Thread: CC3200

Hello,

I have an application with a CC3200 where I've added LPDS using PMF.  When first powering up, the device is entering sleep mode properly and drawing expected current.  I've been finding that in longer tests (5 hours+), the device stops entering sleep mode and draws more power.  It does not resume sleep until processor is reset.

I am trying to determine why the device fails to enter sleep so I can address the root issue.  But how can I do this?  From looking in middleware project, it looks like cc_are_irqs_pending is the function I want to print more information from.  But I can't find any documentation on the six NVIC_PENDx and NVIC_PENDx_MASK registers.  Table 2-7 in Technical Reference Manual shows Interrupt vector addresses, but I'm not sure how this relates to the bits in PENDx registers.

Any thoughts or links to further documentation would be greatly appreciated.

Kind Regards,
-Jon

  • Hi Jon,

    Make sure you are enabling wake from IRQ from NWP:
    MAP_PRCMLPDSWakeupSourceEnable(PRCM_LPDS_GPIO | PRCM_LPDS_TIMER | PRCM_LPDS_HOST_IRQ);
    Then you can handle the event and return to sleep

    -Aaron
  • Aaron,

    Thanks for your advice, I am excited to try your suggestion. 

    But I'm not sure where to put it.  I do not see that function called anywhere in the RTOS idle_profile example, it only appears in idle_profile_nonos.  Seems like the RTOS project uses "lp3p0_pm_ops->set_up_wkup_srcs = cc_set_up_wkup_srcs;"

    Where do you suggest I put this line?  Is it okay to simply call it once after platform_init()?

    -Jon

  • I've gone ahead and put Aaron's suggested line into my code, just after platform_init, and it seems to be a great fix.  I've had a test running for about 12 hours, and not only does it look like device has remained in low power state, but I have not seen any disconnects (before adding this line I'd see at least 1 disconnect/reconnect every hour).

    Thanks Again Aaron.  If you have any thoughts on why this line has improved connection stability of my device I'd be interested to hear it.

    -Jon

  • Aaron,

    It looks like I talked too soon in thinking this issue was solved. Although that 21 hour test showed promising results, I have run further testing and have found the issue still remains.

    It's interesting to note that this always seems to correlate with a "[WLAN EVENT]Device disconnected" and "[WLAN EVENT] STA Connected".  I have seen device go through this disconnect/reconnect and still return to low power, so it's not a direct correlation.  But once it stops entering sleep, it does not enter sleep again until processor is reset.  Also, I should mention that every time this disconnect/reconnect happens, when I reconnect with AWS I find a "[SOCK ERROR] -close socket (83) operation remote side down from secure to unsecure". (83 is not always 83, but is within range 80-86).  This does not seem to affect device operation, but I'm not sure if it could be contributing to intermittent low power operation or not.

    Do you have any other thoughts on what this may be?
    Is there any reason I should have to call the WakeupSourceEnable function multiple times?
    Should I go back to the original idea of modifying middleware to add debug messages to help figure out what is prevent device from entering sleep?

    I look forward to your advice,
    -Jon

  • Today I tried upgrading my SDK to see if that would help (changing CC3200_SDK_ROOT from 1.2.0 to 1.3.0). I'm finding that with this new SDK, the device only enters sleep when not connected to the network. Once it gains network connection, it does not enter sleep anymore. It's worth noting that I did check to confirm the CC3200_SDK_ROOT is used when locating the middleware.a library.

    Reverting my CC3200_SDK_ROOT back to 1.2.0 shows the intermittent issue as before (sleep works for a little while, but eventually a WiFi disconnect/reconnect gets it into some mode where sleep doesn't work).

    Hopefully this is a good additional clue for you to help me debug this issue. I'm happy to make modifications to the middleware.a library to help debug the situation, but really need help from you to know what to look for.

    Thank you,
    -Jon
  • Hi Jon,

    Make sure you are handling all of your messages, including a small timeout (to handle them when they come one after another). Only exit the message poll loop when there are no messages from any source

    -Aaron
  • Aaron, can you clarify what you mean by message poll loop?  I'm not sure what layer of the application you're referring to, I'd greatly appreciate if you can refer to specific function calls/API.

    Thank you,
    -Jon

  • Aaron,

    Is sl_Recv() the function you are talking about? My code does call this every 700ms, and in most cases I find the return value of 0 bytes received. I do not see any difference in the return of this function before/after the sleep issue occurs.
  • Thank you Aaron.

    I'm not sure I understand though, I've only create 1 msgQ in my application, and my while(1) loop is constantly polling that function. For reference, this is my main loop:

    while(1){
    osi_MsgQRead(&msgQ_wakeSource, &queueMsg, OSI_WAIT_FOREVER);

    switch (queueMsg){
    case WAKE_SOURCE_TIMER:
    applicationAPI_mainLoop(); //Run loop every 100ms
    break;

    case WAKE_SOURCE_GPIO:
    DEBUG_PRINT("gpio wakeup");
    //not currently using this for anything
    break;

    default:
    DEBUG_PRINT_WARN("unknown wakeup %d", queueMsg);
    break;
    }
    }


    The only reason I am using this msgQ is to allow for sleep using RTC clock. It's also worth noting I only have 1 task in my program, so the program is only doing the above loop.
  • HI Jon,

    Are you running the device in AP mode? Note that the power is going to be significantly higher in AP mode.

    -Aaron
  • Aaron,

    We are using AP mode for part of the application, and I understand that requires more power.  But the test I'm performing is only using STA mode, as that is where device will spend the majority of its time.  I am positive that the device isn't switching to AP mode during my testing.

    When I test my code on the Launchpad, I find <1mA when operating normally in STA mode.  When the issue occurs, I find ~10mA.  When I switch to AP mode, I find about 63mA.  (Note all these numbers refer to base current, not including current spikes).

    Is there any polling and/or debug prints I can put into my code to help us figure out the root cause of why the device won't go back to sleep after disconnect/reconnect?

    -Jon

  • Aaron,

    Do you have any thoughts? The only item I can think of is to force reset the network co-processor (sl_stop, sl_start) when case SL_WLAN_DISCONNECT_EVENT occurs in SimpleLinkWlanEventHandler. This seems like a very nasty hack so I'd prefer to try other methods first.

    Looking forward to any thoughts you may have. Please let me know if you'd like me to post additional sections of my code.

    -Jon
  • Aaron,

    I went ahead and tested restarting network coprocessor upon receiving WiFi disconnect (using sl_stop(); sl_start();). It does seem to recover to low power, but I do find some SSL read issues that my mitigation logic has to fix (which it does, but it takes several seconds).

    I'm also concerned of calling sl_stop and sl_start during normal operation as I've seen the CC3200 stall with these commands before.

    Again, I think this is a very nasty hack to solve the problem, and I would appreciate suggestions on other things I might be able to try. Are there any commands to forcefully ask network processor for anything it might be holding onto that would prevent re-entering sleep?

    Thank you,
    -Jon
  • Hi Jon,

    I agree that calling sl_Stop, sl_Start is not a good workaround. however, there should not be any issue in doing this. Perhaps sometimes sl_Stop can return -6, where you will need to call it again to successfully stop it.

    When you get disconnected from the access point, are you still getting 10 mA even after you have reconnected to the AP? The current should be higher when the device is actively trying to connect

    -Aaron
  • Thanks Aaron.

    The issue is indeed that the 10mA remains after re-connection. Note the disconnect/reconnect messages come within 1s of each other. I am not sure what causes this disconnect reconnect...this occurs both with Launchpad and my custom hardware sitting untouched on my lab bench. Sometimes I'll find hours of operation without any disconnect/reconnect, sometimes I'll find several disconnect/reconnects within an hour.

    -Jon
  • Aaron,

    Do you have any further suggestions I can try? Would you like me to post more of my code? Can you recommend some debug message placements to give you more information?

    -Jon
  • Have you tried running the idle_profile example on your board? What current does it give when a disconnect/connect happens?

    -Aaron
  • Yes, I have run the idle_profile example, but it never seems to experience the random disconnect/connects.  I've been running a test since your message yesterday and have not seen any disconnects and current remains low.

    I'd really like to add some debug messages into my code to help us understand the issue and/or show you additional code that might help uncover the issue.  Do you have any thoughts?

    -Jon

  • Aaron,

    This issue is becoming critical as we are ramping up a manufacturer to produce a large quantity of devices. The mechanics, hardware, and firmware features are all implemented and ready for production. We need to address this intermittent sleep issue to avoid having devices not reach the advertised battery life; this extra 10mA draw is very significant to our overall application.

    What can we do to get this issue resolved? Can you suggest where to place debug messages to help understand the root cause? Can we provide you with the source code prepared for use on Launchpad so you can replicate the issue we see?

    -Jon
  • I've noticed that my project properties is prioritizing the OS version of simplelink.a rather than the PM_Framework version of simplelink.a. The idle_profile uses the PM_Framework version, so I attempted switching this, but I find error when calling sl_start (it returns -2004). idle_profile does not have this issue using the PM_Framework version.

    Do you think this could contribute to the issue of not returning to sleep after WiFi disconnect/reconnect? If so, any thoughts on why sl_start has issue in my project and not in idle_profile?

    -Jon
  • Is this the first sl_Start, or is other stuff happening before it?

    and yes you have to link to the PM framework version of simplelink driver to use PM framework
  • This is the first call to sl_Start. Before sl_Start my application looks very similar to idle_profile example, although I'm only using 1 task and idle_profile is using two.

    Note that when I use simplelink.a from OS, there is no issue with sl_Start and LPDS works initially (until the device comes across WiFi disconnect/reconnect).

    Also note that idle_profile example does use simplelink.a from PM_Framework, and it does not experience the sl_Start issue.

    Any thoughts on why sl_Start may be failing? What does return value -2004 mean?
  • I have resolved the issue with PM_Framework's simplelink.a. But it does not fix my root issue; I am still seeing excessive current after WiFi disconnect/reconnect (approximately 10mA increase).

    Please suggest what I can do to debug what is keeping the processor from entering LPDS after disconnect/reconnect.
  • If you are not using the OS version, you manually should re-enter LPDS in your main loop when execution resumes from a wake event (e.g disconnect or connect event)

  • I don't understand your response here.  I'm using FreeRTOS just as idle_profile does. 

    Just as in idle_profile example, my call to go to sleep looks like this:
    void vApplicationIdleHook( void )
    {
        cc_idle_task_pm();
    }

    It's worth noting that I have used test code to prove when the issue occurs this code is still being executed.  This shows it is something within cc_idle_task_pm that is not letting processor go to full LPDS.

    -Jon

  • I've recently been testing with putting a delay after disconnection and found promising results, but am having new issues.

    Recall my issue of device not entering low power only occurs after WLAN has disconnect/reconnect. I previously mentioned this happens quickly (~0.2s). It is worth noting my device is defaulted to enable autoconnect.

    At first, I attempted to disable autoconnect for a little bit after getting a disconnect event. I tried doing this directly in the SL_WLAN_DISCONNECT_EVENT event of SimpleLinkWlanEventHandler (calling sl_WlanPolicySet to change CONNECTION_POLICY). But device always stalled when calling sl_WlanPolicySet...do you know what's going on here, or how I can do this safely?

    Next, I put in a simply 2s delay in the same place (SL_WLAN_DISCONNECT_EVENT event of SimpleLinkWlanEventHandler). I've seen several WLAN disconnect/stall/reconnects, and it's been going back to low power every time! But I'm finding a much higher error rate in connection with AWS server, so I do not think this blocking delay inside SimpleLinkWlanEventHandler is a good solution.

    I think overall it makes sense to delay connection after a disconnect...perhaps something internal to the CC3200 is not able to react to the disconnect before the reconnect occurs, and that's causing the sleep issue. But I'd greatly appreciate your insight on how to cleanly and safely delay reconnecting to WiFi after a disconnect event. Please provide your thoughts on how to accomplish this.

    As always, if there is any additional information I can provide please let me know.

    -Jon
  • I should also mention that I'm printing out 'reason_code' during disconnect and always find return value '7' for this random disconnects. What does reason_code '7' mean?
    (reason code is found in SlWlanEvent_t->SlWlanEventData_u->slWlanConnectAsyncResponse_t->reason_code).