CC3000 Stops Working After a Few Weeks of Operation

Hi Victor,

Is it the first write that is failing for you? From the majority of the description above, it looks lilke the Generic write is failing here. But "sSpiInformation.ulSpiState = eSPI_STATE_INITIALIZED;" is executed only in the beginning.

Whereas, one of the places where the line sSpiInformation.ulSpiState = eSPI_STATE_IDLE; is executed, is inside 'SpiTriggerRxProcessing'. You can move out of the loop 'while (sSpiInformation.ulSpiState != eSPI_STATE_IDLE);' only after completely receiving a pending response/data from the CC3000.

P.S: Modified the comment. Does not look like a missing interrupt issue. With respect to the first while loop in SpiWrite "while (sSpiInformation.ulSpiState != eSPI_STATE_IDLE);" , the above comment holds good only for state change, if a response was received. Tx issue highlighted in the next comment.

Thanks & Regards,
Raghavendra

Hi Victor,

To add to the above comment, you may also have to check the previous SPi write. Your current SPI write is waiting for the last write to complete. For some reason, your spi write is not moving to a completion. And the state is not moving to idle.

Thanks & Regards,
Raghavendra

Raghavendra,

Can you simplify your response to tell me what exactly you want me to do in order to provide more data so that you can help me solve this problem? I have ordered an 8-channel Saleae logic analyzer and a 1.8V FTDI USB-UART cable in the event that either can be helpful in this pursuit.

I want to stress to you that the unit that I'm using *was working fine* for several weeks.

Thanks,

Vic

Raghavendra,

I got my Saleae Logic Analyzer hooked up and working. Please find attached the results of running the buildtest example from the Adafruit_CC3000_Library [https://github.com/adafruit/Adafruit_CC3000_Library/tree/master/examples/buildtest].

The form wouldn't let me upload it as a VCD waveform extension, nor as CSV, so I simply renamed the VCD formatted file with a .TXT extension to make your upload facility happy...

Does this data provide any insigh? Please tell me what the next steps are.

Regards,

Vic

$timescale 1 ns $end
$scope module top $end
$var wire 1 ! CLOCK $end
$var wire 1 " MISO $end
$var wire 1 # MOSI $end
$var wire 1 $ ENABLE $end
$var wire 1 % VBAT $end
$var wire 1 & IRQ $end
$upscope $end
$enddefinitions $end
#3733973125
0!
1"
0#
1$
0%
1&
#4294268687
1#
#4294278250
0#
#4294386312
1%
#4357949187
0&
#4357969062
0$
#4358973062
1!
#4358973125
0!
0"
#4358973187
1!
#4358973250
0!
#4358973312
1!
#4358973375
0!
#4358973437
1!
#4358973500
0!
#4358973562
1!
#4358973625
0!
#4358973687
1!
#4358973750
0!
#4358973812
1!
#4358973875
0!
1"
#4358973937
1!
1#
#4358974000
0!
0"
#4358975312
1!
0#
#4358975375
0!
#4358975437
1!
#4358975500
0!
#4358975562
1!
#4358975625
0!
#4358975687
1!
#4358975750
0!
#4358975812
1!
#4358975875
0!
#4358975937
1!
#4358976000
0!
#4358976062
1!
#4358976125
0!
#4358976187
1!
#4358976250
0!
#4358977562
1!
#4358977625
0!
1"
#4358977687
1!
#4358977750
0!
#4358977812
1!
#4358977875
0!
#4358977937
1!
#4358978000
0!
#4358978062
1!
#4358978125
0!
#4358978187
1!
1#
#4358978250
0!
#4358978312
1!
0#
#4358978375
0!
#4358978437
1!
1#
#4358978500
0!
#4358979812
1!
0#
#4358979875
0!
0"
#4358979937
1!
#4358980000
0!
#4358980062
1!
#4358980125
0!
#4358980187
1!
#4358980250
0!
#4358980312
1!
#4358980375
0!
#4358980437
1!
#4358980500
0!
#4358980562
1!
#4358980625
0!
#4358980687
1!
#4358980750
0!
#4359987812
1!
#4359987875
0!
#4359987937
1!
#4359988000
0!
#4359988062
1!
#4359988125
0!
#4359988187
1!
#4359988250
0!
#4359988312
1!
#4359988375
0!
#4359988437
1!
#4359988500
0!
#4359988562
1!
#4359988625
0!
#4359988687
1!
#4359988750
0!
#4359990000
1!
#4359990062
0!
#4359990125
1!
#4359990187
0!
#4359990250
1!
#4359990312
0!
#4359990375
1!
#4359990437
0!
#4359990500
1!
#4359990562
0!
#4359990625
1!
#4359990687
0!
#4359990750
1!
#4359990812
0!
#4359990875
1!
1#
#4359990937
0!
#4359992187
1!
0#
#4359992250
0!
#4359992312
1!
#4359992375
0!
#4359992437
1!
#4359992500
0!
#4359992562
1!
#4359992625
0!
#4359992687
1!
#4359992750
0!
#4359992812
1!
#4359992875
0!
#4359992937
1!
#4359993000
0!
#4359993062
1!
#4359993125
0!
#4359994375
1!
#4359994437
0!
#4359994500
1!
1#
#4359994562
0!
#4359994625
1!
0#
#4359994687
0!
#4359994750
1!
#4359994812
0!
#4359994875
1!
#4359994937
0!
#4359995000
1!
#4359995062
0!
#4359995125
1!
#4359995187
0!
#4359995250
1!
#4359995312
0!
#4359996562
1!
#4359996625
0!
#4359996687
1!
#4359996750
0!
#4359996812
1!
#4359996875
0!
#4359996937
1!
#4359997000
0!
#4359997062
1!
#4359997125
0!
#4359997187
1!
#4359997250
0!
#4359997312
1!
#4359997375
0!
#4359997437
1!
1#
#4359997500
0!
#4359998750
1!
0#
#4359998812
0!
#4359998875
1!
#4359998937
0!
#4359999000
1!
#4359999062
0!
#4359999125
1!
#4359999187
0!
#4359999250
1!
#4359999312
0!
#4359999375
1!
#4359999437
0!
#4359999500
1!
#4359999562
0!
#4359999625
1!
#4359999687
0!
#4360001500
1$
#4360006062
1&
#4368943312
1"

Hi Victor,

Thanks for the capture. But I am unable to view anything from the attached file. Would you be able to capture it in .logicdata format?

We have to look for the states of CS, IRQ and MOSI when this issue happens. We must see if CC3000 is indeed ready to receive data(at least it reports so). If not, then what made it stop receiving data from the host.

Thanks & Regards,
Raghavendra

Raghavendra,

No problem. Attached in Saleae .logicdata format



	Data save$�@x}
���








��y^
�4�
@\���0


��y^
�4�

�
CLOCK��y^
�4�


�
E

�


MISO��y^
�4�

�
�

MOSI��y^
�4�

�
�
�

ENABLE��y^
�4�

�
�
�

VBAT��y^
�4�

�
�

	Channel 5
T��y^
�4�

�
�

IRQ��y^
�4�

�
 
�
�

	Channel 7
T

T
T
T
T






Edge Finder
T
T









State 1��������






������y^
�4�


�Z�T�1
�i�
T@x}
���
T
T

��y^
�4�
���

T
T
T
T
$�

��y^
�4�










�









�Z��

R12
��




�
�
��
�
�
�
�
�
�
��
�
�
�
�
�
�
��
�
�
�
�
�
�
��
�
�
�
�
�
�
��
�
�
�
�
�
�
��
�
�
�
�
�
�
��
�
�
�
�
�
�
��
�
�
�
�
�
�
��
�
�
�
�
�
�
��
�
�
�
�
�
�
�






�>






S�1





��������S�1

]�1

	
{�1

��1

��1
('
��1
21
��1
<;
��1
>�02
?
�02
IH
�02
SR
�02
]\
�02
gf
�02
qp
	12
{z
&12
��
C12
��
M12
��


�Z��

84
�p}






�:�


S�1�y�


��������S�1

��1



�Z��

C12
��








�.���%�-�


��?


h%"
`�


��������h%"

&"

a�1


��1

�02




�Z��

o12
��









�~





��1�


����������1


�Z��

�,"
?Ï









�,"






��������

	�Z��

�



�Z��

�12
J�






X�1�^�





��������

�Z��

�




��y^
�4���y^
�4�

��y^
�4�
��y^
�4�
��y^
�4�
��y^
�4�
��y^
�4�
��y^
�4�


��y^
�4�

��y^
�4���y^
�4�

��y^
�4�
��y^
�4�
��y^
�4�
��y^
�4�
��y^
�4�
��y^
�4�


$�


SPI


�22 serialization::archive 5 17 SaleaeSpiAnalyzer 0 8 10607357066970005166 2 10607357066970005166 1 10607357066970005166 0 10607357066970005166 3 0 8 0 0 0






Obviously, rename to remove the .txt extension in order to open the file in Logic.

Regards,

Vic

Hi Victor,

I assume that "ENABLE" that you are referring to is the Chip Select.

I see that the First Write is failing for you. I could see a difference in the way MISO is behaving here. After writing the 10bytes (4+6 bytes), and aftrer receiving correct response on MISO, the line still stays low for a considerable time. It does not go high for some time. Can you please check this?

Ideally at this stage the host should get the interrupt to receive the command complete event from CC3000. But instead MISO stays low.

Also, have you tried reflashing the patches using patch programmer?

Thanks & Regards,
Raghavendra

Hi Victor,
I noticed that your Saleae trace had SPI settings for Mode 0 (CPOL=0, CPHA=0).

For CC3000 you should use Mode 1 (CPOL=0, CPHA=1). Although for Saleae trace this does not change the values, maybe your microcontroller is also configured for Mode 0 and this change might make a difference for CC3000?

Just a thought.

Cheers,
Risto

Thanks yes, I'm sure the SPI mode is correct on the actual host device, but thanks for the tip re: the Saleae settings, I'll make sure to configure it properly for future captures.

Raghavendra,

Yes, ENABLE is the chip select signal, I will rename it accordingly in future captures.

I too found it "interesting" that the MISO line remains low for such a long time. That line is an output from the CC3000 and an input to the microcontroller though, so the only explanation I have for it is the CC3000 is driving it low (there is *nothing* else on the SPI bus, if that's where you're heading). From my vantage point, this capture is an accurate reflection of the actual behavior of the device in whatever state it is currently in.

I have not tried reflashing the patches yet, but I would be happy to do so this evening. That being said, if that solves the problem, where does that leave me? If it starts working, I will have an unreproducible latent problem. How did the firmware get "corrupted" in the first place? I can't imagine a workaround for this in production. Surely you don't expect me to ship with a patch programmer fall-back mechanism in my code for when the CC3000 goes wonky? Don't you want to try and get a handle on a root cause?

Are you calling any host driver functions  which write to the NVM? If one of these is called upon each startup you could be hitting the write cycle limit.

E.g. netapp_dhcp() or one of the nvmen_* set/write functions?

seems plausible... see a potentially related cross-posting here http://e2e.ti.com/support/low_power_rf/f/851/p/334303/1171028.aspx#1171028. I'm just using the Adafruit_CC3000_Library and using the pattern in all the examples to connect to the same network over and over again. The call sequence at the top level (above the TI host driver) is I think summarized as:

cc3000.begin()
cc3000.deleteProfiles()
cc3000.connectToAP(WLAN_SSID, WLAN_PASS, WLAN_SECURITY)
cc3000.checkDHCP()

... then post some data to a website, rinse and repeat about once or twice a minute.
I'm not in a position to access the source I'm using in an easily grep-able form right now, but it doesn't seem terribly unlikely that netapp_dhcp is getting called in that sequence, for example

Sorry I am not familiar with the adafruit library internals.

If deleteProfiles() is calling wlan_ioctl_del_profile() then its possible this may be the cause since profiles are stored in NVEM. 

I don't see why you would be calling this with any regularity. Especially if you have not called the equivalent add profile function (wlan_add_profile()). (Unless it does something else completely different in this case).

I don't really want to get into a debate about the merits of the pattern. It was an experiment on whether it was a viable usage pattern to provide long term operation and stability, given the CC3000's relative lack thereof currently. Perhaps you're right and it would work if I simply removed the deleteProfiles call from the pattern.

It's nevertheless important for me / the CC3000 community to understand the implications / constraints if the CC3000 can only connect to a network a limited number of times in its lifetime, or that you can only delete / write profiles a limited number of times, especially if some obvious usage patterns (like what I've described) might exhaust in a matter of weeks.

Is there a proof-positive way to determine that the endurance of the NVRAM is toast?

Victor Aprea said:

It's nevertheless important for me / the CC3000 community to understand the implications / constraints if the CC3000 can only connect to a network a limited number of times in its lifetime, or that you can only delete / write profiles a limited number of times, especially if some obvious usage patterns (like what I've described) might exhaust in a matter of weeks.

I have not used the profiles feature, but to my understanding it is there to permit you to store network details for commonly used networks to permit a faster connect. So you store profiles once and perform some configuration and in future power ons the CC3000 will auto connect to them. Constantly deleting them seems like an improper use case to me.

The alternative is to just call wlan_connect() each time the device starts up.

Victor Aprea said:

Is there a proof-positive way to determine that the endurance of the NVRAM is toast?

Hopefully someone else will be able to answer this, I am not too sure on what happens when the NVM burns out - if it is only the effected locations of if the whole thing can become corrupt (if there is some form of error checking etc)

Probably not a big help, but it looks like the NVRAM is a separate part inside the CC3000 module, see here an image from Adafruit.com:

Cheers,
Risto

Raghavendra, 

FYI the Patch Programmer halts before it even gets into writing any memories. So what do you think, is it bricked?

Regards,

Vic

Hi Victor,
attached you'll find appended CC3000 Patch Programmer V1.12 (Driver section) Code Composer Studio source for MSP430FR5739 LaunchPad that additionally outputs all SPI communication over the virtual UART. Just monitor this with a program (like hTerm, see here: http://www.der-hammer.info/terminal/) and look how far it gets.

Cheers,
Risto

Attachment: 6232.CC3000_Patch_V1.12_DR_debug_out.zip
P.S. While compiling turn on the optimizations, otherwise it does not fit into FRAM (right click the project, choose Properties -> Build -> MSP430 Compiler -> Optimization -> Set level to 4 and slider to 0 (size)).

Hi Victor,

Does it get stuck at the same point? Do you observe the same pattern on the logic analyzer trace?

And how are you using the patch programmer(any modifications or is it the standard one)?

Thanks & Regards,
Raghavendra

The EEPROM is separate and its bus is not even connected internally to the rest of the CC3000 module. Instead you must connect its I2C bus externally as the CC3000 datasheet instructs at the bottom of page 4:

(3) Connect SDA_EEPROM and SDA_CC3000 through a 0-Ω resistor.
(4) Connect SLC_EEPROM and SLC_CC3000 through a 0-Ω resistor.

This is good, since you can attach a digital oscilloscope or a logic analyzer to pins 27 and 28 and monitor the I2C bus. In your case if you do that you can compare the transmitted data of a good CC3000 with the bad one.

Ivor, can you clarify, are you suggesting that I should do a logic capture on the I2C lines at the 0-ohm loop-back resistors? I'm happy to do that, but I would like to understand how that might help.

Victor Aprea said:

Ivor, can you clarify, are you suggesting that I should do a logic capture on the I2C lines at the 0-ohm loop-back resistors? I'm happy to do that, but I would like to understand how that might help.

Hi Victor,

Someone suggested that you might have exhausted your EEPROM. I have been running tests to see what happens then methods that write to the EEPROM are called too many times. I already bricked two separate CC3000 modules by calling one of these methods more than 120,000 times. When this happens the CC3000 will not longer respond. You can tell that this has happened by monitoring the I2C bus.

Ivor: I assume when you exhaust the EEPROM cycles, the module is dead forever? Even after power cycling...

Victor: When you run into a "stop" situation, is the module dead forever or does it work again after power cycling?

Martin Maurer said:

Ivor: I assume when you exhaust the EEPROM cycles, the module is dead forever? Even after power cycling...

Victor: When you run into a "stop" situation, is the module dead forever or does it work again after power cycling?

Well, it is not really dead - its more like stuck internally. When I power it up it gets stuck forever in the while(1) loop in hci_event_handler() after calling wlan_start().

I found that I can un-brick it by disconnecting the SDA line of the EEPROM's I2C bus. (Simply remove the R2 0 Ohm resistor on the CC3000EM board.) In this state the CC3000 can boot, scan for open APs, connect, but it cannot send data, since it has no mac address.

I met some curious maybe in same problem。

one of my broad can‘t connnect AP，and can’t search AP。same ware others two works well。

so i have to reprogram CC3000，by update program。one time can‘t work， two time same，and about 4-5 operate。I’m lucky， it works。

but after power up about 5 minites later，it can‘t response broadcast udp，can do it only in power up right now。

so i guess maybe the model’s nvmem  volatiled。

another guess，when i set wlan_ioctl_set_connection_policy(0, 0, 1);  maybe 30-40 percent can't connect AP，but Add a wlan_ioctl_set_scan_params（）after that and enable search for 200ms，it can get 99 percent to connect。curiously，but maybe help others。

ti‘s engineer always suppose others are beginner，but theirself is really in truth。

cc3000’s firmware failure。

pony han said:

I met some curious maybe in same problem。

one of my broad can‘t connnect AP，and can’t search AP。same ware others two works well。

so i have to reprogram CC3000，by update program。one time can‘t work， two time same，and about 4-5 operate。I’m lucky， it works。

but after power up about 5 minites later，it can‘t response broadcast udp，can do it only in power up right now。

so i guess maybe the model’s nvmem  volatiled。

another guess，when i set wlan_ioctl_set_connection_policy(0, 0, 1);  maybe 30-40 percent can't connect AP，but Add a wlan_ioctl_set_scan_params（）after that and enable search for 200ms，it can get 99 percent to connect。curiously，but maybe help others。

ti‘s engineer always suppose others are beginner，but theirself is really in truth。

cc3000’s firmware failure。

These are precisely the symptoms of exhausted EEPROM. I performed tests on two separate CC3000EM boards to see what happens when an API method that writes to the EEPROM is called more that 100,000 times. They both failed after 120,000 writes. They simply could no longer boot and got stuck in the infamous while(1) loop of hci_event_handler(...).

Today I was able to resurrect one of them by simply re-applying the patches in a loop until they finally succeeded. It took about 10 minutes, but it finally worked. However as soon as I called the method that caused the EEPROM to fail, the CC3000MOD was broken again. Then I resurrected it again using the fore-mentioned approach.

Once resurrected, if I don't call the API method that writes to the exhausted address of the EEPROM, the CC3000 works perfectly.

I also found that if you need to call an API method that writes to the EEPROM over and over, you can prevent the writing and exhausting of that address by disconnecting the SCL line of the I2C bus. On the CC3000EM board this is achieved by removing the R1 jumper that connects pin 29 to pin 30 of the CC3000 module. No clock - nothing can be written to the EEPROM. I also experimented cutting the SDA line, but that caused the EEPROM to become corrupted in some cases.

In order to do this programmatically, I replaced R1 with a bus switch, that allows me to disconnect the SCL line. I used Fairchild's NC7SZ66M5X:

http://www.digikey.com/product-detail/en/NC7SZ66M5X/NC7SZ66M5XCT-ND/673401

So, these are all the API methods that write to the EEPROM even if you keep calling them with the same parameters every time:

netapp_timeout_values(...)

wlan_ioctl_set_connection_policy(...)

wlan_ioctl_set_scan_params(...)

netapp_config_mac_adrress(...)

netapp_dhcp(...)

wlan_add_profile(...)

wlan_ioctl_del_profile(...)

Update: the resurrected CC3000EM board I mentioned in my previous post keeps dying after 10-20 minutes of operation and I have to keep re-patching it. Also the time it takes to successfully re-patch seems to increase.

hi Ivor:

sincerely thank you share your experience. i call these method only at beginning of power up, but UDP fails same,and can't fix it. TI‘s solution kicks engineer into the HELL,  TI focus only money、money、and money，don't take care about rubust of the firmware。i tell my leader and boss its fault, i do my duty, they choice it.

Hello,

It appears that we are also suffering from this same problem. We have had a unit running in the field for almost one month when it suddenly stopped working; it would no longer connect to the access point it had been connecting to since installation.

On interrogation of the EEPROM, we found that there were 32 bytes of (assumed) corrupted data in each successive 128-byte block of the WLAN configuration area. This same corruption was also seen in the WLAN config shadow area.

The corrupted areas included the location of the AP’s SSID, so the unit could not possibly connect in this situation.

I have attached a copy of the NVRAM content of the failing unit (nvram_fail.bin.txt) and a good unit, whose configuration profiles have been flushed, for comparison (nvram_good_after_wifi_flush.bin.txt). I’ve also added a snapshot of the comparison between the two to show the 32-byte blocks of corruption.

We have not yet run any further tests on the CC3000 in case any important data is lost from the EEPROM.

We are using patch SP 1.24 with API Version 1.11.

Is there any information I can provide that will help diagnose this problem further?

During this one month of operation have you called any of the following methods more than 100,000 times:

netapp_timeout_values(...)
wlan_ioctl_set_connection_policy(...)
wlan_ioctl_set_scan_params(...)
netapp_config_mac_adrress(...)
netapp_dhcp(...)
wlan_add_profile(...)
wlan_ioctl_del_profile(...)

Ivor Sargoytchev said:

During this one month of operation have you called any of the following methods more than 100,000 times:

netapp_timeout_values(...)
wlan_ioctl_set_connection_policy(...)
wlan_ioctl_set_scan_params(...)
netapp_config_mac_adrress(...)
netapp_dhcp(...)
wlan_add_profile(...)
wlan_ioctl_del_profile(...)

Before I read your previous post, Ivor, I would have said that we were not writing to the NVRAM very often at all, but now I am thinking differently. I’m setting up a test now that will monitor the I2C bus to find out how often the NVRAM is being written, particularly if the access point gets switched off. I’ll post the results when we have them.

Thanks.

You will not be able to tell if there is an EEPROM write by monitoring the I2C bus. I tried that myself. There is waaaaay too much traffic on the bus and the EEPROM has a dedicated write pin, which you cannot monitor. Also you will see a lot of writes on the bus, which are just commands being sent to the EEPROM, not actual memory writes.

The best way is to look at your code and figure out of you are calling any of the functions that I listed.