accept() stop working after a while

I have a similar issue. I am routinely accepting and processing connections on a TCP port. After some fluctuating amount of time (1/2 hour to 1/2 day) , the accept function will simply start always returning SOC_IN_PROGRESS (-2) even though I see multiple attempts by devices to establish a TCP connection with the CC3000. It never recovers and it never indicates a problem with the listening socket. Even if I close the listening socket and open a new one, the new port continues to ignore connection requests.The CC3000 still broadcasts the UDP messages my application periodically transmits though, so it is not as if it is completely dead.

I am using the latest patch to the CC3000 (1.24) and driver library (v1.11.1). However, I have seen this issue on the previous patch and driver library as well.

WireShark capture:

4150.WireShark capture of CC3000 ignoring TCP connection requests.pdf

Hi,

I would just want to understand more about thids. Can you please tell me if you tried this with SOCKOPT_ACCEPT_NONBLOCK set to '1' or '0'?

Thanks & Regards,

Raghavendra

I tried only with SOCKOPT_ACCEPT_NONBLOCK set to '0'.

I poll accept() so I would like to be nonblocking.

Yes, my listening socket has been configured with the SOCKOPT_ACCEPT_NONBLOCK option set to SOCK_ON (0).

Hi,

Sorry for the delay in response. I tried to reproduce this at my end, but I could not recreate this issue. May be there is something different in what we are doing.

Can you please let me know how you could reproduce this issue? Approximately how long does it take in the accept loop to reproduce this issue?

If possible, can you please capture the firmware logs by following the below link? (http://processors.wiki.ti.com/index.php/CC3000_Logger)

Thanks & Regards,

Raghavendra

Raghavendra,

The conditions leading to this issue in my setup are as follows.

I have a web server running on my platform that will receive HTTP request (establishing a TCP connection in the process), service it and then close the connection. This occurs approximately every 5 seconds.

I also am periodically transmitting a UDP message (which is unaffected by the problems with the TCP listen socket used by the HTTP server).

The listening socket will begin ignoring TCP connection requests after around 1/2 hour. After that, I never see the listen socket report a new TCP connection being established (even if I close the listening socket and open a new one). I've let this run for some time (> 1hr) and it never appears to recover. In the meantime,  I have not problem sending my periodic UDP beacon messages.

So to replicate it, I would suggest repetitively opening and closing TCP connections to the same listening port.. You might occasionally broadcast a UDP packet as well.

If I can find serial communication hardware that will allow me to capture the firmware logs, I will try to gather this for you. However, this is so repeatable on my platform, I think you should be able to

Hi Christopher,

So it is TCP socket create --> receive --> Close followed by a UDP send right? Have you created this issue without involving UDP at all? I will try to reproduce this and then get back to you.

Thanks & Regards,

Raghavendra

Raghavendra,

No. The UDP runs on an independent schedule. I only mentioned it to highlight that it keeps working even when the listen/accept feature appears to "lock up".

Yes, the problem occurs even if the UDP messages are not present.

Again, the problem is that the listen socket I establish for my HTTP server simply will stop indicating any device is attempting to establish a connection with it (it simply keeps returning a SOC_IN_PROGRESS from every accept call). Please refer to my WireShark trace I provided in my first message.

Hi Christopher,

Thanks for the clarification. I shall try it out and get back to you if I need more information when I am trying to reproduce it.

Thanks & Regards,

Raghavendra

Raghavendra,

Do you have any updates on this?

Regards,

Chris

Hi Christopher,

I have tried the scenario for considerably long time, but I could not exactly reproduce the scenario that was captured in the logs above.

Below are few observations and requests:

1) I see that in your case you are trying to connect every 5 seconds in case of a failed connection attempt. I would suggest you to wait a little longer if that is the case. I can see that when the failure happens, you send a SYN and CC3000 replies back with the SYN,ACK. But the remote device does not send an ACK back and tries another connection attempt by sending SYN.

2) I think it would be helpful if you can get me the logger logs as requested above. These logs along with air traces will help us to debug faster.

3) Also, when you are executing your test cases, can you please add the prints in 'HCI_EVNT_DATA_UNSOL_FREE_BUFF' event to print the number of available buffers(tSLInformation.usNumberOfFreeBuffers).

4) Lastly, can you explain a bit more about your test setup? Is there any ARP trafiic? The logs do not show much of non-TCP traffic.

Thanks & Regards,

Raghavendra

I have tried the scenario for considerably long time, but I could not exactly reproduce the scenario that was captured in the logs above.

Could you describe in detail what you have done. Perhaps I can make recommendations that will improve your chances of replicating the problem. Please send me a log of your data traffic so that I can see how closely it matches the scenario I sent you earlier?

1) I see that in your case you are trying to connect every 5 seconds in case of a failed connection attempt. I would suggest you to wait a little longer if that is the case. I can see that when the failure happens, you send a SYN and CC3000 replies back with the SYN,ACK. But the remote device does not send an ACK back and tries another connection attempt by sending SYN.

The device is running an embedded web server. There are multiple clients in this application. There is no way to coordinate these clients, nor should it be necessary. I do not mind if the CC3000 occasionally refuses or ignores connection attempts due to limits on the number of packets it can handle. However, it should not enter a state where it simply refuses to respond to any connection attempt. This is a serious defect in the CC3000 firmware.

Yes, there are cases where TCP connection establishment fails or is interrupted. This happens in the real world. The CC3000 should recover from these situations when new connections are attempted. Perhaps this is a condition you should be trying to replicate in your tests. The happy path is not the only path.
 

2) I think it would be helpful if you can get me the logger logs as requested above. These logs along with air traces will help us to debug faster.

While I am sure it would, I do not have ready access to a serial device that can handle the voltage levels used on these pins nor do I have time to hack one together. Can you recommend a commercially available device I could use to extract this?

3) Also, when you are executing your test cases, can you please add the prints in 'HCI_EVNT_DATA_UNSOL_FREE_BUFF' event to print the number of available buffers(tSLInformation.usNumberOfFreeBuffers).

I will look into trying this when I get some time for this project.

4) Lastly, can you explain a bit more about your test setup? Is there any ARP trafiic? The logs do not show much of non-TCP traffic.

I filtered the traffic in the log I sent you to better reveal what I was seeing. Looking through the old captures there does not appear to be any ARP traffic near the time the CC3000 begins ignoring all connection requests. I have attached a new unfiltered capture of the period around the onset of the CC3000's refusal to respond to connection attempts. Note: The situation preceding the CC3000's refusal to respond to connection attempts is different in this capture than in my prior one; so there appears to be many ways to cause the CC3000 to enter this unresponsive state. Again note that UDP traffic appears unaffected by this.

4061.WireShark capture of CC3000 ignoring TCP connection requests (unfiltered, summary).pdf

0003.WireShark capture of CC3000 ignoring TCP connection requests (unfiltered, full).pdf

Thanks for continuing to look into this. The project has been effectively shelved until this issue is resolved.

- Chris

Hi Christopher,

1) Yes, it should ideally recover. I am trying to understand if this particular behavior is causing it to reach a state of no-recovery. And post that it just rejects the in-coming connections.

On my setup, I am running a TCP server on CC3000 as below: 

i) Open the TCP socket
ii) bind, listen to the port and wait to accept an incoming connect.
iii) Once connected, I will start receiving the data using 'recv', where each packet is in the order of 1K bytes.
iv) After successful reception, close the clientsocket and then close the listeningSocket.

I run step i) to iv) in a while loop. And the client also follows the same sequence.

2) With respect to level shifter, you can use FTDI TTL-232RG-VREG1V8-WE with the CC3000 module, you can refer to the below link: http://processors.wiki.ti.com/index.php/CC3000_Radio_Tool

Thanks & Regards,

Raghavendra

Raghavendra,

Thank you for your response.

To better recreate the environment in my application (and, in general, any application where the CC3000 is hosting a web server), I would invite you to make the following three changes to your test:

1. Do not close the listening socket after every transaction. It is only the client socket that should be closed after every transaction. Simply accept requests as they come in and process the transaction for the new client socket..

2. Try to have multiple clients attempting to establish a TCP connection to the CC3000 simultaneously. This is what a HTTP server is exposed to in a real-world environment and it will better simulate the conditions present in my application.

3. Write data back to the client socket and then wait for the HCI_EVENT_CC3000_CAN_SHUT_DOWN message prior to closing the client socket.

Note: I am not blocking the CC3000 library, so the accept function is polled until a client connects; at which time the client socket's request is processed and then closed. Then polling of the accept function resumes.

I will look into acquiring a FTDI TTL-232RG-VREG1V8-WE, in the meantime, I would invite you to re-run your tests with the changes I suggest above.

Regards,

Chris

Raghavendra,

�����������������������������������������������������������������������������������������������������������������������������������������������������������R+��Ҵ�Ь��Ф��+�
���PRR�
�RRZ�!�ZZRZZ��P��+Іk�+҄�
�rk��X
i���+�iJF�)��X��iR��д�
P�
д�)�ش��
ЬZҴ�*Д�#ؤ�ЄZ�!J�#k�9���R���+
P�F���XМ.��R�Ҵ)|�#�B)0�y�+�{�J�R+�Ҍ)��0�+R+6pBҍ���+#P�k�
�k�����;�k�0��z2X�����Z���)��0�i���6�(P�#�0�ЄZN���咐4�;:R�6�ҤR�

Rk�cڔФ�s�ٶ��3p����*�҄:X��+�Ґ�+.�+*ؔ���м��Ҽ
R�#��x�+�R�
P��++Z�P���В��;�xЄ>�>R��R2c0k�r;")�#Є�Ф���6����J�p{JR����Ќ
ڼR�i
��Ф+Brck��3#Є����+��Q��Ќ�k�ؤq��R��ث�acx��J؞��pА>�
�J��+�9��P�А:R���p���Z�X�ڒ9���д3�3kJ3�
�9��ٶ�kJ����;�cФ+�ZКyҐ)0��#Ќ)+XД!)Rz���+P�Z+kP���
ФJfP��дi++N��2Є+r�P������+a:���;�N
PJPҒJXPв�!��ئ+�P1Ҝ�{
�ЬB)2�
�ؔ��є�д��ڄؒZ��P��Z��Ь����N���:S����дRP�Z�ڱJҐؔ"Z�)�B��ЄBi�R;����Ф�1P+*R�"p
��Jf҄iR��;�Ҥ�:�
�i
�0)Ӿ�spJR��
��X��9Z��P+:ҤҌB��+JЄBi
+�9���)�"Ҍ!Ӿ
�~J�У�;��1�#��؜�k���Ф
�*�..�"XP�١)�*Y��~�2�г\J����!��iR���k�ZpV+�PҤP�ВPQ�s
Pؤ+���ڡ0���6R�N��k�PФ+
�X�R��J�����z��+pRҫИ��+�)��k0)P�k�+Ф)2��
��ҤФ)
�PkА��ҴB9���
)��kP9�Pk��i�д)PjЄشR
P�Ф)���p��3�P�Ғ������,����B�Ґp�Ф�P��
�
RФi�)Ф):+��RP���
�ؐ


�됐R)�9#R�P��;��Ҷ�PPR��!������+�)�PВЄ�ДZ��J�
�kPВ��)��)А�)��Z�P��Д��iж+�3kJЄ+i
)����аP�{9���)P�ڐ��Z�a)+
�ؤڤВ�X��ڬ�+��Ҥ���$�Ф؄)���PJX�+���;+���*�ؐФ{Ҥ�Ґ����Аs�Є�B��Ф�
ЖR��i�Вд�ؔ�j�P�k�N{*��R҄)�ؼ�:�#���В�!26���*

���>)�B)R��
�
)ڤ+�)��#R��N
�ҤpZ�#ФkJ��
Ф3
Ғ����
�R
�Jv)��s��i+�k���Ф9
�*X�Є�����:;�k��ZД9>P�;
�Єz�)NҴڴ�aR���ҡ0~�X+�kز�NА�XR��PҤ��X�Є�Є����*АP���)�)�ڱ�P�:����9��)Є��+��R��;
Дi
���
Ҥ)#):Ф��+���ж��#R�+�ВС�J�)R��X�0x��҄��ЫФ
��;B+��!Ф���P+Ҥ���)R+����kqN���

+��جkNФ{
��)��ŚN;v��s~k~�+v��Ґ^Ь�P�3�R:0p��;�X��
ZR��Z��Ҥ#9�Фk�)+~��)�+
{8Ф�P��R�a�!X>В�J+�RЄ)R��
R)�9�#�&���+�҄��Є��p��+���ФІ�����А�+Ҍ��J0����P�Ф����Ғ��Ь)��+�iP�i0��)�)���#Юᒥ
r�)2���+��Є���+
�)�++0�;Ф�J����q���+�J���Z���)�!�++p�;k��9�Ь��P��А;*P��~PrZ�P��k
9kش)ِ�����
����ҤrP���J�*r�P������+V�.pN�+����k�P��+�����������2�
��+
Pc*Х��v��ːҍ;v�a
�B~���А�{��++�0a*+�)Bv����
Ҭr�����i�Ф��PR��������RPZ����RP�����R����BҫX���.�Ж���Pش1�������м�Ҥ+
뒍2�
�i+Xr
Z���X+А�{ҜЩ+�����R�JP0����к���{^�{)�)#�sҴi���)ش�r�i��;Ж�JZ��;��Bz����+�xP�ЌҬ)
�F�ZZФ;
R�R���!�R�R��

P�)q~��!Pؔ�X+:R�Ф::�ؔ)Z�����И�R��9�s��RҤڔؐ;+�+�+.����Ф��
R�R+ЩJ�Ф�+Fؤ+R+>0�P+)�)R�cо��!Ҥ�+�ФJ0К��)Q�)Pz���شNP����+>й
�����)
+���Z��Фش�P:pP�X��А�2p�*ФP�Z�R+"���9s��k*�����
���)Є)������J)�ج�;P�3���)д��)��!JP�a)���k҅;ҫ����)R�)��p:2�Ф�P���+Є�PЬ�дpz��<k��Ф�xX6��R����J�Z"P�z�
+�P���+��P+
P�ў��ҶRX�R��+�P+���ж��дД�д�д�+Rд�Ҥ!�X!Z��+i)!
�!P���
дP���)P+ҤZF#���(Pk)
������*3��+
�R�Ф�+��JZ�Z�!�k�N)�3Ц�cА�����NФ��!P��c�
��NФ9�N��������д)Ҭ))��;д)��+д!��
Фi
P�)Ф+д)��+Ҽд)��)Х{���{�Ф;>
�Ф+ФФ+��)��)���k��+��+��+��+Ф+��+zЄ+Ф���+�퐭+���:Ц�
��+��
��+���+P�)X�)
Ф��+ؒ�+ҬФ)����+ж�
ж��+д)
��)P�+��+�!�)�+А���+Д)�Є��+�ж)�Ф+���+�ж�F���

Attached is the log file from the CC3000; from power-up right up until the accept function starts refusing any new connections. Please analyze this at your earliest convenience and let me know what corrective actions either I can take or that TI intends to take.

Regards,

Chris

Hi Christopher,

Thank you very much for the logs. We will analyze this and get back to you.

Thanks & Regards,
Raghavendra

Hi Christopher,

The logs look corrupted. It does not have any data. Did you pull it from Pin2, mapping the Network Stack logger pin? May be the below post will provide some direction.
http://e2e.ti.com/support/low_power_rf/f/851/p/315682/1103935.aspx#1103935

As a side note, while capturing the logs please add 'netapp_set_debug_level(0x1ff);' every time after you call wlan_start().

Thanks & Regards,
Raghavendra 

The LSR TiWi SL datasheet shows pin 2 as a no connect and is not accessible on my target platform. Pin 7 is shown as the TEST UART TX (1.8V LOGIC) pin. This is the pin used to capture the previous log file. If this incorrect, there is little I can do on my platform to acquire log files then.

How is your testing going? Have you made the suggested changes to your test.

Hi Christopher,

With respect to the TiWi SL, you can refer to the below post for the NS_UART_DBG pin details. http://e2e.ti.com/support/low_power_rf/f/851/p/252590/888052.aspx#888052

Along with this, I would request you to provide the air trace as you had provided before.

With my testing I could not hit the exact issue, with the testing that I have done so far. I had used the sample HTTP server application with 1.11 release, to match your scenario on MSP430F5529.

By the way, in your test runs do you always see a block in the 'accept' call, or do you see it blocked in the previous closesocket call?

Thanks & Regards,
Raghavendra

Raghavendra,

As I mentioned before, Pin 2 is not accessible on my platform, so I will not be able to capture the log file you seek. Also, as mentioned before, the accept function will simply start always returning SOC_IN_PROGRESS (-2) even though I see multiple attempts by devices to establish a TCP connection with the CC3000. It never recovers and it never indicates a problem with the listening socket. Even if I close the listening socket and open a new one, the new listening socket continues to ignore connection requests.

I have spent countless hours on this. While I have tried to help you solve this issue with your product, I am not TI's debug team. If your current test approach does not recreate the problem on your side, I would invite you to keep trying new variations until it does. If wireshark captures of your "test" do not look like the ones I provided earlier; if you have far less traffic or far more time between events, then I suggest you have not "matched" my scenario. This is your product.. My expectations are that you will do what it takes to fix it.