This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Running out of internal buffer on the CC3000.

Other Parts Discussed in Thread: CC3100
I have now come so far in my project that the main area that is hampering progress is the CC3000. I there for need som help to try and locate the reasons for it crashing.
More related information in this thread: http://e2e.ti.com/support/low_power_rf/f/851/t/312857.aspx
I have a AFE connected to the Tiva C that generates data from 16-channels, this data is then transfered from the Tiva with the help of the CC3000 over to a PC where Matlab imports the data. This is working but there are two areas of problems.
1. The CC3000 will hang and most of the time stop sending data because it no longer has any internal bufferts left.
2. The system it too slow to be able to send data from the data acquisition to the delivery in Matlab. This I should be able to solve during the following two weeks so it is not a problem I need to bring up here right now.
Back to the main problem where the bufferts aren´t cleared. I have been trying to do a stability test to see how long the system can handle to contentiously send data and right now it is around 500 sec before the system fails to clear the internal CC3000 bufferts. Below is the area and the changes I have made to socket.c:
//*****************************************************************************
//
//! HostFlowControlConsumeBuff
//!
//!  @ param  sd  socket descriptor
//!
//!  @return 0 in case there are buffers available,
//!          -1 in case of bad socket
//!          -2 if there are no free buffers present (only when
//!          SEND_NON_BLOCKING is enabled)
//!
//!  @brief  if SEND_NON_BLOCKING not define - block until have free buffer
//!          becomes available, else return immediately  with correct status
//!          regarding the buffers available.
//
//*****************************************************************************
static int
HostFlowControlConsumeBuff( int sd)
{
#define SEND_NON_BLOCKING;
#ifndef SEND_NON_BLOCKING
    /* wait in busy loop */
    do
    {
        // In case last transmission failed then we will return the last failure
        // reason here.
        // Note that the buffer will not be allocated in this case
        if (tSLInformation.slTransmitDataError != 0)
        {
            errno = tSLInformation.slTransmitDataError;
            UARTprintf( "NON: Transmission error code = %d\n" , tSLInformation.usNumberOfFreeBuffers);
            tSLInformation.slTransmitDataError = 0;
            return errno;
        }
 
        if (SOCKET_STATUS_ACTIVE != get_socket_active_status(sd))
            return -1;
    }
    while (0 == tSLInformation.usNumberOfFreeBuffers);
    {
 
       tSLInformation.usNumberOfFreeBuffers--;
       UARTprintf( "NON: Number of buffers free = %d\n" , tSLInformation.usNumberOfFreeBuffers);
        return 0;
    }
#else
 
    // In case last transmission failed then we will return the last failure
    // reason here.
    // Note that the buffer will not be allocated in this case
    if (tSLInformation. slTransmitDataError != 0)
    {
        errno = tSLInformation. slTransmitDataError ;
        UARTprintf ("Transmission error code = %d\n" , tSLInformation.usNumberOfFreeBuffers );
        tSLInformation. slTransmitDataError = 0;
        return errno;
    }
    if(SOCKET_STATUS_ACTIVE != get_socket_active_status(sd))
        return -1;
 
    //If there are no available buffers, return -2. It is recommended to use
    // select or receive to see if there is any buffer occupied with received data
    // If so, call receive() to release the buffer.
    if(0 == tSLInformation. usNumberOfFreeBuffers )
    {
        UARTprintf( "No more buffers left = %d\n" , tSLInformation.usNumberOfFreeBuffers );
        return -2;
    }
    else
    {
        tSLInformation. usNumberOfFreeBuffers --;
        UARTprintf ("Number of buffers left = %d\n" , tSLInformation.usNumberOfFreeBuffers );
        return 0;
    }
#endif
}
As you can see I am mainly printing information to the UART to be able to track when the system hangs, but I have also added a breakpoint so I can take a look at all the variables values at that point. My question is, what can I do to force a clearing of the buffers? It is stated that the system has 5 buffers and I am wondering if you can specify how big those buffers are and if there is any kind of related information that I may benefit from knowing.
According to the information given in the HostFlowControlConsumeBuff I should use receive to see if any buffer is occupied, but there is no information that I am sending over the wifi that is to be received and I have a timer running that activates the recv() every sec. But it doesn´t make any difference because the system still hangs even if I do a recv() between each sending!
  • Hi Martin,

    At present there are no ways to directly handle the internal memory of CC3000. At user level we can only wait till there are buffers available for Tx.

    Is it that when the issue happens, the available buffer event is not reported to the user by 'HCI_EVNT_DATA_UNSOL_FREE_BUFF' event? Does it always show '0' as the value for 'tSLInformation.usNumberOfFreeBuffers' in the evnt_handler.c?

    Do you see ARP activity during your test case? Can you please share the air trace if possible? At any point in your code are you making a call to netapp_arp_flush?

    Thanks & Regards,

    Raghavendra

  • Hello Raghavendra, thank you for starting a dialog!
    The variable tSLInformation.usNumberOfFreeBuffers shows the current available buffers through out the run, printing after each transfer, see below.
    Regarding the ARP command there is one ARP command sent from the CC3000to the PC during the data transfers.
    I am never calling netapp_arp_flush and I cant see the code ever requesting it. How should I use it?
     The HCI_EVNT_DATA_UNSOL_FREE_BUFF is working up till a point, it looks like it stopps working and then the buffert runs out:

    Nr: 770
    Number of buffers left = 3
    Nr: 771
    Number of buffers left = 2

    HCI Event Free buffert!

    Nr: 772
    Number of buffers left = 3
    Nr: 773
    Number of buffers left = 2

    HCI Event Free buffert!

    Nr: 774
    Number of buffers left = 3
    Nr: 775
    Number of buffers left = 2
    HCI Event Free buffert!
    Nr: 776
    Number of buffers left = 3
    Nr: 777
    Number of buffers left = 2
    Nr: 778
    Number of buffers left = 1
    Nr: 779
    Number of buffers left = 0 
    Nr: 780
    No more buffers left = 0 (No more transfers done)

    Every single time I run the transfer it stopps on number 780! Each package is 1440(1494 according to wireshark) byte big.
    How can I force the free buff event to be activated once again?
    How exactly do I do a air trace from Wireshark?
  • Hi Martin,

    Thanks for the information. I was just trying to rule out any of the ARP activities. Can you please share the capture that you already have?

    Also, do you see a change in behavior when the buffer size is reduced from 1440? Does the system recover after sometime with 'HCI_EVNT_DATA_UNSOL_FREE_BUFF' and making buffers available? Or once it hangs, is it forever?

    Thanks & Regards,

    Raghavendra

     

  • Hello Raghavendra, I have tried to inserted a Wireshark capture that contains all information from the startup to the CC3000 stops transmiting. What file ending should it have? I tried using .pcapng.

    The system responds after the transfers but will never send any more information over the CC3000. It uses HCI_EVNT_DATA_UNSOL_FREE_BUFF from time to time but it stopps working after a set of transfers.

    Below are the amount of packages sent according to tSLInformation.

    CASE 1:

    Send Packages: 781

    Released Packages: 775

    Size: 1440

     

    CASE 2:

    Send Packages: 1556

    Released Packages: 1550

    Size: 720

     

  • Here is the function that generats the dummy data that I am transferring over the Wifi:
    void AFE_64( void )  
    {
            int cch = 0;
            unsigned short AFE_Data16 = 0;     //16-bit = 2 byte in size, CC3000 can handle 720 samples maximum
     
            for ( cch = 0; cch < 720; cch++)
           {
                  ulStorage[cch] = AFE_Data16;
                  AFE_Data16++;
           }
           ulDataLength = 720*2;  //2 byte * 720 = 1440
    }
     
    Here is the transfer function I am using:
    {
                   UARTprintf ("PACKAGE SENDING\n" );
                   int c = 0;
                   // AFE DATA GENERATION:
                   AFE_64();
     
                  recvtimeout = 100;    //0.1 Sec
                  setsockopt(SOCKETServerTCP, SOL_SOCKET, SOCKOPT_RECV_TIMEOUT , &recvtimeout, sizeof (recvtimeout));
     
                   // SEND FUNCTION:
                   for (c = 0; c<2000; c++)
                  {
                         SEND_TCP = send(SOCKETClientTCP, ( unsigned short *)ulStorage, ulDataLength, 0);
                  }
                   UARTprintf (" Int Data SENT\n" );
     
                   // CLEAR THE MEM BUFFERT:
                   memset (ulStorage, 0x00, sizeof(ulStorage));
                   free (ulStorage);
          }
     
    What I am finding very interesting is that when I remove the printouts from the functions below that are located in socket.c the system doesn´t crash and doesn´t stopp. BUT it does not send all the data, only about 25% of the data is sent over the network!
    //If there are no available buffers, return -2. It is recommended to use
        // select or receive to see if there is any buffer occupied with received data
        // If so, call receive() to release the buffer.
        if(0 == tSLInformation. usNumberOfFreeBuffers )
        {
            UARTprintf( "No more buffers left = %d\n" , tSLInformation.usNumberOfFreeBuffers );
     
            //RECV_TCP = recv(SOCKETClientTCP, (unsigned short *)uiRx_Buffer, sizeof(uiRx_Buffer), 0);
            //RECV_TCP = recv(SOCKETClientTCP, (unsigned short *)ucRx_Buffer, sizeof(ucRx_Buffer), 0);
            return -2;
        }
        else
        {
            tSLInformation. usNumberOfFreeBuffers --;
            UARTprintf ("Number of buffers left = %d\n" , tSLInformation.usNumberOfFreeBuffers );
            return 0;
        }
     
    So it looks like the CC3000 is losing a large amount of data because it can´t handle the speed of received data... I got no clue why it just drops the data without giving any kind of error message or holding/pausing the system...
  • Martin,

    You have run into the same issue as a lot of us have here within the past 2 months and there seems to be no way around it.  TI says they are looking into it and will respond to a new post but the support will soon cease once they can't figure it out.  This issue presents itself in many different ways, UDP, TCP and MDNS, you always eventually run out of buffer to send data.  

    I wish we could really figure this out already.  If you can provide a debug trace to them from the debug pins that may help, I personally did not route them out from under the module on my pcb so I cant do it.  TI are you actively looking into why the buffer keeps running out?  I eventually gave up and moved on hoping they are going to figure it out and release a patch soon.  

    So TI I asked before what we could do to help troubleshoot this issue and never got a response even though many people wrote in they agreed and wanted a response.  Can we get a clear answer on what is going on with this issue?  Are the design engineers currently working on it or do you still have low level engineers trying to figure it out?  Is there anything we can do to help? Like trying different settings?  Looks like you suspect ARP, mind sharing why?  I personally have set all my socket timeouts to infinite so I can handle it myself, could this be a problem?  Are any resources being used to fix this or has everything moved over the design of the new CC3100 we keep hearing rumors about?  Is the new module going to be pin compatible with the CC3000?  

    TI, ARE WE ALL JUST WASTING OUR TIME WITH THIS MODULE OR WHAT?

    Right from the home page of the CC3000 Wiki, "The main goal of this wiki is to be the first line of support and information for all customers and users of this platform".  Where is the second line of support because I need to talk to them!  No offense to the guys helping in the forum but we need advanced support on this issue as it is a clear SHOWSTOPPER and has been kicking around for at least 2 months.

    Chad

  • Hi Martin,

      Looks like the traces are missing. But as I understand, the internal buffer is getting filled and the host has to wait until you get the unsolicited event for a long time and when it comes, the subsequent transfers result in loss of data. And the recovery does happen eventually right?  And it does not hang, right?

    Would you be able to provide us the logger logs from debug pins?http://processors.wiki.ti.com/index.php/CC3000_Logger

    Hi Chad,

     We are tracking few issues related to the memory overflow. All these issues may not have the same root cause. Hence we would either have to identify them based on the symptoms of the issue, to understand if they are similar to other issues that we have already observed. Or we request for the debug logs from the debug pins, which is a better way get the issue solved. Hence we request you to provide these debug logs to proceed with efficient debugging.

    Thanks & Regards,

    Raghavendra

  • Hi Raghavendra,

    > Hence we would either have to identify them based on the symptoms of the issue

    > to understand if they are similar to other issues that we have already observed.

    it would all be a bit easier, if one or multiple test versions of a patch are released. Then you would see what is already fixed and what is still open and must be fixed in a different way. Or add debug events if you run in certain situation (like a crash dump) which a customer can report to TI.

    Do small steps. Do transparent communication. Then a customer will understand, why it takes so long. I am still missing a (planned) release date for something new (let it be a test or even an official release). There is even no info, if there will be a release at all in near future or CC3000 is suddenly marked as NRND (Not Recommend for New Designs).

    When you do a big step and not fix all errors, do you think customers will wait again till remaining errors are fixed, after waiting for (more or less) 4 month?

    I know it is not your problem, you do the best to give support here in the forum. Many thanks for it!

    Forward it to the TI department, which is responsible for planning the releases.

    Just my 2 cents...

    Best regards,

    Martin

  • Raghavendra,

    First thanks for replying but your reply helps me 0%.  I asked many questions in the post and you could not answer any of them, all you can do is give the canned response of we need debug logs which in your words, "which is a better way get the issue solved", ok but it is not the only way.  I have asked for advanced technical support and all I get is the same bullshit.  Well I am done.  I am moving on to Bluegiga as their modules are similar in price and they actually work hopefully.

    Martin, Sorry to jump in on your post, good luck, you are going to need it, we are all going to need it.

  • Hi,

    I  posted the same issue in the thread "CC3000 sometimes takes long time to release TX buffer" 3 months ago and no answer. I don't know if TI guys do not care or they are not able to resolve the issue. I think and suggest TI should hire better engineers. I'm so surprised TI's management  team's altitude.

    Thanks

  • I will now stop working on the CC3000 since TIs level of support is not up to standard to be looked on as a possible future development partner. I will now write my report on the current project and how it has been hampered by the CC3000, I will also point out that the CC3000 should not be used in future development.
  • Instead of using TCP, I change my design to use UDP and it does solve the issues for my application. However, I do worry if TI will discontinue this product in the near future.

  • Martin, Joo Hong, Chad,

    First, we are very sorry to hear about your experince with the product and support.

    We understand and acknowledge the issues you have high-lighted on this and a couple of other threads.

    I wanted to let you know that we are analyzing these issues and working to understand the root cause.

    We plan to come back to you shortly with outcome of the analysis and next steps.

    Thank you again for high-lighting the issues, we will keep you updated on our progress and findings.

    Adnan

  • Hi Adnan,

    Has this issue been resolved? Will there be an API for forcing buffer clearing? I am running into this issue as well. I can see the amount of buffers slowly dwindle. Even after waiting for a long time I do not see the HCI_EVNT_DATA_UNSOL_FREE_BUFF.

    It seems that HCI_EVNT_DATA_UNSOL_FREE_BUFF is only fired after socket writes, however it fires less frequently than the actual sockets that are being written.

  • It also seems that the buffers are not cleared after the socket closes. So if the buffer does not free before the socket closes, then that buffer is never recovered.

  • I upgraded to driver 1.14 and since then I haven't had HostFlowControlConsumeBuff() hang forever waiting for a buffer. 

    I did then start to have failures in where it hung in hci_event_handler().  It appears to be this issue: https://e2e.ti.com/support/wireless_connectivity/f/851/p/312391/1167512 

    There a fix in my code base that was done by a previous engineer. The change is commented with "fix from forums" but without a link I can't figure out where exactly it came from. 

    Anyway, that fix was: Remove the line tSLInformation.usRxEventOpcode = usOpcode; from SimpleLinkWaitEvent() and add it to hci_command_send() just before the SpiWrite call.  This looks like:

    UINT16 hci_command_send(UINT16 usOpcode, UINT8 *pucBuff, UINT8 ucArgsLength)
    { 
    	UINT8 *stream;
    
    	stream = (pucBuff + SPI_HEADER_SIZE);
    
    	UINT8_TO_STREAM(stream, HCI_TYPE_CMND);
    	stream = UINT16_TO_STREAM(stream, usOpcode);
    	UINT8_TO_STREAM(stream, ucArgsLength);
    
    	// moved from Simplelinkwaitevent()
    	tSLInformation.usRxEventOpcode = usOpcode;
    
    	//Update the opcode of the event we will be waiting for
    	SpiWrite(pucBuff, ucArgsLength + SIMPLE_LINK_HCI_CMND_HEADER_SIZE);
    
    	return(0);
    }

  • or not...

    My image based on 1.14 can't FW update the CC3000 when programmed to a board with 1.13. It hangs forever in hci_event_handler()