CC3100: Increasing payload size in https

Hi Akhi,

MAXSENDLEN is used on the socket level to control how much data can be sent in one packet. You should not modify this value, as 1460 is the max size of the underlying socket's packet.

What HTTP method are you using? There shouldn't be any limitation to sending a request with arbitrary payload size in the HTTP library. This is since you can set the expected content length on the HTTP headers, and then simply use HTTPCli_sendRequestBody() repeatedly until you have finished sending all of the data. So you'd loop through your data and send MAXSENDLEN at a time.

Here is some example code that expands on the HTTP post implementation provided with the library. This code was used by an example I developed to interface with a cloud server so you will see some extra functionality, but the basic concept should demonstrate what you should do with your HTTP application code to perform large payload requests.

int HTTPPostMethod(HTTPCli_Handle httpClient, int postSettings, IBMTestPacket* packetToPost)
{
    unsigned char *pucFileBuffer = NULL;        // Data read or to be written
    unsigned long chunkSize = MAX_BUFF_SIZE;
    long pFileHandle;               // Pointer to file handle
    SlFsFileInfo_t pFsFileInfo;
    char *FileRead = FILE_NAME;             // File to be read
    int fileSize = 0;
    char s_fileSize[15];
    int uiFileSize;
    int i;

    bool moreFlags = 0;
    long lRetVal = 0;

    pucFileBuffer = malloc(chunkSize);
        if(NULL == pucFileBuffer)
        {
            UART_PRINT("Can't Allocate Resources\r\n");
            LOOP_FOREVER();
        }

        memset(pucFileBuffer,'\0',chunkSize);
        memset(s_fileSize,'\0',15);
        /* Read file size if we are posting a file */
        if(postSettings==HTTP_POST_FLACFILE){
            /* open audio file */
            pFileHandle = sl_FsOpen((unsigned char *)FileRead,SL_FS_READ, \
                NULL);
            if(lRetVal < 0)
            {
                free(pucFileBuffer);
                ERR_PRINT(lRetVal);
                LOOP_FOREVER();
            }

            lRetVal = sl_FsGetInfo((unsigned char *)FileRead, NULL, &pFsFileInfo);
            if(lRetVal < 0)
            {
                lRetVal = sl_FsClose(pFileHandle,0,0,0);
                free(pucFileBuffer);
                ERR_PRINT(lRetVal);
                LOOP_FOREVER();
            }

            uiFileSize = (&pFsFileInfo)->Len;
            sprintf(s_fileSize, "%d", uiFileSize);
        }
        /* Get stream size from packet otherwise */
        else if((postSettings==HTTP_POST_RAW)||(postSettings==HTTP_POST_RAW_2CH)){
            uiFileSize = packetToPost->bytesInBuffer;
            sprintf(s_fileSize, "%d", packetToPost->bytesInBuffer);
        }
        HTTPCli_Field fields[7] = {
                                    {HTTPCli_FIELD_NAME_AUTHORIZATION, AUTHENTICATION},
                                    {HTTPCli_FIELD_NAME_HOST, HOST_NAME},
                                {HTTPCli_FIELD_NAME_ACCEPT, "*/*"},
                                {HTTPCli_FIELD_NAME_CONTENT_TYPE, CONTENT_TYPE},
                                {HTTPCli_FIELD_NAME_CONTENT_LENGTH, s_fileSize},
                                {HTTPCli_FIELD_NAME_EXPECT, "100-continue"},
                                    {NULL,NULL}
                            };
        if(postSettings==HTTP_POST_RAW){
            fields[3].value = CONTENT_TYPE_RAW;
        }
        else if(postSettings==HTTP_POST_RAW_2CH){
            fields[3].value = CONTENT_TYPE_RAW_2CH;
        }
    /* Set request header fields to be sent for HTTP request. */
        HTTPCli_setRequestFields(httpClient, fields);

        /* Send POST method request. */
        lRetVal = HTTPCli_sendRequest(httpClient, HTTPCli_METHOD_POST, POST_REQUEST_URI, moreFlags);
        if(lRetVal < 0)
        {
            UART_PRINT("Failed to send HTTP POST request header.\n\r");
            return lRetVal;
        }
        if(postSettings==HTTP_POST_FLACFILE){
        for (i = 0; i < uiFileSize; i = i + fileSize){

            fileSize = sl_FsRead(pFileHandle, i,  pucFileBuffer, chunkSize);
            if(lRetVal < 0)
            {
                lRetVal = sl_FsClose(pFileHandle,0,0,0);
                free(pucFileBuffer);
                ERR_PRINT(lRetVal);
                LOOP_FOREVER();
            }

            /* Send POST data/body */
            lRetVal = HTTPCli_sendRequestBody(httpClient, pucFileBuffer, fileSize);

            if(lRetVal < 0)
            {
                UART_PRINT("Failed to send HTTP POST request body.\n\r");
                    return lRetVal;
            }
        }
        }
        else if((postSettings==HTTP_POST_RAW)||(postSettings==HTTP_POST_RAW_2CH)){
            lRetVal = HTTPCli_sendRequestBody(httpClient, packetToPost->bufferPointer, packetToPost->bytesInBuffer);

            if(lRetVal < 0)
            {
                UART_PRINT("Failed to send HTTP POST request body.\n\r");
                return lRetVal;
            }
        }

        free(pucFileBuffer);
        if(postSettings==HTTP_POST_FLACFILE){
        lRetVal = sl_FsClose(pFileHandle,0,0,0);
        }
        lRetVal = readResponse(httpClient);
        if(lRetVal < 0)
        {
            UART_PRINT("Failed to read response.\n\r");
            return lRetVal;
        }
#if ENABLE_DEBUG_OUTPUT
        UART_PRINT("First Response Received.Waiting a few seconds...\n\r");
#endif
        //Task_sleep(8000);
        int failcounter = 0;
        lRetVal = readResponse(httpClient);
        while(lRetVal<0&&failcounter<HTTP_RETRY_LIMIT){
            //UART_PRINT("Failed to read response.\n\r");
            failcounter++;
            lRetVal = readResponse(httpClient);
        }
//        if(lRetVal < 0)
//        {
//            UART_PRINT("Failed to read response.\n\r");
//            return lRetVal;
//        }

        return lRetVal;
}

Let me know if you need more clarification or have further questions on using the HTTP client library.

Regards,

Michael

Hi michael,

I am using mqtt using httpvli library. For that, I need to send more than 1460B of data. This is my requirement. MQTT is not like HTTP where you can dump as many data of 1460B. There we need to send 1 time and we want this size to be more, something around 10KB. I can try changing this value if you help me how to modify this hardcoded field.

Also, I am using AWS IOT for mqtt where 1 message size is 128KN at maximum. That's the reason I want to increase it more.

Thanks

Hi Akhi,

I am aware of the limitations of HTTP, and how you typically need to provide all of the data for a application interaction in a single request.

Now, the underlying transport layer of HTTP is TCP. TCP has the hard limitation of only 64KB in a packet, but this isn't the primary limit as lower layers such as the Wi-Fi MAC layer or the ETH layer that is used in the TCP/IP interface will have lower MTU limits. This is why you have the 1460B limit in the library, to take into account MTU limitations.

As HTTP does require large payloads, there are a variety of methods to work around the MTU limitation. For example, the code I provided above explicitly provides the content-length in the HTTP header, so that the HTTP server you are connecting to will expect more data on the same socket, and receive TCP packets while you perform HTTPCli_sendRequestBody() for the full data payload. Another method you can use is a chunked data transfer.

Can you please provide me the API documentation for the server you are using? That will allow me to see which of those workarounds will work best.

Also, if you are using MQTT over HTTP, is there a reason why you don't simply use the MQTT library provided with the CC3100 SDK?

Regards,

Michael

Hi Michael,

I am aware of the above-mentioned method to dump more data by providing the content length in the header. I was just wondering if we can increase that. 

I can try the same thing for MQTT.

For mqtt, I am using the library provided by aws sdk. There is a network interface layer provided by aws sdk where we need to provide socket layer. In that layer, I am using httpcli library's function such as connect, read_Rawbody, sendrequest, etc. 

The main reason I am using httpcli library as a network interface is that it works on both- ethernet and wifi (cc3100). As we are already using this library for both, so I used this. 

I can try the same thing with mqtt as well and will update you. The main point in mqtt is that we send something to a topic and the max data to the mqtt broker (aws iot mqtt broker) allowed is 128KB. 

https://docs.aws.amazon.com/iot/latest/developerguide/what-is-aws-iot.html

Thanks

akhi

Hi Akhi,

To clarify, are you trying to run MQTT on top of HTTP? Or are you simply using the TCP socket functions that are implemented in the HTTP library for MQTT as well?

As for socket layer limitations, this is something that the application layer, in this case the AWS MQTT library, needs to work around. This is not a limitation that is unique to the CC3100 - at a low level every network application that relies on ETH frames will need to segment packets.

Indeed, in the CC3100 MQTT client code, there are provisions to split up TX packets into parts, in order to handle the case where the network layer only supports small segments. Are you sure that this code isn't present in the AWS MQTT code?

As for increasing the amount of data per HTTP chunk, unfortunately the limit you currently have with the chunk sizes is not able to be adjusted - each TCP packet you send will have a max of 1460 bytes. I suggest you explore using the fixed content length header to using HTTP chunks.

Regards,

Michael

Hi Michael,

I am simply using TCP socket functions and TLS implemented in the httpcli library which are- 

HTTPCli_construct, HTTPCli_setRequestFields, TLS_create, HTTPCli_initSockAddr, HTTPCli_connect, HTTPCli_getSocketError

HTTPCli_sendRequestBody, HTTPCli_readRawResponseBody and few more.. There is an interface layer on asws mqtt sdk where we have to provide our socket functions. Since I am already using httpcli library for https and ftps, I have integrated this into that and it is working fine. 

The same way I am doing FTPS using the same functions.  

Right now, I am splitting the big buffer into 1024B chunks and sending it over mqtt. This is working fine now and I think it will resolve my issues. I understand that there is a limitation of 1460 and that's fine. I can now split big chunks and send it in multiples of 1024. 

Thanks