MSP432E401Y: JSON parsing issue in corner case.

Giuseppe Cantavenera

Expert 1060 points

Part Number: MSP432E401Y

Hello,

We are using the JSON lib included in the SDK and we see a property parsing issue.

It was possible to reproduce it on the EVM board reworking the example included in the MSP432E401Y SDK as well.

Basically, we have:

1- a template

2- a working piece of JSON that is parsed correctly by the library

3- a non working piece of the JSON with the same semantics, but with a property just moved 2 lines down at the same tree level.

When we use 3 instead of 2 the same property reads as random value rather than the expected one.

The other values seem to read as expected instead.

Here below the template, the working JSON, the same non working JSON, and the output on the UART.

--> the property that holds the random value in 3 but not in 2 is "\"wifi\".\"WiFiOn\"".

We can provide the full .c rewoked example if needed.

Any suggestion? Could this be a corner case bug in the library?

Thanks,

Giuseppe

------------------------------------------------------------------------------------------------------------------------

#define EXAMPLE_TEMPLATE                                          \
"{                                                          "\
"   \"ethIPv4Conf\": {                                      "\
"                       \"ipv4Addr\"        : string,       "\
"                       \"ipv4Netmask\"         : string,   "\
"                       \"ipv4Gateway\"     : string,       "\
"                       \"ipv4socketTimeout\"   : uint32,    "\
"                       \"DHCPenable\"      : boolean      "\
"                   },                                      "\
"                                                            "\
" \"wifi\" : {                                              "\
"       \"SSID\" : string,                                  "\
"       \"passwd\" : string,                                "\
"       \"WiFiOn\" : boolean                               "\
"   }                                                      "\
"}"

---------------------------------------------------------------------

#define EXAMPLE_JSONBUF_WORKING                                                 \
"{                                                                              "\
"   \"ethIPv4Conf\": {                                                          "\
"                       \"ipv4Addr\"        : \"192.168.1.100\",                "\
"                       \"ipv4Netmask\"         : \"255.255.255.0\",            "\
"                       \"ipv4Gateway\"     : \"192.168.1.1\",                  "\
"                       \"ipv4socketTimeout\"   : 10,                           "\
"                       \"DHCPenable\"      : false                             "\
"                   },                                                          "\
"                                                                               "\
"   \"wifi\" : {                                                                "\
"                       \"WiFiOn\" : false,                                     "\
"                       \"SSID\" : \"mynetwork\",                               "\
"                       \"passwd\" : \"abcdefghil\"                             "\
"                   }                                                           "\
"}"

-------------------------------------------------------------------------------------------------

#define EXAMPLE_JSONBUF_NON_WORKING                                                 \
"{                                                                              "\
"   \"ethIPv4Conf\": {                                                          "\
"                       \"ipv4Addr\"        : \"192.168.1.100\",                "\
"                       \"ipv4Netmask\"         : \"255.255.255.0\",            "\
"                       \"ipv4Gateway\"     : \"192.168.1.1\",                  "\
"                       \"ipv4socketTimeout\"   : 10,                           "\
"                       \"DHCPenable\"      : false                             "\
"                   },                                                          "\
"                                                                               "\
"   \"wifi\" : {                                                                "\
"                       \"SSID\" : \"mynetwork\",                               "\
"                       \"passwd\" : \"abcdefghil\",                            "\
"                       \"WiFiOn\" : false                                     "\
"                   }                                                           "\
"}"

********************* OUTPUT ********************************

USING(2)

JSON object created from template
JSON buffer parsed
DHCPenable value: 0
SSID value: mynetwork
passwd value: abcdefghil
WiFiOn value: 0
Finished JSON example

****************************************************************

USING (3)

JSON object created from template
JSON buffer parsed
DHCPenable value: 0
SSID value: mynetwork
passwd value: abcdefghil
WiFiOn value: 768
Finished JSON example

over 7 years ago

0 PM over 7 years ago

TI__Mastermind 27145 points

Hi,

I sent it to a concerned engineer. We will get back to you ASAP. Please bear with us.

Thanks,

0 Giuseppe Cantavenera over 6 years ago in reply to PM

Expert 1060 points

Hello,

any news on this?

Thanks,

Giuseppe C.

0 Sai Reddy over 6 years ago in reply to Giuseppe Cantavenera

TI__Mastermind 22640 points

Hello Giuseppe,

Apologize for the delay. I will work on this and get back by end of day.

Thanks,

Sai

0 Brandon_Green over 6 years ago in reply to Giuseppe Cantavenera

TI__Intellectual 1390 points

Hi Giuseppe,

I'm guessing these values:

DHCPenable value: 0
SSID value: mynetwork
passwd value: abcdefghil
WiFiOn value: 768
Finished JSON example

are produced by calling `Json_getValue` followed by a call to `printf`?

It is odd that the issue only appears when the position of the "WiFiOn" field inside the object is shifted. I was able to reproduce something similar to your results after messing around with the printing of the parsed value:

// Casting the returned void* as a uint16_t produces incorrect output
retVal = Json_getValue(objectHandle, "\"wifi\".\"WiFiOn\"", voidp, &retSize);
printf("\nwifi.WiFiOn value: %d ret: %d\n", *(uint16_t *)voidp, retVal);

// The below casting always prints the correct result
retVal = Json_getValue(objectHandle, "\"wifi\".\"WiFiOn\"", voidp, &retSize);
printf("\nwifi.WiFiOn value: %d ret: %d\n", *(bool *)voidp, retVal);

Could the issue be something like this? If not, please feel free to share your code.

Best,

Brandon

0 Giuseppe Cantavenera over 6 years ago in reply to Brandon_Green

Expert 1060 points

Hello Brandon,

thanks for looking into it, but actually I think casting to bool is not a good idea:

Json_getValue expects a 2 bytes buffer (it's documented only for the Json_setValue actually..), I verified with CCS memory browser that 2 bytes of memory get changed when Json_getValue is exectuted. In my case these 2 bytes get assigned to (0x0, 0x3)

On the other hand, sizeof(bool) is equal to 1 and Json_getValue will fail if retSize is 1 when it's invoked. Casting to bool after reading the param, probably ensues in printing 0, but it's a side effect of the incorrect cast.

Please find attached the modified example that should reproduce the problem.

Thanks,

Giuseppe C.

Fullscreen json.c Download

/*
 * Copyright (c) 2019, Texas Instruments Incorporated
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 * *  Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 *
 * *  Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 *
 * *  Neither the name of Texas Instruments Incorporated nor the names of
 *    its contributors may be used to endorse or promote products derived
 *    from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
 * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
 * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
 * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

/*
 *  ======== json.c ========
 */
#include <stdint.h>
#include <stddef.h>
#include <stdio.h>

/* Display Header files */
#include <ti/display/Display.h>

#include <ti/utils/json/json.h>

////////////////////////  CUSTOMISED PART /////////////////////////


/// change this to select working vs non working case

#define USED_JSON EXAMPLE_JSONBUF_NON_WORKING
//#define USED_JSON EXAMPLE_JSONBUF_WORKING

#define EXAMPLE_JSONBUF_NON_WORKING                                                 \
"{                                                                              "\
"   \"ethIPv4Conf\": {                                                          "\
"                       \"ipv4Addr\"        : \"192.168.1.100\",                "\
"                       \"ipv4Netmask\"         : \"255.255.255.0\",            "\
"                       \"ipv4Gateway\"     : \"192.168.1.1\",                  "\
"                       \"ipv4socketTimeout\"   : 10,                           "\
"                       \"DHCPenable\"      : false                             "\
"                   },                                                          "\
"                                                                               "\
"   \"wifi\" : {                                                                "\
"                       \"SSID\" : \"mynetwork\",                               "\
"                       \"passwd\" : \"abcdefghil\",                            "\
"                       \"WiFiOn\" : false                                     "\
"                   }                                                           "\
"}"

#define EXAMPLE_JSONBUF_WORKING                                                 \
"{                                                                              "\
"   \"ethIPv4Conf\": {                                                          "\
"                       \"ipv4Addr\"        : \"192.168.1.100\",                "\
"                       \"ipv4Netmask\"         : \"255.255.255.0\",            "\
"                       \"ipv4Gateway\"     : \"192.168.1.1\",                  "\
"                       \"ipv4socketTimeout\"   : 10,                           "\
"                       \"DHCPenable\"      : false                             "\
"                   },                                                          "\
"                                                                               "\
"   \"wifi\" : {                                                                "\
"                       \"WiFiOn\" : false,                                     "\
"                       \"SSID\" : \"mynetwork\",                               "\
"                       \"passwd\" : \"abcdefghil\"                             "\
"                   }                                                           "\
"}"

#define EXAMPLE_TEMPLATE                                          \
"{                                                          "\
"   \"ethIPv4Conf\": {                                      "\
"                       \"ipv4Addr\"        : string,       "\
"                       \"ipv4Netmask\"         : string,   "\
"                       \"ipv4Gateway\"     : string,       "\
"                       \"ipv4socketTimeout\"   : uint32,    "\
"                       \"DHCPenable\"      : boolean      "\
"                   },                                      "\
"                                                            "\
"  \"wifi\" : {                                              "\
"       \"SSID\" : string,                                  "\
"       \"passwd\" : string,                                "\
"       \"WiFiOn\" : boolean                               "\
"   }                                                      "\
"}"


//////////////////////////////////////////////////////////////////////////////

/***********************************  ORIGINAL EXAMPLE *************************/

//#define EXAMPLE_TEMPLATE                \
//"{"                                     \
//  "\"firstName\": string,"              \
//  "\"lastName\": string,"               \
//  "\"age\": int32,"                     \
//"\"address\": {"                      \
//    "\"streetAddress\": string,"        \
//    "\"city\": string,"                 \
//    "\"state\": string,"                \
//    "\"postalCode\": string"            \
//  "},"                                  \
//  "\"phoneNumbers\": ["                 \
//    "{"                                 \
//      "\"type\": string,"               \
//      "\"number\": string"              \
//    "},"                                \
//    "{"                                 \
//      "\"type\": string,"               \
//      "\"number\": string"              \
//    "},"                                \
//    "{"                                 \
//      "\"type\": string,"               \
//      "\"number\": string"              \
//    "}"                                 \
//  "],"                                  \
//  "\"children\": [raw],"                \
//  "\"spouse\": boolean,"                 \
//    "\"isAlive\": boolean"               \
//"}"
//
//#define EXAMPLE_JSONBUF                     \
//"{"                                         \
//  "\"firstName\": \"John\","                \
//  "\"lastName\": \"Smith\","                \
//  "\"age\": 25,"                            \
//  "\"isAlive\": true,"                      \
//"\"address\": {"                          \
//    "\"streetAddress\": \"21 2nd Street\"," \
//    "\"city\": \"New York\","               \
//    "\"state\": \"NY\","                    \
//    "\"postalCode\": \"10021-3100\""        \
//  "},"                                      \
//  "\"phoneNumbers\": ["                     \
//    "{"                                     \
//      "\"type\": \"home\","                 \
//      "\"number\": \"212 555-1234\""        \
//    "},"                                    \
//    "{"                                     \
//      "\"type\": \"office\","               \
//      "\"number\": \"646 555-4567\""        \
//    "},"                                    \
//    "{"                                     \
//      "\"type\": \"mobile\","               \
//      "\"number\": \"123 456-7890\""        \
//    "}"                                     \
//  "],"                                      \
//  "\"children\": [],"                       \
//  "\"spouse\": null"                        \
//"}"

/***************************************************************************/

static char keyBuf[37];     /* max string to hold ("phoneNumbers".[X]."type") */
static char jsonBuf[1024];  /* max string to hold serialized JSON buffer */
static char phoneType[10];  /* max string to hold phone types (e.g. "mobile"); */

/*
 *  ======== mainThread ========
 */
void *mainThread(void *arg0)
{
    int i;
    void *valueBuf;
    int16_t retVal;
    int32_t age;
    int32_t phoneNumbers;
    uint16_t valueSize;
    uint16_t jsonBufSize;
    Json_Handle hTemplate;
    Json_Handle hObject;
    Display_Handle display;

    /***** customised ****/
    char str[200];
    uint16_t boolValue;
    /**********************/

    Display_init();

    /* Open an available UART display using default params. */
    display = Display_open(Display_Type_UART, NULL);
    if (display == NULL) {
        /* Failed to open a display */
        while (1);
    }

    /* create a template from a buffer containing a template */
    retVal = Json_createTemplate(&hTemplate, EXAMPLE_TEMPLATE,
            strlen(EXAMPLE_TEMPLATE));
    if (retVal != 0) {
        Display_printf(display, 0, 0, "Error creating the JSON template");
        while (1);
    }
    else {
        Display_printf(display, 0, 0, "JSON template created");
    }

    /* create a default-sized JSON object from the template */
    retVal = Json_createObject(&hObject, hTemplate, 0);
    if (retVal != 0) {
        Display_printf(display, 0, 0, "Error creating JSON object");
        while (1);
    }
    else {
        Display_printf(display, 0, 0, "JSON object created from template");
    }


    /***********  ADAPTED FROM EXAMPLE   ***********/


    /* parse EXAMPLE_JSONBUF */
    retVal = Json_parse(hObject, USED_JSON, strlen(USED_JSON));
    if (retVal != 0) {
        Display_printf(display, 0, 0, "Error parsing the JSON buffer");
        while (1);
    }
    else {
        Display_printf(display, 0, 0, "JSON buffer parsed");
    }

    valueSize = 2;
    retVal = Json_getValue(hObject, "\"ethIPv4Conf\".\"DHCPenable\"", &boolValue, &valueSize);
    if (retVal != 0) {
        Display_printf(display, 0, 0, "Error getting DHCPenable ");
        while (1);
    }
    else {
        Display_printf(display, 0, 0, "DHCPenable  value: %d", boolValue);
    }

    valueSize = sizeof(str);
    retVal = Json_getValue(hObject, "\"wifi\".\"SSID\"", str, &valueSize);
    if (retVal != 0) {
        Display_printf(display, 0, 0, "Error getting SSID ");
        while (1);
    }
    else {
        str[valueSize] = '\0';
        Display_printf(display, 0, 0, "SSID  value: %s", str);
    }

    valueSize = sizeof(str);
    retVal = Json_getValue(hObject, "\"wifi\".\"passwd\"", str, &valueSize);
    if (retVal != 0) {
        Display_printf(display, 0, 0, "Error getting passwd ");
        while (1);
    }
    else {
        str[valueSize] = '\0';
        Display_printf(display, 0, 0, "passwd  value: %s", str);
    }

    valueSize = 2;
    retVal = Json_getValue(hObject, "\"wifi\".\"WiFiOn\"", &boolValue, &valueSize);
    if (retVal != 0) {
        Display_printf(display, 0, 0, "Error getting WiFiOn ");
        while (1);
    }
    else {
        Display_printf(display, 0, 0, "WiFiOn  value: %d", boolValue);
    }
/*****************************************************************************/


/***************************** ORIGINAL EXAMPLE  **************************************************/


//    /*
//     * Get the "firstName" value.  Note that this approach first
//     * detects the size of the result, then calloc()'s memory for it
//     * and retrieves the value.
//     */
//    retVal = Json_getValue(hObject, "\"firstName\"", NULL, &valueSize);
//    if (retVal != 0) {
//        Display_printf(display, 0, 0, "Error getting the firstName buffer size");
//        while (1);
//    }
//    else {
//        Display_printf(display, 0, 0, "firstName buffer size: %d", valueSize);
//    }
//
//    valueBuf = calloc(1, valueSize + 1);
//    retVal = Json_getValue(hObject, "\"firstName\"", valueBuf, &valueSize);
//    if (retVal != 0) {
//        Display_printf(display, 0, 0, "Error getting the firstName value");
//        while (1);
//    }
//    else {
//        Display_printf(display, 0, 0, "JSON buffer parsed, firstName: %s",
//                valueBuf);
//    }
//    free(valueBuf);
//
//    /* increment the "age" value by one */
//    valueSize = sizeof(age);
//    retVal = Json_getValue(hObject, "\"age\"", &age, &valueSize);
//    if (retVal != 0) {
//        Display_printf(display, 0, 0, "Error getting the age value");
//        while (1);
//    }
//    else {
//        Display_printf(display, 0, 0, "age value: %d", age);
//    }
//
//    age++;
//
//    valueSize = sizeof(age);
//    retVal = Json_setValue(hObject, "\"age\"", &age, valueSize);
//    if (retVal != 0) {
//        Display_printf(display, 0, 0, "Error setting the age value");
//        while (1);
//    }
//
//    valueSize = sizeof(age);
//    retVal = Json_getValue(hObject, "\"age\"", &age, &valueSize);
//    if (retVal != 0) {
//        Display_printf(display, 0, 0, "Error getting the new age value");
//        while (1);
//    }
//    else {
//        Display_printf(display, 0, 0, "new age value: %d", age);
//    }
//
//    /* serialize the current JSON object (now with a new age) */
//    jsonBufSize = 1024;
//    retVal = Json_build(hObject, jsonBuf, &jsonBufSize);
//    if (retVal != 0) {
//        Display_printf(display, 0, 0, "Error serializing JSON data");
//        while (1);
//    }
//    else {
//        Display_printf(display, 0, 0, "serialized data:\n%s", jsonBuf);
//    }
//
//    /* determine how many phone numbers are present */
//    phoneNumbers = Json_getArrayMembersCount(hObject, "\"phoneNumbers\"");
//    if (retVal != 0) {
//        Display_printf(display, 0, 0,
//                "Error getting phoneNumbers array size");
//        while (1);
//    }
//    else {
//        Display_printf(display, 0, 0, "phoneNumbers array size: %d",
//                phoneNumbers);
//    }
//
//    /* print the mobile phone number */
//    for (i = 0; i < phoneNumbers; i++) {
//        sprintf(keyBuf, "\"phoneNumbers\".[%d].\"type\"", i);
//
//        valueSize = sizeof(phoneType);
//        memset(phoneType, 0, valueSize);
//
//        retVal = Json_getValue(hObject, keyBuf, phoneType, &valueSize);
//        if (retVal != 0) {
//            Display_printf(display, 0, 0, "Error getting phoneNumbers[%d]",
//                    i);
//            while (1);
//        }
//
//        if (strcmp(phoneType, "mobile") == 0) {
//            /* found it, get the value, then stop looking */
//            sprintf(keyBuf, "\"phoneNumbers\".[%d].\"number\"", i);
//
//            retVal = Json_getValue(hObject, keyBuf, NULL, &valueSize);
//
//            valueBuf = calloc(1, valueSize + 1);
//            retVal = Json_getValue(hObject, keyBuf, valueBuf, &valueSize);
//            if (retVal != 0) {
//                Display_printf(display, 0, 0,
//                        "Error getting phoneNumbers[%d]", i);
//                while (1);
//            }
//            else {
//                Display_printf(display, 0, 0,
//                        "mobile phone number: %s", valueBuf);
//            }
//
//            free(valueBuf);
//            break;
//        }
//    }
//
//    /* print the second (index #1) phone number */
//    retVal = Json_getValue(hObject, "\"phoneNumbers\".[1].\"number\"", NULL,
//            &valueSize);
//    if (retVal != 0) {
//        Display_printf(display, 0, 0, "Error getting phoneNumbers[1]");
//        while (1);
//    }
//
//    valueBuf = calloc(1, valueSize + 1);
//    retVal = Json_getValue(hObject, "\"phoneNumbers\".[1].\"number\"", valueBuf,
//            &valueSize);
//    if (retVal != 0) {
//        Display_printf(display, 0, 0, "Error getting phoneNumbers[1] value");
//        while (1);
//    }
//    else {
//        Display_printf(display, 0, 0, "phoneNumbers[1] value: %s", valueBuf);
//    }
//    free(valueBuf);
    /*******************************************************************************************/

        /* done, cleanup */
    retVal = Json_destroyObject(hObject);
    retVal = Json_destroyTemplate(hTemplate);


    Display_printf(display, 0, 0, "Finished JSON example");

    return (0);
}

0 Brandon_Green over 6 years ago in reply to Giuseppe Cantavenera

TI__Intellectual 1390 points

Hi Giuseppe,

Thanks for the code. I think you're getting lucky/unlucky with the values in memory when using the EXAMPLE_JSONBUF_WORKING and I think that the size of the variable you choose to use to store the value is the source of the bad output value. Like you said, sizeof(bool) is 1, so from that I would think we should only attempt to read a single byte of the memory retrieved by Json_getValue. I believe that by using sizeof(uint16_t) to interpret the return value (whether explicitly through a cast to a uint16_t pointer or implicitly through using a uint16_t variable as storage) you are reading one more byte than necessary. The fact that two bytes are copied over by Json_getValue is an unfortunate consequence of how the library internally handles boolean values. I believe the code you've provided works in one case, but not the other because 0x0 was in memory next to the boolean value as opposed to 0x3, so when reading two bytes instead of one the printed value still appeared like the expected boolean value.

If you take the EXAMPLE_JSONBUF_WORKING and increase the value of ipv4socketTimeout to some value that takes more than a single byte to store, you will start getting strange results for the following boolean field DHCPenable, if you also choose to read two bytes from DHCPenable's memory address instead of one.

I am only suggesting to use a size value of 1 when interpreting the return value of Json_getValue called on a boolean json field. If you were to use a boolean variable as the input to Json_getValue (instead of a more generic variable like a character array) and then output this value, you will get correct results because the boolean only takes up one byte.

This would seem contradictory or, at least, not quite correct based on Json_getValue failing if you specify a maxValueSize of 1, like you say. But if you look in the code around the size check it does, there is no special case for a boolean and it instead chooses to use sizeof(uint16_t) for its error checking. I can see why this would lead someone to use a uint16_t when attempting to retrieve a boolean value, but I don't think that's how booleans should be treated when using this library. This is my opinion based on usage and testing. The original documentation around certain details within the library is lacking, so I cannot say for sure what the intended usage pattern was.

Thank you for the note on Json_getValue. We will update the documentation.

Best,

Brandon

0 Giuseppe Cantavenera over 6 years ago in reply to Brandon_Green

Expert 1060 points

Hello Brandon,

thank you for the explanation.

We made some tests on the EVM example also playing with the other values and were able to come to the same conclusions:

it actually seems that 2 bytes are copied onto the input buffer but only one is to be considered, like the library copies one more byte of unrelated data to the user buffer.

Based on the tests so far I haven't spotted any side effect or corruption caused by the suggested approach,

therefore we'll try this on the real application we are developing and mark this thread as answered for now.

Still, this library behaviour looks pretty weird...

Thanks,

Giuseppe

**Attention** This is a public forum

MSP low-power microcontrollers

MSP low-power microcontroller forum

MSP432E401Y: JSON parsing issue in corner case.