This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC3235MODSF: Rollback of firmware after OTA

Part Number: CC3235MODSF
Other Parts Discussed in Thread: UNIFLASH

Hi,

I have an issue with the automatic rollback when doing an OTA update of the cc3235modsf. 

I'm writing the file /sys/mcuflashimg.bin, calling sl_Stop, PRCMHibernateCycleTrigger and testing the new firmware. If it succeed, I call sl_FsCtl(SL_FS_CTL_BUNDLE_COMMIT, ...).

I am expecting the file to be automatically reverted to the previous one if it does not get validated by the commit during its first start, or automatically rollbacked after a few seconds if the file is not functional at all.  When testing with a functional firmware it works fine. But when writing a bad file I get two unexpected behavior:

1: When closing the file with sl_FsClose(FileHandle, FS_CERTIFICATE_FILE_NAME, buff, KEY_SIZE); It returns 0 even though the key is not the proper one.

2: The cc does not wake up afterwards. I need to reset it externally

Some more info:

To open the file, I'm using the flags:

uint32_t flags = SL_FS_OVERWRITE;
flags |= SL_FS_CREATE;
flags |= SL_FS_CREATE_FAILSAFE;
flags |= SL_FS_WRITE_BUNDLE_FILE;
flags |= SL_FS_CREATE_MAX_SIZE(FW_MAX_FILE_SIZE);
flags |= SL_FS_CREATE_SECURE;
flags |= SL_FS_CREATE_PUBLIC_WRITE;

The SlFsFileInfo_t.Flags is 0x258 after the new file has been written.

The bundle state is 1 after the file has been written, and 2 after sl_Stop has been called.

I have tried as well with calling Platform_CommitWdtConfig (as found in ota_cloud example) to set the watchdog, but without success.

I'm using the latest sdk 3_30_01_02 and service pack 4.4.1.3_3.1.0.5_3.1.0.19

With a similar procedure on the cc3220modasf, I get the error SL_ERROR_FS_WRONG_SIGNATURE_SECURITY_ALERT when calling sl_FsClose()

Do you have any idea what could be the issue? 

Do not hesitate to ask for more information.

Best regards,

Cédric

  • Hi Cedric,.

    1. What do you mean by "the key is not the proper one"? what key? You are supposed to provide a certificate (filename) and signature (of the new file content signed with a corresponding private key)?

    2. When do you call the "Platform_CommitWdtConfig"? it should be called at the beginning of the OTA procedure (before the MCU reset).

    what is the timeout you've provided?

    Br,

    Kobi  

  • Hi Kobi,

    Thanks for the prompt feedback.

    1: Indeed my wording was bad. I am using a certificate which is stored in flash, and the signature of the file is generated using the corresponding private key.  When the signature is the correct one everything works fine, but when the signature is incorrect I do not get any error on the cc3235 (on the cc3220 I do get the error)

    2: My current procedure is:

    • Write new file
    • Platform_CommitWdtConfig
    • sl_Stop
    • PRCMHibernateCycleTrigger

    3: Currently I set it to 10 secondes

    Additional tests:

    I've tried to test with the SL_FS_WRITE_MUST_COMMIT flag instead of SL_FS_WRITE_BUNDLE_FILE flag and get the same behavior.

    I've played with the bootloader watchdog (Platform_CommitWdtConfig). I tried to set it at the start of my application and restart with PRCMHibernateCycleTrigger to see if it would reset during the next wake up phase, I tried with and without using the application watchdog. I also tried to reset it externally after having called Platform_CommitWdtConfig to insure it did see a boot. I did not get a reset in any scenario.

    Is there a way to check that the bootloader watchdog is indeed activated?

    Best regards,

    Cédric

  • Hi Cedric,

    Sequence seems OK.

    Does the watchdog works for you using CC3220? I've personally verified it with CC3220 in the past (haven't done so with CC3235 yet).

    Please provide NWP logs of the OTA process (see: http://processors.wiki.ti.com/index.php/CC3120_%26_CC3220_Capture_NWP_Logs ).

    Br,

    Kobi

  • Hi Kobi,

    I will have to check for the cc3220 watchdog. Unfortunately on our board getting the nwp log is complicated, and there is quite a lot to adapt to make it run on the launchpad.

    Actually my main issue is more with non-error when writing a bad secure file. If I can insure that bad files are not used, the watchdog will only be a backup solution.  And this works on the cc3220.

    I've done some additional tests and do not understand much of the results.

    Service pack:

    When I write the correct file with its signature appended, the service pack is correctly installed. However when I write a bad file (I used a 75KB html file with the signature), I also get no error. After restart the service pack has been rollbacked to an old service pack:

    CHIP 0x31000019
    MAC 31.3.0.0.19
    PHY 0x3.1.0.0
    NWP 0x4.0.1.21
    ROM 0
    HOST 0x3.0.1.36

    I tried with other flags ( SL_FS_CREATE_SECURE) but got the same issue.

    Here is the code showing how the sp is opened/closed:

        FileHandle = sl_FsOpen(SP_FILE_NAME, SL_FS_WRITE | SL_FS_CREATE_MAX_SIZE( SP_MAX_FILE_SIZE ), NULL);
        DBG_PRINT("lRetVal OPEN: %d", FileHandle);
        if (FileHandle < 0)
        {
            ERROR_PRINT("Error: FW %d", FileHandle);
            return -1;
        }
    
        WatchdogAck();          // ack watchdog so it doesn't reset
        lRetVal = HTTPGetServicePack(g_buff, updateFirmwareWriteCallback, FileHandle);
        if (lRetVal < 0)
        {
            ERROR_PRINT("Error: FW %d", lRetVal);
            sl_FsClose(FileHandle, NULL, NULL, 0);
            return -1;
        }
    
        //TODO: Handle signature (SL_ERROR_FS_WRONG_CERTIFICATE_FILE_NAME)
        lRetVal = sl_FsClose(FileHandle, NULL, g_buff, SIGNATURE_SIZE);
        DBG_PRINT("lRetVal CLOSE: %d", lRetVal);
        if (lRetVal < 0)
        {
            ERROR_PRINT("Error: FW %d", lRetVal);
            return -1;
        }

    Dummy file:

    I tried to write a dummy file in secure mode and write it once with a bad signature and once with a good signature.

    I create the file with uniflash like: 

    C:\\ti\\uniflash_5.1.0\\simplelink\\imagecreator\\bin\\SLImageCreator.exe project add_file --name Display_v2 --file dummy.bin --fs_path dummy.bin --priv key.der --cert code_signing.der --overwrite --max_size 7500 --flags publicread,publicwrite,secure

    And I have the same thing, the file is always written without error even if the key is bad. 

    As it works when the file is correct, I do not understand how my code could have an issue. Should I use another set of flags to have the signature check upon close? Or is there a bug in the sdk or nwp?

    Can you reproduce this issue?

  • I've reproduced the same issue on the launchpad with a slightly modified example project.

    In the project the file msp_measure.bin is overwritten by the file msp_measure_copy.bin using a signature generated by another key/cert pair. It should fail but instead succeeds.

    Here are the ccs project and the uniflash project:

    /cfs-file/__key/communityserver-discussions-components-files/968/Display_5F00_v2_5F00_test_5F00_191107155049.zip

    /cfs-file/__key/communityserver-discussions-components-files/968/httpget_5F00_CC3235SF_5F00_LAUNCHXL_5F00_tirtos_5F00_ccs.zip

    Can you reproduce it?

    Best regards,

    Cédric

  • Hi, 

    Have you been able to reproduce the issue?

    Best regards,

    Cédric

  • I have the same issue on the cc3220, a bad signature is not detected with the latest sp. But when reverting to previous sp it works ok with cc3220

    sdk_3_20_01_01 sdk_3_30_01_02
    sp_3.12.0.1_2.0.0.0_2.2.0.6 Error SL_ERROR_FS_WRONG_SIGNATURE_SECURITY_ALERT as expected Error SL_ERROR_FS_WRONG_SIGNATURE_SECURITY_ALERT as expected
    sp_3.13.0.3_2.0.0.0_2.2.0.6 No error, fail at reboot No error, fail at reboot

    For the cc3220 I will then work with the sdk_3_30_01_02 and sp_3.12.0.1_2.0.0.0_2.2.0.6 until this is fixed. Is there any expected incompatibility?

    But for the cc3235 it never works. I have tried with several sp:

    sp_4.2.0.3_3.1.0.5_3.1.0.17

    sp_4.3.0.5_3.1.0.5_3.1.0.18

    sp_4.4.0.3_3.1.0.5_3.1.0.19

    sp_4.4.1.3_3.1.0.5_3.1.0.19

    Any ideas what I could more?

    We need a solution soon!

    Best regards,

    Cédric

  • I have done more tests on the watchdog and cc3220modasf.

    With the following procedure it works:

    • Platform_CommitWdtConfig
    • Write new file
    • sl_Stop
    • PRCMHibernateCycleTrigger

    It does reset after 5s as set in Platform_CommitWdtConfig and the fw is rollbacked.

    I could not make this work on the cc3235.

  • Hi Cedric,

    Does your new ota content contain a service pack?

    If yes, is it SP of CC3235?

    I was able to reproduce the issue, but need more time to debug the problem. I'll send an update next week.

    Br,

    Kobi

  • Hi Kobi,

    I'm glad that you could reproduce the issue! 

    Our current OTA process is separated for both the SP and FW.

    Both fails silently if the file is incorrect with bad signature, with the FW it gets stuck and with the SP it gets reverted to an old one (as described above)

    Keep me posted, I'll gladly take any fix or workaround!

    Cédric

  • Hi Cedric,

    So just to verify we see the same thing: the FW update is working correctly (with automatic reversion based on the watchdog), but the updating an erroneous SP causes issues (as the watchdog is not triggered). Is that what you see?

    How do you generate the wrong SP?

    Br,

    Kobi

  • Hi Kobi,

    Not exactly.

    To test a bad file I took a random file of a few KB. (it was an html file) on my computer. It resulted in:

    • FW update: no error is raised and watchdog does not kick so it stays in non functional state until external reset
    • SP update: No error is raised and SP reverts to an old one

    This is what I have as root cause:

    • Writing any secure file with a bad signature does not raise any error.
    • Watchdog does not kick

    Have you been able to reproduce the writing of a secured file with a bad signature (With the project I provided above)? 

    Cédric

  • Hi Cedric,


    The issues was reported as internally and we will handle it in one of the next releases. 

    We completely understand the importance but I can't suggest an immediate fix.

    I'll close the thread for now and will provide an update once it is fixed.

    br,

    Kobi

     

  • Hi Kobi,

    Thanks for the update, as we talked about several things can you describe the issues you have found and reported?

    What about the watchdog issue?

    Best regards,

    Cédric

  • Hi Cedric,

    The issues reported are:

    1. CC3235 doesn't identify a bad signature during an OTA update.

    2. If the SP is wrong, then the missing failure detection will end by switching to the wrong that will not trigger the OTA watchdog (so no automatic reversion gets triggered). 

    I'll mark this thread as resolved, but the fix will only be available in one of the next SDK releases (unfortunately not the one at the end of this month). 

    Br,

    Kobi

  • Hi Kobi,

    Thanks for the update!

    I'll accept the thread as resolved when the issue is fixed ; )

    Thanks!

    Cédric