CCS/CC3220SF-LAUNCHXL: OTA - Over the Air - Signature Verification fails after many successful attempts

Hi Zacson Skaria,

Are you always using the same OTA image  (probably not since at least the timestamp needs to be updated).

This error was known while using older service packs. Are you using the latest one? (the new SP is required for the Secure OTA).

Can it be that the previous (successful) OTA process reverted to an older SP?

Br,

Kobi

Adding to our earlier finding.(Zacson and I were working together on this)

The issue is observed to be caused after the internet connection is lost when the write of the mcuimg is in progress.
---------
Logs:
OtaArchive_RunParseTar: FileType=0, FileName=/sys/mcuimg.bin,
FileSize=170696
Create/Open for write file /sys/mcuimg.bin
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 1440.
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 2880.
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 4320.
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 5760.
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 7200.
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 8640.
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 10080.
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 11520.
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 12960.
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 14400.
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 15840.
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 17280.
OtaArchive_RunParseTar: Write size 1440 to file /sys/mcuimg.bin total 18720.
OtaArchive_RunParseTar: Write size 504 to file /sys/mcuimg.bin total 19224.
HttpClient_RecvAppend: ERROR HttpClient_Recv, status=-20312
OTA_run: ERROR downloading, CdnClient_RecvAppend Status=-20312
_OtaCheckConsecutiveErrors: ConsecutiveOtaErrors=1/5, return only
WARNNING
OtaRunStep: WARNING Ota_run, Status=20003, continue for next OTA retry
-----------

This OTARunStep will continue retry as no internet connection. When internet connectivity returns, the OTA is reattempted. But this time and all future attempts, we get this error as shown below:
-----------
Logs:
OtaArchive_RunParseTar: parsing archive file header
OtaArchive_RunParseTar: skip block align RecvBufLen=863,
SkipAlignSize=275
OtaArchive_RunParseTar: FileType=0, FileName=ota.sign, FileSize=71
[_BundleCmdSignatureFile_Parse] signature verification failed!
OtaArchive_RunParseTar: ERROR in _BundleCmdSignatureFile_Parse, Status=-
12291
OTA_run: ERROR OtaArchive_RunParse, Status=-12291
_OtaCheckConsecutiveErrors: ConsecutiveOtaErrors=1/5, return only
WARNNING
OtaRunStep: WARNING Ota_run, Status=20009, continue for next OTA retry
---------
This error continues for all future attempts, even if we update the CDN with a new OTA.
This issue is observed in the cloud_ota CCS example project too. (This error vanishes only if we re-flash the device with a new build using Uniflash directly)

Our assumption is that the earlier failure during write to file /sys/mcuimg.bin is corrupting the signature somehow, and is causing all the future attempts for certificate verification to fail.

Has anyone observed any similar issue? Any suggestions for a possible fix?

We are also using the latest cc3220 SDK&SP from this link -> www.ti.com/.../SIMPLELINK-CC3220-SDK

Dear Kobi,

We are using the same OTA image through out our testing. Since the OTA_LOOP_TESTING is enabled, it will ignore the timestamp (older version) and will keep updating.

We are using the latest service pack, i.e in the initial version as well as OTA image uses the latest SDK and service pack.

It seems like the dowloaded image would get corrupted sometime due to a network issue(we tried to disconnect the n/w while OTA download was in progress), and there after it will keep trying to update with the same corrupted image(which is downloaded to the chip) hence resulting in signature verification.

So basically if any download fails, the OTA will never work even if we reset the board. Factory reseting the board seems to clear the corrupted image.

Regards

Zac

Team,

We have tried the cloud ota example provided in the SDK and found similar issue appearing. 

While the firmware/archive is being downloaded, if network is interrupted then the image would get corrupted and would never recover even if we reboot the chip. After reboot the chip would still try to update with the partly downloaded (corrupted) firmware and throws signature verification failed error. This can be cleared only if we do a hard reset or re-flash via uniflash.

Please share if others have experienced similar issue and have found any solution.

Thanks

Zac

Hi Zac.

Do you mean that you tried the new SDK 1.60.00.04? It includes a fix to the Secure OTA library that is very relevant to this. 

BTW. do you include the "ota vendor certificate" within your OTA image (or just install it once using Uniflash)?

Do you see this error only after facing the network issues?

Please provide a full  OTA log of the failure as well as the previous  ota-cycle with the network issues.

Br,

Kobi.

Hi Kobi,
I am using the SDK 1.50.00.06 (www.ti.com/.../simplelink-cc3220-sdk). This seems to be the latest SDK available in TI site.
Could you please direct us to SDK 1.60.00.04 ?

Thanks
Zac

Hi Kobi,
I will try with the new SDK (I accidentally marked this as resolved).
Is this a known issue in SDK 1.5 ?
Thanks
Zac

Yes, it seems like a known issue (see release notes of the 1.60 SDK).
I'll know for sure (that yours is the same issue) - if it will work with the new SDK or if you provide the details I've asked before.

Br,
Kobi

Hi Zac,
Noob question...Could you explain how to get the OTA demo running with the latest SDK (1.60)? When I browse to the demo project in Resource Explorer and import the code into CSS, it specifically uses 1.50 SDK version... Is there a different way to get this project into CSS? (I'm using the project that utilizes FreeRTOS)

Hi Brian,

I worked with the Ti-RTOS version of the SDK (not FreeRTOS).

Nevertheless, I think most of the configurations while importing a new SDK example would be the same. I remember a few changes which I did when i tried using the new SDK:

1. In CCS project settings, i set the SDK as the latest one. "CCS General" >> "Products" >> SimpleLink c3220 SDK >> 1.60.0.04.

2. After this, the path variables became updated with the new SDK path.  (Properties >> Resources >> Linked Resources) 

3. I edited the new ota library from the new SDK 1.60 and updated the CDN details and added the same to the project. (<SDK path>/source/ti/net/ota). Added the ota library in the Project properties > ARM Linker >File search path. 

4. Another point to note is that while flashing the build using UniFlash is to select the new Service pack and also the new certificates and keys.

Please do check and let me know if it works. 

Regards,

Zacson

Hi Zac,
I made some progress with this OTA demo using dropbox as the CDN. I am able to successfully connect to the server and download the new image, but I am receiving an error when the program attempts to parse the image and write it to /sys/mcuflashimg.bin.... Here are the logs:

HandlePingComplete: OTA Command arrived
OtaInit: statistics = 0, 0, 0
OtaInit: call Ota_init
OTA_init: sizeof CdnClient=576, sizeof OtaArchive=4956
OTA_init: sizeof OtaLib_t=7736, sizeof OTA_memBlock=7800
OTA_init: OTA lib version = OTA_LIB_2.0.0.7
OtaArchive_Init: OTA archive version = OTA_ARCHIVE_2.0.0.4
OtaConfig: call OTA_set EXTLIB_OTA_SET_OPT_SERVER_INFO, ServerName=api.dropboxapi.com
OtaConfig: call OTA_set EXTLIB_OTA_SET_OPT_VENDOR_ID, VendorDir=OTA-firmware
OTA_run: call CdnClient_ConnectServer OTA server=api.dropboxapi.com
CdnClient_ConnectServer: HttpClient_Connect api.dropboxapi.com
HttpClient_Connect: IP_ADDR=162.125.6.7
HttpClient_Connect: WARNING Socket Connect, status=-468, Ignored...
OTA_run: CdnClient_ReqOtaDir, VendorDir=OTA-firmware
CdnDropbox_SendReqDir: uri=/2/files/list_folder
RespLen is 702, ProcessedSize is: 697
the entire JSON pRespBuf is: OtaDir FileName=/OTA-firmware/20180123170612_CC3220SF_TI-OTA-test.tar, FileSize=163840
OTA_run: CdnClient_ReqOtaDir, NumDirFiles=1
OTA_run: CdnClient_GetNextDirFile
OTA_run: CdnClient_GetNextDirFile: file=/OTA-firmware/20180123170612_CC3220SF_TI-OTA-test.tar, size=163840
OtaArchive_Init: OTA archive version = OTA_ARCHIVE_2.0.0.4
_ReadOtaVersionFile: file ota.dat, status=SL_ERROR_FS_FILE_NOT_EXISTS
OtaArchive_CheckVersion: can't open version file, sign it as old version
OtaArchive_CheckVersion: accept the new version = 20180123170612_CC3220SF_TI-OTA-test.tar
OtaRunStep: status from Ota_run: OTA_RUN_STATUS_CHECK_NEWER_VERSION, accept and continue
_ReadOtaVersionFile: file ota.dat, status=SL_ERROR_FS_FILE_NOT_EXISTS
OtaRunStep: CurrentVersion=00000000000000, NewVersion=20180123170612, Start download ...
OTA_run: Call CdnClient_ReqFileUrl, filename = /OTA-firmware/20180123170612_CC3220SF_TI-OTA-test.tar
CdnDropbox_SendReqFileUrl: uri=/2/files/get_temporary_link
HTTP request is:
POST /2/files/get_temporary_link HTTP/1.1
host: api.dropboxapi.com
Authorization: Bearer 5iGgEwYQNbAAAAAXXXXXXXXXXXX-hidden-XXXXXXXXXXXXXXXwNQpUROYCc3pO
Content-Type: Application/Json
Content-Length: 65

{"path": "/OTA-firmware/20180123170612_CC3220SF_TI-OTA-test.tar"}

OTA_run: Call CdnClient_ConnectFileServer, url = dl.dropboxusercontent.com/.../AAD8FKC7K1GzPRnDlymFCK-98zwYUQBQ5Cv5e5Dtwdh7Y4qdifk648aHyCD_109qkPAX0ruJb2DCywjDKZFnvhXj5YKqd2WZJByok_KZeqGsIiL_gb3vCneNulBguFOvFS4BVXSShSo1z4X-EgDV-L7kry3gFPW8e9he88GPQrWhW1yN-Iww2mSRRqBq28zBZs_7tc4PH-uo9H--T8HERQgApgSj6uZJYW2hqUEyeh8nV-yCctJ_YFJLPiiv56g5A2_hq5R7ApfF6-W63ttXxaJtN5davgFesPw0Om9TX2_5r0gSw5z0YxGKGFSkqa9qtfg
HttpClient_Connect: IP_ADDR=162.125.6.6
HttpClient_Connect: WARNING Socket Connect, status=-468, Ignored...
OTA_run: Call CdnClient_ReqFileContent, url = dl.dropboxusercontent.com/.../AAD8FKC7K1GzPRnDlymFCK-98zwYUQBQ5Cv5e5Dtwdh7Y4qdifk648aHyCD_109qkPAX0ruJb2DCywjDKZFnvhXj5YKqd2WZJByok_KZeqGsIiL_gb3vCneNulBguFOvFS4BVXSShSo1z4X-EgDV-L7kry3gFPW8e9he88GPQrWhW1yN-Iww2mSRRqBq28zBZs_7tc4PH-uo9H--T8HERQgApgSj6uZJYW2hqUEyeh8nV-yCctJ_YFJLPiiv56g5A2_hq5R7ApfF6-W63ttXxaJtN5davgFesPw0Om9TX2_5r0gSw5z0YxGKGFSkqa9qtfg
CdnDropbox_SendReqFileContent: file=/apitl/1/AAD8FKC7K1GzPRnDlymFCK-98zwYUQBQ5Cv5e5Dtwdh7Y4qdifk648aHyCD_109qkPAX0ruJb2DCywjDKZFnvhXj5YKqd2WZJByok_KZeqGsIiL_gb3vCneNulBguFOvFS4BVXSShSo1z4X-EgDV-L7kry3gFPW8e9he88GPQrWhW1yN-Iww2mSRRqBq28zBZs_7tc4PH-uo9H--T8HERQgApgSj6uZJYW2hqUEyeh8nV-yCctJ_YFJLPiiv56g5A2_hq5R7ApfF6-W63ttXxaJtN5davgFesPw0Om9TX2_5r0gSw5z0YxGKGFSkqa9qtfg
OtaArchive_RunParse: set state=ARCHIVE_STATE_PARSE_HDR
OtaArchive_RunParseTar: parsing archive file header
OtaArchive_RunParseTar: filetype=5, directory=20180123170612_CC3220SF_TI-OTA-test/
OtaArchive_RunParseTar: parsing archive file header
OtaArchive_RunParseTar: filetype=5, directory=20180123170612_CC3220SF_TI-OTA-test/0/
OtaArchive_RunParseTar: parsing archive file header
OtaArchive_RunParseTar: filetype=5, directory=20180123170612_CC3220SF_TI-OTA-test/1/
OtaArchive_RunParseTar: parsing archive file header
OtaArchive_RunParseTar: filetype=5, directory=20180123170612_CC3220SF_TI-OTA-test/2/
OtaArchive_RunParseTar: parsing archive file header
OtaArchive_RunParseTar: FileType=0, FileName=ota.cmd, FileSize=2153
[_BundleCmdFile_Parse] bundle cmd file=/sys/mcuflashimg.bin, sig_len=256, SHA_256_Digets=28a2bb6520f9fc57592ecbfc17ab066e115d139e1a82d5b26648eae5de77b015, cert=dummy-root-ca-cert, secured=1, bundle=1
[_BundleCmdFile_Parse] bundle cmd file=digicerthighassuranceevrootca.der, sig_len=0, SHA_256_Digets=7431e5f4c3c1ce4690774f0b61e05440883ba9a01ed00ba6abd7806ed3b118cf, cert=, secured=0, bundle=0
[_BundleCmdFile_Parse] bundle cmd file=dummy-root-ca-cert, sig_len=0, SHA_256_Digets=34941765501d16a4ab776c3a74d89945f1a2575c5893069f62ffbe803f344549, cert=, secured=0, bundle=0
[_BundleCmdFile_Parse] bundle cmd file=dummy-trusted-ca-cert, sig_len=0, SHA_256_Digets=07ad6ba74b3921009edf184cb382c05a32baacf9c767f74427c094b2c56aa135, cert=, secured=0, bundle=0
[_BundleCmdFile_Parse] bundle cmd file=dummy_ota_vendor_cert.der, sig_len=0, SHA_256_Digets=a160b855d7a00a6002922181377249a80ecd6a738d23e1dd8976c8bb7fad1bcb, cert=, secured=0, bundle=0
[_BundleCmdFile_Parse] bundle cmd file=/sys/servicepack.ucf, sig_len=256, SHA_256_Digets=5c842c3139ecda90f0c15723f920a392b8d4d19f03281ee226359308f758f06e, cert=, secured=1, bundle=1
OtaArchive_RunParseTar: parsing archive file header
OtaArchive_RunParseTar: skip block align RecvBufLen=1440, SkipAlignSize=408
OtaArchive_RunParseTar: FileType=0, FileName=ota.sign, FileSize=72
OtaArchive_RunParseTar: parsing archive file header
OtaArchive_RunParseTar: skip block align RecvBufLen=1440, SkipAlignSize=440
OtaArchive_RunParseTar: FileType=0, FileName=dummy-root-ca-cert, FileSize=975
Create/Open for write file dummy-root-ca-cert
OtaArchive_RunParseTar: Write size 975 to file dummy-root-ca-cert total 975.

Hash verification succeeded.
Total archive file bytes 7631.
OtaArchive_RunParseTar: 1 files that are mentioned in the ota.cmd were saved
OtaArchive_RunParseTar: Downloading File Completed - Size=975
OtaArchive_RunParseTar: parsing archive file header
OtaArchive_RunParseTar: skip block align RecvBufLen=1440, SkipAlignSize=49
OtaArchive_RunParseTar: FileType=0, FileName=/sys/mcuflashimg.bin, FileSize=116124
Create/Open for write file /sys/mcuflashimg.bin

OtaArchive_RunParseTar: !!!!!! SECURITY ALERT !!!!! on pOpenFile, Status=-10365

OTA_run: SECURITY ALERT OtaArchive_RunParse, Status=-20199

OtaRunStep: FATAL ERROR from Ota_run -21004 !!!!!!!!!!!!!!!!!!!!!!!!!!!

Test failed: State = 6, Event = 17
Event handler failed..!!
-----------------------------------------------------------------------------

Any idea what's causing this?

Hi Brian,

I'll appreciate if you can send the link to the page with the wrong instructions.

Thanks,

Kobi

Here are links to the two documents I referenced above:
dev.ti.com/.../Over_The_Air_Update_Guide.html
dev.ti.com/.../wifi_ota.html

Brian Harrah said:

Ok... I was able to figure it out! Hooray! Turns out I was using the wrong key file when signing the image in Uniflash. This actually happened because I was following a tutorial that instructed me one way. Then I found a different tutorial for this demo with slightly different instructions that ended up working....  So this begs the question, why are there two very similar (but not identical) sets of instructions for the OTA demo?  Neither set of instructions seem to work correctly on their own, but using bits and pieces from both you can derive enough correct information to get the demo working. Whats up with that?

This was my experience as well.  The documentation is not clear in either case, but if you manage to piece the two together you can get something to work.  Really needs to get fixed.

Hi Andrew, Brian,

Looking at both (one is the OTA application notes which supposed to be more generic while the 2nd is our online OTA lab that takes you through specific example) I couldn't find the issue you are reporting. Do you find any wrong instruction in any of them or you just find them not clear enough.

Br,
Kobi

Hi Kobi,

It needs to be corrected to tell github users to use their username for the OTA_VENDOR_TOKEN.