SK-AM62A-LP: sigmoid bug in AM62A PSDK 10.01

Hello Youngheon,

This issue is still being addressed by our development team. I have indicated to them that this issue is still prevalent in the 10.1 SDK. They have indicated that the next SDK release will carry a fix for this issue. 

BR,
Reese

Hi Youngheon,

There will not be an 11.0 SDK for AM62A, unfortunately.

However, this does not mean that the fix will not be available sooner than the following SDK release. 

We plan to have a backwards compatible release of 11.0 TIDL tools that will work with 10.1 SDK. This is a practice that we've been following in the past 3-4 release. This is marked in a version compatibility table / document:

• https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/docs/version_compatibility_table.md
• Does not yet show 11.0 since that is unreleased overall

So basically, I expect to see a release tag for 11_00_00_01, where the odd fix number (1) indicates it is compatible with previous SDK release 10.1. You can see similar for 10_01_03_00 that is compatible with 10.0 SDK. 

This release would have the Sigmoid issues fixed, which I've confirmed to be merged. 

What does that mean for the 10.1 SDK you have now?

There is a script for patching the SDK (requires internet access) within the edgeai-tidl-tools repo

• https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/update_target.sh
• note that this has gone by a different name for the .SH script in earlier release (setup_target_device.sh)
• This will update the C7x firmware (which is critical), a few libraries TIDL needs, related headers, and python packages

Then, you would compile your model with the tidl_tools downloaded from that 11_00_00_01 tag in the edgeai-tidl-tools repository. 

Please also see my FAQ on SDK and TIDL versioning: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1455079/faq-edge-ai-studio-is-sdk-version-important-for-edge-ai-and-ti-deep-learning-tidl-with-c7x-on-am6xa-socs-am62a-am67a-am68a-am68pa-am69a 

Note that the path here for custom hardware using different memory map (DDR <2GB) may have to consider an extra step here to patch the firmware-builder with 11.0 compatible sources for TIDL and dependencies. This is not relevant until you have custom PCB with AM62A to optimize the memory map for. 

BR,
Reese

Hi Reese,

I really appreciate your detailed explanation.
I'll look forward to the release of 11_00_00_01 TIDL tool.

Best Regards,
Youngheon

Hello,

I wanted to provide further update that these TIDL tools are currently expected to be available at the end of May. 

This timeline is determined by other TDA4x devices' 11.0 release schedule

Hi Reese,

Thank you for the additional update.
I understand that TIDL_tool 11_00_00_01 will be available by the end of May.
Thanks again!

Hi Reese,

Could you please share the current status of the sigmoid bug fix for AM62, if available?
I noticed that TIDL_tool 11_00_06_00 has been updated in edgeai-tidl-tools, but it seems that AM62A is not yet supported in this version.
- https://github.com/TexasInstruments/edgeai-tidl-tools/blob/11_00_06_00/docs/version_compatibility_table.md

Best Regards,
Youngheon

Hello,

There will be a backwards compatible release with that last non-zero value has an odd number -- I presume this will be 11_00_05_00. I expect this to be released within the next couple of days. I appreciate your patience. 

As stated before, AM62A will not have a public 11.0 SDK release. We are waiting for the backwards-compatible version to be released. 

This will include the sigmoid accuracy fix.

BR,
Reese

Hi Reese,

Thanks for your response.
I'll wait for the backwards-compatible version for AM62A.

Best Regards,
Youngheon

Youngheon, 

Quick status update after discussing this with Reese: this backward-compatible version was not in the TIDL tools 11_00_06 release last week.  We are now tracking 11_00_07 which is targeted the end of next week.

John

Hi John,

Thank you for letting us know about the upcoming new version.

Best Regards,
Youngheon

Update - due to some testing issues on this end, we have had to push out the TIDL release by a couple of days. The revised date is now middle of next week.  

John

Youngheon, 

Reese and I are still tracking this with the TIDL dev team, the release has encountered integration issues; consequently, the new date is 6/23.  We'll continue to keep open here until release.  

John

Hi John,

Thank you for the update. I’ll wait for the release on 6/23.

Best Regards,
Youngheon

Is it ready?

I looked for a release tag as described but didn't see one but maybe I'm not interpreting correctly.

Reese Grimsley said:

So basically, I expect to see a release tag for 11_00_00_01, where the odd fix number (1) indicates it is compatible with previous SDK release 10.1. You can see similar for 10_01_03_00 that is compatible with 10.0 SDK. 

Devin, 
I should have updated this yesterday.  As communicated by the TIDL dev team to me then, the patch got held up in regression testing. It is now slated for 6/30.  I recognize that this is a week-to-week slip for the past month. We can talk about this on the call tomorrow.  

John 

Hi John,

What can we do to improve the timing?

We need the patch ASAP.

Regards

Devin

Per 6/25 week,

What you are looking at in the image above is the correct location. However, if it does not appear there on 6/30, I will post here.  

As noted in a sidebar email with Reese, we do suggest to simply pick up the patch immediately upon availability and test it.  There should be no additional effort required. 

John 

Please see the latest release: 

github.com/.../11_00_07_00

Hi John,

I tested the sigmoid layer with edgai-tidl-tools 11_00_07_00, but the issue still persists.
When I compared the board dump files with the PC dump files generated by edgeai-tidl-tools, I found that some values in the sigmoid results appeared as 0xFF instead of 0x00.

I conducted the following test steps.

[TIDL Import]

- Installed edgeai-tidl-tools with tag 11_00_07_00, then imported the model.
- Then, dumped result files with PC_dsp_test_dl_algo.out.

[Board Setup]

- Downloaded tisdk-edgeai-image-am62a-evm.wic.xz (version 10.00.01.05) from TI website(https://www.ti.com/tool/PROCESSOR-SDK-AM62A)
- Created an SD Card using BalenaEtcher with the downloaded image.
- Installed edgeai-tidl-tools with tag 11_00_07_00 and ran update_target.sh after setting the following environment variables.
    - export UPDATE_OSRT_COMPONENTS=1
    - export UPDATE_FIRMWARE_AND_LIB=1
    - export SDK_VERSION=10_1
    - export SOC=am62a
    - export TISDK_IMAGE=edgeai
- Rebooted the board and dumped result files with /opt/tidl_test/TI_DEVICE_armv8_test_dl_algo_host_rt.out.

I would appreciate it if you could confirm whether these steps were performed correctly.

Best Regards,
YoungHeon

Hello Youngheon, 

Your steps look correct. Let's make sure that those critical files were in fact patched correctly.

Can you confirm the firmware's MD5SUM?

md5sum /lib/firmware/vision_apps_eaik/vx_app_rtos_linux_c7x_1.out
##unpatched:
bf8c9a403433557a5f7b234e01e83308  /lib/firmware/vision_apps_eaik/vx_app_rtos_linux_c7x_1.out
##patched
be35e318a440799133a2ba9192c14480  /lib/firmware/vision_apps_eaik/vx_app_rtos_linux_c7x_1.out

BR,
Reese

Hi Reese,

Thank you for confirming the steps.

I have verified the MD5 checksum of the firmware, and it matches.

root@am62axx-evm:~/workspace/youngheon# md5sum /lib/firmware/vision_apps_eaik/vx_app_rtos_linux_c7x_1.out
be35e318a440799133a2ba9192c14480  /lib/firmware/vision_apps_eaik/vx_app_rtos_linux_c7x_1.out

Then, I created a sample case for the sigmoid result mismatch on AM62A between the PC and the board.
I’ve attached the file containing the sample case. Please take a look and let me know if you need anything further.

am62a-sigmoid-sample.tar

Best Regards,
Youngheon

Hi Youngheon, 

Understood -- you have patched correctly then. Thank you for including the attached files, this saves us time and gets to a resolution faster. I have notified the TIDL team that this issue is still present. I will reproduce the error in parallel and we will work to address this

Edit: I've reproduced your traces exactly -- I see these 3 instances where the wrong mapping occurred. The previous layer value here was 0x80, so minimum signed int, and Sigmoid should map that to 0x00. Instead it is mapping to 0xff (wrong, regardless of dtype's sign) -- since this is unsigned output from Sigmoid, it is therefore max int. I'll raise this again with dev team. 

BR,
Reese

Per a Reese / John discussion, a few updates / clarifications:

1) The tool release at the end of June did address one corner case of the sigmoid issue.  

Details: in the original issue, we found that values of 0xC3 were mapping incorrectly to 0xFF. This was resolved going into the 11.0 release, and supplied as part of 11_00_07_00 TIDL tools

2) However, there is another corner case that Stradvision is encountering (and that TI / Reese has reproduced successfully): 

Details: In the now 11_00_07, we now see that values 0x80 mapping incorrectly to 0xFF. This is considered a separate issue

3) On this second issue, we are tracking via JIRA - TIDL-7892. This is planned for the Linux PSDK 11.1 release. 

Per 8/6 weekly, an update:

John H. Gardner said:

On this second issue, we are tracking via JIRA - TIDL-7892. This is planned for the Linux PSDK 11.1 release. 

This is the Edge AI SDK 11.1 for the AM62A device which is slated for the e/o August.  Early testing shows that this 11.1 release is fixing this corner case.