• State TI Thinks Resolved
  • Locked Locked
  • Replies 8 replies
  • Answers 1 answer
  • Subscribers 100 subscribers
  • Views 244 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VH-Q1: Artifacts built with the leakage patch do not run on device.

Alex Spivakovsky
Prodigy 160 points
Part Number: TDA4VH-Q1

readme_9.2.md.tar.gz

Tool/software:

Hi Christina,

First, I would like to give you an insight on the problem and the work that was done already. There is an memory leakage issue, the solution was provided by TI was a patch for both SDK9.2 and also tidl tools. Those

tools version (10.01.03.03) is not listed on tidl_tools github and was a part of the patch. In total I received the following packages:

1. c7x-mma-tidl.tar.gz

2. mmalib_obj_C7120_10_01_01_00.tar.gz

3. tidl_tools.tar.gz

4. ONNX_Dependencies.zip with the following contents:

  • onnx_1.15.0_aragoj7.tar.gz
  • onnx_1.15.0_x86_u22.tar.gz
  • onnxruntime_tidl-1.15.0-cp310-cp310-linux_aarch64.whl
  • onnxruntime_tidl-1.15.0-cp310-cp310-linux_x86_64.whl

I laso received a readme file (attached) with exact instructions how to compile the SDK and also build the tidl_tools environment.

The SDK was built sucessfuly and the created SDK card is up and running. I followed all the requirements in steps SDK SETUP and EVM SETUP. 

For the tidl tools there is a separate section in the readme: TIDL HOST EMULATION SETUP. I think there is a typo in step 2, as it says to replace the existing onnx_1.14.0_aragoj7 with onnx_1.15.0_aragoj7. 

However for tidl_tools the osrt_deps folder does not include those, so I just used the onnx_1.15.0_x86_u22 included in the ONNX_Dependencies.zip.

Also I used the latest 10.01.04.00 version, downloaded it and used it as a starting point to tidl_tools. I still followed the steps 1 and 2 even though I already had the onnxruntime_tidl1.15 and onnx_1.15.0_x86_u22.

I finished with replacing the tidl folder with the one provided by TI as requested in step 3.

The artifacts were built without issues, however I receive errors like Graph failed on 2 different boards built with the patch.

Thank you,

Alex.

  • 0
    TI__Expert 4280 points

    Hi Alex,

    I will add this to the related Jira TIDL-7166 and keep you updated on any updates I have when recreating.

    Warm regards,

    Christina

  • 0 in reply to Christina Kuruvilla
    TI__Expert 4280 points

    Hi Alex,

    When using the tools version 10.01.03.03, were you able to see the issues? Also, could you share all the files you received for the patch that you mentioned?  Please send this through email if possible.

    My colleague Wen will also be assisting with this issue, and verifying if the issue is related to the patch or not.

    Also, is the model the same model FvModel_epoch10_1739095521.onnx ?

    Warm regards,

    Christina 

  • 0 in reply to Christina Kuruvilla
    Prodigy 160 points

    Hi Christina,

    I see the issues while running the compiled artifacts on the device.

    Yes, the artifacts were built for the same ONNX (I built artifacts also for other models in our pipeline - neither of them work).

    TI PATCH.tar.gz

    I attached the TI's patch here as the contents size is large to be sent over mail.

    The zip file contains files and instructions how to build the SDK, prepare the EVM and tidl tools setup.

    Also, the provided tools 10_01_03_03 were compiled for CPU and not for GPU, so the compilation runs very slow. However this is a side effect issue.

    Thank you,

    Alex. 

  • 0 in reply to Alex Spivakovsky
    TI__Expert 4280 points

    Hi Alex,

    Thank you for the information. We are setting up the EVM and SDK based on your setup, and will try to reproduce. I will send you updates as we move forward.

    Warm regards,

    Christina

  • 0 in reply to Christina Kuruvilla
    TI__Expert 4280 points

    Hi Alex,

    Could you share the steps you take to run on the board? Are you using your full_test script? Can you share all the errors you get?

    We want to make sure we are aligned on how you recreated it. Currently, we have been testing it out with our TIDL examples with the patch as a baseline. 

    Warm regards,

    Christina

  • 0 in reply to Christina Kuruvilla
    Prodigy 160 points

    Hi Christina,

    I'm using a script pretty similar to a full_test, but targeted to build a real artifacts - it uses a 128 images calibration dataset I shared with you. In the second stage I used the compiled artifacts to run inference on the real board while running our pipeline. I received errors like: graph is not ready to run, I can send you the exact errors I got on Tuesday only, as I need to replace the SD card manually on the board and I'm working at home till then.

    Thank you,

    Alex.

  • 0 in reply to Alex Spivakovsky
    TI__Expert 4280 points

    Thanks Alex. 

    We are now testing this on the 11.0 prerelease to validate that there is no memory leak and everything under TIDL works properly. No rush in getting the exact errors but please send whenever it is convenient for you. 

    Warm regards,

    Christina

  • 0 in reply to Christina Kuruvilla
    TI__Expert 4280 points

    Hi Alex,

    11.0 release is available. Please check on the new version and let me know if you have any issues. It has the fix for the memory leak.

    Warm regards,

    Christina