TDA4VH-Q1: The model converted from Import and Edge AI are different

Kuan-Hung Liu

Part Number: TDA4VH-Q1

Hi, I found that there are two tools to quantize model to TIDL format.
1. TIDL Import.out
2. Edge AI tool

We encounter that the model converted from Edge AI tool has poor accuracy.
The model converted from Import.out is good but it can't inference with Edge AI code.

Is there an example code that can inference the model from Import?

Or modify the Import model to Edge AI format.

over 1 year ago

0 Pratik Kedar over 1 year ago

TI__Mastermind 24041 points

Hi,

Can you please clarify the exact ask, so i can assist better.

0 Kuan-Hung Liu over 1 year ago in reply to Pratik Kedar

Prodigy 100 points

We have tried two methods and compared the results with ONNX's inference output:

We used the importer to convert the ONNX model and employed 'PC_dsp_test_dl_algo.out' for inference.
We utilized the 'Edge AI onnxrt_ep.py' to convert the ONNX model and employed the same script for inference.

The first method is similar to the output from ONNX, but the second method is almost completely unrelated. In addition, the article you mentioned only seems to discuss the comparison between TIDL float and quantized models. What I want to know is what went wrong with the second method? Or how can I make the results produced by the importer compatible with edge AI inference?

0 Pratik Kedar over 1 year ago in reply to Kuan-Hung Liu

TI__Mastermind 24041 points

Hi,

Lets address the first issue,

Kuan-Hung Liu said:
What I want to know is what went wrong with the second method?

The notebook by default clones tidl tools 9.0 SDK, one can check the SDK version for import tool usage, Secondly Calibration Image Set should be exactly same as well as the number of calibration iteration. Thirdly import/compilation params/flags should be exactly same.

0 Kuan-Hung Liu over 1 year ago in reply to Pratik Kedar

Prodigy 100 points

Pratik Kedar said:
Thirdly

Hi, regarding the third point about ensuring that both sides have the same params/flags, Edge AI's parameters are quite different from the importer, especially the default settings. I've noticed that Edge AI adds an extra data converter layer after the input layer and before the output layer. I would like to ask how to configure Edge AI's config to match the one below.

tidl_model_import.out /content/resnet18/config --modelType 2 \

--inputNetFile /content/resnet18_opset9.onnx --outputNetFile /content/resnet18/tidl_net.bin \

--outputParamsFile /content/resnet18/tidl_io_buff --inDataNorm 1 \

--inMean 123.675 116.28 103.53 --inScale 0.017125 0.017507 0.017429 \

--inData /content/resnet18/in_data_list.txt --inFileFormat 2 \

--tidlStatsTool /content/tidl_tools/PC_dsp_test_dl_algo.out \

--perfSimTool /content/tidl_tools/ti_cnnperfsim.out \

--graphVizTool /content/tidl_tools/tidl_graphVisualiser.out \

--inHeight 224 --inWidth 224 --inNumChannels 3 --numFrames 1 --numParamBits 8

0 Pratik Kedar over 1 year ago in reply to Kuan-Hung Liu

TI__Mastermind 24041 points

Can you check the TIDL-RT documentation here : https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-j721s2/08_05_00_11/exports/docs/tidl_j721s2_08_05_00_16/ti_dl/docs/user_guide_html/md_tidl_model_import.html

You can look for Config Pram For Format Conversion table for addDataConvertToNet flag.

Thanks

0 Kuan-Hung Liu over 1 year ago in reply to Pratik Kedar

Prodigy 100 points

Hi

Thanks for the document.

I noticed that the document of the TIDL importer has disappeared since 08_06_00_10 version.
Is there a difference between the 09_00_01_01 and the 08_06_00_10 version?

0 Pratik Kedar over 1 year ago in reply to Kuan-Hung Liu

TI__Mastermind 24041 points

Hi,

We expect user to use OSRT (tidl tools) for there custom model bring up use case, thus we have stopped adding TIDL RT doc post 8.5 SDK release.

You can use above link for TIDL RT documentation.

0 Kuan-Hung Liu over 1 year ago in reply to Pratik Kedar

Prodigy 100 points

Hi,
We discovered that the reason for the compiled model breaking (resulting in almost no correlation with ONNX) is the Dataconverter option. I'd like to inquire whether this could be due to the Dataconverter or if it's an option within the experiment. Alternatively, could there be another possible reason?

0 Pratik Kedar over 1 year ago in reply to Kuan-Hung Liu

TI__Mastermind 24041 points

Hi,

Currently am Out Of Office, please expect delay in my responses until 20/11/23.

Thank you for your patience.

0 Kuan-Hung Liu over 1 year ago

Prodigy 100 points

Hi,
This is the scatter plot which compare the output tensor between 32 bit TIDL model and ONNX model.
I assume they should be exactly same but it it not.
What thing can I do for fixing it?

0 Pratik Kedar over 1 year ago in reply to Kuan-Hung Liu

TI__Mastermind 24041 points

Not able to see the attached image, can you re share the same

0 Kuan-Hung Liu over 1 year ago in reply to Pratik Kedar

Prodigy 100 points

0 Pratik Kedar over 1 year ago in reply to Kuan-Hung Liu

TI__Mastermind 24041 points

Hi Kuan,

Can you let us know what value did you set for quantization bits for above exp ?

This to cross out possibility of you not comparing 16bit inference result in case.

Thank you

0 Kuan-Hung Liu over 1 year ago in reply to Pratik Kedar

Prodigy 100 points

Hi

Below is our config for compiling

32-bit

modelType          = 2
inputNetFile       = "../../../models/3DOD/resize_ratio.onnx"
outputNetFile      = "./tidl_models/resizeModelOutput_200img_2iter_16_16bits/resizeModel.bin"
outputParamsFile   = "./tidl_models/resizeModelOutput_200img_2iter_16_16bits/resizeModel_io_"
inDataNorm  = 1
numParamBits = 32
numFeatureBits = 16
inMean = 123.675 116.28 103.53
inScale = 0.017125 0.017507 0.017429
resizeWidth = 320
resizeHeight = 192
biasCalibrationIterations = 2
inWidth  = 320
inHeight = 192 
inNumChannels = 3
inFileFormat = 2
numFrames = 200
inData = ../../../datasets/3DOD/resize_img/input_lists/all_compile_img.txt
perfSimConfig = ./config/device_config.cfg

8bit

modelType          = 2
inputNetFile       = "../../../models/3DOD/resize_ratio.onnx"
outputNetFile      = "./tidl_models/resizeModelOutput_200img_2iter_16_16bits/resizeModel.bin"
outputParamsFile   = "./tidl_models/resizeModelOutput_200img_2iter_16_16bits/resizeModel_io_"
inDataNorm  = 1
numParamBits = 8
numFeatureBits = 8
inMean = 123.675 116.28 103.53
inScale = 0.017125 0.017507 0.017429
resizeWidth = 320
resizeHeight = 192
biasCalibrationIterations = 2
inWidth  = 320
inHeight = 192 
inNumChannels = 3
inFileFormat = 2
numFrames = 200
inData = ../../../datasets/3DOD/resize_img/input_lists/all_compile_img.txt
perfSimConfig = ./config/device_config.cfg

Processors

Processors forum

TDA4VH-Q1: The model converted from Import and Edge AI are different