TDA4VH-Q1: Compiled onnx .bin artifact cannot has poor accuracy compared to the original model.

ST T

Prodigy 190 points

Part Number: TDA4VH-Q1

Hi,

This is the continuation discussion of this thread:

e2e.ti.com/.../4871255

I compiled a onnx model to .bin file and do inference using the compiled artifacts.

I compared the result of tidl with the original onnx model output, this is my graph comparison graph:

I have 21 outputs, I would expect the output will be a bottom-left to top-right line.

The experiment I was running is using a single image as calibration input, and I also test the result with the exact same image.

I think the above setting is the easiest case and the quantization should be doing good, but it turns out not.

This is the link to my model:

drive.google.com/.../1mDr9f7LKvgiXw9hf6fpAuHZYWCuCIw7U

This is my config for `delegate_options` (github.com/.../onnxrt_ep.py) for compiling, this is config can compile

{'tidl_tools_path': '/home/root/tidl_tools', 'artifacts_folder': '../../../model-artifacts//modelResize_onnx/', 'platform': 'J7', 'version': '7.2', 'tensor_bits': 8, 'debug_level': 0, 'max_num_subgraphs': 16, 'deny_list': '', 'deny_list:layer_type': '', 'deny_list:layer_name': '', 'model_type': '', 'accuracy_level': 1, 'advanced_options:calibration_frames': 1, 'advanced_options:calibration_iterations': 1, 'advanced_options:output_feature_16bit_names_list': '', 'advanced_options:params_16bit_names_list': '', 'advanced_options:mixed_precision_factor': -1, 'advanced_options:quantization_scale_type': 0, 'advanced_options:high_resolution_optimization': 0, 'advanced_options:pre_batchnorm_fold': 1, 'ti_internal_nc_flag': 1601, 'advanced_options:activation_clipping': 1, 'advanced_options:weight_clipping': 1, 'advanced_options:bias_calibration': 1, 'advanced_options:add_data_convert_ops': 3, 'advanced_options:channel_wise_quantization': 0, 'advanced_options:inference_mode': 0, 'advanced_options:num_cores': 1}

btw, if i change it to

"advanced_options:quantization_scale_type": 0,

"advanced_options:add_data_convert_ops": 3,

just for testing, it cannot compile, the error:

Running_Model :  modelResize_onnx  


Running shape inference on model ../../../models/public/../onnx/resizeStrip.onnx 


Preliminary subgraphs created = 1 
Final number of subgraphs created are : 1, - Offloaded Nodes - 143, Total Nodes - 143 

 ************** Frame index 1 : Running float import ************* 
INFORMATION: [TIDL_ResizeLayer] Resize_82 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] Resize_85 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
****************************************************
**          2 WARNINGS          0 ERRORS          **
****************************************************
The soft limit is 2048
The hard limit is 2048
MEM: Init ... !!!
MEM: Init ... Done !!!
 0.0s:  VX_ZONE_INIT:Enabled
 0.5s:  VX_ZONE_ERROR:Enabled
 0.6s:  VX_ZONE_WARNING:Enabled
 0.1818s:  VX_ZONE_INIT:[tivxInit:185] Initialization Done !!!
 0.157478s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = -1121
Segmentation fault (core dumped)

Could you help me to compile the model?

Thanks!

over 1 year ago

0 Pratik Kedar over 1 year ago

TI__Mastermind 24041 points

Hi,

Lets address the spotting issue of this thread (accuracy)

Let me summaries this issue first,

Per your above description you are observing differences between ONNX output of model vs TIDL RT output.

Please note that, there will be slight deviation in output dumps, as model inference on target ran in 8 bit quantized flow and inference result your referring, if they are float 32.

Here the colab notebook we have create for float vs quantized comparison.

https://colab.research.google.com/github/kumardesappan/colab-notebooks/blob/main/tidlrt_tensor_debug.ipynb#scrollTo=xzB8l5BIgQ3V

Could you use your model for the same and let us know the observations.

0 ST T over 1 year ago in reply to Pratik Kedar

Prodigy 190 points

Hi Pratik,

We can convert the model by mimicking your notebook example.

May I know the difference between compiling models using edgeai-tidl-tools vs tidl-importer?
Will edgeai-tidl-tools call tidl_model_import.out internally?

Some confusion I notice:
1. If I want to use edgeai-tidl-tools with onnx after "compiling", it will still need a .onnx file when inferencing, why?

I will expect we don't need the original .onnx file after we do the compilation

2. Can I use tidl_model_import.out to convert an .onnx model, then use Ort::Session cpp code similar like in this example (https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/examples/osrt_cpp/ort/onnx_main.cpp)?

Thanks for your help!!!

0 Pratik Kedar over 1 year ago in reply to ST T

TI__Mastermind 24041 points

ST T said:
May I know the difference between compiling models using edgeai-tidl-tools vs tidl-importer?

Per my understanding, there shouldn't be differences, If this observation coming from edgeai-tidl-tools usage in model accuracy vs using model import flow based in notebooks, then this might be because of compilation flags related setting or num of calibration iteration or calib images related differences.

ST T said:
1. If I want to use edgeai-tidl-tools with onnx after "compiling", it will still need a .onnx file when inferencing, why?

Per my understanding for ARM only mode this is required, however i believe for TIDL Offload generated bin files are only required.

ST T said:
2. Can I use tidl_model_import.out to convert an .onnx model, then use Ort::Session cpp code similar like in this example (https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/examples/osrt_cpp/ort/onnx_main.cpp)?

You can refer to cpp based flow to achieve model compilation and inference as well.

Let me know if you need any assistance on model accuracy front, if this issue is addressed can we close this thread ?

Thank you

0 ST T over 1 year ago in reply to Pratik Kedar

Prodigy 190 points

Hi patrik,

The current status of ours is that we can achieve better quantization by using your notebook, but cannot with https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/examples/osrt_python/ort/onnxrt_ep.py

So our decision we be using `tidl_model_import.out` similar to your notebook to compile and using Ort::Session to read.

But `Ort::Session` seems to have to provide the onnx model.

btw, if i use `tidl_model_import.out`, it can accept onnx or tflite, what is the tensor layout (nchw or nhwc) of the output .bin

when importing the model, how can it know it is in nchw or nhwc?

Thanks!

0 Pratik Kedar over 1 year ago in reply to ST T

TI__Mastermind 24041 points

Hi STT,

ST T said:
The current status of ours is that we can achieve better quantization by using your notebook, but cannot with https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/examples/osrt_python/ort/onnxrt_ep.py

Can you confirm few things mentioned in my reply here :

Pratik Kedar said:
ST T said:
May I know the difference between compiling models using edgeai-tidl-tools vs tidl-importer?

Per my understanding, there shouldn't be differences, If this observation coming from edgeai-tidl-tools usage in model accuracy vs using model import flow based in notebooks, then this might be because of compilation flags related setting or num of calibration iteration or calib images related differences.

The default TIDL Format is NCHW, and per my understanding same is maintained trough out.

Processors

Processors forum

TDA4VH-Q1: Compiled onnx .bin artifact cannot has poor accuracy compared to the original model.