Hi,
Please help me resolve this problem on edgeai-tidl-tools(https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/docs/version_compatibility_table.md TAG 09_00_00_06).
I am constantly trying to compile models on my Linux machine using the ubuntu22 docker image for edgeai-tidl-tools(https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/docs/advanced_setup.md). The model for which the logs are pasted below is
Here are the compile logs:
root@a47824e03edc:/home/root/examples/osrt_python/ort# python3 onnxrt_ep.py -c -m od-ort-ssd-lite_mobilenetv2_fpn Available execution providers : ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider'] Running 1 Models - ['od-ort-ssd-lite_mobilenetv2_fpn'] Running_Model : od-ort-ssd-lite_mobilenetv2_fpn Downloading ../../../models/public/ssd-lite_mobilenetv2_fpn.onnx Converted model is valid! Downloading ../../../models/public/ssd-lite_mobilenetv2_fpn.prototxt Running shape inference on model ../../../models/public/ssd-lite_mobilenetv2_fpn.onnx ssd is meta arch name Number of OD backbone nodes = 159 Size of odBackboneNodeIds = 159 Preliminary subgraphs created = 1 Final number of subgraphs created are : 1, - Offloaded Nodes - 494, Total Nodes - 494 /home/root/tidl_tools/tidl_graphVisualiser_runtimes.out: error while loading shared libraries: libcgraph.so.6: cannot open shared object file: No such file or directory TIDL Meta PipeLine (Proto) File : ../../../models/public/ssd-lite_mobilenetv2_fpn.prototxt ssd Warning :: img_w & img_h or img_size is not provided as part of prior_box_param, hence using img_w = 512 and img_h = 512 in prior box decoding Warning :: img_w & img_h or img_size is not provided as part of prior_box_param, hence using img_w = 512 and img_h = 512 in prior box decoding Warning :: img_w & img_h or img_size is not provided as part of prior_box_param, hence using img_w = 512 and img_h = 512 in prior box decoding Warning :: img_w & img_h or img_size is not provided as part of prior_box_param, hence using img_w = 512 and img_h = 512 in prior box decoding Warning :: img_w & img_h or img_size is not provided as part of prior_box_param, hence using img_w = 512 and img_h = 512 in prior box decoding Warning :: img_w & img_h or img_size is not provided as part of prior_box_param, hence using img_w = 512 and img_h = 512 in prior box decoding Warning : Requested Output Data Convert Layer is not Added to the network, It is currently not Optimal Warning : Requested Output Data Convert Layer is not Added to the network, It is currently not Optimal ************** Frame index 1 : Running float import ************* /home/root/tidl_tools/tidl_graphVisualiser.out: error while loading shared libraries: libcgraph.so.6: cannot open shared object file: No such file or directory INFORMATION: [TIDL_ResizeLayer] Resize_153 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize. INFORMATION: [TIDL_ResizeLayer] Resize_156 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize. **************************************************** ** 2 WARNINGS 0 ERRORS ** **************************************************** The soft limit is 2048 The hard limit is 2048 MEM: Init ... !!! MEM: Init ... Done !!! 0.0s: VX_ZONE_INIT:Enabled 0.3s: VX_ZONE_ERROR:Enabled 0.7s: VX_ZONE_WARNING:Enabled 0.1892s: VX_ZONE_INIT:[tivxInit:185] Initialization Done !!! ********** Frame Index 1 : Running float inference ********** ********** Frame Index 2 : Running fixed point mode for calibration ********** Empty prototxt path, running calibration ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~ Processing config file #0 : /home/root/model-artifacts/od-ort-ssd-lite_mobilenetv2_fpn/tempDir/boxeslabels_tidl_io_.qunat_stats_config.txt Illegal instruction (core dumped) ***************** Calibration iteration number 0 started ************************ ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~ Processing config file #0 : /home/root/model-artifacts/od-ort-ssd-lite_mobilenetv2_fpn/tempDir/boxeslabels_tidl_io_.qunat_stats_config.txt Illegal instruction (core dumped) ***************** Calibration iteration number 0 completed ************************ ***************** Calibration iteration number 1 started ************************ ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~ Processing config file #0 : /home/root/model-artifacts/od-ort-ssd-lite_mobilenetv2_fpn/tempDir/boxeslabels_tidl_io_.qunat_stats_config.txt Illegal instruction (core dumped) ***************** Calibration iteration number 1 completed ************************ ***************** Calibration iteration number 2 started ************************ ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~ Processing config file #0 : /home/root/model-artifacts/od-ort-ssd-lite_mobilenetv2_fpn/tempDir/boxeslabels_tidl_io_.qunat_stats_config.txt Illegal instruction (core dumped) ***************** Calibration iteration number 2 completed ************************ ***************** Calibration iteration number 3 started ************************ ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~ Processing config file #0 : /home/root/model-artifacts/od-ort-ssd-lite_mobilenetv2_fpn/tempDir/boxeslabels_tidl_io_.qunat_stats_config.txt Illegal instruction (core dumped) ***************** Calibration iteration number 3 completed ************************ ***************** Calibration iteration number 4 started ************************ ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~ Processing config file #0 : /home/root/model-artifacts/od-ort-ssd-lite_mobilenetv2_fpn/tempDir/boxeslabels_tidl_io_.qunat_stats_config.txt Illegal instruction (core dumped) ***************** Calibration iteration number 4 completed ************************ ------------------ Network Compiler Traces ----------------------------- successful Memory allocation /home/root/tidl_tools/tidl_graphVisualiser.out: error while loading shared libraries: libcgraph.so.6: cannot open shared object file: No such file or directory INFORMATION: [TIDL_ResizeLayer] Resize_153 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize. INFORMATION: [TIDL_ResizeLayer] Resize_156 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize. TIDL ALLOWLISTING LAYER CHECK: TIDL_E_QUANT_STATS_NOT_AVAILABLE] tidl_quant_stats_tool.out fails to collect dynamic range. Please look into quant stats log. This model will get fault on target. **************************************************** ** 2 WARNINGS 1 ERRORS ** **************************************************** Completed_Model : 1, Name : od-ort-ssd-lite_mobilenetv2_fpn , Total time : 7182.27, Offload Time : 6312.97 , DDR RW MBs : 0, Output File : py_out_od-ort-ssd-lite_mobilenetv2_fpn_ADE_val_00001801.jpg MEM: Deinit ... !!! MEM: Alloc's: 27 alloc's of 276244535 bytes MEM: Free's : 27 free's of 276244535 bytes MEM: Open's : 0 allocs of 0 bytes MEM: Deinit ... Done !!!
The compile logs contain an error:
```
~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
Processing config file #0 : /home/root/model-artifacts/od-ort-ssd-lite_mobilenetv2_fpn/tempDir/boxeslabels_tidl_io_.qunat_stats_config.txt
Illegal instruction (core dumped)
```
Followed by:
```
TIDL ALLOWLISTING LAYER CHECK: TIDL_E_QUANT_STATS_NOT_AVAILABLE] tidl_quant_stats_tool.out fails to collect dynamic range. Please look into quant stats log. This model will get fault on target.
****************************************************
** 2 WARNINGS 1 ERRORS **
****************************************************
```
Following are the execution logs for the same model. It executes without error but gives incorrect output as warned by the compilation errors. On the other hand, the ARM execution gives correct output.
root@a47824e03edc:/home/root/examples/osrt_python/ort# python3 onnxrt_ep.py -m od-ort-ssd-lite_mobilenetv2_fpn Available execution providers : ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider'] Running 1 Models - ['od-ort-ssd-lite_mobilenetv2_fpn'] Running_Model : od-ort-ssd-lite_mobilenetv2_fpn libtidl_onnxrt_EP loaded 0x562b10a8fcc0 Final number of subgraphs created are : 1, - Offloaded Nodes - 494, Total Nodes - 494 The soft limit is 2048 The hard limit is 2048 MEM: Init ... !!! MEM: Init ... Done !!! 0.0s: VX_ZONE_INIT:Enabled 0.20s: VX_ZONE_ERROR:Enabled 0.22s: VX_ZONE_WARNING:Enabled 0.1964s: VX_ZONE_INIT:[tivxInit:185] Initialization Done !!! Saving image to ../../../output_images/ Completed_Model : 1, Name : od-ort-ssd-lite_mobilenetv2_fpn , Total time : 775.84, Offload Time : 775.76 , DDR RW MBs : 0, Output File : py_out_od-ort-ssd-lite_mobilenetv2_fpn_ADE_val_00001801.jpg MEM: Deinit ... !!! MEM: Alloc's: 27 alloc's of 90019807 bytes MEM: Free's : 27 free's of 90019807 bytes MEM: Open's : 0 allocs of 0 bytes MEM: Deinit ... Done !!!
Please let me know if this is some fault with the edgeai-tidl-tools or is there a step taken wrong.
Regards,
Vaibhav Kashera
