I'm trying to build the custom model from ONNX using edgeai-tidl-tools, but encountered the problem.
Here is the visualization of ONNX layers:
Here is the log message with max debug level:
tidl_tools_path = /home/root/tidl_tools artifacts_folder = /home/root/model-artifacts/search/ tidl_tensor_bits = 8 debug_level = 6 num_tidl_subgraphs = 16 tidl_denylist = tidl_denylist_layer_name = tidl_denylist_layer_type = tidl_allowlist_layer_name = model_type = tidl_calibration_accuracy_level = 7 tidl_calibration_options:num_frames_calibration = 20 tidl_calibration_options:bias_calibration_iterations = 50 mixed_precision_factor = -1.000000 model_group_id = 0 power_of_2_quantization = 2 enable_high_resolution_optimization = 0 pre_batchnorm_fold = 1 add_data_convert_ops = 0 output_feature_16bit_names_list = m_params_16bit_names_list = reserved_compile_constraints_flag = 1601 ti_internal_reserved_1 = ****** WARNING : Network not identified as Object Detection network : (1) Ignore if network is not Object Detection network (2) If network is Object Detection network, please specify "model_type":"OD" as part of OSRT compilation options****** Supported TIDL layer type --- Conv -- /Conv Supported TIDL layer type --- Reshape -- /connect_model/cls_encode/Reshape Unsupported (TIDL check) TIDL layer type --- Transpose Preliminary subgraphs created = 1 Final number of subgraphs created are : 1, - Offloaded Nodes - 2, Total Nodes - 3 TIDL ALLOWLISTING LAYER CHECK -- [TIDL_TransposeLayer] should be removed in import process. If not, this model will not work! /home/root/tidl_tools/tidl_graphVisualiser_runtimes.out: error while loading shared libraries: libcgraph.so.6: cannot open shared object file: No such file or directory Running runtimes graphviz - /home/root/tidl_tools/tidl_graphVisualiser_runtimes.out /home/root/model-artifacts/search//allowedNode.txt /home/root/model-artifacts/search//tempDir/graphvizInfo.txt /home/root/model-artifacts/search//tempDir/runtimes_visualization.svg *** In TIDL_createStateImportFunc *** Compute on node : TIDLExecutionProvider_TIDL_0_0 0, Conv, 3, 1, template_features, /Conv_output_0 1, Reshape, 2, 1, /Conv_output_0, /connect_model/cls_encode/Reshape_output_0 Input tensor name - template_features Output tensor name - /connect_model/cls_encode/Reshape_output_0 Initialized In TIDL_onnxRtImportInit subgraph_name=_connect_model_cls_encode_Reshape_output_0 Layer 0, subgraph id _connect_model_cls_encode_Reshape_output_0, name=/connect_model/cls_encode/Reshape_output_0 Layer 1, subgraph id _connect_model_cls_encode_Reshape_output_0, name=template_features In TIDL_runtimesOptimizeNet: LayerIndex = 4, dataIndex = 3 ************** Frame index 1 : Running float import ************* In TIDL_runtimesPostProcessNet /home/root/tidl_tools/tidl_graphVisualiser.out: error while loading shared libraries: libcgraph.so.6: cannot open shared object file: No such file or directory **************************************************** ** ALL MODEL CHECK PASSED ** **************************************************** ************ in TIDL_subgraphRtCreate ************ The soft limit is 2048 The hard limit is 2048 MEM: Init ... !!! MEM: Init ... Done !!! 0.0s: VX_ZONE_INIT:Enabled 0.7s: VX_ZONE_ERROR:Enabled 0.12s: VX_ZONE_WARNING:Enabled 0.2171s: VX_ZONE_INIT:[tivxInit:184] Initialization Done !!! TIDL_initDebugTraceParams Done Alg Alloc for Layer # - 0 Alg Alloc for Layer # - 1 Alg Alloc for Layer # - 2 Alg Alloc for Layer # - 3 Alg Alloc for Layer # - 4 Alg Alloc for Layer # - 5 -------------------------------------------- TIDL Memory size requiement (record wise): MemRecNum , Space , Attribute , Size(KBytes) 0 , DDR Cacheable, Persistent , 14.84 1 , DDR Cacheable, Persistent , 0.14 2 , DDR Cacheable, Scratch , 16.00 3 , DDR Cacheable, Scratch , 4.00 4 , DDR Cacheable, Scratch , 56.00 5 , DDR Cacheable, Persistent , 258.00 6 , DDR Cacheable, Scratch , 448.62 7 , DDR Cacheable, Scratch , 0.12 8 , DDR Cacheable, Scratch , 96.12 9 , DDR Cacheable, Scratch , 515.00 10 , DDR Cacheable, Scratch , 0.12 11 , DDR Cacheable, Persistent , 201.12 12 , DDR Cacheable, Scratch , 512.25 13 , DDR Cacheable, Persistent , 0.12 14 , DDR Cacheable, Persistent , 675.81 -------------------------------------------- Total memory size requirement (space wise): Mem Space , Size(KBytes) DDR Cacheable, 2798.29 -------------------------------------------- NOTE: Memory requirement in host emulation can be different from the same on EVM To get the actual TIDL memory requirement make sure to run on EVM with debugTraceLevel = 2 -------------------------------------------- Alg Init for Layer # - 0 out of 5 Alg Init for Layer # - 1 out of 5 Alg Init for Layer # - 2 out of 5 Alg Init for Layer # - 3 out of 5 Alg Init for Layer # - 4 out of 5 Alg Init for Layer # - 5 out of 5 PREEMPTION: Adding a new priority object for targetPriority = 2, handle = 0x7feee47a3000 PREEMPTION: Now total number of priority objects = 1 at priorityId = 2, with new memRec of base = 0x7fee30017000 and size = 128 PREEMPTION: Requesting context memory addr for handle 0x7feee47a3000, return Addr = 0x7fee30017000 ************ TIDL_subgraphRtCreate done ************ Warning : Couldn't find corresponding ioBuf tensor for onnx tensor with matching name ******* In TIDL_subgraphRtInvoke ******** TIDL_activate is called with handle : e47a3000 Starting Layer # - 1 Processing Layer # - 1 End of Layer # - 1 with outPtrs[0] = 0x7fee30167000 Starting Layer # - 2 Processing Layer # - 2 End of Layer # - 2 with outPtrs[0] = 0x7fee30177000 Starting Layer # - 3 Processing Layer # - 3 End of Layer # - 3 with outPtrs[0] = 0x7fee30187000 Starting Layer # - 4 Processing Layer # - 4 End of Layer # - 4 with outPtrs[0] = 0x7fee302e0000 TIDL_process is completed with handle : e47a3000 Layer, Layer Cycles,kernelOnlyCycles, coreLoopCycles,LayerSetupCycles,dmaPipeupCycles, dmaPipeDownCycles, PrefetchCycles,copyKerCoeffCycles,LayerDeinitCycles,LastBlockCycles, paddingTrigger, paddingWait,LayerWithoutPad,LayerHandleCopy, BackupCycles, RestoreCycles, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Sum of Layer Cycles 0 Sub Graph Stats 12.000000 38971.000000 164.000000 ******* TIDL_subgraphRtInvoke done ******** ********** Frame Index 1 : Running float inference ********** Segmentation fault (core dumped)
Importantly, that if Transpose layer is removed from the onnx model, tidl compilation is successful. Here is the log message of succesfull compilation when "Transpose" layer is removed from model:
tidl_tools_path = /home/root/tidl_tools artifacts_folder = /home/root/model-artifacts/search/ tidl_tensor_bits = 8 debug_level = 6 num_tidl_subgraphs = 16 tidl_denylist = tidl_denylist_layer_name = tidl_denylist_layer_type = tidl_allowlist_layer_name = model_type = tidl_calibration_accuracy_level = 7 tidl_calibration_options:num_frames_calibration = 20 tidl_calibration_options:bias_calibration_iterations = 50 mixed_precision_factor = -1.000000 model_group_id = 0 power_of_2_quantization = 2 enable_high_resolution_optimization = 0 pre_batchnorm_fold = 1 add_data_convert_ops = 0 output_feature_16bit_names_list = m_params_16bit_names_list = reserved_compile_constraints_flag = 1601 ti_internal_reserved_1 = ****** WARNING : Network not identified as Object Detection network : (1) Ignore if network is not Object Detection network (2) If network is Object Detection network, please specify "model_type":"OD" as part of OSRT compilation options****** Supported TIDL layer type --- Conv -- /Conv Supported TIDL layer type --- Reshape -- /connect_model/cls_encode/Reshape Preliminary subgraphs created = 1 Final number of subgraphs created are : 1, - Offloaded Nodes - 2, Total Nodes - 2 /home/root/tidl_tools/tidl_graphVisualiser_runtimes.out: error while loading shared libraries: libcgraph.so.6: cannot open shared object file: No such file or directory Running runtimes graphviz - /home/root/tidl_tools/tidl_graphVisualiser_runtimes.out /home/root/model-artifacts/search//allowedNode.txt /home/root/model-artifacts/search//tempDir/graphvizInfo.txt /home/root/model-artifacts/search//tempDir/runtimes_visualization.svg *** In TIDL_createStateImportFunc *** Compute on node : TIDLExecutionProvider_TIDL_0_0 0, Conv, 3, 1, template_features, /Conv_output_0 1, Reshape, 2, 1, /Conv_output_0, /connect_model/cls_encode/Reshape_output_0 Input tensor name - template_features Output tensor name - /connect_model/cls_encode/Reshape_output_0 Initialized In TIDL_onnxRtImportInit subgraph_name=_connect_model_cls_encode_Reshape_output_0 Layer 0, subgraph id _connect_model_cls_encode_Reshape_output_0, name=/connect_model/cls_encode/Reshape_output_0 Layer 1, subgraph id _connect_model_cls_encode_Reshape_output_0, name=template_features In TIDL_runtimesOptimizeNet: LayerIndex = 4, dataIndex = 3 ************** Frame index 1 : Running float import ************* In TIDL_runtimesPostProcessNet /home/root/tidl_tools/tidl_graphVisualiser.out: error while loading shared libraries: libcgraph.so.6: cannot open shared object file: No such file or directory **************************************************** ** ALL MODEL CHECK PASSED ** **************************************************** ************ in TIDL_subgraphRtCreate ************ The soft limit is 2048 The hard limit is 2048 MEM: Init ... !!! MEM: Init ... Done !!! 0.0s: VX_ZONE_INIT:Enabled 0.6s: VX_ZONE_ERROR:Enabled 0.10s: VX_ZONE_WARNING:Enabled 0.1829s: VX_ZONE_INIT:[tivxInit:184] Initialization Done !!! TIDL_initDebugTraceParams Done Alg Alloc for Layer # - 0 Alg Alloc for Layer # - 1 Alg Alloc for Layer # - 2 Alg Alloc for Layer # - 3 Alg Alloc for Layer # - 4 Alg Alloc for Layer # - 5 -------------------------------------------- TIDL Memory size requiement (record wise): MemRecNum , Space , Attribute , Size(KBytes) 0 , DDR Cacheable, Persistent , 14.84 1 , DDR Cacheable, Persistent , 0.14 2 , DDR Cacheable, Scratch , 16.00 3 , DDR Cacheable, Scratch , 4.00 4 , DDR Cacheable, Scratch , 56.00 5 , DDR Cacheable, Persistent , 258.00 6 , DDR Cacheable, Scratch , 448.62 7 , DDR Cacheable, Scratch , 0.12 8 , DDR Cacheable, Scratch , 96.12 9 , DDR Cacheable, Scratch , 515.00 10 , DDR Cacheable, Scratch , 0.12 11 , DDR Cacheable, Persistent , 201.12 12 , DDR Cacheable, Scratch , 512.25 13 , DDR Cacheable, Persistent , 0.12 14 , DDR Cacheable, Persistent , 675.81 -------------------------------------------- Total memory size requirement (space wise): Mem Space , Size(KBytes) DDR Cacheable, 2798.29 -------------------------------------------- NOTE: Memory requirement in host emulation can be different from the same on EVM To get the actual TIDL memory requirement make sure to run on EVM with debugTraceLevel = 2 -------------------------------------------- Alg Init for Layer # - 0 out of 5 Alg Init for Layer # - 1 out of 5 Alg Init for Layer # - 2 out of 5 Alg Init for Layer # - 3 out of 5 Alg Init for Layer # - 4 out of 5 Alg Init for Layer # - 5 out of 5 PREEMPTION: Adding a new priority object for targetPriority = 2, handle = 0x7f8a4b00d000 PREEMPTION: Now total number of priority objects = 1 at priorityId = 2, with new memRec of base = 0x7f8998084000 and size = 128 PREEMPTION: Requesting context memory addr for handle 0x7f8a4b00d000, return Addr = 0x7f8998084000 ************ TIDL_subgraphRtCreate done ************ Warning : Couldn't find corresponding ioBuf tensor for onnx tensor with matching name ******* In TIDL_subgraphRtInvoke ******** TIDL_activate is called with handle : 4b00d000 Starting Layer # - 1 Processing Layer # - 1 End of Layer # - 1 with outPtrs[0] = 0x7f89981d4000 Starting Layer # - 2 Processing Layer # - 2 End of Layer # - 2 with outPtrs[0] = 0x7f89981e4000 Starting Layer # - 3 Processing Layer # - 3 End of Layer # - 3 with outPtrs[0] = 0x7f89981f4000 Starting Layer # - 4 Processing Layer # - 4 End of Layer # - 4 with outPtrs[0] = 0x7f899834d000 TIDL_process is completed with handle : 4b00d000 Layer, Layer Cycles,kernelOnlyCycles, coreLoopCycles,LayerSetupCycles,dmaPipeupCycles, dmaPipeDownCycles, PrefetchCycles,copyKerCoeffCycles,LayerDeinitCycles,LastBlockCycles, paddingTrigger, paddingWait,LayerWithoutPad,LayerHandleCopy, BackupCycles, RestoreCycles, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Sum of Layer Cycles 0 Sub Graph Stats 12.000000 38914.000000 179.000000 ******* TIDL_subgraphRtInvoke done ******** ********** Frame Index 1 : Running float inference ********** [array(0.646822, dtype=float32)] Video processed and saved to outputs/test.mp4 ************ in TIDL_subgraphRtDelete ************ TIDL_deactivate is called with handle : 4b00d000 PREEMPTION: Removing priroty object with handle = 0x7f8a4b00d000 and targetPriority = 2, Number of obejcts left are = 0, removed object with base = 0x7f8998084000 and size =128 MEM: Deinit ... !!! MEM: Alloc's: 26 alloc's of 4658204 bytes MEM: Free's : 26 free's of 4658204 bytes MEM: Open's : 0 allocs of 0 bytes MEM: Deinit ... Done !!!
During the import process next two steps caught my attention:
Unsupported (TIDL check) TIDL layer type --- Transpose
TIDL ALLOWLISTING LAYER CHECK -- [TIDL_TransposeLayer] should be removed in import process. If not, this model will not work!
And I am wondering how the compilation process should be changed ?