root@tda4vm-sk:/opt/edgeai-tidl-tools/examples/osrt_python/ort# python3 onnxrt_ep.py -m cl-ort-deit-tiny Available execution providers : ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider'] Running 1 Models - ['cl-ort-deit-tiny'] Running_Model : cl-ort-deit-tiny libtidl_onnxrt_EP loaded 0x3c426370 artifacts_folder = ../../../model-artifacts//cl-ort-deit-tiny/ debug_level = 2 target_priority = 0 max_pre_empt_delay = 340282346638528859811704183484516925440.000000 Final number of subgraphs created are : 1, - Offloaded Nodes - 580, Total Nodes - 580 In TIDL_createStateInfer Compute on node : TIDLExecutionProvider_TIDL_0_0 ************ in TIDL_subgraphRtCreate ************ APP: Init ... !!! 373.990213 s: MEM: Init ... !!! 373.990272 s: MEM: Initialized DMA HEAP (fd=5) !!! 373.990410 s: MEM: Init ... Done !!! 373.990429 s: IPC: Init ... !!! 374.060763 s: IPC: Init ... Done !!! REMOTE_SERVICE: Init ... !!! REMOTE_SERVICE: Init ... Done !!! 374.072971 s: GTC Frequency = 200 MHz APP: Init ... Done !!! 374.076349 s: VX_ZONE_INIT:Enabled 374.076381 s: VX_ZONE_ERROR:Enabled 374.076387 s: VX_ZONE_WARNING:Enabled 374.083326 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-0 374.083466 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-1 374.083546 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-2 374.083617 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-3 374.083627 s: VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!! 374.086810 s: VX_ZONE_INIT:[tivxHostInitLocal:106] Initialization Done for HOST !!! [C7x_1 ] 374.713895 s: PREEMPTION: Requesting memory of size 8159232 for targetPriority = 512 [C7x_1 ] 374.713921 s: [C7x_1 ] 374.713940 s: -------------------------------------------- [C7x_1 ] 374.713964 s: TIDL Memory size requiement (record wise): [C7x_1 ] 374.714005 s: MemRecNum , Space , Attribute , Alignment , Size(KBytes), BasePtr [C7x_1 ] 374.714050 s: 0 , DDR Cacheable , Persistent , 128, 19.27 , 0x00000000 [C7x_1 ] 374.714092 s: 1 , DDR Cacheable , Persistent , 128, 0.65 , 0x00000000 [C7x_1 ] 374.714133 s: 2 , L1D , Scratch , 128, 16.00 , 0x00000000 [C7x_1 ] 374.714173 s: 3 , L2 , Scratch , 128, 448.00 , 0x00000000 [C7x_1 ] 374.714213 s: 4 , L3/MSMC , Scratch , 128, 7968.00 , 0x00000000 [C7x_1 ] 374.714253 s: 5 , DDR Cacheable , Persistent , 128, 20158.13, 0x00000000 [C7x_1 ] 374.714297 s: 6 , DDR Cacheable , Scratch , 128, 67.75 , 0x00000000 [C7x_1 ] 374.714340 s: 7 , DDR Cacheable , Persistent , 128, 589.25 , 0x00000000 [C7x_1 ] 374.714381 s: 8 , DDR Cacheable , Scratch , 128, 0.13 , 0x00000000 [C7x_1 ] 374.714421 s: 9 , DDR Cacheable , Scratch , 128, 3.13 , 0x00000000 [C7x_1 ] 374.714461 s: 10 , DDR Cacheable , Persistent , 128, 5355.03 , 0x00000000 [C7x_1 ] 374.714502 s: 11 , DDR Cacheable , Scratch , 128, 512.25 , 0x00000000 [C7x_1 ] 374.714542 s: 12 , DDR Cacheable , Persistent , 128, 7968.00 , 0x00000000 [C7x_1 ] 374.714581 s: 13 , DDR Cacheable , Persistent , 128, 2770.78 , 0x00000000 [C7x_1 ] 374.714621 s: 14 , DDR Cacheable , Persistent , 128, 0.00 , 0x00000000 [C7x_1 ] 374.714661 s: 15 , DDR Cacheable , Persistent , 128, 5648.38 , 0x00000000 [C7x_1 ] 374.714689 s: -------------------------------------------- [C7x_1 ] 374.714713 s: Total memory size requirement (space wise): [C7x_1 ] 374.714732 s: Mem Space , Size(KBytes) [C7x_1 ] 374.714748 s: L1D , 16.00 [C7x_1 ] 374.714764 s: L2 , 448.00 [C7x_1 ] 374.714780 s: L3/MSMC , 7968.00 [C7x_1 ] 374.714797 s: DDR Cacheable, 43092.73 [C7x_1 ] 374.714819 s: -------------------------------------------- [C7x_1 ] 374.714854 s: NOTE: Memory requirement in host emulation can be different from the same on EVM [C7x_1 ] 374.714891 s: To get the actual TIDL memory requirement make sure to run on EVM with [C7x_1 ] 374.714914 s: debugTraceLevel = 2 [C7x_1 ] 374.714923 s: [C7x_1 ] 374.714942 s: -------------------------------------------- [C7x_1 ] 374.715405 s: TIDL init call from ivision API [C7x_1 ] 374.715422 s: [C7x_1 ] 374.715439 s: -------------------------------------------- [C7x_1 ] 374.715463 s: TIDL Memory size requiement (record wise): [C7x_1 ] 374.715503 s: MemRecNum , Space , Attribute , Alignment , Size(KBytes), BasePtr [C7x_1 ] 374.715547 s: 0 , DDR Cacheable , Persistent , 128, 19.27 , 0x17027000 [C7x_1 ] 374.715588 s: 1 , DDR Cacheable , Persistent , 128, 0.65 , 0x1702bf00 [C7x_1 ] 374.715628 s: 2 , L1D , Scratch , 128, 16.00 , 0x64e00000 [C7x_1 ] 374.715668 s: 3 , L2 , Scratch , 128, 448.00 , 0x64800000 [C7x_1 ] 374.715708 s: 4 , L3/MSMC , Scratch , 128, 7968.00 , 0x70020000 [C7x_1 ] 374.715748 s: 5 , DDR Cacheable , Persistent , 128, 20158.13, 0x1702c300 [C7x_1 ] 374.715788 s: 6 , DDR Cacheable , Scratch , 128, 67.75 , 0x00000000 [C7x_1 ] 374.715828 s: 7 , DDR Cacheable , Persistent , 128, 589.25 , 0x183dbd00 [C7x_1 ] 374.715868 s: 8 , DDR Cacheable , Scratch , 128, 0.13 , 0x00011000 374.744659 s: VX_ZONE_ERROR:[ownContextSendCmd:885] Command ack message returned failure cmd_status: -1 [C7x_1 ] 374.715907 s: 9 , DDR Cacheable , Scratch , 128, 3.13 , 0x00011400 375.049871 s: VX_ZONE_ERROR:[ownNodeKernelInit:592] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode 375.049882 s: VX_ZONE_ERROR:[ownNodeKernelInit:593] Please be sure the target callbacks have been registered for this core 375.049890 s: VX_ZONE_ERROR:[ownNodeKernelInit:594] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel [C7x_1 ] 374.715948 s: 10 , DDR Cacheable , Persistent , 128, 5355.03 , 0x1846f300 375.049900 s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:620] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!! [C7x_1 ] 374.715987 s: 11 , DDR Cacheable , Scratch , 128, 512.25 , 0x00012400 [C7x_1 ] 374.716027 s: 12 , DDR Cacheable , Persistent , 128, 7968.00 , 0x189aa100 375.049934 s: VX_ZONE_ERROR:[vxVerifyGraph:2254] Node kernel init failed 375.049945 s: VX_ZONE_ERROR:[vxVerifyGraph:2311] Graph verify failed [C7x_1 ] 374.716067 s: 13 , DDR Cacheable , Persistent , 128, 2770.78 , 0x19172200 [C7x_1 ] 374.716107 s: 14 , DDR Cacheable , Persistent , 128, 0.00 , 0x19426f00 [C7x_1 ] 374.716147 s: 15 , DDR Cacheable , Persistent , 128, 5648.38 , 0x19427100 [C7x_1 ] 374.716175 s: -------------------------------------------- [C7x_1 ] 374.716199 s: Total memory size requirement (space wise): [C7x_1 ] 374.716218 s: Mem Space , Size(KBytes) TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!! [C7x_1 ] 374.716234 s: L1D , 16.00 TIDL_RT_OVX: ERROR: Verify OpenVX graph failed [C7x_1 ] 374.716250 s: L2 , 448.00 [C7x_1 ] 374.716266 s: L3/MSMC , 7968.00 [C7x_1 ] 374.716286 s: DDR Cacheable, 43092.73 [C7x_1 ] 374.716310 s: -------------------------------------------- [C7x_1 ] 374.716345 s: NOTE: Memory requirement in host emulation can be different from the same on EVM [C7x_1 ] 374.716383 s: To get the actual TIDL memory requirement make sure to run on EVM with [C7x_1 ] 374.716406 s: debugTraceLevel = 2 [C7x_1 ] 374.716415 s: [C7x_1 ] 374.716434 s: -------------------------------------------- [C7x_1 ] 374.719384 s: Alg Init for Layer # - 102 [C7x_1 ] 374.719862 s: Alg Init for Layer # - 104 [C7x_1 ] 374.720461 s: Alg Init for Layer # - 105 [C7x_1 ] 374.721035 s: Alg Init for Layer # - 106 [C7x_1 ] 374.721613 s: Alg Init for Layer # - 107 [C7x_1 ] 374.722186 s: Alg Init for Layer # - 108 [C7x_1 ] 374.722766 s: Alg Init for Layer # - 100 [C7x_1 ] 374.723164 s: Alg Init for Layer # - 109 [C7x_1 ] 374.724209 s: Alg Init for Layer # - 101 [C7x_1 ] 374.724462 s: Alg Init for Layer # - 110 [C7x_1 ] 374.725549 s: Alg Init for Layer # - 111 [C7x_1 ] 374.726124 s: Alg Init for Layer # - 112 [C7x_1 ] 374.726733 s: Alg Init for Layer # - 113 [C7x_1 ] 374.727318 s: Alg Init for Layer # - 1 [C7x_1 ] 374.727487 s: Alg Init for Layer # - 114 [C7x_1 ] 374.728504 s: Alg Init for Layer # - 2 [C7x_1 ] 374.728746 s: Alg Init for Layer # - 115 [C7x_1 ] 374.729824 s: Alg Init for Layer # - 116 [C7x_1 ] 374.730470 s: Alg Init for Layer # - 3 [C7x_1 ] 374.730820 s: Alg Init for Layer # - 117 [C7x_1 ] 374.731871 s: Alg Init for Layer # - 4 [C7x_1 ] 374.732052 s: Alg Init for Layer # - 118 [C7x_1 ] 374.733073 s: Alg Init for Layer # - 119 [C7x_1 ] 374.733664 s: Alg Init for Layer # - 120 RT-Profile: TIDLRT_init_profiling [C7x_1 ] 374.734242 s: Alg Init for Layer # - 121 tidlrt_create : 1064156949 ns, tidl_rt_ovx_Init : 99469105 ns, [C7x_1 ] 374.734828 s: Alg Init for Layer # - 122 vxCreateContext : 2886777 ns, init_tidl_tiovx : 10672902 ns, [C7x_1 ] 374.735431 s: Alg Init for Layer # - 125 create_graph_tidl_tiovx : 6072789 ns, verify_graph_tidl_tiovx : 943590495 ns, [C7x_1 ] 374.736206 s: Alg Init for Layer # - 123 tivxTIDLLoadKernels : 19815 ns, mapConfig : 457444 ns, [C7x_1 ] 374.736800 s: Alg Init for Layer # - 126 tivxAddKernelTIDL : 37770 ns, mapNetwork : 9712845 ns, [C7x_1 ] 374.737388 s: Alg Init for Layer # - 127 setCreateParams : 192693 ns, setArgs : 250345 ns, [C7x_1 ] 374.737994 s: Alg Init for Layer # - 128 vxCreateUserDataObject : 23811 ns, vxMapUserDataObject : 6489296 ns, [C7x_1 ] 374.738585 s: Alg Init for Layer # - 129 memcopy_network_buffer : 3194023 ns, vxUnmapUserDataObject : 3625 ns, [C7x_1 ] 374.739158 s: Alg Init for Layer # - 130 ************ TIDL_subgraphRtCreate done ************ [C7x_1 ] 374.739931 s: Alg Init for Layer # - 131 [C7x_1 ] 374.740956 s: Alg Init for Layer # - 132 [C7x_1 ] 374.741552 s: Alg Init for Layer # - 133 [C7x_1 ] 374.742160 s: Alg Init for Layer # - 134 [C7x_1 ] 374.742752 s: Alg Init for Layer # - 135 [C7x_1 ] 374.743335 s: Alg Init for Layer # - 136 [C7x_1 ] 374.744056 s: Output Transpose is not supported on this device.. [C7x_1 ] 374.744361 s: WorkloadUnitExec_Init: initParams->linkInitParams[linkIdx].initFuncPtr Failed, Link Id 72611793024 [C7x_1 ] 374.744392 s: VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = 1 [C7x_1 ] 374.744444 s: Error: handle (117027000) doesn't exist in priority table [C7x_1 ] 374.744470 s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:961] tivxAlgiVisionCreate returned NULL ******* In TIDL_subgraphRtInvoke ******** 375.459705 s: VX_ZONE_ERROR:[ownContextSendCmd:885] Command ack message returned failure cmd_status: -1 375.459733 s: VX_ZONE_ERROR:[ownNodeKernelInit:592] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode 375.459740 s: VX_ZONE_ERROR:[ownNodeKernelInit:593] Please be sure the target callbacks have been registered for this core 375.459747 s: VX_ZONE_ERROR:[ownNodeKernelInit:594] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel 375.459756 s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:620] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!! 375.459766 s: VX_ZONE_ERROR:[vxVerifyGraph:2254] Node kernel init failed 375.459773 s: VX_ZONE_ERROR:[vxVerifyGraph:2311] Graph verify failed 375.459832 s: VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:919] graph is not in a state required to be scheduled 375.459840 s: VX_ZONE_ERROR:[vxProcessGraph:844] schedule graph failed 375.459846 s: VX_ZONE_ERROR:[vxProcessGraph:849] wait graph failed ERROR: Running TIDL graph ... Failed !!! Sub Graph Stats 381.000000 7506.000000 10847995625212612.000000 ******* TIDL_subgraphRtInvoke done ******** 2024-09-30 16:12:42.917974373 [E:onnxruntime:, sequential_executor.cc:494 ExecuteKernel] Non-zero status code returned while running TIDL_0 node. Name:'TIDLExecutionProvider_TIDL_0_0' Status Message: TID. [C7x_1 ] 375.459546 s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:945] Network version - 0x00000000, Expected version - 0x20240719 Traceback (most recent call last): File "/opt/edgeai-tidl-tools/examples/osrt_python/ort/onnxrt_ep.py", line 328, in run_model(model, mIdx) File "/opt/edgeai-tidl-tools/examples/osrt_python/ort/onnxrt_ep.py", line 239, in run_model imgs, output, proc_time, sub_graph_time, height, width = infer_image(sess, input_images, config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/edgeai-tidl-tools/examples/osrt_python/ort/onnxrt_ep.py", line 129, in infer_image output = list(sess.run(None, {input_name: input_data})) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 200, in run return self._sess.run(output_names, input_feed, run_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running TIDL_0 node. Name:'TIDLExecutionProvider_TIDL_0_0' Status Message: TIDL Compu. ************ in TIDL_subgraphRtDelete ************ 375.839446 s: VX_ZONE_INIT:[tivxHostDeInitLocal:120] De-Initialization Done for HOST !!! 375.843930 s: VX_ZONE_INIT:[tivxDeInitLocal:206] De-Initialization Done !!! APP: Deinit ... !!! REMOTE_SERVICE: Deinit ... !!! REMOTE_SERVICE: Deinit ... Done !!! 375.846130 s: IPC: Deinit ... !!! 375.846756 s: IPC: DeInit ... Done !!! 375.846788 s: MEM: Deinit ... !!! 375.846799 s: DDR_SHARED_MEM: Alloc's: 8 alloc's of 18957332 bytes 375.846806 s: DDR_SHARED_MEM: Free's : 8 free's of 18957332 bytes 375.846812 s: DDR_SHARED_MEM: Open's : 0 allocs of 0 bytes 375.846822 s: MEM: Deinit ... Done !!! APP: Deinit ... Done !!!