SK-TDA4VM: Compiled Vision Transformer does not work in TDA4VM.

Vivek Patel

Part Number: SK-TDA4VM

Tool/software:

Hello,

Recently, I have updated my board to SDK 10.00.00.08, and I have updated my Docker container accordingly from edgeai-tidl-tools

I can compile and run ResNet-18 and MobileNetV2 on my TDA4 board without any issues.

I can also compile deit_tiny_patch16_224 and some other transformer model, but that compile artifact does not work on the board.

I am following this example (as starting point) from your official repository - DeiT Transformer Example

I am running following GStreamer pipeline to run the artifacts

gst-launch-1.0 -v videotestsrc num-buffers=-1 pattern=ball is-live=false ! \
    video/x-raw,height=224,width=224,format=RGB ! \
    tidlpreproc ! tidlinferer model=../models/deit_tiny_patch16_224 ! \
    fpsdisplaysink text-overlay=0 video-sink=fakesink sync=false

I am getting following error on my TDA4

libtidl_onnxrt_EP loaded 0x20ae18c0 
Final number of subgraphs created are : 1, - Offloaded Nodes - 592, Total Nodes - 592 
APP: Init ... !!!
  3274.520923 s: MEM: Init ... !!!
  3274.520969 s: MEM: Initialized DMA HEAP (fd=8) !!!
  3274.521065 s: MEM: Init ... Done !!!
  3274.521075 s: IPC: Init ... !!!
  3274.593911 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3274.602514 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3274.602595 s:  VX_ZONE_INIT:Enabled
  3274.602605 s:  VX_ZONE_ERROR:Enabled
  3274.602614 s:  VX_ZONE_WARNING:Enabled
  3274.603527 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-0 
  3274.603644 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-1 
  3274.603730 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-2 
  3274.603812 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-3 
  3274.603823 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
  3274.604218 s:  VX_ZONE_INIT:[tivxHostInitLocal:106] Initialization Done for HOST !!!
  3275.550322 s:  VX_ZONE_ERROR:[ownContextSendCmd:885] Command ack message returned failure cmd_status: -1
[C7x_1 ]   3275.550098 s:  VX_ZONE_ERROR:[tivxAlgiVisionAllocMem:194] Failed to Allocate memory record 5 @ space = 17 and size = 8670480 !!! 
  3275.550787 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  3275.550821 s:  VX_ZONE_ERROR:[ownNodeKernelInit:593] Please be sure the target callbacks have been registered for this core
  3275.550830 s:  VX_ZONE_ERROR:[ownNodeKernelInit:594] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
[C7x_1 ]   3275.550128 s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:358] tivxAlgiVisionAllocMem Failed
  3275.550842 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:620] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
  3275.550876 s:  VX_ZONE_ERROR:[vxVerifyGraph:2254] Node kernel init failed
[C7x_1 ]   3275.550153 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:961] tivxAlgiVisionCreate returned NULL
  3275.550885 s:  VX_ZONE_ERROR:[vxVerifyGraph:2311] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
  3275.561652 s:  VX_ZONE_ERROR:[ownContextSendCmd:885] Command ack message returned failure cmd_status: -1
  3275.561683 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  3275.561694 s:  VX_ZONE_ERROR:[ownNodeKernelInit:593] Please be sure the target callbacks have been registered for this core
  3275.561704 s:  VX_ZONE_ERROR:[ownNodeKernelInit:594] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  3275.561712 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:620] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
  3275.561722 s:  VX_ZONE_ERROR:[vxVerifyGraph:2254] Node kernel init failed
  3275.561729 s:  VX_ZONE_ERROR:[vxVerifyGraph:2311] Graph verify failed
  3275.561787 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:919] graph is not in a state required to be scheduled
  3275.561795 s:  VX_ZONE_ERROR:[vxProcessGraph:844] schedule graph failed
  3275.561800 s:  VX_ZONE_ERROR:[vxProcessGraph:849] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
2024-09-25 13:55:35.654023216 [E:onnxruntime:, sequential_executor.cc:494 ExecuteKernel] Non-zero status code returned while running TIDL_0 node. Name:'TIDLExecutionProvider_TIDL_0_0' Status Message: TIDL Compute Invoke Failed.
terminate called after throwing an instance of 'Ort::Exception'
  what():  Non-zero status code returned while running TIDL_0 node. Name:'TIDLExecutionProvider_TIDL_0_0' Status Message: TIDL Compute Invoke Failed.

I have tried compiling various vision transformers with different sizes, most of them I was able to compile successfully.

But none of them run on TDA4.

This is the result of vx_app_arm_remote_log.out

[C7x_1 ]      5.411076 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ 117000000 of size 268435456 bytes !!!
[C7x_1 ]      5.411098 s: MEM: Init ... Done !!!
[C7x_1 ]      5.411107 s: IPC: Init ... !!!
[C7x_1 ]      5.411121 s: IPC: 6 CPUs participating in IPC !!!
[C7x_1 ]      5.411136 s: IPC: Waiting for HLOS to be ready ... !!!
[C7x_1 ]     16.252871 s: IPC: HLOS is ready !!!
[C7x_1 ]     16.254806 s: IPC: Init ... Done !!!
[C7x_1 ]     16.254820 s: APP: Syncing with 5 CPUs ... !!!
[C7x_1 ]     16.706152 s: APP: Syncing with 5 CPUs ... Done !!!
[C7x_1 ]     16.706170 s: REMOTE_SERVICE: Init ... !!!
[C7x_1 ]     16.706310 s: REMOTE_SERVICE: Init ... Done !!!
[C7x_1 ]     16.706330 s:  VX_ZONE_INIT:Enabled
[C7x_1 ]     16.706342 s:  VX_ZONE_ERROR:Enabled
[C7x_1 ]     16.706353 s:  VX_ZONE_WARNING:Enabled
[C7x_1 ]     16.706592 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target DSP_C7-1 
[C7x_1 ]     16.706666 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target DSP_C7-1_PRI_2 
[C7x_1 ]     16.706774 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target DSP_C7-1_PRI_3 
[C7x_1 ]     16.706853 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target DSP_C7-1_PRI_4 
[C7x_1 ]     16.706921 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target DSP_C7-1_PRI_5 
[C7x_1 ]     16.706987 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target DSP_C7-1_PRI_6 
[C7x_1 ]     16.707059 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target DSP_C7-1_PRI_7 
[C7x_1 ]     16.707128 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target DSP_C7-1_PRI_8 
[C7x_1 ]     16.707149 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
[C7x_1 ]     16.707162 s: APP: OpenVX Target kernel init ... !!!
[C7x_1 ]     16.707332 s: APP: OpenVX Target kernel init ... Done !!!
[C7x_1 ]     16.707346 s: APP: Init ... Done !!!
[C7x_1 ]     16.707355 s: APP: Run ... !!!
[C7x_1 ]     16.707364 s: IPC: Starting echo test ...
[C7x_1 ]     16.707461 s: APP: Run ... Done !!!
[C7x_1 ]     16.713198 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[x] C66X_1[P] C66X_2[.] C7X_1[s] 
[C7x_1 ]     16.713400 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[x] C66X_1[P] C66X_2[P] C7X_1[s] 
[C7x_1 ]     16.727360 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[P] C66X_1[P] C66X_2[P] C7X_1[s] 
[C7x_1 ]     16.762010 s: IPC: Echo status: mpu1_0[x] mcu2_0[P] mcu2_1[P] C66X_1[P] C66X_2[P] C7X_1[s] 
[C7x_1 ]    118.503251 s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = 1
[C7x_1 ]    118.503297 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:961] tivxAlgiVisionCreate returned NULL
[C7x_1 ]    118.514061 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:945] Network version - 0x00000000, Expected version - 0x20240719
[C7x_1 ]    236.003623 s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = 1
[C7x_1 ]    236.003670 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:961] tivxAlgiVisionCreate returned NULL
[C7x_1 ]    236.014710 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:945] Network version - 0x00000000, Expected version - 0x20240719
[C7x_1 ]    249.938003 s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = 1
[C7x_1 ]    249.938049 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:961] tivxAlgiVisionCreate returned NULL
[C7x_1 ]    249.948790 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:945] Network version - 0x00000000, Expected version - 0x20240719
[C7x_1 ]    274.237117 s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = 1
[C7x_1 ]    274.237163 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:961] tivxAlgiVisionCreate returned NULL
[C7x_1 ]    274.247866 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:945] Network version - 0x00000000, Expected version - 0x20240719
[C7x_1 ]    292.161099 s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = 1
[C7x_1 ]    292.161146 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:961] tivxAlgiVisionCreate returned NULL
[C7x_1 ]    292.171659 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:945] Network version - 0x00000000, Expected version - 0x20240719
[C7x_1 ]    307.041204 s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = 1
[C7x_1 ]    307.041250 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:961] tivxAlgiVisionCreate returned NULL
[C7x_1 ]    307.051874 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:945] Network version - 0x00000000, Expected version - 0x20240719
[C7x_1 ]    425.098803 s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = 1
[C7x_1 ]    425.099168 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:961] tivxAlgiVisionCreate returned NULL
[C7x_1 ]    425.109739 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:945] Network version - 0x00000000, Expected version - 0x20240719
[C7x_1 ]    658.811138 s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = 1
[C7x_1 ]    658.811185 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:961] tivxAlgiVisionCreate returned NULL
[C7x_1 ]    658.821964 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:945] Network version - 0x00000000, Expected version - 0x20240719
[C7x_1 ]   2596.942467 s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = 1
[C7x_1 ]   2596.942514 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:961] tivxAlgiVisionCreate returned NULL
[C7x_1 ]   2596.953128 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:945] Network version - 0x00000000, Expected version - 0x20240719
[C7x_1 ]   3275.561512 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:945] Network version - 0x00000000, Expected version - 0x20240719

Can you please help me to fix the issue?

Thank you very much in advance.

BR,

Vivek

1 month ago

0 Asha Bhandarkar 1 month ago

TI__Genius 10150 points

Hi Vivek,

Thank you for posting your question. I was able to reproduce your issue by generating the deit_tiny_1.onnx file and running the default model_config that exists as cl-ort-deit-tiny on edgeai-tidl-tools with a 10.0 SDK environment.

Attaching inference logs with debug_level = 2 on SK-TDA4VM for your reference:

tda4vm_inf_log.txt

root@tda4vm-sk:/opt/edgeai-tidl-tools/examples/osrt_python/ort# python3 onnxrt_ep.py -m cl-ort-deit-tiny
Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 1 Models - ['cl-ort-deit-tiny']


Running_Model :  cl-ort-deit-tiny  

libtidl_onnxrt_EP loaded 0x3c426370 
artifacts_folder                                = ../../../model-artifacts//cl-ort-deit-tiny/ 
debug_level                                     = 2 
target_priority                                 = 0 
max_pre_empt_delay                              = 340282346638528859811704183484516925440.000000 
Final number of subgraphs created are : 1, - Offloaded Nodes - 580, Total Nodes - 580 
In TIDL_createStateInfer 
Compute on node : TIDLExecutionProvider_TIDL_0_0
************ in TIDL_subgraphRtCreate ************ 
 APP: Init ... !!!
   373.990213 s: MEM: Init ... !!!
   373.990272 s: MEM: Initialized DMA HEAP (fd=5) !!!
   373.990410 s: MEM: Init ... Done !!!
   373.990429 s: IPC: Init ... !!!
   374.060763 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   374.072971 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   374.076349 s:  VX_ZONE_INIT:Enabled
   374.076381 s:  VX_ZONE_ERROR:Enabled
   374.076387 s:  VX_ZONE_WARNING:Enabled
   374.083326 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-0 
   374.083466 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-1 
   374.083546 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-2 
   374.083617 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-3 
   374.083627 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
   374.086810 s:  VX_ZONE_INIT:[tivxHostInitLocal:106] Initialization Done for HOST !!!
[C7x_1 ]    374.713895 s: PREEMPTION: Requesting memory of size 8159232 for targetPriority = 512
[C7x_1 ]    374.713921 s: 
[C7x_1 ]    374.713940 s: --------------------------------------------
[C7x_1 ]    374.713964 s: TIDL Memory size requiement (record wise):
[C7x_1 ]    374.714005 s: MemRecNum   , Space               , Attribute   , Alignment   , Size(KBytes), BasePtr     
[C7x_1 ]    374.714050 s: 0           , DDR Cacheable       , Persistent  ,  128, 19.27   , 0x00000000
[C7x_1 ]    374.714092 s: 1           , DDR Cacheable       , Persistent  ,  128, 0.65    , 0x00000000
[C7x_1 ]    374.714133 s: 2           , L1D                 , Scratch     ,  128, 16.00   , 0x00000000
[C7x_1 ]    374.714173 s: 3           , L2                  , Scratch     ,  128, 448.00  , 0x00000000
[C7x_1 ]    374.714213 s: 4           , L3/MSMC             , Scratch     ,  128, 7968.00 , 0x00000000
[C7x_1 ]    374.714253 s: 5           , DDR Cacheable       , Persistent  ,  128, 20158.13, 0x00000000
[C7x_1 ]    374.714297 s: 6           , DDR Cacheable       , Scratch     ,  128, 67.75   , 0x00000000
[C7x_1 ]    374.714340 s: 7           , DDR Cacheable       , Persistent  ,  128, 589.25  , 0x00000000
[C7x_1 ]    374.714381 s: 8           , DDR Cacheable       , Scratch     ,  128, 0.13    , 0x00000000
[C7x_1 ]    374.714421 s: 9           , DDR Cacheable       , Scratch     ,  128, 3.13    , 0x00000000
[C7x_1 ]    374.714461 s: 10          , DDR Cacheable       , Persistent  ,  128, 5355.03 , 0x00000000
[C7x_1 ]    374.714502 s: 11          , DDR Cacheable       , Scratch     ,  128, 512.25  , 0x00000000
[C7x_1 ]    374.714542 s: 12          , DDR Cacheable       , Persistent  ,  128, 7968.00 , 0x00000000
[C7x_1 ]    374.714581 s: 13          , DDR Cacheable       , Persistent  ,  128, 2770.78 , 0x00000000
[C7x_1 ]    374.714621 s: 14          , DDR Cacheable       , Persistent  ,  128, 0.00    , 0x00000000
[C7x_1 ]    374.714661 s: 15          , DDR Cacheable       , Persistent  ,  128, 5648.38 , 0x00000000
[C7x_1 ]    374.714689 s: --------------------------------------------
[C7x_1 ]    374.714713 s: Total memory size requirement (space wise):
[C7x_1 ]    374.714732 s: Mem Space , Size(KBytes)
[C7x_1 ]    374.714748 s: L1D       , 16.00   
[C7x_1 ]    374.714764 s: L2        , 448.00  
[C7x_1 ]    374.714780 s: L3/MSMC   , 7968.00 
[C7x_1 ]    374.714797 s: DDR Cacheable, 43092.73
[C7x_1 ]    374.714819 s: --------------------------------------------
[C7x_1 ]    374.714854 s: NOTE: Memory requirement in host emulation can be different from the same on EVM
[C7x_1 ]    374.714891 s:       To get the actual TIDL memory requirement make sure to run on EVM with 
[C7x_1 ]    374.714914 s:       debugTraceLevel = 2
[C7x_1 ]    374.714923 s: 
[C7x_1 ]    374.714942 s: --------------------------------------------
[C7x_1 ]    374.715405 s: TIDL init call from ivision API 
[C7x_1 ]    374.715422 s: 
[C7x_1 ]    374.715439 s: --------------------------------------------
[C7x_1 ]    374.715463 s: TIDL Memory size requiement (record wise):
[C7x_1 ]    374.715503 s: MemRecNum   , Space               , Attribute   , Alignment   , Size(KBytes), BasePtr     
[C7x_1 ]    374.715547 s: 0           , DDR Cacheable       , Persistent  ,  128, 19.27   , 0x17027000
[C7x_1 ]    374.715588 s: 1           , DDR Cacheable       , Persistent  ,  128, 0.65    , 0x1702bf00
[C7x_1 ]    374.715628 s: 2           , L1D                 , Scratch     ,  128, 16.00   , 0x64e00000
[C7x_1 ]    374.715668 s: 3           , L2                  , Scratch     ,  128, 448.00  , 0x64800000
[C7x_1 ]    374.715708 s: 4           , L3/MSMC             , Scratch     ,  128, 7968.00 , 0x70020000
[C7x_1 ]    374.715748 s: 5           , DDR Cacheable       , Persistent  ,  128, 20158.13, 0x1702c300
[C7x_1 ]    374.715788 s: 6           , DDR Cacheable       , Scratch     ,  128, 67.75   , 0x00000000
[C7x_1 ]    374.715828 s: 7           , DDR Cacheable       , Persistent  ,  128, 589.25  , 0x183dbd00
[C7x_1 ]    374.715868 s: 8           , DDR Cacheable       , Scratch     ,  128, 0.13    , 0x00011000
   374.744659 s:  VX_ZONE_ERROR:[ownContextSendCmd:885] Command ack message returned failure cmd_status: -1
[C7x_1 ]    374.715907 s: 9           , DDR Cacheable       , Scratch     ,  128, 3.13    , 0x00011400
   375.049871 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
   375.049882 s:  VX_ZONE_ERROR:[ownNodeKernelInit:593] Please be sure the target callbacks have been registered for this core
   375.049890 s:  VX_ZONE_ERROR:[ownNodeKernelInit:594] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
[C7x_1 ]    374.715948 s: 10          , DDR Cacheable       , Persistent  ,  128, 5355.03 , 0x1846f300
   375.049900 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:620] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
[C7x_1 ]    374.715987 s: 11          , DDR Cacheable       , Scratch     ,  128, 512.25  , 0x00012400
[C7x_1 ]    374.716027 s: 12          , DDR Cacheable       , Persistent  ,  128, 7968.00 , 0x189aa100
   375.049934 s:  VX_ZONE_ERROR:[vxVerifyGraph:2254] Node kernel init failed
   375.049945 s:  VX_ZONE_ERROR:[vxVerifyGraph:2311] Graph verify failed
[C7x_1 ]    374.716067 s: 13          , DDR Cacheable       , Persistent  ,  128, 2770.78 , 0x19172200
[C7x_1 ]    374.716107 s: 14          , DDR Cacheable       , Persistent  ,  128, 0.00    , 0x19426f00
[C7x_1 ]    374.716147 s: 15          , DDR Cacheable       , Persistent  ,  128, 5648.38 , 0x19427100
[C7x_1 ]    374.716175 s: --------------------------------------------
[C7x_1 ]    374.716199 s: Total memory size requirement (space wise):
[C7x_1 ]    374.716218 s: Mem Space , Size(KBytes)
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
[C7x_1 ]    374.716234 s: L1D       , 16.00   
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
[C7x_1 ]    374.716250 s: L2        , 448.00  
[C7x_1 ]    374.716266 s: L3/MSMC   , 7968.00 
[C7x_1 ]    374.716286 s: DDR Cacheable, 43092.73
[C7x_1 ]    374.716310 s: --------------------------------------------
[C7x_1 ]    374.716345 s: NOTE: Memory requirement in host emulation can be different from the same on EVM
[C7x_1 ]    374.716383 s:       To get the actual TIDL memory requirement make sure to run on EVM with 
[C7x_1 ]    374.716406 s:       debugTraceLevel = 2
[C7x_1 ]    374.716415 s: 
[C7x_1 ]    374.716434 s: --------------------------------------------
[C7x_1 ]    374.719384 s: Alg Init for Layer # -  102
[C7x_1 ]    374.719862 s: Alg Init for Layer # -  104
[C7x_1 ]    374.720461 s: Alg Init for Layer # -  105
[C7x_1 ]    374.721035 s: Alg Init for Layer # -  106
[C7x_1 ]    374.721613 s: Alg Init for Layer # -  107
[C7x_1 ]    374.722186 s: Alg Init for Layer # -  108
[C7x_1 ]    374.722766 s: Alg Init for Layer # -  100
[C7x_1 ]    374.723164 s: Alg Init for Layer # -  109
[C7x_1 ]    374.724209 s: Alg Init for Layer # -  101
[C7x_1 ]    374.724462 s: Alg Init for Layer # -  110
[C7x_1 ]    374.725549 s: Alg Init for Layer # -  111
[C7x_1 ]    374.726124 s: Alg Init for Layer # -  112
[C7x_1 ]    374.726733 s: Alg Init for Layer # -  113
[C7x_1 ]    374.727318 s: Alg Init for Layer # -    1
[C7x_1 ]    374.727487 s: Alg Init for Layer # -  114
[C7x_1 ]    374.728504 s: Alg Init for Layer # -    2
[C7x_1 ]    374.728746 s: Alg Init for Layer # -  115
[C7x_1 ]    374.729824 s: Alg Init for Layer # -  116
[C7x_1 ]    374.730470 s: Alg Init for Layer # -    3
[C7x_1 ]    374.730820 s: Alg Init for Layer # -  117
[C7x_1 ]    374.731871 s: Alg Init for Layer # -    4
[C7x_1 ]    374.732052 s: Alg Init for Layer # -  118
[C7x_1 ]    374.733073 s: Alg Init for Layer # -  119
[C7x_1 ]    374.733664 s: Alg Init for Layer # -  120
RT-Profile: TIDLRT_init_profiling 
[C7x_1 ]    374.734242 s: Alg Init for Layer # -  121
tidlrt_create            :     1064156949 ns,
tidl_rt_ovx_Init         :       99469105 ns,
[C7x_1 ]    374.734828 s: Alg Init for Layer # -  122
vxCreateContext          :        2886777 ns,
init_tidl_tiovx          :       10672902 ns,
[C7x_1 ]    374.735431 s: Alg Init for Layer # -  125
create_graph_tidl_tiovx  :        6072789 ns,
verify_graph_tidl_tiovx  :      943590495 ns,
[C7x_1 ]    374.736206 s: Alg Init for Layer # -  123
tivxTIDLLoadKernels      :          19815 ns,
mapConfig                :         457444 ns,
[C7x_1 ]    374.736800 s: Alg Init for Layer # -  126
tivxAddKernelTIDL        :          37770 ns,
mapNetwork               :        9712845 ns,
[C7x_1 ]    374.737388 s: Alg Init for Layer # -  127
setCreateParams          :         192693 ns,
setArgs                  :         250345 ns,
[C7x_1 ]    374.737994 s: Alg Init for Layer # -  128
vxCreateUserDataObject   :          23811 ns,
vxMapUserDataObject      :        6489296 ns,
[C7x_1 ]    374.738585 s: Alg Init for Layer # -  129
memcopy_network_buffer   :        3194023 ns,
vxUnmapUserDataObject    :           3625 ns,
[C7x_1 ]    374.739158 s: Alg Init for Layer # -  130
************ TIDL_subgraphRtCreate done ************ 
[C7x_1 ]    374.739931 s: Alg Init for Layer # -  131
[C7x_1 ]    374.740956 s: Alg Init for Layer # -  132
[C7x_1 ]    374.741552 s: Alg Init for Layer # -  133
[C7x_1 ]    374.742160 s: Alg Init for Layer # -  134
[C7x_1 ]    374.742752 s: Alg Init for Layer # -  135
[C7x_1 ]    374.743335 s: Alg Init for Layer # -  136
[C7x_1 ]    374.744056 s: Output Transpose is not supported on this device.. 
[C7x_1 ]    374.744361 s: WorkloadUnitExec_Init: initParams->linkInitParams[linkIdx].initFuncPtr Failed, Link Id 72611793024 
[C7x_1 ]    374.744392 s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = 1
[C7x_1 ]    374.744444 s: Error: handle (117027000) doesn't exist in priority table
[C7x_1 ]    374.744470 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:961] tivxAlgiVisionCreate returned NULL
 *******   In TIDL_subgraphRtInvoke  ******** 
   375.459705 s:  VX_ZONE_ERROR:[ownContextSendCmd:885] Command ack message returned failure cmd_status: -1
   375.459733 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
   375.459740 s:  VX_ZONE_ERROR:[ownNodeKernelInit:593] Please be sure the target callbacks have been registered for this core
   375.459747 s:  VX_ZONE_ERROR:[ownNodeKernelInit:594] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
   375.459756 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:620] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
   375.459766 s:  VX_ZONE_ERROR:[vxVerifyGraph:2254] Node kernel init failed
   375.459773 s:  VX_ZONE_ERROR:[vxVerifyGraph:2311] Graph verify failed
   375.459832 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:919] graph is not in a state required to be scheduled
   375.459840 s:  VX_ZONE_ERROR:[vxProcessGraph:844] schedule graph failed
   375.459846 s:  VX_ZONE_ERROR:[vxProcessGraph:849] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
Sub Graph Stats 381.000000 7506.000000 10847995625212612.000000 
*******  TIDL_subgraphRtInvoke done  ******** 
2024-09-30 16:12:42.917974373 [E:onnxruntime:, sequential_executor.cc:494 ExecuteKernel] Non-zero status code returned while running TIDL_0 node. Name:'TIDLExecutionProvider_TIDL_0_0' Status Message: TID.
[C7x_1 ]    375.459546 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:945] Network version - 0x00000000, Expected version - 0x20240719
Traceback (most recent call last):
  File "/opt/edgeai-tidl-tools/examples/osrt_python/ort/onnxrt_ep.py", line 328, in <module>
    run_model(model, mIdx)
  File "/opt/edgeai-tidl-tools/examples/osrt_python/ort/onnxrt_ep.py", line 239, in run_model
    imgs, output, proc_time, sub_graph_time, height, width  = infer_image(sess, input_images, config)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/edgeai-tidl-tools/examples/osrt_python/ort/onnxrt_ep.py", line 129, in infer_image
    output = list(sess.run(None, {input_name: input_data}))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 200, in run
    return self._sess.run(output_names, input_feed, run_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running TIDL_0 node. Name:'TIDLExecutionProvider_TIDL_0_0' Status Message: TIDL Compu.
************ in TIDL_subgraphRtDelete ************ 
    375.839446 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:120] De-Initialization Done for HOST !!!
   375.843930 s:  VX_ZONE_INIT:[tivxDeInitLocal:206] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
   375.846130 s: IPC: Deinit ... !!!
   375.846756 s: IPC: DeInit ... Done !!!
   375.846788 s: MEM: Deinit ... !!!
   375.846799 s: DDR_SHARED_MEM: Alloc's: 8 alloc's of 18957332 bytes 
   375.846806 s: DDR_SHARED_MEM: Free's : 8 free's  of 18957332 bytes 
   375.846812 s: DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
   375.846822 s: MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

I have filed a bug for this, and is being tracked internally with TIDL-5134

Best,

Asha

0 Vivek Patel 1 month ago in reply to Asha Bhandarkar

Prodigy 50 points

Hello Asha,

thank you very much for the response.

Now I will set it to resolve the issue.

BR,

Vivek

Processors

Processors forum

SK-TDA4VM: Compiled Vision Transformer does not work in TDA4VM.