• State
  • Locked Locked
  • Replies 3 replies
  • Subscribers 97 subscribers
  • Views 427 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: Edge AI : Mobilnet Model deployment error on EVM board

abhilash sk
Intellectual 530 points

Part Number: TDA4VM

Hello Champs , 

I am trying to run a mobilenet model on the EVM board after compliation and i am getting the following error Please find the shared logs 

Settings : 

SDK : 9.2.7

Docker 

Mode : Inference  

Fullscreen 14_mobile_tda_int8_logs_l5.txt Download 
root@j721e-evm:~/examples/osrt_python/ort# python3 onnxrt_ep_mobilenet_int8.py  
Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 1 Models - ['mobilenet_custom_int8']


Running_Model :  mobilenet_custom_int8  

libtidl_onnxrt_EP loaded 0x139c8690 
artifacts_folder                                = ../../../model-artifacts//mobilenet_custom_int8/ 
debug_level                                     = 5 
target_priority                                 = 0 
max_pre_empt_delay                              = 340282346638528859811704183484516925440.000000 
Final number of subgraphs created are : 1, - Offloaded Nodes - 57, Total Nodes - 57 
In TIDL_createStateInfer 
Compute on node : TIDLExecutionProvider_TIDL_0_0
************ in TIDL_subgraphRtCreate ************ 
 APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  2213.897869 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  2213.897949 s:  VX_ZONE_INIT:Enabled
  2213.897957 s:  VX_ZONE_ERROR:Enabled
  2213.897962 s:  VX_ZONE_WARNING:Enabled
  2213.898472 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-0 
  2213.898578 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-1 
  2213.898668 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-2 
  2213.898752 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-3 
  2213.898761 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
  2213.899162 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
  2213.965080 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  2213.965107 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  2213.965126 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  2213.965132 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  2213.965141 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
  2213.965153 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  2213.965159 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
************ TIDL_subgraphRtCreate done ************ 

***************Running_Inference Section **********

 **********This is Mobilenet Custom Model for image  **********

 *******   In TIDL_subgraphRtInvoke  ******** 
  2214.010611 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  2214.010635 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  2214.010642 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  2214.010649 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  2214.010657 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
  2214.010666 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  2214.010673 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  2214.010723 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  2214.010731 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  2214.010736 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
Sub Graph Stats 412.000000 4975.000000 13458162009366004.000000 
*******  TIDL_subgraphRtInvoke done  ******** 

***************Running_ Benchmark Section **********


***************Running_Inference Section **********

 **********This is Mobilenet Custom Model for image  **********

*******   In TIDL_subgraphRtInvoke  ******** 
  2214.020160 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  2214.020181 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  2214.020187 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  2214.020193 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  2214.020200 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
  2214.020210 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  2214.020216 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  2214.020259 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  2214.020265 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  2214.020271 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
Sub Graph Stats 388.000000 805.000000 13458162009366004.000000 
*******  TIDL_subgraphRtInvoke done  ******** 

***************Running_ Benchmark Section **********


***************Running_Inference Section **********

 **********This is Mobilenet Custom Model for image  **********

*******   In TIDL_subgraphRtInvoke  ******** 
  2214.029583 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  2214.029603 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  2214.029609 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  2214.029615 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  2214.029623 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
  2214.029631 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  2214.029637 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  2214.359387 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  2214.359399 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  2214.359405 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
Sub Graph Stats 558.000000 330484.000000 13458162009366004.000000 
*******  TIDL_subgraphRtInvoke done  ******** 

***************Running_ Benchmark Section **********


 ,  0  0.327665  Shih-Tzu ,,  1  0.250365  white stork, Ciconia ciconia ,,  2  0.238840  chain saw, chainsaw ,,  3  0.224173  obelisk ,,  4  0.192789  drum, membranophone, tympan ,

Saving image to  ../../../output_images/

 
Completed_Model :     1, Name : mobilenet_custom_int8                             , Total time : 4988582064455.85, Offload Time :     112.09 , DDR RW MBs : 0, Output File : py_out_mobilenet_c
ustom_int8_airshow.jpg 
 
 
************ in TIDL_subgraphRtDelete ************ 
   2214.377197 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  2214.381612 s:  VX_ZONE_INIT:[tivxDeInitLocal:204] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 9 alloc's of 79332224 bytes 
DDR_SHARED_MEM: Free's : 9 free's  of 79332224 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
root@j721e-evm:~/examples/osrt_python/ort#

Can you let me knw there reason behind it and the solution for it 

  • 0
    TI__Mastermind 24041 points

    Hi,

    Would appreciate few details, 

    1. Which mobilnet model are you using is this the std one ? the mobilenet model that we have validated can be found here : https://github.com/TexasInstruments/edgeai-modelzoo/tree/main/models/vision/classification/imagenet1k/edgeai-tv we recommend to use the same reference

    2. Can you elaborate your test flow ? Hardware setup and how are you inferencing this model, what sdk version is the target board flashed with etc.

    3. Inference tests : 

    3.1) Is host emulation worked properly for above gen artifacts ?

    3.2) Is OSRT inference was successful ? (without tidl offload) 

    3.3) On target front, can you share the remote core logs ? (run vision_apps.init from /opt/vision_apps, and run the inference with debug_level 2)

    Please provide logs for above listed observations, along with generated artifacts folder and model files

  • 0 in reply to Pratik Kedar
    Intellectual 530 points

    Hi,

    Would appreciate few details, 

    1. Which mobilnet model are you using is this the std one ? the mobilenet model that we have validated can be found here : https://github.com/TexasInstruments/edgeai-modelzoo/tree/main/models/vision/classification/imagenet1k/edgeai-tv we recommend to use the same reference  ----- The mobile net used is the same model you had shared with us with file name "mobile_deploy.onnx"

    2. Can you elaborate your test flow ? Hardware setup and how are you inferencing this model, what sdk version is the target board flashed with etc. ---- we have followed the steps mentioned in the following link 

    3. Inference tests : 

    3.1) Is host emulation worked properly for above gen artifacts ? --- yes on host the generated artifcats are working 

    3.2) Is OSRT inference was successful ? (without tidl offload) --- yes it was successful 

    3.3) On target front, can you share the remote core logs ? (run vision_apps.init from /opt/vision_apps, and run the inference with debug_level 2) --yes shall share it 

    Please provide logs for above listed observations, along with generated artifacts folder and model files

  • 0 in reply to abhilash sk
    TI__Mastermind 24041 points

    Hi Abhilash,

    Look forward for above listed observations.

    Thanks