SK-TDA4VM: Execution hangs when using debug_level=3 in TIDL Tools

Part Number: SK-TDA4VM

Tool/software:

Hi,

I was trying to generate traces for a DL model on the SK-TDA4VM, but the execution hangs and I have to kill the process.

To make sure the problem is not related to my script, I ran

/opt/edgeai-tidl-tools/examples/osrt_python/ort# python3 onnxrt_ep.py -m cl-ort-resnet18-v1

and changed debug_level to 3 in /opt/edgeai-tidl-tools/examples/osrt_python/common_utils.py

This hangs too. Following is the output of the command

root@tda4vm-sk:/opt/edgeai-tidl-tools/examples/osrt_python/ort# python3 onnxrt_ep.py -m cl-ort-resnet18-v1
Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 1 Models - ['cl-ort-resnet18-v1']


Running_Model :  cl-ort-resnet18-v1  

libtidl_onnxrt_EP loaded 0x1246f120 
artifacts_folder                                = ../../../model-artifacts//cl-ort-resnet18-v1/artifacts 
debug_level                                     = 3 
target_priority                                 = 0 
max_pre_empt_delay                              = 340282346638528859811704183484516925440.000000 
Final number of subgraphs created are : 1, - Offloaded Nodes - 52, Total Nodes - 52 
In TIDL_createStateInfer 
Compute on node : TIDLExecutionProvider_TIDL_0_0
************ in TIDL_subgraphRtCreate ************ 
 APP: Init ... !!!
  1013.725896 s: MEM: Init ... !!!
  1013.725943 s: MEM: Initialized DMA HEAP (fd=5) !!!
  1013.726091 s: MEM: Init ... Done !!!
  1013.726114 s: IPC: Init ... !!!
  1013.783908 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  1013.790888 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  1013.790988 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
  1013.790999 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
  1013.791009 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
  1013.791544 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-0 
  1013.791659 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-1 
  1013.791760 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-2 
  1013.791852 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-3 
  1013.791865 s:  VX_ZONE_INFO: [tivxInitLocal:202] Initialization Done !!!
  1013.791881 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
************ TIDL_subgraphRtCreate done ************ 
 *******   In TIDL_subgraphRtInvoke  ******** 


I need the traces to compare a mismatch between model run in host emulation mode inside docker with the model being run on the board.

I am following instructions given in the TIDL repo.

Thank you in advance,

Charanjit