• State
  • Locked Locked
  • Replies 4 replies
  • Subscribers 103 subscribers
  • Views 186 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: “ERROR: Verifying TIDL graph ... Failed !!!” When compiling the model

?? ?
Prodigy 30 points
Part Number: TDA4VM

there are error occurred when compiling the model by edgeai-idle-tools。

The edgeai-idle-tools compiled and output bin files, but there are error messages in the log.

The model has two inputs

Error log:

****************************************************
** 45 WARNINGS 0 ERRORS **
****************************************************
In TIDL_runtimesPostProcessNet 4
************ in TIDL_subgraphRtCreate ************
The soft limit is 2048
The hard limit is 2048
MEM: Init ... !!!
MEM: Init ... Done !!!
0.0s: VX_ZONE_INIT:Enabled
0.3s: VX_ZONE_ERROR:Enabled
0.6s: VX_ZONE_WARNING:Enabled
0.1166s: VX_ZONE_INIT:[tivxInit:185] Initialization Done !!!
0.6535s: VX_ZONE_ERROR:[tivxAlgiVisionCreate:363] Calling ialg.algAlloc failed with status = -1120
0.6553s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:912] tivxAlgiVisionCreate returned NULL
0.6599s: VX_ZONE_ERROR:[ownContextSendCmd:822] Command ack message returned failure cmd_status: -1
0.6604s: VX_ZONE_ERROR:[ownContextSendCmd:862] tivxEventWait() failed.
0.6607s: VX_ZONE_ERROR:[ownNodeKernelInit:584] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
0.6608s: VX_ZONE_ERROR:[ownNodeKernelInit:585] Please be sure the target callbacks have been registered for this core
0.6610s: VX_ZONE_ERROR:[ownNodeKernelInit:586] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
0.6612s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl:2:1 ... failed !!!
0.6615s: VX_ZONE_ERROR:[vxVerifyGraph:2059] Node kernel init failed
0.6617s: VX_ZONE_ERROR:[vxVerifyGraph:2113] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
************ TIDL_subgraphRtCreate done ************
2024-04-22 06:22:14.865456099 [W:onnxruntime:, execution_frame.cc:835 VerifyOutputSizes] Expected shape from model of {1,1,320,640} does not match actual shape of {1,1,1,1,320,640} for output disp5
******* In TIDL_subgraphRtInvoke ********
0.11067s: VX_ZONE_ERROR:[tivxAlgiVisionCreate:363] Calling ialg.algAlloc failed with status = -1120
0.11081s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:912] tivxAlgiVisionCreate returned NULL
0.11149s: VX_ZONE_ERROR:[ownContextSendCmd:822] Command ack message returned failure cmd_status: -1
0.11154s: VX_ZONE_ERROR:[ownContextSendCmd:862] tivxEventWait() failed.
0.11156s: VX_ZONE_ERROR:[ownNodeKernelInit:584] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
0.11158s: VX_ZONE_ERROR:[ownNodeKernelInit:585] Please be sure the target callbacks have been registered for this core
0.11160s: VX_ZONE_ERROR:[ownNodeKernelInit:586] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
0.11162s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl:2:1 ... failed !!!
0.11165s: VX_ZONE_ERROR:[vxVerifyGraph:2059] Node kernel init failed
0.11167s: VX_ZONE_ERROR:[vxVerifyGraph:2113] Graph verify failed
0.11179s: VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:803] graph is not in a state required to be scheduled
0.11180s: VX_ZONE_ERROR:[vxProcessGraph:738] schedule graph failed
0.11182s: VX_ZONE_ERROR:[vxProcessGraph:743] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
Sub Graph Stats 567.000000 1309.000000 0.000000
******* TIDL_subgraphRtInvoke done ********

********** Frame Index 1 : Running float inference **********
~~~~~~~~~~~~~~~~~ infer image ~~~~~~~~~~~~~~~~~
模型输入节点数量 input size : 2
当前输入节点: image1
当前输入节点: image2
2024-04-22 06:22:14.925496721 [W:onnxruntime:, execution_frame.cc:835 VerifyOutputSizes] Expected shape from model of {1,1,320,640} does not match actual shape of {1,1,1,1,320,640} for output disp5
Graph Domain TO version : 18******* In TIDL_subgraphRtInvoke ********
0.70977s: VX_ZONE_ERROR:[tivxAlgiVisionCreate:363] Calling ialg.algAlloc failed with status = -1120
0.70997s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:912] tivxAlgiVisionCreate returned NULL
0.71066s: VX_ZONE_ERROR:[ownContextSendCmd:822] Command ack message returned failure cmd_status: -1
0.71072s: VX_ZONE_ERROR:[ownContextSendCmd:862] tivxEventWait() failed.
0.71074s: VX_ZONE_ERROR:[ownNodeKernelInit:584] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
0.71075s: VX_ZONE_ERROR:[ownNodeKernelInit:585] Please be sure the target callbacks have been registered for this core
0.71076s: VX_ZONE_ERROR:[ownNodeKernelInit:586] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
0.71078s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl:2:1 ... failed !!!
0.71080s: VX_ZONE_ERROR:[vxVerifyGraph:2059] Node kernel init failed
0.71081s: VX_ZONE_ERROR:[vxVerifyGraph:2113] Graph verify failed
0.71094s: VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:803] graph is not in a state required to be scheduled
0.71096s: VX_ZONE_ERROR:[vxProcessGraph:738] schedule graph failed
0.71098s: VX_ZONE_ERROR:[vxProcessGraph:743] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
Sub Graph Stats 543.000000 1357.000000 0.000000
******* TIDL_subgraphRtInvoke done ********

********** Frame Index 2 : Running fixed point mode for calibration **********
In TIDL_runtimesPostProcessNet
In TIDL_runtimesPostProcessNet 1
In TIDL_runtimesPostProcessNet 2
In TIDL_runtimesPostProcessNet 3
Asymmetric quantization with calib = 4
Empty prototxt path, running calibration

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/root/metoak-demo/out/model-artifacts/metoak-mostereonet-aanet/tempDir/disp5_tidl_io_.qunat_stats_config.txt
Error at line: 1167 : in file /ti/SDK_9_1/J721E/ti-processor-sdk-rtos-j721e-evm-09_01_00_06/c7x-mma-tidl/ti_dl/test/src/pc_linux/../tidl_rt.c, of function : TIDLRT_create
Invalid Error Type!
Error at line: 527 : in file /ti/SDK_9_1/J721E/ti-processor-sdk-rtos-j721e-evm-09_01_00_06/c7x-mma-tidl/ti_dl/test/src/pc_linux/../tidl_tb.c, of function : tidlMultiInstanceTest
Invalid Error Type!

***************** Calibration iteration number 0 started ************************
Empty prototxt path, running calibration

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/root/metoak-demo/out/model-artifacts/metoak-mostereonet-aanet/tempDir/disp5_tidl_io_.qunat_stats_config.txt
Error at line: 1167 : in file /ti/SDK_9_1/J721E/ti-processor-sdk-rtos-j721e-evm-09_01_00_06/c7x-mma-tidl/ti_dl/test/src/pc_linux/../tidl_rt.c, of function : TIDLRT_create
Invalid Error Type!
Error at line: 527 : in file /ti/SDK_9_1/J721E/ti-processor-sdk-rtos-j721e-evm-09_01_00_06/c7x-mma-tidl/ti_dl/test/src/pc_linux/../tidl_tb.c, of function : tidlMultiInstanceTest
Invalid Error Type!

***************** Calibration iteration number 0 completed ************************

***************** Calibration iteration number 1 started ************************
Empty prototxt path, running calibration
B=-117551423631466643661210517504.000000 C=0.000000 D=0.000000
ScaleX = 1.000000 | scaleW = 127.000000 | scaleW = 127.000000
B=-58775711815733321830605258752.000000 C=0.000000 D=0.000000
ScaleX = 1.000000 | scaleW = 63.500000 | scaleW = 63.500000
B=-29387855907866660915302629376.000000 C=0.000000 D=0.000000
 

Complete logs：

Fullscreen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
root@ai-MS-7C99:~/metoak-demo/osrt_convert_python# python3 ./convert_onnxrt.py -c -m mostereonet
True
~~~~~~~~~~~~~~~~~ models > 0 ~~~~~~~~~~~~~~~~~ 1
['metoak-mostereonet-aanet']
Running_Model: metoak-mostereonet-aanet
Running shape inference on model /home/root/metoak-demo/models/metoak_model_convshift_md48_sim.onnx
TIDLCompilationProvider
tidl_tools_path = /home/root/tidl_tools
artifacts_folder = /home/root/metoak-demo/out/model-artifacts/mostereonet/
tidl_tensor_bits = 8
debug_level = 1
num_tidl_subgraphs = 16
tidl_denylist =
tidl_denylist_layer_name =
tidl_denylist_layer_type =
tidl_allowlist_layer_name =
model_type =
tidl_calibration_accuracy_level = 4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  • 0
    TI__Intellectual 2750 points

    Hi, 

    The problem may comes from this log:

    Can you change the dimension of your node to 6 by adding empty dimensions to it?  The dimension of the output tensor is always 6 in tidl. Also, can you check if the demo runs without error in current environment?

    Regards,

    Adam

  • 0 in reply to Adam Hua
    Prodigy 30 points

    I create a simple model, which will also have the same warning, but it can run normally

  • 0 in reply to Adam Hua
    Prodigy 30 points

    the demo can run successfully and obtain the correct results

  • 0 in reply to Adam Hua
    Prodigy 40 points

    Hi, we have added 2 empty dimensions to model's output node but the same warnings and errors still occur. As for the warning message, the expected shape of output is now a 6D tensor and the actual shape of the output is 8D. 

    Fullscreen
    1
    2
    ********** Frame Index 1 : Running float inference **********
    2024-04-22 10:52:39.330752949 [W:onnxruntime:, execution_frame.cc:835 VerifyOutputSizes] Expected shape from model of {1,1,1,1,320,640} does not match actual shape of {1,1,1,1,1,1,320,640} for output disp5
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX