SK-AM62A-LP: Custom model inference on EVM failed with Graph verify failed.

Akira Sato

Prodigy 10 points

Part Number: SK-AM62A-LP

Tool/software:

Hello TI team,

One of my custom models gets stuck during inference with the TIDL ONNX Runtime Execution Provider.

Could you please provide any hints or suggestions for troubleshooting this issue?

The model is a custom ONNX model compiled with the TIDL compiler.
Compilation completed successfully.

- Inference with x86 Emulator Mode: **Success**
- Inference on EVM: **Failure**

Please see the log output below:

libtidl_onnxrt_EP loaded 0x270aad80
Final number of subgraphs created are : 16, - Offloaded Nodes - 380, Total Nodes - 420
APP: Init ... !!!
 52967.375819 s: MEM: Init ... !!!
 52967.375894 s: MEM: Initialized DMA HEAP (fd=5) !!!
 52967.376090 s: MEM: Init ... Done !!!
 52967.376124 s: IPC: Init ... !!!
 52967.394312 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
 52967.398984 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
 52967.399171 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
 52967.399201 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
 52967.399211 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
 52967.400335 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-0
 52967.401090 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-1
 52967.401509 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-2
 52967.402237 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-3
 52967.402335 s:  VX_ZONE_INFO: [tivxInitLocal:126] Initialization Done !!!
 52967.402374 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
 52967.817573 s:  VX_ZONE_ERROR: [ownContextSendCmd:912] Command ack message returned failure cmd_status: -1
 52967.817842 s:  VX_ZONE_ERROR: [ownNodeKernelInit:604] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
 52967.817871 s:  VX_ZONE_ERROR: [ownNodeKernelInit:605] Please be sure the target callbacks have been registered for this core
 52967.817881 s:  VX_ZONE_ERROR: [ownNodeKernelInit:606] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this k
 52967.817898 s:  VX_ZONE_ERROR: [ownGraphNodeKernelInit:690] kernel init for node 0, kernel com.ti.tidl:4:3 ... failed !!!
 52967.817964 s:  VX_ZONE_ERROR: [ TIDL subgraph cls_signal ] Node kernel init failed
 52967.817976 s:  VX_ZONE_ERROR: [ TIDL subgraph cls_signal ] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed

The inference application is our own custom Python program using the ONNX Runtime API, with the following Execution Provider list:
`EP_list = ["TIDLExecutionProvider", "CPUExecutionProvider"]`

Thank you in advance for your support.

4 months ago

0 Reese Grimsley 4 months ago

TI__Genius 16386 points

Hello,

Let's find more information for where this error is coming from. Please share the logs you receive from the following

Run `echo $EDGEAI_VERSION` to tell me your SDK version

Run `/opt/vx_app_arm_remote_log.out &` to start a logging application in the background

Before starting your application, pass "debug_level":2 in the dictionary of runtime options for the ONNX Inference Session for TIDLExecutionProvider. If you are using edgeai-tidl-tools for this with python API, then easiest way is to globally set from common_utils.py [0]

Run your application and please share the logs.

I do notice that you have the maximal 16 subgraphs for your model. This likely means that there are multiple unsupported layers in the network. In addition to reducing overall performance/inference speed, I tend to notice that larger number of subgraphs results in more opportunities for errors like you are seeing. I will give the general recommendation to alter the network architecture to be as compliant with our supported operators [1] (note that this is set of supported ops is related to your SDK version).

[0] https://github.com/TexasInstruments/edgeai-tidl-tools/blob/95ba2c7ec62bbedeb637d7a5c0273fcede21cac9/examples/osrt_python/common_utils.py#L73

[1] https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/docs/supported_ops_rts_versions.md

BR,
Reese

0 Akira Sato 4 months ago in reply to Reese Grimsley

Prodigy 10 points

Hello Reese,

Thank you for your quick response.

Here are the results based on your instructions:

SDK Version (echo $EDGEAI_VERSION):
→ 10.1
Log Output with vx_app_arm_remote_log.out running in the background and debug_level: 2 enabled:
→ Please see the attached file: log.txt.

8345.log.txt

Regarding the subgraph count:
I understand that the current ONNX model includes many unsupported layers, which results in the maximum number of 16 subgraphs. I will work on optimizing the base model as you suggested, since a large number of subgraphs seems to increase the risk of errors and reduce inference performance.

Thanks again for your support.

Best regards,

0 Reese Grimsley 4 months ago in reply to Akira Sato

TI__Genius 16386 points

Hello,

This log is helpful, although it is difficult to determine what the root cause for this error is.

I see indications that this is failing during network setup in subgraph 10, and then triggering a hardware fault during subgraph 11 setup. That fault is likely because it is trying to reference a data structure related to the previous subgraph 10.

In addition to optimizing your base model, you can also limit then maximum number of subgraphs [0] [1]. This will mean more layers run on the Arm cores, but it is a helpful debugging technique to make the problem smaller. This can help isolate which section of the network was causing errors

You can also set model-specific parameters for compilation using the runtime_options in your model_config; see an example here [2]

[0] docs; see max_num_subgraphs: https://github.com/TexasInstruments/edgeai-tidl-tools/tree/10_01_04_00/examples/osrt_python#optional-options

[1] Can set globally from here: https://github.com/TexasInstruments/edgeai-tidl-tools/blob/19ac4e61b40890537ab67fb3036780f41a7ade3a/examples/osrt_python/common_utils.py#L74

[2] https://github.com/TexasInstruments/edgeai-tidl-tools/blob/19ac4e61b40890537ab67fb3036780f41a7ade3a/examples/osrt_python/model_configs.py#L1015

BR,
Reese

Processors

Processors forum

SK-AM62A-LP: Custom model inference on EVM failed with Graph verify failed.