AM62A7: TVM errors generated by TIDL 9.2 compilation module

Hello Martin,

I see that the script generating the errors here is edgeai-tidl-tools/examples/osrt_python/tvm_dlr/tvm_compilation_onnx_example.py. Let's focus on running this script directly, instead of with the run_python_examples.sh script

It looks like TVM is failing to import TIDL as a backend. What is located at the following directories:

ls $TIDL_TOOLS_PATH
ls $TIDL_TOOLS_PATH/osrt_deps

martin liang said:

I use the following version of TVM

https://github.com/TexasInstruments/tvm/tree/tidl-j7

What are you building from this?

I am not able to replicate your error / warnings. Most likely, there is an installation issue. 

Hello
This is my compilation LOG
Finally, add the command you said

3201.build_rel_09_02_log_5.txt

Hi martin,

Let's take a step back.

I assume your application needs TVM. Please tell me if it does not.

TI SDK provides a compiled TVM compatible with TIDL for the target device to run inference. In edgeai-tidl-tools, it will also setup the required TVM/DLR components for x86 PC to compile a model via TVM. It is not required to build the TI fork of TVM yourself. 

• there is a ti_build.sh build script, but I see some proxy lines that may need to be commented out. This looks like it's made for TI internal usage

Reese Grimsley said:

I see that the script generating the errors here is edgeai-tidl-tools/examples/osrt_python/tvm_dlr/tvm_compilation_onnx_example.py. Let's focus on running this script directly, instead of with the run_python_examples.sh script

Please handle run the TI example this way to simplify the printout. 

Rebuilding TVM seems to be the problem here. My recommendation:

• Setup a new virtual environment for python, and rerun the setup.sh script in edgeai-tidl-tools. Let it provide your TVM libraries, rather than rebuilding yourself
• If you have a specific reason to rebuild TVM yourself, then probably try the ti_build.sh script in that repo.

BR.
Reese

Hi Martin,

   If you use the prebuilt tvm package that gets installed by edgeai-tidl-tools setup.sh.  These out-of-box tvm examples should work.

   We don't expect users to build TVM package themselves.  If you really want to build TVM yourself, you are welcome to try:

• download Processor SDK RTOS j721e and Processor SDK RTOS j784s4
• patch pdk from one SDK to the other: cp -nr   <j784s4_psdk_rtos_path>/<pdk_path>/packages/ti/*   <j721e_psdk_rtos_path>/<pdk_path>/packages/ti
• download and install ti-cgt c7000 compiler (should be set up by edgeai-tidl-tools setup.sh)
• cd <j721e psdk rtos path>;  create softline from tidl_j7* to c7x-mma-tidl
• export PSDKR_PATH=<downloaded j721e psdk rtos path>
  export TIDL_DIR_NAME=c7x-mma-tidl   # alt: tidl_j7*
  export CGT7X_ROOT=$HOME/ti/ti-cgt-c7000_4.1.0.LTS
• in tvm, if you build against psdk 9.2, please checkout TVM tag: TIDL_PSDK_9.2.0  Otherwise, you will run into incompatibility issues between TVM and TIDL TOOLS
• cmake -DUSE_MICRO=ON -DUSE_SORT=ON -DUSE_TIDL=ON -DUSE_LLVM="llvm-config --link-static" -DHIDE_PRIVATE_SYMBOLS=ON -DUSE_TIDL_RT_PATH=$(ls -d ${PSDKR_PATH}/${TIDL_DIR_NAME}/arm-tidl/rt) -DUSE_TIDL_PSDKR_PATH=${PSDKR_PATH} -DUSE_CGT7X_ROOT=${CGT7X_ROOT} ..
• make -j16

    Just want to point out that this is not a supported use mode.  I still recommend the prebuilt tvm package associated with the edgeai-tidl-tools releases.

-Yuan

Hi

OK, I will change TVM to the default compilation as you requested.
But there is still a compilation error. This question seems to have been raised before but no response

Follow the default instructions source ./setup.sh
The following failure will still occur

TIDL_RT_OVX: ERROR: Invalid core number specified - 0 , expected core number is 1
TIDL_RT_OVX: ERROR: Create OpenVX graph failed
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
0.64326s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:907] Network version - 0x00000000, Expected version - 0x20240401
0.64517s: VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
0.64524s: VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node node_79
0.64530s: VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
0.64534s: VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
0.64541s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
0.64548s: VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
0.64570s: VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
0.64592s: VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
0.64596s: VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
0.64612s: VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
2025-04-17 10:19:31,738 ERROR error in running inference DLRModelImpl TVMError: TIDLRT_invoke failed

build_rel_09_02_log_6.txt

Okay, the compile log looks much better. Thank you for changing the setup to use prebuilt packages as suggested. This resolves the original error, but not sufficient to pass the example scripts. 

Your inference is running on the host, I assume

martin liang said:

TIDL_RT_OVX: ERROR: Invalid core number specified - 0 , expected core number is 1

I had seen this error before in your logs but needed to resolve the build issues first.  It looks like an invalid ID was used for which accelerator the model should run on (for AM62A, only one instance of C7xMMA, so only one value possible). I assume you did not touch anything related to core_number parameter for runtime, right? I'm looking in the compilation and inference scripts used here, and this parameter is not exposed anywhere.

TVM Is not my area of expertise, so I will need to ask another expert to weigh in. Again, I will ask you to confirm that TVM is important to your application.

• TFLite and ONNX interfaces are more stable and are strongly preferred over TVM. Even on latest 10.1 SDK, I see TVM does compile and perform inference without error, but outputs are not correct. 

BR,
Reese

Hi Martin,

    Looks like your dlr package is not installed properly, maybe you had an old dlr package installed, then setup.sh was not able to install the dlr package that is compatible with rel_09_02 branch?

    I tested locally using the docker approach.  I didn't see the issue that you encountered.  See enclosed run log.

-Yuan

Running Inference on Model - ../../../model-artifacts/cl-dlr-onnx_mobilenetv2

2025-04-17 21:20:57,217 INFO Could not find libdlr.so in model artifact. Using dlr from /usr/local/lib/python3.10/dist-packages/dlr/libdlr.so
The soft limit is 2048
The hard limit is 2048
MEM: Init ... !!!
MEM: Init ... Done !!!
0.0s: VX_ZONE_INIT:Enabled
0.6s: VX_ZONE_ERROR:Enabled
0.8s: VX_ZONE_WARNING:Enabled
0.1794s: VX_ZONE_INIT:[tivxInit:190] Initialization Done !!!

Processing time in ms : 182.6

, 0 13.257734 missile ,, 1 12.934375 projectile, missile ,, 2 12.287656 warplane, military plane ,, 3 12.287656 aircraft carrier, carrier, flattop, attack aircraft carrier ,, 4 10.347500 airship, dirigible ,

Saving image to ../../../output_images/

Completed_Model : 2, Name : cl-dlr-onnx_mobilenetv2 , Total time : 182.62, Offload Time : 182.62 , DDR RW MBs : 0, Output File : py_out_cl-dlr-onnx_mobilenetv2_airshow.jpg


MEM: Deinit ... !!!
MEM: Alloc's: 25 alloc's of 37856006 bytes
MEM: Free's : 25 free's of 37856006 bytes
MEM: Open's : 0 allocs of 0 bytes
MEM: Deinit ... Done !!!

Hello

You said there is a problem with libdlr.so
I checked and the file exists.

log1

2025-04-18 12:46:52,558 INFO Could not find libdlr.so in model artifact. Using dlr from /home/user001/.local/lib/python3.10/site-packages/dlr/libdlr.so

log2

user001@user001-CoffeeLake:~/edgeai-tidl-tools$ ls /home/user001/.local/lib/python3.10/site-packages
attr fsspec ninja platformdirs synr
attrs fsspec-2025.3.0.dist-info ninja-1.11.1.3.dist-info platformdirs-4.3.6.dist-info synr-0.6.0.dist-info
attrs-25.1.0.dist-info functorch numpy pluggy test
autocfg gluoncv numpy-1.23.0.dist-info pluggy-1.5.0.dist-info tests
autocfg-0.0.8.dist-info gluoncv-0.10.5.post0.dist-info numpy.libs portalocker tflite
caffe2onnx google nvidia portalocker-3.1.1.dist-info tflite-2.18.0.dist-info
caffe2onnx-1.0.2.dist-info graphviz nvidia_cublas_cu12-12.4.5.8.dist-info protobuf-3.20.3.dist-info tflite_runtime
cloudpickle graphviz-0.20.3.dist-info nvidia_cuda_cupti_cu12-12.4.127.dist-info protobuf-3.20.3-py3.10-nspkg.pth tflite_runtime-2.12.0.dist-info
cloudpickle-3.1.1.dist-info huggingface_hub nvidia_cuda_nvrtc_cu12-12.4.127.dist-info psutil timm
coloredlogs huggingface_hub-0.29.2.dist-info nvidia_cuda_runtime_cu12-12.4.127.dist-info psutil-7.0.0.dist-info timm-1.0.15.dist-info
coloredlogs-15.0.1.dist-info humanfriendly nvidia_cudnn_cu12-9.1.0.70.dist-info pybind11 tomli
coloredlogs.pth humanfriendly-10.0.dist-info nvidia_cufft_cu12-11.2.1.3.dist-info pybind11-2.13.6.dist-info tomli-2.2.1.dist-info
contourpy iniconfig nvidia_curand_cu12-10.3.5.147.dist-info pybind11_global-2.13.6.dist-info torch
contourpy-1.3.1.dist-info iniconfig-2.0.0.dist-info nvidia_cusolver_cu12-11.6.1.9.dist-info __pycache__ torch-2.6.0.dist-info
cusparselt isympy.py nvidia_cusparse_cu12-12.3.1.170.dist-info pyelftools-0.32.dist-info torchgen
cv2 jinja2 nvidia_cusparselt_cu12-0.6.2.dist-info pygments torchvision
cycler jinja2-3.1.6.dist-info nvidia_nccl_cu12-2.21.5.dist-info pygments-2.19.1.dist-info torchvision-0.21.0.dist-info
cycler-0.12.1.dist-info jsonschema nvidia_nvjitlink_cu12-12.4.127.dist-info pylab.py torchvision.libs
dataclasses-0.6.dist-info jsonschema-4.23.0.dist-info nvidia_nvtx_cu12-12.4.127.dist-info py.py tornado
dataclasses.py jsonschema_specifications onnx _pytest tornado-6.4.2.dist-info
dateutil jsonschema_specifications-2024.10.1.dist-info onnx-1.13.0.dist-info pytest tqdm
decorator-5.2.1.dist-info kiwisolver onnx_graphsurgeon pytest-8.3.5.dist-info tqdm-4.67.1.dist-info
decorator.py kiwisolver-1.4.8.dist-info onnx_graphsurgeon-0.3.26.dist-info python_dateutil-2.9.0.post0.dist-info triton
distlib markdown_it onnxruntime referencing triton-3.2.0.dist-info
distlib-0.3.9.dist-info markdown_it_py-3.0.0.dist-info onnxruntime_tidl-1.14.0.dist-info referencing-0.36.2.dist-info tvm
_distutils_hack matplotlib onnxsim rich tvm-0.12.0.dist-info
distutils-precedence.pth matplotlib-3.10.1.dist-info onnxsim-0.4.35.dist-info rich-13.9.4.dist-info typing_extensions-4.12.2.dist-info
dlr mdurl opencv_python-4.11.0.86.dist-info rpds typing_extensions.py
dlr-1.13.0.dist-info mdurl-0.1.2.dist-info opencv_python.libs rpds_py-0.24.0.dist-info tzdata
elftools meson-1.7.0.dist-info packaging safetensors tzdata-2025.1.dist-info
exceptiongroup mesonbuild packaging-24.2.dist-info safetensors-0.5.3.dist-info virtualenv
exceptiongroup-1.2.2.dist-info mpl_toolkits pandas scipy virtualenv-20.29.3.dist-info
filelock mpmath pandas-2.2.3.dist-info scipy-1.13.1.dist-info yacs
filelock-3.17.0.dist-info mpmath-1.3.0.dist-info pathspec scipy.libs yacs-0.1.8.dist-info
flatbuffers netron pathspec-0.12.1.dist-info setuptools yamllint
flatbuffers-1.12.dist-info netron-8.2.2.dist-info pip setuptools-78.1.0.dist-info yamllint-1.37.0.dist-info
fontTools networkx pip-25.0.1.dist-info sympy
fonttools-4.56.0.dist-info networkx-3.4.2.dist-info pkg_resources sympy-1.13.1.dist-info
user001@user001-CoffeeLake:~/edgeai-tidl-tools$ ls /home/user001/.local/lib/python3.10/site-packages/dlr/
api.py counter dlr_model.py include __init__.py libdlr.so libpath.py metadata.py neologger.py __pycache__
user001@user001-CoffeeLake:~/edgeai-tidl-tools$

Is it related to TIDL_RT_OVX failure?

TIDL_RT_OVX: ERROR: Invalid core number specified - 0 , expected core number is 1
TIDL_RT_OVX: ERROR: Create OpenVX graph failed
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
0.65594s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:907] Network version - 0x00000000, Expected version - 0x20240401
0.65733s: VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
0.65739s: VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node node_79
0.65743s: VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
0.65748s: VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
0.65753s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
0.65762s: VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
0.65765s: VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
0.65785s: VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
0.65789s: VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
0.65792s: VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
2025-04-18 12:46:53,501 ERROR error in running inference DLRModelImpl TVMError: TIDLRT_invoke failed

How to modify libdlr.so to the correct version

user001@user001-CoffeeLake:~/edgeai-tidl-tools$ find . -name libdlr.so
./tidl_tools/osrt_deps/libdlr.so
user001@user001-CoffeeLake:~/edgeai-tidl-tools$ cd tidl_tools/osrt_deps/
user001@user001-CoffeeLake:~/edgeai-tidl-tools/tidl_tools/osrt_deps$ ls -al
總用量 3584
drwxrwxr-x 6 user001 user001 4096 4月 18 12:08 .
drwxrwxr-x 3 user001 user001 4096 4月 18 12:36 ..
drwxrwxr-x 3 user001 user001 4096 4月 18 12:08 dlr_1.10.0_x86_u22
-rwxrwxr-x 1 user001 user001 3643752 4月 18 12:08 libdlr.so
drwxr-xr-x 3 user001 user001 4096 4月 18 12:07 onnx_1.14.0_x86_u22
drwxrwxr-x 4 user001 user001 4096 4月 18 12:08 opencv_4.2.0_x86_u22
drwxr-xr-x 4 user001 user001 4096 3月 21 2024 tflite_2.12_x86_u22
user001@user001-CoffeeLake:~/edgeai-tidl-tools/tidl_tools/osrt_deps$ pwd
/home/user001/edgeai-tidl-tools/tidl_tools/osrt_deps
user001@user001-CoffeeLake:~/edgeai-tidl-tools/tidl_tools/osrt_deps$

user001@user001-CoffeeLake:~/.local/lib/python3.10/site-packages/dlr$ ls
api.py counter dlr_model.py include __init__.py libdlr.so libpath.py metadata.py neologger.py __pycache__
user001@user001-CoffeeLake:~/.local/lib/python3.10/site-packages/dlr$ ls -al
總用量 3636
drwxrwxr-x 5 user001 user001 4096 3月 10 10:02 .
drwxrwxr-x 181 user001 user001 12288 4月 17 09:33 ..
-rw-rw-r-- 1 user001 user001 9291 3月 10 10:02 api.py
drwxrwxr-x 4 user001 user001 4096 3月 10 10:02 counter
-rw-rw-r-- 1 user001 user001 18476 3月 10 10:02 dlr_model.py
drwxrwxr-x 4 user001 user001 4096 3月 10 10:02 include
-rw-rw-r-- 1 user001 user001 223 3月 10 10:02 __init__.py
-rwxrwxr-x 1 user001 user001 3643752 3月 10 10:02 libdlr.so
-rw-rw-r-- 1 user001 user001 2815 3月 10 10:02 libpath.py
-rw-rw-r-- 1 user001 user001 40 3月 10 10:02 metadata.py
-rw-rw-r-- 1 user001 user001 1072 3月 10 10:02 neologger.py
drwxrwxr-x 2 user001 user001 4096 3月 10 10:02 __pycache__
user001@user001-CoffeeLake:~/.local/lib/python3.10/site-packages/dlr$ echo LD_LIBRARY_PATH
LD_LIBRARY_PATH
user001@user001-CoffeeLake:~/.local/lib/python3.10/site-packages/dlr$ echo $LD_LIBRARY_PATH
:/home/user001/edgeai-tidl-tools/tidl_tools:/home/user001/edgeai-tidl-tools/tidl_tools/osrt_deps:/home/user001/edgeai-tidl-tools/tidl_tools/osrt_deps/opencv/
user001@user001-CoffeeLake:~/.local/lib/python3.10/site-packages/dlr$

build_rel_09_02_log_7.txt

I checked the libdlr.so file I have here
It turns out that they may all use the same file
If the versions are different, will the size of libdlr.so also be different?
As shown in the figure below

I'm checking the cause of this error.

2025-04-18 14:26:39,987 INFO Could not find libdlr.so in model artifact. Using dlr from /home/user001/tflite_env/lib/python3.10/site-packages/dlr/libdlr.so

Is libdlr.so an important library for TIDL_RT_OVX?
Is it directly related to these issues?

TIDL_RT_OVX: ERROR: Invalid core number specified - 0 , expected core number is 1
TIDL_RT_OVX: ERROR: Create OpenVX graph failed
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed