TDA4VM: edgeai-modelmaker

Makoto Nakamura

Part Number: TDA4VM

Tool/software:

Hello.

I am using edgeai-modelmaker. And I followed the step in README.
When I handling the Step 3, console showed me SUCCESS, but the model I compiled cannot work successfully.
So I checked the run.log which is generated by this tool, then I found some errors.

What shoud I do?

※The run.log is attached.

The pip information maybe useful, I atattch it as pip.txt.

※The pip.tx is attached.

commandline:
./run_modelmaker.sh TDA4VM config_detection.yaml > command.log

※The command.log is attached.

To repeat my errors,
put my 20240604-111044_yolox_nano_lite_onnxrt_TDA4VM.tar.gz into the board, and unzip it

※The 20240604-111044_yolox_nano_lite_onnxrt_TDA4VM.tar.gz is attached.

root@tda4vm-sk:/opt/edgeai-gst-apps/apps_python# ls /opt/model_zoo/custom_model/
artifacts config.yaml dataset.yaml model param.yaml run.log

Edit the /opt/edgeai-gst-apps/configs/object_detection.yaml
※The object_detection.yaml is attached as object_detection.yaml.txt because the limit of file type when uploading

Then run:

root@tda4vm-sk:/opt/edgeai-gst-apps/apps_python# python3 app_edgeai.py ../configs/object_detection.yaml
libtidl_onnxrt_EP loaded 0x27433e70
Final number of subgraphs created are : 1, - Offloaded Nodes - 283, Total Nodes - 283
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
596100.197813 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
596100.198001 s: VX_ZONE_INIT:Enabled
596100.198047 s: VX_ZONE_ERROR:Enabled
596100.198772 s: VX_ZONE_WARNING:Enabled
596100.199339 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-0
596100.199471 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-1
596100.199553 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-2
596100.199643 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-3
596100.199655 s: VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
596100.200244 s: VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
596100.215100 s: VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
596100.215149 s: VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
596100.215167 s: VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
596100.215184 s: VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
596100.215203 s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
596100.215225 s: VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
596100.215243 s: VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
[ERROR] Flow output resolution can not be greater than input resolution
596100.253998 s: VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
596100.258457 s: VX_ZONE_INIT:[tivxDeInitLocal:204] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 8 alloc's of 9687228 bytes
DDR_SHARED_MEM: Free's : 8 free's of 9687228 bytes
DDR_SHARED_MEM: Open's : 0 allocs of 0 bytes
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
command.log 20240604-111044_yolox_nano_lite_onnxrt_TDA4VM.tar.gz

Fullscreen object_detection.yaml.txt Download

title: "Object Detection"
log_level: 2
inputs:
    input1:
        source: /opt/edgeai-test-data/videos/video0_1280_768.h264
        format: h264
        width: 1280
        height: 768
        framerate: 30
        loop: True
models:
    model2:
        model_path: /opt/model_zoo/custom_model
        viz_threshold: 0.6
outputs:
    output3:
        sink: remote
        width: 1920
        height: 1080
        port: 8081
        host: 127.0.0.1
        encoding: jpeg
        overlay-perf-type: graph
    output0:
        sink: kmssink
        width: 1920
        height: 1080
        overlay-perf-type: graph

flows:
    flow0: [input1,model2,output3]

Fullscreen pip.txt Download

absl-py==2.1.0
addict==2.4.0
appdirs==1.4.4
asgiref==3.8.1
attr==0.3.1
attrs==23.2.0
azure-core==1.30.1
azure-storage-blob==12.20.0
bleach==5.0.1
boto==2.49.0
boto3==1.16.63
botocore==1.19.63
boxing==0.1.4
cachetools==5.3.3
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==3.0.0
colorama==0.4.6
coloredlogs==15.0.1
contourpy==1.2.1
cryptography==42.0.7
cycler==0.12.1
Cython==3.0.10
debugpy==1.8.1
decorator==5.1.1
defusedxml==0.7.1
distro==1.9.0
Django==3.2.20
django-annoying==0.10.6
django-cors-headers==3.6.0
django-debug-toolbar==3.2.1
django-environ==0.10.0
django-extensions==3.1.0
django-filter==2.4.0
django-model-utils==4.1.1
django-ranged-fileresponse==0.1.2
django-rq==2.5.1
django-storages==1.12.3
django-user-agents==0.4.0
djangorestframework==3.13.1
dlr @ https://software-dl.ti.com/jacinto7/esd/tidl-tools/09_02_06_00/OSRT_TOOLS/X86_64_LINUX/UBUNTU_22_04/dlr-1.13.0-py3-none-any.whl#sha256=f2a6f2814c60ceb2d2aa2de92714f559dc10d16919b7dbc73f733e7605f93176
drf-dynamic-fields==0.3.0
drf-flex-fields==0.9.5
drf-generators==0.3.0
-e git+https://gitclone.com/github.com/TexasInstruments/edgeai-tensorlab@9d595574ee4eeebb145a56376ac91a60c68d7873#egg=edgeai_benchmark&subdirectory=edgeai-benchmark
-e git+https://gitclone.com/github.com/TexasInstruments/edgeai-tensorlab@9d595574ee4eeebb145a56376ac91a60c68d7873#egg=edgeai_modelmaker&subdirectory=edgeai-modelmaker
edgeai-torchmodelopt @ git+https://github.com/TexasInstruments/edgeai-modeloptimization.git@c6958abce6a177d90add2b21bedd1f8b77c44354#subdirectory=torchmodelopt
-e git+https://gitclone.com/github.com/TexasInstruments/edgeai-tensorlab@9d595574ee4eeebb145a56376ac91a60c68d7873#egg=edgeai_xvision&subdirectory=edgeai-torchvision/references/edgeailite
exceptiongroup==1.2.1
expiringdict==1.2.2
filelock==3.14.0
flatbuffers==1.12
fonttools==4.51.0
google-api-core==2.11.0
google-auth==2.14.1
google-auth-oauthlib==0.4.6
google-cloud-appengine-logging==1.1.0
google-cloud-audit-log==0.2.0
google-cloud-core==2.3.2
google-cloud-logging==2.7.2
google-cloud-storage==2.5.0
google-crc32c==1.5.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.4
graphviz==0.20.3
grpc-google-iam-v1==0.12.4
grpcio==1.64.0
grpcio-status==1.48.2
h5py==3.11.0
htmlmin==0.1.12
humanfriendly==10.0
humansignal-drf-yasg==1.21.9
idna==3.7
ijson==3.2.3
imageio==2.34.1
importlib_metadata==7.1.0
inflection==0.5.1
iniconfig==2.0.0
isodate==0.6.1
Jinja2==3.1.4
jmespath==0.10.0
joblib==1.4.2
json-tricks==3.17.3
jsonschema==3.2.0
kiwisolver==1.4.5
label-studio==1.9.0
label-studio-converter==0.0.55
label-studio-tools==0.0.4
launchdarkly-server-sdk==7.5.0
lazy_loader==0.4
lockfile==0.12.2
loguru==0.7.2
lxml==5.2.2
Markdown==3.6
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.9.0
mdurl==0.1.2
mmcv-full==1.4.8
-e git+https://gitclone.com/github.com/TexasInstruments/edgeai-tensorlab@9d595574ee4eeebb145a56376ac91a60c68d7873#egg=mmdet&subdirectory=edgeai-mmdetection
mpmath==1.3.0
munkres==1.1.4
networkx==3.3
ninja==1.11.1.1
nltk==3.6.7
numpy==1.23.0
oauthlib==3.2.2
onnx==1.13.0
onnx-simplifier==0.4.10
onnxruntime-tidl @ https://software-dl.ti.com/jacinto7/esd/tidl-tools/09_02_06_00/OSRT_TOOLS/X86_64_LINUX/UBUNTU_22_04/onnxruntime_tidl-1.14.0-cp310-cp310-linux_x86_64.whl#sha256=4f3fe7407a7892bea916f63d796f25fa4062f519a92863ebc6da02ba3f5c657d
onnxsim==0.4.36
opencv-python==4.9.0.80
opencv-python-headless==4.9.0.80
ordered-set==4.0.2
osrt-model-tools @ git+https://github.com/TexasInstruments/edgeai-tidl-tools.git@196a4f1258e289ad08058c8db8251d7f3ca85f71#subdirectory=scripts
packaging==24.0
pandas==2.2.2
Pillow==9.3.0
Pillow-SIMD==9.0.0.post1
platformdirs==4.2.2
pluggy==1.5.0
plyfile==0.7.4
progiter==2.0.0
progressbar==2.5
proto-plus==1.23.0
protobuf==3.20.2
psutil==5.9.8
psycopg2-binary==2.9.6
pyasn1==0.6.0
pyasn1_modules==0.4.0
pycocotools==2.0.7
pycparser==2.22
pydantic==1.10.15
Pygments==2.18.0
pyparsing==3.1.2
pyRFC3339==1.1
pyrsistent==0.20.0
pytest==8.2.1
python-dateutil==2.9.0.post0
python-json-logger==2.0.4
pytz==2022.7.1
PyYAML==6.0.1
redis==3.5.3
regex==2024.5.15
requests==2.31.0
requests-oauthlib==2.0.0
rich==13.7.1
rq==1.10.1
rsa==4.9
rules==2.2
s3transfer==0.3.7
scikit-image==0.23.2
scikit-learn==1.5.0
scipy==1.13.1
seaborn==0.11.2
semver==2.13.0
sentry-sdk==2.3.1
six==1.16.0
sqlparse==0.5.0
sympy==1.12
tabulate==0.9.0
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
terminaltables==3.1.10
tflite==2.10.0
tflite-runtime @ https://software-dl.ti.com/jacinto7/esd/tidl-tools/09_02_06_00/OSRT_TOOLS/X86_64_LINUX/UBUNTU_22_04/tflite_runtime-2.8.2-cp310-cp310-linux_x86_64.whl#sha256=a5bf53a437f48324a61c6f315c1fdb79cde7d8ec87689c1efa8f14f621152b0d
thop==0.1.1.post2209072238
threadpoolctl==3.5.0
tifffile==2024.5.22
tomli==2.0.1
torch==2.0.1+cpu
torchinfo==1.8.0
torchvision==0.15.2+cpu
tornado==6.4
tqdm==4.66.4
tvm @ https://software-dl.ti.com/jacinto7/esd/tidl-tools/09_02_06_00/OSRT_TOOLS/X86_64_LINUX/UBUNTU_22_04/tvm-0.12.0-cp310-cp310-linux_x86_64.whl#sha256=7af4695a6c377e34c47c688a1b60329dfcde6db657f2915efaea9b58813009eb
typing_extensions==4.12.0
tzdata==2024.1
ua-parser==0.18.0
ujson==5.10.0
uritemplate==4.1.1
urllib3==1.26.16
user-agents==2.2.0
webencodings==0.5.1
Werkzeug==3.0.3
wurlitzer==3.1.0
xmljson==0.2.0
yapf==0.40.0
-e git+https://gitclone.com/github.com/TexasInstruments/edgeai-tensorlab@9d595574ee4eeebb145a56376ac91a60c68d7873#egg=yolox&subdirectory=edgeai-yolox
zipp==3.18.2

Fullscreen 5554.run.log Download

INFO:20240604-113753: running - od-8200
INFO:20240604-113753: pipeline_config - {'task_type': 'detection', 'dataset_category': 'coco', 'calibration_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7848bfefb970>, 'input_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7848f2918070>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x784898e3ac50>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x784898e3acb0>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x784898e3af50>, 'metric': {'label_offset_pred': 1}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': None}, 'model_shortlist': 10}}
INFO:20240604-113753: import  - od-8200 - this may take some time...
Processing config file #0 : /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_io_.qunat_stats_config.txt 
 Freeing memory for user provided Net
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T    1372.75  .... ..... ... .... .....
#    1 . .. T    1353.69  .... ..... ... .... .....
#    2 . .. T    1368.71  .... ..... ... .... .....
#    3 . .. T    1340.36  .... ..... ... .... .....
#    4 . .. T    1332.33  .... ..... ... .... .....
#    5 . .. T    1355.60  .... ..... ... .... .....
#    6 . .. T    1363.89  .... ..... ... .... .....
#    7 . .. T    1358.27  .... ..... ... .... .....
#    8 . .. T    1360.99  .... ..... ... .... .....
#    9 . .. T    1328.45  .... ..... ... .... .....
Processing config file #0 : /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_io_.qunat_stats_config.txt 
 Freeing memory for user provided Net
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T     520.74  .... ..... ... .... .....
#    1 . .. T     508.98  .... ..... ... .... .....
#    2 . .. T     520.40  .... ..... ... .... .....
#    3 . .. T     510.23  .... ..... ... .... .....
#    4 . .. T     521.07  .... ..... ... .... .....
#    5 . .. T     509.37  .... ..... ... .... .....
#    6 . .. T     516.73  .... ..... ... .... .....
#    7 . .. T     511.12  .... ..... ... .... .....
#    8 . .. T     510.52  .... ..... ... .... .....
#    9 . .. T     511.61  .... ..... ... .... .....
Processing config file #0 : /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_io_.qunat_stats_config.txt 
 Freeing memory for user provided Net
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T     521.54  .... ..... ... .... .....
#    1 . .. T     509.28  .... ..... ... .... .....
#    2 . .. T     510.79  .... ..... ... .... .....
#    3 . .. T     510.01  .... ..... ... .... .....
#    4 . .. T     509.81  .... ..... ... .... .....
#    5 . .. T     512.77  .... ..... ... .... .....
#    6 . .. T     511.57  .... ..... ... .... .....
#    7 . .. T     515.17  .... ..... ... .... .....
#    8 . .. T     508.99  .... ..... ... .... .....
#    9 . .. T     510.94  .... ..... ... .... .....
Processing config file #0 : /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_io_.qunat_stats_config.txt 
 Freeing memory for user provided Net
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T     509.84  .... ..... ... .... .....
#    1 . .. T     508.33  .... ..... ... .... .....
#    2 . .. T     507.98  .... ..... ... .... .....
#    3 . .. T     513.50  .... ..... ... .... .....
#    4 . .. T     508.66  .... ..... ... .... .....
#    5 . .. T     517.78  .... ..... ... .... .....
#    6 . .. T     510.58  .... ..... ... .... .....
#    7 . .. T     513.58  .... ..... ... .... .....
#    8 . .. T     513.75  .... ..... ... .... .....
#    9 . .. T     512.76  .... ..... ... .... .....
Processing config file #0 : /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_io_.qunat_stats_config.txt 
 Freeing memory for user provided Net
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T     513.77  .... ..... ... .... .....
#    1 . .. T     509.26  .... ..... ... .... .....
#    2 . .. T     514.88  .... ..... ... .... .....
#    3 . .. T     508.51  .... ..... ... .... .....
#    4 . .. T     519.64  .... ..... ... .... .....
#    5 . .. T     508.66  .... ..... ... .... .....
#    6 . .. T     510.75  .... ..... ... .... .....
#    7 . .. T     510.82  .... ..... ... .... .....
#    8 . .. T     508.89  .... ..... ... .... .....
#    9 . .. T     508.28  .... ..... ... .... .....yolox is meta arch name 
yolox
Number of OD backbone nodes = 192 
Size of odBackboneNodeIds = 192 

Preliminary subgraphs created = 1 
Final number of subgraphs created are : 1, - Offloaded Nodes - 283, Total Nodes - 283 
 Graph Domain TO version : 11TIDL Meta PipeLine (Proto) File  : /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/model/model.prototxt  
yolox
yolox
Warning : Requested Output Data Convert Layer is not Added to the network, It is currently not Optimal
Warning : Requested Output Data Convert Layer is not Added to the network, It is currently not Optimal

 ************** Frame index 1 : Running float import ************* 
INFORMATION: [TIDL_ResizeLayer] /neck/upsample/Resize Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] /neck/upsample_1/Resize Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
****************************************************
**          2 WARNINGS          0 ERRORS          **
****************************************************
The soft limit is 2048
The hard limit is 2048
MEM: Init ... !!!
MEM: Init ... Done !!!
 0.0s:  VX_ZONE_INIT:Enabled
 0.9s:  VX_ZONE_ERROR:Enabled
 0.10s:  VX_ZONE_WARNING:Enabled
 0.4710s:  VX_ZONE_INIT:[tivxInit:185] Initialization Done !!!

**********  Frame Index 1 : Running float inference **********
 Graph Domain TO version : 11
**********  Frame Index 2 : Running float inference **********
 Graph Domain TO version : 11
**********  Frame Index 3 : Running float inference **********
 Graph Domain TO version : 11
**********  Frame Index 4 : Running float inference **********
 Graph Domain TO version : 11
**********  Frame Index 5 : Running float inference **********
 Graph Domain TO version : 11
**********  Frame Index 6 : Running float inference **********
 Graph Domain TO version : 11
**********  Frame Index 7 : Running float inference **********
 Graph Domain TO version : 11
**********  Frame Index 8 : Running float inference **********
 Graph Domain TO version : 11
**********  Frame Index 9 : Running float inference **********
 Graph Domain TO version : 11
**********  Frame Index 10 : Running fixed point mode for calibration **********
Parameters unavailable, running calibration!

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

 
 
 *****************   Calibration iteration number 0 started ************************ 
 
 
 
Parameters unavailable, running calibration!

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

 
 
 *****************   Calibration iteration number 0 completed ************************ 
 
 
 

 
 
 *****************   Calibration iteration number 1 started ************************ 
 
 
 
Parameters unavailable, running calibration!

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

 
 
 *****************   Calibration iteration number 1 completed ************************ 
 
 
 

 
 
 *****************   Calibration iteration number 2 started ************************ 
 
 
 
Parameters unavailable, running calibration!

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

 
 
 *****************   Calibration iteration number 2 completed ************************ 
 
 
 

 
 
 *****************   Calibration iteration number 3 started ************************ 
 
 
 
Parameters unavailable, running calibration!

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

 
 
 *****************   Calibration iteration number 3 completed ************************ 
 
 
 

 
 
 *****************   Calibration ite
Processing config file #0 : /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_io_.qunat_stats_config.txt 
 Freeing memory for user provided Net
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T     513.30  .... ..... ... .... .....
#    1 . .. T     508.92  .... ..... ... .... .....
#    2 . .. T     510.15  .... ..... ... .... .....
#    3 . .. T     511.18  .... ..... ... .... .....
#    4 . .. T     512.36  .... ..... ... .... .....
#    5 . .. T     517.61  .... ..... ... .... .....
#    6 . .. T     509.62  .... ..... ... .... .....
#    7 . .. T     512.09  .... ..... ... .... .....
#    8 . .. T     509.60  .... ..... ... .... .....
#    9 . .. T     511.74  .... ..... ... .... .....
Processing config file #0 : /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_io_.qunat_stats_config.txt 
 Freeing memory for user provided Net
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T     513.99  .... ..... ... .... .....
#    1 . .. T     510.15  .... ..... ... .... .....
#    2 . .. T     512.10  .... ..... ... .... .....
#    3 . .. T     511.35  .... ..... ... .... .....
#    4 . .. T     515.11  .... ..... ... .... .....
#    5 . .. T     511.54  .... ..... ... .... .....
#    6 . .. T     518.69  .... ..... ... .... .....
#    7 . .. T     510.90  .... ..... ... .... .....
#    8 . .. T     509.48  .... ..... ... .... .....
#    9 . .. T     511.21  .... ..... ... .... .....
Processing config file #0 : /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_io_.qunat_stats_config.txt 
 Freeing memory for user provided Net
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T     525.72  .... ..... ... .... .....
#    1 . .. T     508.73  .... ..... ... .... .....
#    2 . .. T     508.22  .... ..... ... .... .....
#    3 . .. T     510.48  .... ..... ... .... .....
#    4 . .. T     507.76  .... ..... ... .... .....
#    5 . .. T     509.05  .... ..... ... .... .....
#    6 . .. T     508.69  .... ..... ... .... .....
#    7 . .. T     513.02  .... ..... ... .... .....
#    8 . .. T     507.40  .... ..... ... .... .....
#    9 . .. T     510.77  .... ..... ... .... .....
Processing config file #0 : /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_io_.qunat_stats_config.txt 
 Freeing memory for user provided Net
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T     514.55  .... ..... ... .... .....
#    1 . .. T     509.63  .... ..... ... .... .....
#    2 . .. T     511.27  .... ..... ... .... .....
#    3 . .. T     514.48  .... ..... ... .... .....
#    4 . .. T     513.76  .... ..... ... .... .....
#    5 . .. T     512.73  .... ..... ... .... .....
#    6 . .. T     508.84  .... ..... ... .... .....
#    7 . .. T     514.83  .... ..... ... .... .....
#    8 . .. T     513.76  .... ..... ... .... .....
#    9 . .. T     509.66  .... ..... ... .... .....
Processing config file #0 : /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_io_.qunat_stats_config.txt 
 Freeing memory for user provided Net
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T     525.33  .... ..... ... .... .....
#    1 . .. T     508.45  .... ..... ... .... .....
#    2 . .. T     509.07  .... ..... ... .... .....
#    3 . .. T     511.64  .... ..... ... .... .....
#    4 . .. T     509.95  .... ..... ... .... .....
#    5 . .. T     508.44  .... ..... ... .... .....
#    6 . .. T     511.87  .... ..... ... .... .....
#    7 . .. T     512.12  .... ..... ... .... .....
#    8 . .. T     509.16  .... ..... ... .... .....
#    9 . .. T     508.25  .... ..... ... .... .....
Processing config file #0 : /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_io_.qunat_stats_config.txt 
 Freeing memory for user provided Net
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T     511.01  .... ..... ... .... .....
#    1 . .. T     511.14  .... ..... ... .... .....
#    2 . .. T     508.89  .... ..... ... .... .....
#    3 . .. T     511.59  .... ..... ... .... .....
#    4 . .. T     509.40  .... ..... ... .... .....
#    5 . .. T     514.80  .... ..... ... .... .....
#    6 . .. T     512.21  .... ..... ... .... .....
#    7 . .. T     515.02  .... ..... ... .... .....
#    8 . .. T     509.90  .... ..... ... .... .....
#    9 . .. T     509.34  .... ..... ... .... .....
------------------ Network Compiler Traces -----------------------------
================================ import model =============
ration number 4 started ************************ 
 
 
 
Parameters unavailable, running calibration!

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

 
 
 *****************   Calibration iteration number 4 completed ************************ 
 
 
 

 
 
 *****************   Calibration iteration number 5 started ************************ 
 
 
 
Parameters unavailable, running calibration!

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

 
 
 *****************   Calibration iteration number 5 completed ************************ 
 
 
 

 
 
 *****************   Calibration iteration number 6 started ************************ 
 
 
 
Parameters unavailable, running calibration!

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

 
 
 *****************   Calibration iteration number 6 completed ************************ 
 
 
 

 
 
 *****************   Calibration iteration number 7 started ************************ 
 
 
 
Parameters unavailable, running calibration!

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

 
 
 *****************   Calibration iteration number 7 completed ************************ 
 
 
 

 
 
 *****************   Calibration iteration number 8 started ************************ 
 
 
 
Parameters unavailable, running calibration!

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

 
 
 *****************   Calibration iteration number 8 completed ************************ 
 
 
 

 
 
 *****************   Calibration iteration number 9 started ************************ 
 
 
 
Parameters unavailable, running calibration!

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

 
 
 *****************   Calibration iteration number 9 completed ************************ 
 
 
 
Parameters unavailable, running calibration!
Output network quant params prototxt file path: /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_net_quant_params.prototxt
Calibrated Quant Parameters stored in protoTxt format
Could not open /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_net/perfSimInfo.bin
INFORMATION: [TIDL_ResizeLayer] /neck/upsample/Resize Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] /neck/upsample_1/Resize Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
WARNING: [TIDL_E_DATAFLOW_INFO_NULL] Network compiler returned with error or didn't executed, this model can only be used on PC/Host emulation mode, it is not expected to work on target/EVM.
****************************************************
**          3 WARNINGS          0 ERRORS          **
****************************************************

INFO:20240604-113919: import completed  - od-8200 - 86 sec
INFO:20240604-113919: import completed  - od-8200 - 86 sec

SUCCESS:20240604-113919: benchmark results - {}


SUCCESS:20240604-113919: benchmark results - {}

INFO:20240604-113919: running - od-8200
INFO:20240604-113919: pipeline_config - {'task_type': 'detection', 'dataset_category': 'coco', 'calibration_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7848bfefb970>, 'input_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7848f2918070>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x784898e3ac50>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x784898e3acb0>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x784898e3af50>, 'metric': {'label_offset_pred': 1}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': None}, 'model_shortlist': 10}}
INFO:20240604-113919: infer  - od-8200 - this may take some time...
infer : od-8200                                             |   0%|          || 0/107 [00:00<?, ?it/s]
infer : od-8200                                             |          |     0% 0/107| [< ]
infer : od-8200                                             |   0%|          || 0/107 [00:00<?, ?it/s]
infer : od-8200                                             |   0%|          || 0/107 [00:00<?, ?it/s]

Traceback (most recent call last):
  File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/pipelines/pipeline_runner.py", line 203, in _run_pipeline
    result = cls._run_pipeline_impl(basic_settings, pipeline_config, description)
  File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/pipelines/pipeline_runner.py", line 176, in _run_pipeline_impl
    accuracy_result = accuracy_pipeline(description)
  File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 87, in __call__
    param_result = self._run(description=description)
  File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 138, in _run
    output_list = self._infer_frames(description)
  File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 231, in _infer_frames
    output, info_dict = postprocess(output, info_dict)
  File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/utils/transforms_utils.py", line 41, in __call__
    tensor, info_dict = t(tensor, info_dict)
  File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/postprocess/transforms.py", line 599, in __call__
    self.color_map = info_dict['dataset_info']['color_map']
KeyError: 'color_map'
'color_map'

SUCCESS: ModelMaker - Compilation completed.

over 1 year ago

0 Fabiana Jaimes over 1 year ago

TI__Mastermind 19670 points

Hi Makoto,

Which SDK version are you using? Assuming you are using the latest SDK version, could you try running the application with the configuration file below?

root@tda4vm-sk:/opt/edgeai-gst-apps/apps_python# ./app_edgeai.py ../configs/object_detection.yaml

title: "Object Detection"
log_level: 2
inputs:
    input0:
        source: /dev/video-usb-cam0
        format: jpeg
        width: 1280
        height: 720
        framerate: 30
    input1:
        source: /opt/edgeai-test-data/videos/video0_1280_768.h264
        format: h264
        width: 1280
        height: 768
        framerate: 30
        loop: True
    input2:
        source: /opt/edgeai-test-data/images/%04d.jpg
        width: 1280
        height: 720
        index: 0
        framerate: 1
        loop: True
models:
    model0:
        model_path: /opt/model_zoo/TVM-OD-5120-ssdLite-mobDet-DSP-coco-320x320
        viz_threshold: 0.6
    model1:
        model_path: /opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320
        viz_threshold: 0.6
    model2:
        model_path: /opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416
        viz_threshold: 0.6
    model3:
        model_path: /opt/model_zoo/custom_model
        viz_threshold: 0.6
outputs:
    output0:
        sink: kmssink
        width: 1920
        height: 1080
        overlay-perf-type: graph
    output1:
        sink: /opt/edgeai-test-data/output/output_video0.mkv
        width: 1920
        height: 1080
    output2:
        sink: /opt/edgeai-test-data/output/output_image_%04d.jpg
        width: 1920
        height: 1080
    output3:
        sink: remote
        width: 1920
        height: 1080
        port: 8081
        host: 127.0.0.1
        encoding: jpeg
        overlay-perf-type: graph

flows:
    flow0: [input1,model3,output0,[320,150,1280,720]]

Thank you,

Fabiana

0 Manu Mathew over 1 year ago in reply to Fabiana Jaimes

TI__Genius 11506 points

https://github.com/TexasInstruments/edgeai-tensorlab has branch names which corresponds to SDK versions. For example the branch r9.2 corresponds to SDK version 09.02 version of SDK. As of now the main branch in github correspond to r9.2

Does this match with what you are using?

There is an error at the end of your compilation. Any other details that you can give to replicate the setup that you have?

infer : od-8200 | 0%| || 0/107 [00:00<?, ?it/s]

Traceback (most recent call last):
File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/pipelines/pipeline_runner.py", line 203, in _run_pipeline
result = cls._run_pipeline_impl(basic_settings, pipeline_config, description)
File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/pipelines/pipeline_runner.py", line 176, in _run_pipeline_impl
accuracy_result = accuracy_pipeline(description)
File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 87, in __call__
param_result = self._run(description=description)
File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 138, in _run
output_list = self._infer_frames(description)
File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 231, in _infer_frames
output, info_dict = postprocess(output, info_dict)
File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/utils/transforms_utils.py", line 41, in __call__
tensor, info_dict = t(tensor, info_dict)
File "/home/arvis/projects/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/postprocess/transforms.py", line 599, in __call__
self.color_map = info_dict['dataset_info']['color_map']
KeyError: 'color_map'
'color_map'

0 Makoto Nakamura over 1 year ago in reply to Fabiana Jaimes

Prodigy 50 points

Hi Fabiana,
I am using SDK 9.2 to deal with the work about edgeai-modelmaker.

I tried your configuration.
I attached it's log.

Is this helpful to resolve my problem about using edgeai-modelmaker?

Thank you,
Mokoto

Fullscreen 8507.output.txt Download

root@tda4vm-sk:/opt/edgeai-gst-apps/apps_python# cat ../configs/object_detection.yaml
title: Object Detection
log_level: 2
inputs:
    input0:
        source: /dev/video-usb-cam0
        format: jpeg
        width: 1280
        height: 720
        framerate: 30
    input1:
        source: /opt/edgeai-test-data/videos/video0_1280_768.h264
        format: h264
        width: 1280
        height: 768
        framerate: 30
        loop: True
    input2:
        source: /opt/edgeai-test-data/images/%04d.jpg
        width: 1280
        height: 720
        index: 0
        framerate: 1
        loop: True
models:
    model0:
        model_path: /opt/model_zoo/TVM-OD-5120-ssdLite-mobDet-DSP-coco-320x320
        viz_threshold: 0.6
    model1:
        model_path: /opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320
        viz_threshold: 0.6
    model2:
        model_path: /opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416
        viz_threshold: 0.6
    model3:
        model_path: /opt/model_zoo/custom_model
        viz_threshold: 0.6
outputs:
    output0:
        sink: kmssink
        width: 1920
        height: 1080
        overlay-perf-type: graph
    output1:
        sink: /opt/edgeai-test-data/output/output_video0.mkv
        width: 1920
        height: 1080
    output2:
        sink: /opt/edgeai-test-data/output/output_image_%04d.jpg
        width: 1920
        height: 1080
    output3:
        sink: remote
        width: 1920
        height: 1080
        port: 8081
        host: 127.0.0.1
        encoding: jpeg
        overlay-perf-type: graph

flows:
    flow0: [input1,model3,output0,[320,150,1280,720]]
root@tda4vm-sk:/opt/edgeai-gst-apps/apps_python# python3 app_edgeai.py ../configs/object_detection.yaml
libtidl_onnxrt_EP loaded 0x937d250 
Final number of subgraphs created are : 1, - Offloaded Nodes - 283, Total Nodes - 283 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
100558.513974 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
100558.515128 s:  VX_ZONE_INIT:Enabled
100558.515182 s:  VX_ZONE_ERROR:Enabled
100558.515221 s:  VX_ZONE_WARNING:Enabled
100558.515788 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-0 
100558.516732 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-1 
100558.516839 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-2 
100558.517296 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-3 
100558.517316 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
100558.518040 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
100558.531873 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
100558.531922 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
100558.531942 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
100558.531958 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
100558.531977 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
100558.531999 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
100558.532015 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
[ERROR] Could not get allowed GstCaps of device
100558.734194 s:  VX_ZONE_ERROR:[vxGetStatus:1020] Reference is NULL
100558.737897 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
100558.742355 s:  VX_ZONE_INIT:[tivxDeInitLocal:204] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 15 alloc's of 37680924 bytes 
DDR_SHARED_MEM: Free's : 15 free's  of 37680924 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
root@tda4vm-sk:/opt/edgeai-gst-apps/apps_python#

0 Makoto Nakamura over 1 year ago in reply to Manu Mathew

Prodigy 50 points

Hi Manu,
Before I post this question, I already used github.com/.../edgeai-tensorlab repo, To check I're right, I contained command output.
$ git remote get-url origin
gitclone.com/.../edgeai-tensorlab
$ git branch
r9.0
* r9.2
$ cat docs/release_notes.md | grep 9.2
Note: I follow the EdgeAI-SDK (Processor SDK Linux with Edge AI) release numbering. For example, the models compiled with 9.2.0 version (r9.2 branch) works in 09.02.00 (09_02_00) SDK release.
## 9.2.0
### Git branch name: r9.2
- Updated to support TIDL & SDK version 9.2

It's correctly? I think already used correctly SDK version.

Thank you
Makoto

0 Fabiana Jaimes over 1 year ago in reply to Makoto Nakamura

TI__Mastermind 19670 points

Hi Makoto,

Thank you for specifying the SDK and git versions. Could you please call out any other details/steps followed to replicate the issue?

Thanks,

Fabiana

0 Manu Mathew over 1 year ago in reply to Fabiana Jaimes

TI__Genius 11506 points

Due to the recent version updates in pip and setuptools, we need to update the repository in github anyway (Otherwise the setup wouldn't even complete as of now). So may be what I can do is push a new version to github and we can base our conclusions on that.

0 Manu Mathew over 1 year ago in reply to Manu Mathew

TI__Genius 11506 points

New commits have been pushed to r9.2 and main branches.

Makoto, please try again after pulling r9.2 branch and let us know if you face any issues with that.

0 Makoto Nakamura over 1 year ago in reply to Manu Mathew

Prodigy 50 points

Hi Manu,

I tried the newest r9.2, then the compile errors were disappeared.
But when running the model, I got many errors.

※I attached the log as 202406014.txt

Fullscreen 202406014.txt Download

root@tda4vm-sk:/opt/edgeai-gst-apps/apps_python# uname -a
Linux tda4vm-sk 6.1.80-ti-g2e423244f8c0 #1 SMP PREEMPT Wed Mar 20 14:43:33 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

root@tda4vm-sk:/opt/edgeai-gst-apps/apps_python# cat ../configs/object_detection.yaml
title: Object Detection
log_level: 2
inputs:
    input0:
        source: /dev/video-usb-cam0
        format: jpeg
        width: 1280
        height: 720
        framerate: 30
    input1:
        source: /opt/edgeai-test-data/videos/video0_1280_768.h264
        format: h264
        width: 1280
        height: 768
        framerate: 30
        loop: True
    input2:
        source: /opt/edgeai-test-data/images/%04d.jpg
        width: 1280
        height: 720
        index: 0
        framerate: 1
        loop: True
models:
    model0:
        model_path: /opt/model_zoo/TVM-OD-5120-ssdLite-mobDet-DSP-coco-320x320
        viz_threshold: 0.6
    model1:
        model_path: /opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320
        viz_threshold: 0.6
    model2:
        model_path: /opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416
        viz_threshold: 0.6
    model3:
        model_path: /opt/model_zoo/custom_model
        viz_threshold: 0.6
outputs:
    output0:
        sink: kmssink
        width: 1920
        height: 1080
        overlay-perf-type: graph
    output1:
        sink: /opt/edgeai-test-data/output/output_video0.mkv
        width: 1920
        height: 1080
    output2:
        sink: /opt/edgeai-test-data/output/output_image_%04d.jpg
        width: 1920
        height: 1080
    output3:
        sink: remote
        width: 1920
        height: 1080
        port: 8081
        host: 127.0.0.1
        encoding: jpeg
        overlay-perf-type: graph

flows:
    flow0: [input1,model3,output1,[320,150,1280,720]]

root@tda4vm-sk:/opt/edgeai-gst-apps/apps_python# python3 app_edgeai.py ../configs/object_detection.yaml
libtidl_onnxrt_EP loaded 0x139e4f30 
Final number of subgraphs created are : 1, - Offloaded Nodes - 283, Total Nodes - 283 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  7947.968951 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  7947.969038 s:  VX_ZONE_INIT:Enabled
  7947.969049 s:  VX_ZONE_ERROR:Enabled
  7947.969074 s:  VX_ZONE_WARNING:Enabled
  7947.969596 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-0 
  7947.972405 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-1 
  7947.972592 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-2 
  7947.972762 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-3 
  7947.972826 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
  7947.973349 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
  7947.998854 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7947.999033 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7947.999078 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7947.999128 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7947.999170 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7947.999216 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7947.999255 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
==========[INPUT PIPELINE(S)]==========

[PIPE-0]

multifilesrc location=/opt/edgeai-test-data/videos/video0_1280_768.h264 loop=True ! h264parse ! v4l2h264dec capture-io-mode=5 ! tiovxmemalloc pool-size=12 ! capsfilter caps="video/x-raw, format=(string)NV12;" ! tiovxmultiscaler name=split_01
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)1280, height=(int)720;" ! tiovxdlcolorconvert out-pool-size=4 ! capsfilter caps="video/x-raw, format=(string)RGB;" ! appsink max-buffers=2 drop=True name=sen_0
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)416, height=(int)416;" ! tiovxdlpreproc out-pool-size=4 data-type=3 tensor-format=1 ! capsfilter caps="application/x-tensor-tiovx;" ! appsink max-buffers=2 drop=True name=pre_0


==========[OUTPUT PIPELINE]==========

appsrc do-timestamp=True format=3 block=True name=post_0 ! tiovxdlcolorconvert ! capsfilter caps="video/x-raw, format=(string)NV12, width=(int)1280, height=(int)720;" ! queue ! mosaic_0.sink_0

tiovxmosaic target=1 background=/tmp/background_0 name=mosaic_0 src::pool-size=2
sink_0::startx="<320>" sink_0::starty="<150>" sink_0::widths="<1280>" sink_0::heights="<720>"
! capsfilter caps="video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080;" ! v4l2h264enc extra-controls="controls, frame_level_rate_control_enable=(int)1, video_bitrate=(int)10000000, video_gop_size=(int)30;" ! h264parse ! matroskamux writing-app=GStreamer Matroska muxer ! filesink sync=False location=/opt/edgeai-test-data/output/output_video0.mkv

  7948.240359 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.240496 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.240541 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.240581 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.240622 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.240668 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.240706 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.240797 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.240836 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.240875 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.262624 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.262665 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.262692 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.262710 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.262730 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.262757 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.262773 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.262878 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.262907 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.262922 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.296805 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.296847 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.296861 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.296876 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.296887 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.296906 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.296915 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.296980 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.296989 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.296999 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.329408 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.329442 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.329474 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.329492 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.329513 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.329538 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.329555 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.329620 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.329629 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.329637 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.362205 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.362403 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.362471 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.362527 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.362583 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.362651 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.362704 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.362819 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.362887 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.362938 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.452437 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.452624 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.452681 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.452733 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.452783 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.452844 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.452893 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.452998 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.453051 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.453165 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.476271 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.476325 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.476343 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.476360 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.476378 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.476401 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.476416 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.476483 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.476503 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.476517 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.486257 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.486286 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.486316 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.486334 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.486352 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.486373 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.486390 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.486454 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.486472 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.486487 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.501048 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.501161 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.501181 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.501249 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.501298 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.501324 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.501340 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.501408 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.501430 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.501446 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.531013 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.531067 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.531085 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.531101 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.531120 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.531142 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.531159 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.531224 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.531248 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.531264 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.563366 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.563418 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.563438 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.563453 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.563474 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.563501 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.563518 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.563584 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.563601 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.563616 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.595828 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.595860 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.595882 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.595900 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.596525 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.596568 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.596585 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.596655 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.596673 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.596687 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.629258 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.629290 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.629302 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.629313 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.629323 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.629338 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.629346 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.629403 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.629412 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.629420 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.662512 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.662665 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.662676 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.662717 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.662751 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.662837 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.662876 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.662981 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.662991 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.663023 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.695805 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.695838 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.695944 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.695954 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.696089 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.696114 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.696151 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.696259 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.696269 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.696300 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.729195 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.729227 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.729352 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.729361 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.729368 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.729381 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.729387 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.729443 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.729452 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.729457 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.762493 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.762527 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.762683 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.762718 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.762753 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.762817 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.762826 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.762931 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.762940 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.762971 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.795793 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.795825 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.795977 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.796014 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.796085 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.796149 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.796158 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.796261 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.796271 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.796277 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
^C  7948.829308 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  7948.829340 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  7948.829492 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  7948.829501 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  7948.829539 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  7948.829604 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  7948.829614 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  7948.829718 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  7948.829727 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  7948.829758 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  7948.973350 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  7948.977863 s:  VX_ZONE_INIT:[tivxDeInitLocal:204] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 75 alloc's of 132240740 bytes 
DDR_SHARED_MEM: Free's : 75 free's  of 132240740 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

Thank you,

Mokoto

0 Fabiana Jaimes over 1 year ago in reply to Makoto Nakamura

TI__Mastermind 19670 points

Hi Makoto,

These errors seem to be due to some model issue rather than edgeai-gst-apps. I will let Manu comment on this.

Thank you,

Fabiana

0 Manu Mathew over 1 year ago in reply to Fabiana Jaimes

TI__Genius 11506 points

Hi Makoto,

Is it possible to attach the compiled model?

edgeai-modelmaker/data/projects/<dtaset name>/run/<run id>/<model name>/compilation/<target device>/pkg/

There will be the compiled packaged model (.tar.gz file)

0 Makoto Nakamura over 1 year ago in reply to Manu Mathew

Prodigy 50 points

Hi Manu,

I've attached that before.
Attach again.

4214.20240604-111044_yolox_nano_lite_onnxrt_TDA4VM.tar.gz

Thank you,

Makoto

0 Fabiana Jaimes over 1 year ago in reply to Makoto Nakamura

TI__Mastermind 19670 points

Thank you for sharing. Please allow some time for review.

-Fabiana

0 Manu Mathew over 1 year ago in reply to Fabiana Jaimes

TI__Genius 11506 points

Hi Makoto,

Attached is a model that I generated yesterday by running config_detection.yaml It is confirmed to be running fine on the 9.2 SDK. Attaching the model:

manu_ti_20240618-135553_yolox_nano_lite_onnxrt_TDA4VM.tar.gz

But the model that you generated is not running. From this I suspect that the something is not correct in the environment that you are using. Can you do a fresh start i.e. delete the edgeai-tensorlab and clone it again. Create a new python environment and try again.

0 Manu Mathew over 1 year ago in reply to Manu Mathew

TI__Genius 11506 points

In the compilation log that you attached above, can you see the lines:

Could not open /home/arvis/projects/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240604-111044/yolox_nano_lite/compilation/TDA4VM/work/od-8200/artifacts/tempDir/detslabels_tidl_net/perfSimInfo.bin

WARNING: [TIDL_E_DATAFLOW_INFO_NULL] Network compiler returned with error or didn't executed, this model can only be used on PC/Host emulation mode, it is not expected to work on target/EVM.

That indicates that the compilation did not work as expected. (The compilation should have stopped at the point, but it did not stop - that is something for the TIDL to fix).

Once you try with a fresh environment, please check the logs and see if that error is there or not.

0 Makoto Nakamura over 1 year ago in reply to Manu Mathew

Prodigy 50 points

Hello Manu,

I tried to create new clear Ubuntu environment. And I following the steps on the github of edgeai-tensorlab. The build proccess was seemed OK. But when running the model, different errors appeared, and program exited.
※I've recreated the environment for many times.

Attached the history.txt, which shows how I create the enviroment with newly installed Ubuntu. Is there any problem with my step?

And also attach the model I build, and the log when I run it.

Fullscreen history.txt Download

  366  sudo apt update
  367  sudo apt install build-essential curl libbz2-dev libffi-dev liblzma-dev libncursesw5-dev libreadline-dev libsqlite3-dev libssl-dev libxml2-dev libxmlsec1-dev llvm make tk-dev xz-utils wget curl
  368  curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
  371  git config --global --unset http.proxy
  372  git config --global --unset https.proxy
  373  curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
  374  echo '# pyenv settings ' >> ${HOME}/.bashrc
  375  echo 'command -v pyenv >/dev/null || export PATH=":${HOME}/.pyenv/bin:$PATH"' >> ${HOME}/.bashrc
  376  echo 'eval "$(pyenv init -)"' >> ${HOME}/.bashrc
  377  echo 'eval "$(pyenv virtualenv-init -)"' >> ${HOME}/.bashrc
  378  echo '' >> ${HOME}/.bashrc
  379  exec ${SHELL}
  380  pyenv install 3.10
  381  pyenv virtualenv 3.10 py310
  382  pyenv rehash
  383  pyenv activate py310
  384  pip install --upgrade pip setuptools
  385  git clone https://github.com/TexasInstruments/edgeai-tensorlab
  386  cd edgeai-tensorlab/
  387  ./setup_all.sh
  388  cd edgeai-modelmaker/
  389  ./setup_all.sh
  401  ./run_modelmaker.sh TDA4VM config_detection.yaml
  402  nano requirements_freeze.txt 
  403  pip install mmcv-full==1.4.8
  404  pyenv activate py310
  405  pip install mmcv-full==1.4.8
  406  cd edgeai-tensorlab/edgeai-modelmaker/
  407  ls
  408  ./run_modelmaker.sh TDA4VM config_detection.yaml
  412  ls -l /home/wjx/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240620-151129/yolox_nano_lite/compilation/TDA4VM/pkg/20240620-151129_yolox_nano_lite_onnxrt_TDA4VM.tar.gz
  413  ping 192.168.1.110
  414  scp /home/wjx/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20240620-151129/yolox_nano_lite/compilation/TDA4VM/pkg/od-8200.tar.gz root@192.168.1.110:~/

Fullscreen 6330.output1.txt Download

root@tda4vm-sk:/opt/edgeai-gst-apps/apps_python# python3 app_edgeai.py ../configs/object_detection.yam                                                                                                                                       l
libtidl_onnxrt_EP loaded 0x3f244d90
ERROR : artifacts_folder not a directory
******** WARNING ******* : Could not open ConstantFolding/allowedNode.txt for reading... Entire model                                                                                                                                        will run on ARM without any delegation to TIDL !
Final number of subgraphs created are : 1, - Offloaded Nodes - 0, Total Nodes - 0
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
 12879.285192 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
 12879.290680 s:  VX_ZONE_INIT:Enabled
 12879.290691 s:  VX_ZONE_ERROR:Enabled
 12879.290696 s:  VX_ZONE_WARNING:Enabled
 12879.292946 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-0
 12879.293045 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-1
 12879.293117 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-2
 12879.293494 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-3
 12879.293508 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
 12879.298019 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
==========[INPUT PIPELINE(S)]==========

[PIPE-0]

multifilesrc location=/opt/edgeai-test-data/videos/video0_1280_768.h264 loop=True ! h264parse ! v4l2h2                                                                                                                                       64dec capture-io-mode=5 ! tiovxmemalloc pool-size=12 ! capsfilter caps="video/x-raw, format=(string)NV                                                                                                                                       12;" ! tiovxmultiscaler name=split_01
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)1280, height=(int)720;" ! tiovxdlcolorcon                                                                                                                                       vert out-pool-size=4 ! capsfilter caps="video/x-raw, format=(string)RGB;" ! appsink max-buffers=2 drop                                                                                                                                       =True name=sen_0
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)416, height=(int)416;" ! tiovxdlpreproc o                                                                                                                                       ut-pool-size=4 data-type=3 tensor-format=1 ! capsfilter caps="application/x-tensor-tiovx;" ! appsink m                                                                                                                                       ax-buffers=2 drop=True name=pre_0


==========[OUTPUT PIPELINE]==========

appsrc do-timestamp=True format=3 block=True name=post_0 ! tiovxdlcolorconvert ! capsfilter caps="vide                                                                                                                                       o/x-raw, format=(string)NV12, width=(int)1280, height=(int)720;" ! queue ! mosaic_0.sink_0

tiovxmosaic target=1 background=/tmp/background_0 name=mosaic_0 src::pool-size=2
sink_0::startx="<320>" sink_0::starty="<150>" sink_0::widths="<1280>" sink_0::heights="<720>"
! capsfilter caps="video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080;" ! v4l2h264enc                                                                                                                                        extra-controls="controls, frame_level_rate_control_enable=(int)1, video_bitrate=(int)10000000, video_                                                                                                                                       gop_size=(int)30;" ! h264parse ! matroskamux writing-app=GStreamer Matroska muxer ! filesink sync=Fals                                                                                                                                       e location=/opt/edgeai-test-data/output/output_video0.mkv

[   82.024228] unclassified data detected!
[   82.072778] unclassified data detected!
[   82.110373] kauditd_printk_skb: 1 callbacks suppressed
[   82.110381] audit: type=1701 audit(1718861225.813:24): auid=4294967295 uid=0 gid=0 ses=4294967295 p                                                                                                                                       id=1485 comm="python3" exe="/usr/bin/python3.10" sig=11 res=1
[   82.146287] audit: type=1334 audit(1718861225.849:25): prog-id=17 op=LOAD
[   82.153114] audit: type=1334 audit(1718861225.857:26): prog-id=18 op=LOAD
Segmentation fault (core dumped)

Fullscreen 6330.output2.txt Download

root@tda4vm-sk:/opt/edgeai-gst-apps/apps_python# cat ../configs/object_detection.yaml
title: Object Detection
log_level: 2
inputs:
    input0:
        source: /dev/video-usb-cam0
        format: jpeg
        width: 1280
        height: 720
        framerate: 30
    input1:
        source: /opt/edgeai-test-data/videos/video0_1280_768.h264
        format: h264
        width: 1280
        height: 768
        framerate: 30
        loop: True
    input2:
        source: /opt/edgeai-test-data/images/%04d.jpg
        width: 1280
        height: 720
        index: 0
        framerate: 1
        loop: True
models:
    model0:
        model_path: /opt/model_zoo/TVM-OD-5120-ssdLite-mobDet-DSP-coco-320x320
        viz_threshold: 0.6
    model1:
        model_path: /opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320
        viz_threshold: 0.6
    model2:
        model_path: /opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416
        viz_threshold: 0.6
    model3:
        model_path: /opt/model_zoo/custom_model
        viz_threshold: 0.6
outputs:
    output0:
        sink: kmssink
        width: 1920
        height: 1080
        overlay-perf-type: graph
    output1:
        sink: /opt/edgeai-test-data/output/output_video0.mkv
        width: 1920
        height: 1080
    output2:
        sink: /opt/edgeai-test-data/output/output_image_%04d.jpg
        width: 1920
        height: 1080
    output3:
        sink: remote
        width: 1920
        height: 1080
        port: 8081
        host: 127.0.0.1
        encoding: jpeg
        overlay-perf-type: graph

flows:
    flow0: [input1,model3,output1,[320,150,1280,720]]
root@tda4vm-sk:/opt/edgeai-gst-apps# ls /opt/model_zoo/custom_model
config.yaml  dataset.yaml  model  param.yaml  run.log
root@tda4vm-sk:/opt/model_zoo# tar -zcvf custom_model.tgz custom_model/
custom_model/
custom_model/dataset.yaml
custom_model/model/
custom_model/model/model.prototxt
custom_model/model/model.onnx
custom_model/param.yaml
custom_model/config.yaml
custom_model/run.log

6330.custom_model.zip

Thank you,

Makoto

0 Manu Mathew over 1 year ago in reply to Makoto Nakamura

TI__Genius 11506 points

Inside the zip file that you shared, I can see the model folder. But the artifacts folder is missing. Wondering why is it so.

0 Makoto Nakamura over 1 year ago in reply to Manu Mathew

Prodigy 50 points

Hello Manu,

Yes, the log also show that problem (artifacts folder is missing). I am sure I followed the steps on the github of edgeai-tensorlab. So I don't know why this happened.
Can you tell me how should I do to deal with this?

Thank you,

Makoto

0 Manu Mathew over 1 year ago in reply to Makoto Nakamura

TI__Genius 11506 points

Given that it works in my machine and it doesn't work in your case, I can think of two options:

(1) How about asking one of your colleague to try it. We can see if it is something specific to your machine.

(2) Or is it possible to share a sample or a small subset of your dataset? I can train/compile it and share with you for you to check.

Processors

Processors forum

TDA4VM: edgeai-modelmaker