TDA4VM: Custom model creation problem (edge-ai tidl-tools)

Hi Sefa,

Can you share the error that you get?

Can you please try yolov5s6

You can first try the yolov5s6 model that we have shared and after that you can try the model that you will train.

Just trying to understand if this crash is due to memory shortage, because yolov5l6 is a large model.

Adding "onnx shape inference" part in onnxrt_ep.py file fixed the problem. Now the artifacts folder is created.

    #onnx shape inference
    if not os.path.isfile(os.path.join(models_base_path, model + '_shape.onnx')):
        print("Writing model with shapes after running onnx shape inference -- ", os.path.join(models_base_path, model + '_shape.onnx'))
        onnx.shape_inference.infer_shapes_path(config['model_path'], config['model_path'])#os.path.join(models_base_path, model + '_shape.onnx'))

sefau18@ubuntu:~/edgeai-tidl-tools/examples/osrt_python/ort$ python3 onnxrt_ep.py -c
Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 1 Models - ['best']


Running_Model :  best  

Writing model with shapes after running onnx shape inference --  ../../../models/public/best_shape.onnx
TIDL Meta PipeLine (Proto) File  : /home/sefau18/edgeai-tidl-tools/models/public/best.prototxt  
yolo_v3
yolo_v3
Number of OD backbone nodes = 0 
Size of odBackboneNodeIds = 0 

Preliminary subgraphs created = 0 
Final number of subgraphs created are : 0, - Offloaded Nodes - 0, Total Nodes - 1 
TIDL Meta PipeLine (Proto) File  : /home/sefau18/edgeai-tidl-tools/models/public/best.prototxt  
yolo_v3
yolo_v3
Number of OD backbone nodes = 0 
Size of odBackboneNodeIds = 0 

Preliminary subgraphs created = 0 
Final number of subgraphs created are : 0, - Offloaded Nodes - 0, Total Nodes - 1 
TIDL Meta PipeLine (Proto) File  : /home/sefau18/edgeai-tidl-tools/models/public/best.prototxt  
yolo_v3
yolo_v3
Number of OD backbone nodes = 328 
Size of odBackboneNodeIds = 328 

Preliminary subgraphs created = 1 
Final number of subgraphs created are : 1, - Offloaded Nodes - 434, Total Nodes - 434 
TIDL Meta PipeLine (Proto) File  : /home/sefau18/edgeai-tidl-tools/models/public/best.prototxt  
yolo_v3
yolo_v3
Warning : Requested Output Data Convert Layer is not Added to the network, It is currently not Optimal

 ************** Frame index 1 : Running float import ************* 
INFORMATION: [TIDL_ResizeLayer] Resize_195 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] Resize_219 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] Resize_243 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
WARNING: [TIDL_E_DATAFLOW_INFO_NULL] ti_cnnperfsim.out fails to allocate memory in MSMC. Please look into perfsim log. This model can only be used on PC emulation, it will get fault on target.
****************************************************
**          4 WARNINGS          0 ERRORS          **
****************************************************
 0.0s:  VX_ZONE_INIT:Enabled
 0.36s:  VX_ZONE_ERROR:Enabled
 0.40s:  VX_ZONE_WARNING:Enabled
 0.1132s:  VX_ZONE_INIT:[tivxInit:178] Initialization Done !!!

**********  Frame Index 1 : Running float inference **********

**********  Frame Index 2 : Running fixed point mode for calibration **********

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/sefau18/edgeai-tidl-tools/model-artifacts/best/tempDir/detections_tidl_io_.qunat_stats_config.txt 
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   62484.14  .... ..... ... .... .....
#    1 . .. T   64024.67  .... ..... ... .... .....
~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/sefau18/edgeai-tidl-tools/model-artifacts/best/tempDir/detections_tidl_io_.qunat_stats_config.txt 
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   38282.48  .... ..... ... .... .....
#    1 . .. T   38302.39  .... ..... ... .... .....
 
 
 *****************   Calibration iteration number 0 completed ************************ 
 
 
 

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/sefau18/edgeai-tidl-tools/model-artifacts/best/tempDir/detections_tidl_io_.qunat_stats_config.txt 
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   38131.61  .... ..... ... .... .....
#    1 . .. T   38735.09  .... ..... ... .... .....
 
 
 *****************   Calibration iteration number 1 completed ************************ 
 
 
 

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/sefau18/edgeai-tidl-tools/model-artifacts/best/tempDir/detections_tidl_io_.qunat_stats_config.txt 
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   39747.21  .... ..... ... .... .....
#    1 . .. T   40037.01  .... ..... ... .... .....
 
 
 *****************   Calibration iteration number 2 completed ************************ 
 
 
 

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/sefau18/edgeai-tidl-tools/model-artifacts/best/tempDir/detections_tidl_io_.qunat_stats_config.txt 
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   39015.66  .... ..... ... .... .....
#    1 . .. T   39431.54  .... ..... ... .... .....
 
 
 *****************   Calibration iteration number 3 completed ************************ 
 
 
 

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/sefau18/edgeai-tidl-tools/model-artifacts/best/tempDir/detections_tidl_io_.qunat_stats_config.txt 
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   38673.23  .... ..... ... .... .....
#    1 . .. T   36513.44  .... ..... ... .... .....
 
 
 *****************   Calibration iteration number 4 completed ************************ 
 
 
 

------------------ Network Compiler Traces -----------------------------
successful Memory allocation
substitute string tidl_net_ not found
INFORMATION: [TIDL_ResizeLayer] Resize_195 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] Resize_219 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] Resize_243 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
****************************************************
**          3 WARNINGS          0 ERRORS          **
****************************************************

 
Completed_Model :     1, Name : best                                              , Total time :  333155.71, Offload Time :   62897.73 , DDR RW MBs : 0, Output File : py_out_best_ADE_val_00001801.jpg 
 
 
sefau18@ubuntu:~/edgeai-tidl-tools/examples/osrt_python/ort$ 

best/
├── allowedNode.txt
├── best.onnx
├── detections_tidl_io_1.bin
├── detections_tidl_net.bin
├── onnxrtMetaData.txt
├── param.yaml
└── tempDir
    ├── detections_calib_raw_data.bin
    ├── detections_tidl_io_1.bin
    ├── detections_tidl_io__LayerPerChannelMean.bin
    ├── detections_tidl_io_.perf_sim_config.txt
    ├── detections_tidl_io_.qunat_stats_config.txt
    ├── detections_tidl_io__stats_tool_out.bin
    ├── detections_tidl_net
    │   ├── bufinfolog.csv
    │   ├── bufinfolog.txt
    │   └── perfSimInfo.bin
    ├── detections_tidl_net.bin
    ├── detections_tidl_net.bin.layer_info.txt
    ├── detections_tidl_net.bin_netLog.txt
    ├── detections_tidl_net.bin_paramDebug.csv
    ├── detections_tidl_net.bin.svg
    ├── graphvizInfo.txt
    └── runtimes_visualization.svg

2 directories, 22 files

https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-sk-tda4vm/latest/exports/docs/inference_models.html

As mentioned in the above document, I manually create the folder structure for inference using edgeai apps python on TDA4VM.

Then I copy the folder I created to TDA4M.

Then I edit the onject_detection.yaml file on TDA4VM.

title: "YoloV5 Object Detection Test"
log_level: 2
inputs:
    input0:
        source: /dev/video2
        format: jpeg
        width: 1280
        height: 720
        framerate: 30
    input1:
        source: /opt/edge_ai_apps/data/videos/video_0000_h264.mp4
        format: h264_sw
        width: 1280
        height: 720
        framerate: 30
        loop: True
    input2:
        source: /dev/video18
        width: 1936
        height: 1100
        format: rggb12
        subdev-id: 2
        framerate: 30
        sen-id: imx390
        ldc: True 
    input3:
        source: /opt/edge_ai_apps/data/images/%04d.jpg
        width: 1280
        height: 720
        index: 0
        framerate: 1
        loop: True
models:
    model0:
        model_path: /opt/model_zoo/best_onnx
        viz_threshold: 0.6
    model1:
        model_path: /opt/model_zoo/od-8100_onnxrt_coco_edgeai-yolov5_yolov5s6_640_ti_lite_37p4_56p0_onnx
        viz_threshold: 0.6
    model2:
        model_path: /opt/model_zoo/ONR-OD-8050-ssd-lite-regNetX-800mf-fpn-bgr-mmdet-coco-512x512
        viz_threshold: 0.6
outputs:
    output0:
        sink: kmssink
        width: 1920
        height: 1080
    output1:
        sink: /opt/edge_ai_apps/data/output/videos/output_video2.mkv
        width: 1920
        height: 1080
    output2:
        sink: /opt/edge_ai_apps/data/output/images/output_image_%04d.jpg
        width: 1920
        height: 1080

flows:
    flow0:
        input: input2
        models: [model0]
        outputs: [output0]
        mosaic:
            mosaic0:
                width:  1280
                height: 720
                pos_x:  320
                pos_y:  180

Here is the error I got on TDA4VM.

root@j7-evm:/opt/edge_ai_apps# ./init_script.sh 
IMX390 Camera 0 detected
    device = /dev/video18
    name = imx390 10-0021
    format = [fmt:SRGGB12_1X12/1936x1100 field: none]
    subdev_id = /dev/v4l-subdev7
    isp_required = yes
    ldc_required = yes
root@j7-evm:/opt/edge_ai_apps# cd apps_python/
root@j7-evm:/opt/edge_ai_apps/apps_python# ./app_edgeai.py ../configs/object_detection.yaml 
libtidl_onnxrt_EP loaded 0x2a7d53c0 
Final number of subgraphs created are : 1, - Offloaded Nodes - 434, Total Nodes - 434 
2022-06-15 05:46:45.764130779 [E:onnxruntime:, inference_session.cc:1310 operator()] Exception during initialization: /usr/src/debug/onnxruntime/1.7.0-r0_psdkla_4/git/include/onnxruntime/core/graph/graph.h:1299 onnxruntime::Node* onnxruntime::Graph::NodeAtIndexImpl(onnxruntime::NodeIndex) const node_index < nodes_.size() was false. Validating no unexpected access using an invalid node_index. Got:28271 Max:1

Traceback (most recent call last):
  File "./app_edgeai.py", line 71, in <module>
    main(sys.argv)
  File "./app_edgeai.py", line 45, in main
    demo = EdgeAIDemo(config)
  File "/opt/edge_ai_apps/apps_python/edge_ai_class.py", line 74, in __init__
    model_obj = config_parser.Model(model_config)
  File "/opt/edge_ai_apps/apps_python/config_parser.py", line 136, in __init__
    self.run_time = RunTime(self)
  File "/opt/edge_ai_apps/apps_python/run_times.py", line 109, in __init__
    self.interpreter = onnxruntime.InferenceSession(params.model_path,\
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 283, in __init__
    self._create_inference_session(providers, provider_options)
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 315, in _create_inference_session
    sess.initialize_session(providers, provider_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /usr/src/debug/onnxruntime/1.7.0-r0_psdkla_4/git/include/onnxruntime/core/graph/graph.h:1299 onnxruntime::Node* onnxruntime::Graph::NodeAtIndexImpl(onnxruntime::NodeIndex) const node_index < nodes_.size() was false. Validating no unexpected access using an invalid node_index. Got:28271 Max:1

root@j7-evm:/opt/edge_ai_apps/apps_python# 

I'm not sure I'm following the right steps.

Thank you for your answers

Does the model inference (using the compiled artifacts) work when using the script provided in edgeai-tidl-tools?

As i suggested earlier please try yolov5s6 and make sure it works.

https://github.com/TexasInstruments/edgeai-yolov5/tree/master/pretrained_models/models/yolov5s6_640_ti_lite/weights

If I download the "best.pt" file from the above link and do exactly the same, I get the same error.

root@j7-evm:/opt/edge_ai_apps/apps_python# ./app_edgeai.py ../configs/object_detection.yaml 
libtidl_onnxrt_EP loaded 0x1973190 
Final number of subgraphs created are : 1, - Offloaded Nodes - 298, Total Nodes - 298 
2022-06-15 07:30:20.151819751 [E:onnxruntime:, inference_session.cc:1310 operator()] Exception during initialization: /usr/src/debug/onnxruntime/1.7.0-r0_psdkla_4/git/include/onnxruntime/core/graph/graph.h:1299 onnxruntime::Node* onnxruntime::Graph::NodeAtIndexImpl(onnxruntime::NodeIndex) const node_index < nodes_.size() was false. Validating no unexpected access using an invalid node_index. Got:28271 Max:1

Traceback (most recent call last):
  File "./app_edgeai.py", line 71, in <module>
    main(sys.argv)
  File "./app_edgeai.py", line 45, in main
    demo = EdgeAIDemo(config)
  File "/opt/edge_ai_apps/apps_python/edge_ai_class.py", line 74, in __init__
    model_obj = config_parser.Model(model_config)
  File "/opt/edge_ai_apps/apps_python/config_parser.py", line 136, in __init__
    self.run_time = RunTime(self)
  File "/opt/edge_ai_apps/apps_python/run_times.py", line 109, in __init__
    self.interpreter = onnxruntime.InferenceSession(params.model_path,\
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 283, in __init__
    self._create_inference_session(providers, provider_options)
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 315, in _create_inference_session
    sess.initialize_session(providers, provider_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /usr/src/debug/onnxruntime/1.7.0-r0_psdkla_4/git/include/onnxruntime/core/graph/graph.h:1299 onnxruntime::Node* onnxruntime::Graph::NodeAtIndexImpl(onnxruntime::NodeIndex) const node_index < nodes_.size() was false. Validating no unexpected access using an invalid node_index. Got:28271 Max:1

https://github.com/TexasInstruments/edgeai-yolov5/blob/master/pretrained_models/modelartifacts/8bits/od-8100_onnxrt_coco_edgeai-yolov5_yolov5s6_640_ti_lite_37p4_56p0_onnx.tar.gz.link

But I have no problem running the previously compiled model on TDA4VM. Everything is working as it should and I can see the inference results on the TDA4M screen.

With my custom model, I can infer with the following code.

sefau18@ubuntu:~/edgeai-tidl-tools/examples/osrt_python/ort$ python3 onnxrt_ep.py -d
Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 1 Models - ['best']


Running_Model :  best  

Writing model with shapes after running onnx shape inference --  ../../../models/public/best_shape.onnx

Saving image to  ../../../output_images/

 
Completed_Model :     1, Name : best                                              , Total time :     459.70, Offload Time :       0.00 , DDR RW MBs : 0, Output File : py_out_best_ADE_val_00001801.jpg

py_out_best_ADE_val_00001801.jpg

Likewise, the output of the yolov5s6 model, which I compiled myself, works just as well.

py_out_yolov5s6_ADE_val_00001801.jpg

shape_inference is required for onnx models - for TIDL to be able to compile them correctly.

Has your issue been resolved?

Inference works fine on PC. But if I copy the folders to the EVM and run the script mentioned in the repo, I get an error.

Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 1 Models - ['best']


Running_Model :  best  

libtidl_onnxrt_EP loaded 0x2d61e220 

******** WARNING ******* : Could not open ../../../model-artifacts//best//allowedNode.txt for reading... Entire model will run on ARM without any delegation to TIDL !
Final number of subgraphs created are : 1, - Offloaded Nodes - 0, Total Nodes - 0 

******** WARNING ******* : Could not open ../../../model-artifacts//best//allowedNode.txt for reading... Entire model will run on ARM without any delegation to TIDL !
Final number of subgraphs created are : 1, - Offloaded Nodes - 0, Total Nodes - 0 

******** WARNING ******* : Could not open ../../../model-artifacts//best//allowedNode.txt for reading... Entire model will run on ARM without any delegation to TIDL !
Final number of subgraphs created are : 1, - Offloaded Nodes - 0, Total Nodes - 65535 
./scripts/run_python_examples.sh: line 23:  3351 Killed                  python3 onnxrt_ep.py

If I do the same operations with the pre-trained yolov5s6 model, I get a different error.

Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 1 Models - ['yolov5s6']


Running_Model :  yolov5s6  

Traceback (most recent call last):
  File "onnxrt_ep.py", line 251, in <module>
    run_model(model, mIdx)
  File "onnxrt_ep.py", line 168, in run_model
    sess = rt.InferenceSession(config['model_path'] ,providers=EP_list, provider_options=[delegate_options, {}], sess_options=so)
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 283, in __init__
    self._create_inference_session(providers, provider_options)
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 310, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from /opt/edgeai-tidl-tools/model-artifacts/yolov5s6/model/yolov5s6.onnx failed:/usr/src/debug/onnxruntime/1.7.0-r0_psdkla_4/git/onnxruntime/core/graph/model.cc:101 onnxruntime::Model::Model(onnx::ModelProto&&, const PathString&, const IOnnxRuntimeOpSchemaRegistryList*, const onnxruntime::logging::Logger&) ModelProto does not have a graph.

Hi ,

Looks like model-artifacts are not copied to expected path. Before concluding anything,  for a sanity check of the setup could you please run the default model examples in the python by reverting back the models to 

models = ['cl-ort-resnet18-v1', 'cl-ort-caffe_squeezenet_v1_1', 'ss-ort-deeplabv3lite_mobilenetv2', 'od-ort-ssd-lite_mobilenetv2_fp']

İşte orijinal sonuçlar. Çıkarım sonuçları "output_images" klasörüne eklenmiş gibi görünüyor.

APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
MEM: Alloc's: 10 alloc's of 18610376 bytes 
MEM: Free's : 10 free's  of 18610376 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 4 Models - ['cl-ort-resnet18-v1', 'cl-ort-caffe_squeezenet_v1_1', 'ss-ort-deeplabv3lite_mobilenetv2', 'od-ort-ssd-lite_mobilenetv2_fpn']


Running_Model :  cl-ort-resnet18-v1  

2022-06-22 13:57:50.723167216 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer4.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723244407 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer4.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723301998 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer4.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723323513 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer2.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723345268 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer1.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723362286 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer3.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723379089 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer1.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723396927 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer1.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723414165 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer2.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723432629 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer1.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723450827 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer2.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723469241 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer4.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723486188 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer4.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723505532 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer3.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723523096 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer2.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723540684 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723562079 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer2.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723582698 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer3.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723603603 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer3.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-22 13:57:50.723622707 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer3.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
libtidl_onnxrt_EP loaded 0x33a7c3c0 
Final number of subgraphs created are : 1, - Offloaded Nodes - 52, Total Nodes - 52 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=4) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
 11432.463707 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
 11432.463787 s:  VX_ZONE_INIT:Enabled
 11432.463806 s:  VX_ZONE_ERROR:Enabled
 11432.463823 s:  VX_ZONE_WARNING:Enabled
 11432.464893 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
 11432.465145 s:  VX_ZONE_INIT:[tivxHostInitLocal:86] Initialization Done for HOST !!!

 ,  0  23.579361  warplane, military plane ,,  1  22.507572  aircraft carrier, carrier, flattop, attack aircraft carrier ,,  2  18.756310  projectile, missile ,,  3  18.488363  missile ,,  4  15.808890  airliner ,

Saving image to  ../../../output_images/

 
Completed_Model :     1, Name : cl-ort-resnet18-v1                                , Total time :       2.41, Offload Time :       2.33 , DDR RW MBs : 0, Output File : py_out_cl-ort-resnet18-v1_airshow.jpg 
 
 
 11433.459357 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:100] De-Initialization Done for HOST !!!
 11433.462062 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
MEM: Alloc's: 7 alloc's of 12769620 bytes 
MEM: Free's : 7 free's  of 12769620 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

Running_Model :  cl-ort-caffe_squeezenet_v1_1  

libtidl_onnxrt_EP loaded 0x33a7c3c0 
Final number of subgraphs created are : 1, - Offloaded Nodes - 68, Total Nodes - 68 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=4) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
 11433.527551 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
 11433.527587 s:  VX_ZONE_INIT:Enabled
 11433.528162 s:  VX_ZONE_ERROR:Enabled
 11433.528758 s:  VX_ZONE_WARNING:Enabled
 11433.529373 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
 11433.529616 s:  VX_ZONE_INIT:[tivxHostInitLocal:86] Initialization Done for HOST !!!

 ,  0  0.516560  warplane, military plane ,,  1  0.319705  aircraft carrier, carrier, flattop, attack aircraft carrier ,,  2  0.108509  airliner ,,  3  0.022795  missile ,,  4  0.017946  projectile, missile ,

Saving image to  ../../../output_images/

 
Completed_Model :     2, Name : cl-ort-caffe_squeezenet_v1_1                      , Total time :       1.16, Offload Time :       1.08 , DDR RW MBs : 0, Output File : py_out_cl-ort-caffe_squeezenet_v1_1_airshow.jpg 
 
 
 11434.332022 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:100] De-Initialization Done for HOST !!!
 11434.335435 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
MEM: Alloc's: 7 alloc's of 2388592 bytes 
MEM: Free's : 7 free's  of 2388592 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

Running_Model :  ss-ort-deeplabv3lite_mobilenetv2  

libtidl_onnxrt_EP loaded 0x33a7c3c0 
Final number of subgraphs created are : 1, - Offloaded Nodes - 124, Total Nodes - 124 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=4) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
 11434.469180 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
 11434.470050 s:  VX_ZONE_INIT:Enabled
 11434.470063 s:  VX_ZONE_ERROR:Enabled
 11434.470070 s:  VX_ZONE_WARNING:Enabled
 11434.470448 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
 11434.470515 s:  VX_ZONE_INIT:[tivxHostInitLocal:86] Initialization Done for HOST !!!

Saving image to  ../../../output_images/

 
Completed_Model :     3, Name : ss-ort-deeplabv3lite_mobilenetv2                  , Total time :      10.99, Offload Time :      10.85 , DDR RW MBs : 0, Output File : py_out_ss-ort-deeplabv3lite_mobilenetv2_ADE_val_00001801.jpg 
 
 
 11452.035294 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:100] De-Initialization Done for HOST !!!
 11452.038730 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
MEM: Alloc's: 7 alloc's of 5321724 bytes 
MEM: Free's : 7 free's  of 5321724 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

Running_Model :  od-ort-ssd-lite_mobilenetv2_fpn  

libtidl_onnxrt_EP loaded 0x33a7c3c0 
Final number of subgraphs created are : 1, - Offloaded Nodes - 494, Total Nodes - 494 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=4) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
 11452.875865 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
 11452.876499 s:  VX_ZONE_INIT:Enabled
 11452.876788 s:  VX_ZONE_ERROR:Enabled
 11452.877037 s:  VX_ZONE_WARNING:Enabled
 11452.877645 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
 11452.878139 s:  VX_ZONE_INIT:[tivxHostInitLocal:86] Initialization Done for HOST !!!

Saving image to  ../../../output_images/

 
Completed_Model :     4, Name : od-ort-ssd-lite_mobilenetv2_fpn                   , Total time :      11.99, Offload Time :      11.70 , DDR RW MBs : 0, Output File : py_out_od-ort-ssd-lite_mobilenetv2_fpn_ADE_val_00001801.jpg 
 
 
 11467.142170 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:100] De-Initialization Done for HOST !!!
 11467.146557 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
MEM: Alloc's: 8 alloc's of 5527700 bytes 
MEM: Free's : 8 free's  of 5527700 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!


Running Inference on Model -  ../../../model-artifacts/cl-dlr-tflite_inceptionnetv3_device

2022-06-22 13:58:26,334 ERROR error in DLRModel instantiation model_path ../../../model-artifacts/cl-dlr-tflite_inceptionnetv3_device doesn't exist
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/dlr/api.py", line 89, in __init__
    self._impl = DLRModelImpl(model_path, dev_type, dev_id, error_log_file, use_default_dlr)
  File "/usr/lib/python3.8/site-packages/dlr/dlr_model.py", line 65, in __init__
    raise ValueError("model_path %s doesn't exist" % model_path)
ValueError: model_path ../../../model-artifacts/cl-dlr-tflite_inceptionnetv3_device doesn't exist
Traceback (most recent call last):
  File "dlr_inference_example.py", line 192, in <module>
    model_create_and_run(model_output_directory, 'input',
  File "dlr_inference_example.py", line 157, in model_create_and_run
    model = DLRModel(model_dir, 'cpu')
  File "/usr/lib/python3.8/site-packages/dlr/api.py", line 92, in __init__
    raise ex
  File "/usr/lib/python3.8/site-packages/dlr/api.py", line 89, in __init__
    self._impl = DLRModelImpl(model_path, dev_type, dev_id, error_log_file, use_default_dlr)
  File "/usr/lib/python3.8/site-packages/dlr/dlr_model.py", line 65, in __init__
    raise ValueError("model_path %s doesn't exist" % model_path)
ValueError: model_path ../../../model-artifacts/cl-dlr-tflite_inceptionnetv3_device doesn't exist

Hi,

Thanks. Looks like setup is good. 

Now let us figure out the error : " Could not open ../../../model-artifacts//best//allowedNode.tx"

could you verify these file  exist :

1)model-artifacts//best//allowedNode.txt

2)the model exist at models/public/best_shape.onnx

Also jus to avoid confusion can you please update the dictionary in /edgeai-tidl-tools/examples/osrt_python/common_utils.py

'best' : {
'model_path' : os.path.join('/home/sefau18/edgeai-tidl-tools/models/public/best.onnx'),

}

Here is the error I got after making the change you mentioned.

Running 1 Models - ['best']


Running_Model :  best  

libtidl_onnxrt_EP loaded 0x39b3120 
Final number of subgraphs created are : 1, - Offloaded Nodes - 434, Total Nodes - 434 
2022-06-23 06:37:14.949809627 [E:onnxruntime:, inference_session.cc:1310 operator()] Exception during initialization: /usr/src/debug/onnxruntime/1.7.0-r0_psdkla_4/git/include/onnxruntime/core/graph/graph.h:1299 onnxruntime::Node* onnxruntime::Graph::NodeAtIndexImpl(onnxruntime::NodeIndex) const node_index < nodes_.size() was false. Validating no unexpected access using an invalid node_index. Got:28271 Max:1

Traceback (most recent call last):
  File "onnxrt_ep.py", line 251, in <module>
    run_model(model, mIdx)
  File "onnxrt_ep.py", line 168, in run_model
    sess = rt.InferenceSession(config['model_path'] ,providers=EP_list, provider_options=[delegate_options, {}], sess_options=so)
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 283, in __init__
    self._create_inference_session(providers, provider_options)
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 315, in _create_inference_session
    sess.initialize_session(providers, provider_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /usr/src/debug/onnxruntime/1.7.0-r0_psdkla_4/git/include/onnxruntime/core/graph/graph.h:1299 onnxruntime::Node* onnxruntime::Graph::NodeAtIndexImpl(onnxruntime::NodeIndex) const node_index < nodes_.size() was false. Validating no unexpected access using an invalid node_index. Got:28271 Max:1

Hi , 

Could you run the model without delegate mode. By running python3 onnxrt_ep.py -d.

root@j7-evm:/opt/edgeai-tidl-tools/examples/osrt_python/ort# python3 onnxrt_ep.py -d
Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 1 Models - ['best']


Running_Model :  best  

oot@j7-evm:/opt/edgeai-tidl-tools/examples/osrt_python/ort# python3 onnxrt_ep.py -d
Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 4 Models - ['cl-ort-resnet18-v1', 'cl-ort-caffe_squeezenet_v1_1', 'ss-ort-deeplabv3lite_mobilenetv2', 'od-ort-ssd-lite_mobilenetv2_fpn']


Running_Model :  cl-ort-resnet18-v1  

2022-06-24 06:48:00.375196036 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer4.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.375263804 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer4.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.375909706 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer4.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.375963148 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer2.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.376396804 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer1.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.376982218 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer3.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.377027725 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer1.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.377247281 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer1.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.377464381 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer2.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.377673067 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer1.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.377883242 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer2.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.378252270 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer4.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.378544674 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer4.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.378951874 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer3.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.378998031 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer2.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.379209111 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.379431622 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer2.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.379679714 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer3.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.379716016 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer3.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2022-06-24 06:48:00.379738702 [W:onnxruntime:, graph.cc:3106 CleanUnusedInitializers] Removing initializer 'layer3.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.

 ,  0  25.051779  warplane, military plane ,,  1  22.756052  aircraft carrier, carrier, flattop, attack aircraft carrier ,,  2  19.245899  projectile, missile ,,  3  19.136238  missile ,,  4  16.698288  wing ,

Saving image to  ../../../output_images/

 
Completed_Model :     1, Name : cl-ort-resnet18-v1                                , Total time :     391.39, Offload Time :       0.00 , DDR RW MBs : 0, Output File : py_out_cl-ort-resnet18-v1_airshow.jpg 
 
 

Running_Model :  cl-ort-caffe_squeezenet_v1_1  


 ,  0  0.627038  warplane, military plane ,,  1  0.209702  aircraft carrier, carrier, flattop, attack aircraft carrier ,,  2  0.055806  projectile, missile ,,  3  0.053149  airliner ,,  4  0.045723  missile ,

Saving image to  ../../../output_images/

 
Completed_Model :     2, Name : cl-ort-caffe_squeezenet_v1_1                      , Total time :      97.00, Offload Time :       0.00 , DDR RW MBs : 0, Output File : py_out_cl-ort-caffe_squeezenet_v1_1_airshow.jpg 
 
 

Running_Model :  ss-ort-deeplabv3lite_mobilenetv2  


Saving image to  ../../../output_images/

 
Completed_Model :     3, Name : ss-ort-deeplabv3lite_mobilenetv2                  , Total time :    1888.53, Offload Time :       0.00 , DDR RW MBs : 0, Output File : py_out_ss-ort-deeplabv3lite_mobilenetv2_ADE_val_00001801.jpg 
 
 

Running_Model :  od-ort-ssd-lite_mobilenetv2_fpn  


Saving image to  ../../../output_images/

 
Completed_Model :     4, Name : od-ort-ssd-lite_mobilenetv2_fpn                   , Total time :    1050.49, Offload Time :       0.00 , DDR RW MBs : 0, Output File : py_out_od-ort-ssd-lite_mobilenetv2_fpn_ADE_val_00001801.jpg

Hi BEKER,

As we verified the setup is good to go and this particular model is failing. Mostly this is coz of insufficient memory. Could you run the same command in a PC environment (X86) to verify the model? 

Hi Muhammed,
I had already inferred in an x86 environment. You can check the comment I shared the traffic signs. I believe my problem is with creating folders for model artifacts.

Thank you in advance for your help.

Hi BEKER,

model-artifacts folder will be generated when you run the python with -c option, as you have done above.

From the error it doesn't look like model-artifacts folder misplace issue. We can try a smaller textbook model and do the same procedure, so that we can narrow down the issue. (you can refer to ti model zoo https://github.com/TexasInstruments/edgeai-modelzoo for models)

Hi Muhammed,
If I do the same operations with the pre-trained yolov5s6 model, I get a different error.

Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 1 Models - ['yolov5s6']


Running_Model :  yolov5s6  

Traceback (most recent call last):
  File "onnxrt_ep.py", line 251, in <module>
    run_model(model, mIdx)
  File "onnxrt_ep.py", line 168, in run_model
    sess = rt.InferenceSession(config['model_path'] ,providers=EP_list, provider_options=[delegate_options, {}], sess_options=so)
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 283, in __init__
    self._create_inference_session(providers, provider_options)
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 310, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from /opt/edgeai-tidl-tools/model-artifacts/yolov5s6/model/yolov5s6.onnx failed:/usr/src/debug/onnxruntime/1.7.0-r0_psdkla_4/git/onnxruntime/core/graph/model.cc:101 onnxruntime::Model::Model(onnx::ModelProto&&, const PathString&, const IOnnxRuntimeOpSchemaRegistryList*, const onnxruntime::logging::Logger&) ModelProto does not have a graph.

Hi Muhammed,

Let me explain what we did step-by-step from the very beginning so that we can better understand the problem.

We have the EVM kit, whose images I have shared. We also have one GMSL camera. We have one screen for the output of the EVM kit. We connect to the EVM with a pc using ubuntu 18.04 via ssh method using VS code. Using edge_ai_apps we can take images from the camera and test the demos live.

Step 1

https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-SK-TDA4VM

We wrote the image file we downloaded via the relevant link to the sd card with balena.

Step 2

https://github.com/TexasInstruments/edgeai-yolov5

We train the yolov5 model using our custom data with the relevant repostory. Again, we are doing the onnx model conversion with the same repo. Now we have two files.

1- best.onnx

2- best.prototxt

Step 3

https://github.com/TexasInstruments/edgeai-tidl-tools

We create the artifacts folder of the "best.onnx" and "best.prototxt" files using the edgeai-tidl-tools repository.

sefau18@ubuntu:~/edgeai-tidl-tools/examples/osrt_python/ort$ python3 onnxrt_ep.py -c
Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 1 Models - ['best']


Running_Model :  best  

Writing model with shapes after running onnx shape inference --  ../../../models/public/best_shape.onnx
TIDL Meta PipeLine (Proto) File  : ../../../models/public/best.prototxt  
yolo_v3
yolo_v3
Number of OD backbone nodes = 0 
Size of odBackboneNodeIds = 0 

Preliminary subgraphs created = 0 
Final number of subgraphs created are : 0, - Offloaded Nodes - 0, Total Nodes - 1 
TIDL Meta PipeLine (Proto) File  : ../../../models/public/best.prototxt  
yolo_v3
yolo_v3
Number of OD backbone nodes = 0 
Size of odBackboneNodeIds = 0 

Preliminary subgraphs created = 0 
Final number of subgraphs created are : 0, - Offloaded Nodes - 0, Total Nodes - 1 
TIDL Meta PipeLine (Proto) File  : ../../../models/public/best.prototxt  
yolo_v3
yolo_v3
Number of OD backbone nodes = 328 
Size of odBackboneNodeIds = 328 

Preliminary subgraphs created = 1 
Final number of subgraphs created are : 1, - Offloaded Nodes - 434, Total Nodes - 434 
TIDL Meta PipeLine (Proto) File  : ../../../models/public/best.prototxt  
yolo_v3
yolo_v3
Warning : Requested Output Data Convert Layer is not Added to the network, It is currently not Optimal

 ************** Frame index 1 : Running float import ************* 
INFORMATION: [TIDL_ResizeLayer] Resize_195 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] Resize_219 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] Resize_243 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
WARNING: [TIDL_E_DATAFLOW_INFO_NULL] ti_cnnperfsim.out fails to allocate memory in MSMC. Please look into perfsim log. This model can only be used on PC emulation, it will get fault on target.
****************************************************
**          4 WARNINGS          0 ERRORS          **
****************************************************
 0.0s:  VX_ZONE_INIT:Enabled
 0.24s:  VX_ZONE_ERROR:Enabled
 0.26s:  VX_ZONE_WARNING:Enabled
 0.456s:  VX_ZONE_INIT:[tivxInit:178] Initialization Done !!!

**********  Frame Index 1 : Running float inference **********

**********  Frame Index 2 : Running fixed point mode for calibration **********

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/sefau18/edgeai-tidl-tools/model-artifacts/best/tempDir/tempDir/detections_tidl_io_.qunat_stats_config.txt 
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   50800.94  .... ..... ... .... .....
#    1 . .. T   50743.26  .... ..... ... .... .....
~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/sefau18/edgeai-tidl-tools/model-artifacts/best/tempDir/tempDir/detections_tidl_io_.qunat_stats_config.txt 
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   31451.69  .... ..... ... .... .....
#    1 . .. T   31204.08  .... ..... ... .... .....
 
 
 *****************   Calibration iteration number 0 completed ************************ 
 
 
 

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/sefau18/edgeai-tidl-tools/model-artifacts/best/tempDir/tempDir/detections_tidl_io_.qunat_stats_config.txt 
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   31393.10  .... ..... ... .... .....
#    1 . .. T   31123.50  .... ..... ... .... .....
 
 
 *****************   Calibration iteration number 1 completed ************************ 
 
 
 

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/sefau18/edgeai-tidl-tools/model-artifacts/best/tempDir/tempDir/detections_tidl_io_.qunat_stats_config.txt 
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   31272.20  .... ..... ... .... .....
#    1 . .. T   32056.80  .... ..... ... .... .....
 
 
 *****************   Calibration iteration number 2 completed ************************ 
 
 
 

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/sefau18/edgeai-tidl-tools/model-artifacts/best/tempDir/tempDir/detections_tidl_io_.qunat_stats_config.txt 
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   31252.65  .... ..... ... .... .....
#    1 . .. T   31130.10  .... ..... ... .... .....
 
 
 *****************   Calibration iteration number 3 completed ************************ 
 
 
 

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/sefau18/edgeai-tidl-tools/model-artifacts/best/tempDir/tempDir/detections_tidl_io_.qunat_stats_config.txt 
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   31182.16  .... ..... ... .... .....
#    1 . .. T   31008.60  .... ..... ... .... .....
 
 
 *****************   Calibration iteration number 4 completed ************************ 
 
 
 

------------------ Network Compiler Traces -----------------------------
successful Memory allocation
substitute string tidl_net_ not found
INFORMATION: [TIDL_ResizeLayer] Resize_195 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] Resize_219 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] Resize_243 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
****************************************************
**          3 WARNINGS          0 ERRORS          **
****************************************************

 
Completed_Model :     1, Name : best                                              , Total time :  268678.36, Offload Time :   50674.55 , DDR RW MBs : 0, Output File : py_out_best_ADE_val_00001801.jpg 

Step 4

We can extract inferences to the x86 environment. The model works as expected.

sefau18@ubuntu:~/edgeai-tidl-tools/examples/osrt_python/ort$ python3 onnxrt_ep.py -d
Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 1 Models - ['best']


Running_Model :  best  

Writing model with shapes after running onnx shape inference --  ../../../models/public/best_shape.onnx

Saving image to  ../../../output_images/

 
Completed_Model :     1, Name : best                                              , Total time :     301.03, Offload Time :       0.00 , DDR RW MBs : 0, Output File : py_out_best_ADE_val_00001801.jpg 

Step 5

Our artifacts folder created in step 3 looks like this.

sefau18@ubuntu:~/edgeai-tidl-tools/model-artifacts/best$ tree

sefau18@ubuntu:~/edgeai-tidl-tools/model-artifacts/best$ tree

 .
 ├── tempDir
 │   ├── detections_calib_raw_data.bin
 │   ├── detections_tidl_io_1.bin
 │   ├── detections_tidl_io__LayerPerChannelMean.bin
 │   ├── detections_tidl_io_.perf_sim_config.txt
 │   ├── detections_tidl_io_.qunat_stats_config.txt
 │   ├── detections_tidl_io__stats_tool_out.bin
 │   ├── detections_tidl_net
 │   │   ├── bufinfolog.csv
 │   │   ├── bufinfolog.txt
 │   │   └── perfSimInfo.bin
 │   ├── detections_tidl_net.bin
 │   ├── detections_tidl_net.bin.layer_info.txt
 │   ├── detections_tidl_net.bin_netLog.txt
 │   ├── detections_tidl_net.bin_paramDebug.csv
 │   ├── detections_tidl_net.bin.svg
 │   ├── graphvizInfo.txt
 │   └── runtimes_visualization.svg
 ├── allowedNode.txt
 ├── best.onnx
 ├── detections_tidl_io_1.bin
 ├── detections_tidl_net.bin
 ├── onnxrtMetaData.txt
 └── param.yaml

2 directories, 22 files

Step 6

https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-sk-tda4vm/latest/exports/docs/inference_models.html

We manually create the artifacts folder shown in the link above.

sefau18@ubuntu:~/edgeai-tidl-tools/model-artifacts/best-manual$ tree
.
├── artifacts
│   ├── allowedNode.txt
│   ├── detections_tidl_io_1.bin
│   ├── detections_tidl_net.bin
│   ├── detections_tidl_net.bin_netLog.txt
│   ├── detections_tidl_net.bin.svg
│   ├── onnxrtMetaData.txt
│   └── runtimes_visualization.svg
├── model
│   ├── best.onnx
│   └── best.prototxt
└── param.yaml

2 directories, 10 files

Step 7

We install the edgeai_tidl-tools repository on the EVM. Then we copy the artifacts folder we created manually to the EVM.

root@j7-evm:/opt/edgeai-tidl-tools# export DEVICE=j7
root@j7-evm:/opt/edgeai-tidl-tools# source ./setup.sh
.
.
.
inflating: opencv-4.1.0/samples/wp8/OpenCVXaml/OpenCVXaml/Properties/AppManifest.xml  
inflating: opencv-4.1.0/samples/wp8/OpenCVXaml/OpenCVXaml/Properties/AssemblyInfo.cs  
inflating: opencv-4.1.0/samples/wp8/OpenCVXaml/OpenCVXaml/Properties/WMAppManifest.xml  
 creating: opencv-4.1.0/samples/wp8/OpenCVXaml/OpenCVXaml/Resources/
inflating: opencv-4.1.0/samples/wp8/OpenCVXaml/OpenCVXaml/Resources/AppResources.Designer.cs  
inflating: opencv-4.1.0/samples/wp8/OpenCVXaml/OpenCVXaml/Resources/AppResources.resx  
inflating: opencv-4.1.0/samples/wp8/readme.txt  

Step 8

examples/osrt_python/common_utils.py edit

models_configs = {
    # ONNX RT OOB Models
    'best' : {
        'model_path' : os.path.join(models_base_path, 'best.onnx'),
        'mean': [0, 0, 0],
        'std' : [0.003921568627,0.003921568627,0.003921568627],
        'num_images' : numImages,
        'num_classes': 36,
        'model_type': 'od',
        'od_type' : 'YoloV5',
        'framework' : '',
        'meta_layers_names_list' : os.path.join(models_base_path, 'best.prototxt'),
        'session_name' : 'onnxrt' ,
        'meta_arch_type' : 6
    },

Step 9

examples/osrt_python/ort/onnx_ep.py edit

Step 10

We can make inferences on the EVM. It works fine just like the x86 environment.

root@j7-evm:/opt/edgeai-tidl-tools/examples/osrt_python/ort# python3 onnxrt_ep.py -d
Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 1 Models - ['best']


Running_Model :  best  


Saving image to  ../../../output_images/

 
Completed_Model :     1, Name : best                                              , Total time :   12447.09, Offload Time :       0.00 , DDR RW MBs : 0, Output File : py_out_best_ADE_val_00001801.jpg 
 
 

Step 11

In the EVM, we manually copy the /op/model_zoo folder to the "best" folder we created in step 6.

object_detection.yaml edit

title: "Object Detection Demo"
log_level: 2
inputs:
    input0:
        source: /dev/video2
        format: jpeg
        width: 1280
        height: 720
        framerate: 30
    input1:
        source: /opt/edge_ai_apps/data/videos/video_0000_h264.mp4
        format: h264_sw
        width: 1280
        height: 720
        framerate: 30
        loop: True
    input2:
        source: /opt/edge_ai_apps/data/images/%04d.jpg
        width: 1280
        height: 720
        index: 0
        framerate: 1
        loop: True
    input3:
        source: /dev/video18
        width: 1936
        height: 1100
        format: rggb12
        subdev-id: 2
        framerate: 30
        sen-id: imx390
        ldc: True
models:
    model0:
        model_path: /opt/model_zoo/TVM-OD-5020-yolov3-mobv1-gluon-mxnet-416x416
        viz_threshold: 0.6
    model1:
        model_path: /opt/model_zoo/od-8130_onnxrt_coco_edgeai-yolov5_yolov5l6_640_ti_lite_47p1_65p6_onnx
        viz_threshold: 0.6
    model2:
        model_path: /opt/model_zoo/best
        viz_threshold: 0.6
outputs:
    output0:
        sink: kmssink
        width: 1920
        height: 1080
    output1:
        sink: /opt/edge_ai_apps/data/output/videos/output_video.mkv
        width: 1920
        height: 1080
    output2:
        sink: /opt/edge_ai_apps/data/output/images/output_image_%04d.jpg
        width: 1920
        height: 1080

flows:
    flow0:
        input: input3
        models: [model2]
        outputs: [output0]
        mosaic:
            mosaic0:
                width:  1280
                height: 720
                pos_x:  320
                pos_y:  180

Step 12

We are running the edge_ai_apps instance on the EVM.

You see the error we got.

root@j7-evm:/opt/edge_ai_apps/apps_python# ./app_edgeai.py ../configs/object_detection.yaml 
Traceback (most recent call last):
  File "./app_edgeai.py", line 71, in <module>
    main(sys.argv)
  File "./app_edgeai.py", line 45, in main
    demo = EdgeAIDemo(config)
  File "/opt/edge_ai_apps/apps_python/edge_ai_class.py", line 74, in __init__
    model_obj = config_parser.Model(model_config)
  File "/opt/edge_ai_apps/apps_python/config_parser.py", line 104, in __init__
    self.mean = params['session']['input_mean']
KeyError: 'input_mean'

Step 13

Here is the original param.yaml file.

postprocess:
  data_layout: NCHW
  detection_thr: 0.3
preprocess:
  crop:
  - 640
  - 640
  data_layout: NCHW
  mean:
  - 0
  - 0
  - 0
  resize:
  - 640
  - 640
  scale:
  - 0.003921568627
  - 0.003921568627
  - 0.003921568627
session:
  artifacts_folder: ''
  model_folder: model
  model_path: best.onnx
  session_name: onnxrt
target_device: pc
task_type: detection

Step 14

Then we edit the param.yaml file using other examples.

postprocess:
  data_layout: NCHW
  detection_thr: 0.3
  formatter:
    dst_indices:
    - 4
    - 5
    name: DetectionBoxSL2BoxLS
    src_indices:
    - 5
    - 4
  ignore_index: null
  normalized_detections: false
  resize_with_pad: true
  save_output: false
  shuffle_indices: null
  squeeze_axis: null
preprocess:
  crop:
  - 640
  - 640
  data_layout: NCHW
  mean:
  - 0
  - 0
  - 0
  resize:
  - 640
  - 640
  scale:
  - 0.003921568627
  - 0.003921568627
  - 0.003921568627

  add_flip_image: false
  backend: cv2
  interpolation: null
  pad_color:
  - 114
  - 114
  - 114
  resize_with_pad: true
  reverse_channels: false
session:
  artifacts_folder: 'artifacts'
  dir_tree_depth: 3
  extra_inputs: null
  input_data_layout: NCHW
  input_mean: null
  input_optimization: true
  input_scale: null
  model_folder: model
  model_path: best.onnx
  session_name: onnxrt
target_device: pc
task_type: detection

Step 15

This is the error we get when we try again.

root@j7-evm:/opt/edge_ai_apps/apps_python# ./app_edgeai.py ../configs/object_detection.yaml
libtidl_onnxrt_EP loaded 0xdc00a70 
Final number of subgraphs created are : 1, - Offloaded Nodes - 434, Total Nodes - 434 
2022-07-06 12:51:44.042351690 [E:onnxruntime:, inference_session.cc:1310 operator()] Exception during initialization: /usr/src/debug/onnxruntime/1.7.0-r0_psdkla_4/git/include/onnxruntime/core/graph/graph.h:1299 onnxruntime::Node* onnxruntime::Graph::NodeAtIndexImpl(onnxruntime::NodeIndex) const node_index < nodes_.size() was false. Validating no unexpected access using an invalid node_index. Got:28271 Max:1

Traceback (most recent call last):
  File "./app_edgeai.py", line 71, in <module>
    main(sys.argv)
  File "./app_edgeai.py", line 45, in main
    demo = EdgeAIDemo(config)
  File "/opt/edge_ai_apps/apps_python/edge_ai_class.py", line 74, in __init__
    model_obj = config_parser.Model(model_config)
  File "/opt/edge_ai_apps/apps_python/config_parser.py", line 136, in __init__
    self.run_time = RunTime(self)
  File "/opt/edge_ai_apps/apps_python/run_times.py", line 109, in __init__
    self.interpreter = onnxruntime.InferenceSession(params.model_path,\
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 283, in __init__
    self._create_inference_session(providers, provider_options)
  File "/usr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 315, in _create_inference_session
    sess.initialize_session(providers, provider_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /usr/src/debug/onnxruntime/1.7.0-r0_psdkla_4/git/include/onnxruntime/core/graph/graph.h:1299 onnxruntime::Node* onnxruntime::Graph::NodeAtIndexImpl(onnxruntime::NodeIndex) const node_index < nodes_.size() was false. Validating no unexpected access using an invalid node_index. Got:28271 Max:1

At this point, our search for a solution continues. We doubt these are the right steps. Thank you for your help in advance.

Fadil Muhammed Manu Mathew  Do you have any ideas on the subject?

Let's continue the discussion if the other thread that you started: e2e.ti.com/.../4164033