SK-TDA4VM: edgeai-tidl-tools benchmark problems

T S

Part Number: SK-TDA4VM
Other Parts Discussed in Thread: TDA4VM

I’m running into errors trying to benchmark the edgeai-tidl-tools (https://github.com/TexasInstruments/edgeai-tidl-tools) on my SK-TDA4VM and am looking for help. My board is running the following:

root@tda4vm-sk:/opt/edgeai-gst-apps# set | grep SDK

EDGEAI_SDK_VERSION=09_01_00

It was programmed with the SD card image from: https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-SK-TDA4VM/09.01.00.06

I’ve cloned the edgeai-tidl-tools repo to my Linux development machine and checked out tag 09_01_00_05. I’ve been able to build everything in a Docker container and have successfully run the operations in the “Compile and Validate on X86_PC” section of the repo’s README.

So far, so good.

When I try to follow the “Benchmark on TI SOC” section of the repo, however, I see a huge number of errors appear when I run the “python3 ./scripts/gen_test_report.py” script. See attached for two back-to-back runs. The first test shows some initial tests that look successful, but the majority fail. Running the test again shows that everything fails which makes it appear the test puts the board into a bad state.

Any ideas? Should I change to a different SDK or target image? I noticed today that tag 09_01_00_05 is marked as “pre-release”. Maybe I’m too close to the bleeding edge of development?

I want to at least verify I can run the out of the box models successfully before I start working on deploying my re-trained and custom models.

Thanks.

gen_test_report.zip

over 2 years ago

0 Fabiana Jaimes over 2 years ago

TI__Mastermind 19670 points

Hi Tim,

There shouldn't be an issue with using tag 09_01_00_05 with SDK v09.01.00.06, but I do see that tag 09_01_01_01 has some bug fixes which is also compatible with the SDK version you have. Could you try using this tag and let us know if this clears up the errors?

Thank you,

Fabiana

0 T S over 2 years ago in reply to Fabiana Jaimes

Intellectual 310 points

Fabiana,

There is virtually no difference between 09_01_01_01 and 09_01_00_05. From what I see for the commits for the edgeai-tidl-tools repo, the only differences are in some revision numbering updates. I don't see any bug fix updates.

Have you (or anyone from TI) recently built everything from scratch and verified that deployment to the SK-TDA4VM works? I know of one other issue relating to Docker creation that I ran into, so I get the feeling the repo hasn't been fully proven out from a customer perspective.

Thanks.

0 T S over 2 years ago in reply to T S

Intellectual 310 points

FYI--

I reverted to 09_01_00_02 and rebuilt everything inside the x86 docker. As before, the compile and validate steps ran successfully. Redeploying model-artifacts and models to the TDA4VM-SK and then rebuilding edgeai-tidl-tools still gives me errors. My expectation is that I'm able to "Benchmark on TI SOC" without error.

Here's an error excerpt:

Command : python3 onnxrt_ep.py in Dir : examples/osrt_python/ort Started
libtidl_onnxrt_EP loaded 0x22283720
Final number of subgraphs created are : 1, - Offloaded Nodes - 52, Total Nodes - 52
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
2173.001495 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
2173.001552 s: VX_ZONE_INIT:Enabled
2173.001555 s: VX_ZONE_ERROR:Enabled
2173.001558 s: VX_ZONE_WARNING:Enabled
2173.002406 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
2173.002785 s: VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
2174.054157 s: VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
2174.058526 s: VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 7 alloc's of 15581092 bytes
DDR_SHARED_MEM: Free's : 7 free's of 15581092 bytes
DDR_SHARED_MEM: Open's : 0 allocs of 0 bytes
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
libtidl_onnxrt_EP loaded 0x22283720
Final number of subgraphs created are : 1, - Offloaded Nodes - 68, Total Nodes - 68
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
2174.420128 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
2174.420141 s: VX_ZONE_INIT:Enabled
2174.420143 s: VX_ZONE_ERROR:Enabled
2174.420146 s: VX_ZONE_WARNING:Enabled
2174.420552 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
2174.420576 s: VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
2174.438194 s: VX_ZONE_ERROR:[ownContextSendCmd:822] Command ack message returned failure cmd_status: -1
2174.438201 s: VX_ZONE_ERROR:[ownContextSendCmd:862] tivxEventWait() failed.
2174.438204 s: VX_ZONE_ERROR:[ownNodeKernelInit:584] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
2174.438207 s: VX_ZONE_ERROR:[ownNodeKernelInit:585] Please be sure the target callbacks have been registered for this core
2174.438210 s: VX_ZONE_ERROR:[ownNodeKernelInit:586] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
2174.438214 s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
2174.438225 s: VX_ZONE_ERROR:[vxVerifyGraph:2059] Node kernel init failed
2174.438228 s: VX_ZONE_ERROR:[vxVerifyGraph:2113] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed

0 T S over 2 years ago in reply to T S

Intellectual 310 points

Not sure if it helps, but based on another thread in the forum, I ran "/opt/vision_apps/vx_app_arm_remote_log.out" from one shell on the target and then ran "python3 ./scripts/gen_test_report.py" from another shell. See attached for logs.

A few highlights from vx_app_arm_remote_log.out are below. Note that each of these appears several times in the log output:

[C7x_1 ] 3801.267750 s: VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = 1
[C7x_1 ] 3801.267921 s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:912] tivxAlgiVisionCreate returned NULL

[C7x_1 ] 3801.642299 s: VX_ZONE_ERROR:[tivxAlgiVisionAllocMem:194] Failed to Allocate memory record 12 @ space = 17 and size = 8159232 !!!
[C7x_1 ] 3801.642328 s: VX_ZONE_ERROR:[tivxAlgiVisionCreate:358] tivxAlgiVisionAllocMem Failed
[C7x_1 ] 3801.642352 s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:912] tivxAlgiVisionCreate returned NULL

[C7x_1 ] 3802.340561 s: VX_ZONE_ERROR:[tivxAlgiVisionAllocMem:194] Failed to Allocate memory record 5 @ space = 17 and size = 2059328 !!!
[C7x_1 ] 3802.340588 s: VX_ZONE_ERROR:[tivxAlgiVisionCreate:358] tivxAlgiVisionAllocMem Failed
[C7x_1 ] 3802.340611 s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:912] tivxAlgiVisionCreate returned NULL

[C7x_1 ] 3850.629474 s: VX_ZONE_ERROR:[tivxAlgiVisionAllocMem:194] Failed to Allocate memory record 0 @ space = 17 and size = 15616 !!!
[C7x_1 ] 3850.629504 s: VX_ZONE_ERROR:[tivxAlgiVisionCreate:358] tivxAlgiVisionAllocMem Failed
[C7x_1 ] 3850.629527 s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:912] tivxAlgiVisionCreate returned NULL

edgeai-tidl-tools-problems.zip

0 Fabiana Jaimes over 2 years ago in reply to T S

TI__Mastermind 19670 points

Hi Tim,

Could you please share the executed commands in the logs?

Thank you,

Fabiana

0 T S over 2 years ago in reply to Fabiana Jaimes

Intellectual 310 points

All operations were executed on the TDA4VM-SK. Here they are:

Shell #1:

cd /opt/vision_apps

./vx_app_arm_remote_log.out

Shell #2:

cd /home/projects/edgeai-tidl-tools

export SOC=am68pa

export TIDL_TOOLS_PATH=$(pwd)

python3 ./scripts/gen_test_report.py > /tmp/gen_test_report_run1.txt

0 Fabiana Jaimes over 2 years ago in reply to T S

TI__Mastermind 19670 points

Hi,

Could you share the executed commands for the initial attached logs?

Thank you,

Fabiana

0 T S over 2 years ago in reply to Fabiana Jaimes

Intellectual 310 points

I issued the commands after copying the x86 model-artifacts and models directories to the board, and compiled everything. This is spelled out in the Github repo's README:

# scp -r <pc>/edgeai-tidl-tools/model-artifacts/  <dev board>/edgeai-tidl-tool/
# scp -r <pc>/edgeai-tidl-tools/models/  <dev board>/edgeai-tidl-tool/
mkdir build && cd build
cmake ../examples && make -j && cd ..
python3 ./scripts/gen_test_report.py

The logs in my initial post followed this. I rebooted the board and did the following for those initial logs:

cd /home/projects/edgeai-tidl-tools

export SOC=am68pa

export TIDL_TOOLS_PATH=$(pwd)

python3 ./scripts/gen_test_report.py > /tmp/gen_test_report_run1.txt

python3 ./scripts/gen_test_report.py > /tmp/gen_test_report_run2.txt

One additional note-- inferencing "breaks" as a result of running the gen_test_report.py on my TDA4VM-CK board until I reboot the board. Here's how I've verified it:

1) Boot the board

2) Run the "Custom" demo app with the USB cameras as input for the ONR-OD-8220-yolox-s-lite-mmdet-coco-640x640 model.

3) Observe bounding boxes appear as expected.

4) Terminate the "Custom" demo app.

4) From a shell run the gen_test_report.py script.

5) Observe the errors reported by running the script.

6) Attempt to run the same "Custom" demo app as in step 2) above.

7) Observe video is shown as expected, but bounding boxes no longer appear as expected. Inferencing is "broken".

8) Reboot the board and run the "Custom" demo app again, as above.

9) Observe the bounding boxes now appear as expected.

0 Fabiana Jaimes over 2 years ago in reply to T S

TI__Mastermind 19670 points

Hi Tim,

Thank you for providing more details. I will be trying this on my end today.

Thank you,

Fabiana

0 Fabiana Jaimes over 2 years ago in reply to Fabiana Jaimes

TI__Mastermind 19670 points

Hi Tim,

When running source ./setup.sh, did you see any warnings or errors?

Thank you,

Fabiana

0 T S over 2 years ago in reply to Fabiana Jaimes

Intellectual 310 points

The instructions on Github don't say to run `source ./setup.sh` on the TI SOC. I thought that was only for the X86_PC platform?

Can you clarify the exact steps you follow for benchmarking things on the TDA4VM-SK?

0 Fabiana Jaimes over 2 years ago in reply to T S

TI__Mastermind 19670 points

Hi Tim,

Yes, that is what I am referring to.

Resource is being allocated to look into this and in 2-3 days' time frame needed to get back to you due to bandwidth.

Thank you,

Fabiana

0 T S over 2 years ago in reply to Fabiana Jaimes

Intellectual 310 points

On my dev machine, I'm running things inside of Docker (note: see https://github.com/TexasInstruments/edgeai-tidl-tools/issues/67). When I start the container from scratch and do the following:

cd /home/root/

export SOC=am68pa

source ./setup.sh

...I see no errors. I do see repeated instances of this warning which I assume is benign:

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: pip.pypa.io/.../venv

Just to re-iterate from my original post, I'm having no issues compiling or validating on the PC. It's the "Benchmark on TI SOC" section where errors crop up.

0 Fabiana Jaimes over 2 years ago in reply to T S

TI__Mastermind 19670 points

Hi Tim,

Thank you for the clarification and additional information. I have wrapped up my other work and will get back to you tomorrow with my findings on this issue.

Thank you for your patience.

- Fabiana

0 Fabiana Jaimes over 2 years ago in reply to Fabiana Jaimes

TI__Mastermind 19670 points

Hi Tim,

Please expect a response after this weekend. I apologize for the delay.

Thank you,

Fabiana

0 Pratik Kedar over 2 years ago in reply to Fabiana Jaimes

TI__Mastermind 24041 points

Hi Tim,

There are 2 prime issues with the current setup of edgeai benchmark that i was able to point out.

1. Infinite halt occurred while building the example on target core : TDA4VM

RCA : Am suspecting this is could be because of -j is not picking correct number of cores which ideally it should have.

Fix/ Way Around : Since TDA4VM has dual core arm, you can leverage 2 cores for parallel build, by explicitly specifying as -j2 Or can simply avoid setting parallel core (removing -j) I was able to build the example with below cmd :

cmake ../examples && make -j2 && cd .. 
Or
cmake ../examples && make && cd ..

2. Issue while running the gen report script.

RCA : Model cl-ort-caffe_squeezenet_v1_1 is failing to infer on target, since this was in model pipeline, once this got halted abruptly it leaves the c7x core in halt state, other model in model pipeline fails to execute because of this.

Fix/ Way Around :

As of now there are 2 options i would like to suggest:

1. Completely remove the above mentioned model and run the gen report script, your model pipeline should look like :

models = ['cl-ort-resnet18-v1', 'ss-ort-deeplabv3lite_mobilenetv2', 'od-ort-ssd-lite_mobilenetv2_fpn']

You can add the above change in onnxrt_ep.py file at target before running the gen report script.

2. After doing RCA for above model inference halt I found that the issue is in layer number 40 SoftMax operator is not supported along axis 3, but some how this operator is delegated for C7x core instead of ARM core for unsupported layer inference.

Fix/ Way Around : For caffe_squeezenet_v1_1 model you can explicitly add compiler flag to offload this operator to ARM by putting this in Deny List, and recompile the model artifacts on host machine.

optional_options = {
# "priority":0,
#delay in ms
# "max_pre_empt_delay":10
"platform":"J7",
"version":"7.2",
"tensor_bits":tensor_bits,
"debug_level":debug_level,
"max_num_subgraphs":max_num_subgraphs,
"deny_list": "",
"deny_list:layer_type": "Softmax", 
"deny_list:layer_name":"",
"model_type":"",#OD
"accuracy_level":accuracy_level,
"advanced_options:calibration_frames": calibration_frames,
"advanced_options:calibration_iterations": calibration_iterations,
"advanced_options:output_feature_16bit_names_list" : output_feature_16bit_names_list,
"advanced_options:params_16bit_names_list" : params_16bit_names_list,
"advanced_options:mixed_precision_factor" :  mixed_precision_factor,
"advanced_options:quantization_scale_type": quantization_scale_type,
#"object_detection:meta_layers_names_list" : meta_layers_names_list,  -- read from models_configs dictionary below
#"object_detection:meta_arch_type" : meta_arch_type,                  -- read from models_configs dictionary below
"advanced_options:high_resolution_optimization": high_resolution_optimization,
"advanced_options:pre_batchnorm_fold" : pre_batchnorm_fold,
"ti_internal_nc_flag" : ti_internal_nc_flag,
# below options will be read only if accuracy_level = 9, else will be discarded.... for accuracy_level = 0/1, these are preset internally
"advanced_options:activation_clipping" : activation_clipping,
"advanced_options:weight_clipping" : weight_clipping,
"advanced_options:bias_calibration" : bias_calibration,
"advanced_options:add_data_convert_ops" : data_convert,
"advanced_options:channel_wise_quantization" : channel_wise_quantization,
# Advanced options for SOC 'am69a' 
"advanced_options:inference_mode" : inference_mode,
"advanced_options:num_cores" : num_cores
}

You can change the common_utils.py file on you host machine to add above flag setting, post this run the model compilation to generate model artifacts and remove the model artifacts for squeeze net on target and re upload the new one that we are have generated for this, post this you can run the gen report script.

Note : Please Update the model artifacts for squeezenet excluding others.

Hope this unblocks you in your current work, meanwhile i will file JIRA to track this change request.

0 T S over 2 years ago in reply to Pratik Kedar

Intellectual 310 points

I modified onnxrt_ep.py as described above. I'm still seeing failures with other tests the gen_test_report.py script executes. The script ends up hanging while it attempts to run "od-ort-ssd-lite_mobilenetv2_fpn".

If I reboot the TDA4VM, configure the environment, and then explicitly execute `./bin/Release/ort_main -f model-artifacts/od-ort-ssd-lite_mobilenetv2_fpn/ -v 1 -i test_data/ADE_val_00001801.jpg -c 20`, it passes, so something further upstream in the gen_test_report.py script is causing issues.

See attached for the latest run output.

Fullscreen gen_test_report_output.txt Download

Command : python3 tflrt_delegate.py in Dir : examples/osrt_python/tfl Started

 Number of subgraphs:1 , 34 nodes delegated out of 34 nodes 
 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3291.623065 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3291.623156 s:  VX_ZONE_INIT:Enabled
  3291.623159 s:  VX_ZONE_ERROR:Enabled
  3291.623161 s:  VX_ZONE_WARNING:Enabled
  3291.626036 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  3291.631264 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
  3292.655146 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  3292.659497 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 7 alloc's of 7607170 bytes 
DDR_SHARED_MEM: Free's : 7 free's  of 7607170 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

 Number of subgraphs:1 , 81 nodes delegated out of 81 nodes 
 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3292.972593 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3292.972609 s:  VX_ZONE_INIT:Enabled
  3292.972611 s:  VX_ZONE_ERROR:Enabled
  3292.972614 s:  VX_ZONE_WARNING:Enabled
  3292.972981 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  3292.973005 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
  3311.052032 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  3311.056373 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 7 alloc's of 7145076 bytes 
DDR_SHARED_MEM: Free's : 7 free's  of 7145076 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

 Number of subgraphs:1 , 107 nodes delegated out of 107 nodes 
 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3311.732352 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3311.732365 s:  VX_ZONE_INIT:Enabled
  3311.732368 s:  VX_ZONE_ERROR:Enabled
  3311.732370 s:  VX_ZONE_WARNING:Enabled
  3311.732772 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  3311.732798 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
  3322.899088 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  3322.903436 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 10 alloc's of 20444424 bytes 
DDR_SHARED_MEM: Free's : 10 free's  of 20444424 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

 Number of subgraphs:1 , 129 nodes delegated out of 129 nodes 
 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3323.390885 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3323.390899 s:  VX_ZONE_INIT:Enabled
  3323.390901 s:  VX_ZONE_ERROR:Enabled
  3323.390904 s:  VX_ZONE_WARNING:Enabled
  3323.391283 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  3323.391308 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
  3334.558246 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115]Running 4 Models - ['cl-tfl-mobilenet_v1_1.0_224', 'ss-tfl-deeplabv3_mnv2_ade20k_float', 'od-tfl-ssd_mobilenet_v2_300_float', 'od-tfl-ssdlite_mobiledet_dsp_320x320_coco']


Running_Model :  cl-tfl-mobilenet_v1_1.0_224

 ,  0  0.691726  warplane, military plane ,,  1  0.181373  missile ,,  2  0.109571  projectile, missile ,,  3  0.006352  cannon ,,  4  0.005370  aircraft carrier, carrier, flattop, attack aircraft carrier ,

Saving image to  ../../../output_images/

 
Completed_Model :     1, Name : cl-tfl-mobilenet_v1_1.0_224                       , Total time :       1.72, Offload Time :       1.70 , DDR RW MBs : 18446744073709.52, Output File : py_out_cl-tfl-mobilenet_v1_1.0_224_airshow.jpg
 
 

Running_Model :  ss-tfl-deeplabv3_mnv2_ade20k_float

Saving image to  ../../../output_images/

 
Completed_Model :     2, Name : ss-tfl-deeplabv3_mnv2_ade20k_float                , Total time :      10.55, Offload Time :      10.54 , DDR RW MBs : 18446744073709.52, Output File : py_out_ss-tfl-deeplabv3_mnv2_ade20k_float_ADE_val_00001801.jpg
 
 

Running_Model :  od-tfl-ssd_mobilenet_v2_300_float

Saving image to  ../../../output_images/

 
Completed_Model :     3, Name : od-tfl-ssd_mobilenet_v2_300_float                 , Total time :       6.35, Offload Time :       6.33 , DDR RW MBs : 18446744073709.52, Output File : py_out_od-tfl-ssd_mobilenet_v2_300_float_ADE_val_00001801.jpg
 
 

Running_Model :  od-tfl-ssdlite_mobiledet_dsp_320x320_coco

Saving image to  ../../../output_images/

 
Completed_Model :     4, Name : od-tfl-ssdlite_mobiledet_dsp_320x320_coco         , Total time :       5.61, Offload Time :       5.59 , DDR RW MBs : 18446744073709.52, Output File : py_out_od-tfl-ssdlite_mobiledet_dsp_320x320_coco_ADE_val_00001801.jpg
 
 
 De-Initialization Done for HOST !!!
  3334.559806 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 10 alloc's of 10800264 bytes 
DDR_SHARED_MEM: Free's : 10 free's  of 10800264 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
Command : python3 onnxrt_ep.py in Dir : examples/osrt_python/ort Started
Available execution providers :  ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Running 3 Models - ['cl-ort-resnet18-v1', 'ss-ort-deeplabv3lite_mobilenetv2', 'od-ort-ssd-lite_mobilenetv2_fpn']


Running_Model :  cl-ort-resnet18-v1  


 ,  0  23.847309  warplane, military plane ,,  1  22.507572  aircraft carrier, carrier, flattop, attack aircraft carrier ,,  2  19.024258  projectile, missile ,,  3  18.756310  missile ,,  4  15.808890  airliner ,

Saving image to  ../../../output_images/

 
Completed_Model :     1, Name : cl-ort-resnet18-v1                                , Total time :       2.62, Offload Time :       2.51 , DDR RW MBs : 0, Output File : py_out_cl-ort-resnet18-v1_airshow.jpg 
 
 

Running_Model :  ss-ort-deeplabv3lite_mobilenetv2  


Saving image to  ../../../output_images/

 
Completed_Model :     2, Name : ss-ort-deeplabv3lite_mobilenetv2                  , Total time :       9.39, Offload Time :       9.22 , DDR RW MBs : 0, Output File : py_out_ss-ort-deeplabv3lite_mobilenetv2_ADE_val_00001801.jpg 
 
 

Running_Model :  od-ort-ssd-lite_mobilenetv2_fpn  

2

Saving image to  ../../../output_images/

 
Completed_Model :     3, Name : od-ort-ssd-lite_mobilenetv2_fpn                   , Total time :       6.34, Offload Time :       6.10 , DDR RW MBs : 0, Output File : py_out_od-ort-ssd-lite_mobilenetv2_fpn_ADE_val_00001801.jpg 
 
 
libtidl_onnxrt_EP loaded 0x17955000 
Final number of subgraphs created are : 1, - Offloaded Nodes - 52, Total Nodes - 52 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3337.565660 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3337.565711 s:  VX_ZONE_INIT:Enabled
  3337.565714 s:  VX_ZONE_ERROR:Enabled
  3337.565716 s:  VX_ZONE_WARNING:Enabled
  3337.566789 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  3337.566926 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
  3338.613612 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  3338.617997 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 7 alloc's of 15581092 bytes 
DDR_SHARED_MEM: Free's : 7 free's  of 15581092 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
libtidl_onnxrt_EP loaded 0x17955000 
Final number of subgraphs created are : 1, - Offloaded Nodes - 124, Total Nodes - 124 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3339.174326 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3339.174338 s:  VX_ZONE_INIT:Enabled
  3339.174341 s:  VX_ZONE_ERROR:Enabled
  3339.174343 s:  VX_ZONE_WARNING:Enabled
  3339.174927 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  3339.174952 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
  3355.570767 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  3355.575163 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 7 alloc's of 7296436 bytes 
DDR_SHARED_MEM: Free's : 7 free's  of 7296436 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
libtidl_onnxrt_EP loaded 0x17955000 
Final number of subgraphs created are : 1, - Offloaded Nodes - 482, Total Nodes - 482 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3356.537065 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3356.537077 s:  VX_ZONE_INIT:Enabled
  3356.537080 s:  VX_ZONE_ERROR:Enabled
  3356.537082 s:  VX_ZONE_WARNING:Enabled
  3356.537509 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  3356.537538 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
  3369.445315 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  3369.449703 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 8 alloc's of 7256996 bytes 
DDR_SHARED_MEM: Free's : 8 free's  of 7256996 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
Command : python3 dlr_inference_example.py in Dir : examples/osrt_python/tvm_dlr Started
2024-02-19 14:13:45,145 INFO Could not find libdlr.so in model artifact. Using dlr from /usr/dlr/libdlr.so
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3376.394962 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3376.395017 s:  VX_ZONE_INIT:Enabled
  3376.395020 s:  VX_ZONE_ERROR:Enabled
  3376.395023 s:  VX_ZONE_WARNING:Enabled
  3376.395726 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  3376.396257 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
  3376.618019 s:  VX_ZONE_ERROR:[ownContextSendCmd:822] Command ack message returned failure cmd_status: -1
  3376.618027 s:  VX_ZONE_ERROR:[ownContextSendCmd:862] tivxEventWait() failed.
  3376.618031 s:  VX_ZONE_ERROR:[ownNodeKernelInit:584] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  3376.618033 s:  VX_ZONE_ERROR:[ownNodeKernelInit:585] Please be sure the target callbacks have been registered for this core
  3376.618036 s:  VX_ZONE_ERROR:[ownNodeKernelInit:586] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  3376.618040 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
  3376.618055 s:  VX_ZONE_ERROR:[vxVerifyGraph:2059] Node kernel init failed
  3376.618058 s:  VX_ZONE_ERROR:[vxVerifyGraph:2113] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
  3376.637550 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  3376.641913 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 7 alloc's of 28308270 bytes 
DDR_SHARED_MEM: Free's : 7 free's  of 28308270 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
[14:13:51] /home/kumar/work/tvm_dlr_91/neo-ai-dlr/src/dlr.cc:343: Error: [14:13:51] /home/kumar/work/tvm_dlr_91/neo-ai-dlr/3rdparty/tvm/src/runtime/contrib/tidl/tidl_runtime.cc:260: Failed to initialize TIDLRT for subgraph 0

Stack trace:
  [bt] (0) /usr/dlr/libdlr.so(+0x13d844) [0xffff8534d844]
  [bt] (1) /usr/dlr/libdlr.so(tvm::runtime::detail::LogFatal::Entry::Finalize()+0x6c) [0xffff8524e8fc]
  [bt] (2) /usr/dlr/libdlr.so(tvm::runtime::TIDLJ7Module::GetFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)+0x7e4) [0xffff8531794c]
  [bt] (3) /usr/dlr/libdlr.so(+0x140084) [0xffff85350084]
  [bt] (4) /usr/dlr/libdlr.so(tvm::runtime::ConstLoaderModuleNode::GetFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)+0x328) [0xffff85265750]
  [bt] (5) /usr/dlr/libdlr.so(+0x140084) [0xffff85350084]
  [bt] (6) /usr/dlr/libdlr.so(+0x1a5f00) [0xffff853b5f00]
  [bt] (7) /usr/dlr/libdlr.so(+0x1a9aa0) [0xffff853b9aa0]
  [bt] (8) /usr/dlr/libdlr.so(+0x1ab27c) [0xffff853bb27c]


2024-02-19 14:13:51,597 ERROR error in DLRModel instantiation 
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/dlr/api.py", line 89, in __init__
    self._impl = DLRModelImpl(model_path, dev_type, dev_id, error_log_file, use_default_dlr)
  File "/usr/lib/python3.10/site-packages/dlr/dlr_model.py", line 79, in __init__
    self._check_call(self._lib.CreateDLRModel(byref(self.handle),
  File "/usr/lib/python3.10/site-packages/dlr/dlr_model.py", line 160, in _check_call
    raise DLRError(self._lib.DLRGetLastError().decode('ascii'))
dlr.dlr_model.DLRError


Running Inference on Model -  ../../../model-artifacts/cl-dlr-tflite_inceptionnetv3_device

Traceback (most recent call last):
  File "/home/projects/edgeai-tidl-tools/examples/osrt_python/tvm_dlr/dlr_inference_example.py", line 199, in <module>
    model_create_and_run(model_output_directory, 'input',
  File "/home/projects/edgeai-tidl-tools/examples/osrt_python/tvm_dlr/dlr_inference_example.py", line 164, in model_create_and_run
    model = DLRModel(model_dir, 'cpu')
  File "/usr/lib/python3.10/site-packages/dlr/api.py", line 92, in __init__
    raise ex
  File "/usr/lib/python3.10/site-packages/dlr/api.py", line 89, in __init__
    self._impl = DLRModelImpl(model_path, dev_type, dev_id, error_log_file, use_default_dlr)
  File "/usr/lib/python3.10/site-packages/dlr/dlr_model.py", line 79, in __init__
    self._check_call(self._lib.CreateDLRModel(byref(self.handle),
  File "/usr/lib/python3.10/site-packages/dlr/dlr_model.py", line 160, in _check_call
    raise DLRError(self._lib.DLRGetLastError().decode('ascii'))
dlr.dlr_model.DLRError
Command : bash run_tfl_models.sh in Dir : scripts/osrt_cpp_scripts/ Started
cl-tfl-mobilenet_v1_1.0_224

***** Display run Config: start *****
verbose level set to: 1
accelerated mode set to: 1
device mem set to: 1
loop count set to: 20
model path set to: 
model artifacts path set to: model-artifacts/cl-tfl-mobilenet_v1_1.0_224/
image path set to: test_data/airshow.jpg
device_type set to: cpu
labels path set to: test_data/labels.txt
num of threads set to: 4
num of results set to: 5
num of warmup runs set to: 2

***** Display run Config: end *****
[14:13:51.000.000000]:INFO:[runInference:0199] Loaded model model-artifacts/cl-tfl-mobilenet_v1_1.0_224//mobilenet_v1_1.0_224.tflite 
[14:13:51.000.000029]:INFO:[runInference:0201] resolved reporter
[14:13:51.089.089590]:INFO:[runInference:0213] tensors size: 93 
[14:13:51.089.089621]:INFO:[runInference:0214] nodes size: 34
[14:13:51.089.089629]:INFO:[runInference:0215] number of inputs: 1
[14:13:51.089.089636]:INFO:[runInference:0216] number of outputs: 1
[14:13:51.089.089641]:INFO:[runInference:0217] input(0) name: input
[14:13:51.089.089649]:INFO:[runInference:0246] input: 87
[14:13:51.089.089655]:INFO:[runInference:0252] TIDL delegate mode

 Number of subgraphs:1 , 34 nodes delegated out of 34 nodes 
 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3377.130821 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3377.130897 s:  VX_ZONE_INIT:Enabled
  3377.130900 s:  VX_ZONE_ERROR:Enabled
  3377.130902 s:  VX_ZONE_WARNING:Enabled
  3377.131672 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  3377.131832 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
[14:13:52.255.255482]:INFO:[runInference:0262] ModifyGraphWithDelegate - Done 
[14:13:52.255.255519]:INFO:[runInference:0309] device mem enabled
[14:13:52.283.283607]:INFO:[preprocImage:0115] template NHWC
[14:13:52.284.284024]:INFO:[runInference:0382] interpreter->Invoke - Started 
[14:13:52.320.320843]:INFO:[runInference:0402] interpreter->Invoke - Done 
[14:13:52.320.320872]:INFO:[runInference:0404] average time:1.621300 ms
[14:13:52.320.320881]:INFO:[prepClassificationResult:0127] preparing clasification result 
[14:13:52.332.332476]:INFO:[prepClassificationResult:0169] 0.705051: 896 :warplane, military plane
[14:13:52.332.332501]:INFO:[prepClassificationResult:0169] 0.184867: 658 :missile
[14:13:52.332.332510]:INFO:[prepClassificationResult:0169] 0.094558: 745 :projectile, missile
[14:13:52.332.332520]:INFO:[prepClassificationResult:0169] 0.006475: 472 :cannon
[14:13:52.332.332527]:INFO:[prepClassificationResult:0169] 0.003915: 404 :aircraft carrier, carrier, flattop, attack aircraft carrier
[14:13:52.336.336160]:INFO:[runInference:0491] saving image result file 
[14:13:52.342.342662]:INFO:[runInference:0541] 
Completed_Model : 0, Name : cl-tfl-mobilenet_v1_1.0_224, Total time : 1.621300, Offload Time : 0 , DDR RW MBs : 0, Output File : cpp_out_cl-tfl-mobilenet_v1_1.0_224.jpg 
 
  3377.283757 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  3377.288171 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 9 alloc's of 7761830 bytes 
DDR_SHARED_MEM: Free's : 9 free's  of 7761830 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
od-tfl-ssd_mobilenet_v2_300_float

***** Display run Config: start *****
verbose level set to: 1
accelerated mode set to: 1
device mem set to: 1
loop count set to: 20
model path set to: 
model artifacts path set to: model-artifacts/od-tfl-ssd_mobilenet_v2_300_float/
image path set to: test_data/ADE_val_00001801.jpg
device_type set to: cpu
labels path set to: test_data/labels.txt
num of threads set to: 4
num of results set to: 5
num of warmup runs set to: 2

***** Display run Config: end *****
[14:13:52.000.000000]:INFO:[runInference:0199] Loaded model model-artifacts/od-tfl-ssd_mobilenet_v2_300_float//ssd_mobilenet_v2_300_float.tflite 
[14:13:52.000.000030]:INFO:[runInference:0201] resolved reporter
[14:13:52.136.136311]:INFO:[runInference:0213] tensors size: 272 
[14:13:52.136.136347]:INFO:[runInference:0214] nodes size: 107
[14:13:52.136.136356]:INFO:[runInference:0215] number of inputs: 1
[14:13:52.136.136363]:INFO:[runInference:0216] number of outputs: 4
[14:13:52.136.136369]:INFO:[runInference:0217] input(0) name: normalized_input_image_tensor
[14:13:52.136.136377]:INFO:[runInference:0246] input: 0
[14:13:52.136.136383]:INFO:[runInference:0252] TIDL delegate mode

 Number of subgraphs:1 , 107 nodes delegated out of 107 nodes 
 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3377.591268 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3377.591347 s:  VX_ZONE_INIT:Enabled
  3377.591350 s:  VX_ZONE_ERROR:Enabled
  3377.591352 s:  VX_ZONE_WARNING:Enabled
  3377.592128 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  3377.592878 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
[14:13:52.343.343500]:INFO:[runInference:0262] ModifyGraphWithDelegate - Done 
[14:13:52.343.343544]:INFO:[runInference:0309] device mem enabled
[14:13:52.434.434367]:INFO:[preprocImage:0115] template NHWC
[14:13:52.435.435108]:INFO:[runInference:0382] interpreter->Invoke - Started 
[14:13:52.569.569312]:INFO:[runInference:0402] interpreter->Invoke - Done 
[14:13:52.569.569345]:INFO:[runInference:0404] average time:6.055250 ms
[14:13:52.569.569355]:INFO:[runInference:0428] detected objects:2 
[14:13:52.569.569393]:INFO:[prepDetectionResult:0448] preparing detection result 
[14:13:52.569.569410]:INFO:[prepDetectionResult:0513] box with score:0.684622, threshold set to:0.300000
[14:13:52.569.569419]:INFO:[prepDetectionResult:0513] box with score:0.499985, threshold set to:0.300000
[14:13:52.569.569488]:INFO:[runInference:0491] saving image result file 
[14:13:52.575.575973]:INFO:[runInference:0541] 
Completed_Model : 0, Name : od-tfl-ssd_mobilenet_v2_300_float, Total time : 6.055250, Offload Time : 0 , DDR RW MBs : 0, Output File : cpp_out_od-tfl-ssd_mobilenet_v2_300_float.jpg 
 
  3377.917813 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  3377.922239 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 15 alloc's of 20714988 bytes 
DDR_SHARED_MEM: Free's : 15 free's  of 20714988 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
ss-tfl-deeplabv3_mnv2_ade20k_float

***** Display run Config: start *****
verbose level set to: 1
accelerated mode set to: 1
device mem set to: 1
loop count set to: 20
model path set to: 
model artifacts path set to: model-artifacts/ss-tfl-deeplabv3_mnv2_ade20k_float/
image path set to: test_data/ADE_val_00001801.jpg
device_type set to: cpu
labels path set to: test_data/labels.txt
num of threads set to: 4
num of results set to: 5
num of warmup runs set to: 2

***** Display run Config: end *****
[14:13:52.000.000000]:INFO:[runInference:0199] Loaded model model-artifacts/ss-tfl-deeplabv3_mnv2_ade20k_float//deeplabv3_mnv2_ade20k_float.tflite 
[14:13:52.000.000031]:INFO:[runInference:0201] resolved reporter
[14:13:53.088.088515]:INFO:[runInference:0213] tensors size: 208 
[14:13:53.088.088546]:INFO:[runInference:0214] nodes size: 81
[14:13:53.088.088553]:INFO:[runInference:0215] number of inputs: 1
[14:13:53.088.088560]:INFO:[runInference:0216] number of outputs: 1
[14:13:53.088.088566]:INFO:[runInference:0217] input(0) name: MobilenetV2/MobilenetV2/input
[14:13:53.088.088574]:INFO:[runInference:0246] input: 0
[14:13:53.088.088581]:INFO:[runInference:0252] TIDL delegate mode

 Number of subgraphs:1 , 81 nodes delegated out of 81 nodes 
 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3378.163452 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3378.163528 s:  VX_ZONE_INIT:Enabled
  3378.163531 s:  VX_ZONE_ERROR:Enabled
  3378.163534 s:  VX_ZONE_WARNING:Enabled
  3378.164908 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  3378.164986 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
[14:13:53.237.237969]:INFO:[runInference:0262] ModifyGraphWithDelegate - Done 
[14:13:53.238.238005]:INFO:[runInference:0309] device mem enabled
[14:13:53.304.304965]:INFO:[preprocImage:0115] template NHWC
[14:13:53.307.307059]:INFO:[runInference:0382] interpreter->Invoke - Started 
[14:13:53.534.534572]:INFO:[runInference:0402] interpreter->Invoke - Done 
[14:13:53.534.534611]:INFO:[runInference:0404] average time:10.287300 ms
[14:13:53.534.534621]:INFO:[prepSegResult:0086] preparing segmentation result 
[14:13:53.537.537399]:INFO:[runInference:0491] saving image result file 
[14:13:53.543.543698]:INFO:[runInference:0541] 
Completed_Model : 0, Name : ss-tfl-deeplabv3_mnv2_ade20k_float, Total time : 10.287300, Offload Time : 0 , DDR RW MBs : 0, Output File : cpp_out_ss-tfl-deeplabv3_mnv2_ade20k_float.jpg 
 
  3378.515964 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  3378.519378 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 9 alloc's of 8193780 bytes 
DDR_SHARED_MEM: Free's : 9 free's  of 8193780 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
/home/projects/edgeai-tidl-tools/scripts/osrt_cpp_scripts
Command : bash run_onnx_models.sh in Dir : scripts/osrt_cpp_scripts/ Started
cl-ort-resnet18-v1

***** Display run Config: start *****
verbose level set to: 1
accelerated mode set to: 1
device mem set to: 1
loop count set to: 20
model path set to: 
model artifacts path set to: model-artifacts/cl-ort-resnet18-v1/
image path set to: test_data/airshow.jpg
device_type set to: cpu
labels path set to: test_data/labels.txt
num of threads set to: 4
num of results set to: 5
num of warmup runs set to: 2

***** Display run Config: end *****
[14:13:53.000.000000]:INFO:[runInference:0316] accelerated mode
[14:13:53.002.002385]:INFO:[runInference:0319] artifacts: model-artifacts/cl-ort-resnet18-v1/
libtidl_onnxrt_EP loaded 0x23e28120 
Final number of subgraphs created are : 1, - Offloaded Nodes - 52, Total Nodes - 52 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3380.362452 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3380.362532 s:  VX_ZONE_INIT:Enabled
  3380.362536 s:  VX_ZONE_ERROR:Enabled
  3380.362538 s:  VX_ZONE_WARNING:Enabled
  3380.363328 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  3380.364060 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
[14:13:55.1738.738320]:INFO:[runInference:0342] Loaded model model-artifacts/cl-ort-resnet18-v1//resnet18_opset9.onnx
[14:13:55.1738.738394]:INFO:[printTensorInfo:0216] number of inputs:1 
[14:13:55.1738.738402]:INFO:[printTensorInfo:0217] number of outputs: 1
[14:13:55.1738.738409]:INFO:[printTensorInfo:0218] input(0) name: input.1Net_IN
[14:13:55.1738.738414]:INFO:[printTensorInfo:0223] Input 0 : name=input.1Net_IN
[14:13:55.1738.738423]:INFO:[printTensorInfo:0230] Input 0 : type=2
[14:13:55.1738.738430]:INFO:[printTensorInfo:0233] Input 0 : num_dims=4
[14:13:55.1738.738437]:INFO:[printTensorInfo:0236] Input 0 : dim 0=1
[14:13:55.1738.738443]:INFO:[printTensorInfo:0236] Input 0 : dim 1=3
[14:13:55.1738.738448]:INFO:[printTensorInfo:0236] Input 0 : dim 2=224
[14:13:55.1738.738455]:INFO:[printTensorInfo:0236] Input 0 : dim 3=224
[14:13:55.1744.744697]:INFO:[preprocImage:0127] template NCHW
[14:13:55.1797.797631]:INFO:[runInference:0474] invoked
[14:13:55.1797.797663]:INFO:[runInference:0475] average time: 2.445450 ms 
[14:13:55.1797.797671]:INFO:[prepClassificationResult:0093] preparing classification result 
[14:13:55.1801.801873]:INFO:[prepClassificationResult:0137] 22.507572: 895 warplane, military plane
[14:13:55.1801.801897]:INFO:[prepClassificationResult:0137] 21.435783: 403 aircraft carrier, carrier, flattop, attack aircraft carrier
[14:13:55.1801.801906]:INFO:[prepClassificationResult:0137] 16.880678: 744 projectile, missile
[14:13:55.1801.801918]:INFO:[prepClassificationResult:0137] 16.880678: 404 airliner
[14:13:55.1801.801925]:INFO:[prepClassificationResult:0137] 16.612732: 657 missile
[14:13:55.1808.808135]:INFO:[runInference:0607] 
Completed_Model : 0, Name : cl-ort-resnet18-v1, Total time : 2.445450, Offload Time : 0 , DDR RW MBs : 0, Output File : cpp_out_cl-ort-resnet18-v1.jpg 
 
  3380.501550 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
  3380.506040 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 10 alloc's of 15739812 bytes 
DDR_SHARED_MEM: Free's : 10 free's  of 15739812 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
od-ort-ssd-lite_mobilenetv2_fpn

0 Pratik Kedar over 2 years ago in reply to T S

TI__Mastermind 24041 points

Hi Tim,

After carefully looking into logs it seems that ort and tfl model inference completed correctly, thanks to your changes in onnxrt_ep.py script.

As i see, the issue is specific to tvm_dlr runtime side, where both the models are failing to get inferred on target, and I can confirm the same observation at my end, log ref : line 294 Command : python3 dlr_inference_example.py in Dir : examples/osrt_python/tvm_dlr Started

I will file a bug on this (dlr runtime issues) and check with dev team internally, mean while you can benchmark the ONNX and TFL examples on target.

In order to do this, you can remove the dlr inference execution calls from gen_test_report.py.

You can comment the dlr specific calls from elif SOC == "am68pa" : as described below.

global test_configs
if SOC == "am62":
    device = 'am62'
    test_configs = [
                    {'script_name':'tflrt_delegate.py', 'script_dir':'tfl','lang':'py', 'rt_type':'tfl-py'},
                    {'script_name':'onnxrt_ep.py', 'script_dir':'ort','lang':'py', 'rt_type':'ort-py'},
                    #{'script_name':'dlr_inference_example.py', 'script_dir':'tvm_dlr','lang':'py', 'rt_type':
                    {'script_name':'run_tfl_models.sh', 'script_dir':'osrt_cpp_scripts/','lang':'bash','rt_typ
                    {'script_name':'run_onnx_models.sh', 'script_dir':'osrt_cpp_scripts/','lang':'bash','rt_ty
                    #{'script_name':'run_dlr_models.sh', 'script_dir':'osrt_cpp_scripts/','lang':'bash','rt_ty
        ]
elif SOC == "am68pa" :
    device = 'am68pa'
    test_configs = [
                    {'script_name':'tflrt_delegate.py', 'script_dir':'tfl','lang':'py', 'rt_type':'tfl-py'},
                    {'script_name':'onnxrt_ep.py', 'script_dir':'ort','lang':'py', 'rt_type':'ort-py'},
                    #{'script_name':'dlr_inference_example.py', 'script_dir':'tvm_dlr','lang':'py', 'rt_type':
                    {'script_name':'run_tfl_models.sh', 'script_dir':'osrt_cpp_scripts/','lang':'bash','rt_typ
                    {'script_name':'run_onnx_models.sh', 'script_dir':'osrt_cpp_scripts/','lang':'bash','rt_ty
                    #{'script_name':'run_dlr_models.sh', 'script_dir':'osrt_cpp_scripts/','lang':'bash','rt_ty
        ]
elif SOC == "am68a" :
    device = 'am68a'
    test_configs = [
                    {'script_name':'tflrt_delegate.py', 'script_dir':'tfl','lang':'py', 'rt_type':'tfl-py'},
                    {'script_name':'onnxrt_ep.py', 'script_dir':'ort','lang':'py', 'rt_type':'ort-py'},
                    {'script_name':'dlr_inference_example.py', 'script_dir':'tvm_dlr','lang':'py', 'rt_type':'
                    {'script_name':'run_tfl_models.sh', 'script_dir':'osrt_cpp_scripts/','lang':'bash','rt_typ
                    {'script_name':'run_onnx_models.sh', 'script_dir':'osrt_cpp_scripts/','lang':'bash','rt_ty
                    {'script_name':'run_dlr_models.sh', 'script_dir':'osrt_cpp_scripts/','lang':'bash','rt_typ
        ]

With above change i can see gen_test_report.py getting executed and benchmarks .csv file created.

Hope this unblocks you as of now.

Best,

Pratik

0 Pratik Kedar over 2 years ago in reply to Pratik Kedar

TI__Mastermind 24041 points

I have raised 2 JIRAs highlighting the issue specific to Onnx and tvm_dlr runtimes,

Adding here for TIs internal tracking purpose.

https://jira.itg.ti.com/browse/TIDL-3812

https://jira.itg.ti.com/browse/TIDL-3807

Processors

Processors forum

SK-TDA4VM: edgeai-tidl-tools benchmark problems