Tool/software:
Hi all,
I have run compiles of OD-8220 yolox network with edgeai-benchmark as a pipe cleaner. In the past I have done this with branch r10.0 and have not seen any issues.
However today I attempted to upgrade to branch r10.1 and wanted to replicate that the compile still works, however am seeing an import error as follows. BTW, during this run I have not touched the python environment coming from r10.0.
(ti-edgeai-benchmark) gunter@Linux-005:~/ti-edgeai/edgeai-tensorlab/edgeai-benchmark$ ./run_benchmarks_pc.sh AM69A
TARGET_SOC: AM69A
TARGET_MACHINE: pc
TIDL_TOOLS_PATH=/home/gunter/ti-edgeai/edgeai-tensorlab/edgeai-benchmark/tools/tidl_tools_package/AM69A/tidl_tools
LD_LIBRARY_PATH=/home/gunter/ti-edgeai/edgeai-tensorlab/edgeai-benchmark/tools/tidl_tools_package/AM69A/tidl_tools:
PYTHONPATH=:
INFO: settings the correct symlinks in tvmdlr compiled artifacts
===================================================================
argv: ['./scripts/benchmark_modelzoo.py', 'settings_import_on_pc.yaml', '--target_device', 'AM69A', '--model_shortlist', '120']
settings: {'include_files': None, 'pipeline_type': 'accuracy', 'c7x_firmware_version': None, 'num_frames': 100, 'calibration_frames': 12, 'calibration_iterations': 12, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/AM69A', 'modelpackage_path': './work_dirs/modelpackage/AM69A', 'datasets_path': './dependencies/datasets', 'target_device': 'AM69A', 'target_machine': 'pc', 'run_suffix': None, 'parallel_devices': None, 'parallel_processes': 12, 'tensor_bits': 8, 'runtime_options': {'advanced_options:quantization_scale_type': 4}, 'run_import': True, 'run_inference': True, 'run_incremental': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'od-8220', 'model_shortlist': 120, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': None, 'dataset_selection': 'coco', 'dataset_loading': True, 'config_range': None, 'write_results': True, 'verbose': True, 'log_file': True, 'additional_models': True, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'target_device_preset': True, 'calibration_iterations_factor': None, 'instance_timeout': None, 'overall_timeout': None, 'sort_pipeline_configs': True, 'check_errors': True, 'param_template_file': None, 'enable_logging': True, 'basic_keys': ['include_files', 'pipeline_type', 'c7x_firmware_version', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'modelpackage_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'parallel_processes', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_incremental', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'write_results', 'verbose', 'log_file', 'additional_models', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'target_device_preset', 'calibration_iterations_factor', 'instance_timeout', 'overall_timeout', 'sort_pipeline_configs', 'check_errors', 'param_template_file', 'enable_logging'], 'dataset_cache': {'imagenet': {'calibration_dataset': 'imagenet', 'input_dataset': 'imagenet'}, 'coco': {'calibration_dataset': 'coco', 'input_dataset': 'coco'}, 'widerface': {'calibration_dataset': 'widerface', 'input_dataset': 'widerface'}, 'ade20k32': {'calibration_dataset': 'ade20k32', 'input_dataset': 'ade20k32'}, 'ade20k': {'calibration_dataset': 'ade20k', 'input_dataset': 'ade20k'}, 'voc2012': {'calibration_dataset': 'voc2012', 'input_dataset': 'voc2012'}, 'cocoseg21': {'calibration_dataset': 'cocoseg21', 'input_dataset': 'cocoseg21'}, 'ti-robokit_semseg_zed1hd': {'calibration_dataset': 'ti-robokit_semseg_zed1hd', 'input_dataset': 'ti-robokit_semseg_zed1hd'}, 'ti-robokit_visloc_zed1hd': {'calibration_dataset': 'ti-robokit_visloc_zed1hd', 'input_dataset': 'ti-robokit_visloc_zed1hd'}, 'cocokpts': {'calibration_dataset': 'cocokpts', 'input_dataset': 'cocokpts'}, 'nyudepthv2': {'calibration_dataset': 'nyudepthv2', 'input_dataset': 'nyudepthv2'}, 'ycbv': {'calibration_dataset': 'ycbv', 'input_dataset': 'ycbv'}}}
INFO: model compilation in PC can use CUDA gpus (if it is available) - setup using setup_pc_gpu.sh
/bin/sh: 1: nvidia-smi: not found
INFO: setting parallel_devices to the number of cuda gpus found - 0
work_dir: ./work_dirs/modelartifacts/AM69A/8bits
INFO: using model configs from Python module: ./configs
INFO:20250506-173602: loading dataset - category:coco variant:coco
INFO:20250506-173602: dataset exists - will reuse - ./dependencies/datasets/coco
loading annotations into memory...
Done (t=0.42s)
creating index...
index created!
loading annotations into memory...
Done (t=0.46s)
creating index...
index created!
WARNING:20250506-173604: model_shortlist=120 - this will cause only a subset of models to be selected for run
WARNING:20250506-173604: if the model that you wish is not being selected for run, then remove this model_shortlist -
WARNING:20250506-173604: this model_shortlist could be being set in settings_base.yaml or passed inside run_benchmarks_pc.sh -
INFO:20250506-173604: number of configs - 1
TASKS TOTAL=1, NUM_RUNNING=1: 0%| | 0/1 [00:00<?, ?it/s, postfix={'RUNNING': ['od-8220:import'], 'COMPLETED': []}]
ERROR:20250506-173605: Error occurred: od-8220:import - Error Code: -11 at /home/gunter/ti-edgeai/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/utils/parallel_runner.py
TASKS TOTAL=1, NUM_RUNNING=1: 0%| | 0/1 [00:01<?, ?it/s, postfix={'RUNNING': ['od-8220:infer'], 'COMPLETED': []}]
ERROR:20250506-173606: Error occurred: od-8220:infer - Error Code: -11 at /home/gunter/ti-edgeai/edgeai-tensorlab/edgeai-benchmark/edgeai_benchmark/utils/parallel_runner.py
TASKS TOTAL=1, NUM_RUNNING=0: 100%|████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 6.59it/s, postfix={'RUNNING': [], 'COMPLETED': ['od-8220']}]
WARNING: Benchmark - completed: 0/1
TASKS TOTAL=1, NUM_RUNNING=0: 100%|████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.53s/it, postfix={'RUNNING': [], 'COMPLETED': ['od-8220']}]
settings: {'include_files': None, 'pipeline_type': 'accuracy', 'c7x_firmware_version': None, 'num_frames': 100, 'calibration_frames': 12, 'calibration_iterations': 12, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/', 'modelpackage_path': './work_dirs/modelpackage/', 'datasets_path': './dependencies/datasets', 'target_device': None, 'target_machine': 'pc', 'run_suffix': None, 'parallel_devices': None, 'parallel_processes': 12, 'tensor_bits': 8, 'runtime_options': {}, 'run_import': True, 'run_inference': True, 'run_incremental': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'od-8220', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': None, 'dataset_selection': 'coco', 'dataset_loading': True, 'config_range': None, 'write_results': True, 'verbose': True, 'log_file': True, 'additional_models': True, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'target_device_preset': True, 'calibration_iterations_factor': None, 'instance_timeout': None, 'overall_timeout': None, 'sort_pipeline_configs': True, 'check_errors': True, 'param_template_file': None, 'enable_logging': True, 'basic_keys': ['include_files', 'pipeline_type', 'c7x_firmware_version', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'modelpackage_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'parallel_processes', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_incremental', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'write_results', 'verbose', 'log_file', 'additional_models', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'target_device_preset', 'calibration_iterations_factor', 'instance_timeout', 'overall_timeout', 'sort_pipeline_configs', 'check_errors', 'param_template_file', 'enable_logging'], 'dataset_cache': {'imagenet': {'calibration_dataset': 'imagenet', 'input_dataset': 'imagenet'}, 'coco': {'calibration_dataset': 'coco', 'input_dataset': 'coco'}, 'widerface': {'calibration_dataset': 'widerface', 'input_dataset': 'widerface'}, 'ade20k32': {'calibration_dataset': 'ade20k32', 'input_dataset': 'ade20k32'}, 'ade20k': {'calibration_dataset': 'ade20k', 'input_dataset': 'ade20k'}, 'voc2012': {'calibration_dataset': 'voc2012', 'input_dataset': 'voc2012'}, 'cocoseg21': {'calibration_dataset': 'cocoseg21', 'input_dataset': 'cocoseg21'}, 'ti-robokit_semseg_zed1hd': {'calibration_dataset': 'ti-robokit_semseg_zed1hd', 'input_dataset': 'ti-robokit_semseg_zed1hd'}, 'ti-robokit_visloc_zed1hd': {'calibration_dataset': 'ti-robokit_visloc_zed1hd', 'input_dataset': 'ti-robokit_visloc_zed1hd'}, 'cocokpts': {'calibration_dataset': 'cocokpts', 'input_dataset': 'cocokpts'}, 'nyudepthv2': {'calibration_dataset': 'nyudepthv2', 'input_dataset': 'nyudepthv2'}, 'ycbv': {'calibration_dataset': 'ycbv', 'input_dataset': 'ycbv'}}}
results found for 4 models
Report generated at ./work_dirs/modelartifacts/
===================================================================
Here is a diff of the settings_base.yaml and detection.py
(ti-edgeai-benchmark) gunter@Linux-005:~/ti-edgeai/edgeai-tensorlab/edgeai-benchmark$ git diff HEAD
diff --git a/edgeai-benchmark/configs/detection.py b/edgeai-benchmark/configs/detection.py
index c12936325d..606fdfb2fc 100644
--- a/edgeai-benchmark/configs/detection.py
+++ b/edgeai-benchmark/configs/detection.py
@@ -197,7 +197,7 @@ def get_configs(settings, work_dir):
# this causes issue in SoCs that do not use tidl_offload
'od-8220':utils.dict_update(common_cfg,
preprocess=preproc_transforms.get_transform_onnx(640, 640, reverse_channels=True, resize_with_pad=[True, "corner"], backend='cv2', pad_color=[114, 114, 114]),
- session=onnx_session_type(**sessions.get_common_session_cfg(settings, work_dir=work_dir),
+ session=onnx_session_type(**sessions.get_common_session_cfg(settings, work_dir=work_dir, input_optimization=False),
runtime_options=settings.runtime_options_onnx_np2(
det_options=True, ext_options={'object_detection:meta_arch_type': 6,
'object_detection:meta_layers_names_list': f'{settings.models_path}/vision/detection/coco/edgeai-mmdet/yolox_s_lite_640x640_20220221_model.prototxt',
@@ -243,6 +243,35 @@ def get_configs(settings, work_dir):
metric=dict(label_offset_pred=datasets.coco_det_label_offset_80to90(label_offset=1)),
model_info=dict(metric_reference={'accuracy_ap[.5:.95]%': 17.9}, model_shortlist=20, compact_name='yolox-pico-lite-mmdet-coco-320x320', shortlisted=True, recommended=True)
),
+ #yolov7-tiny
+ 'od-8850':utils.dict_update(common_cfg,
+ preprocess=preproc_transforms.get_transform_onnx(640, 640, resize_with_pad=True, backend='cv2', pad_color=[114,114,114]),
+ session=onnx_session_type(**sessions.get_onnx_session_cfg(settings, work_dir=work_dir, input_optimization=False, input_mean=(0.0, 0.0, 0.0), input_scale=(0.003921568627, 0.003921568627, 0.003921568627)),
+ runtime_options=settings.runtime_options_onnx_np2(
+ det_options=True, ext_options={'object_detection:meta_arch_type': 6,
+ 'object_detection:meta_layers_names_list':f'../edgeai-modelzoo-cl/models/vision/detection/coco/edgeai-mmyolo-gplv3/yolov7_tiny_lite_640x640_20230830_model.prototxt',
+ 'advanced_options:output_feature_16bit_names_list':'1,137,147,157'},
+ fast_calibration=True),
+ model_path=f'../edgeai-modelzoo-cl/models/vision/detection/coco/edgeai-mmyolo-gplv3/yolov7_tiny_lite_640x640_20230830_model.onnx'),
+ postprocess=postproc_transforms.get_transform_detection_yolov5_onnx(squeeze_axis=None, normalized_detections=False, resize_with_pad=True, formatter=postprocess.DetectionBoxSL2BoxLS()), #TODO: check this
+ metric=dict(label_offset_pred=datasets.coco_det_label_offset_80to90(label_offset=1)),
+ model_info=dict(metric_reference={'accuracy_ap[.5:.95]%':36.7}, model_shortlist=70, compact_name='yolov7-tiny-lite-640x640-gplv3', shortlisted=False)
+ ),
+ #yolov7-large
+ 'od-8860':utils.dict_update(common_cfg,
+ preprocess=preproc_transforms.get_transform_onnx(640, 640, resize_with_pad=True, backend='cv2',pad_color=[114,114,114]),
+ session=onnx_session_type(**sessions.get_onnx_session_cfg(settings, work_dir=work_dir, input_optimization=False, input_mean=(0.0, 0.0, 0.0), input_scale=(0.003921568627, 0.003921568627, 0.003921568627)),
+ runtime_options=settings.runtime_options_onnx_np2(
+ det_options=True, ext_options={'object_detection:meta_arch_type': 6,
+ 'object_detection:meta_layers_names_list':f'../edgeai-modelzoo-cl/models/vision/detection/coco/edgeai-mmyolo-gplv3/yolov7_large_lite_640x640_20240119_model.prototxt',
+ 'advanced_options:output_feature_16bit_names_list':'2,221,251,277'
+ },
+ fast_calibration=True),
+ model_path=f'../edgeai-modelzoo-cl/models/vision/detection/coco/edgeai-mmyolo-gplv3/yolov7_large_lite_640x640_20240119_model.onnx'),
+ postprocess=postproc_transforms.get_transform_detection_yolov5_onnx(squeeze_axis=None, normalized_detections=False, resize_with_pad=True, formatter=postprocess.DetectionBoxSL2BoxLS()), #TODO: check this
+ metric=dict(label_offset_pred=datasets.coco_det_label_offset_80to90(label_offset=1)),
+ model_info=dict(metric_reference={'accuracy_ap[.5:.95]%':48.1}, model_shortlist=70, compact_name='yolov7-large-lite-640x640-gplv3', shortlisted=False)
+ ),
#################################################################
# MXNET MODELS
#################################################################
diff --git a/edgeai-benchmark/settings_base.yaml b/edgeai-benchmark/settings_base.yaml
index 2d83571190..aa9c18b0a8 100644
--- a/edgeai-benchmark/settings_base.yaml
+++ b/edgeai-benchmark/settings_base.yaml
@@ -13,7 +13,7 @@ target_device : null
tensor_bits : 8
# number of frames for inference
-num_frames : 1000 #5000 #10000 #50000
+num_frames : 100 #1000 #5000 #10000 #50000
# number of frames to be used for post training quantization / calibration
calibration_frames : 12 #25 #50
@@ -52,7 +52,7 @@ session_type_dict : {'onnx':'onnxrt', 'tflite':'tflitert', 'mxnet':'tvmdlr'}
# examples: ['resnet18.onnx', 'resnet50_v1.tflite'] ['classification'] ['imagenet1k'] ['torchvision'] ['coco']
# examples: [cl-0000, od-2020, ss-2580, cl-3090, od-5120, ss-5710, cl-6360, od-8020, od-8200, od-8220, od-8420, ss-8610, kd-7060, 3dod-7100, 6dpose-7200, ss-7618]
# examples: [cl-0000, od-2020, cl-6360, od-8200, od-8270, od-8410, ss-8610, ss-8630, ss-8710, ss-8720]
-model_selection : null
+model_selection : od-8220
# model_shortlist can be a number, which indicates a predefined shortlist, and a fraction of the models will be selected
# model_shortlist and model_selection are complimentary - they can be used together.
@@ -78,7 +78,7 @@ runtime_selection : null
# set this to null to run all the models
# example: ['coco']
# example: ['imagenet', 'coco', 'cocoseg21', 'ade20k', 'cocokpts', 'kitti_lidar_det', 'ti-robokit_semseg_zed1hd', 'ycbv']
-dataset_selection : null
+dataset_selection : coco
# use TIDL offload to speedup inference
tidl_offload : True
@@ -97,7 +97,7 @@ detection_threshold : 0.3
detection_top_k : 200
# verbose mode - print out more information
-verbose : False
+verbose : True
# save detection, segmentation, human pose estimation output
save_output : False
@@ -112,7 +112,7 @@ enable_logging: True
write_results: True
# enable use of additional models - the actual model files may be in a different modelzoo repo (for example edgeai-modelzoo-cl)
-additional_models : False
+additional_models : True
# enable use of experimental models - these model files may not be available in modelzoo in some cases
experimental_models : False
Is there something obvious that was missed?
Thanks!
--Gunter
