This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

SK-TDA4VM: TDA4VM -benchMark deployment problem

Part Number: SK-TDA4VM

weights.zipHi,

My sdk version is  Processor SDK Linux for Edge AI 08.04.00. I trained a custom model following the https://github.com/TexasInstruments/edgeai-yolov5Then apply the onnx conversion with " python3 export.py --weights run/exp5/weights/best.pt --img 640 --batch 1 --simplify --export-nms --opset 11" as a result I copied the .prototxt and .onnx files.  I copied them to the "opt/edgeai-benchmark-r8.4/work_dirs/modelartifacts_yolov5s6_640" folder and made the following changes. But the creation of model artifacts is interrupted. What could be the problem?

“benchmark_custom.py”  file:
  'imagedet-7': dict(
            task_type='detection',
            calibration_dataset=imagedet_calib_dataset,
            input_dataset=imagedet_val_dataset,
            preprocess=preproc_transforms.get_transform_onnx(640, 640,  resize_with_pad=True, backend='cv2', pad_color=[114,114,114]),
            session=sessions.ONNXRTSession(**utils.dict_update(onnx_session_cfg, input_optimization=False, input_mean=(0.0, 0.0, 0.0), input_scale=(0.003921568627, 0.003921568627, 0.003921568627)),
                runtime_options=settings.runtime_options_onnx_np2(
                    det_options=True, ext_options={'object_detection:meta_arch_type': 6,
                     'object_detection:meta_layers_names_list':f'/home/machunlei/opt/edgeai-benchmark-r8.4/work_dirs/modelartifacts_yolov5s6_640/best.prototxt',
                     'advanced_options:output_feature_16bit_names_list':''
                     }),
                model_path=f'/home/machunlei/opt/edgeai-benchmark-r8.4/work_dirs/modelartifacts_yolov5s6_640/best.onnx'),
            postprocess=postproc_transforms.get_transform_detection_yolov5_onnx(squeeze_axis=None, normalized_detections=False, resize_with_pad=True, formatter=postprocess.DetectionBoxSL2BoxLS()), #TODO: check this
            metric=dict(label_offset_pred=datasets.coco_det_label_offset_80to90(label_offset=1)),
            model_info=dict(metric_reference={'accuracy_ap[.5:.95]%':37.4})
        ),

"setting_base.yaml" file:

modelartifacts_path : './work_dirs/modelartifacts_yolov5s6_640'

(py36) root@iZ2zegfoso26e6c51v05bnZ:/home/machunlei/opt/edgeai-benchmark-r8.4# ./run_custom_pc.sh
find: ‘./work_dirs/modelartifacts/8bits/’: No such file or directory
TIDL_TOOLS_PATH=/home/machunlei/opt/edgeai-benchmark-r8.4/tidl_tools
LD_LIBRARY_PATH=/home/machunlei/opt/edgeai-benchmark-r8.4/tidl_tools
PYTHONPATH=::
===================================================================
work_dir = ./work_dirs/modelartifacts_yolov5s6_640/8bits
packaged_dir = ./work_dirs/modelartifacts_yolov5s6_640_package/8bits
loading annotations into memory...
Done (t=0.60s)
creating index...
index created!
loading annotations into memory...
Done (t=0.65s)
creating index...
index created!
configs to run: ['imagedet-7_onnxrt_work_dirs_modelartifacts_yolov5s6_640_best_onnx']
number of configs: 1
TASKS | | 0% 0/1| [< ]
INFO:20230227-165139: starting process on parallel_device - 0 0%| || 0/1 [00:00<?, ?it/s]

INFO:20230227-165148: starting - imagedet-7_onnxrt_work_dirs_modelartifacts_yolov5s6_640_best_onnx
INFO:20230227-165148: model_path - /home/machunlei/opt/edgeai-benchmark-r8.4/work_dirs/modelartifacts_yolov5s6_640/best.onnx
INFO:20230227-165148: model_file - /home/machunlei/opt/edgeai-benchmark-r8.4/work_dirs/modelartifacts_yolov5s6_640/8bits/imagedet-7_onnxrt_work_dirs_modelartifacts_yolov5s6_640_best_onnx/model/best.onnx

INFO:20230227-165148: running - imagedet-7_onnxrt_work_dirs_modelartifacts_yolov5s6_640_best_onnx
INFO:20230227-165148: pipeline_config - {'task_type': 'detection', 'calibration_dataset': <edgeai_benchmark.datasets.coco_det.COCODetection object at 0x7f2ae3ff8208>, 'input_dataset': <edgeai_benchmark.datasets.coco_det.COCODetection object at 0x7f2aff1020f0>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7f2acdb6b400>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7f2acdb6b828>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7f2acdb6b898>, 'metric': {'label_offset_pred': {0: 1, 1: 2, 2: 3, 3: 4, 4: 5, 5: 6, 6: 7, 7: 8, 8: 9, 9: 10, 10: 11, 11: 13, 12: 14, 13: 15, 14: 16, 15: 17, 16: 18, 17: 19, 18: 20, 19: 21, 20: 22, 21: 23, 22: 24, 23: 25, 24: 27, 25: 28, 26: 31, 27: 32, 28: 33, 29: 34, 30: 35, 31: 36, 32: 37, 33: 38, 34: 39, 35: 40, 36: 41, 37: 42, 38: 43, 39: 44, 40: 46, 41: 47, 42: 48, 43: 49, 44: 50, 45: 51, 46: 52, 47: 53, 48: 54, 49: 55, 50: 56, 51: 57, 52: 58, 53: 59, 54: 60, 55: 61, 56: 62, 57: 63, 58: 64, 59: 65, 60: 67, 61: 70, 62: 72, 63: 73, 64: 74, 65: 75, 66: 76, 67: 77, 68: 78, 69: 79, 70: 80, 71: 81, 72: 82, 73: 84, 74: 85, 75: 86, 76: 87, 77: 88, 78: 89, 79: 90, 80: 91}}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': 37.4}}}
INFO:20230227-165148: import - imagedet-7_onnxrt_work_dirs_modelartifacts_yolov5s6_640_best_onnx - this may take some time.../root/anaconda3/envs/py36/lib/python3.6/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:56: UserWarning: Specified provider 'TIDLCompilationProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
"Available providers: '{}'".format(name, ", ".join(available_provider_names)))
Traceback (most recent call last):
File "/home/machunlei/opt/edgeai-benchmark-r8.4/edgeai_benchmark/pipelines/pipeline_runner.py", line 154, in _run_pipeline
result = cls._run_pipeline_impl(settings, pipeline_config, description)
File "/home/machunlei/opt/edgeai-benchmark-r8.4/edgeai_benchmark/pipelines/pipeline_runner.py", line 125, in _run_pipeline_impl
accuracy_result = accuracy_pipeline(description)
File "/home/machunlei/opt/edgeai-benchmark-r8.4/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 121, in __call__
param_result = self._run(description=description)
File "/home/machunlei/opt/edgeai-benchmark-r8.4/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 147, in _run
self._import_model(description)
File "/home/machunlei/opt/edgeai-benchmark-r8.4/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 201, in _import_model
self._run_with_log(session.import_model, calib_data)
File "/home/machunlei/opt/edgeai-benchmark-r8.4/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 301, in _run_with_log
return func(*args, **kwargs)
File "/home/machunlei/opt/edgeai-benchmark-r8.4/edgeai_benchmark/sessions/onnxrt_session.py", line 53, in import_model
self.interpreter = self._create_interpreter(is_import=True)
File "/home/machunlei/opt/edgeai-benchmark-r8.4/edgeai_benchmark/sessions/onnxrt_session.py", line 132, in _create_interpreter
provider_options=[runtime_options, {}], sess_options=sess_options)
File "/root/anaconda3/envs/py36/lib/python3.6/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 335, in __init__
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "/root/anaconda3/envs/py36/lib/python3.6/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 379, in _create_inference_session
sess.initialize_session(providers, provider_options, disabled_optimizers)
RuntimeError: Unknown Provider Type: TIDLCompilationProvider
Unknown Provider Type: TIDLCompilationProvider
TASKS | 100%|██████████|| 1/1 [00:12<00:00, 12.57s/it]
TASKS | 100%|██████████|| 1/1 [00:12<00:00, 12.57s/it]

packaging artifacts to ./work_dirs/modelartifacts_yolov5s6_640_package/8bits please wait...
WARNING:20230227-165149: could not package - ./work_dirs/modelartifacts_yolov5s6_640/8bits/imagedet-7_onnxrt_work_dirs_modelartifacts_yolov5s6_640_best_onnx
Traceback (most recent call last):
File "./scripts/benchmark_custom.py", line 299, in <module>
tools.run_package(settings, work_dir, packaged_dir, custom_model=True)
File "/home/machunlei/opt/edgeai-benchmark-r8.4/edgeai_benchmark/tools/run_package.py", line 42, in run_package
package_artifacts(settings, work_dir, out_dir, include_results=include_results, custom_model=custom_model)
File "/home/machunlei/opt/edgeai-benchmark-r8.4/edgeai_benchmark/tools/run_package.py", line 269, in package_artifacts
packaged_artifacts_keys = list(packaged_artifacts_dict.values())[0].keys()
IndexError: list index out of range
-------------------------------------------------------------------
===================================================================
settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 10000, 'calibration_frames': 25, 'calibration_iterations': 25, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts_yolov5s6_640', 'datasets_path': './dependencies/datasets', 'target_device': None, 'target_machine': 'pc', 'run_suffix': None, 'parallel_devices': [0], 'tensor_bits': 8, 'runtime_options': None, 'run_import': True, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': None, 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_import_on_pc.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None}
no results found - no report to generate.
Report generated at ./work_dirs/modelartifacts_yolov5s6_640

Attachment is my custom model.  Please help check what is the problem ?

Maiunlei,