Hello,TI
We want to use the benchmark tool to generate some artifacts that can be deployed on the sk-tda4vm board using the .onnx model. So we download Mobilenet_v2_tv.onnx model from（ software-dl.ti.com/.../mobilenet_v2_tv.onnx . And run ./ run_ custom_ cc.sh.
Why won't param.yaml, dataset. yaml, and artifacts folder be generated? It seems that only execute to the pipeline_ config.
[16:04:29] (benchmark) cxee@cxee-VirtualBox:~/yyl/fce_sdk/mmc/tool/edgeai-benchmark-r8.6$ ./run_custom_pc.sh
target_device/SOC: TDA4VM
Pass the appropriate commandline argument to use another target_device
Entering: ./work_dirs/modelartifacts/TDA4VM/8bits/ss-5710_tvmdlr_cocoseg21_edgeai-tv_deeplabv3plus_mobilenetv2_edgeailite_512x512_20210405_onnx.tar.gz.link
./run_set_target_machine.sh: line 66: cd: ./work_dirs/modelartifacts/TDA4VM/8bits/ss-5710_tvmdlr_cocoseg21_edgeai-tv_deeplabv3plus_mobilenetv2_edgeailite_512x512_20210405_onnx.tar.gz.link/artifacts: Not a directory
Entering: ./work_dirs/modelartifacts/TDA4VM/8bits/od-5120_tvmdlr_coco_tf1-models_ssdlite_mobiledet_dsp_320x320_coco_20200519_tflite.tar.gz.link
./run_set_target_machine.sh: line 66: cd: ./work_dirs/modelartifacts/TDA4VM/8bits/od-5120_tvmdlr_coco_tf1-models_ssdlite_mobiledet_dsp_320x320_coco_20200519_tflite.tar.gz.link/artifacts: Not a directory
Entering: ./work_dirs/modelartifacts/TDA4VM/8bits/cl-3090_tvmdlr_imagenet1k_torchvision_mobilenet_v2_tv_onnx.tar.gz.link
./run_set_target_machine.sh: line 66: cd: ./work_dirs/modelartifacts/TDA4VM/8bits/cl-3090_tvmdlr_imagenet1k_torchvision_mobilenet_v2_tv_onnx.tar.gz.link/artifacts: Not a directory
TIDL_TOOLS_PATH=/home/cxee/yyl/fce_sdk/mmc/tool/edgeai-benchmark-r8.6/tools/TDA4VM/tidl_tools
LD_LIBRARY_PATH=/home/cxee/yyl/fce_sdk/mmc/tool/edgeai-benchmark-r8.6/tools/TDA4VM/tidl_tools
PYTHONPATH=:
===================================================================
settings_file = settings_import_on_pc.yaml
model_selection = None
target_device = TDA4VM
work_dir = ./work_dirs/modelartifacts/TDA4VM/8bits
modelartifacts_path=./work_dirs/modelartifacts/TDA4VM
packaged_dir = ./work_dirs/modelartifacts/TDA4VM_package/8bits
ffffffffffffffffffffffffffffffffffffff
dff={'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 10000, 'calibration_frames': 25, 'calibration_iterations': 25, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/TDA4VM', 'datasets_path': './dependencies/datasets', 'target_device': 'TDA4VM', 'target_machine': 'pc', 'run_suffix': None, 'parallel_devices': 1, 'tensor_bits': 8, 'runtime_options': None, 'run_import': True, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': None, 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth'], 'dataset_cache': None}
!!!!!!!!!!!!!!!!!!!!!!!
modelpath=../edgeai-modelzoo/models
configs to run: ['imagecls-1_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx']
number of configs: 1
INFO:20231104-160431: starting process on parallel_device - 0
INFO:20231104-160431: starting - imagecls-1_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx
INFO:20231104-160431: model_path - /home/cxee/yyl/fce_sdk/mmc/tool/edgeai-modelzoo/models/vision/classification/imagenet1k/torchvision/mobilenet_v2_tv.onnx
INFO:20231104-160431: model_file - /home/cxee/yyl/fce_sdk/mmc/tool/edgeai-benchmark-r8.6/work_dirs/modelartifacts/TDA4VM/8bits/imagecls-1_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx/model/mobilenet_v2_tv.onnx
INFO:20231104-160431: running - imagecls-1_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx
INFO:20231104-160431: pipeline_config - {'task_type': 'classification', 'calibration_dataset': <edgeai_benchmark.datasets.image_cls.ImageClassification object at 0x7fabba043710>, 'input_dataset': <edgeai_benchmark.datasets.image_cls.ImageClassification object at 0x7fabba0435f8>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7fabba0437b8>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7fabba043b00>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7fabba0439b0>, 'model_info': {'metric_reference': {'accuracy_top1%': 71.88}}}
TASKS | | 0% 0/1| [< ]Traceback (most recent call last):
File "/home/cxee/yyl/fce_sdk/mmc/tool/edgeai-benchmark-r8.6/edgeai_benchmark/pipelines/pipeline_runner.py", line 154, in _run_pipeline
result = cls._run_pipeline_impl(settings, pipeline_config, description)
File "/home/cxee/yyl/fce_sdk/mmc/tool/edgeai-benchmark-r8.6/edgeai_benchmark/pipelines/pipeline_runner.py", line 125, in _run_pipeline_impl
accuracy_result = accuracy_pipeline(description)
File "/home/cxee/yyl/fce_sdk/mmc/tool/edgeai-benchmark-r8.6/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 122, in __call__
param_result = self._run(description=description)
File "/home/cxee/yyl/fce_sdk/mmc/tool/edgeai-benchmark-r8.6/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 148, in _run
self._import_model(description)
File "/home/cxee/yyl/fce_sdk/mmc/tool/edgeai-benchmark-r8.6/edgeai_benchmark/pipelines/accuracy_pipeline.py", line 197, in _import_model
data = calibration_dataset[data_index]
File "/home/cxee/yyl/fce_sdk/mmc/tool/edgeai-benchmark-r8.6/edgeai_benchmark/datasets/image_cls.py", line 75, in __getitem__
words = self.imgs[idx].split(' ')
IndexError: list index out of range
list index out of range
TASKS | 100%|██████████|| 1/1 [00:00<00:00, 376.68it/s]
packaging artifacts to ./work_dirs/modelartifacts/TDA4VM_package/8bits please wait...
WARNING:20231104-160432: could not package - ./work_dirs/modelartifacts/TDA4VM/8bits/imagecls-1_onnxrt_imagenet1k_torchvision_mobilenet_v2_tv_onnx
Traceback (most recent call last):
File "./scripts/benchmark_fce.py", line 226, in <module>
tools.run_package(settings, work_dir, packaged_dir, custom_model=True)
File "/home/cxee/yyl/fce_sdk/mmc/tool/edgeai-benchmark-r8.6/edgeai_benchmark/tools/run_package.py", line 42, in run_package
package_artifacts(settings, work_dir, out_dir, include_results=include_results, custom_model=custom_model)
File "/home/cxee/yyl/fce_sdk/mmc/tool/edgeai-benchmark-r8.6/edgeai_benchmark/tools/run_package.py", line 274, in package_artifacts
packaged_artifacts_keys = list(packaged_artifacts_dict.values())[0].keys()
IndexError: list index out of range
-------------------------------------------------------------------
===================================================================
settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 10000, 'calibration_frames': 25, 'calibration_iterations': 25, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/', 'datasets_path': './dependencies/datasets', 'target_device': None, 'target_machine': 'pc', 'run_suffix': None, 'parallel_devices': 1, 'tensor_bits': 8, 'runtime_options': None, 'run_import': True, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'onnx', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_import_on_pc.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None}
no results found - no report to generate.
Report generated at ./work_dirs/modelartifacts/
-------------------------------------------------------------------
