(benchmark) machida@machida-VirtualBox:~/ti-processor-sdk-linux-edgeai-am62a-evm-10_01_00_05/edgeai-tensorlab/edgeai-benchmark$ ./run_custom_pc.sh AM62A TARGET_SOC: AM62A TARGET_MACHINE: pc DEBUG MODE: false @ machida-VirtualBox:5678 TIDL_TOOLS_PATH=/home/machida/ti-processor-sdk-linux-edgeai-am62a-evm-10_01_00_05/edgeai-tensorlab/edgeai-benchmark/tools/AM62A/FM_edgeai_tensorlab/tidl_tools LD_LIBRARY_PATH=/home/machida/ti-processor-sdk-linux-edgeai-am62a-evm-10_01_00_05/edgeai-tensorlab/edgeai-benchmark/tools/AM62A/FM_edgeai_tensorlab/tidl_tools PYTHONPATH=: =================================================================== work_dir = ./work_dirs/modelartifacts/AM62A/8bits packaged_dir = ./work_dirs/modelpackage/AM62A/8bits loading annotations into memory... Done (t=0.00s) creating index... index created! loading annotations into memory... Done (t=0.00s) creating index... index created! configs to run: ['imagedet-custom_cat_ORG_20250206_CH_TIDLTOOLS_onnxrt_DGX_yolov8_s_syncbn_fast_8xb16-100e_coco_optimized_best_coco_bbox_mAP_epoch_93_onnx'] number of configs: 1 INFO:20250206-094727: parallel_run - parallel_processes:1 parallel_devices=[0] TASKS | | 0% 0/1| [< ] INFO:20250206-094727: starting process on parallel_device - 0 0%| || 0/1 [00:00, 'input_dataset': , 'preprocess': , 'session': , 'postprocess': , 'metric': {'label_offset_pred': {0: 1, 1: 2, 2: 3, 3: 4, 4: 5, 5: 6, 6: 7, 7: 8, 8: 9, 9: 10, 10: 11, 11: 13, 12: 14, 13: 15, 14: 16, 15: 17, 16: 18, 17: 19, 18: 20, 19: 21, 20: 22, 21: 23, 22: 24, 23: 25, 24: 27, 25: 28, 26: 31, 27: 32, 28: 33, 29: 34, 30: 35, 31: 36, 32: 37, 33: 38, 34: 39, 35: 40, 36: 41, 37: 42, 38: 43, 39: 44, 40: 46, 41: 47, 42: 48, 43: 49, 44: 50, 45: 51, 46: 52, 47: 53, 48: 54, 49: 55, 50: 56, 51: 57, 52: 58, 53: 59, 54: 60, 55: 61, 56: 62, 57: 63, 58: 64, 59: 65, 60: 67, 61: 70, 62: 72, 63: 73, 64: 74, 65: 75, 66: 76, 67: 77, 68: 78, 69: 79, 70: 80, 71: 81, 72: 82, 73: 84, 74: 85, 75: 86, 76: 87, 77: 88, 78: 89, 79: 90, 80: 91}}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': 42.4}, 'model_shortlist': 70}} INFO:20250206-094729: import - imagedet-custom_cat_ORG_20250206_CH_TIDLTOOLS_onnxrt_DGX_yolov8_s_syncbn_fast_8xb16-100e_coco_optimized_best_coco_bbox_mAP_epoch_93_onnx - this may take some time.../home/machida/.pyenv/versions/benchmark/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:115: UserWarning: Specified provider 'TIDLCompilationProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider' warnings.warn( *************** EP Error *************** EP Error Unknown Provider Type: TIDLCompilationProvider when using ['TIDLCompilationProvider', 'CPUExecutionProvider'] Falling back to ['CPUExecutionProvider'] and retrying. **************************************** ================================ import model ============= INFO:20250206-094731: import completed - imagedet-custom_cat_ORG_20250206_CH_TIDLTOOLS_onnxrt_DGX_yolov8_s_syncbn_fast_8xb16-100e_coco_optimized_best_coco_bbox_mAP_epoch_93_onnx - 2 sec INFO:20250206-094731: infer - imagedet-custom_cat_ORG_20250206_CH_TIDLTOOLS_onnxrt_DGX_yolov8_s_syncbn_fast_8xb16-100e_coco_optimized_best_coco_bbox_mAP_epoch_93_onnx - this may take some time.../home/machida/.pyenv/versions/benchmark/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:115: UserWarning: Specified provider 'TIDLExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider' warnings.warn( *************** EP Error *************** EP Error Unknown Provider Type: TIDLExecutionProvider when using ['TIDLExecutionProvider', 'CPUExecutionProvider'] Falling back to ['CPUExecutionProvider'] and retrying. **************************************** infer : imagedet-custom_cat_ORG_20250206_CH_TIDLTOOLS_onnxrt| 64%|######3 || 74/116 [00:53<00:29, 1.41it/s] INFO:20250206-094827: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0 infer : imagedet-custom_cat_ORG_20250206_CH_TIDLTOOLS_onnxrt| 100%|##########|| 116/116 [01:22<00:00, 1.40it/s] INFO:20250206-094853: infer completed - imagedet-custom_cat_ORG_20250206_CH_TIDLTOOLS_onnxrt_DGX_yolov8_s_syncbn_fast_8xb16-100e_coco_optimized_best_coco_bbox_mAP_epoch_93_onnx - 83 secLoading and preparing results... DONE (t=0.00s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=0.06s). Accumulating evaluation results... DONE (t=0.01s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.020 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.080 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.020 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.027 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.030 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.030 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.030 SUCCESS:20250206-094854: benchmark results - {'infer_path': 'imagedet-custom_cat_ORG_20250206_CH_TIDLTOOLS_onnxrt_DGX_yolov8_s_syncbn_fast_8xb16-100e_coco_optimized_best_coco_bbox_mAP_epoch_93_onnx', 'accuracy_ap[.5:.95]%': 1.986918, 'accuracy_ap50%': 7.986799, 'num_subgraphs': 0, 'infer_time_core_ms': 0.0, 'infer_time_subgraph_ms': 0.0, 'ddr_transfer_mb': 0.0, 'perfsim_time_ms': 0.0, 'perfsim_ddr_transfer_mb': 0.0, 'perfsim_gmacs': 0.0} TASKS | 100%|██████████|| 1/1 [01:26<00:00, 86.53s/it] TASKS | 100%|██████████|| 1/1 [01:26<00:00, 86.42s/it] packaging artifacts to ./work_dirs/modelpackage/AM62A/8bits please wait... SUCCESS:20250206-094856: finished packaging - /home/machida/ti-processor-sdk-linux-edgeai-am62a-evm-10_01_00_05/edgeai-tensorlab/edgeai-benchmark/work_dirs/modelartifacts/AM62A/8bits/imagedet-custom_cat_3_onnxrt_DGX_yolov8_s_syncbn_fast_8xb16-100e_coco_optimized_best_coco_bbox_mAP_epoch_93_onnx SUCCESS:20250206-094858: finished packaging - /home/machida/ti-processor-sdk-linux-edgeai-am62a-evm-10_01_00_05/edgeai-tensorlab/edgeai-benchmark/work_dirs/modelartifacts/AM62A/8bits/imagedet-custom_cat_CH_param_1_onnxrt_DGX_yolov8_s_syncbn_fast_8xb16-100e_coco_optimized_best_coco_bbox_mAP_epoch_93_onnx SUCCESS:20250206-094900: finished packaging - /home/machida/ti-processor-sdk-linux-edgeai-am62a-evm-10_01_00_05/edgeai-tensorlab/edgeai-benchmark/work_dirs/modelartifacts/AM62A/8bits/imagedet-custom_cat_CH_param_onnxrt_DGX_yolov8_s_syncbn_fast_8xb16-100e_coco_optimized_best_coco_bbox_mAP_epoch_93_onnx SUCCESS:20250206-094902: finished packaging - /home/machida/ti-processor-sdk-linux-edgeai-am62a-evm-10_01_00_05/edgeai-tensorlab/edgeai-benchmark/work_dirs/modelartifacts/AM62A/8bits/imagedet-custom_cat_ORG_20250203_onnxrt_DGX_yolov8_s_syncbn_fast_8xb16-100e_coco_optimized_best_coco_bbox_mAP_epoch_93_onnx SUCCESS:20250206-094904: finished packaging - /home/machida/ti-processor-sdk-linux-edgeai-am62a-evm-10_01_00_05/edgeai-tensorlab/edgeai-benchmark/work_dirs/modelartifacts/AM62A/8bits/imagedet-custom_cat_ORG_20250206_CH_TIDLTOOLS_onnxrt_DGX_yolov8_s_syncbn_fast_8xb16-100e_coco_optimized_best_coco_bbox_mAP_epoch_93_onnx ------------------------------------------------------------------- settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5000, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/', 'modelpackage_path': './work_dirs/modelpackage/', 'datasets_path': './dependencies/datasets', 'target_device': None, 'target_machine': 'pc', 'run_suffix': None, 'parallel_devices': None, 'parallel_processes': 1, 'tensor_bits': 8, 'runtime_options': None, 'run_import': True, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': None, 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'additional_models': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'target_device_preset': True, 'fast_calibration_factor': None, 'param_template_file': None, 'skip_pattern': '_package', 'settings_file': 'settings_import_on_pc.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'modelpackage_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'parallel_processes', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'additional_models', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'target_device_preset', 'fast_calibration_factor', 'param_template_file', 'skip_pattern', 'settings_file'], 'dataset_cache': None} results found for 5 models Report generated at ./work_dirs/modelartifacts/ ------------------------------------------------------------------- =================================================================== (benchmark) machida@machida-VirtualBox:~/ti-processor-sdk-linux-edgeai-am62a-evm-10_01_00_05/edgeai-tensorlab/edgeai-benchmark$