This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

[FAQ] TDA4VM: How to inference/benchmark on TDA4VM SoC with precompiled model artifacts using edgeai-benchmark repo ?

Part Number: TDA4VM

Hi,

How to benchmark Object Detection, Classification and Segmentation models which are part of edgeai-benchmark repos on target SoC ?

What KPIs are supported for benchmarking model on target SoC ?

Thanks for the help. 

  • Hi,

    One can benchmark all the supported models on edgeai-benchmark repo using following steps:

    Clone edgeai-benchmark repo on your PC, and follow the setup instructions listed here : link

    For this FAQ we have used linux sdk version 8.6, and hence we are on branch r8.6.

    We have plenty of supported models inside edgeai-benchmark repo, you can check file setting_base.yaml for reference.

    For this FAQ we have considered classification model movileNet V1 which is present inside model zoo with reference tag cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite

    Change setting_base.yaml file line model_selection : null to model_selection : cl-0000 this will consume only selected model for compilation part.

    If model_selection param set to null then by default all model will be considered for compilation.

    Make sure there are no any links file present inside edgeai-benchmark/work_dirs/modelartifacts folder, else compilation will be skipped.

    Run benchmark script as,

    run_benchmarks_pc.sh TDA4VM

    After successful execution of run_becnhmark_pc.sh script console logs will look like below..

    (benchmark) user@udtensorlab1:~/model-training/edgeai-benchmark$ ./run_benchmarks_pc.sh TDA4VM
    
    target_device/SOC: TDA4VM
    Pass the appropriate commandline argument to use another target_device
    find: ‘./work_dirs/modelartifacts/TDA4VM/8bits/’: No such file or directory
    TIDL_TOOLS_PATH=/home/user/model-training/edgeai-benchmark/tools/TDA4VM/tidl_tools
    LD_LIBRARY_PATH=/home/user/model-training/edgeai-benchmark/tools/TDA4VM/tidl_tools
    PYTHONPATH=:
    -------------------------------------------------------------------
    argv: ['./scripts/benchmark_modelzoo.py', 'settings_import_on_pc.yaml', '--target_device', 'TDA4VM']
    settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/TDA4VM_package', 'datasets_path': './dependencies/datasets', 'target_device': 'TDA4VM', 'target_machine': 'pc', 'run_suffix': None, 'parallel_devices': 1, 'tensor_bits': 8, 'runtime_options': None, 'run_import': True, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'cl-0000', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_import_on_pc.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None}
    work_dir: ./work_dirs/modelartifacts/TDA4VM_package/8bits
    
    INFO:20230607-173711: dataset exists - will reuse - ./dependencies/datasets/imagenetv2c/val
    
    INFO:20230607-173711: dataset exists - will reuse - ./dependencies/datasets/coco
    loading annotations into memory...
    Done (t=0.30s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.36s)
    creating index...
    index created!
    
    INFO:20230607-173711: dataset exists - will reuse - ./dependencies/datasets/ycbv
    loading annotations into memory...
    Done (t=0.04s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.03s)
    creating index...
    index created!
    
    INFO:20230607-173717: dataset exists - will reuse - ./dependencies/datasets/coco
    loading annotations into memory...
    Done (t=0.58s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.76s)
    creating index...
    index created!
    
    INFO:20230607-173720: dataset exists - will reuse - ./dependencies/datasets/widerface
    loading annotations into memory...
    Done (t=0.49s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.12s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.12s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.12s)
    creating index...
    index created!
    
    INFO:20230607-173722: dataset exists - will reuse - ./dependencies/datasets/coco
    loading annotations into memory...
    Done (t=0.46s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.46s)
    creating index...
    index created!
    
    INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/ADEChallengeData2016
    
    INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/ADEChallengeData2016
    
    INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/VOCdevkit/VOC2012
    
    INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/nyudepthv2
    
    INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/ti-robokit_semseg_zed1hd
    
    INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/ti-robokit_semseg_zed1hd
    download_ok: True
    configs to run: ['cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite']
    number of configs: 1
    TASKS                                                       |          |     0% 0/1| [< ]
    INFO:20230607-173733: starting process on parallel_device - 0   0%|          || 0/1 [00:00<?, ?it/s]
    
    INFO:20230607-173734: starting - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite
    INFO:20230607-173734: model_path - /home/user/model-training/edgeai-modelzoo/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite
    INFO:20230607-173734: model_file - /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/model/mobilenet_v1_1.0_224.tflite
    Downloading 1/1: /home/user/model-training/edgeai-modelzoo/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite
    Downloading http://software-dl.ti.com/jacinto7/esd/modelzoo/latest/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite to /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/model/mobilenet_v1_1.0_224.tflite
    16908288it [00:08, 1992659.04it/s]                                                                                                                                                                                 
    Download done for /home/user/model-training/edgeai-modelzoo/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite
    /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/model/mobilenet_v1_1.0_224.tflite
    
    INFO:20230607-173742: running - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite
    INFO:20230607-173742: pipeline_config - {'task_type': 'classification', 'dataset_category': 'imagenet', 'calibration_dataset': <edgeai_benchmark.datasets.imagenetv2.ImageNetV2C object at 0x7f84f3177250>, 'input_dataset': <edgeai_benchmark.datasets.imagenetv2.ImageNetV2C object at 0x7f84f3177190>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7f84f3177110>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7f84f33c9c50>, 'session': <edgeai_benchmark.sessions.tflitert_session.TFLiteRTSession object at 0x7f845a1b5cd0>, 'metric': {'label_offset_pred': -1}, 'model_info': {'metric_reference': {'accuracy_top1%': 71.676}, 'model_shortlist': 10}}
    INFO:20230607-173742: import  - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - this may take some time...
     Preliminary number of subgraphs:1 , 34 nodes delegated out of 34 nodes 
     
    Warning : Requested Output Data Convert Layer is not Added to the network, It is currently not Optimal
    
     ************** Frame index 1 : Running float import ************* 
    ****************************************************
    **                ALL MODEL CHECK PASSED          **
    ****************************************************
    
    The soft limit is 2048
    The hard limit is 2048
    MEM: Init ... !!!
    MEM: Init ... Done !!!
     0.0s:  VX_ZONE_INIT:Enabled
     0.21s:  VX_ZONE_ERROR:Enabled
     0.48s:  VX_ZONE_WARNING:Enabled
     0.2456s:  VX_ZONE_INIT:[tivxInit:184] Initialization Done !!!
    
     ************ Frame index 1 : Running float inference **************** 
    
     ************ Frame index 2 : Running float inference **************** 
    
     ************ Frame index 3 : Running float inference **************** 
    
     ************ Frame index 4 : Running float inference **************** 
    
     ************ Frame index 5 : Running fixed point mode for calibration **************** 
    
    ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
    
    Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt 
     Freeing memory for user provided Net
     ----------------------- TIDL Process with REF_ONLY FLOW ------------------------
    
    #    0 . .. T    1822.48  .... ..... ... .... .....
    #    1 . .. T    1824.50  .... ..... ... .... .....
    #    2 . .. T    1841.23  .... ..... ... .... .....
    #    3 . .. T    1804.06  .... ..... ... .... .....
    #    4 . .. T    1822.88  .... ..... ... .... .....
    ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
    
    Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt 
     Freeing memory for user provided Net
     ----------------------- TIDL Process with REF_ONLY FLOW ------------------------
    
    #    0 . .. T    1622.77  .... ..... ... .... .....
    #    1 . .. T    1624.90  .... ..... ... .... .....
    #    2 . .. T    1569.75  .... ..... ... .... .....
    #    3 . .. T    1561.55  .... ..... ... .... .....
    #    4 . .. T    1617.24  .... ..... ... .... .....
     
     
     *****************   Calibration iteration number 0 completed ************************ 
     
     
     
    
    ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
    
    Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt 
     Freeing memory for user provided Net
     ----------------------- TIDL Process with REF_ONLY FLOW ------------------------
    
    #    0 . .. T    1572.60  .... ..... ... .... .....
    #    1 . .. T    1578.75  .... ..... ... .... .....
    #    2 . .. T    1552.07  .... ..... ... .... .....
    #    3 . .. T    1563.12  .... ..... ... .... .....
    #    4 . .. T    1559.22  .... ..... ... .... .....
     
     
     *****************   Calibration iteration number 1 completed ************************ 
     
     
     
    
    ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
    
    Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt 
     Freeing memory for user provided Net
     ----------------------- TIDL Process with REF_ONLY FLOW ------------------------
    
    #    0 . .. T    1623.64  .... ..... ... .... .....
    #    1 . .. T    1603.79  .... ..... ... .... .....
    #    2 . .. T    1608.56  .... ..... ... .... .....
    #    3 . .. T    1628.29  .... ..... ... .... .....
    #    4 . .. T    1609.37  .... ..... ... .... .....
     
     
     *****************   Calibration iteration number 2 completed ************************ 
     
     
     
    
    ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
    
    Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt 
     Freeing memory for user provided Net
     ----------------------- TIDL Process with REF_ONLY FLOW ------------------------
    
    #    0 . .. T    1562.18  .... ..... ... .... .....
    #    1 . .. T    1567.63  .... ..... ... .... .....
    #    2 . .. T    1552.51  .... ..... ... .... .....
    #    3 . .. T    1611.97  .... ..... ... .... .....
    #    4 . .. T    1550.12  .... ..... ... .... .....
     
     
     *****************   Calibration iteration number 3 completed ************************ 
     
     
     
    
    ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
    
    Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt 
     Freeing memory for user provided Net
     ----------------------- TIDL Process with REF_ONLY FLOW ------------------------
    
    #    0 . .. T    1565.97  .... ..... ... .... .....
    #    1 . .. T    1599.54  .... ..... ... .... .....
    #    2 . .. T    1612.83  .... ..... ... .... .....
    #    3 . .. T    1561.91  .... ..... ... .... .....
    #    4 . .. T    1553.31  .... ..... ... .... .....
     
     
     *****************   Calibration iteration number 4 completed ************************ 
     
     
     
    
    ------------------ Network Compiler Traces -----------------------------
    Error : Error Code = <ERR_UNSUPPORTED_DATA_TYPE>
    Could not open /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_net/perfSimInfo.bin
    Rerunning network compiler for reshape
    
    ------------------ Network Compiler Traces -----------------------------
    successful Memory allocation
    ****************************************************
    **                ALL MODEL CHECK PASSED          **
    ****************************************************
    
    
     Final number of subgraphs:1 , 34 nodes delegated to accelerator 
     
    
    INFO:20230607-173854: import completed  - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - 72 sec
    INFO:20230607-173854: infer  - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - this may take some time...
     Number of subgraphs:1 , 34 nodes delegated out of 34 nodes 
     
    infer : cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_| 100%|##########|| 5/5 [00:01<00:00,  4.76it/s]
    
    INFO:20230607-173856: infer completed  - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - 1 sec
    
    SUCCESS:20230607-173856: benchmark results - {'infer_path': 'cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite', 'accuracy_top1%': 100.0, 'num_subgraphs': 1, 'infer_time_core_ms': 198.201599, 'infer_time_subgraph_ms': 198.192176, 'ddr_transfer_mb': 18446744073709.55, 'perfsim_time_ms': 0.0, 'perfsim_ddr_transfer_mb': 0.0, 'perfsim_gmacs': 0.0}
    MEM: Deinit ... !!!
    MEM: Alloc's: 52 alloc's of 147541958 bytes 
    MEM: Free's : 52 free's  of 147541958 bytes 
    MEM: Open's : 0 allocs  of 0 bytes 
    MEM: Deinit ... Done !!!
    TASKS                                                       | 100%|██████████|| 1/1 [01:27<00:00, 87.60s/it]
    TASKS                                                       | 100%|██████████|| 1/1 [01:27<00:00, 87.60s/it]
    
    -------------------------------------------------------------------
    -------------------------------------------------------------------
    settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/_package', 'datasets_path': './dependencies/datasets', 'target_device': None, 'target_machine': 'pc', 'run_suffix': None, 'parallel_devices': 1, 'tensor_bits': 8, 'runtime_options': None, 'run_import': True, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'cl-0000', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_import_on_pc.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None}
    no results found - no report to generate.
    Report generated at ./work_dirs/modelartifacts/_package
    -------------------------------------------------------------------
    

    Please note that, this logs contains lesser number of iterations as this can differ based on calibration_frames setting.

    Once the model compilation completed you can check the generated model artifacts in edgeai-benchmark/work_dirs/modelartifacts/TDA4VM directory.

    Next step is to package this model artifacts so that they can be used on target for benchmarking.

    Run packaging script as,

    run_package_artifacts_for_evm.sh TDA4VM

    After successful execution of above script a package will be created at edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package directory

    Mount the edgeai-benchmark repo present in PC on the target device use mounted directory for running further commands.

    On target side, we need to install all the dependency packages, available inside requirements_evm.txt file.

    Run below command to install the dependency list as,

    pip3 install -r requirements_evm.txt

    Before running run_benchmarks_evm.sh make sure that modelartifacts_path is set correctly from setting_base.yaml

    Here in case since we have stored compiled model artifacts at TDA4VM_package we can set path as

    modelartifacts_path : './work_dirs/modelartifacts/{target_device}_package'

    lastly, to run the benchmark script on target run below command as,

    ./run_benchmarks_evm.sh TDA4VM

    After execution above script you will see the below listed KPIs.

    'infer_path': 'cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite',

    'accuracy_top1%': 80.0,

    'num_subgraphs': 1,

    'infer_time_core_ms': 1.965482,

    'infer_time_subgraph_ms': 1.950902,

    'ddr_transfer_mb': 6.163737,

    'perfsim_time_ms': 0.0,

    'perfsim_ddr_transfer_mb': 0.0,

    'perfsim_gmacs': 0.0

    Refer below logs for reference.

    root@tda4vm-sk:/opt/mounted-benchmark# ./run_benchmarks_evm.sh TDA4VM
    TARGET_SOC: TDA4VM
    Pass the appropriate commandline argument to use another one.
    TIDL_TOOLS_PATH=/opt/mounted-benchmark/tools/TDA4VM/tidl_tools
    LD_LIBRARY_PATH=/opt/mounted-benchmark/tools/TDA4VM/tidl_tools
    PYTHONPATH=:/usr/lib/python3.8/site-packages/
    ===================================================================
    Please install EdgeAI Linux SDK StarterKit for TDA4VM
    from: https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-SK-TDA4VM
    All the required dependencies are available in the SDK
    For more information, please visit: https://www.ti.com/tool/SK-TDA4VM
    -------------------------------------------------------------------
    ===================================================================
    argv: ['./scripts/benchmark_modelzoo.py', 'settings_infer_on_evm.yaml', '--target_device', 'TDA4VM']
    settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/TDA4VM_package', 'datasets_path': './dependencies/datasets', 'target_device': 'TDA4VM', 'target_machine': 'evm', 'run_suffix': None, 'parallel_devices': None, 'tensor_bits': 8, 'runtime_options': None, 'run_import': False, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'cl-0000', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_infer_on_evm.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None}
    work_dir: ./work_dirs/modelartifacts/TDA4VM_package/8bits
    
    INFO:20230607-130458: dataset exists - will reuse - ./dependencies/datasets/imagenetv2c/val
    
    INFO:20230607-130459: dataset exists - will reuse - ./dependencies/datasets/coco
    loading annotations into memory...
    Done (t=0.76s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.76s)
    creating index...
    index created!
    
    INFO:20230607-130500: dataset exists - will reuse - ./dependencies/datasets/ycbv
    loading annotations into memory...
    Done (t=0.12s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.10s)
    creating index...
    index created!
    
    INFO:20230607-130523: dataset exists - will reuse - ./dependencies/datasets/coco
    loading annotations into memory...
    Done (t=2.36s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=2.39s)
    creating index...
    index created!
    
    INFO:20230607-130533: dataset exists - will reuse - ./dependencies/datasets/widerface
    loading annotations into memory...
    Done (t=1.55s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=1.66s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.40s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.38s)
    creating index...
    index created!
    
    INFO:20230607-130541: dataset exists - will reuse - ./dependencies/datasets/coco
    loading annotations into memory...
    Done (t=2.88s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=1.59s)
    creating index...
    index created!
    
    INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/ADEChallengeData2016
    
    INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/ADEChallengeData2016
    
    INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/VOCdevkit/VOC2012
    
    INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/nyudepthv2
    
    INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/ti-robokit_semseg_zed1hd
    
    INFO:20230607-130552: dataset exists - will reuse - ./dependencies/datasets/ti-robokit_semseg_zed1hd
    download_ok: True
    configs to run: ['cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite']
    number of configs: 1
    
    INFO:20230607-130552: starting - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite
    INFO:20230607-130555: model_path - /opt/edgeai-modelzoo/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite
    INFO:20230607-130555: model_file - /opt/mounted-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/model/mobilenet_v1_1.0_224.tflite
    
    INFO:20230607-130555: running - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite
    INFO:20230607-130555: pipeline_config - {'task_type': 'classification', 'dataset_category': 'imagenet', 'calibration_dataset': <edgeai_benchmark.datasets.imagenetv2.ImageNetV2C object at 0xffff5eae7550>, 'input_dataset': <edgeai_benchmark.datasets.imagenetv2.ImageNetV2C object at 0xffff5eae7610>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0xffff5eae7640>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0xffff5eae76d0>, 'session': <edgeai_benchmark.sessions.tflitert_session.TFLiteRTSession object at 0xffff5eae7880>, 'metric': {'label_offset_pred': -1}, 'model_info': {'metric_reference': {'accuracy_top1%': 71.676}, 'model_shortlist': 10}}
    INFO:20230607-130555: infer  - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - this may take some time...
     Number of subgraphs:1 , 34 nodes delegated out of 34 nodes 
     
    APP: Init ... !!!
    MEM: Init ... !!!
    MEM: Initialized DMA HEAP (fd=6) !!!
    MEM: Init ... Done !!!
    IPC: Init ... !!!
    IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
     21226.775752 s: GTC Frequency = 200 MHz
    APP: Init ... Done !!!
     21226.787708 s:  VX_ZONE_INIT:Enabled
     21226.787741 s:  VX_ZONE_ERROR:Enabled
     21226.788333 s:  VX_ZONE_WARNING:Enabled
     21226.789310 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
     21226.789824 s:  VX_ZONE_INIT:[tivxHostInitLocal:93] Initialization Done for HOST !!!
    infer : cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_| 100%|##########|| 5/5 [00:00<00:00, 19.63it/s]
    
    INFO:20230607-130557: infer completed  - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - 1 sec
    
    SUCCESS:20230607-130557: benchmark results - {'infer_path': 'cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite', 'accuracy_top1%': 80.0, 'num_subgraphs': 1, 'infer_time_core_ms': 1.965482, 'infer_time_subgraph_ms': 1.950902, 'ddr_transfer_mb': 6.163737, 'perfsim_time_ms': 0.0, 'perfsim_ddr_transfer_mb': 0.0, 'perfsim_gmacs': 0.0}
     21227.153310 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:107] De-Initialization Done for HOST !!!
     21227.155829 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
    APP: Deinit ... !!!
    REMOTE_SERVICE: Deinit ... !!!
    REMOTE_SERVICE: Deinit ... Done !!!
    IPC: Deinit ... !!!
    IPC: DeInit ... Done !!!
    MEM: Deinit ... !!!
    DDR_SHARED_MEM: Alloc's: 7 alloc's of 5358322 bytes 
    DDR_SHARED_MEM: Free's : 7 free's  of 5358322 bytes 
    DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
    DDR_SHARED_MEM: Total size: 536870912 bytes 
    MEM: Deinit ... Done !!!
    APP: Deinit ... Done !!!
    -------------------------------------------------------------------
    ===================================================================
    settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/_package', 'datasets_path': './dependencies/datasets', 'target_device': None, 'target_machine': 'evm', 'run_suffix': None, 'parallel_devices': None, 'tensor_bits': 8, 'runtime_options': None, 'run_import': False, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'cl-0000', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_infer_on_evm.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None}
    no results found - no report to generate.
    Report generated at ./work_dirs/modelartifacts/_package
    -------------------------------------------------------------------