Part Number: SK-TDA4VM
1. situation
I download ssd_mobilenet_v1_coco_2018_01_28 tflite model(link) from ti model zoo and try to compile it following the official guide notebook(link).
then , I can get though the whole process and come up with a packaged model with artifacts.
However, mAP generated by Ti edgeai benchmark even in your original official notebook(In 9) mentioned above is quite low:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.028 (this is too low and should be about 0.23 )
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.057
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.027
2. replacement
As this model is from Ti modelzoo, I also follow your guide of Compile models in the model zoo to run the Ti edgeai benchmark script below to compile the same ssd_mobilenet_v1_2018 tflite model and get the right mAP of 20.
https://github.com/TexasInstruments/edgeai-benchmark/blob/master/run_benchmarks_pc.sh
Benchmark result:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.204
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.329
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.226
SUCCESS:20220314-102614: benchmark results - {'infer_path': 'od-2000_tflitert_mlperf_ssd_mobilenet_v1_coco_20180128_tflite', 'accuracy_ap[.5:.95]%': 20.38576, 'accuracy_ap50%': 32.880828, 'num_subgraphs': 0, 'infer_time_core_ms': 347.285729, 'infer_time_subgraph_ms': 0.0, 'ddr_transfer_mb': 0.0, 'perfsim_time_ms': 0.0, 'perfsim_ddr_transfer_mb': 0.0, 'perfsim_gmacs': 0.0}
3. question
I test the compiled model created by official notebook on tda4 sk and hardly detect any object and could get good detect result created by run_benchmarks_pc.sh
I guess there must be wrong settings in your guide notebook and please tell me how to compile ssd_mobilenet_v1 to get acceptable mAP, as I might use other models which are not from TI' s modelzoo and have to be compiled in this guide notebook.
Thank you in advance.
