• State TI Thinks Resolved
  • Locked Locked
  • Replies 4 replies
  • Answers 1 answer
  • Subscribers 99 subscribers
  • Views 534 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: Questions about official pre-trained models.

Hao Liu
Prodigy 10 points
Part Number: TDA4VM

Hello,

I'm testing object detection models on EdgeAI Cloud but just found a strange issue.

As illustrated in the page of Model selection tool, the model "ONR-OD-8100-yolov5-s6-ti-lite-coco-640x640" got 37.4AP on float and 35.6AP on int8.

However, the run log "prebuilt-models/8bits/od-8100_onnxrt_weights_yolov5s6_640_ti_lite_37p4_56p0_onnx/run.log" shows that the model get only 32.122AP.

Why are the results not consistent???

  • 0
    TI__Intellectual 2406 points

    Hi,

     If you open the corresponding prototxt file,you can see that the confidence_threshold is set to 0.05 whereas top_k and keep_top_k are set to 500 and 200 respectively. For any practical deployment and while measuring latency, this is a suitable value.

      However, while measuring accuracy, we need to reduce the threshold to a really low value like 0.005. As in the official repo, top_k and keep_top_k has to be set to 30000 and 300 respectively. With this threshold, you can recreate the reported accuracy. This will however require more time for the evaluation to complete.

       I will rerun to cross-check the accuracy once again and confirm shortly.

    Regards, Debapriya

  • 0 in reply to Debapriya Maji
    TI__Intellectual 2406 points

    Hi,

    You can reproduce the accuracy with confidence_threshold=0.005 and top_k=30000 and keep_top_k=300.

    Regards, Debapriya

  • 0 in reply to Debapriya Maji
    Prodigy 10 points

    Hi,

    Thanks a lot for your reply.

    Yes, the threshold and top_k influent the measuring accuracy. However, the influence should not be such huge (>5 AP gap).

    I had attempt to reproduce the measuring accuracy by using edgeAI cloud, but the reserving time of EVM is not enough for running.

    Could you please also upload the run log with settings (confidence_threshold=0.005 and top_k=30000 and keep_top_k=300)

    Thanks!

    Regards,

    Hao

  • 0 in reply to Hao Liu
    TI__Intellectual 2406 points

    Hi Hao,

    PFA log for  top_k=30000 and keep_top_k=300. In8 AP for the latest release is 36.3.


    7853.run.log