TDA4VM: Query regarding multiple detections

Hi Gina,
     Please follow the steps mentioned in below documentation and let us know if you can identify the layer which is causing the issue : 

https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/08_01_00_11/exports/docs/tidl_j7_08_01_00_05/ti_dl/docs/user_guide_html/md_tidl_fsg_steps_to_debug_mismatch.html

Regards,

Anshu

Hi Anshu,

Ok. We checked on the points from the url you shared, but couldn't find the layer which is causing the multiple detection issue.

But when we checked paramDebug.csv file, it is seen that in some of the initial convolution layers, orgmax is much higher than the meanOrigFloat. 

We also tried inference using INT16 quantized model. 

We find that with INT16 and INT8 model, orgmax has almost the same value and it is higher than meanOrigFloat. 

Does this mean that quantization loss is not reduced with INT16? We found that multiple detection is less in INT16 than INT8.  Also mAP value is improved in INT16 compared to INT8. 

regards,

Gina

Hi Gina,

     Can you confirm if you are using advanced calibration for 8 bit quantization ( i.e. calibrationOption = 7) with 8 bit quantization. Recommendation is to use 50 images with this option.

Regards,

Anshu

Hi Anshu,

Yes advanced calibration was used with 50 images.

regards,

Gina

Hi Gina,

  As per from what I understand so far is float accuracy is as per expectation, 16-bit accuracy is also close to floating point but 8 bit accuracy is not as per expectation. This can happen because of following reason: 

1. Genuine quantization issue where there is accuracy drop because of quantization. For this our recommendation will be to use mixed precision. Details for the same can be find here (section A.4): 

https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/08_01_00_11/exports/docs/tidl_j7_08_01_00_05/ti_dl/docs/user_guide_html/md_tidl_fsg_quantization.html#did_tidl_quantization_1

2. Functional issue with 8 bit, for this there is no easy way but to compare the floating point traces and 8 bit traces (de-quantized traces). You can refer the following documentation for some sample script for such comparison : 

https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/08_01_00_11/exports/docs/tidl_j7_08_01_00_05/ti_dl/docs/user_guide_html/md_tidl_fsg_steps_to_debug_mismatch.html#did_tidl_debug_fm_quant

Regards,

Anshu

Hi Gina, is this issue resolved?

Hi Anand,

No this issue is not resolved.

Precision and recall is improved with INT16 model. But multiple detection issue is still there in INT16, although not as high as INT8 model. Also we cant try mixed precision as we understand mixed precision not supported in SDK v7.01. And also we understand mixed precision doesn't serve our purpose as both INT8 and INT16 has multiple detections at the output. 

As multiple detection occur in some images after quantization which is is not there in original pytorch model, customer is also concerned about it. 

Can you help why multiple detection occurs after quantization?

regards,

Gina

Hi Gina,

Can you try running the model on latest released SDK 8.2 and check if the issue still exists there? The SDK version you are using is quite old, and it's possible there have been some fixes which already resolve this issue.

Regards,

Anand