TDA4VM: Query regarding PTQ

Hi Gina,

     Our benchmarking results are always with the best settings for a given model so we will not have any specific benchmarks for power of 2 quantization. I didn't understand the part about linear quantization. Can you tell us why exactly you need this information and then we can guide you accordingly.

Regards,

Anshu

Hi Anshu,

Linear mode is setting quantizationStyle = 2. We are trying to compare the results from TIDL model for centernet, with linear and power of 2 scale quantized model. We would like to know in general which quantization mode would be performing better.

regards,

Gina

Hi Gina,

     If you are using Processor SDK RTOS 8.0 onwards then for most of the networks quantizationStyle = 2 should work. quantizationStyle = 3 (power of 2 quantization) is mainly required when we use QAT models. 

Regards,
Anshu

Hi Anshu,

We are using PSDKRTOS 7.1 and when we measured the accuracy we observed with one of the centernet model, power of 2 scale quantization is having 10% more accuracy than linear quantization(quantizationStyle = 2). In another centernet model, with quantization style = 2 (linear quantization) recall is only 1%, while for the same model after power of 2 scale quantization we are getting 50% recall. Can I get any inputs for this change in results?

regards,

Gina

Hi Gina,

    Does this model has Relu6 or any other Clip operator?

Regards,

Anshu

Hi Anshu,

Yes Relu6 is used. 

regards,

Gina

Hi Anshu,

Both centernet models using Relu6. Only change is with weights.

regards,

Gina

Hi Gina,

      We had an issue in the SDK which you are using related to Relu6 where in the output was not getting saturated at a value of 6.0. That's why you are seeing variation when you use power of 2 or non-power of 2 quantization. Some time one quantization results better accuracy than other depending on whether output is close to 6.0 or not. The right fix for this was done as part of SDK 8.0 and our recommendation would be to move to the latest SDK which SDK 8.1.

Regards,

Anshu

Hi Anshu,

In this case, can we convert the model to TIDL using SDK 8.1 and then do the inference in TIOVX application in SDK7.1 itself? 

regards,

Gina

Hi Gina,

   No we cannot do this. 

Regards,

Anshu

One option (as you might have guessed) may be to finetune the model with ReLU (instead of ReLU6) and try.

Ok 

Hi Gina,
    Let me know if you still have anything open in this thread, if not can we close it?

Regards,

Anshu

Hi Anshu,

You can close the thread. Thanks for your support. 

regards,

Gina

Hi Anshu,

I have found below option, in the documentation page 

• If fine tuning is not desired solution, we propose replacing ReLU6 with ReLU8 in inference (instead of ReLU). This is automatically done by TIDL

Whether ReLU6 are all replaced to ReLU8 automatically during TIDL Model Import or TIDL Inference, or do we need to change something to enable this?

regards,

Gina

Hi Gina,

    This recommendation is to change ReLU6 to ReLU8 in your original model itself and retrain it. This was a way to get around the ReLU6 bug which I talked about earlier. If you use SDK 8.0 onwards then you need not  update your model as the bug has been fixed now.

Regards,
Anshu

Hi Anshu,

We had checked the accuracy with power of 2 scale and linear quantization in RTOS SDK 8.01 and as you have said, we found that there is not much variation in the results between the two in 8.01 unlike that in 7.01 version. 

regards,

Gina

Gina,
    With SDK 8.1 you should use non-power to 2 quantization ( i.e. quantizationStyle = 2 which is default value if you dont set it ).

Regards,
Anshu

Hi Anshu,

Ok. We also tried to quantize our model to INT16 and computed the metrics. When checked, we didnt find any deviation in results between power of 2 and non power of 2 scale in 7.01 SDK. Considering your reply, I thought that due to SDK bug in 7.01 SDK, even in INT16 results will too deviate between quantization style. Is my understanding correct? Or this behavior is only for INT8 quantization.

regards,

Gina

Hi Gina,

    Whether results will deviate or not will depend on the statistics, for some cases deviation may not be large and hence gives decent results with 16 bit. But its not guaranteed, so because of the mentioned bug even 16 bit can give poor accuracy.

Regards,

Anshu