TDA4VM: After QAT, the Accuracy drop is still more than 10%

That's interesting. Although you said QAT document was followed, I still would like to ask all the usual questions - to be sure:

1. Is everything being used in the module form and not functional form in PyTorch code? For example are you using torch.nn.ReLU() - make sure torch.nn.functional.relu is not used.

2. Is there any repetition of same instances - i.e. let's say you have self.relu = nn.ReLU(). Are you using that self.relu more than once? - that's not allowed.

3. Is there anything else that you are using in functional form instead of module form?

4. What was the weight decay used in the original floating point training? Prefer 1e-4. At least 4e-5 is required. Are all the parameters regularized?

5. What was the learning rate used for QAT? 1e-5 or 5e-5 are good.

6. When you test on TIDL, what are the values of calibrationOption (should be 0) and quantizationStyle (should be 3)

Best regards,

Manu

1. I am sure that we haven't use any torch.nn.functional module

2. Our backbone is mobilenetv2+DLAseg, I follow the mobilenetv2 in your example here. For DLAseg part I will check if there are any repetition of same instances

https://git.ti.com/cgit/jacinto-ai/pytorch-jacinto-ai-devkit/tree/modules/pytorch_jacinto_ai/xvision/models/mobilenetv2.py

3. The weight decay is 1e-4 and learning rate is 1e-5

4. The main problem is when test on TIDL, the mAP drop to 0.234 from 0.317. I'm sure that I have followed the guide and set calibrationOption=0 and quantizationStyle=3. Any other advise? In addition, our model has 7 outputs. I wonder if there are some problem when outputs are more than one.

Is there is a public source code that you are using as reference to this DLASeg? If so can you share the link? Or do you think you can share the onnx model (only if it is okay).

Just to understand more - without quantization on the training framework, what is the accuracy for the float model on Pytorch? Also what is the accuracy on TIDL with the float model?

Hi, for the DLASeg part you can refer to the git link below in line 533.

https://github.com/xingyizhou/CenterNet/blob/master/src/lib/models/networks/dlav0.py

And the mAP is 0.437 for the float model on Pytorch, the accuracy is also 0.437 on TIDL with the float model and 0.416 with the int16 model. But the accuracy drop a lot on int8 model, so we do QAT to improve the accuracy.

Is there a model & checkpoint in that repository that would have the issue that you are reporting? Can you share the links? I would like to try it out.

Can I have your email address and I will share the onnx model.

Please send it to: jacinto-ai-devkit@list.ti.com

Please share both float and qat onnx models.

Also if you can point to a public domain PyTorch model and checkpoint (if it is publicly available - if not exactly same, but similar), that will be useful too.

Hi the public domain PyTorch model is in the link below

https://drive.google.com/file/d/1pl_-ael8wERdUREEnaIfqOV_VF2bEVRT/view

https://github.com/xingyizhou/CenterNet

And the model link is

https://github.com/xingyizhou/CenterNet/blob/master/src/lib/models/networks/dlav0.py

In this public domain, they use ResNet and we use MobilenetV2 instead. As for the dla part we use similar code.

Thank you. I shall take a look.

I have not tried quantizing CenterNet - so not sure exactly what is going on. But I do know that the peaks of the confidence score is extremely important for CenterNet as it doesn't use the common Box based NMS, but instead relies solely on the peaks of the confidence score.

I tried the source code that you shared - but it is for PyTorch 0.4 and I had quite some struggle to get it working. Still not able to do ONNX export. Do you have an ONNX model that I can look at? You can send it to the email id that I shared above if you don't want to post it in the forum.