TDA4VM: ONNX Export issue for QATv1

Can you elaborate the issue in details ? especially what are you comparing against what ?

Thanks 

Hi Pratik,

This is a Object detection model trained on a multiclass dataset using QAT.

I have infered the above pytorch model on dataset and calculated F1 score. ---- A

Now, I exported the pytorch model file in ONNX format. And calculated the F1 score again using this ONNX file on the same dataset. -----B

There is a substantial performance drop between from A to B. So, I wanted to know what could be the possible reasons for this.

Thanks,

Adarsh

Hi Adarsh,

QAT adds the quantizer and dequantizer layers this will add the precision loss and our expectation is n/w should learn to accommodate that and tune the n/w weights based on it, so that trained n/w will be good for quantization, accuracy drop is expected but on what scale for that let me add algo expert in this conversation to reply

Thanks 

Wen you are exporting an onnx model trained with QATv1 from edgeaimodeloptimization, that QAT onnx model will not give good accuracy when running with plain onnxruntime. That model is exported specifically for TIDL. Hence you have to use it with onnxruntime-tidl.

Hi Manu,

I am using TI SDK 8.02.

Can you please share the related link to run the onnx model using onnxruntime-tidl?

Thanks,

Adarsh

edgeai-tidl-tools can be used for model compilation: https://github.com/TexasInstruments/edgeai-tidl-tools

You can use an appropriate tag or branch corresponding to that release: https://github.com/TexasInstruments/edgeai-tidl-tools/releases?page=3

For example this tag is listed in the above page: 08_02_00_05

Hi Manu,

When using edgeai-tidl-tools, I first run python3 onnxrt_ep.py - d and output the model results in arm mode. This output was same which I got though plain onnxruntime inference earlier.

Then, I run python3 onnxrt_ep. py - c again and output the model results using tidlnode. The outputs coming are incorrect.

Can you please point out possible reasons for this?

Thanks and Regards,

Adarsh Kumar

"The outputs coming are incorrect."

What exactly does this mean? Which outputs? In PC or in EVM? 

But here are some debugging tips when you get incorrect output with edgeai-tidl-tools:

1. set tensor_bits to 32 and check the output in PC. If the output is correct, then proceed to next step

2. set tensor_bits to 16 and check the output in PC.. If the output is correct, then proceed to next step

3. set tensor_bits to 8 and check the output in PC.. If the output is correct, then proceed to next step

4. Run the inference in EVM and check if the output matches with that of PC.

Hi Manu,

1. I am inferencing in PC Emulation Mode and have set tensor_bits to 32.

2. I am dumping the sess.run() results in bin files and running custom postprocessing script over it.

3. I am getting the same postprocessing output which I got through plain onnxruntime inference earlier by running 'python3 onnxrt_ep.py -d'.
But getting no bounding boxes after postprocessing when running 'python3 onnxrt_ep.py -c' and output the model results using tidlnode.

4. The postprocessing code is same in both the cases mentioned in '3'.

Regards,

Adarsh

I think what you are saying is: arm mode(python3 onnxrt_ep.py -d) and tidl mode(python3 onnxrt_ep.py -c and then python3 onnxrt_ep.py) generates different output. This is happening on a QATv1 model. I hope I captured your observation right.

This is as per our expectation.

QATv1 model is for TIDL - not for plain onnxruntime or for arm mode. It will only produce expected output in TIDL. Also if the QATv1 training was done for 8bits (default settings), it will only produce expected output with tensor_bits=8 

Hi Manu,

Thank you for the quick replies.

Manu Mathew said:

I hope I captured your observation right.

Yes you got my issue right.

Manu Mathew said:

This is as per our expectation.

The QATv1 training was done for 8 bits. I have tested in tidl mode with tensor_bits=8, but still getting no postprocessing output in my case which was not as exxpected. 

(Note: I am using default parameters in tidl  mode.)

What could be the possible reasons for this?

Regards,

Adarsh

Can you try one of the QATv1 models hat we have in the modelzoo,

perhaps this one: https://github.com/TexasInstruments/edgeai-tensorlab/blob/main/edgeai-modelzoo/models/vision/classification/imagenet1k/edgeai-tv/mobilenet_v2_qat-p2_20201213.onnx.link

The compilation parameters that we used is here: 

https://github.com/TexasInstruments/edgeai-tensorlab/blob/main/edgeai-modelzoo/models/vision/classification/imagenet1k/edgeai-tv/mobilenet_v2_qat-p2_20201213_config.yaml

The most important part is for QATv1 models is, the compilation options should have: 
  tensor_bits: 8
  accuracy_level: 0

Hi Manu,

1.

Manu Mathew said:

Can you try one of the QATv1 models hat we have in the modelzoo,

I have tried this model as provided by you and it's working fine. Also, I have found my mistake while running in TIDL mode. So, now I am able to get outputs same as arm mode.

2. 

Manu Mathew said:

QAT onnx model will not give good accuracy when running with plain onnxruntime. That model is exported specifically for TIDL. Hence you have to use it with onnxruntime-tidl

Coming to the initial issue of performance drop from pth file to onnx file. You have quoted the above line for that.
But, now even after inferencing in TIDL mode, I am still getting a performance drop of approx. 30% in my case.

Is this much drop expected in some cases or is there still something which we can do to improve the performance in tidl mode?
Also, if there some compilation parameters that you have used for better performance in Object detection?

Regards,

Adarsh

Accuracy measurement of detection model is a complicated topic. Parameters such as top_k and confidence_treshold have to be tuned properly. Also mixed precision may have to be used. 

Are you using any prototxt to compile the model for TIDL? 

Can you see the following thread that discussed a similar topic: e2e.ti.com/.../sk-am68-how-to-change-trainning-and-compilation-parameters-in-model-maker