TDA4VM: why model output is worry when using 8bits quantization in edgeai-tidl-tools, while 16-bits is correct?

Chris

Part Number: TDA4VM

Dear author,

When I tried to use OSRT to compile Superpoint model (please find this network in https://github.com/eric-yyjau/pytorch-superpoint), I always got worry output when seting number of bits for quantization as 8 bits.

But When number of bits for quantization is 16bits, the output is correct, please what is the reason for this? How can I got correct results by using 8 bits quantization?

over 3 years ago

0 Anand Pathak over 3 years ago

TI__Genius 9065 points

Hi,

16 bit working and 8 bit not giving good results is an indication that there is quantization loss in the model and it requires advanced calibration.

You need to set "accuracy_level" to 1 (https://github.com/TexasInstruments/edgeai-tidl-tools/blob/3dc359873f0b80c1e1a0b4eec5d1d02e35d4e532/examples/osrt_python/common_utils.py#L36) and we recommend using 50 calibration_frames and 50 calibration_iterations from the dataset you are using to get good accuracy in such scenarios.

edgeai-tidl-tools (https://github.com/TexasInstruments/edgeai-tidl-tools) repo is intended to give examples of how models work using OSRT and for quick prototyping and debugging.

You can also checkout edgeai-benchmark repo (https://github.com/TexasInstruments/edgeai-benchmark) which has more tools on top of edgeai-tidl-tools repo to ease the task of compilation and benchmarking on larger datasets.

Feel free to continue thread in case you face any issues in above.

Regards,

Anand

0 Chris over 3 years ago in reply to Anand Pathak

Prodigy 30 points

Hi Anand,

Thanks for your reply! I have set the calibration_frames and calibration_iterations as 50 in advanced_options, "accuracy_level" to 1, but I still got the worry output. I have tried to quantize model with 50 calibration images, but still the got worry output for 8bit. Please is there any other solution to this problem?

Many thanks,

Chris

0 Anand Pathak over 3 years ago in reply to Chris

TI__Genius 9065 points

Hi Chris,

Just want to confirm : Did you make sure to use 50 different images for calibration? If you use edgeai-tidl-tools, it may re-use the base image present in scripts and not yield good results. You need to do some changes to scripts to make sure all images are used.

Regards,

Anand

0 Chris over 3 years ago in reply to Anand Pathak

Prodigy 30 points

Hi Anand,

Yes, I have tried to calibrate the model with 50 different images in edgeai-tidl-tools (the specific osrt used here is tvm-dlr). when compiling the model, the terminal did show that there were 50 diffrent images calibrated with calibration iteration set as 50. You can see my config setting as below:

Regards,

Chris

0 Anand Pathak over 3 years ago in reply to Chris

TI__Genius 9065 points

Hi Chris,

Ok, that means some layers need higher precision and are not able to work well with 8 bit quantization. The next step would be to try out the mixed precision feature, which enables you to specify and run selected layers in 16 bit precision and the remaining ones in 8 bit precision.

You can refer to these options : https://github.com/TexasInstruments/edgeai-tidl-tools/blob/3dc359873f0b80c1e1a0b4eec5d1d02e35d4e532/examples/osrt_python/common_utils.py#L69

Since TIDL importer merges certain layers, this list should correspond to the last layer after merging. You can find this mapping by compiling the model once without this parameter which generates ***.layer_info.txt file in the artifacts folder This *.layer_info.txt file contains three columns the first one is the layer number, second one is unique data id and last one is the name as given in the original network's model format. This third column gives the name which will be present in the TIDL imported network after merging any layer. User should use this value for "advanced_options:output_feature_16bit_names_list".

Regards,

Anand

Processors

Processors forum

TDA4VM: why model output is worry when using 8bits quantization in edgeai-tidl-tools, while 16-bits is correct?