Part Number: TDA4VEN-Q1
Dear TI Experts,
We are using edgeai-tidl-tools to compile and quantize our model, which is YOLOv5-nano. After configuring edgeai-tidl-tools/examples/osrt_python/model_configs.py, we used edgeai-tidl-tools/examples/osrt_python/ort/onnx2bin.py to convert the trained ONNX model into a .bin file, and then ran edgeai-tidl-tools/examples/osrt_python/ort/infer_on_pc.py to simulate the quantization results on the PC.
We found that when the model is quantized to INT16, the inference results from infer_on_pc.py are very close to the on-board inference results after deployment on the TDA4 VEN board. However, when quantizing to INT8, or when using mixed INT8 & INT16 quantization, although the inference results from infer_on_pc.py show only some acceptable deviations compared with the original FP32 model, the inference results on the board are completely wrong. In particular, when we adopt mixed INT8 & INT16 quantization, the inference results from infer_on_pc.py are already on par with those of the original FP32 model, yet the on-board inference results are completely distorted.
Is this behavior normal? Why is there such a large discrepancy between the simulated quantization results from infer_on_pc.py on the PC and the quantized inference results on the board?