SK-AM62A-LP: import pre-quant tflite model

Hello Jin Peng,

Thank you for your question and the tools configuration.

This model is already quantized. There is a similar thread: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1232355/processor-sdk-j721s2-how-to-import-pre-quantized-network-tflite-cnn-models-using-tidl-without-calibration

Can you try this and let me know if this solves the problem you are seeing?

Here is another doc on quantization: https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/docs/tidl_fsg_quantization.md

Jin Peng said:

Does tidl support asymmetric quantization?

- No, not currently. We intend to enable this in the future, but I would need to contact our development team to understand their plan/timeline to enable this feature.

The thread uses "TIDL OSRT" to import model.

I wish to accelerate my models through C7x and MMA to free CPU.

"TIDL OSRT" is an API deployed on CPU, not with C7x and MMA. Is that right?

Jin Peng said:

"TIDL OSRT" is an API deployed on CPU, not with C7x and MMA. Is that right?

Correct, we have extended open source runtimes (OSRT) like tensorflow lite, onnx runtime, and TVM/DLR to use their APIs for invoking the C7xMMA for accelerating model inference. It adds a few lines of code for configuring options that the C7xMMA needs to run the model, but otherwise uses the APIs of those runtimes.

The API runs from CPU, transfers data to the C7xMMA using TI's OpenVX implementation[1], executes the exported/compiled model on the C7xMMA, transfers output tensors back to the CPU, and returns that data from the OSRT API call. These APIs help abstract those OpenVX transfers. We also use gstreamer plugins for a similar form of abstraction when invoking other IP like ISP and video encoder/decoder [2] (preinstalled in edgeai SDK linux image).

[1] https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/latest/exports/docs/tiovx/docs/user_guide/index.html  

[2] https://github.com/TexasInstruments/edgeai-gst-plugins

Best,
Reese

Hi Jin Peng,

Quick update on asymmetric quantization:

For TFLite, asymmetric quantization done within tensorflow/tflite will hold for the compiled model. As per the linked e2e above, make sure to set the advanced_options:quantization_scale_type option to 3 

The next SDK release (9.0) will have support for asymmetric quantization using TIDL, meaning it is not required to quantize the model before compiling it with TIDL tools. This is aimed at enabling more quantization aware training (QAT)

Hi,

I am using TIDL-RT(SDK 08_06_00_00), not with edgeai-tidl-tools, and I am using tidl_tools to convert tflite model to TIDL model.

Could I import tflite pre-quant model using config which mentioned at beginning? I think the config "advanced_options:quantization_scale_type" is same as "quantizationStyle=5"

Hi Jin Peng,

quantizationStyle should not be necessary for what you're trying to do here, but the configuration should be fine otherwise (assuming you add advanced_options:quantization_scale_type=3). Those two quantization options are not the same.

Technically, tidl_tools is all you need. The edgeai-tidl-tools just provides scripts and examples to download tidl_tools, setup the environment, and compile several models.

Best,
Reese