TDA4VM: How to instruct TIDL to run particular layer with 16 bits accuracy

While we are waiting for Anshu to respond, please have a look at the some of the example usages of advanced_options:output_feature_16bit_names_list

in these set of configs: https://github.com/TexasInstruments/edgeai-benchmark/blob/master/configs/detection.py

The corresponding models are here: https://github.com/TexasInstruments/edgeai-modelzoo/tree/master/models/vision/detection/coco

and 

here: https://github.com/TexasInstruments/edgeai-yolov5/tree/master/pretrained_models/models/detection/coco/

>>>The documentation instructs to set the advanced_options:output_feature_16bit_names_list with the layer names of the original model.

This should be the layer names in the onnx model that you are giving to TIDL.

Yet another details is that in case TIDL is merging layers, TIDL expects the last of those merged layers' names to be given in advanced_options:output_feature_16bit_names_list

So if a convolution and subsequent clip layer is merged by TIDL, then the name of the clip layer has to be populated in advanced_options:output_feature_16bit_names_list

Since you are able to use mixed precision now, do you still need to use QAT? Why not just try PTQ with mixed precision?

A heuristic setting for mixed precision layer selection that works quite often is to keep the first and last convolution/fully-connected layers in 16-bits.

Mostly because our pipeline consists of 2 model with additional logic between them.  From my previous experience with the system with Nvidia quantization tools, the PQT wasn't enough to achieve desired accuracy. the first model is based on MobileNetV2 and a simple PQT causes very big drop in it.

I'm achieving acceptable accuracy using QAT, but as you can see, we are facing accuracy issues between the torchvision and TIDL results already at the output of the first model. This is also true with 16 bit QAT (all layers quantized with 16 bit weights and activations).

The first model outputs the predicted depth in the input image. I'm observing errors up to 20 meters with 8-bit quantization and up to 4m with 16-bit. Pretty much unacceptable.

I also did the experiment with the toy model without the input Clip as you suggested feeding it with values between 0-255. The results are strange:

The output Clip has range [0-256], leading to scale 1, the expected output should be exactly the same as the input.

8-bit artifacts:

Input in range [0-63] - input and output match exactly

Input in range [64-127]- difference of 1 between the input and output

Input in range [128-255] - the output is zero.

16-bit artifacts:

Errors up to 1e-2, which are also quite big for a single convolution experiment. 

>>Input in range [128-255] - the output is zero.

I think it is about what data format TIDL has assumed while converting the input to integer. I should have suggested you try with input of -128 to 127 (since most models have a signed input, I am assuming TIDL assumes int8 input and not uint8 input). Also, you can check the input data type assumed by TIDL.

Hi Manu,

No, the TIDL expects the data to be uint8, here is the details. The negative input is clipped to zero.

Starting Layer # - 1
0 1.00000 0.00000 255.00000 1
Processing Layer # - 1
1 1.00000 0.00000 256.00000 0

Starting Layer # - 1

0 1.00000 0.00000 255.00000 1

As per the TIDL documentation (below), the highlighted 1 indicates signed char (int8). (Ignore the range 0 - 255 in the above line as it is not computed value, but just default value - only for the input layer)

So -128 to 127 input should work - please train once again with this input and try it.

software-dl.ti.com/.../itidl__ti_8h.html

TIDL Element Type

This group defines the different types of element type used by TIDL library

#define TIDL_UnsignedChar ((int32_t) 0)

#define TIDL_SignedChar ((int32_t) 1)

#define TIDL_UnsignedShort ((int32_t) 2)

#define TIDL_SignedShort ((int32_t) 3)

#define TIDL_UnsignedWord ((int32_t) 4)

#define TIDL_SignedWord ((int32_t) 5)

#define TIDL_SinglePrecFloat ((int32_t) 6)

#define TIDL_UnsignedDoubleWord ((int32_t) 7)

#define TIDL_SignedDoubleWord ((int32_t) 8)

In order for TIDL to use uint8 for input type, the type of onnx input layer has to be changed to uint8

An example where we have done it is here: https://github.com/TexasInstruments/edgeai-modelzoo/blob/master/modelartifacts/8bits/cl-6070_onnxrt_imagenet1k_edgeai-tv_mobilenet_v2_20191224_onnx.tar.gz.link

This was done by using: https://github.com/TexasInstruments/edgeai-tidl-tools/tree/master/scripts/osrt_model_tools

which is used here: https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/examples/osrt_python/common_utils.py#L196

and here: https://github.com/TexasInstruments/edgeai-benchmark/blob/master/edgeai_benchmark/sessions/basert_session.py#L444

But this script also brings the mean/scale inside the model - so do not use this script if you do not want to apply that mean/scale - (using mean/scale introduces that approximation that I mentioned earlier). I am just showing an example of how you can use unsinged input type.

Hi Manu,

I did some experiments with what you mentioned above - I added Cast to FLOAT and Mul l(1/255) layers to my toy example. The graph looks as shown below: 

I would expect the TIDL inference to support the whole UINT8 range now, but it still shows as the input is signed and clips all output values bigger than 127.

Another test was insert directly the input into the convolution without using any input Clip as you suggested it uses some hardware related computations 

which might lead to those differences I'm observing. The graph now looks as below:

Now I would expect the input and output to be exactly the same as both convolutional layer and the output clip (scale of 1) should be "transparent" in terms of computations. However, the output graph of TIDL result (blue) vs. expected result (orange) looks as below:

As you can see the values in range of [-64:64]  are similar, however in ranges [-127:-65] there is a constant error of 1 and in range [65:127] a constant error of -1.

Let us discuss more tomorrow - but need to understand what you mean by "expected".

I think you are using a weight value of 1.0

We need to think how can we represent 1.0 in signed 8-bit fixed point with values from -128 to 127 - with power-of-2 scale constraints.

scale of 64 and shift of 6 is a possibility - but then we are not using the full 8bit range. 

scale of 128 and shift of 7 (with saturation between -128 to 127) is another possibility - but then 1.0 is will be represented as 127/128 which is an approximation.