TIDL Inference Int8 preprocessing error

Hi,

Can you please refer to our detailed documentation on accuracy debugs here : https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/docs/tidl_osr_debug.md

We have created layer level debug specific notebook you might want to check the same here : https://github.com/kumardesappan/colab-notebooks/blob/main/ti_edgeai_tidl_tools.ipynb

yes i have referred the doccumentation and extracted the plots above can u please look into the clearly and let me knw for int8 inference should we change the input or preprocessing modules 

To understand the issue better can you able to share the jupyter notebook which has layer level plots so I can analyze the same at my end.

You can share the notebook of same format we have shared above.

Thanks 

Hi Abhilash,

Is it possible for you to send the import/infer configs that you are using? As you mentioned there is something wrong with the input data layers itself, we can look at the inElementTypes that you are setting.

Thanks,

Sai

Also, I see you have shared multiple plots, these plots belong to which layers?

Hello Saideepak please find the below response 

• The config files for both import and infe
  Fullscreen tidl_infer_config.txt Download

  inFileFormat    = 1
  numFrames   = 1
  netBinFile      = "~/TI/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/ti_dl/test/testvecs/config/tidl_models/onnx/customer_mode_v5.bin"
  ioConfigFile   = "~/TI/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/ti_dl/test/testvecs/config/tidl_models/onnx/customer_mode_v5_1.bin"
  inFileFormat = 1 
  rawDataInElementType = 6
  inElementType = 2
  inData = ~/TI/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/lucid_laneDetection/inputs/Tidl_input.raw
  writeOutput = 2
  outData =   ~/TI/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/tidlInputOutputs/outputs/sample_output.bin
  writeTraceLevel = 3
  

  rence is also attached 
  Fullscreen 8737.tidl_import_config.txt Download

  modelType          = 2
  numParamBits       = 8
  inputNetFile      = "~/Documents/onnxmodel/customer_model_v6.onnx"
  outputNetFile      = "~/TI/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/ti_dl/test/testvecs/config/tidl_models/onnx/customer_mode_v5.bin"
  outputParamsFile   = "~/TI/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/ti_dl/test/testvecs/config/tidl_models/onnx/customer_mode_v5_"
  inDataNamesList = image,input_471,Gather_207_input,Mul_199_input,ones_input
  #outDataNamesList = seeds,output_dict_list,523,587,599
  numBatches=1
  inNumChannels = 3 2 1 1 1
  inHeight =  320 40 1 1 66
  inWidth  = 800 100 1 134 1
  inFileFormat = 1 
  inData = ~/TI/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/tidlInputOutputs/rawInputs/test_input3_tidl.raw
  
  

• Our model input is image data and additional data to the layer operators  which is present in the .raw file which is shared 
• The shared multiple plots in this forum is wrt to the input layers comparison 
• test_input3_tidl.raw.txt

Hi,

If you are planning to run the network in 8-bit, you should set the inElementType to 0/1 in the import config. Following is the list of data types that TIDL supports,

If the data present in the inData file is in float, and you are planning to run the network in 8-bit, please set the following parameter in both import/infer files

rawDataInElementType = 6

inElementType = 0/1

You can set the writeTraceLevel flag to enable the layer level traces, I suggest you first make sure the input is correctly passed on to the network. If you set this flag to a higher level (2/3), it should dump float traces (fixed point converted to float), you can now compare the input trace to the actual input file.

Let me know if you face any issues.

Thanks,

Sai

I have followed the steps suggested and i am still facing the same issues with the input , i have added all the plots here for more analysis 

comparision_outs.zip.txt

modelType          = 2
numParamBits       = 32
numFrames = 2
inputNetFile      = "/home//TI_SDK/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/ti_dl/test/testvecs/models/public/onnx/customer_model_v6.onnx"
outputNetFile      = "/home//TI_SDK/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/ti_dl/test/testvecs/config/tidl_models/onnx/customer_mode_v6_fp32.bin"
outputParamsFile   = "/home//TI_SDK/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/ti_dl/test/testvecs/config/tidl_models/onnx/customer_mode_v6_fp32"
inDataNamesList = image,input_471,Gather_207_input,Mul_199_input,ones_input
#outDataNamesList = seeds,output_dict_list,523,587,599
numBatches=1
inNumChannels = 3 2 1 1 1
inHeight =  320 40 1 1 66
inWidth  = 800 100 1 134 1
inFileFormat = 1 
inData = /home//TI_SDK/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/tidlInputOutputs/rawInputs/test_input_2_3_tidl.raw

modelType          = 2
numParamBits       = 8
numFrames = 2
inputNetFile      = "/home//TI_SDK/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/ti_dl/test/testvecs/models/public/onnx/customer_model_v6.onnx"
outputNetFile      = "/home//TI_SDK/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/ti_dl/test/testvecs/config/tidl_models/onnx/customer_mode_v6_int8.bin"
outputParamsFile   = "/home//TI_SDK/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/ti_dl/test/testvecs/config/tidl_models/onnx/customer_mode_v6_int8_"
inDataNamesList = image,input_471,Gather_207_input,Mul_199_input,ones_input
#outDataNamesList = seeds,output_dict_list,523,587,599
numBatches=1
inNumChannels = 3 2 1 1 1
inHeight =  320 40 1 1 66
inWidth  = 800 100 1 134 1
inFileFormat = 1 
inData = /home//TI_SDK/ti-processor-sdk-rtos-j721e-evm-09_00_00_02/c7x-mma-tidl/tidlInputOutputs/rawInputs/test_input_2_3_tidl.raw
rawDataInElementType = 6
inElementType = 0

Hi Abhilash,

Can you also share the 8-bit svg (last one you shared is float svg) and toy model if it's ready. I can look at them before call so that we can debug better.

Thanks,

Sai

customer_mode_v6_.zip

Please find both the svg files for fp32 and int8 

It is not clear from the thread that which version are you using.

can you confirm below

1. Are you using the latest tool chain: 09_01_06_00

https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/docs/version_compatibility_table.md

2. Are you able to get good accuracy in INT8 mode with a known good reference run time such as ONNX run time using PTQ provided by them. see below

https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html#static-quantization

Many times we have seen that network is not trained with good regularization and results in poor accuracy with INT8 mode so request you to confirm that by using external environment mentioned above

Thanks,

with Regards,

Pramod

THe sdk version is 9.0 , i am not getting the any output from the int8 inference only 

I am not clear on what you meant by not getting any output with int8. If that is the case, then how did you get the mentioned plots in the first part of the thread. I will suggest to do below 

1. Refer the documentation https://github.com/TexasInstruments/edgeai-tidl-tools/

2. Use 9.1 SDK and latest tool chain (09_01_06_00) available in above repo 

3. Make sure that you are able to run OOB examples first

4. Then try your model

5. If you face issue then please enable debug_level and share the traces, dumps and comparison plots

Thanks

Pramod

I also learnt that you are looking for an example

I strongly suggest that you please go through the documentation at below link

https://github.com/TexasInstruments/edgeai-tidl-tools

There are several examples available for you to try out.

https://github.com/TexasInstruments/edgeai-tidl-tools/tree/master/examples/osrt_python

There are also google colab notebook for you to try out

https://github.com/TexasInstruments/edgeai-tidl-tools/tree/master/examples/jupyter_notebooks/colab

Refer below python file and you can find the dictionary of models, you can change with your model and try out 

 https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/examples/osrt_python/ort/onnxrt_ep.py

Thanks,
with Regards,

pramod