TDA4VM: Guidelines on maximizing accuracy from calibration.

Hi Akshat,

Could you share your model and the full import configuration you are using? 

Akshat Dogra said:

the results are best when I am using 50 images following by 10

Just to make sure I understood, did you calibrate with 50 images, and then ran inference on 10? Also are the images you are using to inference found in the images you use to calibrate, or are they completely different images?

Also, which version of TIDL/SDK are you using?

Some other factors include the use of inQuantFactor if your model is trained from 0-1, as well as whether some layers when implemented in 8bit result in precision loss. There is a number of different factors, and I will only be able to comment better once I am able to run your model on my end to investigate more. 

Warm regards,

Christina

Hi,

I apologize but I can't give you the model.

I have kept the infer size constant in all the 3 iterations (1000 images). I am just running 3 different tests which have different number of calibration images(10,50 and 100), rest everything is the same.

I am using SDK 8.2. 

I also in-housed mixed precision in these 3 tests. I basically added every layer of the model, still I was getting a loss of ~10% in the best case scenario as compared to 16 bit.

This is my import file:

modelType          = 2
numParamBits       = 8
numFeatureBits     = 8
quantizationStyle  = 3
calibrationOption = 7
inputNetFile        = "../../test/testvecs/pmsa/models/model.onnx"
outputNetFile       = "../../test/testvecs/pmsa/import/output/8_bit_all_rgb_duplicates_removed_100_calib/8_bit_all_rgb_duplicates_removed_100_calib.bin"
outputParamsFile  = "../../test/testvecs/pmsa/import/output/8_bit_all_rgb_duplicates_removed_100_calib/8_bit_all_rgb_duplicates_removed_100_calib_paramfile"
inDataNorm  = 1
inMean = 118.065 120.615 124.185
inScale =  0.024978 0.024357 0.023068
inWidth  = 512
inHeight = 320
inNumChannels = 3
inDataFormat = 1
inFileFormat = 2
writeOutput = 2
outputFeature16bitNamesList = "All the layers of the model"
inData  =   ../../test/testvecs/proj/input/lists/calib_list.txt
outDataNamesList = "hm,wh,reg"

Hi Akshat,

8.2 is an extremely old SDK version and we have had multiple bug fixes/improvements since. This issue is most likely fixed in a newer release. Are you able to upgrade to a newer one? 

Warm regards,

Christina

Hi Christina,

Due to some constraints, I can't change my SDK version. Would it be possible to infer a model on 8.2 sdk that is imported and calibrated in a newer sdk version?  

If not, then can you please tell me if there is a guideline on the nature of images that we should use for calibration.