TIDL-Caffe Result Matching

William Wong

Prodigy 70 points

Hi,

With a CNN model, is it possible to produce the exact same output using TIDL and using caffe-jacinto with quantization?

If yes, what are the quantization parameters I need to set in caffe-jacinto?

William

over 6 years ago

0 kumar.desappan over 6 years ago

TI__Mastermind 22105 points

Hi,

Same quantization logic is implemented in caffe-jacinto and TIDL. We did not spend effort to match the outputs. The caffe-jacinto will provide close indication of expected accuracy when it is deployed using TIDL.

Thanks and Regards,

Kumar.D

0 William Wong over 6 years ago in reply to kumar.desappan

Prodigy 70 points

Hi,

I have further tested the said scenario with TIDL and pre-trained jsegnet21v2 initial model (downloaded from: https://github.com/tidsp/caffe-jacinto-models/tree/caffe-0.16/trained/image_segmentation/cityscapes5_jsegnet21v2/initial).

Below are the examples of outputs. Although Caffe with quantization produces coarser segmentation result than Caffe without quantization (which is expected), it is still pretty different from TIDL output. Is there any advice to perform a closer estimation of TIDL accuracy using Caffe-jacinto?

Caffe result without quantization:

Caffe result with quantization (12-bit for weights, 8-bit for activations, power2 scaling set to true):

TIDL result (bitwidth: 12):

0 kumar.desappan over 6 years ago in reply to William Wong

TI__Mastermind 22105 points

TIDL uses non-power 2 scaling. Please try with this configuration in caffe-jacinto.

Could you confirm that you have successfully imported the same network and executed the same on TIDL (The default network available in the release is not this one)? If yes, did you try execute TIDL for more number of frames. TIDL quantization uses statistic from previous executions. We expect better results over the execution.

Can you also try with 8-bitwidth for model ?

0 William Wong over 6 years ago in reply to kumar.desappan

Prodigy 70 points

I have tried to turn off power2 scaling but the results are similar.

Before trying out the execution-over-time experiment, I might need you to point me to the right direction regarding some parameters in caffe-jacinto to obtain the closest possible output to TIDL: Does TIDL apply offset on activations and weights while performing dynamic fixed point quantization?

0 kumar.desappan over 6 years ago in reply to William Wong

TI__Mastermind 22105 points

Please match quality after 10 or 20 frames. There are differences in the initial frames. For the first frame quantization is not applied in caffe-jacinto where as the TIDL uses the statistics estimated by import tool an does quantization even for the first frame.

BTW, TIDL apply offset only on weights.

Please set the below in TIDL interface.
quantMargin = 20;

0 kumar.desappan over 6 years ago in reply to kumar.desappan

TI__Mastermind 22105 points

I have tried the same model on TIDL without any code changes (8-bitwidth for model). Below is the result that we are observing for frame 0.

Please confirm that you could get the same results with 8-bitwidth (numParamBits = 8) for this model by following the steps mentioned in the users guide.

0 William Wong over 6 years ago in reply to kumar.desappan

Prodigy 70 points

Hi Kumar,

Thanks for the help. I could now obtain identical result as yours in TIDL with the suggested settings. With this, it at least provides us a clue on how to use caffe to better estimate the accuracy while porting to TIDL and embedded system.

William

0 Eric Lai over 6 years ago in reply to kumar.desappan

Expert 1140 points

Hello kumar,

Can you tell me how to get the .yuv file like what you show above??

I use the model import tool and I only got the .bin files and some dump .y files.

and I open the .y files using the raw data format it looks like this one.

please share with me .thank you!

Regards,

Jimmy Lai

0 kumar.desappan over 6 years ago in reply to Eric Lai

TI__Mastermind 22105 points

Jimmy Lai
I have used the overlay code that used in out vision SDk demo to overlay output on the YUV

Regards,
Kumar.D

0 Eric Lai over 6 years ago in reply to kumar.desappan

Expert 1140 points

Kumar.D,

Okay,I got it.
Thanks.

Regards,
Jimmy Lai

Processors

Processors forum

TIDL-Caffe Result Matching