[TDA4EVM] How to debug TIDL model in TDA4

Hi, 

When I use standalone test, the process is stocked at 1st "Deconvolution Layer".

I know the document said "Recommend to use Resize/Upsample to get better performance. Only 4x4 kernel with 2x2 stride is supported".

But is the "Deconvolution Layer" verified?

Chares

Hi Charles,

What are the parameters of Deconvolution layer present in your network?

Regards,

Rishabh 

Hi,

The parameters:

param {
      lr_mult: 0.0
      decay_mult: 0.0
}
convolution_param {
      num_output: 48
      bias_term: false
      pad: 1
      kernel_size: 4
      group: 48
      stride: 2
      weight_filler {
            type: "bilinear"
      }
}

Hi,

For your reference.

After processing for a long time, it finished.

However the two results between PC and TDA4 are not the same.

Input:

PC:

TDA4_EVM:

Charles

Charles,

There is an issue with deconvolution that can cause mismatches between PC and TDA4x. We are in the process of fixing it and will let you know once the release is available.

regards,

Victor

Hi Victor,

Are you also verifying the execution time issue as Rishabh said ?

Do you have a target date to release?

Charles

Hi Charles,

The release will be available in the beginning of Jan 2020. It will fix some performance issue we have been seeing with deconvolution.

However you are observing processing time of couple of minutes for 3 images, which is a long time. We have not encountered for any model such a long processing time on our end.

Can you enable trace in the inference config file by setting:

enableLayerPerfTraces = 1

debugTraceLevel = 1

Please refer to the section 'Steps to Analyze Performance' in the TIDL's user's guide 'Steps to Debug' chapter.

You can share the performance log with us so we can see which layers take a long time to execute.

regards,

Victor

Hi,

The standalone test log is as follow.

Layer 32, 37, 39 are Deconvolutional layers. Looks like layer 32 and 37 take a longer time.

The overall TSC Mega Cycles is 315097 Mega. The TSC Mega Cycles of the same model test in TDA2x is ~26 Mega.

The graphs generated by tidlModelGraphviz are as follow:

Hi Charles,

Yes I can see that the deconvolution layer takes a long time. Can you try to rerun the test with the attached library ? Thanks.

regards,

Victortidl_algo.lib

Hi Victor,

Please see the following table.

Total Megacycles is 2/3 of previous test. 

The values in "LayerSetupCycles" field are reduced a lot. 

The others have no big difference.

Charles

Hi Charles,

Please ignore the setup column as I modified the code to measure something else for my own debugging. We are still improving the deconvolution layer so the final release may be faster.

However you reported execution time in the order of "minutes". I suspect it is because you have trace enabled. Can you try to run after disabling trace in the config file ?

regards,

Victor

Hi,

Here is my config setting:

inFileFormat = 2
postProcType = 3
numFrames = 1
padInBuffInTB = 1
netBinFile = "testvecs/config/tidl_models/caffe/tidl_net_FS.bin"
ioConfigFile = "testvecs/config/tidl_models/caffe/tidl_io_FS.bin"
totNumClasses = 8
outData = testvecs/output/FreeSpace.bin
inData = testvecs/config/test.txt
debugTraceLevel = 0
writeTraceLevel = 0
enableLayerPerfTraces = 0

And the process time for single image in standalone test is around 3.5 mins.

The  "tidl_infer_adjsegNet" model is also test, the process time is very short.

Charles

Here is the console output of my test.

Hi Charles, 

Would it be possible for you to make a dummy network that only includes layers 32 to 37 and share it with TI so we can reproduce the issue on your side ? 

regards,

Victor

Hi Victor,

Will update to you once prepared.

Charles

Hi Victor,

Thanks.

Is this bug affect to Deconvolutional Layer only ?

Charles