TDA4VM: MaxPool layer outputs all zeros during execution

F_Fontana

Prodigy 140 points

Part Number: TDA4VM

Goodmorning,

I have an issue executing a model with a MaxPool layer (3x3 kernel, pads=[1, 1, 1, 1], strides=[1, 1]):

conversion is successful but during execution on the TDA4VM board, the MaxPool layer gives all zeros as output.

Do you have any idea about what could cause this problem? Is there any limitation or unsupported parameter?

Attached you can find a very simple model with few layers that shows the issue together with our conversion log and script.

Regards,

Federico

maxpool_model_issue.zip

over 4 years ago

0 Praveen Eppa1 over 4 years ago

TI__Genius 17580 points

Hi Federico,

Sorry for the late reply.

I am checking this issue but not able to reproduce the issue at our end, so please share below additional data to replicate the issue.

1. Import and infer config files

2. Input data file

Thanks,

Praveen

0 F_Fontana over 4 years ago

Prodigy 140 points

Hello Praveen,

Attached to the previous message you can find a compressed folder containing:

- test input data (numpy)

- quantization data (raw)

- import config file (conversion_script.txt)

I'm now attaching also the inference config file as you requested, plus the binary model generated by TIDL Import Tool.

If you need any other file or information, let me know.

Thanks,

Federico

maxpool_model_inference_files.zip

0 F_Fontana over 4 years ago in reply to Praveen Eppa1

Prodigy 140 points

Hello Praveen,

Were you able to reproduce the issue with the files provided in previous messages?

Thanks,

Federico

0 Praveen Eppa1 over 4 years ago in reply to F_Fontana

TI__Genius 17580 points

Hi Federico,

Sorry for the late reply.

I tried with the files you shared but not able to reproduce the issue. I am able to see non-zero values in the max pool layer output.

So, kindly share the layer level outputs from TDA4 board execution so that we can compare them with the output traces at our end.

Please set debugTraceLevel = 3 writeTraceLevel = 3 in the infer config file to get the layer level traces.

Thanks,

Praveen

0 F_Fontana over 4 years ago in reply to Praveen Eppa1

Prodigy 140 points

Hello Praveen,

Attached you can find either the trace coming from the execution on the TDA4VM board and the simulation ones.
As you can see, last layer's trace in simulation is not all zeros, while on the board's trace we have all zeros.
Could it be due to an issue present only on the target? 
Furthermore, should the simulation be bit-exact?
I want also to add a further information: it seems that this kind of issue appears when a Pooling layer is the last layer of a model

Thanks,
Federico

maxpool_simul_and_trace.zip

0 Praveen Eppa1 over 4 years ago in reply to F_Fontana

TI__Genius 17580 points

Hi Federico,

Thanks for sharing the traces. We could reproduce the issue with maxpool 3x3 kernel, stride =1 on the EVM.

We have fixed this max pool issue and fix will be available in the coming SDK 7.3 release planned in early next month.

Regards,

Praveen

0 F_Fontana over 4 years ago in reply to Praveen Eppa1

Prodigy 140 points

Hi Praveen,

That's a good news, thanks a lot!

I would add a related question then:

Does that fix affect MaxPool operation only? Because we faced a similar issue on an AveragePool layer too when it's in the same position of the MaxPool one (at the end of the model), but in that case instead of having all zeros as output, we have all NaN values.

I tried to save the trace as I did for the MaxPool and I discovered that no trace was created for that layer at all when running it on the EVM! During simulation, instead, everything works fine.

Could it be related to the same issue of the MaxPool layer and, then, be fixed in SDK v7.3 as well?

Thanks for your support,

Federico

0 Praveen Eppa1 over 4 years ago in reply to F_Fontana

TI__Genius 17580 points

Hi Federico,

The max pool fix I did affect only maxPool operation, it will not impact or fix the average pool layer issue.

Kindly share the model with Average pool so that I will try to replicate the issue at our end.

Thanks,

Praveen

+1 Praveen Eppa1 over 4 years ago in reply to Praveen Eppa1

TI__Genius 17580 points

Hi Federico,

I received model with Average pooling and could replicate and fix this issue in the Avg pooling layer, but we can't include this bug fix in the upcoming 7.3 release as code freeze happened last week itself and hence I am sharing the fix here, so please try with this fix.

The bug fix is in the import tool code,

Folder Name : ti_dl/utils/tidlModelImport

File Name : tidl_import_core.cpp

Function Name : tidlRunPerfSimTool

In the "tidlRunPerfSimTool" function around line # 832 replace below existing if condition with new if condition,

Existing if condition:

if((tidlNet->TIDLLayers[i].layerType == TIDL_PoolingLayer) && (tidlNet->TIDLLayers[i].layerParams.poolParams.kernelW == 0))
{
tidlNet->TIDLLayers[i].outData[0].pitch[TIDL_CHANNEL_PITCH] = perSimInfo->sdataFlowInfo[i].bufInfo[OUT_FEAT_MAP][WRITE].bufWidth = tidlNet->TIDLLayers[i].outData[0].dimValues[TIDL_DIM_WIDTH];
tidlNet->TIDLLayers[i].outData[0].pitch[TIDL_ROI_PITCH] = perSimInfo->sdataFlowInfo[i].bufInfo[OUT_FEAT_MAP][WRITE].bufSize = tidlNet->TIDLLayers[i].outData[0].pitch[TIDL_CHANNEL_PITCH]*tidlNet->TIDLLayers[i].outData[0].dimValues[TIDL_DIM_NUMCH];
tidlNet->TIDLLayers[i].outData[0].padW = perSimInfo->sdataFlowInfo[i].bufInfo[OUT_FEAT_MAP][WRITE].padC = 0;
tidlNet->TIDLLayers[i].outData[0].padH = perSimInfo->sdataFlowInfo[i].bufInfo[OUT_FEAT_MAP][WRITE].padR = 0;
}

New if condition:

if((tidlNet->TIDLLayers[i].layerType == TIDL_PoolingLayer) && (tidlNet->TIDLLayers[i].layerParams.poolParams.kernelW == 0))
{
perSimInfo->sdataFlowInfo[i].bufInfo[OUT_FEAT_MAP][READ].bufWidth = 1;
if (tidltb_isOutDataBuff(tidlNet, tidlNet->TIDLLayers[i].outData[0].dataId,currLayersGroupId))
{
tidlNet->TIDLLayers[i].outData[0].pitch[TIDL_LINE_PITCH] = tidlNet->TIDLLayers[i].outData[0].dimValues[TIDL_DIM_WIDTH];
}
tidlNet->TIDLLayers[i].outData[0].pitch[TIDL_CHANNEL_PITCH] = perSimInfo->sdataFlowInfo[i].bufInfo[OUT_FEAT_MAP][WRITE].bufWidth = tidlNet->TIDLLayers[i].outData[0].dimValues[TIDL_DIM_WIDTH];
tidlNet->TIDLLayers[i].outData[0].pitch[TIDL_ROI_PITCH] = perSimInfo->sdataFlowInfo[i].bufInfo[OUT_FEAT_MAP][WRITE].bufSize = tidlNet->TIDLLayers[i].outData[0].pitch[TIDL_CHANNEL_PITCH]*tidlNet->TIDLLayers[i].outData[0].dimValues[TIDL_DIM_NUMCH];
tidlNet->TIDLLayers[i].outData[0].padW = perSimInfo->sdataFlowInfo[i].bufInfo[OUT_FEAT_MAP][WRITE].padC = 0;
tidlNet->TIDLLayers[i].outData[0].padH = perSimInfo->sdataFlowInfo[i].bufInfo[OUT_FEAT_MAP][WRITE].padR = 0;
}

After replacing with above if condition, please build import tool and then import your model again and check if it is (Avg pooling) working.

Thanks,

Praveen

0 F_Fontana over 4 years ago in reply to Praveen Eppa1

Prodigy 140 points

Hello Praveen,

I confirm that this bugfix solved the AvgPool issue!

Thanks a lot for the support,

Federico

Processors

Processors forum

TDA4VM: MaxPool layer outputs all zeros during execution