TDA4AH-Q1: When we test our model in SDK 9.2, we find that the results on the board does not match the simulation results and custom layer cannot remove the pad of the model output

Hi Guang;

We have downloaded your zip file, thanks.

We want to try to run the .bin files in the target EVM. Could you let us know what these bin files are? And how you run them.

We saw the onnx model files as well, but could you tell us how you linked/built with your C7x-MMA layer code?

Best regards

Wen Li

Hi:

For the first question: We have provided files in a similar format for the previous issues, and your colleagues have already run them correctly. You can consult them for the specific way to run them.

previous issue: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1494305/tda4ah-q1-the-consistency-issues-between-simulation-and-on-board-inference-as-well-as-the-consistency-issues-of-multiple-inferences-on-the-board

For the second question: The c7x-mma-tidl file only contains the custom layer code. We are using the method of adding the custom layer code that default code for TDA4 recommended. You can directly add it after comparing.

Best regards

Guang Ren

Hi

Please file separate e2e for different issues. 

GUANG REN said:

• the result is different between board and simulation(inconsistent_board_and_simulation folder)
• when we add a simple custom layer on the output layer of the network which we provide, the result is different between board and simulation, however it is always right when we add the same custom layer after other network(inconsistent_board_and_simulation_after_add_custom_layer folder)

We will address these two problems in this e2e. Please file another one for the third. 

For the inconsistent problem, please set debugTraceLevel to 1 and this will generate the intermediate results.

"

• If the user is finding target(EVM) execution output for a given network to be different from the host emulation output for the same, then they should follow the following steps to identify which layer causes the first mismatch:
  • Enable layer level fixed point traces in both host emulation and EVM execution. This can be done by setting writeTraceLevel = 1 in the TIDL-RT inference configuration file.
  • Run the TIDL-RT inference on host emulation mode. Layer level traces will be generated inside ti_dl/test/traces folder. If this folder is missing, please create the same. Keep a copy of this trace folder, lets call it traces_ref
  • Repeat the above step for target(EVM) execution. Keep a copy of this trace folder, lets call it traces_target
  • Use any binary comparison tool (e.g. beyond compare, diff, cmp etc) to compare the traces_ref and traces_target folder
  • The outputs are expected to bit match and if any particular layer output is mismatching in these two modes then the first layer where mismatch is observed is the layer which needs to be reported.

"

Regards,

Adam

Hi

For the first one, we have submitted another e2e in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1507146/tda4ah-q1-when-we-test-our-model-in-sdk-9-2-we-find-that-custom-layer-cannot-remove-the-pad-of-the-model-output

For the second one, we have tested this model before, inconsistencies in simulation and board-end consistency generally start from the deconvolutional layers or InnerProduct layers.

Best regards

Guang Ren

Hi 

GUANG REN said:

we have tested this model before, inconsistencies in simulation and board-end consistency generally start from the deconvolutional layers or InnerProduct layers

Can you provide more detailed information for this deconvolutional layer or innerProduct layer, like partial onnx model contains these layers？Or can you point out which specific layer in your model?

Regards,

Adam

Hi 

Could you also help fill the list for the first mismatch issue without custom layer?

As for the mismatch with custom layer, we have discussed locally for work around. Please try removing padding on arm without data type converting. 

Regards,

Adam

Hi 

To update on this issue, the mismatch problem is reproduced in "inconsistent_board_and_simulation folder". This issue is tracked internally in Jira.

Regards,

Adam

Hi

Regarding this issue "inconsistent_board_and_simulation folder", the difference starts from transed_conv106 layer. Please help resolve this issue.

Hi

We will try to solve this problem by adding crop layer in the caffe network, as you suggested in another e2e. 

But we have the following doubts:

1.It is normal in SDK8.6 when we use the same method to remove pad in the same model, so it is a version compatibility issue. We hope that it can be resolved

2.We still don't know what caused the error this time. So we are not sure if we will encounter the same issue when adding a custom layer in the future.

Best regards

Guang Ren

We are looking at the inconsistencies between target/board and emulation runs. We were not able to reproduce the issue on our side. We are getting a matching result for the final output and for every layer. We have used import and inference tools from SDK for emulation runs. For target runs, we have used inference tool packaged with the J784S4 Linux image. In order to understand a bit more, can you please provide the following details:
1) Ubuntu Version
2) SDK and TIDL version
3) Build steps followed for building import tool and C7x firmware
4) How was import and inference performed
5) There was a tidl_net_onnx.bin.txt provided in "cannot_remove_pad" folder. Can you please provide a binary text version for the inconsistency case also
6) SVG file generated from the import
7) The trace files generated in target/board and emulation while execution of the net file.

The bin file shared under inconsistent_board_and_simulation , seems to have custom layer at layer 55 , but the caffe model doesn't seem to have that. Please confirm this also. Please clarify if the issue 1 mentioned is with out custom layer or not.

Hi Jijo,

Please upload your artifacts, pc infer results and evm infer results thus we can compare with customer's results.

Regards,

Adam

Import logs , infer logs , artifacts out files from PC (ref and CI) and EVM

tidrive.itg.ti.com/.../deecf608-6e53-4b24-a80b-7743bc344ad5

Access code: @JMfFF72

Hi 

The link is abnormal, please try edit again. This is seen sometimes with e2e.

The link is abnormal. Paste it from email:

https://tidrive.itg.ti.com/a/NZK_UaoSvw4nkr8I/deecf608-6e53-4b24-a80b-7743bc344ad5?l 

Access code: @JMfFF72

Hi GUANG, TI team,

I am little confused on this thread because of multiple issues reported in this thread an trying to summarize here to make sure we all are on same page. This E2E has below issues

1. Inconsistent board and simulation output - TI is not able to reproduce - Pending on customer/Hik Auto
2. Inconsistent board and simulation with custom Layer --> Solution shall be provided which doesn't need custom layer to remove padded data. Basically there will not be padded data b/w channels in output - Action on  TI to provide release. This issue links to other E2E - e2e.ti.com/.../tda4ah-q1-when-we-test-our-model-in-sdk-9-2-we-find-that-custom-layer-cannot-remove-the-pad-of-the-model-output
3. Can not remove padding with custom layer - Not relevant as (2) shall address it 

So in summary this E2E is going to focus on point (1) Inconsistent board and simulation output and TI is waiting for customer input as we are not able to reproduce

Thanks

With Regards

Pramod

1. Below release (10_01_03_08) shall address your issue of having extra padded data between channels, internal TI JIRA record (TIDL-7531)

Link: https://tidrive.ext.ti.com/u/hxnztNuZ86yVGfO5/32455f3b-eca9-4359-a054-2340aee34bd6?l

Access code: M_K3AH61

2. Inconsistent board and simulation output (internal TI JIR record TIDL-7470) :TI is waiting for customer input as we are not able to reproduce

~Pramod

Hi GUANG, Harry, Adam,

On below

2. Inconsistent board and simulation output (internal TI JIR record TIDL-7470) :TI is waiting for customer input as we are not able to reproduce

Since we didn't hear from you, we plan to close our internal record and mark this issue as rejected.

Will wait for 1 week before doing this to allow you raise your concern

Thanks,

with Regards,

Pramod