TDA4AH-Q1: The consistency issues between simulation and on-board inference, as well as the consistency issues of multiple inferences on the board

We are looking into both the issues (A) simulation vs onboard difference (B) multiple inference having different result

We are able to reproduce both of these issues and should get back on both of them tomorrow

Thanks,

With Regards,

Pramod

Hi Guang
This is about the models with consistency issues between simulation and on-board inference.
Model 1 is resolved.
On compilation of model 2, i am getting error.
Error: 

: (index) < (current_size_):
terminate called after throwing an instance of 'google::protobuf::FatalException'
what(): CHECK failed: (index) < (current_size_):
Aborted (core dumped)
make: *** [makefile:413: it_run] Error 134


are you able to compile the model?

Thanks and Regards
Vijay

We verified all the models that were previously capable of correct reasoning and found that one of them had a consistency issue after the patch update.

The patch we used is 

Link: https://tidrive.ext.ti.com/u/h1dm0WNXZvWVV2bW/7e46c098-8c53-4eb0-84c3-be47c8f34b04?l
Access Code: |wH7N3VZ

And The materials for reproduction have also been provided. This is the newly added model, not the one previously provided.

/cfs-file/__key/communityserver-discussions-components-files/791/different_2D00_between_2D00_simulation_2D00_and_2D00_board.zip

Sorry, this is our mistake. We provided the wrong import.config.

We have reorganized the materials for model 2, which you can download from the link below:

/cfs-file/__key/communityserver-discussions-components-files/791/2425.our_5F00_model.zip

Issue Tracking in TI system: TIDL-7340, TIDL-7369

Hello GUANG REN 

Thanks for these additional inputs.

From TI side a release (10_01_03_06) available with at below path

Link:

tidrive.ext.ti.com/.../0838da3a-df57-4f19-92db-0f5850348be5
Code: 5AT)S1zc

somehow if above link is not fully appearing in E2E then sharing a tinyurl :https://tinyurl.com/3cf7k4p9

Please note: This is still undergoing more validation and we will be confirming tomorrow about the test results. As of now sharing here in case you want to try in advance

It should address all of below E2Es

1. Current(this) E2E: e2e.ti.com/.../tda4ah-q1-the-consistency-issues-between-simulation-and-on-board-inference-as-well-as-the-consistency-issues-of-multiple-inferences-on-the-board

Other E2Es:
2. e2e.ti.com/.../tda4vh-q1-when-disable-otf-buffer-optimizations-the-result-of-deconv2dlayer-is-error-on-board
3. e2e.ti.com/.../tda4vh-q1-c7x-encountered-an-error-during-the-initialization-of-deconvolution-with-group
4. e2e.ti.com/.../tda4ah-q1-when-the-group-is-not-equal-to-1-the-simulation-inference-results-of-deconvolution-in-sdk-9-2-are-abnormal
5. e2e.ti.com/.../tda4ah-q1-when-we-inference-on-the-board-with-a-caffe-model-which-includes-deconvolution-layers-in-sdk9-2-the-process-gets-stuck-at-the-deconvolution-step
---
6. e2e.ti.com/.../tda4ah-q1-when-we-test-our-model-in-sdk-9-2-we-find-that-the-results-on-the-board-does-not-match-the-simulation-results
---
7. e2e.ti.com/.../tda4ah-q1-the-consistency-issues-between-simulation-and-on-board-inference-as-well-as-the-consistency-issues-of-multiple-inferences-on-the-board
---
below is partially addressed
8. e2e.ti.com/.../tda4ah-q1-when-we-test-innerproduct-layer-in-sdk-9-2-we-find-that-the-computation-time-of-this-layer-increases-significantl
---

Thanks,
With Regards,

Pramod

Hi GUANG REN

We did the Validation, and all models are working fine. There is not mismatch.

Thanks and Regards
Vijay

Hi Vijay，

         Based on the patch provided on the 26th, we have discovered a new consistency issue. In yesterday's post, we provided the corresponding model structure.  The new patch which release today  still has the same issues. 

        Network structures similar to 'Flatten + Innerproduct +Relu + Innerproduct'  all have consistency issues during inference.

GUANG REN said:

We verified all the models that were previously capable of correct reasoning and found that one of them had a consistency issue after the patch update.

The patch we used is 

Fullscreen

1

2

Link: tidrive.ext.ti.com/.../7e46c098-8c53-4eb0-84c3-be47c8f34b04

Access Code: |wH7N3VZ

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Link: tidrive.ext.ti.com/.../7e46c098-8c53-4eb0-84c3-be47c8f34b04
Access Code: |wH7N3VZ

And The materials for reproduction have also been provided. This is the newly added model, not the one previously provided.

/cfs-file/__key/communityserver-discussions-components-files/791/different_2D00_between_2D00_simulation_2D00_and_2D00_board.zip

Hello, Pramod

While verifying the patch, we found that there are still inconsistencies between simulation and board-side performance in other models, as well as issues with board-side execution failures.The models are shown as follows. Please help us resolve these issues

the patch we use is 10_01_03_06

the model we use is as below:

/cfs-file/__key/communityserver-discussions-components-files/791/Supplementary_2D00_model_2D00_0402.zip

Hi Guang

/cfs-file/__key/communityserver-discussions-components-files/791/different_2D00_between_2D00_simulation_2D00_and_2D00_board.zip
I tested this model with the latest patch ( 10_01_03_06). Outputs are matching.

Thanks and Regards
Vijay

Hi Guang 

/cfs-file/__key/communityserver-discussions-components-files/791/Supplementary_2D00_model_2D00_0402.zip

 1) mod_seg_board_error :
          1) I tested this model, it is running fine on EVM and output is matching.
          2) have you did make sdk_scrub -j and compiled the model again with latest patch ?
          3) from logs you shared, its looks like , SD-card not flashed correctly.

 2) difference_between_simulation_and_board: 
          1) i am not able to compile the model using the configs and model you shared.
          2) error : [TIDL Import] [PARSER] ERROR: Cannot import this model. Some data is required but not provided as Input data layer. Please double check your                          model's input data top names -- [tidl_caffeImport_v2.cpp, 538]
              [TIDL Import] FATAL ERROR: Caffe model import failed -- [tidl_import_main.cpp, 482]
              [TIDL Import] Aborting

Thanks and Regards
Vijay

Hi, VIJAY SINGHAL

Difference_between_simulation_and_board issue has a higher priority. We have tested the model locally again, and it can compile normally. Can you provide the complete log for us.

Thanks!

Hi Gaung
I tested it again, sorry that was my mistake. 
Now i am able to compile and run the model, output is matching for me.

can you please recompile the model again and test it locally ?
If you are still facing issue, we can debug over call. 

Thanks and Regards
Vijay

Hi, VIJAY SINGHAL

I test it again, and the result is still different between simulation and board. Can you provide the simulation output result and board output result of the input data which I provide ? 

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/evm_5F00_tidl_5F00_res_5F00_onnx_5F00_model_5F00_7                           https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/ref_5F00_tidl_5F00_res_5F00_onnx_5F00_model_5F00_7

Hello, VIJAY 

We will further verify it as quick as we can.  We find anthor Consistency issue . When we test model 1 in the below link which we have provided, we find the result is different between SDK8.6 ans SDK10.1(10_01_03_06). Please help us solve it as soon as possible.

/cfs-file/__key/communityserver-discussions-components-files/791/the_5F00_consistency_5F00_issues_5F00_between_5F00_simulation_5F00_and_5F00_on_2D00_board_5F00_inference.zip

Hello Guang,

Just want to clarify on your latest query/issue:

Output of 8.6 and 10.1 can be different due to quantization changes across these versions . We have not been tracking to have same bit matching output b/w 8.6 and 10.1. But we have been tracking our functional behavior is not degrading due to change in quantization by comparing against ONNX run time.

So I want to understand the issue better with respect to absolute behavior of 10.1 instead of comparing with 8.6 : do you see any issue with functional behavior of 10.1 even though it is not matching with 8.6?

Thanks,

with Regards,

Pramod

The results of simulation and on-board inference can currently be kept consistent. However, consistency with the inference results from SDK 8.6 cannot be guaranteed. We have compared the quantization coefficients of the two versions and found them to be consistent. The inference results should also be consistent in theory.

We are still in the process of further identifying the root cause of the issue.

Hi Guang

Hope you are well and doing well.

we need some clarification from your side. Can you please confirm?

Thanks, and Regards
Vijay

Hi Vijay

    Make a statement about the current situation.

•     The following issues have been resolved:
  1. GUANG REN said:

     These two models are provided to reproduce the consistency issues between simulation and on-board inference.

     /cfs-file/__key/communityserver-discussions-components-files/791/the_5F00_consistency_5F00_issues_5F00_between_5F00_simulation_5F00_and_5F00_on_2D00_board_5F00_inference.zip
  • GUANG REN said:

    These two models are suitable for reproducing the issue of inconsistent results from multiple inferences of the model.

    /cfs-file/__key/communityserver-discussions-components-files/791/the_5F00_consistency_5F00_issues_5F00_of_5F00_multiple-inferences.zip
  • the following issues which you are unable to reproduce
    1. GUANG REN said:

       And The materials for reproduction have also been provided. This is the newly added model, not the one previously provided.

       /cfs-file/__key/communityserver-discussions-components-files/791/different_2D00_between_2D00_simulation_2D00_and_2D00_board.zip
    We will further investigate the cause of this issue.
  • Pramod Kumar Swami said:

    8. e2e.ti.com/.../tda4ah-q1-when-we-test-innerproduct-layer-in-sdk-9-2-we-find-that-the-computation-time-of-this-layer-increases-significantl
  The issue of increased time consumption has not been resolved in our testing, and we will reconfirm it.
  • The following issues are defects found in SDK 8.6, not in version 10.01.03.06
    1. GUANG REN said:

       We will further verify it as quick as we can.  We find anthor Consistency issue . When we test model 1 in the below link which we have provided, we find the result is different between SDK8.6 ans SDK10.1(10_01_03_06). Please help us solve it as soon as possible.

       /cfs-file/__key/communityserver-discussions-components-files/791/the_5F00_consistency_5F00_issues_5F00_between_5F00_simulation_5F00_and_5F00_on_2D00_board_5F00_inference.zip
    The current version's simulation and the board-side are consistent. However, they were inconsistent when it was in SDK 8.6.The simulation results of SDK 8.6 and 10.01.03.06 are consistent.