TDA4AH-Q1: When we test InnerProduct layer in SDK 9.2, we find that the computation time of this layer increases significantly

Hello Guang,

Please give me until end of next week to get a response for you. 

Thank you,

Christina

InnerProduct_model.zip

This is our test model which includes InnerProduct Layer

This issue is blocking the progress of the project and needs to be resolved as soon as possible.

Hi Guang,

Please set debug_level = 2, which will give you a breakdown of the layer level performance with more detailed traces. More information can be found here. This will better determine which is the underlying cause of the issue. 

Warm regards,

Christina

The detailed log is here.

/cfs-file/__key/communityserver-discussions-components-files/791/vh_5F00_9.2_5F00_2025_2D00_03_2D00_21_5F00_14_2D00_25_2D00_51.log

Hello Guang,

This is caffe model. For this example model, we can convert to ONNX but wanted to check if you have any issue in converting your full caffe model to ONNX? There are some offline tools available for such conversion.

Quick update on this issue

We had some instability issues in inner product ( even in 8.6) for some specific dimensions as written below so later versions use a  generic un-optimal code resulting in poor performance

In > 2048  || out > 1024 ||  out < 16U

For your case it is as below and hits this condition

In: 4800, out: 64

But inner product can be converted to matmul and then it should provide better performance

If you are able to use ONNX model then we can provide simple surgery rule to convert inner product (gemm in ONNX) to matmul.

Thanks,

With Regards,

Pramod

But in TIDL 8.6 and 9.2, the time cost is less than 1ms.

Yes we found stability issues during 9.2 and changed it so any release starting from 9.2 (including 9.2), you should see the lesser performance.

Can you confirm on possibility of converting your model to ONNX instead of caffe? You can use tools like caffe2onnx · PyPI

The complete model contains operators that are not supported by caffe2onnx, such as YoloProposal and Argmax.

Pramod Kumar Swami said:

convert inner product (gemm in ONNX) to matmul.

How to do this？

Hi Guang,

Please look at onnx model optimizer on how it is converted gemm to matmul for ONNX. I personally am not sure on how to convert directly from caffe but maybe the Dev team can provide more context soon.

Hi Guang
we already shared the patch.
Hope you received it, and patch is working fine.
If you need any further assistance, kindly let us know.

Thanks, and Regards
Vijay

The Patch has consistency issues.We are currently pruning the model to reproduce the issue.

Hi Guang,

Thanks for letting us know. Just for transparency, our TI India team is on vacation on Monday. Please keep us updated and we will get back to you as soon as we can. 

Warm regards,

Christina

Link: https://tidrive.ext.ti.com/u/h1dm0WNXZvWVV2bW/7e46c098-8c53-4eb0-84c3-be47c8f34b04?l
Access Code: |wH7N3VZ

Post that Guang you mentioned about consistency issue and mentioned that will be sharing a model to highlight the issue, Can you please provide the pruned model?

As Christina mentioned, we have holiday in India Monday 31/Mar/2025 but we will be still trying to find a solution for this issue given the urgency of the issue. So please share the relevant details to us so we can progress further

Thanks,

Pramod

Hello, Pramod

We have raised this consistency issue in another ticket which contains the pruned models. Please refer to it.

e2e.ti.com/.../tda4ah-q1-the-consistency-issues-between-simulation-and-on-board-inference-as-well-as-the-consistency-issues-of-multiple-inferences-on-the-board

Thanks, 

GUANG

Thanks GUANG, we will consider this thread as closed assuming that the performance wise it is fine, will respond back on other thread about inconsistency 

thanks

with regrds

Pramod

Hello, Pramod

Sorry, I mean that the performance issue still exists, and there is also the consistency issue

Thanks, 

GUANG

Hello,Pramod

We have still reproduced the issue of a dramatic increase in time consumption based on the new patch. Please help us resolve it.

The patch we use is:

Link: tidrive.ext.ti.com/.../7e46c098-8c53-4eb0-84c3-be47c8f34b04
Access Code: |wH7N3VZ

The phenomenon of the dramatic increase in time consumption is as follows:

The model we use is InnerProduct_model.zip which we have provided

Issue tracking in TI system TIDL-7346

Hi GUANG REN

I run innnerproduct model locally, and i am getting ~0.6 ms as execution time for innerproduct layer.
Complete network execution time is 1.03 ms.

 layer execution time = ( (layer cycles of that layer) / 1GHz ) seconds

Thanks and Regards
Vijay

Will close the ticket, please feel free to create a new ticket if there is new issue.

Br, Tommy