AM68A: *** buffer overflow detected ***: python3 terminated

Hi,

Can you share the model compilation logs ? 

Also have you tried compiling custom model using tidl tools : https://github.com/TexasInstruments/edgeai-tidl-tools

I don't try to compile models with edgeai-tidl-tools. I use edgeai-benchmark which internally uses the tidl-tools, according to the documentation. However, as you asked for edgeai-tidl-tools, I tried to compile the model using the edgeai-tidl-tools, I get this error,

=====================================================================================================================

Preliminary subgraphs created = 1
Final number of subgraphs created are : 1, - Offloaded Nodes - 1, Total Nodes - 1

Preliminary subgraphs created = 1
Final number of subgraphs created are : 1, - Offloaded Nodes - 1, Total Nodes - 1
Traceback (most recent call last):
File "onnxrt_ep.py", line 284, in <module>
run_model(model, mIdx)
File "onnxrt_ep.py", line 185, in run_model
sess = rt.InferenceSession(config['model_path'] ,providers=EP_list, provider_options=[delegate_options, {}], sess_options=so)
File "/home/ti/anaconda3/envs/TI_08_06_00_03/lib/python3.6/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 283, in __init__
self._create_inference_session(providers, provider_options)
File "/home/ti/anaconda3/envs/TI_08_06_00_03/lib/python3.6/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 315, in _create_inference_session
sess.initialize_session(providers, provider_options)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to add kernel for TIDL_0 com.microsoft TIDLExecutionProvider: Conflicting with a registered kernel with op versions.
Available execution providers : ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

=====================================================================================================================

This error is coming from the script ""onnxrt_ep.py" on line,

sess = rt.InferenceSession(config['model_path'] ,providers=EP_list, provider_options=[delegate_options, {}], sess_options=so)

where, 

Available execution providers : ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

Thanks,

Ahmed

Thanks for sharing the above observation.

Could you try out running the default examples ? This will null out environment setup related possibility and narrow down the debug further.

If this has verified by you earlier may i know what is opset version of your model ?

Yes, I have already tried the default examples and they are running perfectly fine. 

From the documentation, I know that TIDL support onnx with opset 9 and 11. The current opset of my model is 11. Still, I am facing the model loading issue.

Thanks,

Ahmed

Thanks for confirmation.

I will assign this thread to our OSRT expert.

He will get back to you on this.

Hi Ahmed,

The network you are using seems to have multiple subgraphs as part of the exported network itself. This could possibly lead to above issues you are observing. Can you please re-check the pytorch code used to export this model and avoid generation of these subgraphs as part of onnx export?

Regards,

Anand

Hi Anand,

Can you please direct me to the documentation where I can avoid the generation of subgraphs for TI? As far as I know, to convert the model from PyTorch to ONNX, I need to set the input size and the opset. The opset is set to 9 and 11.

Are there any other parameters I should set to convert the model such as export_params, do_constant_folding, etc.?

Thanks,

Ahmed

Hi Anand,

Why the model should not have multiple subgraphs?

Some subgraphs which are supported can work on C7x/MMA accelerators, while others which are not supported can work on ARM cores.

As a results, the number of subgraphs might not always be 1.

-- Joy

Hi Anand,

Do you have any update regarding this issue?

Thanks,

Ahmed

Hi Ahmed, Joy,

Apologies for delay in getting back. Probably I did not explain this clearly - ONNX graph is provided to TIDL, which then creates subgraphs based on supported layers which can be delegated to DSP and infers the remaining subgraphs on ARM. However assumption is ONNX graph itself comes as a single subgraph. Below statements in log:

"

Preliminary subgraphs created = 1
Final number of subgraphs created are : 1, - Offloaded Nodes - 1, Total Nodes - 1

Preliminary subgraphs created = 1
Final number of subgraphs created are : 1, - Offloaded Nodes - 1, Total Nodes - 1

"

These statements indicate ONNX graph itself has 2 subgraphs of 1 node each, before TIDL even accessing it. (there would be a 3rd subgraph, did not hit that due to error in 2nd subgraph itself).

If you click on the input/output nodes in netron in the network, as part of the properties, you would see "subgraph" as one of the network properties, which would be more than 1. Unfortunately we don't have this case supported where ONNX graph itself has multiple subgraphs. In most of the use cases, I have observed only one of these subgraphs is the actual network to be inferred on (which TIDL would create subgraphs from and infer on DSP/ARM), and the others are dummy nodes generated as part of model export. These must be getting generated from somewhere in pytorch code, can you please review the pytorch code for source of these dummy nodes?

Hope the explanation is clear, please ping back in case of any confusion.

Regards,

Anand