[FAQ] TDA4AH-Q1: How to edit/change ONNX graph to remove unsupported layers by TIDL and add supported ones

Hi,

The list of supported operators can be found in edgeai tidl tools gitgub repo at location : https://github.com/TexasInstruments/edgeai-tidl-tools/blob/09_02_09_00/docs/supported_ops_rts_versions.md

After doing due diligence its clear that activation HardSwish is not supported (as on 9.2.9.0) release, the edgeai tidl repos supports this type of scenarios where unsupported layers are delegated to arm for inference by effectively creating subgraphs with locus of supported layers which delegate to DSP for acceleration.

Case 1 (Importing Onnx model without modification)

1. Follow the model compilation steps mentioned here : https://github.com/TexasInstruments/edgeai-tidl-tools/blob/09_02_09_00/examples/osrt_python/README.md

2. Post model compilation check out the model artifacts directory where subgraph svg files are located.

For above model there will be 2 sub graph created, for 5 nodes where 4 nodes are offloaded(delegated to dsp)

Preliminary subgraphs created = 2
Final number of subgraphs created are : 2, - Offloaded Nodes - 4, Total Nodes - 5

In above case, it is expected to offload the unsupported layer to arm for inference, this might not be optimal but it ensures the end to end model inference despite of unsupported nodes by TIDL.

Case 2 (Change the Onnx graph, replace the unsupported nodes by supported nodes)

1. The open source tools like onnx-graphsurgeon can be leveraged here, please refer the readme https://github.com/NVIDIA/TensorRT/tree/master/tools/onnx-graphsurgeon

2. Here is the sample script that removes the hardswish node from above graph and replace it with relu activation for reference, that you can use for reference.

Feel free to change or update the same for your custom usecase.

!pip install onnx_graphsurgeon --quiet

import onnx_graphsurgeon as gs
import onnx

# Load the ONNX model
graph = gs.import_onnx(onnx.load("test_model_Opset_18_IR_8.onnx"))

# Find the HardSwish node
hardswish_node = [node for node in graph.nodes if node.op == "HardSwish"][0]

# Create a new ReLU node with new output tensor
relu_node = gs.Node(op="Relu", inputs=hardswish_node.inputs, outputs=[gs.Variable(name="relu_out")]) 

# Replace the HardSwish node's output with the new ReLU node's output
for node in graph.nodes:
    for i, inp in enumerate(node.inputs):
        if inp == hardswish_node.outputs[0]:
            node.inputs[i] = relu_node.outputs[0]

# Remove the HardSwish node 
graph.nodes.remove(hardswish_node)
graph.nodes.append(relu_node)

# Clean up the graph and export the modified ONNX model
graph.cleanup()  
onnx.save(gs.export_onnx(graph), "test_model_Opset_18_IR_8_modified.onnx")

# Load and verify the modified model
modified_model = onnx.load("test_model_Opset_18_IR_8_modified.onnx")
print(modified_model.opset_import)
print(modified_model.ir_version)

Once the above script execution is completed the modified model file will be stored that will look like below.

Use this modified onnx graph and recompile the model (Refer Case 1, (1)). Once the model is being compiled the runtime_visualuzation.svg graph as seen below contains single subgraph indicating all layers are supported.

Open subgraph_0_tidl_net.bin.svg as seem the below subgraph the Conv + Relu are combined together and single subgraph is created.

Please do diligence specific to speed and accuracy trade offs of such modification before hand. The above is just a examples to illustrate specific use case.