AM67A: Model Compilation Issue

Hi Zhanying Chen,

This looks like the TIDL_TOOLS_PATH environment variable is not set, since it is just recognizing CPUExecutionProvider. 

Before you ran this example, you should run `source setup_env.sh` from the root of edgeai-gst-apps repository so that TIDL_TOOLS_PATH is set to point to the right tidl_tools directory. 

By the way, the latest AM67A SDK for this was 11.0, and the rel_11_01 (same as tag 11_01_07_00) [1] will require you run the update_target.sh script on the EVM board before using the compiled model artifacts. 

[1] https://github.com/TexasInstruments/edgeai-tidl-tools/tree/11_01_07_00 

Thank you very much for your reply. The issue has been resolved. The root cause was that some files were missing from the downloaded onnxruntime-tidl package when executing source ./setup.sh. Retrying the download and setup process a few times solved the problem.

The version of edgeai-tidl-tools I am currently using is rel_11_01_06_00.

In addition, when compiling DeiT Transformer and Swin models, I noticed that there are no model-specific configurations in onnxrt_ep.py or model_configs.py. Does this mean that I need to download the models from the model zoo and modify the code according to the corresponding YAML configuration files?

For example, should I download the model using swin_base_patch4_window7_224_simp.onnx.link, place it under the models/public/ directory, and then manually add the preprocessing and session-related configurations in model_configs.py based on the settings provided in swin_base_patch4_window7_224_simp_config.yaml?

Are there any TI-provided reference implementations that have already been adapted for these models, or any more detailed documentation and reference materials that you would recommend?

Thank you very much for your reply. The issue has been resolved. The root cause was that some files were missing from the downloaded onnxruntime-tidl package when executing source ./setup.sh. Retrying the download and setup process a few times solved the problem.

The version of edgeai-tidl-tools I am currently using is rel_11_01_06_00.

In addition, when compiling DeiT Transformer and Swin models, I noticed that there are no model-specific configurations in onnxrt_ep.py or model_configs.py. Does this mean that I need to download the models from the model zoo and modify the code according to the corresponding YAML configuration files?

For example, should I download the model using swin_base_patch4_window7_224_simp.onnx.link, place it under the models/public/ directory, and then manually add the preprocessing and session-related configurations in model_configs.py based on the settings provided in swin_base_patch4_window7_224_simp_config.yaml?

Are there any TI-provided reference implementations that have already been adapted for these models, or any more detailed documentation and reference materials that you would recommend?

Hello, 
I encountered an issue while performing int8 quantization testing. The model being tested is cl-ort-resnet18-v1. I set debug_level=4, accuracy_level=1, quantization_scale_type = 0, and ran it in compile mode. After completion, under the tmp path, there are float traces for each layer, as well as a file named tidl_trace_subgraph_0_0000_0001_0001_00001_00003_00224x00224.y. Is this normal? After generating the artifacts, I ran python3 onnxrt_ep.py -m cl-ort-resnet18-v1, and all layer traces, including float.bin and .y files, were generated in the tmp directory. At this point, there is one more float.bin than .y files — specifically, one extra float.bin for the last layer. Is this also normal?

However, I’ve discovered a more obvious issue. After running python3 onnxrt_ep.py -m cl-ort-resnet18-v1, the file tidl_trace_subgraph_0_0031_0001_0001_00001_00512_00001x00001_float.bin is generated in the tmp directory. When reading it with dtype=np.float32, I found abnormal data, with a max absolute value of 1.1183523e+27. As a result, when comparing the quantized and floating-point outputs per layer, the correlation is very poor. Additionally, using hexdump -C tidl_trace_subgraph_0_0031_0001_0001_00001_00512_00001x00001_float.bin | head shows that the data does not appear to be in normal float32 format. When I read the tidl_trace_subgraph_0_0031_0001_0001_00001_00512_00001x00001_float.bin generated in compile mode with dtype=np.float32, the data range appears normal. Could you explain the reason for this? Or is there an issue with my parameter settings? Apart from this specific file, most of the other trace files are relatively normal, with correlations reaching 1.

Hi Zhanying Chen,

Okay, I assume your model is not prequantized (otherwise, you will need to set input settings for TIDL accordingly [1] 

When you run with debug_level 4, it creates layer traces for every layer that TIDL runs. The .Y files are the fixed point values, and the _float.bin files are the dequantized  version of those .Y files.

This means that a direct comparison between the two will yield confusing results, as the .BIN files are the same data as .Y, just having been converted to fp32 by rescaling and applying a bias term. Comparing traces should be done with float.bin files from different quantization settings, or vs. directly dumping intermediate oiutputs at the model leve. while running on CPU. Please refer to  [2].

Zhanying Chen said:

At this point, there is one more float.bin than .y files — specifically, one extra float.bin for the last layer. Is this also normal?

Yes, this is typical. There should be extra float.bin for input and output tensors, especially if the output to the network is (per the .ONNX model) in float32. 

Zhanying Chen said:

When reading it with dtype=np.float32, I found abnormal data, with a max absolute value of 1.1183523e+27. As a result, when comparing the quantized and floating-point outputs per layer, the correlation is very poor. Additionally, using hexdump -C tidl_trace_subgraph_0_0031_0001_0001_00001_00512_00001x00001_float.bin | head shows that the data does not appear to be in normal float32 format. When I read the tidl_trace_subgraph_0_0031_0001_0001_00001_00512_00001x00001_float.bin generated in compile mode with dtype=np.float32, the data range appears normal.

A max value that high is strange, yes. That would mean the dequantized from .Y fixed-point traces was scaled by extremely large value. This does not seem typical. Is this the only layer that occurs for? Some outliers are possible without large change to the quantization parameters. 

Also, is this the last layer of your network, or are there other after that behave ordinarily? Is accuracy recovering for later layers? I would also want to know what layer type this layer 31 is (e.g. Convolution with fused activation)

[1] https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/docs/model_compilation.md#quantization-specific-options -- For ONNX prequantized, see advanced_options:prequantized_model 

[2] https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/scripts/debug/layer_trace_inspector/README.md 

BR,
Reese

Thank you very much for your reply.

I tried the method you suggested in your previous response, but I am still seeing the same issue.

The float.bin trace files were generated as follows:

1. Set debug_level=4 and tensor_bits=32.

2. Run:

   python3 onnxrt_ep.py -c -m cl-ort-resnet18-v1
   

3. Then run:

   python3 onnxrt_ep.py -m cl-ort-resnet18-v1
   

4. Copy the generated float.bin files from the /tmp directory.

The .y trace files were generated as follows:

1. Set debug_level=4 and tensor_bits=8.

2. Run:

   python3 onnxrt_ep.py -c -m cl-ort-resnet18-v1
   

3. Then run:

   python3 onnxrt_ep.py -m cl-ort-resnet18-v1
   

4. Copy the generated .y files from the /tmp directory.

After comparing the correlation between the corresponding traces, I found that only Layer 31 (TIDL_PoolingLayer) has a negative correlation coefficient. For all other layers, the correlation coefficients are around 0.98 or higher.

What is particularly strange is that Layer 32 is a TIDL_ReshapeLayer, and its correlation coefficient is about 0.99. Since Layer 32 only performs a reshape operation on the output of Layer 31, I would expect their data to be identical apart from the tensor shape. However, the data in the traces are not the same.

According to subgraph_0_tidl_net.bin.svg, Layers 31 and 32 use the same quantization scale, which seems reasonable.

At this point, the most likely explanations appear to be:

1. The Layer 31 trace is being read incorrectly.

2. The Layer 31 trace data is stored in a special layout or ordering that differs from the other layers.

Could you please help explain what might be causing this behavior?

I have also attached the comparison results. Could you please take a look and help analyze the issue?

Any insights would be greatly appreciated.

====================================================================================================
TIDL Layer Comparison Detailed Summary
====================================================================================================

OVERALL STATISTICS
--------------------------------------------------
Total layers processed: 33
Average Correlation: 0.956779
Average SNR: 16.92 dB
Average Mean Abs Diff: 0.106787
Average Max Abs Diff: 0.731458

BEST LAYER (highest correlation)
--------------------------------------------------
Layer ID: 1
Layer Type: TIDL_DataConvertLayer
Correlation: 0.999983
SNR: 33.63 dB
Mean Abs Diff: 0.887097
Max Abs Diff: 2.000000

WORST LAYER (largest error)
--------------------------------------------------
Layer ID: 31
Layer Type: TIDL_PoolingLayer
Correlation: -0.079204
SNR: -1.71 dB
Mean Abs Diff: 1.201779
Max Abs Diff: 6.981273

LAYER BY LAYER DETAILS
====================================================================================================
ID Type Scale Corr SNR MeanDiff MaxDiff
----------------------------------------------------------------------------------------------------
1 TIDL_DataConvertLayer 1.0000 0.999983 33.6 0.887097 2.000000
2 TIDL_ConvolutionLayer 72.5102 0.997279 22.3 0.011974 0.113018
3 TIDL_PoolingLayer 72.5102 0.998211 24.0 0.012815 0.109948
4 TIDL_ConvolutionLayer 96.8083 0.994732 19.5 0.009364 0.073665
5 TIDL_ConvolutionLayer 31.3994 0.996618 21.6 0.023083 0.144577
6 TIDL_EltWiseLayer 31.3994 0.995041 19.6 0.027936 0.198592
7 TIDL_ConvolutionLayer 112.1579 0.987091 15.6 0.013332 0.162436
8 TIDL_ConvolutionLayer 26.7708 0.991234 17.4 0.042589 0.375309
9 TIDL_EltWiseLayer 53.5416 0.991059 17.4 0.043243 0.372609
10 TIDL_ConvolutionLayer 125.8642 0.990888 17.4 0.011744 0.205541
11 TIDL_ConvolutionLayer 38.6721 0.987486 16.0 0.024017 0.336742
12 TIDL_ConvolutionLayer 43.4524 0.990537 17.2 0.034056 0.482496
13 TIDL_EltWiseLayer 77.3441 0.986302 15.6 0.022858 0.426664
14 TIDL_ConvolutionLayer 124.3273 0.986506 15.7 0.008314 0.231640
15 TIDL_ConvolutionLayer 35.4379 0.989855 16.9 0.034745 0.494941
16 TIDL_EltWiseLayer 35.4379 0.987521 16.0 0.024220 0.494941
17 TIDL_ConvolutionLayer 150.3406 0.987814 16.1 0.012051 0.894124
18 TIDL_ConvolutionLayer 86.8233 0.993675 18.9 0.010532 0.167100
19 TIDL_ConvolutionLayer 33.3250 0.990988 17.4 0.034477 0.424768
20 TIDL_EltWiseLayer 66.6499 0.988109 16.2 0.016939 0.411054
21 TIDL_ConvolutionLayer 122.3459 0.986370 15.6 0.005592 0.200055
22 TIDL_ConvolutionLayer 31.5285 0.990145 16.9 0.030157 0.615458
23 TIDL_EltWiseLayer 63.0570 0.985977 15.5 0.017564 0.466482
24 TIDL_ConvolutionLayer 104.5296 0.980982 14.2 0.006974 0.222567
25 TIDL_ConvolutionLayer 63.1702 0.987106 15.9 0.031259 0.406923
26 TIDL_ConvolutionLayer 45.6950 0.985622 15.4 0.059418 0.466563
27 TIDL_EltWiseLayer 45.6950 0.978477 13.7 0.015119 0.558569
28 TIDL_ConvolutionLayer 155.8160 0.975288 13.0 0.006958 0.216080
29 TIDL_ConvolutionLayer 5.3386 0.983551 14.6 0.298886 1.991327
30 TIDL_EltWiseLayer 10.6773 0.983073 14.6 0.172177 2.074892
31 TIDL_PoolingLayer 21.3546 -0.079204 -1.7 1.201779 6.981273
32 TIDL_ReshapeLayer 21.3546 0.991290 17.0 0.102834 0.620338
33 TIDL_InnerProductLayer 3.7321 0.994110 19.0 0.269856 1.197404

====================================================================================================
QUALITY ASSESSMENT
====================================================================================================
 GOOD: Int8 model matches float model reasonably well

Correlation > 0.999: 1 layers
Correlation > 0.99: 15 layers
Correlation > 0.95: 32 layers
Correlation < 0.90: 1 layers

Hi Zhanying,

Zhanying Chen said:

The .y trace files were generated as follows:

1. Set debug_level=4 and tensor_bits=8.

2. Run:

   python3 onnxrt_ep.py -c -m cl-ort-resnet18-v1
   

3. Then run:

   python3 onnxrt_ep.py -m cl-ort-resnet18-v1
   

4. Copy the generated .y files from the /tmp directory

I will note that the .y files will also be dequantized into _float.bin when debug_level=4, even for tensor_bits=8

Zhanying Chen said:

At this point, the most likely explanations appear to be:

1. The Layer 31 trace is being read incorrectly.

2. The Layer 31 trace data is stored in a special layout or ordering that differs from the other layers.

I think it is the former, but let me follow up with the backend development team to confirm this point. 

Since your output is reasonably accurate here, then I think the issue with layer 31 comparison is not critical to debug. 

Zhanying Chen said:

31 TIDL_PoolingLayer 21.3546 -0.079204 -1.7 1.201779 6.981273
32 TIDL_ReshapeLayer 21.3546 0.991290 17.0 0.102834 0.620338
33 TIDL_InnerProductLayer 3.7321 0.994110 19.0 0.269856 1.197404

If anything, I would say layer 33 could be used in 16-bit quantization with the settings "advanced_options:output_feature_16bit_names_list" and "advanced_options:params_16bit_names_list" [1]

[1] github.com/.../model_compilation.md

Thank you for your reply.

I am currently using the 11_01_06_00 release tag, and my debugging work is based on the onnxrt_ep.py script.

In your recent recommendations, most of the reference materials seem to come from the master branch, where the debugging examples are based on basic_example.py instead of onnxrt_ep.py.

I would like to understand whether it is safe to use the scripts from the master branch while keeping the rest of my environment on 11_01_06_00. Specifically:

1. Can I use the master branch's basic_example.py script directly in my current 11_01_06_00 environment?

2. If I do so, could it generate artifacts that are incompatible with the 11_01_06_00 toolchain or runtime?

3. Our target SoC is AM67A, and all of our current compilation and deployment environments are based on 11_01_06_00. For debugging accuracy and quantization issues, which scripts would you recommend using: onnxrt_ep.py from 11_01_06_00 or basic_example.py from master?

4. Since the master branch contains much more debugging documentation and troubleshooting guidance than the 11_01_06_00 release, which materials should I follow when investigating trace, quantization, and accuracy issues? Should I rely only on documentation corresponding to 11_01_06_00, or are the master branch debugging guides generally applicable to 11_01_06_00 as well?

I would appreciate your guidance on the recommended debugging workflow for an AM67A project based on the 11_01_06_00 release.

Hello,

Please stick to the examples within your 11_01_06_00 branch. They did a major overhaul of the examples and documentation in 11_02_xx and onwards. The major points within the documentation still stand, but exact scripts are a bit different (especially examples).

The majority of the documentation is still there in 11_01_06_00, but may have a different name for the page or be in a different directory. The newer documentation is more readable, in my opinion. 

Zhanying Chen said:

• Can I use the master branch's basic_example.py script directly in my current 11_01_06_00 environment?

You likely can, but there is no guarantee this will work. I think it should function

Zhanying Chen said:

• If I do so, could it generate artifacts that are incompatible with the 11_01_06_00 toolchain or runtime?

As long as you are pointing TIDL_TOOLS_PATH environment variable to a set of tidl_tools that was downloaded from 11_01_06_00 branch, they the artifacts generated will be compatible with 11.1 SDK

Zhanying Chen said:

• Our target SoC is AM67A, and all of our current compilation and deployment environments are based on 11_01_06_00. For debugging accuracy and quantization issues, which scripts would you recommend using: onnxrt_ep.py from 11_01_06_00 or basic_example.py from master?

Okay, here's a key point. AM67A does not have an 11.1 SDK release. The last Edge AI Linux SDK for this part was 11.0, so you should either use branches from rel_11_0 or use backwards compatible releases (last number is odd, like 11_01_07 and use appropriate update_target.sh on the EVM). 

In this scenario onnxrt_ep.py is the same in 11_01_07, 11_01_06, and prior releases. I would recommend using onnxrt_ep.py in your case. 

Zhanying Chen said:

1. Since the master branch contains much more debugging documentation and troubleshooting guidance than the 11_01_06_00 release, which materials should I follow when investigating trace, quantization, and accuracy issues? Should I rely only on documentation corresponding to 11_01_06_00, or are the master branch debugging guides generally applicable to 11_01_06_00 as well?

Debugging guidelines in newer releases are also correct for 11_01_06. The only difference is when the debugging guidelines point you to additional scripts (Python scripts, usually) that may only exist in the newer release of the repository

Reference for versioning and compatibility: 

• https://github.com/TexasInstruments/edgeai-tidl-tools/blob/11_02_07_00/docs/sdk_version_compatibility_table.md
• https://github.com/TexasInstruments/edgeai-tidl-tools/blob/11_02_07_00/scripts/setup/README.md#target-setup-scripts-overview 
• https://github.com/TexasInstruments/edgeai-tidl-tools/blob/11_01_07_00/docs/update_target.md 
  • This one is most correct in your AM67A scenario. AM67A uses TISDK_IMAGE="edgeai"

Thank you very much for your reply.

I tried using the basic_example.py script from the master branch for both compilation and inference, while still using the 11_01_06_00 release tag and runtime environment.

I compared:

1. The golden output and the TIDL float output for every layer.

2. The TIDL float output and the TIDL fixed output for every layer.

(The comparison results from the second case were already provided in my previous reply.)

In both comparisons, Layer 31 (Pooling) consistently shows the lowest correlation, while the correlations of almost all other layers are around 0.98.

Since the issue is consistently observed at the same layer, I would like to understand whether there is any special handling for the cl-ort-resnet18-v1 (resnet18_opset9.onnx) model during TIDL compilation or trace generation.

Specifically:

• Is it possible that the trace output for this layer does not directly correspond to the output of the original ONNX pooling layer?

• Is there any layer fusion, optimization, data layout conversion, or other internal processing that could affect the trace output associated with Layer 31?

• Is it possible that the backend is reading or generating the trace data incorrectly, resulting in a mismatch between the trace output and the actual layer output?

• Could the trace data be mapped to a different internal layer than expected?

Since this issue is blocking our accuracy analysis, could you please help confirm this as soon as possible?

Any suggestions on how to further investigate this issue would be greatly appreciated.

Hi Zhanying,

Zhanying Chen said:

• The golden output and the TIDL float output for every layer.

• The TIDL float output and the TIDL fixed output for every layer.

Just as a quick clarification: If you run the model wtih debug_level=4, you will get .Y files and _float.bin files under /tmp. In this case, the _float.bin file is the same data as the .Y file, but it has been dequantized (cast to fp32, multiply by scaler, and optionally add a bias term if the quantization settings were for asymmetric quantization). 

The first bullet is a good comparison, but the second is not, because you would be comparing the same data against itself. For point 2, the standard approach is to compile the model with tensor_bits=32, run the model to collect _float.bin files, and then RECOMPILE with tensor_bits=8 (or 16), run the model to collect _float.bin's that have been dequantized, and finally compare between these two sets of _float.bin files for each layer. 

I am not sure if your correlation data was from the approach in bullet 1 or 2

 I do recall seeing similar behavior before for certain layers like pooling. I have seen layers like data-shaping layers not directly correlate with the original floating point from the model in OSRT (e.g. ONNXRT). Let me see if there is any workaround to enable this correlation or for a more concrete explanation. 

Layer 32 has very high correlation. This removes concern I have about layer 31's poor correlation. 

I do not think that different layers are being correlated (i.e. layers themselves mismatch). It is more likely do to with how pooling layer data is represented

BR,
Reese

Hi Zhanying,

I spoke with our development team. This is a known issue for GlobalAveragePool (which I believe is the case for this last pooling layer within a resnet classification model). It is writing incorrect trace data to that output file. This issue has not been fixed in an available release yet. 

Since the following layer shows correct accuracy, I would recommend ignoring this poor correlation on the GlobalAveragePool -- it is the result of a bug.