PROCESSOR-SDK-AM62A: the AM62A model compiled by edgeai-benchmark outputs all zeros on AM62A board

Yue Lee

Part Number: PROCESSOR-SDK-AM62A
Other Parts Discussed in Thread: AM68A

Tool/software:

When using edgeai-benchmark to compile semantic segmentation models for AM62A and AM68A, then running them on development boards with edgeai-gst-apps, the AM68A model works normally on j721s2 board but the AM62A model outputs all zeros on AM62A board. How to resolve this issue?

edgeai-benchmark version is: 09_02_00_06
evm sdk version is :09_02_00_05

the command to compile is :
./run_custom_pc_seg.sh AM62A

the config files are here seg_sample.tar.gz

seg_sample.tar.gz

1 month ago

0 Qutaiba Saleh 1 month ago

TI__Expert 4670 points

Hi Yue,

It is our pleasure to help you.

Would you please try the following:

use the ./run_custom_pc_seg.sh on the evm so that we use the same code in the inference. The goal is to make sure the model itself is compiled with good accuracy using the same code.
Another experiment is to run the model on the ARM cores of the EVM just to see what kind of output you will get. This is to test if the wrong results you are getting are from the model itself or caused by the compilation. To run the model on the ARM cores using edgeai-gst-apps, set the target value in the configs/gst_plugins_map.yaml to arm instead of dsp: https://github.com/TexasInstruments/edgeai-gst-apps/blob/main/configs/gst_plugins_map.yaml#L211

Please, provide more details of the results you get from the experiments above.

Best regards,

Qutaiba

0 Yue Lee 1 month ago in reply to Qutaiba Saleh

Prodigy 20 points

Thank you very much for your suggestions.‌ I followed your advice and conducted two separate experiments.

‌First Experiment:‌
Ran the ./run_custom_pc_seg.sh script for inference on the EVM, but it did not produce the correct results. The output image is as follows:

‌Second Experiment:‌
Modified the target value in configs/gst_plugins_map.yaml to arm, and this time, it produced the correct results. The output image is as follows:

‌Now, can you tell me what the issue is?‌ What should I modify to get the correct results when the target is set to dsp?

+1 Qutaiba Saleh 1 month ago in reply to Yue Lee

TI__Expert 4670 points

Hi Yue Lee,

Thank for providing the results of the experiment. I ran the model you shared with me and I was able to reproduce the issue you are seeing.

Debugging the model shows that there is an issue in the very last layer. All the layers in the model has non-zero output but the last layer has zero output.

A quick way to fix this is to compile that last a few layers with 16 bits instead of 8 bits. For a quick test, I suggest that you recompile the entire model with 16 bits to make sure that the model will work fine. After that try to see if only compiling the last layers with 16 bits will work.

Here are some resourced for you:

Debug the model and looking at intermediate output of the layers: https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/docs/tidl_osr_debug.md#steps-to-debug-error-scenarios-for-targetevmdevice-execution
Changing quantization parameters 8 bits to 16 bits: https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/docs/tidl_fsg_quantization.md#c-mixed-precision
Changing the quantization bits in edgeai-benchmark for the entire model can be done by changing a single line of code at: https://github.com/TexasInstruments/edgeai-tensorlab/blob/main/edgeai-benchmark/settings_base.yaml#L13.

Please, share any results or questions you might have.

Best regards,

Qutaiba

Processors

Processors forum

PROCESSOR-SDK-AM62A: the AM62A model compiled by edgeai-benchmark outputs all zeros on AM62A board