PROCESSOR-SDK-AM69A: Optimization iResnet101

I am running on TIDLRT and have not changed anything in the model, you can see more information here:  PROCESSOR-SDK-AM69A: Issue when switching to using the tidlcpp library . 

Sorry I mistakenly ticked Resolved so I will create a new topic: e2e.ti.com/.../processor-sdk-am69a-optimization-resnet101

Hi Khoa,

I am going to close the new ticket and continue discussion on this one to minimize confusion. We are recreating your issue right now. Rohit will update on his status.

Warm regards,

Christina

Hello Khoa,

I am looking into running your Resnet101 model on the AM69A device. Our open source runtime currently doesn’t support compilation for your model so TIDLRT is viable here.

Can you share your compilation steps. More specifically your import and inference txt files and respective artifacts that you are using to run on the SOC. This will help diagnose the issue.

Best,

Rohit

Hi Rohit, 
i have build my application 

on the device and compiled resnet101 model in a 10_01_00_02 environment and everything is working.  It produces the out files in edgeai-tidl-tools/tidl_tools/out.   The directory out/ must be created before the import.  Please use the following import configuration file and scp -r out to <your_device_ip>/opt/arrow (example: scp -r out root@128.247.79.163:/opt/arrow) on the device. 

Import:

./tidl_model_import.out import_resnet101.txt
========================= [Model Compilation Started] =========================

Model compilation will perform the following stages:
1. Parsing
2. Graph Optimization
3. Quantization & Calibration
4. Memory Planning

============================== [Version Summary] ==============================

-------------------------------------------------------------------------------
| TIDL Tools Version | 10_01_00_01 |
-------------------------------------------------------------------------------
| C7x Firmware Version | 10_00_02_00 |
-------------------------------------------------------------------------------

ONNX model (Proto) file : /shared/resnet101_simp.onnx
TIDL network file : out/tidl_net.bin
TIDL IO info file : out/tidl_io_buff
Current ONNX OpSet version : 11
============================ [Optimization started] ============================

[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_1's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_4's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_10's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_15's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_20's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_26's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_31's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_36's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_41's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_46's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_51's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_56's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_61's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_66's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_71's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_76's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_81's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_86's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_92's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_97's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_102's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_107's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_112's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_117's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_122's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_127's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_132's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_137's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_142's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_147's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_152's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_157's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_162's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_167's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_172's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_177's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_182's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_187's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_192's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_197's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_202's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_207's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_212's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_217's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_222's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_227's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_232's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_237's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_243's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: PReLU Layer PRelu_248's bias cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random
[TIDL Import] [PARSER] WARNING: Batch Norm Layer BatchNormalization_254's coeff cannot be found(or not match) in coeff file, Random bias will be generated only for evaluation usage. Results are all random

----------------------------- Optimization Summary -----------------------------
---------------------------------------------------------------------------------
| Layer | Nodes before optimization | Nodes after optimization |
---------------------------------------------------------------------------------
| TIDL_PReLULayer | 50 | 0 |
| TIDL_EltWiseLayer | 49 | 49 |
| TIDL_ConvolutionLayer | 103 | 103 |
| TIDL_FlattenLayer | 1 | 1 |
| TIDL_InnerProductLayer | 1 | 1 |
| TIDL_BatchNormLayer | 51 | 100 |
---------------------------------------------------------------------------------

=========================== [Optimization completed] ===========================

-------- Running Calibration in Float Mode to Collect Tensor Statistics --------
[=============================================================================] 100 %

------------------ Fixed-point Calibration Iteration [1 / 1]: ------------------
[=============================================================================] 100 %

==================== [Quantization & Calibration Completed] ====================

========================== [Memory Planning Started] ==========================

------------------------- Network Compiler Traces ------------------------------
Successful Memory Allocation
Successful Workload Creation

========================= [Memory Planning Completed] =========================

[TIDL Import] WARNING: Size larger than (2048*2048) in [Gemm_253] is not optimal
======================== Subgraph Compiled Successfully ========================

modelType           = 2
numParamBits            = 8
numFeatureBits          = 8


inputNetFile        = "/shared/resnet101_simp.onnx"
outputNetFile = "out/tidl_net.bin"
outputParamsFile = "out/tidl_io_buff"


addDataConvertToNet = 0
foldPreBnConv2D         = 0
rawDataInElementType = 0
inElementType = 0
outElementType = 0


tidlStatsTool = "./PC_dsp_test_dl_algo.out"
perfSimTool = "./ti_cnnperfsim.out" 
graphVizTool = "./tidl_graphVisualiser.out"

inWidth             = 112
inHeight            = 112
inNumChannels       = 3
numFrames           = 1
inFileFormat        = 0
inData  =   "./airshow.jpg"
perfSimConfig = "device_config.cfg"
debugTraceLevel   

e2e.ti.com/.../resnet101.onnx

root@am69-sk:/home/root/edgeai-tidl-tools/bin/Release# ./tidlrt_resnet -f /opt/arrow/out --image face_aligned_1.jpg
Model Files names : /opt/arrow/out/tidl_net.bin,/opt/arrow/out/tidl_io_buff1.bin
APP: Init ... !!!
81266.466307 s: MEM: Init ... !!!
81266.466366 s: MEM: Initialized DMA HEAP (fd=5) !!!
81266.466511 s: MEM: Init ... Done !!!
81266.466533 s: IPC: Init ... !!!
81266.501083 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
81266.509069 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
81266.509174 s: VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
81266.509187 s: VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
81266.509197 s: VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
81266.510039 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-0
81266.510170 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-1
81266.510263 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-2
81266.510369 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-3
81266.510381 s: VX_ZONE_INFO: [tivxInitLocal:126] Initialization Done !!!
81266.510394 s: VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
invoked
average time: 332.367 ms
Feature vector statistics:
Dimension: 512
Min value: -3.00667e+23
Max value: 2.8864e+23
Mean value: 1.70299e+21
Feature vector saved to face_aligned_1.jpg.features.txt
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
81267.488617 s: IPC: Deinit ... !!!
81267.489629 s: IPC: DeInit ... Done !!!
81267.489661 s: MEM: Deinit ... !!!
81267.489672 s: DDR_SHARED_MEM: Alloc's: 7 alloc's of 68044312 bytes
81267.489680 s: DDR_SHARED_MEM: Free's : 7 free's of 68044312 bytes
81267.489686 s: DDR_SHARED_MEM: Open's : 0 allocs of 0 bytes
81267.489696 s: MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

Regards,

Tung le

Hi Tung,

I've tried running the model on 10_01_04, but it doesn't seem to be working. Could you provide more information about the executable you're using to run on EVM? Specifically, I noticed you have an executable called ./tidlrt_resnet. When we use ./TI_DEVICE_armv8_test_dl_algo_host_rt.out in /opt/tidl_test/, the model doesn't execute correctly.

I'd appreciate any insight you can provide to help us determine if this issue needs to be escalated to our development team.

Best,
Rohit

Hi Rohit, 

Previously, I successfully tested the TI_DEVICE_armv8_test_dl_algo_host_rt target. Now I need to run the model using the TIDL Runtime from source code. I’m taking the following steps:

1. Cloned the repository:

   git clone github.com/.../edgeai-tidl-tools.git
   

2. Checked out branch 10_01_00_02.

3. Added my code into /opt/edgeai-tidl-tools/examples/tidlrt_cpp to build an application that runs the CMC ResNet-101 model.
   7652.resnet.zip
   you can see more information here: e2e.ti.com/.../processor-sdk-am69a-issue-when-switching-to-using-the-tidlcpp-library

Hi Rohit Rao , We are waiting for your response

Hey it's not within our scope to debug your code's issues. If you have any specific questions on how to implement one of our frameworks, I can help with that.

 Dear Rohit Rao Christina Kuruvilla 

We have an iResNet-101 model that has already been trained and runs very well on platforms such as Nvidia, Qualcomm, and Rockchip. We are currently developing a new product based on the TI platform, and it is encountering issues.

I believe that we are not the only ones — your new customers may also face similar problems. Therefore, we hope that TI can provide us with guidance and documentation that directly relates to this issue, so that we can study and resolve it effectively:

1. Analyze the causes of long inference time (e.g., model analysis, layers, functions)

2. Methods to optimize the model for the TI platform

3. The process for retraining a model to be compatible with the TI platform.

I would like to emphasize that we are extremely serious about our product development. We have already purchased the SK-AM69 and received 15 sets of AM69A chips for SMT. The design has reached the PCB stage, and we are in the process of purchasing the SK-AM68.

Hello Khoa,

Understood, I am looking into a potential solution for you. Please give me some time to address your concerns and devise a solution.

Best,

Rohit

Hi Rohit Rao 

We have reimplemented the TIDL version 11_00_06_00,
and the firmware version is 11_00_00_00.
The inference time has significantly improved to 36ms, which is acceptable.

However, the accuracy is extremely poor — currently only 1%,
while the accuracy of the original ONNX model is 99.8%.

Could you please suggest possible causes of this issue and how we might resolve it?