Part Number: AM69A
Tool/software:
I created a simple model with single convolution layer, two kernels, one input and two output channels:
import tensorflow as tf
# Define the model
model = tf.keras.Sequential([
tf.keras.layers.Input(shape=(5, 5, 1)),
tf.keras.layers.Conv2D(
filters=2,
kernel_size=(9, 1),
padding='same'
)
])
# Print the model summary
model.summary()
# Convert the model to TFLite format
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
# Save the TFLite model
with open('single_conv_layer_2_kernels.tflite', 'wb') as f:
f.write(tflite_model)
You can find single_conv_layer_2_kernels.tflite file in the archive:
single_conv_layer_2_kernels_tflite.zip
Here is the Python code used for model compilation:
import tflite_runtime.interpreter as tf
import numpy as np
##########################################
### Script parameters ###
##########################################
tflite_model_path = 'single_conv_layer_2_kernels.tflite'
options = {}
options['artifacts_folder'] = './model-artifacts-dir/'
options['tidl_tools_path'] = '/home/root/tidl_tools'
options['debug_level'] = 3
##########################################
# Load the TFLite model
# interpreter = tf.lite.Interpreter(model_path=tflite_model_path)
interpreter = tf.Interpreter(
model_path=tflite_model_path,
experimental_delegates=[tf.load_delegate('tidl_model_import_tflite.so', options)])
# interpreter = tf.Interpreter(
# model_path=tflite_model_path,
# experimental_delegates=[tf.load_delegate('libtidl_tfl_delegate.so', options)])
interpreter.allocate_tensors()
# Get input and output tensors
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Prepare a sample input (5x5 matrix with a single channel)
input_data = np.random.rand(1, 5, 5, 1).astype(np.float32)
# Set the tensor to point to the input data to be inferred
interpreter.set_tensor(input_details[0]['index'], input_data)
# Run the inference
interpreter.invoke()
# Get the output tensor
output_data = interpreter.get_tensor(output_details[0]['index'])
print("Output:")
print(output_data)
And complete console log for model compilation with debug_level=3:
tidl_tools_path = /home/root/tidl_tools
artifacts_folder = ./model-artifacts-dir/
tidl_tensor_bits = 8
debug_level = 3
num_tidl_subgraphs = 16
tidl_denylist =
tidl_denylist_layer_name =
tidl_denylist_layer_type =
tidl_allowlist_layer_name =
model_type =
tidl_calibration_accuracy_level = 7
tidl_calibration_options:num_frames_calibration = 20
tidl_calibration_options:bias_calibration_iterations = 50
mixed_precision_factor = -1.000000
model_group_id = 0
power_of_2_quantization = 2
ONNX QDQ Enabled = 0
enable_high_resolution_optimization = 0
pre_batchnorm_fold = 1
add_data_convert_ops = 0
output_feature_16bit_names_list =
m_params_16bit_names_list =
m_single_core_layers_names_list =
Inference mode = 0
Number of cores = 1
reserved_compile_constraints_flag = 1601
partial_init_during_compile = 0
ti_internal_reserved_1 =
========================= [Model Compilation Started] =========================
Model compilation will perform the following stages:
1. Parsing
2. Graph Optimization
3. Quantization & Calibration
4. Memory Planning
============================== [Version Summary] ==============================
-------------------------------------------------------------------------------
| TIDL Tools Version | 10_00_04_00 |
-------------------------------------------------------------------------------
| C7x Firmware Version | 10_00_02_00 |
-------------------------------------------------------------------------------
============================== [Parsing Started] ==============================
[TIDL Import] [PARSER] WARNING: Network not identified as Object Detection network : (1) Ignore if network is not Object Detection network (2) If network is Object Detection network, please specify "model_type":"OD" as part of OSRT compilation options
[TIDL Import] WARNING: Parameters - Kernel 9x1, Stride 1x1, dilation 1x1, Pad 4x0, Bias 1 in [] has gone through limited verification
[TIDL Import] [PARSER] SUPPORTED: Supported TIDL layer type --- 1 Tflite layer type --- 3 layer output name--- StatefulPartitionedCall_1:0 -- [tidl_tfLiteRtImport_core.cpp, 3090]
Total Nodes = 1
-------------------------------------------------------------------------------
| Core | No. of Nodes | Number of Subgraphs |
-------------------------------------------------------------------------------
| C7x | 1 | 1 |
| CPU | 0 | x |
-------------------------------------------------------------------------------
============================= [Parsing Completed] =============================
In TIDL_tfliteRtImportInit subgraph_id=3
Layer 0, subgraph id 3, name=StatefulPartitionedCall_1:0
Layer 1, subgraph id 3, name=serving_default_keras_tensor:0
In TIDL_tfliteRtImportNode, TIDL Layer type - 1, Tflite builtin code type - 3
==================== [Optimization for subgraph_3 started] ====================
In TIDL_runtimesOptimizeNet: LayerIndex = 3, dataIndex = 2
----------------------------- Optimization Summary -----------------------------
--------------------------------------------------------------------------------
| Layer | Nodes before optimization | Nodes after optimization |
--------------------------------------------------------------------------------
| TIDL_ConvolutionLayer | 1 | 1 |
--------------------------------------------------------------------------------
=================== [Optimization for subgraph_3 completed] ===================
In TIDL_runtimesPostProcessNet
[TIDL Import] WARNING: Parameters - Kernel 9x1, Stride 1x1, dilation 1x1, Pad 4x0, Bias 1 in [StatefulPartitionedCall_1:0] has gone through limited verification
************ in TIDL_subgraphRtCreate ************
The soft limit is 10240
The hard limit is 10240
MEM: Init ... !!!
MEM: Init ... Done !!!
0.0s: VX_ZONE_INIT:Enabled
0.2s: VX_ZONE_ERROR:Enabled
0.2s: VX_ZONE_WARNING:Enabled
0.3210s: VX_ZONE_INIT:[tivxInit:190] Initialization Done !!!
************ TIDL_subgraphRtCreate done ************
tidl_tfLiteRtImport_delegate.cpp Invoke 526
******* In TIDL_subgraphRtInvoke ********
0 1.00000 0.01707 0.99761 6
1 1.00000 -1.20642 1.02418 6
Layer, Layer Cycles,kernelOnlyCycles, coreLoopCycles,LayerSetupCycles,dmaPipeupCycles, dmaPipeDownCycles, PrefetchCycles,copyKerCoeffCycles,LayerDeinitCycles,LastBlockCycles, paddingTrigger, paddingWait,LayerWithoutPad,LayerHandleCopy, BackupCycles, RestoreCycles,Multic7xContextCopyCycles,
1, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
Sum of Layer Cycles 0
Sub Graph Stats 11.000000 3356.000000 246.000000
******* TIDL_subgraphRtInvoke done ********
************ Frame index 1 : Running float inference ****************
tidl_tfLiteRtImport_delegate.cpp Invoke 647
Output:
[[[[-0.4975459 0.15360159]
[-0.74099416 0.54164875]
[-0.8277656 0.54985464]
[-1.1647784 0.7238448 ]
[-1.2064233 1.0241802 ]]
[[-0.33638442 0.32774824]
[-0.3055907 -0.08622443]
[-0.35362354 0.1582561 ]
[-0.3864597 -0.10611226]
[-0.7554928 0.09201628]]
[[-0.3587169 -0.05090892]
[-0.13165918 -0.03363836]
[-0.36763018 -0.09886585]
[-0.38701996 0.02807827]
[-0.43973437 -0.02877097]]
[[-0.14468591 0.13824506]
[-0.53635097 0.39574966]
[-0.6342068 0.5474889 ]
[-0.7425537 0.59489006]
[-1.0733385 0.7317146 ]]
[[-0.4419533 0.14727505]
[-0.32524627 0.06330962]
[-0.55514824 0.09715606]
[-0.57541275 0.03066342]
[-0.4509401 0.06172246]]]]
************ in TIDL_subgraphRtDelete ************
MEM: Deinit ... !!!
MEM: Alloc's: 27 alloc's of 115313861 bytes
MEM: Free's : 27 free's of 115313861 bytes
MEM: Open's : 0 allocs of 0 bytes
MEM: Deinit ... Done !!!
I want to draw your attention that there is no Quantization & Calibration and Memory Planning compilation stages in the console log above. Only Parsing and Graph Optimization stages are present. Here is how artifacts directory looks like after the compilation:
root@1ae02087d69a:/home/root/custom_models/model-artifacts-dir# ls
tempDir
root@1ae02087d69a:/home/root/custom_models/model-artifacts-dir# cd tempDir/
root@1ae02087d69a:/home/root/custom_models/model-artifacts-dir/tempDir# ls
3_calib_raw_data.bin 3_tidl_net.bin 3_tidl_net.bin.svg graphvizInfo.txt
3_tidl_io_1.bin 3_tidl_net.bin.layer_info.txt 3_tidl_net.bin_netLog.txt
I used the following Python code for inference:
import tflite_runtime.interpreter as tf
import numpy as np
##########################################
### Script parameters ###
##########################################
tflite_model_path = 'single_conv_layer_2_kernels.tflite'
options = {}
options['artifacts_folder'] = './model-artifacts-dir/'
options['tidl_tools_path'] = '/home/root/tidl_tools'
options['debug_level'] = 3
##########################################
# Load the TFLite model
# interpreter = tf.lite.Interpreter(model_path=tflite_model_path)
# interpreter = tf.Interpreter(
# model_path=tflite_model_path,
# experimental_delegates=[tf.load_delegate('tidl_model_import_tflite.so', options)])
interpreter = tf.Interpreter(
model_path=tflite_model_path,
experimental_delegates=[tf.load_delegate('libtidl_tfl_delegate.so', options)])
interpreter.allocate_tensors()
# Get input and output tensors
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Prepare a sample input (5x5 matrix with a single channel)
input_data = np.random.rand(1, 5, 5, 1).astype(np.float32)
# Set the tensor to point to the input data to be inferred
interpreter.set_tensor(input_details[0]['index'], input_data)
# Run the inference
interpreter.invoke()
# Get the output tensor
output_data = interpreter.get_tensor(output_details[0]['index'])
print("Output:")
print(output_data)
Here is the full inference log with debug_level=3:
****** In DelegatePrepare ******
******** WARNING ******* : Could not open ./model-artifacts-dir///allowedNode.txt for reading... Entire model will run on ARM without any delegation to TIDL !
Number of subgraphs:1 , 0 nodes delegated out of 1 nodes
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Output:
[[[[-1.0127023 0.47760093]
[-1.0161562 0.3824313 ]
[-0.8853292 0.75817144]
[-0.9115631 0.43699983]
[-1.0880611 0.5449653 ]]
[[-0.6603909 0.325401 ]
[-0.97620225 0.740861 ]
[-0.5387642 -0.09967738]
[-0.7799023 0.5132353 ]
[-0.36896044 -0.06385579]]
[[-0.2453596 -0.17629509]
[-0.5779659 -0.13249311]
[-0.29097825 0.09676507]
[-0.8343738 0.13079968]
[-0.91081744 0.39155906]]
[[-0.35439658 0.24490732]
[-0.23750313 0.0976485 ]
[-0.74538195 0.44951007]
[-0.42277795 0.24323536]
[-0.607256 0.45970833]]
[[-0.7101433 0.36863968]
[-0.7668315 0.421872 ]
[-0.17297046 -0.03946714]
[-0.4834563 0.0604281 ]
[-0.37863034 -0.28085268]]]]
Hence, inference with delegation to TIDL is not possible due to the absence of allowedNode.txt file in the artifacts directory.
