This thread has been locked.
If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.
Tool/software:
Hi team:
I try to run example code to convert model by https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/examples/jupyter_notebooks/custom-model-onnx.ipynb
but I get error below:
*************** EP Error *************** EP Error Unknown Provider Type: TIDLCompilationProvider when using ['TIDLCompilationProvider', 'CPUExecutionProvider'] Falling back to ['CPUExecutionProvider'] and retrying. ****************************************
I check the TIDL_TOOLS_PATH is set, and make sure the path has "tidl_model_import_onnx.so"
Please help me to fix this symptom
Thanks for your kindly help.
[backup]
code below:
import os import tqdm import cv2 import numpy as np import onnxruntime as rt import shutil from scripts.utils import imagenet_class_to_name, download_model import matplotlib.pyplot as plt from pathlib import Path from IPython.display import Markdown as md from scripts.utils import loggerWritter from scripts.utils import get_svg_path import onnx def preprocess(image_path): # read the image using openCV img = cv2.imread(image_path) # convert to RGB img = img[:,:,::-1] # Most of the onnx models are trained using # 224x224 images. The general rule of thumb # is to scale the input image while preserving # the original aspect ratio so that the # short edge is 256 pixels, and then # center-crop the scaled image to 224x224 orig_height, orig_width, _ = img.shape short_edge = min(img.shape[:2]) new_height = (orig_height * 256) // short_edge new_width = (orig_width * 256) // short_edge img = cv2.resize(img, (new_width, new_height), interpolation=cv2.INTER_CUBIC) startx = new_width//2 - (224//2) starty = new_height//2 - (224//2) img = img[starty:starty+224,startx:startx+224] # apply scaling and mean subtraction. # if your model is built with an input # normalization layer, then you might # need to skip this img = img.astype('float32') for mean, scale, ch in zip([128, 128, 128], [0.0078125, 0.0078125, 0.0078125], range(img.shape[2])): img[:,:,ch] = ((img.astype('float32')[:,:,ch] - mean) * scale) img = np.expand_dims(img,axis=0) img = np.transpose(img, (0, 3, 1, 2)) return img calib_images = [ 'sample-images/elephant.bmp', 'sample-images/bus.bmp', 'sample-images/bicycle.bmp', 'sample-images/zebra.bmp', ] output_dir = 'custom-artifacts/onnx/resnet18_opset9.onnx' onnx_model_path = 'models/public/onnx/resnet18_opset9.onnx' download_model(onnx_model_path) onnx.shape_inference.infer_shapes_path(onnx_model_path, onnx_model_path) #compilation options - knobs to tweak num_bits =8 accuracy =1 log_dir = Path("logs").mkdir(parents=True, exist_ok=True) # stdout and stderr saved to a *.log file. with loggerWritter("logs/custon-model-onnx"): # model compilation options compile_options = { 'tidl_tools_path' : os.environ['TIDL_TOOLS_PATH'], 'artifacts_folder' : output_dir, 'tensor_bits' : num_bits, 'accuracy_level' : accuracy, 'advanced_options:calibration_frames' : len(calib_images), 'advanced_options:calibration_iterations' : 3, # used if accuracy_level = 1 'advanced_options:add_data_convert_ops' : 1, 'debug_level' : 1, #'deny_list' : "MaxPool" #Comma separated string of operator types as defined by ONNX runtime, ex "MaxPool, Concat" } # create the output dir if not present # clear the directory os.makedirs(output_dir, exist_ok=True) for root, dirs, files in os.walk(output_dir, topdown=False): [os.remove(os.path.join(root, f)) for f in files] [os.rmdir(os.path.join(root, d)) for d in dirs] so = rt.SessionOptions() EP_list = ['TIDLCompilationProvider','CPUExecutionProvider'] sess = rt.InferenceSession(onnx_model_path ,providers=EP_list, provider_options=[compile_options, {}], sess_options=so) input_details = sess.get_inputs() for num in tqdm.trange(len(calib_images)): output = list(sess.run(None, {input_details[0].name : preprocess(calib_images[num])}))[0] ##optional #subgraph_link =get_svg_path(output_dir) #for sg in subgraph_link: # hl_text = os.path.join(*Path(sg).parts[4:]) # sg_rel = os.path.join('../', sg) # display(md("[{}]({})".format(hl_text,sg_rel))) # #EP_list = ['TIDLExecutionProvider','CPUExecutionProvider'] #print("[ken debug] 5 ") #sess = rt.InferenceSession(onnx_model_path ,providers=EP_list, provider_options=[compile_options, {}], sess_options=so) ##Running inference several times to get an stable performance output #for i in range(5): # output = list(sess.run(None, {input_details[0].name : preprocess('sample-images/elephant.bmp')})) # #for idx, cls in enumerate(output[0].squeeze().argsort()[-5:][::-1]): # print('[%d] %s' % (idx, '/'.join(imagenet_class_to_name(cls)))) # #from scripts.utils import plot_TI_performance_data, plot_TI_DDRBW_data, get_benchmark_output #stats = sess.get_TI_benchmark_data() #fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(10,5)) #plot_TI_performance_data(stats, axis=ax) #plt.show() #print("[ken debug] 6 ") #tt, st, rb, wb = get_benchmark_output(stats) #print(f'Statistics : \n Inferences Per Second : {1000.0/tt :7.2f} fps') #print(f' Inference Time Per Image : {tt :7.2f} ms \n DDR BW Per Image : {rb+ wb : 7.2f} MB')
Hi, please check two things.
1. Make sure your <edgeai-tidl-tools>/tidl_tools/ is there and has some version of the following.
tidl_tools$ ls -1
device_config.cfg
ignore.log
itidl_rt.h
itvm_rt.h
libtidl_onnxrt_EP.so
libtidl_tfl_delegate.so
libvx_tidl_rt.so
libvx_tidl_rt.so.1.0
nc_infofile.txt
netbufinfo.bin
osrt_deps
PC_dsp_test_dl_algo.out
ti_cnnperfsim.out
tidl_graphVisualiser.out
tidl_graphVisualiser_runtimes.out
tidl_model_import_onnx.so
tidl_model_import.out
tidl_model_import_relay.so
tidl_model_import.so
tidl_model_import_tflite.so
tidl_model_secure.out
version.txt
If not, set SOC to your device and run "source ./setup.sh" from our edgeai-tidl-tools/ directory (this will take some time).
2. If edgeai-tidl-tools/tidl_tools exists and it has the above files, then set TIDL_TOOLS_PATH=/<FULL_PATH_TO_EDGEAI_TIDL_TOOLS>/edgeai-tidl-tools/tidl_tools.
Hi Chris:
I check the tidl_tools file below, the file is correct (SOC = am67a)
and already set the TIDL_TOOLS_PATH to full path of tidl_tools
My TIDL version is 10.00.04.00, and run on PC(ubuntu 22.04)
please help me to check,
Thanks for your kindly help.
Please try this:
Change:
from scripts.utils import loggerWritter log_dir = Path("logs").mkdir(parents=True, exist_ok=True) # stdout and stderr saved to a *.log file. with loggerWritter("logs/custon-model-onnx"): # model compilation options
Add the line in bold:
from scripts.utils import loggerWritter log_dir = Path("logs").mkdir(parents=True, exist_ok=True) print(os.environ['TIDL_TOOLS_PATH'])
# stdout and stderr saved to a *.log file.
with loggerWritter("logs/custon-model-onnx"):
# model compilation options
Make sure the output of the value of os.environ['TIDL_TOOLS_PATH'] matches your tidl_tools/ directory. It should
point to your edgeai-tidl-tools/tidl_tools/ directory.
if not you may have started 'jupyter notebook' before the variable was set.
Or if you want more control that jupyter notebook, you can just run your model from the
command line in edgeai-tidl-tools/example/osrt_python/ort/ by:
python3 ./onnxrt_ep.py -m cl-ort-resnet18-v1 -c (First Compile the model)
python3 ./onnxrt_ep.py -m cl-ort-resnet18-v1 -d (Run the model on the x86 processor)
python3 ./onnxrt_ep.py -m cl-ort-resnet18-v1 (Run the model on the PC with C7x emulation)
Hi Chris:
I print the os.environ['TIDL_TOOLS_PATH'] below, The path appears to be correct.
Thanks for your kindly help.
Hi Chris,
I tried running the command python3 ./onnxrt_ep.py -m cl-ort-resnet18-v1 -c
and encountered the following error:
AttributeError: 'InferenceSession' object has no attribute 'get_TI_benchmark_data'
.
Could this be because the TIDLCompilationProvider
is not being used?
Thanks for your kindly help.
What version of TIDL are you running? I would recommend a 10.0 variant. And it looks like you are running this from a Docker container; is that correct? Can you run it from outside the container to eliminate some variability?
I am certain this works, so the only other thing I can think of is to re-install tidl_tools. To do this move/remove tidl_tools (that directory should not exist for reinstall). Ensure SOC is set and run 'source ./setup.sh' from the edgeai-tidl-tools directory.
Hi Chris:
I run TIDL version 10.00.04.00.
Yes, I run this from docker container.
Thank you very much for your suggestion. I will try running it directly on the Ubuntu system without using a container. Additionally, may I confirm if the SOC should be set to "am67a"?
Thanks for your kindly help.
Hi Chris:
I would like to double-check the steps I followed to ensure they are correct:
[env]
without container
Ubuntu : 22.04
TIDL : 10.00.04.00
[steps]
Could you kindly confirm if these steps are correct?
Additionally, I wanted to ask if the output file will be located in the custom-artifacts/onnx/
directory as specified in the notebook?
Thank you for your assistance!
Best regards,
Ken
Hi Ken,
I ran all your steps, but I am confused about how you run the jupyter ipynb file. I am using Ubuntu 22.04 and exec does not work. How are you running it? I have heard of using papermill to run a notebook from the command line but exec is not doing anything.
Chris
Hi Chris:
I copied the code from the Jupyter Notebook file into a .py
file and executed it using Python 3.
Thank you for your assistance!
Best regards,
Ken
Hi Chris:
I try to run TIDL without container, the onnx convert tool can work,
But I get the new issue from
and
Thanks for your kindly help.