TDA4VM: EdgeAI Cloud TIDL with EVM

Hi Sugawara-san,

Assuming customers want to use the TDA4VM EVM like "EdgeAI Cloud" where they have machine learning models and Jupyter notebook installed.

a) I would recommend installing the SK-TDA4VM https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-SK-TDA4VM#downloads. This is a prebuilt image that includes everything needed to run deep learning AI on our TDA4VM processors including on the TDA4VM EVMs.

b) No SW license fee is required when using software from TI. As a general rule, all software developed by TI is free and does not need licensing fee. The machine learning engine such as TensorFlowLite, ONNX, and TVM/Neo are open source and free too.

c) https://github.com/TexasInstruments/edgeai-modelzoo/tree/master/models/vision/detection - This has the performance benchmarks of the different models that are in TIDL (TI Deep Learning) for object detection that has been optimized for the TDA4VM. The github should have benchmarks for other use cases such as semantic segmentation or image classification in other folder if they are needed.

d) Yes, they convert the model data to make it more into a form that gives you better performance for TDA4VM. I recommend reading this documentation over quantization: https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/06_01_01_12/exports/docs/tidl_j7_01_00_01_00/ti_dl/docs/user_guide_html/md_tidl_fsg_quantization.html

e) Link for c) should have performance speed for each model for object detection.

Hopefully this answered all your questions!

Regards,

Takuma

Hi Sugawara-san,

I will check with the team whether there is a typical use case for each run-time and if there is a procedure to select between TensorFlowLite, ONNX, or TVM/Neo other than performing benchmarks on all the models using datasets for specific customer use cases. I will update you tomorrow with the response I get.

In the meantime, I have found a few tools and documentation that may be of interest to you.

For conducting benchmarks on the different models: https://github.com/TexasInstruments/edgeai-benchmark

For learning more about TensorFlowLite, ONNX, or TVM/Neo: 

"

tensorflow: TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries, and community resources that lets researchers push the state-of-the-art in ML and developers easily build and deploy ML-powered applications. Please see the branch tidl-j7 for changes specific to our device.

onnxruntime: ONNX Runtime is a cross-platform inference and training machine-learning accelerator. Please see the branch tidl-j7 for changes specific to our device.

neo-ai-dlr: DLR is a compact, common runtime for deep learning models and decision tree models compiled by AWS SageMaker Neo, TVM, or Treelite. Please see the branch tidl-j7 for changes specific to our device.

tvm: Apache TVM (incubating) is a compiler stack for deep learning systems. It is designed to close the gap between the productivity-focused deep learning frameworks, and the performance- and efficiency-focused hardware backends. TVM works with deep learning frameworks to provide end to end compilation to different backends. Please see the branch tidl-j7 for changes specific to our device.

"

I found the above from here: (https://github.com/TexasInstruments/edgeai/blob/master/readme_sdk.md)

Regards,

Takuma

Hi Sugawara-san,

Apologies for the delay. I was able to gather information from the experts on TIDL (TI Deep Learning). I will quote their response here:

"

• If TIDL supports all the layers of the model on hardware accelerator, the entire model is delegated to TIDL, and in this case the performance will be the same irrespective of runtime used. This is the best case scenario for the customer in terms of performance. In case some of the layers in the models are not supported by TIDL, these layers will be delegated to ARM and in this case, performance may slightly differ across runtimes based on their implementations of these layers.
• I think you also must be aware of our Model Selection Tool as part of our Edge AI offering (https://dev.ti.com/edgeai/) which helps customers in model selection from a pre compiled set of models.

"

Also from their explanation, the three runtimes are usually chosen when:

• Tensorflow-lite runtime: Model is trained using tensorflow framework
• ONNX runtime: Model is trained using pytorch or similar frameworks which can be exported to ONNX format
• TVM/Neo runtime: Some layers cannot be delegated to TIDL, and are delegated to ARM processor running Linux. Additionally TVM generated code gives better performance than respective tensorflow-lite or ONNX runtime.

Regards,

Takuma