EDGE-AI-STUDIO: CL-0160 evaluation not working on Model Analyzer for AM69 EVM

Hello,
Apologies for the delay. This was just now assigned to the correct organization.

Please be aware of the current support model for AM69: [FAQ] Important Notice for Support transition for AM68/AM68A and AM69/AM69A devices 

I can only offer limited support here, per the support model Anshu mentioned.

I estimate the model did not compile correctly. Some of the logs may have been suppressed  / piped into a logfile. What you show here suggests to me that the model did not initialize correctly and is returning random output. CL-0160 should be a resnet50 model, which should compile trivially. Perhaps that step within this .IPYNB was skipped

Hi Anshu, Reese,

Thank you for clarifying the support model for the AM69. Probably, I should have worded the original request somewhat differently, but I am actually working on the J784S4X EVM for the TDA4VH device, as shown in the picture below. And the AM69A EVM is the only cloud-based device available to run the Model Analyzer. As such, I am not really looking for AM69A support.




Let me clarify the context of my request. We're investigating whether the TDA4VH device would be suitable replacement for the Edge AI FPGA platform we're currently using. I have been tasked to verify whether the performance numbers presented in application notes SPRACZ2 Application note | TI.com and SPRY344A, which are quite impressive, are in fact achievable in real-life, with a focus on the 3 MLPerf AI models mentioned in these application notes, i.e., CL-0160, OD-2000, and CL-0000.

I ran the recommended TIDL Runner benchmark tooling on the J784S4X EVM, but get 30-35% lower performance than the application notes claim.

j784s4xevm-performance-gap-on-mlperf-models

So, I started investigating with the additional tooling that TI makes available. I have checked the data sets with Edge AI Studio's Model Selector tool. For two of the 3 relevant models, the results are missing in the TDA4VH data set (CL-0160 and OD-2000). The only available data set point CL-0000 shows a 20% lower performance than the application notes.

edge-ai-studio-cl-0160-performance-on-tda4vm

Hence, I have switched to the Model Analyzer tool, which is only available on an AM69 EVM, and which turns out to have a broken evaluation notebook for the CL-0160 and CL0000 models. Note that the original notebook was used "as is", without any modification. And this cloud-based EVM is even no longer supported.

Don't get me wrong, I fully understand that this vast ecosystem is a tough job to support and maintain. Nonetheless, at the very least, I would have hoped that Texas Instruments would have a working toolchain around the models they are explicitly putting in the spotlight in their application notes. Though, taking into account the lower TIDL Runner benchmark results, the incorrect/missing data points in Model Selection tool, and broken evaluation notebook in the Model Analyzer, I am not given much useful data to make a compelling case for the TDA4VH device at the moment.

Kind regards,

Christophe