• State Resolved
  • Replies 3 replies
  • Subscribers 99 subscribers
  • Views 60 views
  • Users 0 members are here
Support feedback
Options
Options
Related

EDGE-AI-STUDIO: Excessive AM69A Benchmark Results in Model Analyzer

Christophe Van Delsen
Prodigy 70 points

Part Number: EDGE-AI-STUDIO
Other Parts Discussed in Thread: AM69A

Hi,

I want to evaluate the performance of some AI models on the J784S4 SoC. I am using the AM69A EVM in Edge AI Studio to get some reference results. However, when running the evalution Jupyter notebooks in Model Analyzer, I am getting quite excessive Inferences Per Seconds numbers for the models I am interested in.

E.g., when running the Object Detection notebook (vdet-tfl.ipynb) on the OD-2000 model, it returns 990 fps, which is an Inference Time per Image of 1.01 ms. Comparing that with the 385 fps mentioned in SPRACZ2 Application note | TI.com , this seems quite excessive.

Additionally, the value of 1.01 ms doesn't match the green bar in the chart, which indicates the C7x MMA operation duration. It looks to be closer to 3.25 ms. See the below screenshot. I have printed the return values tt and st of the get_benchmark_output() function call. It looks like the st value of 3.2146 ms is much closer to what is expected. This would give us 331 fps, more in line with the application note.

image.png


I got a similar result when running the Image Classification notebook (vcls-tfl.ipynb) on the CL-0000 model. Here, I am getting 1896 fps, versus 741 fps in application note SPRY344A. Again, using the st variable instead of tt gives a more appropriate result.

Could you confirm whether or not the results from the evaluation notebooks are correct?

Kind regards,

Christophe

  • 0
    TI__Mastermind 23805 points

    Hello Christophe,

    Apologies for the delay. This was just now assigned to the correct organization.

    Please be aware of the current support model: [FAQ] Important Notice for Support transition for AM68/AM68A and AM69/AM69A devices 

  • 0 in reply to Anshu Madwesh
    TI__Mastermind 19726 points

    The AM69A has 4x C7 cores which can each run a model independently, including in a batched mode. My estimate is that this is running in a multi-batch mode (or perhaps the benchmark FPS are being computed under this assumption), in which case the numbers make sense. In that way, 1ms latency is not right, but 1000FPS throughput is. 

    But otherwise, you are correct that these latencies do not match up, at least for single-core performance. 

  • +1 in reply to Reese Grimsley
    Prodigy 70 points

    Multi-batch mode could be possible, but I find this doubtful given that

    • The OD-2000 model in the Object Detection notebook runs at a 20% lower performance than advertised in the TI application notes. Therefore, likely running on a single core. Why run the models in one notebook on a single core and in multi-core on another? It would not make much sense.

    • Reporting the performance as raw FPS instead of FPS/TOPS violates TI's own recommendation in the application notes regarding apples-to-apples comparisons.

    I consider this a broken notebook. And given there is no direct support anymore for it, this effectively means that the J784S4 SoC has no viable cloud-based evaluation platform anymore. Closing this topic.