TDA4VM: Benchmarking of TI model zoo models on TDA4VM

Hi Vishnu,

We have some benchmarking numbers in emulation I can share. What type of model are you looking for (Classification, Detection, Seg)?

If you need any specific ones, please let me know the model name. 

Warm regards,

Christina

Hi  Christina,

Thanks for replying.

I mainly need for all the SSD models specified in model zoo.

Warm Regards,

Vishnu

Hi Vishnu, 

This is on the last SDK and on emulation. These numbers were done to document a baseline. 

If there is a model you want to be tested on board with some more robust testing, please let me know.

Warm regards,

Christina

Hi Christina,

Thanks for your valuable reply.

This data definitely ease our work.

I have few more doubts as given below one by one.

1.  The models which you used for benchmarking are float32 or UINT8

2.  The benchmark values of SSD with resNet and mobilenet backend seems to be against our intuition. Mobilenet is a light weight model when compared to resNet and expected a high FPS value for MobileNet when compared to resNet. But what is the reason of not following this intuition while emulating on PC.

3. Do these models behave the same way as shown in the benchmark while  implementing on device ( TDA4VM )

Warms Regards,

Vishnu 

Hi Vishnu,

1. I believe all of these were run in int8.

2. It is possible certain layers may be causing this difference. We have also optimized the resnet models in our modelzoo to better run on the TIDL. Is there a specific mobilenet and resnet models are you interested in? 

3. Our PC emulation for our model zoo models are usually pretty similar to the device runs. This is just what I had at hand, but I can run benchmarks on specific models you are interested in to further verify.

Warm regards,

Christina

Hi Christina,

I am getting more and more clarity now. 

Can you please share the benchmarks of all the SSD models on device( TDA4VM ). 

This would be a great help for my team

Warm Regards,

Vishnu 

Hi Vishnu,

Please give me some time to get these benchmark's done, especially since there are many models. Is there a specific SDK you are looking for? These numbers are from our latest SDK (11.0.6.0), however we have a new release coming soon. 

I can give an update on my status latest by Friday.

Warm regards,

Christina

Hi Christina,

Sure, I sincerely appreciate your assistance

We are using SDK version 11.00.06.00

Warms Regards,

Vishnu 

Perfect! Thanks for the verification.

Will keep you updated.

Warm regards,

Christina

Hi Christina,

Meanwhile, I have two doubts about the benchmark data which you provided.

1. Those values are really FPS ?  Or is it time taken per frame ?

2. If it is time taken per frame, then what is the unit of those values ( second, millisecond, etc ) ?

Warm Regards,

Vishnu

Hi Vishnu,

Sorry for the delay in response. I wasn't the person who did this testing (it was one of the summer interns) so I am not able to confirm. It may actually be time taken per frames but the units for our testing is usually done in millisecond. 

I will redo the tests you are interested in anyways and be able to give you better information regarding. 

Warm regards,

Christina

Hi Christina,

Thanks for the reply.

Warm Regards,

Vishnu

For published FPS numbers of various models, you can navigate to Model Selection Tool from this link: https://dev.ti.com/edgeaistudio/

Or go directly to: https://dev.ti.com/gallery/view/edgeai/edgeai-modelselection/ver/1.0.0/