TDA4VM: HWA performance issue

Hi,

no, we have not seen such timeout issue in Scalar. Scalar should not such large time in processing. Are you running anything else on the R5F core, which probably does not allow running MSC task? This could be one reason why MSC task is not getting scheduled and why it is taking time in processing the frame. 

Regards,

Brijesh

Hi，thank you for reply timely.

> Are you running anything else on the R5F core

Yes, there are several tasks running such as camera data capturing，ldc and viss and ethernet driver

if it is related with task  scheduling, the load of the tasks may be abnormal.

We have checked the load of them, including MSC1, the load data is normal value when it happened.

for example: compared the load data of timeout with the one of not timeout. The difference is too small.

How about IPC? Is there any performance issue already known at present?

additional information:

The car which occurred timeout had a capture device for ethernet data, it didn't connect to TDA4 directly but OBD,  and the device  captured the ethernet data that TDA4 output. And the ethernet driver running on r5f.  Does the capture influence on the scaler?

Frank DU said:

How about IPC? Is there any performance issue already known at present?

No, IPC is another higher priority task, so will not be blocked due to other tasks. 

Do you have detailed performance statistics when the issue comes? also any logs available from mcu2_0? 

Also which SDK release are you using? 

Regards,

Brijesh

Hi, 

SDK release version: SDK8.2

> Do you have detailed performance statistics when the issue comes? also any logs available from mcu2_0? 

We have done it but hard to show it since we haven't an accurate measurement of performance in a release version and did not find any obvious helpful clue including mcu2_0 log

please reference:

comment.xlsx: it helps to understand the original problem and get the date time while it happened. 

mcu_log_46_2024_0904_072312.log: MCU2_0 log is in it, I am not sure if have any available information 

sys_log_38_2024_0904_072319.log: the original task load data with date time (keyword: taskload, the  log content is redirected from vx_app_task_load.out  of SDK)

Note: the log of task load output per 30 seconds. That means load value may be an average value in 30 seconds period, not so accurate.

noa_failed_tidl_often_timeout.zipcomment.xlsx

Hi，

Would you have any other idea about the issue or some suggestion?
If it might be a scheduling problem. What's the action for us to prove it.
We have no idea to find an efficient way since it happened with a low incidence.

We also may focus on checking health status of the scheduling when app tasks runs.
Are there any systematic check tools or methods from TI? As we know:
vx_app_arm_remote_log.out: which can output RTOS log via shared memory.
vx_app_task_print.out: which can output the load of tasks or other statistics.

Both of them work well, but difficult to trace scheduling problem.
We may add statistics log to trace the time consumption of tasks and interrupts,
but it is not efficient or involve performance issue by itself.

Frank DU said:

We have done it but hard to show it since we haven't an accurate measurement of performance in a release version and did not find any obvious helpful clue including mcu2_0 log

The performance stats reports max/min and average performance numbers. So from this, we can determine if the MSC tool more than expected time once in a while. 

In the comments.xls file, i see you have printed the MSC load, but can you also include MSC's performance at different instances? This will help to determine if this is really related to MSC. 

Do you see any other error from MSC? Can you also check what's the value of VHWA_M2M_MSC_MAX_WAIT_LOOP_CNT macro in the VHWA driver? Since it is old release, it might be set to less value and might be timing out. 

In order to check the scheduling issue, is it possible to first see if we can reproduce the issue consistently? Also is it possible to get detailed performance statistics? Also how much time it takes to reproduce this issue?  

Regards,

Brijesh

Hi,

>In the comments.xls file, i see you have printed the MSC load, but can you also >include MSC's performance at different instances? This will help to determine if this is >really related to MSC. 

-> We have compared the load with normal ones, no much difference of MSC itself.
But from our log, we have found some other abnormal things.
The frame rate of TIDL model(by C71) and pyramid(by C66) was down a little bit in the period which the scaler timeout had happened in.
The frame rate are calculated by A72, based on the response time from C71 or C66 including cores IPC communication.

>Do you see any other error from MSC? Can you also check what's the value of >VHWA_M2M_MSC_MAX_WAIT_LOOP_CNT macro in the VHWA driver?

->Thanks, we will trace VHWA_M2M_MSC_MAX_WAIT_LOOP_CNT later. We have found no errors from MSC.

>In order to check the scheduling issue, is it possible to first see if we can reproduce the issue consistently?

-> It is very hard to reproduce, It has not happened yet in several weeks.
We will try to reproduce it by more cars and collect more output log data including the system running situation and try to figure out the precondition for reproducing it with 100% percent.

Hi Frank,

Frank DU said:

The frame rate of TIDL model(by C71) and pyramid(by C66) was down a little bit in the period which the scaler timeout had happened in.
The frame rate are calculated by A72, based on the response time from C71 or C66 including cores IPC communication.

this is strange, this should not have affected MSC, unless they are using high DDR BW and delaying MSC read/write operations. But again delaying by 150ms or even 250ms seems very high in this case. It is not DDR BW issue, and core loading is also in control, then probably some scheduling issue on mcu2_0. If you have something high priority task taking longer time, then it can affect MSC (and probably other tasks) performance also. 

Regards,

Brijesh

I have confirmed the communication between A72 and R5f are using spinlock in TIOVX， and it seems the side of A72 has not disabled all interrupts before acquire spin lock. Is it possible that scheduling of A72 occurs while A72 is holding the spin lock and  cannot release the spin lock in time, in which period R5f is just spinning, no any scheduling?

Please see the picture below(a internal abstracted spin lock interface for all cores,  SDK8.2). Interrupts is not disabled.  I may misunderstand something as I could not understand TIOVX well.

Hi Frank,

Yes, that's possible. but do you have other high priority tasks running on A72 Linux? 

Btw, its not possible to disable interrupts from user application on Linux. 

Regards,

Brijesh