This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: Slowdown when using five cameras and TIDL

Part Number: TDA4VM

Hello,

I'm having a problem where it seems there is a slowdown in processing when I'm running cameras and TIDL in parallel. I'm using five D3RCM imx390 cameras on EVM board with fusion board. I'm using SDK 7.01.

My application has a two modes, first mode is cameras feeding to TIDL, 2nd mode is images are sent over network to TIDL. In both modes the only difference is the source of the images and the related nodes (capture nodes vs network node), the TIDL node and everything after (pre processing, post processing, display nodes) are all identical.

When I run the network mode, I see the TIDL processing time is around 90 ms. When I'm running the camera mode it increases to 122 ms. I also ran against the multi cam app and found the VISS bandwidth also decreases from 318 MP/s to 286 MP/s, and I measured camera frame rate drops from 30 fps to 27 fps (which match the MP/s numbers, ie 1936*1096*30*5=318MP/s).  I'm guessing the VISS and C7x are struggling to get enough memory bandwidth to run "full speed".

So a few questions. Is there a way to allocate more bandwidth to each HW unit? I'm not using all the HW units, like C6x or LDC so if reallocation is possible it may help. What is the max memory bandwidth on the EVM? I tried a few tests but it seems it maxes out around 5500 MB/s (~25%). How can I reach higher utilization numbers? Is there a way to measure memory bandwidth usage of the A72 CPU?

I added a monitor function to call appPerfStatsPrintAll() roughly every 10 sec, here is what I'm seeing:

Network mode:

Summary of CPU load,
====================

CPU: mpu1_0: TOTAL LOAD =  66.90 % ( HWI =   2.67 %, SWI =   7. 2 % )
CPU: mcu2_0: TOTAL LOAD =   4.52 % ( HWI =   1.43 %, SWI =   0.22 % )
CPU: mcu2_1: TOTAL LOAD =   0.12 % ( HWI =   0. 6 %, SWI =   0. 3 % )
CPU:  c6x_1: TOTAL LOAD =   0. 6 % ( HWI =   0. 2 %, SWI =   0. 2 % )
CPU:  c6x_2: TOTAL LOAD =   0. 6 % ( HWI =   0. 2 %, SWI =   0. 2 % )
CPU:  c7x_1: TOTAL LOAD =  72.32 % ( HWI =   0.12 %, SWI =   0. 9 % )


HWA performance statistics,
===========================



DDR performance statistics,
===========================

DDR: READ  BW: AVG =   2749 MB/s, PEAK =   4651 MB/s
DDR: WRITE BW: AVG =    823 MB/s, PEAK =   1372 MB/s
DDR: TOTAL BW: AVG =   3572 MB/s, PEAK =   6023 MB/s


Detailed CPU performance/memory statistics,
===========================================

CPU: mcu2_0: TASK:           IPC_RX:   0. 3 %
CPU: mcu2_0: TASK:       REMOTE_SRV:   0. 3 %
CPU: mcu2_0: TASK:         TIVX_CPU:   0. 0 %
CPU: mcu2_0: TASK:          TIVX_NF:   0. 0 %
CPU: mcu2_0: TASK:        TIVX_LDC1:   0. 0 %
CPU: mcu2_0: TASK:        TIVX_MSC1:   0. 0 %
CPU: mcu2_0: TASK:        TIVX_MSC2:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_VISS1:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CAPT1:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CAPT2:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_DISP1:   0.21 %
CPU: mcu2_0: TASK:       TIVX_DISP2:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CSITX:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CAPT3:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CAPT4:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CAPT5:   0. 0 %

CPU: mcu2_0: HEAP:   DDR_SHARED_MEM: size =    8388608 B, free =    8338688 B ( 99 % unused)
CPU: mcu2_0: HEAP:           L3_MEM: size =     131072 B, free =     131072 B (100 % unused)
CPU: mcu2_0: HEAP:  DDR_NON_CACHE_M: size =      65536 B, free =      65536 B (100 % unused)

CPU: mcu2_1: TASK:           IPC_RX:   0. 0 %
CPU: mcu2_1: TASK:       REMOTE_SRV:   0. 1 %
CPU: mcu2_1: TASK:         TIVX_SDE:   0. 0 %
CPU: mcu2_1: TASK:         TIVX_DOF:   0. 0 %
CPU: mcu2_1: TASK:       TIVX_VDEC1:   0. 0 %
CPU: mcu2_1: TASK:       TIVX_VDEC2:   0. 0 %
CPU: mcu2_1: TASK:       TIVX_VENC1:   0. 0 %
CPU: mcu2_1: TASK:       TIVX_VENC2:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_RX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %

CPU: mcu2_1: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
CPU: mcu2_1: HEAP:           L3_MEM: size =     131072 B, free =     131072 B (100 % unused)
CPU: mcu2_1: HEAP:  DDR_NON_CACHE_M: size =   67043328 B, free =   47144960 B (  6 % unused)

CPU:  c6x_1: TASK:           IPC_RX:   0. 0 %
CPU:  c6x_1: TASK:       REMOTE_SRV:   0. 0 %
CPU:  c6x_1: TASK:         TIVX_CPU:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_RX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %

CPU:  c6x_1: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
CPU:  c6x_1: HEAP:           L2_MEM: size =     229376 B, free =     229376 B (100 % unused)
CPU:  c6x_1: HEAP:  DDR_SCRATCH_MEM: size =   50331648 B, free =   50331648 B ( 14 % unused)

CPU:  c6x_2: TASK:           IPC_RX:   0. 0 %
CPU:  c6x_2: TASK:       REMOTE_SRV:   0. 0 %
CPU:  c6x_2: TASK:         TIVX_CPU:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_RX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %

CPU:  c6x_2: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
CPU:  c6x_2: HEAP:           L2_MEM: size =     229376 B, free =     229376 B (100 % unused)
CPU:  c6x_2: HEAP:  DDR_SCRATCH_MEM: size =   50331648 B, free =   50331648 B ( 14 % unused)

CPU:  c7x_1: TASK:           IPC_RX:   0. 5 %
CPU:  c7x_1: TASK:       REMOTE_SRV:   0. 0 %
CPU:  c7x_1: TASK:         TIVX_CPU:  72. 4 %
CPU:  c7x_1: TASK:      IPC_TEST_RX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %

CPU:  c7x_1: HEAP:   DDR_SHARED_MEM: size =  268435456 B, free =  211463168 B ( 14 % unused)
CPU:  c7x_1: HEAP:           L3_MEM: size =    8159232 B, free =          0 B (  0 % unused)
CPU:  c7x_1: HEAP:           L2_MEM: size =     491520 B, free =     491520 B (100 % unused)
CPU:  c7x_1: HEAP:           L1_MEM: size =      16384 B, free =          0 B (  0 % unused)
CPU:  c7x_1: HEAP:  DDR_SCRATCH_MEM: size =  234881024 B, free =  233944638 B (  8 % unused)

Camera mode:

Summary of CPU load,
====================

CPU: mpu1_0: TOTAL LOAD =  79.25 % ( HWI =   3.59 %, SWI =   1.94 % )
CPU: mcu2_0: TOTAL LOAD =  59.64 % ( HWI =   9.52 %, SWI =   1.77 % )
CPU: mcu2_1: TOTAL LOAD =   0.15 % ( HWI =   0. 7 %, SWI =   0. 4 % )
CPU:  c6x_1: TOTAL LOAD =   0. 6 % ( HWI =   0. 2 %, SWI =   0. 2 % )
CPU:  c6x_2: TOTAL LOAD =   0. 6 % ( HWI =   0. 2 %, SWI =   0. 2 % )
CPU:  c7x_1: TOTAL LOAD =  80.85 % ( HWI =   0.15 %, SWI =   0.11 % )


HWA performance statistics,
===========================

HWA:   VISS: LOAD =  54.30 % ( 522 MP/s )
HWA:   MSC0: LOAD =  48.76 % ( 699 MP/s )
HWA:   MSC1: LOAD =  49. 3 % ( 696 MP/s )


DDR performance statistics,
===========================

DDR: READ  BW: AVG =   3543 MB/s, PEAK =   5154 MB/s
DDR: WRITE BW: AVG =   1841 MB/s, PEAK =   2722 MB/s
DDR: TOTAL BW: AVG =   5384 MB/s, PEAK =   7876 MB/s


Detailed CPU performance/memory statistics,
===========================================

CPU: mcu2_0: TASK:           IPC_RX:   0.32 %
CPU: mcu2_0: TASK:       REMOTE_SRV:   0. 8 %
CPU: mcu2_0: TASK:         TIVX_CPU:  10.85 %
CPU: mcu2_0: TASK:          TIVX_NF:   0. 0 %
CPU: mcu2_0: TASK:        TIVX_LDC1:   0. 0 %
CPU: mcu2_0: TASK:        TIVX_MSC1:   9.41 %
CPU: mcu2_0: TASK:        TIVX_MSC2:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_VISS1:  16.99 %
CPU: mcu2_0: TASK:       TIVX_CAPT1:   2.64 %
CPU: mcu2_0: TASK:       TIVX_CAPT2:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_DISP1:   0.35 %
CPU: mcu2_0: TASK:       TIVX_DISP2:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CSITX:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CAPT3:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CAPT4:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CAPT5:   0. 0 %

CPU: mcu2_0: HEAP:   DDR_SHARED_MEM: size =    8388608 B, free =    6796288 B ( 81 % unused)
CPU: mcu2_0: HEAP:           L3_MEM: size =     131072 B, free =      49152 B ( 37 % unused)
CPU: mcu2_0: HEAP:  DDR_NON_CACHE_M: size =      65536 B, free =      65536 B (100 % unused)

CPU: mcu2_1: TASK:           IPC_RX:   0. 0 %
CPU: mcu2_1: TASK:       REMOTE_SRV:   0. 2 %
CPU: mcu2_1: TASK:         TIVX_SDE:   0. 0 %
CPU: mcu2_1: TASK:         TIVX_DOF:   0. 0 %
CPU: mcu2_1: TASK:       TIVX_VDEC1:   0. 0 %
CPU: mcu2_1: TASK:       TIVX_VDEC2:   0. 0 %
CPU: mcu2_1: TASK:       TIVX_VENC1:   0. 0 %
CPU: mcu2_1: TASK:       TIVX_VENC2:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_RX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %

CPU: mcu2_1: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
CPU: mcu2_1: HEAP:           L3_MEM: size =     131072 B, free =     131072 B (100 % unused)
CPU: mcu2_1: HEAP:  DDR_NON_CACHE_M: size =   67043328 B, free =   47144960 B (  6 % unused)

CPU:  c6x_1: TASK:           IPC_RX:   0. 0 %
CPU:  c6x_1: TASK:       REMOTE_SRV:   0. 0 %
CPU:  c6x_1: TASK:         TIVX_CPU:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_RX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %

CPU:  c6x_1: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
CPU:  c6x_1: HEAP:           L2_MEM: size =     229376 B, free =     229376 B (100 % unused)
CPU:  c6x_1: HEAP:  DDR_SCRATCH_MEM: size =   50331648 B, free =   50331648 B ( 14 % unused)

CPU:  c6x_2: TASK:           IPC_RX:   0. 0 %
CPU:  c6x_2: TASK:       REMOTE_SRV:   0. 0 %
CPU:  c6x_2: TASK:         TIVX_CPU:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_RX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %

CPU:  c6x_2: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
CPU:  c6x_2: HEAP:           L2_MEM: size =     229376 B, free =     229376 B (100 % unused)
CPU:  c6x_2: HEAP:  DDR_SCRATCH_MEM: size =   50331648 B, free =   50331648 B ( 14 % unused)

CPU:  c7x_1: TASK:           IPC_RX:   0. 7 %
CPU:  c7x_1: TASK:       REMOTE_SRV:   0. 0 %
CPU:  c7x_1: TASK:         TIVX_CPU:  80.50 %
CPU:  c7x_1: TASK:      IPC_TEST_RX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %

CPU:  c7x_1: HEAP:   DDR_SHARED_MEM: size =  268435456 B, free =  211463168 B ( 14 % unused)
CPU:  c7x_1: HEAP:           L3_MEM: size =    8159232 B, free =          0 B (  0 % unused)
CPU:  c7x_1: HEAP:           L2_MEM: size =     491520 B, free =     491520 B (100 % unused)
CPU:  c7x_1: HEAP:           L1_MEM: size =      16384 B, free =          0 B (  0 % unused)
CPU:  c7x_1: HEAP:  DDR_SCRATCH_MEM: size =  234881024 B, free =  233944638 B (  8 % unused)

Running run_app_multi_cam.sh with 5 cameras enabled:

Summary of CPU load,
====================

CPU: mpu1_0: TOTAL LOAD =   0.98 % ( HWI =   0. 9 %, SWI =   0. 4 % )
CPU: mcu2_0: TOTAL LOAD =  39.76 % ( HWI =   6.17 %, SWI =   1.10 % )
CPU: mcu2_1: TOTAL LOAD =   0.13 % ( HWI =   0. 6 %, SWI =   0. 3 % )
CPU:  c6x_1: TOTAL LOAD =   0. 6 % ( HWI =   0. 2 %, SWI =   0. 2 % )
CPU:  c6x_2: TOTAL LOAD =   0. 6 % ( HWI =   0. 2 %, SWI =   0. 2 % )
CPU:  c7x_1: TOTAL LOAD =   0. 8 % ( HWI =   0. 3 %, SWI =   0. 2 % )


HWA performance statistics,
===========================

HWA:   VISS: LOAD =  56.27 % ( 567 MP/s )
HWA:   MSC0: LOAD =  48.70 % ( 785 MP/s )
HWA:   MSC1: LOAD =  32.51 % ( 784 MP/s )


DDR performance statistics,
===========================

DDR: READ  BW: AVG =   1649 MB/s, PEAK =   1734 MB/s
DDR: WRITE BW: AVG =   1216 MB/s, PEAK =   1266 MB/s
DDR: TOTAL BW: AVG =   2865 MB/s, PEAK =   3000 MB/s


Detailed CPU performance/memory statistics,
===========================================

CPU: mcu2_0: TASK:           IPC_RX:   0. 7 %
CPU: mcu2_0: TASK:       REMOTE_SRV:   0. 5 %
CPU: mcu2_0: TASK:         TIVX_CPU:   5.45 %
CPU: mcu2_0: TASK:          TIVX_NF:   0. 0 %
CPU: mcu2_0: TASK:        TIVX_LDC1:   0. 0 %
CPU: mcu2_0: TASK:        TIVX_MSC1:   5.51 %
CPU: mcu2_0: TASK:        TIVX_MSC2:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_VISS1:  12.92 %
CPU: mcu2_0: TASK:       TIVX_CAPT1:   1.99 %
CPU: mcu2_0: TASK:       TIVX_CAPT2:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_DISP1:   0.99 %
CPU: mcu2_0: TASK:       TIVX_DISP2:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CSITX:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CAPT3:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CAPT4:   0. 0 %
CPU: mcu2_0: TASK:       TIVX_CAPT5:   0. 0 %

CPU: mcu2_0: HEAP:   DDR_SHARED_MEM: size =    8388608 B, free =    6760960 B ( 80 % unused)
CPU: mcu2_0: HEAP:           L3_MEM: size =     131072 B, free =      49152 B ( 37 % unused)
CPU: mcu2_0: HEAP:  DDR_NON_CACHE_M: size =      65536 B, free =      65536 B (100 % unused)

CPU: mcu2_1: TASK:           IPC_RX:   0. 0 %
CPU: mcu2_1: TASK:       REMOTE_SRV:   0. 1 %
CPU: mcu2_1: TASK:         TIVX_SDE:   0. 0 %
CPU: mcu2_1: TASK:         TIVX_DOF:   0. 0 %
CPU: mcu2_1: TASK:       TIVX_VDEC1:   0. 0 %
CPU: mcu2_1: TASK:       TIVX_VDEC2:   0. 0 %
CPU: mcu2_1: TASK:       TIVX_VENC1:   0. 0 %
CPU: mcu2_1: TASK:       TIVX_VENC2:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_RX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %

CPU: mcu2_1: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
CPU: mcu2_1: HEAP:           L3_MEM: size =     131072 B, free =     131072 B (100 % unused)
CPU: mcu2_1: HEAP:  DDR_NON_CACHE_M: size =   67043328 B, free =   47144960 B (  6 % unused)

CPU:  c6x_1: TASK:           IPC_RX:   0. 0 %
CPU:  c6x_1: TASK:       REMOTE_SRV:   0. 0 %
CPU:  c6x_1: TASK:         TIVX_CPU:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_RX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %

CPU:  c6x_1: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
CPU:  c6x_1: HEAP:           L2_MEM: size =     229376 B, free =     229376 B (100 % unused)
CPU:  c6x_1: HEAP:  DDR_SCRATCH_MEM: size =   50331648 B, free =   50331648 B ( 14 % unused)

CPU:  c6x_2: TASK:           IPC_RX:   0. 0 %
CPU:  c6x_2: TASK:       REMOTE_SRV:   0. 0 %
CPU:  c6x_2: TASK:         TIVX_CPU:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_RX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %

CPU:  c6x_2: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
CPU:  c6x_2: HEAP:           L2_MEM: size =     229376 B, free =     229376 B (100 % unused)
CPU:  c6x_2: HEAP:  DDR_SCRATCH_MEM: size =   50331648 B, free =   50331648 B ( 14 % unused)

CPU:  c7x_1: TASK:           IPC_RX:   0. 0 %
CPU:  c7x_1: TASK:       REMOTE_SRV:   0. 0 %
CPU:  c7x_1: TASK:         TIVX_CPU:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_RX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %

CPU:  c7x_1: HEAP:   DDR_SHARED_MEM: size =  268435456 B, free =  268435456 B (  4 % unused)
CPU:  c7x_1: HEAP:           L3_MEM: size =    8159232 B, free =    8159232 B (100 % unused)
CPU:  c7x_1: HEAP:           L2_MEM: size =     491520 B, free =     491520 B (100 % unused)
CPU:  c7x_1: HEAP:           L1_MEM: size =      16384 B, free =      16384 B (100 % unused)
CPU:  c7x_1: HEAP:  DDR_SCRATCH_MEM: size =  234881024 B, free =  234881024 B (  8 % unused)

  • Hi,

    Do you see any issue when you run less number of channels, lets say 3 or 4 channels? 

    capture and VISS will definitely add to the total BW, but we need to first understand the usecase, then we can try to optimize it. 

    Can you please check/contact your Local support for the further support?

    Regards,

    Brijesh

  • Hi,

    With 3 or 4 cameras the application runs fine, with the expected camera fps and TIDL processing time.

  • Hi gordon huang,

    Could you please share complete performance stats for 5 channel? Would like to see which component is slowing down the entire chain.

    Regards,

    Brijesh 

  • What do you mean by complete performance stats? Which APIs to get that information?

  • Please press 'p' on the console and share the log output.

    Rgds,

    Brijesh

  • Hi,

    My application doesn't have command line interface, but I did add the same calls used in the example program 'p'. Also, in my app there are 3 TIOVX graphs, so the output is somewhat duplicated:

    camera mode:


    Summary of CPU load,
    ====================

    CPU: mpu1_0: TOTAL LOAD =  76.13 % ( HWI =   3.50 %, SWI =   1.94 % )
    CPU: mcu2_0: TOTAL LOAD =  60.17 % ( HWI =   9.54 %, SWI =   1.78 % )
    CPU: mcu2_1: TOTAL LOAD =   0.15 % ( HWI =   0. 7 %, SWI =   0. 4 % )
    CPU:  c6x_1: TOTAL LOAD =   0. 6 % ( HWI =   0. 2 %, SWI =   0. 2 % )
    CPU:  c6x_2: TOTAL LOAD =   0. 6 % ( HWI =   0. 2 %, SWI =   0. 2 % )
    CPU:  c7x_1: TOTAL LOAD =  80.98 % ( HWI =   0.15 %, SWI =   0.11 % )


    HWA performance statistics,
    ===========================

    HWA:   VISS: LOAD =  54.68 % ( 521 MP/s )
    HWA:   MSC0: LOAD =  49.15 % ( 694 MP/s )
    HWA:   MSC1: LOAD =  49.43 % ( 690 MP/s )


    DDR performance statistics,
    ===========================

    DDR: READ  BW: AVG =   3566 MB/s, PEAK =   5567 MB/s
    DDR: WRITE BW: AVG =   1854 MB/s, PEAK =   2885 MB/s
    DDR: TOTAL BW: AVG =   5420 MB/s, PEAK =   8452 MB/s


    Detailed CPU performance/memory statistics,
    ===========================================

    CPU: mcu2_0: TASK:           IPC_RX:   0.33 %
    CPU: mcu2_0: TASK:       REMOTE_SRV:   0. 7 %
    CPU: mcu2_0: TASK:         TIVX_CPU:  11. 2 %
    CPU: mcu2_0: TASK:          TIVX_NF:   0. 0 %
    CPU: mcu2_0: TASK:        TIVX_LDC1:   0. 0 %
    CPU: mcu2_0: TASK:        TIVX_MSC1:   9.40 %
    CPU: mcu2_0: TASK:        TIVX_MSC2:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_VISS1:  17.20 %
    CPU: mcu2_0: TASK:       TIVX_CAPT1:   2.73 %
    CPU: mcu2_0: TASK:       TIVX_CAPT2:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_DISP1:   0.34 %
    CPU: mcu2_0: TASK:       TIVX_DISP2:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CSITX:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CAPT3:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CAPT4:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CAPT5:   0. 0 %

    CPU: mcu2_0: HEAP:   DDR_SHARED_MEM: size =    8388608 B, free =    6725632 B ( 80 % unused)
    CPU: mcu2_0: HEAP:           L3_MEM: size =     131072 B, free =      49152 B ( 37 % unused)
    CPU: mcu2_0: HEAP:  DDR_NON_CACHE_M: size =      65536 B, free =      65536 B (100 % unused)

    CPU: mcu2_1: TASK:           IPC_RX:   0. 0 %
    CPU: mcu2_1: TASK:       REMOTE_SRV:   0. 2 %
    CPU: mcu2_1: TASK:         TIVX_SDE:   0. 0 %
    CPU: mcu2_1: TASK:         TIVX_DOF:   0. 0 %
    CPU: mcu2_1: TASK:       TIVX_VDEC1:   0. 0 %
    CPU: mcu2_1: TASK:       TIVX_VDEC2:   0. 0 %
    CPU: mcu2_1: TASK:       TIVX_VENC1:   0. 0 %
    CPU: mcu2_1: TASK:       TIVX_VENC2:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_RX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %

    CPU: mcu2_1: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
    CPU: mcu2_1: HEAP:           L3_MEM: size =     131072 B, free =     131072 B (100 % unused)
    CPU: mcu2_1: HEAP:  DDR_NON_CACHE_M: size =   67043328 B, free =   47144960 B (  6 % unused)

    CPU:  c6x_1: TASK:           IPC_RX:   0. 0 %
    CPU:  c6x_1: TASK:       REMOTE_SRV:   0. 0 %
    CPU:  c6x_1: TASK:         TIVX_CPU:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_RX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %

    CPU:  c6x_1: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
    CPU:  c6x_1: HEAP:           L2_MEM: size =     229376 B, free =     229376 B (100 % unused)
    CPU:  c6x_1: HEAP:  DDR_SCRATCH_MEM: size =   50331648 B, free =   50331648 B ( 14 % unused)

    CPU:  c6x_2: TASK:           IPC_RX:   0. 0 %
    CPU:  c6x_2: TASK:       REMOTE_SRV:   0. 0 %
    CPU:  c6x_2: TASK:         TIVX_CPU:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_RX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %

    CPU:  c6x_2: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
    CPU:  c6x_2: HEAP:           L2_MEM: size =     229376 B, free =     229376 B (100 % unused)
    CPU:  c6x_2: HEAP:  DDR_SCRATCH_MEM: size =   50331648 B, free =   50331648 B ( 14 % unused)

    CPU:  c7x_1: TASK:           IPC_RX:   0. 7 %
    CPU:  c7x_1: TASK:       REMOTE_SRV:   0. 0 %
    CPU:  c7x_1: TASK:         TIVX_CPU:  80.63 %
    CPU:  c7x_1: TASK:      IPC_TEST_RX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %

    CPU:  c7x_1: HEAP:   DDR_SHARED_MEM: size =  268435456 B, free =  211463168 B ( 14 % unused)
    CPU:  c7x_1: HEAP:           L3_MEM: size =    8159232 B, free =          0 B (  0 % unused)
    CPU:  c7x_1: HEAP:           L2_MEM: size =     491520 B, free =     491520 B (100 % unused)
    CPU:  c7x_1: HEAP:           L1_MEM: size =      16384 B, free =          0 B (  0 % unused)
    CPU:  c7x_1: HEAP:  DDR_SCRATCH_MEM: size =  234881024 B, free =  233944638 B (  8 % unused)


    GRAPH: app_multi_cam_graph (#nodes =   4, #executions =   1499)
     NODE:   CAPTURE1:             capture_node: avg =  23637 usecs, min/max =    714 /  76921 usecs, #executions =       1499
     NODE: VPAC_VISS1:                viss_node: avg =  26400 usecs, min/max =  21811 /  29184 usecs, #executions =       1499
     NODE:     IPU1-0:                aewb_node: avg =   6371 usecs, min/max =    902 /  23849 usecs, #executions =       1499
     NODE:  VPAC_MSC1:              mosaic_node: avg =  20342 usecs, min/max =  16833 /  25614 usecs, #executions =       1499

     PERF:           FILEIO: avg =      0 usecs, min/max = 4294967295 /      0 usecs, #executions =          0
     PERF:            TOTAL: avg =  30415 usecs, min/max =   2112 /  65666 usecs, #executions =        500

     PERF:            TOTAL:   32.87 FPS

    GRAPH:        graph_157 (#nodes =   1, #executions =   1001)
     NODE:   DSP_C7-1:                 node_177: avg =  18917 usecs, min/max =  16580 /  21841 usecs, #executions =       1001

     PERF:             TIVX: avg =  23274 usecs, min/max =  18454 /  39954 usecs, #executions =        500

     PERF:             TIVX:   42.96 FPS

    GRAPH:        graph_179 (#nodes =   1, #executions =    501)
     NODE:   DISPLAY1:                 node_180: avg =   9321 usecs, min/max =    120 /  33441 usecs, #executions =        501

     PERF:             DISP: avg =  19132 usecs, min/max =   5206 /  55118 usecs, #executions =        500

     PERF:             DISP:   52.26 FPS

    network mode:


    Summary of CPU load,
    ====================

    CPU: mpu1_0: TOTAL LOAD =  68.78 % ( HWI =   2.72 %, SWI =   6.89 % )
    CPU: mcu2_0: TOTAL LOAD =   4.56 % ( HWI =   1.44 %, SWI =   0.22 % )
    CPU: mcu2_1: TOTAL LOAD =   0.12 % ( HWI =   0. 6 %, SWI =   0. 3 % )
    CPU:  c6x_1: TOTAL LOAD =   0. 6 % ( HWI =   0. 2 %, SWI =   0. 2 % )
    CPU:  c6x_2: TOTAL LOAD =   0. 6 % ( HWI =   0. 2 %, SWI =   0. 2 % )
    CPU:  c7x_1: TOTAL LOAD =  75. 6 % ( HWI =   0.12 %, SWI =   0. 9 % )


    HWA performance statistics,
    ===========================



    DDR performance statistics,
    ===========================

    DDR: READ  BW: AVG =   2796 MB/s, PEAK =   4070 MB/s
    DDR: WRITE BW: AVG =    843 MB/s, PEAK =   1213 MB/s
    DDR: TOTAL BW: AVG =   3639 MB/s, PEAK =   5283 MB/s


    Detailed CPU performance/memory statistics,
    ===========================================

    CPU: mcu2_0: TASK:           IPC_RX:   0. 4 %
    CPU: mcu2_0: TASK:       REMOTE_SRV:   0. 3 %
    CPU: mcu2_0: TASK:         TIVX_CPU:   0. 0 %
    CPU: mcu2_0: TASK:          TIVX_NF:   0. 0 %
    CPU: mcu2_0: TASK:        TIVX_LDC1:   0. 0 %
    CPU: mcu2_0: TASK:        TIVX_MSC1:   0. 0 %
    CPU: mcu2_0: TASK:        TIVX_MSC2:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_VISS1:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CAPT1:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CAPT2:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_DISP1:   0.21 %
    CPU: mcu2_0: TASK:       TIVX_DISP2:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CSITX:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CAPT3:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CAPT4:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CAPT5:   0. 0 %

    CPU: mcu2_0: HEAP:   DDR_SHARED_MEM: size =    8388608 B, free =    8268032 B ( 98 % unused)
    CPU: mcu2_0: HEAP:           L3_MEM: size =     131072 B, free =     131072 B (100 % unused)
    CPU: mcu2_0: HEAP:  DDR_NON_CACHE_M: size =      65536 B, free =      65536 B (100 % unused)

    CPU: mcu2_1: TASK:           IPC_RX:   0. 0 %
    CPU: mcu2_1: TASK:       REMOTE_SRV:   0. 1 %
    CPU: mcu2_1: TASK:         TIVX_SDE:   0. 0 %
    CPU: mcu2_1: TASK:         TIVX_DOF:   0. 0 %
    CPU: mcu2_1: TASK:       TIVX_VDEC1:   0. 0 %
    CPU: mcu2_1: TASK:       TIVX_VDEC2:   0. 0 %
    CPU: mcu2_1: TASK:       TIVX_VENC1:   0. 0 %
    CPU: mcu2_1: TASK:       TIVX_VENC2:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_RX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %

    CPU: mcu2_1: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
    CPU: mcu2_1: HEAP:           L3_MEM: size =     131072 B, free =     131072 B (100 % unused)
    CPU: mcu2_1: HEAP:  DDR_NON_CACHE_M: size =   67043328 B, free =   47144960 B (  6 % unused)

    CPU:  c6x_1: TASK:           IPC_RX:   0. 0 %
    CPU:  c6x_1: TASK:       REMOTE_SRV:   0. 0 %
    CPU:  c6x_1: TASK:         TIVX_CPU:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_RX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %

    CPU:  c6x_1: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
    CPU:  c6x_1: HEAP:           L2_MEM: size =     229376 B, free =     229376 B (100 % unused)
    CPU:  c6x_1: HEAP:  DDR_SCRATCH_MEM: size =   50331648 B, free =   50331648 B ( 14 % unused)

    CPU:  c6x_2: TASK:           IPC_RX:   0. 0 %
    CPU:  c6x_2: TASK:       REMOTE_SRV:   0. 0 %
    CPU:  c6x_2: TASK:         TIVX_CPU:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_RX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %

    CPU:  c6x_2: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
    CPU:  c6x_2: HEAP:           L2_MEM: size =     229376 B, free =     229376 B (100 % unused)
    CPU:  c6x_2: HEAP:  DDR_SCRATCH_MEM: size =   50331648 B, free =   50331648 B ( 14 % unused)

    CPU:  c7x_1: TASK:           IPC_RX:   0. 6 %
    CPU:  c7x_1: TASK:       REMOTE_SRV:   0. 0 %
    CPU:  c7x_1: TASK:         TIVX_CPU:  74.77 %
    CPU:  c7x_1: TASK:      IPC_TEST_RX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %

    CPU:  c7x_1: HEAP:   DDR_SHARED_MEM: size =  268435456 B, free =  211463168 B ( 14 % unused)
    CPU:  c7x_1: HEAP:           L3_MEM: size =    8159232 B, free =          0 B (  0 % unused)
    CPU:  c7x_1: HEAP:           L2_MEM: size =     491520 B, free =     491520 B (100 % unused)
    CPU:  c7x_1: HEAP:           L1_MEM: size =      16384 B, free =          0 B (  0 % unused)
    CPU:  c7x_1: HEAP:  DDR_SCRATCH_MEM: size =  234881024 B, free =  233944638 B (  8 % unused)


    GRAPH:         graph_72 (#nodes =   1, #executions =   1001)
     NODE:   DSP_C7-1:                  node_92: avg =  13993 usecs, min/max =  13102 /  16126 usecs, #executions =       1001

     PERF:             TIVX: avg =  16381 usecs, min/max =  14852 /  44038 usecs, #executions =        500

     PERF:             TIVX:   61. 4 FPS

    GRAPH:        graph_109 (#nodes =   1, #executions =    501)
     NODE:   DISPLAY1:                 node_110: avg =   8419 usecs, min/max =     85 /  16947 usecs, #executions =        501

     PERF:             DISP: avg =  13299 usecs, min/max =   2978 /  42295 usecs, #executions =        500

     PERF:             DISP:   75.19 FPS

  • Hi gordon,

    but according to these stats, the graph is is running at correct fps, ie 30fps. Do you expect higher fps in this case? or are you running 5 camera for this stats? 

    GRAPH: app_multi_cam_graph (#nodes =   4, #executions =   1499)
     NODE:   CAPTURE1:             capture_node: avg =  23637 usecs, min/max =    714 /  76921 usecs, #executions =       1499
     NODE: VPAC_VISS1:                viss_node: avg =  26400 usecs, min/max =  21811 /  29184 usecs, #executions =       1499
     NODE:     IPU1-0:                aewb_node: avg =   6371 usecs, min/max =    902 /  23849 usecs, #executions =       1499
     NODE:  VPAC_MSC1:              mosaic_node: avg =  20342 usecs, min/max =  16833 /  25614 usecs, #executions =       1499

     PERF:           FILEIO: avg =      0 usecs, min/max = 4294967295 /      0 usecs, #executions =          0
     PERF:            TOTAL: avg =  30415 usecs, min/max =   2112 /  65666 usecs, #executions =        500

     PERF:            TOTAL:   32.87 FPS

    Regards,

    Brijesh

  • Hi,

    This is running the 5 camera mode. I'm not sure how the TI lib is calculating the FPS, but when I measure the rate from application side it is not right. Maybe it's only counting frames it processes and not what it's skipping? Looking at the data for HWA,

    HWA:   VISS: LOAD =  54.68 % ( 521 MP/s )

    54.78% * 521 = 285.4 MP/s

    1936 (width) * 1096 (height) * 5 (cam) * N fps = 285.4 MP/s

    N = 26.9 FPS

    This FPS number matches what the frame rate I receive on my application side, which is 27 FPS. Also, check the DSP_C7-1 TIDL numbers, they have increased by 40%. They are both running the same models.

    PERF:             TIVX: avg =  16381 usecs, min/max =  14852 /  44038 usecs, #executions =        500

    PERF:             TIVX: avg =  23274 usecs, min/max =  18454 /  39954 usecs, #executions =        500

    The slowdown in TIDL is more critical for me, since we can't run entire pipeline at 27 or 30 fps anyway. The TIDL processing is the bottleneck for entire pipeline, I'd like to at least keep the TIDL processing time as short as possible.

  • Hi,

    So I reran test on the D3 RVP-TDA4VM board, and found some different results. On that board, the 5 camera setup runs fine without any TIDL slow down. The frame rate I measured was around 28.5 FPS.

    Looking at the statistics results, it looks like both DDR bandwidth and VISS are both higher than on TI EVM board. I'm wondering, maybe D3 has different DDR config to allow higher throughput? Why does the VISS have higher MP/s value (574 vs 521)?

    Since our application runs as expected on D3, this issue isn't so critical, but it would be nice to know why there is a difference and how to configure both boards to behave similarly.

    Camera mode on D3 board:

    Summary of CPU load,
    ====================

    CPU: mpu1_0: TOTAL LOAD =  60.97 % ( HWI =   1.91 %, SWI =   1.52 % )
    CPU: mcu2_0: TOTAL LOAD =  38.42 % ( HWI =   5.12 %, SWI =   1.27 % )
    CPU: mcu2_1: TOTAL LOAD =   0.12 % ( HWI =   0. 6 %, SWI =   0. 3 % )
    CPU:  c6x_1: TOTAL LOAD =   0. 6 % ( HWI =   0. 2 %, SWI =   0. 2 % )
    CPU:  c6x_2: TOTAL LOAD =   0. 6 % ( HWI =   0. 2 %, SWI =   0. 2 % )
    CPU:  c7x_1: TOTAL LOAD =  77.36 % ( HWI =   0.15 %, SWI =   0.11 % )


    HWA performance statistics,
    ===========================

    HWA:   VISS: LOAD =  51.90 % ( 574 MP/s )
    HWA:   MSC0: LOAD =  45.28 % ( 791 MP/s )
    HWA:   MSC1: LOAD =  45.48 % ( 787 MP/s )


    DDR performance statistics,
    ===========================

    DDR: READ  BW: AVG =   3918 MB/s, PEAK =   6574 MB/s
    DDR: WRITE BW: AVG =   1971 MB/s, PEAK =   3289 MB/s
    DDR: TOTAL BW: AVG =   5889 MB/s, PEAK =   9863 MB/s


    Detailed CPU performance/memory statistics,
    ===========================================

    CPU: mcu2_0: TASK:           IPC_RX:   0.18 %
    CPU: mcu2_0: TASK:       REMOTE_SRV:   0. 4 %
    CPU: mcu2_0: TASK:         TIVX_CPU:   7.27 %
    CPU: mcu2_0: TASK:          TIVX_NF:   0. 0 %
    CPU: mcu2_0: TASK:        TIVX_LDC1:   0. 0 %
    CPU: mcu2_0: TASK:        TIVX_MSC1:   6. 1 %
    CPU: mcu2_0: TASK:        TIVX_MSC2:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_VISS1:  12. 2 %
    CPU: mcu2_0: TASK:       TIVX_CAPT1:   1.83 %
    CPU: mcu2_0: TASK:       TIVX_CAPT2:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_DISP1:   0.24 %
    CPU: mcu2_0: TASK:       TIVX_DISP2:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CSITX:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CAPT3:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CAPT4:   0. 0 %
    CPU: mcu2_0: TASK:       TIVX_CAPT5:   0. 0 %

    CPU: mcu2_0: HEAP:   DDR_SHARED_MEM: size =    8388608 B, free =    6791168 B ( 80 % unused)
    CPU: mcu2_0: HEAP:           L3_MEM: size =     131072 B, free =      49152 B ( 37 % unused)
    CPU: mcu2_0: HEAP:  DDR_NON_CACHE_M: size =      65536 B, free =      65536 B (100 % unused)

    CPU: mcu2_1: TASK:           IPC_RX:   0. 0 %
    CPU: mcu2_1: TASK:       REMOTE_SRV:   0. 1 %
    CPU: mcu2_1: TASK:         TIVX_SDE:   0. 0 %
    CPU: mcu2_1: TASK:         TIVX_DOF:   0. 0 %
    CPU: mcu2_1: TASK:       TIVX_VDEC1:   0. 0 %
    CPU: mcu2_1: TASK:       TIVX_VDEC2:   0. 0 %
    CPU: mcu2_1: TASK:       TIVX_VENC1:   0. 0 %
    CPU: mcu2_1: TASK:       TIVX_VENC2:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_RX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU: mcu2_1: TASK:      IPC_TEST_TX:   0. 0 %

    CPU: mcu2_1: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
    CPU: mcu2_1: HEAP:           L3_MEM: size =     131072 B, free =     131072 B (100 % unused)
    CPU: mcu2_1: HEAP:  DDR_NON_CACHE_M: size =   67043328 B, free =   47144960 B (  6 % unused)

    CPU:  c6x_1: TASK:           IPC_RX:   0. 0 %
    CPU:  c6x_1: TASK:       REMOTE_SRV:   0. 0 %
    CPU:  c6x_1: TASK:         TIVX_CPU:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_RX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_1: TASK:      IPC_TEST_TX:   0. 0 %

    CPU:  c6x_1: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
    CPU:  c6x_1: HEAP:           L2_MEM: size =     229376 B, free =     229376 B (100 % unused)
    CPU:  c6x_1: HEAP:  DDR_SCRATCH_MEM: size =   50331648 B, free =   50331648 B ( 14 % unused)

    CPU:  c6x_2: TASK:           IPC_RX:   0. 0 %
    CPU:  c6x_2: TASK:       REMOTE_SRV:   0. 0 %
    CPU:  c6x_2: TASK:         TIVX_CPU:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_RX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c6x_2: TASK:      IPC_TEST_TX:   0. 0 %

    CPU:  c6x_2: HEAP:   DDR_SHARED_MEM: size =   16777216 B, free =   16774912 B ( 99 % unused)
    CPU:  c6x_2: HEAP:           L2_MEM: size =     229376 B, free =     229376 B (100 % unused)
    CPU:  c6x_2: HEAP:  DDR_SCRATCH_MEM: size =   50331648 B, free =   50331648 B ( 14 % unused)

    CPU:  c7x_1: TASK:           IPC_RX:   0. 7 %
    CPU:  c7x_1: TASK:       REMOTE_SRV:   0. 0 %
    CPU:  c7x_1: TASK:         TIVX_CPU:  77. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_RX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %
    CPU:  c7x_1: TASK:      IPC_TEST_TX:   0. 0 %

    CPU:  c7x_1: HEAP:   DDR_SHARED_MEM: size =  268435456 B, free =  211463168 B ( 14 % unused)
    CPU:  c7x_1: HEAP:           L3_MEM: size =    8159232 B, free =          0 B (  0 % unused)
    CPU:  c7x_1: HEAP:           L2_MEM: size =     491520 B, free =     491520 B (100 % unused)
    CPU:  c7x_1: HEAP:           L1_MEM: size =      16384 B, free =          0 B (  0 % unused)
    CPU:  c7x_1: HEAP:  DDR_SCRATCH_MEM: size =  234881024 B, free =  233944638 B (  8 % unused)

    GRAPH: app_multi_cam_graph (#nodes =   4, #executions =   1499)
     NODE:   CAPTURE1:             capture_node: avg =  27812 usecs, min/max =   9198 /  59283 usecs, #executions =       1499
     NODE: VPAC_VISS1:                viss_node: avg =  22780 usecs, min/max =  21402 /  23952 usecs, #executions =       1499
     NODE:     IPU1-0:                aewb_node: avg =   2963 usecs, min/max =    642 /  13985 usecs, #executions =       1499
     NODE:  VPAC_MSC1:              mosaic_node: avg =  17256 usecs, min/max =  16580 /  23650 usecs, #executions =       1499

     PERF:           FILEIO: avg =      0 usecs, min/max = 4294967295 /      0 usecs, #executions =          0
     PERF:            TOTAL: avg =  30362 usecs, min/max =  10210 /  50254 usecs, #executions =        500

     PERF:            TOTAL:   32.93 FPS



    GRAPH:        graph_157 (#nodes =   1, #executions =   1001)
     NODE:   DSP_C7-1:                 node_177: avg =  16011 usecs, min/max =  13844 /  18093 usecs, #executions =       1001

     PERF:             TIVX: avg =  19013 usecs, min/max =  15481 /  27952 usecs, #executions =        500

     PERF:             TIVX:   52.59 FPS

    GRAPH:        graph_179 (#nodes =   1, #executions =    501)
     NODE:   DISPLAY1:                 node_180: avg =   8600 usecs, min/max =     90 /  16851 usecs, #executions =        501

     PERF:             DISP: avg =  15626 usecs, min/max =   3980 /  41033 usecs, #executions =        500

     PERF:             DISP:   63.99 FPS