am62axx-evm: cannot allocate MSMC memory

Hello,

Please tell us more about what you are doing.

Which version of the AM62Ax Linux SDK and MCU+ SDK are you using?

What are you trying to accomplish?

What changes did you make? Specifically, I am curious if you made any modifications to the Linux devicetree files, and the memory allocations there. If you made any changes to the projects running on the remote cores, tell us what you changed there as well.

Regards,

Nick

hello，I use ti-firmware-builder-am62axx-evm-08.06.00.41/tidl_am62a_08_06_00_27 and i want use this demo named TI_DEVICE_armv8_test_dl_algo_host_rt.out  to run on the device of am62a 

i replace tidl_am62a_08_06_00_10 with tidl_am62a_08_06_00_27 .

Hello Shuai,

Based on your response, I assume you did not make any modifications to the Linux devicetree files.

Can you please confirm which EVM board you are using, and which filesystem image is getting loaded onto the EVM?

Regards,

Nick

Hello Shuai Wang,

You say that you are using a newer version of the tidl-am62a source code (8.6.0.27 vs. 8.6.0.41). Did you update the C7x firmware or use the version as-is in the SDK? There could be compatibility issues between artifacts generated for build-version 27 vs. 41. There is not compatibility between versions like 8.6 and 9.0

I'm not sure what you mean by output node limitation issue. Do you mean that your network has more than 16 outputs? It might also help to see the entirety of your output log where this error prints.

Could you also provide any of the compilation logs?

Best,
Reese

may be difference between 8.6.0.27 and 8.6.0.41 are small，but this is other Colleague pass it to me，i can’t ensure the compatibility issues。

i have not update the C7x firmware。do you mean i have to update it？

Yes，my network has more than 16 outputs，i try to run it on the  am62a board ，but it coredump like this，

so i use tool of

the source vison output as bellow:

and the TI_DEVICE_armv8_test_dl_algo_host_rt .out tool

output as bellow:

i use the AM62A board and the Official filesystem image 

Hello,

Shuai Wang said:

i have not update the C7x firmware。do you mean i have to update it？

I don't think this should be required, but it would be good to update this firmware and rule it out.

To do that, you would need to build that firmware with the firmware-builder and copy that into /lib/firmware. There is a symbolic link called am62a-c71_0-fw that points to the actual file, which defaults to /lib/firmware/vision_apps_eaik/vx_app_rtos_linux_c7x_1.out. You can change this symbolic link to point to your updated FW, and reboot for the changes to take effect.

Allowing > 16 outputs would certainly require further changes to the C7 firmware, and probably the arm-tidl code that runs from the Arm cores to communicate with the accelerator. To give further steps to solve this, I will need to involve engineers with more insight to the inner workings. This may take a few days to get a response from them.

-Reese

Hello,

Would you please look into the itidl_ti.h file in your version? There is a parameter "TIDL_NUM_OUT_BUFS" that is probably set to 16. Could you check this value? Please also run your test with TIDL_RT_DEBUG=1 exported. I would also recommend running /opt/vx_app_arm_remote_log.out in the background. This may give more information on why MSMC if failing to allocate, just in case it's an unrelated issue (although it is still necessary to increase the # outputs bufs if you have a 24 output model).

in general, to use the bugfix version of TIDL with the LInux SDK, you will need to change the firmware for the C7x. It is likely that there are too many changes to ensure compatibility.

You will also need to update the components that exist on Arm/Linux (there are mostly header files and compiled .SO libraries). Those exist in subdirectories of arm-tidl. Some of the top level scripts in the firmware builder should be able to help with this build process (e.g. make_sdk.sh in vision_apps directory of fw-builder root.

Best,
Reese

yes, parameter "TIDL_NUM_OUT_BUFS" is 16. and outputs bufs is 32

 Are  the "TIDL_NUM_OUT_BUFS" and the " TIDL_MAX_ALG_OUT_BUFS"  need not to change?

/opt/vx_app_arm_remote_log.out outputs as state:

[MCU1_0] 104020.614211 s: CIO: Init ... Done !!!
[MCU1_0] 104020.614272 s: ### CPU Frequency = 800000000 Hz
[MCU1_0] 104020.614304 s: APP: Init ... !!!
[MCU1_0] 104020.614325 s: MEM: Init ... !!!
[MCU1_0] 104020.614357 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ af000000 of size 16777216 bytes !!!
[MCU1_0] 104020.614425 s: MEM: Init ... Done !!!
[MCU1_0] 104020.614447 s: IPC: Init ... !!!
[MCU1_0] 104020.614506 s: IPC: 3 CPUs participating in IPC !!!
[MCU1_0] 104020.614540 s: IPC: Waiting for HLOS to be ready ... !!!
[MCU1_0] 104032.036071 s: IPC: HLOS is ready !!!
[MCU1_0] 104032.039736 s: IPC: Init ... Done !!!
[MCU1_0] 104032.039775 s: APP: Syncing with 2 CPUs ... !!!
[MCU1_0] 104032.818552 s: APP: Syncing with 2 CPUs ... Done !!!
[MCU1_0] 104032.818733 s: REMOTE_SERVICE: Init ... !!!
[MCU1_0] 104032.820471 s: REMOTE_SERVICE: Init ... Done !!!
[MCU1_0] 104032.820511 s: FVID2: Init ... !!!
[MCU1_0] 104032.820603 s: FVID2: Init ... Done !!!
[MCU1_0] 104032.820630 s: VHWA: VPAC Init ... !!!
[MCU1_0] 104032.820651 s: SCICLIENT: Sciclient_pmSetModuleState module=219 state=2
[MCU1_0] 104032.820766 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU1_0] 104032.820797 s: VHWA: LDC Init ... !!!
[MCU1_0] 104032.821108 s: VHWA: LDC Init ... Done !!!
[MCU1_0] 104032.821143 s: VHWA: MSC Init ... !!!
[MCU1_0] 104032.823096 s: VHWA: MSC Init ... Done !!!
[MCU1_0] 104032.823129 s: VHWA: VISS Init ... !!!
[MCU1_0] 104032.823459 s: VHWA: VISS Init ... Done !!!
[MCU1_0] 104032.823488 s: VHWA: VPAC Init ... Done !!!
[MCU1_0] 104032.823523 s: VX_ZONE_INIT:Enabled
[MCU1_0] 104032.823547 s: VX_ZONE_ERROR:Enabled
[MCU1_0] 104032.823570 s: VX_ZONE_WARNING:Enabled
[MCU1_0] 104032.824327 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target MCU1-0
[MCU1_0] 104032.824551 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target VPAC_LDC1
[MCU1_0] 104032.824765 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target VPAC_MSC1
[MCU1_0] 104032.824979 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target VPAC_MSC2
[MCU1_0] 104032.825196 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target VPAC_VISS1
[MCU1_0] 104032.825237 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
[MCU1_0] 104032.825268 s: APP: OpenVX Target kernel init ... !!!
[MCU1_0] 104032.853000 s: APP: OpenVX Target kernel init ... Done !!!
[MCU1_0] 104032.853032 s: VISS REMOTE SERVICE: Init ... !!!
[MCU1_0] 104032.853084 s: VISS REMOTE SERVICE: Init ... Done !!!
[MCU1_0] 104032.853130 s: APP: Init ... Done !!!
[MCU1_0] 104032.853155 s: APP: Run ... !!!
[MCU1_0] 104032.853177 s: IPC: Starting echo test ...
[MCU1_0] 104032.855025 s: APP: Run ... Done !!!
[MCU1_0] 104032.855143 s: IPC: Echo status: mpu1_0[x] mcu1_0[s] c7x_1[P]
[C7x_1 ] 104032.817535 s: CIO: Init ... Done !!!
[C7x_1 ] 104032.817554 s: ### CPU Frequency = 1000000000 Hz
[C7x_1 ] 104032.817570 s: APP: Init ... !!!
[C7x_1 ] 104032.817582 s: SCICLIENT: Init ... !!!
[C7x_1 ] 104032.817703 s: SCICLIENT: DMSC FW version [8.6.4--v08.06.04 (Chill Capybar]
[C7x_1 ] 104032.817723 s: SCICLIENT: DMSC FW revision 0x8
[C7x_1 ] 104032.817738 s: SCICLIENT: DMSC FW ABI revision 3.1
[C7x_1 ] 104032.817753 s: SCICLIENT: Init ... Done !!!
[C7x_1 ] 104032.817766 s: UDMA: Init ... !!!
[C7x_1 ] 104032.817777 s: UDMA: Init ... Done !!!
[C7x_1 ] 104032.817789 s: MEM: Init ... !!!
[C7x_1 ] 104032.817803 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ b2000000 of size 117440512 bytes !!!
[C7x_1 ] 104032.817831 s: MEM: Created heap (L3_MEM, id=1, flags=0x00000001) @ 7e000000 of size 1048576 bytes !!!
[C7x_1 ] 104032.817854 s: MEM: Created heap (L2_MEM, id=2, flags=0x00000001) @ 7f000000 of size 245760 bytes !!!
[C7x_1 ] 104032.817877 s: MEM: Created heap (L1_MEM, id=3, flags=0x00000001) @ 7f03c000 of size 16384 bytes !!!
[C7x_1 ] 104032.817900 s: MEM: Created heap (DDR_SCRATCH_MEM, id=4, flags=0x00000001) @ b9000000 of size 117440512 bytes !!!
[C7x_1 ] 104032.817924 s: MEM: Created heap (DDR_NON_CACHE_ME, id=5, flags=0x00000001) @ b0000000 of size 16777216 bytes !!!
[C7x_1 ] 104032.817948 s: MEM: Created heap (DDR_SCRATCH_NON_, id=6, flags=0x00000001) @ b1000000 of size 16777216 bytes !!!
[C7x_1 ] 104032.817972 s: MEM: Init ... Done !!!
[C7x_1 ] 104032.817984 s: IPC: Init ... !!!
[C7x_1 ] 104032.818007 s: IPC: 3 CPUs participating in IPC !!!
[C7x_1 ] 104032.818026 s: IPC: Waiting for HLOS to be ready ... !!!
[C7x_1 ] 104032.818043 s: IPC: HLOS is ready !!!
[C7x_1 ] 104032.818521 s: IPC: Init ... Done !!!
[C7x_1 ] 104032.818535 s: APP: Syncing with 2 CPUs ... !!!
[C7x_1 ] 104032.818551 s: APP: Syncing with 2 CPUs ... Done !!!
[C7x_1 ] 104032.818565 s: REMOTE_SERVICE: Init ... !!!
[C7x_1 ] 104032.818732 s: REMOTE_SERVICE: Init ... Done !!!
[C7x_1 ] 104032.818754 s: VX_ZONE_INIT:Enabled
[C7x_1 ] 104032.818769 s: VX_ZONE_ERROR:Enabled
[C7x_1 ] 104032.818781 s: VX_ZONE_WARNING:Enabled
[C7x_1 ] 104032.819401 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1
[C7x_1 ] 104032.819498 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_2
[C7x_1 ] 104032.819592 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_3
[C7x_1 ] 104032.819684 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_4
[C7x_1 ] 104032.819775 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_5
[C7x_1 ] 104032.819864 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_6
[C7x_1 ] 104032.819955 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_7
[C7x_1 ] 104032.820045 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_8
[C7x_1 ] 104032.820072 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
[C7x_1 ] 104032.820088 s: APP: OpenVX Target kernel init ... !!!
[C7x_1 ] 104032.820267 s: APP: OpenVX Target kernel init ... Done !!!
[C7x_1 ] 104032.820283 s: APP: Init ... Done !!!
[C7x_1 ] 104032.820296 s: APP: Run ... !!!
[C7x_1 ] 104032.820307 s: IPC: Starting echo test ...
[C7x_1 ] 104032.820471 s: APP: Run ... Done !!!
[C7x_1 ] 104032.853964 s: IPC: Echo status: mpu1_0[x] mcu1_0[P] c7x_1[s]
[C7x_1 ] 104124.387336 s: Alg Alloc for Layer # - 0
[C7x_1 ] 104124.387364 s: Alg Alloc for Layer # - 1
[C7x_1 ] 104124.387440 s: Alg Alloc for Layer # - 2
[C7x_1 ] 104124.387482 s: Alg Alloc for Layer # - 3
[C7x_1 ] 104124.387520 s: Alg Alloc for Layer # - 4
[C7x_1 ] 104124.387588 s: Alg Alloc for Layer # - 5
[C7x_1 ] 104124.387658 s: Alg Alloc for Layer # - 6
[C7x_1 ] 104124.387724 s: Alg Alloc for Layer # - 7
[C7x_1 ] 104124.387762 s: Alg Alloc for Layer # - 8
[C7x_1 ] 104124.387801 s: Alg Alloc for Layer # - 9
[C7x_1 ] 104124.387864 s: Alg Alloc for Layer # - 10
[C7x_1 ] 104124.387903 s: Alg Alloc for Layer # - 11
[C7x_1 ] 104124.387942 s: Alg Alloc for Layer # - 12
[C7x_1 ] 104124.387992 s: Alg Alloc for Layer # - 13
[C7x_1 ] 104124.388048 s: Alg Alloc for Layer # - 14
[C7x_1 ] 104124.388085 s: Alg Alloc for Layer # - 15
[C7x_1 ] 104124.388149 s: Alg Alloc for Layer # - 16
[C7x_1 ] 104124.388216 s: Alg Alloc for Layer # - 17
[C7x_1 ] 104124.388264 s: Alg Alloc for Layer # - 18
[C7x_1 ] 104124.388327 s: Alg Alloc for Layer # - 19
[C7x_1 ] 104124.388366 s: Alg Alloc for Layer # - 20
[C7x_1 ] 104124.388435 s: Alg Alloc for Layer # - 21
[C7x_1 ] 104124.388479 s: Alg Alloc for Layer # - 22
[C7x_1 ] 104124.388519 s: Alg Alloc for Layer # - 23
[C7x_1 ] 104124.388569 s: Alg Alloc for Layer # - 24
[C7x_1 ] 104124.388623 s: Alg Alloc for Layer # - 25
[C7x_1 ] 104124.388662 s: Alg Alloc for Layer # - 26
[C7x_1 ] 104124.388724 s: Alg Alloc for Layer # - 27
[C7x_1 ] 104124.388792 s: Alg Alloc for Layer # - 28
[C7x_1 ] 104124.388838 s: Alg Alloc for Layer # - 29
[C7x_1 ] 104124.388901 s: Alg Alloc for Layer # - 30
[C7x_1 ] 104124.388967 s: Alg Alloc for Layer # - 31
[C7x_1 ] 104124.389030 s: Alg Alloc for Layer # - 32
[C7x_1 ] 104124.389091 s: Alg Alloc for Layer # - 33
[C7x_1 ] 104124.389130 s: Alg Alloc for Layer # - 34
[C7x_1 ] 104124.389192 s: Alg Alloc for Layer # - 35
[C7x_1 ] 104124.389234 s: Alg Alloc for Layer # - 36
[C7x_1 ] 104124.389277 s: Alg Alloc for Layer # - 37
[C7x_1 ] 104124.389338 s: Alg Alloc for Layer # - 38
[C7x_1 ] 104124.389395 s: Alg Alloc for Layer # - 39
[C7x_1 ] 104124.389468 s: Alg Alloc for Layer # - 40
[C7x_1 ] 104124.389531 s: Alg Alloc for Layer # - 41
[C7x_1 ] 104124.389570 s: Alg Alloc for Layer # - 42
[C7x_1 ] 104124.389613 s: Alg Alloc for Layer # - 43
[C7x_1 ] 104124.389656 s: Alg Alloc for Layer # - 44
[C7x_1 ] 104124.389696 s: Alg Alloc for Layer # - 45
[C7x_1 ] 104124.389735 s: Alg Alloc for Layer # - 46
[C7x_1 ] 104124.389799 s: Alg Alloc for Layer # - 47
[C7x_1 ] 104124.389867 s: Alg Alloc for Layer # - 48
[C7x_1 ] 104124.389929 s: Alg Alloc for Layer # - 49
[C7x_1 ] 104124.389996 s: Alg Alloc for Layer # - 50
[C7x_1 ] 104124.390058 s: Alg Alloc for Layer # - 51
[C7x_1 ] 104124.390125 s: Alg Alloc for Layer # - 52
[C7x_1 ] 104124.390164 s: Alg Alloc for Layer # - 53
[C7x_1 ] 104124.390209 s: Alg Alloc for Layer # - 54
[C7x_1 ] 104124.390254 s: Alg Alloc for Layer # - 55
[C7x_1 ] 104124.390294 s: Alg Alloc for Layer # - 56
[C7x_1 ] 104124.390335 s: Alg Alloc for Layer # - 57
[C7x_1 ] 104124.390399 s: Alg Alloc for Layer # - 58
[C7x_1 ] 104124.390473 s: Alg Alloc for Layer # - 59
[C7x_1 ] 104124.390539 s: Alg Alloc for Layer # - 60
[C7x_1 ] 104124.390607 s: Alg Alloc for Layer # - 61
[C7x_1 ] 104124.390670 s: Alg Alloc for Layer # - 62
[C7x_1 ] 104124.390736 s: Alg Alloc for Layer # - 63
[C7x_1 ] 104124.390793 s: Alg Alloc for Layer # - 64
[C7x_1 ] 104124.390857 s: Alg Alloc for Layer # - 65
[C7x_1 ] 104124.390920 s: Alg Alloc for Layer # - 66
[C7x_1 ] 104124.390976 s: Alg Alloc for Layer # - 67
[C7x_1 ] 104124.391041 s: Alg Alloc for Layer # - 68
[C7x_1 ] 104124.391108 s: Alg Alloc for Layer # - 69
[C7x_1 ] 104124.391172 s: Alg Alloc for Layer # - 70
[C7x_1 ] 104124.391238 s: Alg Alloc for Layer # - 71
[C7x_1 ] 104124.391301 s: Alg Alloc for Layer # - 72
[C7x_1 ] 104124.391369 s: Alg Alloc for Layer # - 73
[C7x_1 ] 104124.391432 s: Alg Alloc for Layer # - 74
[C7x_1 ] 104124.391500 s: Alg Alloc for Layer # - 75
[C7x_1 ] 104124.391564 s: Alg Alloc for Layer # - 76
[C7x_1 ] 104124.391606 s: Alg Alloc for Layer # - 77
[C7x_1 ] 104124.391670 s: Alg Alloc for Layer # - 78
[C7x_1 ] 104124.391737 s: Alg Alloc for Layer # - 79
[C7x_1 ] 104124.391800 s: Alg Alloc for Layer # - 80
[C7x_1 ] 104124.391866 s: Alg Alloc for Layer # - 81
[C7x_1 ] 104124.391930 s: Alg Alloc for Layer # - 82
[C7x_1 ] 104124.391998 s: Alg Alloc for Layer # - 83
[C7x_1 ] 104124.392055 s: Alg Alloc for Layer # - 84
[C7x_1 ] 104124.392119 s: Alg Alloc for Layer # - 85
[C7x_1 ] 104124.392186 s: Alg Alloc for Layer # - 86
[C7x_1 ] 104124.392250 s: Alg Alloc for Layer # - 87
[C7x_1 ] 104124.392290 s: Alg Alloc for Layer # - 88
[C7x_1 ] 104124.392314 s: Alg Alloc for Layer # - 89
[C7x_1 ] 104124.392378 s: Alg Alloc for Layer # - 90
[C7x_1 ] 104124.392419 s: Alg Alloc for Layer # - 91
[C7x_1 ] 104124.392452 s: Alg Alloc for Layer # - 92
[C7x_1 ] 104124.392516 s: Alg Alloc for Layer # - 93
[C7x_1 ] 104124.392558 s: Alg Alloc for Layer # - 94
[C7x_1 ] 104124.392582 s: Alg Alloc for Layer # - 95
[C7x_1 ] 104124.392644 s: Alg Alloc for Layer # - 96
[C7x_1 ] 104124.392686 s: Alg Alloc for Layer # - 97
[C7x_1 ] 104124.392709 s: Alg Alloc for Layer # - 98
[C7x_1 ] 104124.392761 s: Alg Alloc for Layer # - 99
[C7x_1 ] 104124.392813 s: Alg Alloc for Layer # - 100
[C7x_1 ] 104124.392879 s: Alg Alloc for Layer # - 101
[C7x_1 ] 104124.392943 s: Alg Alloc for Layer # - 102
[C7x_1 ] 104124.393015 s: Alg Alloc for Layer # - 103
[C7x_1 ] 104124.393049 s: Alg Alloc for Layer # - 104
[C7x_1 ] 104124.393072 s: Alg Alloc for Layer # - 105
[C7x_1 ] 104124.393134 s: Alg Alloc for Layer # - 106
[C7x_1 ] 104124.393177 s: Alg Alloc for Layer # - 107
[C7x_1 ] 104124.393201 s: Alg Alloc for Layer # - 108
[C7x_1 ] 104124.393263 s: Alg Alloc for Layer # - 109
[C7x_1 ] 104124.393306 s: Alg Alloc for Layer # - 110
[C7x_1 ] 104124.393330 s: Alg Alloc for Layer # - 111
[C7x_1 ] 104124.393392 s: Alg Alloc for Layer # - 112
[C7x_1 ] 104124.393440 s: Alg Alloc for Layer # - 113
[C7x_1 ] 104124.393465 s: Alg Alloc for Layer # - 114
[C7x_1 ] 104124.393528 s: Alg Alloc for Layer # - 115
[C7x_1 ] 104124.393570 s: Alg Alloc for Layer # - 116
[C7x_1 ] 104124.393594 s: Alg Alloc for Layer # - 117
[C7x_1 ] 104124.393655 s: Alg Alloc for Layer # - 118
[C7x_1 ] 104124.393698 s: Alg Alloc for Layer # - 119
[C7x_1 ] 104124.393720 s: Alg Alloc for Layer # - 120
[C7x_1 ] 104124.393781 s: Alg Alloc for Layer # - 121
[C7x_1 ] 104124.393824 s: Alg Alloc for Layer # - 122
[C7x_1 ] 104124.393847 s: Alg Alloc for Layer # - 123
[C7x_1 ] 104124.393908 s: Alg Alloc for Layer # - 124
[C7x_1 ] 104124.393950 s: Alg Alloc for Layer # - 125
[C7x_1 ] 104124.393972 s: Alg Alloc for Layer # - 126
[C7x_1 ] 104124.394033 s: Alg Alloc for Layer # - 127
[C7x_1 ] 104124.394097 s: Alg Alloc for Layer # - 128
[C7x_1 ] 104124.394142 s: Alg Alloc for Layer # - 129
[C7x_1 ] 104124.394165 s: Alg Alloc for Layer # - 130
[C7x_1 ] 104124.394227 s: Alg Alloc for Layer # - 131
[C7x_1 ] 104124.394272 s: Alg Alloc for Layer # - 132
[C7x_1 ] 104124.394295 s: Alg Alloc for Layer # - 133
[C7x_1 ] 104124.394358 s: Alg Alloc for Layer # - 134
[C7x_1 ] 104124.394405 s: Alg Alloc for Layer # - 135
[C7x_1 ] 104124.394433 s: Alg Alloc for Layer # - 136
[C7x_1 ] 104124.394498 s: Alg Alloc for Layer # - 137
[C7x_1 ] 104124.394546 s: Alg Alloc for Layer # - 138
[C7x_1 ] 104124.394569 s: Alg Alloc for Layer # - 139
[C7x_1 ] 104124.394619 s: Alg Alloc for Layer # - 140
[C7x_1 ] 104124.394685 s: Alg Alloc for Layer # - 141
[C7x_1 ] 104124.394734 s: Alg Alloc for Layer # - 142
[C7x_1 ] 104124.394757 s: Alg Alloc for Layer # - 143
[C7x_1 ] 104124.394822 s: Alg Alloc for Layer # - 144
[C7x_1 ] 104124.394870 s: Alg Alloc for Layer # - 145
[C7x_1 ] 104124.394894 s: Alg Alloc for Layer # - 146
[C7x_1 ] 104124.394959 s: Alg Alloc for Layer # - 147
[C7x_1 ] 104124.395009 s: Alg Alloc for Layer # - 148
[C7x_1 ] 104124.395032 s: Alg Alloc for Layer # - 149
[C7x_1 ] 104124.395097 s: Alg Alloc for Layer # - 150
[C7x_1 ] 104124.395148 s: Alg Alloc for Layer # - 151
[C7x_1 ] 104124.395470 s:
[C7x_1 ] 104124.395491 s: --------------------------------------------
[C7x_1 ] 104124.395519 s: TIDL Memory size requiement (record wise):
[C7x_1 ] 104124.395552 s: MemRecNum , Space , Attribute , Size(KBytes)
[C7x_1 ] 104124.395591 s: 0 , DDR Cacheable, Persistent , 14.84
[C7x_1 ] 104124.395628 s: 1 , DDR Cacheable, Persistent , 0.14
[C7x_1 ] 104124.395665 s: 2 , DDR Cacheable, Scratch , 16.00
[C7x_1 ] 104124.395702 s: 3 , DDR Cacheable, Scratch , 4.00
[C7x_1 ] 104124.395738 s: 4 , DDR Cacheable, Scratch , 56.00
[C7x_1 ] 104124.395775 s: 5 , DDR Cacheable, Persistent , 468.21
[C7x_1 ] 104124.395812 s: 6 , DDR Cacheable, Scratch , 0.25
[C7x_1 ] 104124.395848 s: 7 , DDR Cacheable, Scratch , 0.13
[C7x_1 ] 104124.395884 s: 8 , DDR Cacheable, Scratch , 19662.13
[C7x_1 ] 104124.395921 s: 9 , DDR Cacheable, Scratch , 26211.00
[C7x_1 ] 104124.395957 s: 10 , DDR Cacheable, Scratch , 0.13
[C7x_1 ] 104124.395994 s: 11 , DDR Cacheable, Persistent , 1375.88
[C7x_1 ] 104124.396030 s: 12 , DDR Cacheable, Scratch , 512.25
[C7x_1 ] 104124.396065 s: 13 , DDR Cacheable, Persistent , 0.13
[C7x_1 ] 104124.396101 s: 14 , DDR Cacheable, Persistent , 11780.09
[C7x_1 ] 104124.396131 s: --------------------------------------------
[C7x_1 ] 104124.396159 s: Total memory size requirement (space wise):
[C7x_1 ] 104124.396183 s: Mem Space , Size(KBytes)
[C7x_1 ] 104124.396205 s: DDR Cacheable, 60101.15
[C7x_1 ] 104124.396232 s: --------------------------------------------
[C7x_1 ] 104124.396271 s: NOTE: Memory requirement in host emulation can be different from the same on EVM
[C7x_1 ] 104124.396314 s: To get the actual TIDL memory requirement make sure to run on EVM with
[C7x_1 ] 104124.396342 s: debugTraceLevel = 2
[C7x_1 ] 104124.396354 s:
[C7x_1 ] 104124.396376 s: --------------------------------------------
[C7x_1 ] 104124.400978 s: Not able to allocate MSMC memory
[C7x_1 ] 104124.401011 s: VX_ZONE_ERROR:[tivxAlgiVisionCreate:335] Calling ialg.algInit failed with status = -1
[C7x_1 ] 104124.401064 s: Error: Trying to remove the pririoty object without initializing
[C7x_1 ] 104124.401140 s: PREEMPTION: Removing priroty object with handle = b2b90d00 and targetPriority = 0, Number of obejcts left are = -1, removed object with base = 0 and size =0
[C7x_1 ] 104124.401185 s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:720] tivxAlgiVisionCreate returned NULL
[C7x_1 ] 104124.401140 s: PREEMPTION: Removing priroty object with handle = b2b90d00 and targetPriority = 0, Number of obejcts left are = -1, removed object with base = 0 and size =0
[C7x_1 ] 104124.401140 s: PREEMPTION: Removing priroty object with handle = b2b90d00 and targetPriority = 0, Number of obejcts left are = -1, removed object with base = 0 and size =0

You will also need to update the components that exist on Arm/Linux (there are mostly header files and compiled .SO libraries). Those exist in subdirectories of arm-tidl. Some of the top level scripts in the firmware builder should be able to help with this build process (e.g. make_sdk.sh in vision_apps directory of fw-builder root.

TI_DEVICE_armv8_test_dl_algo_host_rt .out outputs as state:

export TIDL_RT_DEBUG=1 

----------------------- TIDL tidlMain ------------------------

Processing config file #0 : out/import_cfg.txt.qunat_stats_config.txt
Input : dataId=0, name=input.1_original, elementType 0, scale=1.000000, zero point=0, layout=0
Ouput : dataId=144, name=828, elementType 1, scale=19.835835, zero point=0, layout=0
Ouput : dataId=145, name=832, elementType 1, scale=6.582086, zero point=0, layout=0
Ouput : dataId=146, name=836, elementType 1, scale=62.680580, zero point=0, layout=0
Ouput : dataId=147, name=840, elementType 1, scale=16.320927, zero point=0, layout=0
Ouput : dataId=117, name=899, elementType 1, scale=4.578856, zero point=0, layout=0
Ouput : dataId=114, name=882, elementType 1, scale=18.303440, zero point=0, layout=0
Ouput : dataId=115, name=886, elementType 1, scale=4.067142, zero point=0, layout=0
Ouput : dataId=116, name=890, elementType 1, scale=27.690706, zero point=0, layout=0
Ouput : dataId=118, name=903, elementType 1, scale=1760.337769, zero point=0, layout=0
Ouput : dataId=119, name=907, elementType 1, scale=9.977260, zero point=0, layout=0
Ouput : dataId=120, name=911, elementType 1, scale=30.790319, zero point=0, layout=0
Ouput : dataId=121, name=915, elementType 1, scale=3.858507, zero point=0, layout=0
Ouput : dataId=110, name=847, elementType 1, scale=23.214573, zero point=0, layout=0
Ouput : dataId=111, name=851, elementType 1, scale=5.261847, zero point=0, layout=0
Ouput : dataId=112, name=855, elementType 1, scale=37.960674, zero point=0, layout=0
Ouput : dataId=113, name=859, elementType 1, scale=7.851976, zero point=0, layout=0
Ouput : dataId=75, name=866, elementType 1, scale=12.686906, zero point=0, layout=0
Ouput : dataId=76, name=870, elementType 1, scale=5.608529, zero point=0, layout=0
Ouput : dataId=77, name=874, elementType 1, scale=64.547394, zero point=0, layout=0
Ouput : dataId=78, name=878, elementType 1, scale=7.224996, zero point=0, layout=0
12062808, 11.504 0xffff76991010
worstCaseDelay for Pre-emption is 3.1522312
Network File Read done
Calling appInit() in TIDL-RT!
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=4) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
104124.271188 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
104124.277303 s: VX_ZONE_INIT:Enabled
104124.277349 s: VX_ZONE_ERROR:Enabled
104124.277366 s: VX_ZONE_WARNING:Enabled
104124.278467 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
104124.279031 s: VX_ZONE_INIT:[tivxHostInitLocal:93] Initialization Done for HOST !!!
TIDL_RT_OVX: Init ...
TIDL_RT_OVX: Mapping config file ...
TIDL_RT_OVX: Mapping config file ... Done. 37256 bytes
TIDL_RT_OVX: Tensors, input = 1, output = 20
TIDL_RT_OVX: Mapping network file
TIDL_RT_OVX: Mapping network file... Done 12062808 bytes
init_tidl_tiovx -------------Init done
TIDL_RT_OVX: Init done.
TIDL_RT_OVX: Creating graph ...
TIDL_RT_OVX: input_sizes[0] = 800, dim = 800 padL = 0 padR = 0
TIDL_RT_OVX: input_sizes[1] = 409600, dim = 512 padT = 0 padB = 0
TIDL_RT_OVX: input_sizes[2] = 3, dim = 3
TIDL_RT_OVX: input_sizes[3] = 1, dim = 1
TIDL_RT_OVX: input_buffer = 0xffff6ecbc000 1228800
TIDL_RT_OVX: Creating graph ... Done.
104124.402092 s: VX_ZONE_ERROR:[ownContextSendCmd:799] Command ack message returned failure cmd_status: -1
104124.402123 s: VX_ZONE_ERROR:[ownContextSendCmd:835] tivxEventWait() failed.
104124.402463 s: VX_ZONE_ERROR:[ownNodeKernelInit:527] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
104124.402491 s: VX_ZONE_ERROR:[ownNodeKernelInit:528] Please be sure the target callbacks have been registered for this core
104124.402510 s: VX_ZONE_ERROR:[ownNodeKernelInit:529] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
104124.402526 s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl:1:20 ... failed !!!
104124.402552 s: VX_ZONE_ERROR:[vxVerifyGraph:2055] Node kernel init failed
104124.402566 s: VX_ZONE_ERROR:[vxVerifyGraph:2109] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
TIDLRT_create ------------- Init done
Error at line: 507 : in file /home/lixing/AM62A/ti-firmware-builder-am62axx-evm-08.06.00.41/tidl_am62a_08_06_00_27/arm-tidl/rt/test/armv8/../src/tidl_tb.c, of function : tidlMultiInstanceTest
Invalid Error Type!

in above ,i have not changed any code , and run the 20 output network output error, do i need to change any code to slove this issue? if it needs,how can i fix the issue? please instruct me  to fix it thanks!

best

shuai

Hello,

I would like to confirm if it is the number of outputs giving you the error or something else with the network. I see the TIDL node is failing to initialize for your network, but the cause it not obvious from the log. Could you test that this is linked to the number of outputs by importing the model with only 16 (or fewer) outputs enabled and see if that initializes correctly without this error? This will require explicitly setting the outputs in the import config (parameter "outDataNamesList" and outputs in comma separated string, e.g. outDataNamesList = "output1, output2, output3")

Shuai Wang said:

 Are  the "TIDL_NUM_OUT_BUFS" and the " TIDL_MAX_ALG_OUT_BUFS"  need not to change?

I will continue my response under the assumption that >16 nodes is the cause for error.

TIDL_NUM_OUT_BUFS will need to change, but TIDL_MAX_ALG_OUT_BUFS does not need to. I would set the former to 24 and leave the latter as is.

With this change done, you would need then to generate the firmware for C7x and copy that into the filesystem under /lib/firmware.The makefiles for this will be under the fw-builder-root/vision_apps directory. The Makefile should have a 'firmware' target, although the exact commands may be in another file under the makerules subdirectory (most likely makefile_linux_arm.mak).

In addition, you will need to update some of the arm-tidl components. These go onto the target file system, mostly under /usr/lib. This will be done when you rebuild c7x-mma-tidl. The makefiles within the vision_apps directory in 8.6-fw-builder can handle this, specifically for the makerules/makefile_linux_arm.mak. I believe the following lines are the main relevant ones (L200-L204, I believe): 

	cp -P $(TIDL_PATH)/../arm-tidl/tfl_delegate/out/$(TARGET_SOC)/A72/LINUX/$(LINUX_APP_PROFILE)/*.so*  $(LINUX_FS_STAGE_PATH)/usr/lib
	cp -P $(TIDL_PATH)/../arm-tidl/onnxrt_ep/out/$(TARGET_SOC)/A72/LINUX/$(LINUX_APP_PROFILE)/*.so*  $(LINUX_FS_STAGE_PATH)/usr/lib
	cp -P $(TIDL_PATH)/../arm-tidl/rt/out/$(TARGET_SOC)/A72/LINUX/$(LINUX_APP_PROFILE)/*.so*  $(LINUX_FS_STAGE_PATH)/usr/lib
	cp $(TIDL_PATH)/../arm-tidl/rt/out/$(TARGET_SOC)/A72/LINUX/$(LINUX_APP_PROFILE)/*.out     $(LINUX_FS_STAGE_PATH)/opt/tidl_test/
	cp -r $(TIDL_PATH)/test/testvecs/ $(LINUX_FS_STAGE_PATH)/opt/tidl_test/

Best Regards,
Reese Grimsley

Reese Grimsley said:

error

i have set "TIDL_NUM_OUT_BUFS" to 24 or 16, and it imports and inferences success on x86 PC ,i need update the component as you say and validate that on a new am62a device ,

Reese Grimsley said:

change

Good progress, Shuai.

Shuai Wang said:

imports and inferences success on x86 PC

Thank you for confirming the number of outputs buffers is the source of this error. Am I right to assume that the error was also showing in x86 PC emulation mode? I cannot tell from past logs, as they all look like they were executed from the target SOC

I can give guidance on updating the AM62A software/firmware as next steps. Try the advice in my previous response once you return next week. We will only need to update TIDL components, and not the entire SDK. Please inform me how this goes!

Best,
Reese

 the error was not  showing in x86 PC emulation mode and that inferences success.

the network 20 output and 16 output have no problems, both can run success on x86 PC,as bellow:

i have try the  advice in your previous response , set "TIDL_NUM_OUT_BUFS" to 24 and compile success , then 

copy

       vision_apps/out/AM62A/C7504/FREERTOS/release/vx_app_rtos_linux_c7x_1.out      to     /lib/firmware/

        onnxrt_ep/out/AM62A/A72/LINUX/release/libtidl_onnxrt_EP.so.1.0                              to   /usr/lib

       rt/out/AM62A/A72/LINUX/release/libvx_tidl_rt.so.1.0                                                      to    /usr/lib

       rt/out/AM62A/A72/LINUX/release/TI_DEVICE_armv8_test_dl_algo_host_rt.out            to    /opt/tidl_test/

run 

     use the 20 output network

     ./TI_DEVICE_armv8_test_dl_algo_host_rt.out s:out/import_cfg.txt.qunat_stats_config.txt

the tool "TI_DEVICE_armv8_test_dl_algo_host_rt.out " has code as this

 i validate it on am62a as bellow:

/opt/vx_app_arm_remote_log.out outputs:

use the 16 output network:

the tool "TI_DEVICE_armv8_test_dl_algo_host_rt.out " can not run successing whatever the network is . 

when i use a tool with no  vxSetNodeTarget function, code as this

use the 16 output network:

it runs ok

but use the 20 output nwtwork:

it coredumps

how could i change the code and update TIDL components , is this a tidl bug ,how could i solve the problem ? please instruct me,  thanks!

best 

shuai

Hi Shuai,

Let me make sure I follow your progress correctly.

• You have modified the model to have a 16 output version. This works on PC and AM62A without the MSMC bug you encounter
• The >16 output version gives error on PC but not AM62A
  • this may be a bug, in that there is difference between emulation and target
• You modified itidl_ti.h to allow 24 output buffers and recompiled the firmware
  • You copied the firmware and .SO files into the device filesystem
    • The problem persists with the 20 output network

Please correct if I am missing any understanding.

The first thing I would check is that the firmware running is in fact what you modified. I see that you copied the firmware binary into /lib/firmware directly. You may be missing a step here. The default firmware is /lib/firmware/am62a-c71_0-fw, which is actually a symbolic link (see screenshot) to a binary in vision_apps_eaik/ subdirectory

You may also try adding a print somewhere that includes the TIDL_NUM_OUT_BUFS variable. A function like TIDLRT_Create in arm-tidl and TIDL_init in ti_dl/algo (runs on C7x) would be worthwhile. If there were errors loading firmware, it should show in dmesg logs for this core (remoteproc0)

Best,
Reese
 

my "/lib/firmware/am62a-c71_0-fw" is linked to this

shoud i change the link?

Hello,

Yes, you should change that. It's strange that it was linked to that binary in the first place, actually. 

hello , now i have changed that,

it also run error and 

/opt/vx_app_arm_remote_log.out outputs:

i use this package as bellow, it was linked to that binary

" The >16 output version gives error on PC but not AM62A" is actually "The >16 output version gives error on AM62A but not PC"

and this tool "TI_DEVICE_armv8_test_dl_algo_host_rt.out" gives error on AM62A

That's the point

can you give  me some advise?

now is the right "vx_app_rtos_linux_c7x_1.out" i have changed it gives info as i have print

the  configure of the sizeL3MemKB is 1024kb as the device_config

991232/1024 + 56 = 1024kb

is that

tisdk-edgeai-image-am62axx-evm.wic.xz package has the problem?

best,

shuai

now i annotate the code

"vxSetNodeTarget(obj->tidl_node, VX_TARGET_STRING, priors[8 * (obj->coreNum - 1) + obj->targetPriority]);"

it gives error 

Hi Shuai,

Thank for the quick response and for trying this. It seems like the changes were implemented for the firmware. It is maybe still good to print that "24" is the setting for TIDL_NUM_OUT_BUFS. 

Shuai Wang said:

" The >16 output version gives error on PC but not AM62A" is actually "The >16 output version gives error on AM62A but not PC"

Oops, typo on my part. Yes, that is what I meant to say that the error is on SoC but not PC.

Have you recompiled the model since changing the itidl_ti.h? If that .H file is changed, then the import tool should as well -- ditto for the compiled model. Please update the import tool, recompile the model, and try again with this firmware.  I want to rule this out. Please also post your inference config.txt

If this does not solve the issue, then I'll need to involve development team to help. Most likely, the first recommendation will be to try this on a newer SDK (v9.1) in case this is a bug that is already fixed.

Shuai Wang said:

"vxSetNodeTarget(obj->tidl_node, VX_TARGET_STRING, priors[8 * (obj->coreNum - 1) + obj->targetPriority]);"

I'm not sure I understand the goal of this change. I'm not surprised that it failed to initialize, because this sets the TIDL node as the target the graph will run on.

hello Reese,

i take back what i  said that "The >16 output version gives error on AM62A but not PC", it's actually "The >16 output version gives error on AM62A both PC"

i set TIDL_NUM_OUT_BUFS = 24 and recompile the model since changing the itidl_ti.h in three places

targetfs/usr/include/processor_sdk/tidl_j7/ti_dl/inc/itidl_ti.h

tidl_am62a_08_06_00_27/ti_dl/inc/itidl_ti.h

tidl_am62a_08_06_00_27/arm-tidl/rt/inc/itidl_ti.h

and update the import tool recompile the model 

 it coredumps as bellow:

import.cfg.txt shows bellow which "./bin/tidl_model_import.out import_cfg.txt" success when set TIDL_NUM_OUT_BUFS = 16 and update the import tool.

so it turns out that  first,  it can not support more than 16 output network both PC and am62a, second, "TI_DEVICE_armv8_test_dl_algo_host_rt.out" tool have a MSMC bug on 8.6? 

this is my inference config.txt

i wish you can give me some support on my accomplish

TI_DEVICE_armv8_test_dl_algo_host_rt.out :s/out/import_cfg.txt.qunat_stats_config.txt

i have try it on SDK (v9.1), it runs successfully ,thanks for your patient advise!!

best regards

shuai,

hello Reese,

yes, it is full 20 output network and doesn't need to change the value of TIDL_NUM_OUT_BUFS ,which seems need not to be changed.

i plan to update the sdk version  to 9.1,

Great, I'm glad we were able to solve this issue! I will mark this as resolved and close the ticket.

If you have more to add, you can make a new thread and link to this one. You can also reply again here and it will reopen the thread.