SK-AM62A-LP: Manually compile U-Boot for EVM

Hi Bob,

I will test this on the EVM and get back to you.

Regards,
Aparna

Hi,

Could you explain about the Edge AI GUI hang issue? Does it occur at every boot?
There seems to be no problem with U-boot build procedure.

Regards,
Aparna

When we try to start any of the AI demos, nothing happens.

This is affecting our custom design with the 09.01 SDK release but we haven't been able to find anything missing between the evm and our design.  Building u-boot manually happens to present the same kind of failure on the evm so we are hoping this leads us to a solution.

Aparna,

Thank you for the help.  Can you share the exact commands you used to perform the build? I'm curious if you compiled your own firmware files or if you used the ones in the 9.1 Processor SDK.  The specific details matter and will help me troubleshoot the process I used.

The documentation I linked to above is not really complete.  It has at least two placeholder items that could be important:

"The location of the toolchain can be found here: location-in-sdk. If the Processor SDK is not used, the toolchain will need to be downloaded and toolchain path will need to be set, more information can be found here: download-arm-toolchains."

In regards to the GUI hang issue, the Edge AI demo comes up and displays the various buttons on the left side of the screen. I am able to move the pointer on top of a button and press it, but when I do so, the GUI hangs and no video or output is displayed in the center of the screen. I have to manually kill the GUI demo process (and relaunching it has the same effect).

Thanks!

Hi Bob,

Bob Duke said:

Can you share the exact commands you used to perform the build? I'm curious if you compiled your own firmware files or if you used the ones in the 9.1 Processor SDK

I have used the TI SDK provided firmware, to re-build U-boot, following the exact commands provided in SDK documentation.

Bob Duke said:

"The location of the toolchain can be found here: location-in-sdk. If the Processor SDK is not used, the toolchain will need to be downloaded and toolchain path will need to be set, more information can be found here: download-arm-toolchains."

Thanks for pointing it out. There is some discrepancy in the documentation. Follow the below link for access: https://software-dl.ti.com/processor-sdk-linux/esd/AM62X/09_01_00_08/exports/docs/linux/Foundational_Components/U-Boot/UG-General-Info.html#build-u-boot

Bob Duke said:

In regards to the GUI hang issue, the Edge AI demo comes up and displays the various buttons on the left side of the screen. I am able to move the pointer on top of a button and press it, but when I do so, the GUI hangs and no video or output is displayed in the center of the screen. I have to manually kill the GUI demo process (and relaunching it has the same effect).

I am assigning the thread to the concerned Edge AI topic expert for the GUI hang related issue.

Regards,
Aparna

Hi Bob and Jonathan,

Are you compiling u-boot for the DM-R5 core as well? Which firmware are you using for this?

We do not have the source for this DM-R5 core's firmware publicly available. There is a repository that hosts the firmware for these cores on a per-release basis. Please see in this directory: https://git.ti.com/cgit/processor-sdk/psdk_fw/tree/am62a/vision_apps_eaik?h=main. The MCU firmware is for the DM-R5, and should be pointed to when building u-boot. Please make sure to use the firmware with the release tag that makes the SDK you are building for (9.1, if I'm not mistaken)

-Reese

The only firmwares we are building ourselves are the OPTEE and TFA firmwares.

BINMAN_INDIRS=/home/tools/mitysom-62x/ti-processor-sdk-linux-edgeai-am62axx-evm-09_01_00_07/board-support/prebuilt-images

The MCU firmwares should be getting pulled from the 09.01.00.07 processor sdk.

Hi Jonathan,

Correct, you should not be directly building the DM-R5 firmware that is active after the boot process.

Jonathan Cormier said:

BINMAN_INDIRS=/home/tools/mitysom-62x/ti-processor-sdk-linux-edgeai-am62axx-evm-09_01_00_07/board-support/prebuilt-images

That looks like the right location for the firmware, and the firmware under the ti-dm/am62axx directory looks to be correct (ignore the name... it is more than an echo test). I had a similar thread that was resolved by pulling in the right firmware. I am not sure if this is relevant for your situation: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1282935/am62a7-cannot-run-example-imx219-pipeline-rpmsg-errors

I would need to see a few more logs to know the source of the issue. I'd like to see where this is hanging/failing within the OOB demo application. When that application is launched from CLI, are there are notable logs / issues? I suggest running /opt/vx_app_arm_remote_log.out in the background during this to check for any OpenVX errors.

- You can also try this demo app to avoid any GUI:

• root@am62axx-evm:/opt/edgeai-gst-apps#   ./optiflow/optiflow.py  ./configs/object_detection.yaml
• The GUI is calling gstreamer commands like this under the hood, so it should be hitting the same components that might be failing.

Reese Grimsley said:

That looks like the right location for the firmware, and the firmware under the ti-dm/am62axx directory looks to be correct (ignore the name... it is more than an echo test). I had a similar thread that was resolved by pulling in the right firmware. I am not sure if this is relevant for your situation: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1282935/am62a7-cannot-run-example-imx219-pipeline-rpmsg-errors

Yes that was us as well, though it was for the 09.00 SDK which got resolved and only affected manual builds of u-boot.  For this error report, both manual and yocto u-boot builds fail in similar ways on our boards.  And I think only the manual u-boot build fails for the EVM...

Were you able to manually build u-boot for the EVM and successfully run the AI demos?

Hi Jonathan,

I was able to use the u-boot binaries from Aparna to boot the 9.1 SDK and run the AI demos. I haven't been able to reproduce the GUI hanging.

I'll note that the first time I booted, I had a IMX219 CSI camera plugged in (and enabled with DTBO). When I opened the tty terminal and logged in with root, the prompt hung. It was probably failing within an init script called from the /home/root/.profile that runs on new user sessions. I rebooted with a few configurations of cameras / mouse plugged in, and I'm still able to run the AI demos. That hang on init-script only happened the one time.

Aparna mentioned to me that the GUI hangs when ethernet is connected to the board, but I haven't been able to reproduce this issue.

-Reese

Aparna and Reese,

Thank you for your help.

Reese, I will try the alternative methods you mentioned to launch the demos and I'll post any relevant logs.

Aparna, the reason I asked for your exact build commands is that the instructions in the SDK are... ambiguous. It states that "The instructions below assume all binaries are built manually." Then, immediately after that it talks about using the firmware included with the Processor SDK. Then the actual commands point to manually compiled firmware.

At this point, I would be happy to just use the prebuilt images in the SDK. You mentioned that that is what you did. However, if you used the pre-built images, you had to change the build commands that were in the documentation because those commands assumed that all binaries are built manually.

If I had your exact commands I could compare them to my build script at the top of this issue. I'm trying to make sure I am matching your process exactly.

Thanks,

-Bob

Hi Bob,

Here are the steps that I have followed to re-build U-boot:

$ export UBOOT_DIR=/home/aparna/ti/ti-processor-sdk-linux-edgeai-am62axx-evm-09_01_00_07/board-support/ti-u-boot-2023.04+gitAUTOINC+b0d717b732-gb0d717b732/
$ export TI_LINUX_FW_DIR=/home/aparna/ti/ti-processor-sdk-linux-edgeai-am62axx-evm-09_01_00_07/board-support/prebuilt-images/

R5
To build tiboot3.bin. Saved in $UBOOT_DIR/out/r5.
$ make ARCH=arm CROSS_COMPILE=/home/aparna/Downloads/sdk/external-toolchain-dir/arm-gnu-toolchain-11.3.rel1-x86_64-arm-none-linux-gnueabihf/bin/arm-none-linux-gnueabihf- am62ax_evm_r5_defconfig O=$UBOOT_DIR/out/r5
$ make ARCH=arm CROSS_COMPILE=/home/aparna/Downloads/sdk/external-toolchain-dir/arm-gnu-toolchain-11.3.rel1-x86_64-arm-none-linux-gnueabihf/bin/arm-none-linux-gnueabihf- O=$UBOOT_DIR/out/r5 BINMAN_INDIRS=$TI_LINUX_FW_DIR

A53
To build tispl.bin and u-boot.img. Saved in $UBOOT_DIR/out/a53. Requires bl31.bin, tee-pager_v2.bin.
$ make ARCH=arm CROSS_COMPILE=/home/aparna/Downloads/sdk/external-toolchain-dir/arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu- am62ax_evm_a53_defconfig O=$UBOOT_DIR/out/a53
$ make ARCH=arm CROSS_COMPILE=/home/aparna/Downloads/sdk/external-toolchain-dir/arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu- BL31=$TI_LINUX_FW_DIR/bl31.bin TEE=$TI_LINUX_FW_DIR/bl32.bin O=$UBOOT_DIR/out/a53 BINMAN_INDIRS=$TI_LINUX_FW_DIR

Aparna,

Many thanks for the details. When I start with your commands (with the Processor SDK and toolchain paths changed to match my system), the resulting binaries boot on the EVM and the GUI does NOT hang.

I need to go back and review my build process and to try figure out what is different. Give me a day or so to do so. I will also try some of the debug suggestions Reese suggested. I will post the results here.

Thanks,

-Bob

Reese,

When I run the GUI app and press the "Image Classification" button, the following text is output from vx_app_arm_remote_log.out:

root@mitysom-am62ax:/opt/edgeai-gst-apps# /etc/init.d/edgeai-launcher.sh start
Starting edgeai-gui-app...
-s option no longer works and will be removed soon.
Please reconfigure to use constraints or trusted servers.
constraint configured without libtls support
constraint configured without libtls support
constraint configured without libtls support
ntpd successful
root@mitysom-am62ax:/opt/edgeai-gst-apps# APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=25) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
440.424743 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
440.424919 s: VX_ZONE_INIT:Enabled
440.424939 s: VX_ZONE_ERROR:Enabled
440.424950 s: VX_ZONE_WARNING:Enabled
440.427134 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
440.427812 s: VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for !

Number of subgraphs:1 , 34 nodes delegated out of 34 nodes

(Note that this is running on our custom 62Ax baseboard)

Hi Bob,

Those logs are ordinary to me. There might have been more log messages if the application progressed, but my guess is that something is failing within the rest of the video pipeline, which is using gstreamer. Looking on my side, I don't see many logs coming out of the base executable /usr/bin/edgeai-gui-app, so they may be suppressed.

This basic OOB GUI application may be difficult to debug directly, so I'd like to consider the components underneath

Your build would include the gstreamer plugins for accessing ISP, DL accelerator, multiscaler engine, correct? You should be able to check like so:

root@am62axx-evm:/opt/edgeai-gst-apps# gst-inspect-1.0 ti
Plugin Details:
  Name                     ti
  Description              GStreamer plugin for TI devices
  Filename                 /usr/lib/gstreamer-1.0/libgstti.so
  Version                  0.7.0
  License                  Proprietary
  Source module            GstTIOVX
  Binary package           GstTIOVX source release
  Origin URL               http://ti.com

  ticolorconvert: TI Color Convert
  tidlinferer: TI DL Inferer
  tidlpostproc: TI DL PostProc
  tidlpreproc: TI DL PreProc
  timosaic: TI Mosaic
  tiperfoverlay: TI Perf Overlay
  tiscaler: TI Scaler

  7 features:
  +-- 7 elements

root@am62axx-evm:/opt/edgeai-gst-apps# gst-inspect-1.0 tiovx
Plugin Details:
  Name                     tiovx
  Description              GStreamer plugin for TIOVX
  Filename                 /usr/lib/gstreamer-1.0/libgsttiovx.so
  Version                  0.7.0
  License                  Proprietary
  Source module            GstTIOVX
  Binary package           GstTIOVX source release
  Origin URL               http://ti.com

  tiovxdelay: TIOVX Delay
  tiovxdemux: TIOVX Demux
  tiovxdlcolorconvert: TIOVX DL ColorConvert
  tiovxdlpreproc: TIOVX DL PreProc
  tiovxisp: TIOVX ISP
  tiovxldc: TIOVX LDC
  tiovxmemalloc: TIOVX Mem Alloc
  tiovxmosaic: TIOVX Mosaic
  tiovxmultiscaler: TIOVX MultiScaler
  tiovxmux: TIOVX Mux
  tiovxpyramid: TIOVX Pyramid

  11 features:
  +-- 11 elements

Assuming these all look right, could you run an equivalent application from command line? It's still useful to run that vx_app_arm_remote_log.out executable in the background

/opt/edgeai-gst-apps/apps_python/app_edgeai.py  /opt/edgeai-gst-apps/configs/image_classification.yaml -n

BR,
Reese

Reese,

I am running with the "full" tisdk-edgeai-image bitbake target.

root@mitysom-am62ax:/opt/edgeai-gst-apps# gst-inspect-1.0 ti
Plugin Details:
Name ti
Description GStreamer plugin for TI devices
Filename /usr/lib/gstreamer-1.0/libgstti.so
Version 0.7.0
License Proprietary
Source module GstTIOVX
Binary package GstTIOVX source release
Origin URL http://ti.com

ticolorconvert: TI Color Convert
tidlinferer: TI DL Inferer
tidlpostproc: TI DL PostProc
tidlpreproc: TI DL PreProc
timosaic: TI Mosaic
tiperfoverlay: TI Perf Overlay
tiscaler: TI Scaler

7 features:
+-- 7 elements

So that looks OK I think.

I also tried your "manual" command, but it produced the same log as when I pressed the button on the GUI:

root@mitysom-am62ax:/opt/edgeai-gst-apps# /opt/edgeai-gst-apps/apps_python/app_edgeai.py /opt/edgeai-gst-apps/configs/image_classification.yaml -n

Number of subgraphs:1 , 34 nodes delegated out of 34 nodes

APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
876.971146 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
876.971299 s: VX_ZONE_INIT:Enabled
876.971320 s: VX_ZONE_ERROR:Enabled
876.971331 s: VX_ZONE_WARNING:Enabled
876.974262 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
876.974399 s: VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!

Given that I am able to compile a working U-Boot for the EVM, there must be something in my U-Boot source/defconfig that is causing this problem as I am pointing to the same pre-built firmware files and using the same build commands as the EVM build.

Thanks!

Aparna Patra said:

Here are the steps that I have followed to re-build U-boot:

Following up on where Bob left off.

diff --git a/board/ti/am62ax/evm.c b/board/ti/am62ax/evm.c
index 71fbc9e9c2b0..906503875644 100644
--- a/board/ti/am62ax/evm.c
+++ b/board/ti/am62ax/evm.c
@@ -15,17 +15,12 @@
 #include <fdt_support.h>
 #include <spl.h>
 
-#include "../common/rtc.c"
-
 #define CTRLMMR_USB0_PHY_CTRL  0x43004008
 #define CTRLMMR_USB1_PHY_CTRL  0x43004018
 #define CORE_VOLTAGE           0x80000000
 
 int board_init(void)
 {
-       if (IS_ENABLED(CONFIG_BOARD_HAS_32K_RTC_CRYSTAL))
-               board_rtc_init();
-
        return 0;
 }
 
diff --git a/configs/am62ax_evm_a53_defconfig b/configs/am62ax_evm_a53_defconfig
index 45bbffada46c..9cb3a17449b3 100644
--- a/configs/am62ax_evm_a53_defconfig
+++ b/configs/am62ax_evm_a53_defconfig
@@ -3,7 +3,6 @@ CONFIG_ARCH_K3=y
 CONFIG_TI_SECURE_DEVICE=y
 CONFIG_SPL_GPIO=y
 CONFIG_SYS_MALLOC_F_LEN=0x8000
-CONFIG_BOARD_HAS_32K_RTC_CRYSTAL=y
 CONFIG_SPL_LIBCOMMON_SUPPORT=y
 CONFIG_SPL_LIBGENERIC_SUPPORT=y
 CONFIG_NR_DRAM_BANKS=2

Alright looks like we are looking at a multi-stage problem.

Using the fixed u-boot and our kernel with the evm sdk 09.01 filesystem, the demo was working.  However when I committed the changes and built our yocto tisdk-edgeai image, the demos no longer worked.

I reflashed the evm sdcard image and replaced the u-boot and kernel from our yocto build and verified the demo started working again so there also seems to be something wrong with our yocto filesystem as well... Drat so close

Now that the u-boot issue is resolved I figured I'd rerun all the commands you asked for above. And they do seem to be failing with various error messages though I'm not sure how to decode them.

The gst-inspect-1.0 commands all looked good.

root@mitysom-am62ax:/opt/edgeai-gst-apps# /opt/edgeai-gst-apps/apps_python/app_edgeai.py  /opt/edgeai-gst-apps/configs/image_classification.yaml -n
[   31.928368] UNUSED: disabling

 Number of subgraphs:1 , 34 nodes delegated out of 34 nodes 
 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
    42.743728 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
    42.746167 s:  VX_ZONE_INIT:Enabled
    42.746217 s:  VX_ZONE_ERROR:Enabled
    42.746227 s:  VX_ZONE_WARNING:Enabled
    42.747747 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
    42.750940 s:  VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!!
    42.769251 s:  VX_ZONE_ERROR:[tivxTIDLValidate:118] 'config' should be a user_data_object of type:
 tivxTIDLParms 
    42.769309 s:  VX_ZONE_ERROR:[ownGraphNodeKernelValidate:531] node kernel validate failed for kernel com.ti.tidl:1:1 at index 0
    42.769321 s:  VX_ZONE_ERROR:[vxVerifyGraph:1941] Node kernel Validate failed
    42.769331 s:  VX_ZONE_ERROR:[vxVerifyGraph:2109] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
    43.080643 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
    43.083208 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
[ERROR] Internal data stream error.
==========[INPUT PIPELINE(S)]==========

[PIPE-0]

multifilesrc location=/opt/edgeai-test-data/images/%04d.jpg index=1 stop-index=-1 loop=True ! jpegdec ! videoscale qos=True ! capsfilter caps="video/x-raw, width=(int)1280, height=(int)720;" ! tiovxdlcolorconvert ! capsfilter caps="video/x-raw, format=(string)NV12;" ! tiovxmultiscaler name=split_01
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)1280, height=(int)720;" ! tiovxdlcolorconvert out-pool-size=4 ! capsfilter caps="video/x-raw, format=(string)RGB;" ! appsink max-buffers=2 drop=True name=sen_0
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)454, height=(int)256;" ! tiovxdlcolorconvert out-pool-size=4 ! capsfilter caps="video/x-raw, format=(string)RGB;" ! videobox qos=True left=115 right=115 top=16 bottom=16 ! tiovxdlpreproc out-pool-size=4 channel-order=1 data-type=3 ! capsfilter caps="application/x-tensor-tiovx;" ! appsink max-buffers=2 drop=True name=pre_0


==========[OUTPUT PIPELINE]==========

appsrc do-timestamp=True format=3 block=True name=post_0 ! tiovxdlcolorconvert ! capsfilter caps="video/x-raw, format=(string)NV12, width=(int)1280, height=(int)720;" ! queue ! mosaic_0.sink_0

tiovxmosaic target=1 background=/tmp/background_0 name=mosaic_0 src::pool-size=4
sink_0::startx="<320>" sink_0::starty="<150>" sink_0::widths="<1280>" sink_0::heights="<720>"
! capsfilter caps="video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080;" ! queue ! tiperfoverlay title=Image Classification ! kmssink sync=False max-lateness=5000000 qos=True processing-deadline=15000000 driver-name=tidss connector-id=40 plane-id=31 force-modesetting=True

[ERROR] Error pulling tensor from GST Pipeline

./optiflow/optiflow.py  ./configs/object_detection.yaml 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=20) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   264.425626 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   264.425757 s:  VX_ZONE_INIT:Enabled
   264.425777 s:  VX_ZONE_ERROR:Enabled
   264.425785 s:  VX_ZONE_WARNING:Enabled
   264.428593 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
   264.429500 s:  VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!!
   264.433666 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
   264.434826 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
   264.435786 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
   264.436688 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
   264.437674 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
   264.438607 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
   264.439518 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
   264.526806 s:  VX_ZONE_WARNING:[vxReleaseContext:1088] Found a reference 0xffff8edbf250 of type 0000080f at external count 1, internal count 0, releasing it
   264.526866 s:  VX_ZONE_WARNING:[vxReleaseContext:1090] Releasing reference (name=image_70) now as a part of garbage collection
   264.526972 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:110] De-Initialization Done for HOST !!!
   264.531682 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 7 alloc's of 10321920 bytes 
DDR_SHARED_MEM: Free's : 7 free's  of 10321920 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=3) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   264.564916 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   264.564973 s:  VX_ZONE_INIT:Enabled
   264.564982 s:  VX_ZONE_ERROR:Enabled
   264.564991 s:  VX_ZONE_WARNING:Enabled
   264.566227 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
   264.566314 s:  VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!!
libtidl_onnxrt_EP loaded 0x3df7a7c0 
Final number of subgraphs created are : 1, - Offloaded Nodes - 283, Total Nodes - 283 
   265.499964 s:  VX_ZONE_ERROR:[tivxTIDLValidate:118] 'config' should be a user_data_object of type:
 tivxTIDLParms 
   265.500023 s:  VX_ZONE_ERROR:[ownGraphNodeKernelValidate:531] node kernel validate failed for kernel com.ti.tidl:1:2 at index 0
   265.500045 s:  VX_ZONE_ERROR:[vxVerifyGraph:1941] Node kernel Validate failed
   265.500056 s:  VX_ZONE_ERROR:[vxVerifyGraph:2109] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
   265.502880 s:  VX_ZONE_ERROR:[tivxTIDLValidate:118] 'config' should be a user_data_object of type:
 tivxTIDLParms 
   265.502931 s:  VX_ZONE_ERROR:[ownGraphNodeKernelValidate:531] node kernel validate failed for kernel com.ti.tidl:1:2 at index 0
   265.502943 s:  VX_ZONE_ERROR:[vxVerifyGraph:1941] Node kernel Validate failed
   265.502953 s:  VX_ZONE_ERROR:[vxVerifyGraph:2109] Graph verify failed
   265.503224 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:799] graph is not in a state required to be scheduled
   265.503236 s:  VX_ZONE_ERROR:[vxProcessGraph:734] schedule graph failed
   265.503246 s:  VX_ZONE_ERROR:[vxProcessGraph:739] wait graph failed
ERROR: Running TIDL graph ... Failed !!!

multifilesrc location=/opt/edgeai-test-data/videos/video0_1280_768.h264 loop=true stop-index=-1 ! h264parse ! v4l2h264dec capture-io-mode=5 ! tiovxmemalloc pool-size=12 ! video/x-raw, format=NV12 ! \
tiovxmultiscaler name=split_01 src_0::roi-startx=0 src_0::roi-starty=0 src_0::roi-width=1280 src_0::roi-height=768 target=0 \
\
split_01. ! queue ! video/x-raw, width=416, height=416 ! tiovxdlpreproc model=/opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416  out-pool-size=4 ! application/x-tensor-tiovx ! tidlinferer target=1  model=/opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416 ! post_0.tensor \
split_01. ! queue ! video/x-raw, width=1280, height=720 ! post_0.sink \
tidlpostproc name=post_0 model=/opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416 alpha=0.400000 viz-threshold=0.600000 top-N=5 display-model=true ! queue ! mosaic_0. \
\
\
tiovxmosaic name=mosaic_0 target=1 src::pool-size=4 \
sink_0::startx="<320>" sink_0::starty="<150>" sink_0::widths="<1280>" sink_0::heights="<720>" \
! video/x-raw,format=NV12, width=1920, height=1080 ! queue ! tiperfoverlay title="Object Detection" overlay-type=graph ! kmssink driver-name=tidss sync=false force-modesetting=true


^C

# Click demo on gui
# /opt/vx_app_arm_remote_log.out 
...

APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=22) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   894.032875 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   894.034102 s:  VX_ZONE_INIT:Enabled
   894.034156 s:  VX_ZONE_ERROR:Enabled
   894.034171 s:  VX_ZONE_WARNING:Enabled
   894.036329 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
   894.040540 s:  VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!!

 Number of subgraphs:1 , 34 nodes delegated out of 34 nodes 
 
   894.529677 s:  VX_ZONE_ERROR:[tivxTIDLValidate:118] 'config' should be a user_data_object of type:
 tivxTIDLParms 
   894.529730 s:  VX_ZONE_ERROR:[ownGraphNodeKernelValidate:531] node kernel validate failed for kernel com.ti.tidl:1:1 at index 0
   894.529757 s:  VX_ZONE_ERROR:[vxVerifyGraph:1941] Node kernel Validate failed
   894.529766 s:  VX_ZONE_ERROR:[vxVerifyGraph:2109] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
   894.534045 s:  VX_ZONE_ERROR:[tivxTIDLValidate:118] 'config' should be a user_data_object of type:
 tivxTIDLParms 
   894.534093 s:  VX_ZONE_ERROR:[ownGraphNodeKernelValidate:531] node kernel validate failed for kernel com.ti.tidl:1:1 at index 0
   894.534104 s:  VX_ZONE_ERROR:[vxVerifyGraph:1941] Node kernel Validate failed
   894.534114 s:  VX_ZONE_ERROR:[vxVerifyGraph:2109] Graph verify failed
   894.534380 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:799] graph is not in a state required to be scheduled
   894.534391 s:  VX_ZONE_ERROR:[vxProcessGraph:734] schedule graph failed
   894.534430 s:  VX_ZONE_ERROR:[vxProcessGraph:739] wait graph failed
ERROR: Running TIDL graph ... Failed !!!

** (edgeai-gui-app:1328): CRITICAL **: 21:10:12.253: gst_v4l2_buffer_pool_orphan: assertion 'bpool' failed
[MCU1_0]    894.778663 s:  VX_ZONE_ERROR:[ownTargetCmdDescHandler:1115] object descriptor type is invalid
[MCU1_0]    894.778713 s:  VX_ZONE_ERROR:[ownTargetCmdDescHandler:1115] object descriptor type is invalid

Hi Jonathan,

Apologies for the radio silence -- I've been away the past week on work travel. You've made good progress in the mean time, but let me try to help push it forward faster

To summarize where we are (and please correct where/if I'm wrong)

• Previously, u-boot building but not functioning correctly, seemingly due to a issue that was later resolved with the RTC and debouncing
  • by removing the RTC from config and/or applying the recent page (verified to be in 9.2 release), u-boot now functioning
• With fixed u-boot and filesystem from stock 9.1 SDK, the demo works.
• When using the filesystem generated by yocto build, the demo returns to not working.
  • Confirmed that kernel and u-boot image are not to blame
  • Unclear if the later logs with "TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!" are from yocto or SDK filesystem

It would help to quickly clarify if these error logs are coming from the SDK filesystem or the yocto-generated FS.

From the logs with the demo, it's always the same error mode. This is shown as

42.769251 s:  VX_ZONE_ERROR:[tivxTIDLValidate:118] 'config' should be a user_data_object of type: tivxTIDLParms
    
AND
    
264.433666 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].

Which is happening during the 'verify' step of the TIOVX graph initialization (details are hidden from you, since it's several layer of abstraction deep). Seems like the APIs were called differently than expected within this build.

My gut reaction is that there is a versioning mismatch between the C7x firmware and the TIDL-related components running on Arm/Linux. That firmware and TIDL libraries should be precompiled and pulled in during the yocto build. Perhaps we could copy those between the yocto and SDK builds to see if that is the root cause here. Files to consider:

• /lib/firmware/vision_apps_eaik/vx_app_rtos_linux_c7x_1.out
• /usr/lib/libvx_tidl_rt.so
• /usr/lib/libonnxruntime.so
• /usr/lib/libtidl_tfl_delegate.so

BR,
Reese

All of the logs I just posted were from the yocto build.

Looks like the only file that doesn't match is libvx_tidl_rt.so

# sdk filesystem
md5sum ./lib/firmware/vision_apps_eaik/vx_app_rtos_linux_c7x_1.out ./usr/lib/libvx_tidl_rt.so ./usr/lib/libonnxruntime.so ./usr/lib/libtidl_tfl_delegate.so
4aab2bfe407ef7cb5af84fca7f20cbcc  ./lib/firmware/vision_apps_eaik/vx_app_rtos_linux_c7x_1.out
38305a658d094db579f3b53c787874bd  ./usr/lib/libvx_tidl_rt.so
fbb6baa56cd449b4be812a2fa9f78775  ./usr/lib/libonnxruntime.so
1e08119c298a907a7b869e29ed4e880c  ./usr/lib/libtidl_tfl_delegate.so

# yocto filesystem
root@mitysom-am62ax:/opt/edgeai-gst-apps# md5sum /lib/firmware/vision_apps_eaik/vx_app_rtos_linux_c7x_1.out /usr/lib/libvx_tidl_rt.so /usr/lib/libonnxruntime.so /usr/lib/libtidl_tfl_delegate.so
4aab2bfe407ef7cb5af84fca7f20cbcc  /lib/firmware/vision_apps_eaik/vx_app_rtos_linux_c7x_1.out
d93a944d7bb56c79a281ef0a74d6db67  /usr/lib/libvx_tidl_rt.so
fbb6baa56cd449b4be812a2fa9f78775  /usr/lib/libonnxruntime.so
1e08119c298a907a7b869e29ed4e880c  /usr/lib/libtidl_tfl_delegate.so

After replacing it, I now seem to be getting a core dump

root@mitysom-am62ax:/opt/edgeai-gst-apps# md5sum /usr/lib/libvx*
38305a658d094db579f3b53c787874bd  /usr/lib/libvx_tidl_rt.so
38305a658d094db579f3b53c787874bd  /usr/lib/libvx_tidl_rt.so.1.0
root@mitysom-am62ax:/opt/edgeai-gst-apps# ./optiflow/optiflow.py  ./configs/object_detection.yaml
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=20) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  4742.133567 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  4742.133723 s:  VX_ZONE_INIT:Enabled
  4742.133740 s:  VX_ZONE_ERROR:Enabled
  4742.133749 s:  VX_ZONE_WARNING:Enabled
  4742.136906 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  4742.137057 s:  VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!!
  4742.142900 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
  4742.144002 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
  4742.144926 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
  4742.145849 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
  4742.146973 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
  4742.148064 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
  4742.149091 s:  VX_ZONE_ERROR:[tivxReferenceImportHandle:1137] num_entries [1] less than num handles expected [2].
  4742.255595 s:  VX_ZONE_WARNING:[vxReleaseContext:1088] Found a reference 0xffffb232f250 of type 0000080f at external count 1, internal count 0, releasing it
  4742.255656 s:  VX_ZONE_WARNING:[vxReleaseContext:1090] Releasing reference (name=image_70) now as a part of garbage collection
  4742.255784 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:110] De-Initialization Done for HOST !!!
  4742.260406 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 7 alloc's of 10321920 bytes
DDR_SHARED_MEM: Free's : 7 free's  of 10321920 bytes
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=3) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  4742.295671 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  4742.295737 s:  VX_ZONE_INIT:Enabled
  4742.295746 s:  VX_ZONE_ERROR:Enabled
  4742.295755 s:  VX_ZONE_WARNING:Enabled
  4742.296629 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  4742.296705 s:  VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!!
libtidl_onnxrt_EP loaded 0x419b4480
Error -   libtivision_apps.so.9.1.0: cannot open shared object file: No such file or directory
Final number of subgraphs created are : 1, - Offloaded Nodes - 283, Total Nodes - 283
[   60.217249] audit: type=1701 audit(67.760:20): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=1354 comm="optiflow.py" exe="/usr/bin/python3.10" sig=11 res=1
[   60.244154] audit: type=1334 audit(67.784:21): prog-id=15 op=LOAD
[   60.250395] audit: type=1334 audit(67.796:22): prog-id=16 op=LOAD
[   74.358207] systemd-journald[177]: Oldest entry in /run/log/journal/1d8ca255c76a44bda835c83c2a842a12/system.journal is older than the configured file retention duration (1month), suggesting rotation.
[   74.376267] systemd-journald[177]: /run/log/journal/1d8ca255c76a44bda835c83c2a842a12/system.journal: Journal header limits reached or header out-of-date, rotating.
Segmentation fault (core dumped)

root@mitysom-am62ax:/opt/edgeai-gst-apps# /opt/edgeai-gst-apps/apps_python/app_edgeai.py  /opt/edgeai-gst-apps/configs/image_classification.yaml -n

 Number of subgraphs:1 , 34 nodes delegated out of 34 nodes
[  126.948283] audit: type=1701 audit(1713198115.614:25): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=1381 comm="app_edgeai.py" exe="/usr/bin/python3.10" sig=11 res=1

[  126.979905] audit: type=1334 audit(1713198115.646:26): prog-id=17 op=LOAD
[  126.986802] audit: type=1334 audit(1713198115.650:27): prog-id=18 op=LOAD
Segmentation fault (core dumped)

Hi Jonathan,

Thanks for trying this. Looks like that error (num_entries is 1 instead of expected 2) is still showing up early in the log with the SDK version of that.

There's another library it mentioned as not existing before it seg-faulted. Is there any instance of libtivision_apps.so.9.1.0 in your system?

Looking at the symbols in that libtivision_apps.so, this is where that function "tivxReferenceImportHandle", which is throwing VX_ZONE_ERROR's, exists.

It's also possible that this tivxReferenceImportHandle function is being called incorrectly (per the current versions of your libraries) so the error may be generated from another library. Let's clear up this libtivision_apps.so first, though.

We may also consider running a simpler pipeline to only touch one component on the TIOVX bus. I am curious if you see a similar error for a pipeline like the following:

gst-launch-1.0 videotestsrc ! video/x-raw, height=1080, width=1920, framerate=30/1 ! tiovxmultiscaler ! video/x-raw, height=1080, width=1920 ! kmssink driver-name=tidss sync=false

Note the sdk filesystem had a newer version of libtivision_apps? 9.1.0 instead of 9.0.0

md5sum ./usr/lib/libtivision_apps.so*
3e1a330c21e2f949881e2e321fe23994  ./usr/lib/libtivision_apps.so
3e1a330c21e2f949881e2e321fe23994  ./usr/lib/libtivision_apps.so.9.1.0


root@mitysom-am62ax:/opt/edgeai-gst-apps# md5sum /usr/lib/libtivision_apps.so*
79c54b6b199577e2b27126223b2bcff7  /usr/lib/libtivision_apps.so
79c54b6b199577e2b27126223b2bcff7  /usr/lib/libtivision_apps.so.9.0.0

After updating that file, I now get a no element tiovxmemalloc

Sorry the above reply got delayed. The e2e has been very intermittent today.  Multiple posts now have failed to send right away.  And I've come back to find they never sent...

I kicked off a clean yocto build. I have a sneaking suspicion that the ti-vision-apps recipes which uses repo tool, isn't working properly with autorev...

Our build servers edgeai...manifest file showed libtivision-apps9.0.0, was installed instead of the expected 9.1.0.  I kicked off a clean build but it still installed the 9.0.0 version, possibly coming from sstate cache... will try again.

Hi Jonathan,

Please try deleting repo related files downloads

DL_DIR/repo*

This will make sure you are pulling latest source

Regards
Rahul T R

You guys should set the PV variable in that recipe. Whatever is setting the 9.0.0 version in the package,  I think has tripped up both the downloads and sstate cache.  I've wiped sstate and downloads, waiting on build to rebuild.

Back from vacation.

Cleaned the sstate cache, DL_DIR/repo*, and the build directory.  Ofcourse now the nodejs recipe failed the build.  Cleaning the recipe and restarting build. Hopefully it completes successfully this time.

Hi Jonathan,

Thanks for the update -- I'm glad to hear this is working now.

Jonathan Cormier said:

The 09.02 release does seem to be handling this by updating the PR in the recipe.

Okay, then my understanding is that these issues would be resolved moving forward within the 9.2 SDK. Please correct me if I misunderstand your comment.

We'll get this feedback to the SDK release team for AM62A

BR,
Reese

Yes, I don't believe we will run into the same issue when moving to the 09.02 release.

Aparna and Reese, thank you for the investigation and feedback. Jon, thanks for getting this over the finish line. I set Jon's summary post as the resolution as our independent yocto build is now working as expected.

Thanks,

-Bob

Maybe I celebrated too soon.  Our AI demo performance is very bad. Created new e2e post. e2e.ti.com/.../am62a7-custom-board-sdk-09-01-poor-object-detection-performance