This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

SK-AM68: Inference problems with Yocto SDK 10.1 build

Part Number: SK-AM68

Tool/software:

If I build our system using Yocto from SDK 10.1 (ti-processor-sdk-linux-edgeai-j721s2-evm-10_01_00_04-Linux-x86-Install.bin), I end up with a system that is unable to perform inferencing.  I've tried using the config file included with the SDK release itself (configs/processor-sdk-analytics/processor-sdk-analytics-10_01-config.txt) and the one available via the repo https://git.ti.com/git/arago-project/oe-layersetup.git (configs/processor-sdk-analytics/processor-sdk-analytics-10.01.00-config.txt). I would expect these to be identical, but they are not (the referenced commit for meta-edgeai differs between them).

Building with either configuration results in errors when tensor data is pulled from the GST pipeline:

root@am68a-sk:/opt/edgeai-gst-apps# cd apps_python/
root@am68a-sk:/opt/edgeai-gst-apps/apps_python# ./app_edgeai.py -n -v ../configs/image_classification.yaml

Number of subgraphs:1 , 34 nodes delegated out of 34 nodes

APP: Init ... !!!
22603.275840 s: MEM: Init ... !!!
22603.275908 s: MEM: Initialized DMA HEAP (fd=6) !!!
22603.276082 s: MEM: Init ... Done !!!
22603.276110 s: IPC: Init ... !!!
22603.312964 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
22603.319090 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
22603.321311 s: VX_ZONE_INIT:Enabled
22603.321544 s: VX_ZONE_ERROR:Enabled
22603.323104 s: VX_ZONE_WARNING:Enabled
22603.328488 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-0
22603.328804 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-1
22603.328938 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-2
22603.329067 s: VX_ZONE_INIT:[tivxPlatformCreateTargetId:124] Added target MPU-3
22603.329085 s: VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
22603.331131 s: VX_ZONE_INIT:[tivxHostInitLocal:106] Initialization Done for HOST !!!
==========[INPUT PIPELINE(S)]==========

[PIPE-0]

multifilesrc location=/opt/edgeai-test-data/images/%04d.jpg index=1 stop-index=-1 loop=True ! jpegdec ! videoscale qos=True ! capsfilter caps="video/x-raw, width=(int)1280, height=(int)720;" ! tiovxdlcolorconvert ! capsfilter caps="video/x-raw, format=(string)NV12;" ! tiovxmultiscaler name=split_01
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)1280, height=(int)720;" ! tiovxdlcolorconvert out-pool-size=4 ! capsfilter caps="video/x-raw, format=(string)RGB;" ! appsink max-buffers=2 drop=True name=sen_0
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)454, height=(int)256;" ! tiovxdlcolorconvert out-pool-size=4 ! capsfilter caps="video/x-raw, format=(string)RGB;" ! videobox qos=True left=115 right=115 top=16 bottom=16 ! tiovxdlpreproc out-pool-size=4 channel-order=1 data-type=3 ! capsfilter caps="application/x-tensor-tiovx;" ! appsink max-buffers=2 drop=True name=pre_0


==========[OUTPUT PIPELINE]==========

appsrc do-timestamp=True format=3 block=True name=post_0 ! tiovxdlcolorconvert ! capsfilter caps="video/x-raw, format=(string)NV12, width=(int)1280, height=(int)720;" ! queue ! mosaic_0.sink_0

tiovxmosaic target=1 background=/tmp/background_0 name=mosaic_0 src::pool-size=4
sink_0::startx="<320>" sink_0::starty="<150>" sink_0::widths="<1280>" sink_0::heights="<720>"
! capsfilter caps="video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080;" ! queue ! tiperfoverlay title=Image Classification ! kmssink sync=False max-lateness=5000000 qos=True processing-deadline=15000000 driver-name=tidss connector-id=40 plane-id=31 force-modesetting=True fd=44

[ERROR] Error pulling tensor from GST Pipeline

If I download the pre-built 10.01.00.04 image and use that, then inferencing works as expected.

Further, if I copy the contents of the pre-built image's `/usr/lib` dir to the SD card with the Yocto build, then inferencing works!

In looking at the contents of the pre-built image, I see discrepancies between several TI-specific libraries.  Here's one example:  The Yocto build has `/usr/lib/libtivision_apps.so.10.0.0`.  The pre-built image has `/usr/lib/libtivision_apps.so.10.1.0`.  I'd expect the versions to be identical, but they are not.  There are other edgeai-related libs that are also different based on their md5sums.  I'd expect everything to be identical but that is not the case.

Any idea what is up?  Can someone try to build from scratch using Yocto for the am68-sk and confirm they end up with something that can perform inferencing?

  • I've got a 'common' download location setup for my bitbake builds to speed up the build process.  I've discovered that if I don't use this, building "edgeai-gst-apps" results in libtivision_apps.so.10.1.0 instead of libtivision_apps.so.10.0.0.  I checked and it looks like the related downloads are different even though they are named identically:

    # my "common" download location:

    ~/downloads/repo_git.ti.com.git.processor-sdk.psdk_repo_manifests.git_vision_apps_yocto.xml_refs/tags$ cd
    tisdk@isgf-adv-eng-v1:~$ md5sum ~/downloads/repo_git.ti.com.git.processor-sdk.psdk_repo_manifests.git_vision_apps_yocto.xml_refs/tags/REL.PSDK.ANALYTICS.10.01.00.04.tar.gz
    8522647addd165134712600f74f3f603 /home/tisdk/downloads/repo_git.ti.com.git.processor-sdk.psdk_repo_manifests.git_vision_apps_yocto.xml_refs/tags/REL.PSDK.ANALYTICS.10.01.00.04.tar.gz

    # the default download location as part of the build/ path:

    tisdk@isgf-adv-eng-v1:~$ md5sum ~/yocto-build/build/downloads/repo_git.ti.com.git.processor-sdk.psdk_repo_manifests.git_vision_apps_yocto.xml_refs/tags/REL.PSDK.ANALYTICS.10.01.00.04.tar.gz
    b8a53367b4b0083fefdfe0af84c828bb /home/tisdk/yocto-build/build/downloads/repo_git.ti.com.git.processor-sdk.psdk_repo_manifests.git_vision_apps_yocto.xml_refs/tags/REL.PSDK.ANALYTICS.10.01.00.04.tar.gz

    Same file naming.  Different md5sum.  Briefly looking at the content of each does indeed show updates to the source contained in the .tar.gz.

    This not a great side effect.

  • I've wiped my download dir and am in the process of a full re-build.  I'll update this thread with the results.  Stay tuned.

  • Hi Tim,

    This has been assigned to Jared (the Yocto build guy), and he is monitoring it.   It also looks like you are trying something first.  If you need my help directly, call or send me an email.

    Regards,

    Chris

  • Hi Tim,

    My guess is that there is nothing specifying which version of libtivision_apps is required, so when Yocto sees there is a previous version already built, it skips building a newer version.

    Clearing the directory and building from scratch (as you've done) should fix it.

    Best,
    Jared

  • Deleting my download directory and building from scratch worked.