SK-AM62A-LP: SDK-analytics-9.2 edge AI issue

Hello and thank you for your query, Mickael.

You are seeing an issue where the compiled AI/CNN model artifacts that you are using do not match the underlying firmware expectations. What is the contents of the image_classification.yaml config file you are using as an argument here? Where did that model come from (perhaps it was added to the Yocto image by default)?

TIDL_RT_OVX: ERROR: Config file size (37912 bytes) does not match size of sTIDL_IOBufDesc_t (93976 bytes)

We have been having storms and power issues locally so I'm unable to do a direct check on what the size of this should be, but there will be an IO binary in the set of compiled artifacts for the binary. The size of this and the structure therein is what it's complaining about.

The error you're seeing indicates (not obviously) that this IO file is the wrong size. This can be due to model artifacts that are complied for the wrong SDK version and/or the wrong device. Given the size of the difference, I posit they are compiled for the wrong device. Fixing this won't require a rebuild at yocto level, but it's possible Yocto was pulling the wrong model_zoo (i.e. for a different device).

You did set the MACHINE for the yocto build to be am62a (or some variation thereof), correct?

BR,
Reese

Hi Mickael,

Just a quick update here since I did a check on the stock 9.2 SDK. The models should have an ..._io_1.bin file (name is model dependent) that is 93976 bytes, exactly as that line says in the error logs

For example:

ls -al /opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416/artifacts/
-rw-r--r-- 1 root root   93976 Mar  9  2018 detslabels_tidl_io_1.bin

So what's probably happened is that your Yocto build pulled an older version of the model zoo than what the 9.2 SDK would expect. Can you verify if all the artifacts under /opt/model_zoo have this same size for the _io_1.bin files, and if these are ~92 kB or ~32kB?

-Reese

Yes there was an issue with the sdk package. I remove all the old build folder in yocto (based on sdk 9.1) and rebuild a new image from sdk9.2
Now i can launch inference ...
But i have this error now.
Below, the procedure i followed

root@am62axx-evm:/opt/edgeai-gst-apps# modprobe -rf ti_k3_dsp_remoteproc
root@am62axx-evm:/opt/edgeai-gst-apps# rtcwake -s 10 -m mem -d rtc1
root@am62axx-evm:/opt/edgeai-gst-apps# dmesg
[  327.480073] k3-dsp-rproc 7e000000.dsp: k3_dsp_rproc_stop: timedout waiting for rproc completion event
[  327.489545] remoteproc remoteproc0: can't stop rproc: -16
[  387.547667] PM: suspend entry (deep)
[  387.552514] Filesystems sync: 0.001 seconds
[  387.562191] remoteproc remoteproc1: stopped remote processor 79000000.r5f
[  387.569489] Freezing user space processes
[  387.575227] Freezing user space processes completed (elapsed 0.001 seconds)
[  387.582219] OOM killer disabled.
[  387.585476] Freezing remaining freezable tasks
[  387.591172] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[  387.598592] printk: Suspending console(s) (use no_console_suspend to debug)
[  387.607146] wlcore: extra spare keys before: 0
[  387.607161] wlcore: mac80211 set key
[  387.607163] wlcore: CMD: 0x1 sta: 0000000000000000
[  387.607169] wlcore: Key: algo:0xfac04, id:1, len:16 flags 0x20
[  387.607179] wlcore: ap key action: 2 id: 1 lid: 1 type: 3 hlid: 18
[  387.607462] wlcore: Reading control info
[  387.607526] wlcore: Processing message type 2, len 4
[  387.607539] wlcore: CMD# 1, len=76
[  387.607545] wlcore: extra spare keys after: 0
[  387.607580] wlcore: mac80211 bss info role 1 changed 0x200
[  387.607588] wlcore: cmd role stop ap 1
[  387.610713] wlcore: Reading control info
[  387.610776] wlcore: Processing message type 2, len 4
[  387.610788] wlcore: CMD# 20, len=16
[  387.610798] wlcore: stopped AP
[  387.610806] wlcore: mac80211 unassign chanctx (role 1) 10 (type 1)
[  387.610812] wlcore: mac80211 bss info role 1 changed 0x4000
[  387.610819] wlcore: mac80211 remove chanctx 10 (type 1)
[  387.610843] wlcore: mac80211 config psm off power 0 idle changed 0x100
[  387.611100] wlcore: mac80211 remove interface 3
[  387.611108] wlcore: mac80211 rm: name1=wlan0, name2=wlan0, name3=wlan0
[  387.611113] wlcore: down
[  387.611116] wlcore: cmd role disable
[  387.622183] wlcore: Reading control info
[  387.622247] wlcore: Processing message type 2, len 4
[  387.622261] wlcore: CMD# 18, len=16
[  387.622269] wlcore: unmasking event_mask 0x3ffef01
[  387.622273] wlcore: acx event mbox mask
[  387.622276] wlcore: cmd configure (31), TSFL 17002402
[  387.622450] wlcore: Reading control info
[  387.622512] wlcore: Processing message type 2, len 4
[  387.622523] wlcore: CMD# 27, len=24
[  387.622534] wlcore: mac80211 stop
[  387.634472] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
[  387.645040] Disabling non-boot CPUs ...
[  387.647025] psci: CPU1 killed (polled 0 ms)
[  387.649628] psci: CPU2 killed (polled 0 ms)
[  387.652411] psci: CPU3 killed (polled 0 ms)
[  387.653344] Enabling non-boot CPUs ...
[  387.653675] Detected VIPT I-cache on CPU1
[  387.653741] GICv3: CPU1: found redistributor 1 region 0:0x00000000018a0000
[  387.653795] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[  387.654573] CPU1 is up
[  387.654807] Detected VIPT I-cache on CPU2
[  387.654846] GICv3: CPU2: found redistributor 2 region 0:0x00000000018c0000
[  387.654883] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
[  387.655546] CPU2 is up
[  387.655791] Detected VIPT I-cache on CPU3
[  387.655836] GICv3: CPU3: found redistributor 3 region 0:0x00000000018e0000
[  387.655880] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[  387.656561] CPU3 is up
[  387.656901] ti-sci 44043000.system-controller: ti_sci_resume: wakeup source: 0x50
[  387.766445] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
[  387.780382] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=POLL)
[  387.780406] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
[  387.802630] wlcore: mac80211 start
[  387.802644] wlcore: acx frag threshold: 2352
[  387.802650] wlcore: cmd configure (53), TSFL 1700250c
[  387.809003] wlcore: Reading control info
[  387.809069] wlcore: Processing message type 2, len 4
[  387.809086] wlcore: CMD# 27, len=20
[  387.809096] wlcore: mac80211 add interface type 3 mac a4:34:f1:b1:62:46
[  387.809108] wlcore: using pre-allocated hw queue base 0
[  387.809112] wlcore: cmd roles enabled: bitmap before: 0, ret=0
[  387.809116] wlcore: cmd role enable, role type 2, addr = a4:34:f1:b1:62:46
[  387.809974] wlcore: Reading control info
[  387.810036] wlcore: Processing message type 2, len 5
[  387.810047] wlcore: CMD# 17, len=20
[  387.810051] wlcore: complete role_id = 1
[  387.810056] wlcore: acx sleep auth 2
[  387.810059] wlcore: cmd configure (9), TSFL 17a36fb5
[  387.810390] wlcore: Reading control info
[  387.810452] wlcore: Processing message type 2, len 4
[  387.810462] wlcore: CMD# 27, len=20
[  387.810467] wlcore: unmasking event_mask 0x3ffff01
[  387.810471] wlcore: acx event mbox mask
[  387.810474] wlcore: cmd configure (31), TSFL 17a37155
[  387.810663] wlcore: Reading control info
[  387.810725] wlcore: Processing message type 2, len 4
[  387.810736] wlcore: CMD# 27, len=24
[  387.810741] wlcore: acx config ap sleep
[  387.810744] wlcore: cmd configure (89), TSFL 17a37266
[  387.810911] wlcore: Reading control info
[  387.810973] wlcore: Processing message type 2, len 4
[  387.810984] wlcore: CMD# 27, len=20
[  387.810989] wlcore: acx ap max tx retry
[  387.810992] wlcore: cmd configure (70), TSFL 17a3735e
[  387.811386] wlcore: Reading control info
[  387.811449] wlcore: Processing message type 2, len 4
[  387.811462] wlcore: CMD# 27, len=20
[  387.811468] wlcore: acx TX_POWER_CFG 21
[  387.811471] wlcore: cmd configure (14), TSFL 17a37539
[  387.811790] wlcore: Reading control info
[  387.811852] wlcore: Processing message type 2, len 4
[  387.811862] wlcore: CMD# 27, len=20
[  387.811867] wlcore: acx slot
[  387.811870] wlcore: cmd configure (5), TSFL 17a376cc
[  387.812378] wlcore: Reading control info
[  387.812441] wlcore: Processing message type 2, len 4
[  387.812451] wlcore: CMD# 27, len=20
[  387.812456] wlcore: acx service period timeout
[  387.812459] wlcore: cmd configure (46), TSFL 17a37919
[  387.813027] wlcore: Reading control info
[  387.813089] wlcore: Processing message type 2, len 4
[  387.813099] wlcore: CMD# 27, len=24
[  387.813104] wlcore: acx rts threshold: 2353
[  387.813108] wlcore: cmd configure (74), TSFL 17a37ba2
[  387.813280] wlcore: Reading control info
[  387.813342] wlcore: Processing message type 2, len 4
[  387.813352] wlcore: CMD# 27, len=20
[  387.813358] wlcore: tx param cfg 0 cw_ming 15 cw_max 63 aifs 3 txop 0
[  387.813363] wlcore: tx param cfg ps_scheme 0 is_mu_edca 0 mu_edca_aifs 0 mu_edca_ecw_min_max 0 mu_edca_timer 0
[  387.813369] wlcore: cmd configure (1), TSFL 17a37c9f
[  387.813903] wlcore: Reading control info
[  387.813965] wlcore: Processing message type 2, len 4
[  387.813976] wlcore: CMD# 27, len=32
[  387.813980] wlcore: tx param cfg 1 cw_ming 15 cw_max 63 aifs 7 txop 0
[  387.813985] wlcore: tx param cfg ps_scheme 0 is_mu_edca 0 mu_edca_aifs 0 mu_edca_ecw_min_max 0 mu_edca_timer 0
[  387.813991] wlcore: cmd configure (1), TSFL 17a37f0e
[  387.814542] wlcore: Reading control info
[  387.814604] wlcore: Processing message type 2, len 4
[  387.814615] wlcore: CMD# 27, len=32
[  387.814620] wlcore: tx param cfg 2 cw_ming 15 cw_max 63 aifs 1 txop 3008
[  387.814625] wlcore: tx param cfg ps_scheme 0 is_mu_edca 0 mu_edca_aifs 0 mu_edca_ecw_min_max 0 mu_edca_timer 0
[  387.814630] wlcore: cmd configure (1), TSFL 17a3818d
[  387.815186] wlcore: Reading control info
[  387.815248] wlcore: Processing message type 2, len 4
[  387.815261] wlcore: CMD# 27, len=32
[  387.815266] wlcore: tx param cfg 3 cw_ming 15 cw_max 63 aifs 1 txop 1504
[  387.815272] wlcore: tx param cfg ps_scheme 0 is_mu_edca 0 mu_edca_aifs 0 mu_edca_ecw_min_max 0 mu_edca_timer 0
[  387.815277] wlcore: cmd configure (1), TSFL 17a38411
[  387.815820] wlcore: Reading control info
[  387.815882] wlcore: Processing message type 2, len 4
[  387.815892] wlcore: CMD# 27, len=32
[  387.815897] wlcore: acx feature cfg
[  387.815899] wlcore: cmd configure (78), TSFL 17a3868b
[  387.816064] wlcore: Reading control info
[  387.816127] wlcore: Processing message type 2, len 4
[  387.816137] wlcore: CMD# 27, len=28
[  387.816143] wlcore: cmd template_set 15 (role 1)
[  387.816148] wlcore: cmd template_set 0 (role 1)
[  387.816152] wlcore: cmd template_set 5 (role 1)
[  387.816156] wlcore: acx beacon filter opt enable=0
[  387.816160] wlcore: cmd configure (25), TSFL 17a3877f
[  387.816337] wlcore: Reading control info
[  387.816399] wlcore: Processing message type 2, len 4
[  387.816410] wlcore: CMD# 27, len=20
[  387.816415] wlcore: acx ba initiator policy
[  387.816418] wlcore: cmd configure (58), TSFL 17a38890
[  387.816602] wlcore: Reading control info
[  387.816664] wlcore: Processing message type 2, len 4
[  387.816674] wlcore: CMD# 27, len=24
[  387.816681] wlcore: mac80211 config psm off power 0 idle changed 0xffffff9d
[  387.816693] wlcore: mac80211 configure filter, FIF_ALLMULTI = 0
[  387.816697] wlcore: acx group address tbl
[  387.816700] wlcore: cmd configure (7), TSFL 17a38998
[  387.817560] wlcore: Reading control info
[  387.817622] wlcore: Processing message type 2, len 4
[  387.817633] wlcore: CMD# 27, len=140
[  387.817640] wlcore: mac80211 conf tx 0
[  387.817643] wlcore: tx param cfg 3 cw_ming 3 cw_max 7 aifs 1 txop 1504
[  387.817649] wlcore: tx param cfg ps_scheme 0 is_mu_edca 0 mu_edca_aifs 0 mu_edca_ecw_min_max 0 mu_edca_timer 0
[  387.817654] wlcore: cmd configure (1), TSFL 17a38d57
[  387.818197] wlcore: Reading control info
[  387.818259] wlcore: Processing message type 2, len 4
[  387.818269] wlcore: CMD# 27, len=32
[  387.818274] wlcore: mac80211 conf tx 1
[  387.818276] wlcore: tx param cfg 2 cw_ming 7 cw_max 15 aifs 1 txop 3008
[  387.818281] wlcore: tx param cfg ps_scheme 0 is_mu_edca 0 mu_edca_aifs 0 mu_edca_ecw_min_max 0 mu_edca_timer 0
[  387.818287] wlcore: cmd configure (1), TSFL 17a38fd4
[  387.818821] wlcore: Reading control info
[  387.818883] wlcore: Processing message type 2, len 4
[  387.818893] wlcore: CMD# 27, len=32
[  387.818898] wlcore: mac80211 conf tx 2
[  387.818901] wlcore: tx param cfg 0 cw_ming 15 cw_max 63 aifs 3 txop 0
[  387.818905] wlcore: tx param cfg ps_scheme 0 is_mu_edca 0 mu_edca_aifs 0 mu_edca_ecw_min_max 0 mu_edca_timer 0
[  387.818911] wlcore: cmd configure (1), TSFL 17a39244
[  387.819450] wlcore: Reading control info
[  387.819513] wlcore: Processing message type 2, len 4
[  387.819526] wlcore: CMD# 27, len=32
[  387.819532] wlcore: mac80211 conf tx 3
[  387.819534] wlcore: tx param cfg 1 cw_ming 15 cw_max 1023 aifs 7 txop 0
[  387.819540] wlcore: tx param cfg ps_scheme 0 is_mu_edca 0 mu_edca_aifs 0 mu_edca_ecw_min_max 0 mu_edca_timer 0
[  387.819545] wlcore: cmd configure (1), TSFL 17a394ba
[  387.820082] wlcore: Reading control info
[  387.820144] wlcore: Processing message type 2, len 4
[  387.820154] wlcore: CMD# 27, len=32
[  388.233254] OOM killer enabled.
[  388.236399] Restarting tasks ... done.
[  388.244426] wlcore: mac80211 configure filter, FIF_ALLMULTI = 0
[  388.245583] wlcore: acx group address tbl
[  388.245588] wlcore: cmd configure (7), TSFL 17a39731
[  388.245749] random: crng reseeded on system resumption
[  388.251029] remoteproc remoteproc1: powering up 79000000.r5f
[  388.256788] remoteproc remoteproc1: Booting fw image am62a-mcu-r5f0_0-fw, size 53172
[  388.264571] wlcore: Reading control info
[  388.264666] wlcore: Processing message type 2, len 4
[  388.264690] wlcore: CMD# 27, len=140
[  388.265651] rproc-virtio rproc-virtio.6.auto: assigned reserved memory node r5f-dma-memory@9b800000
[  388.274985] rproc-virtio rproc-virtio.6.auto: registered virtio0 (type 7)
[  388.281935] remoteproc remoteproc1: remote processor 79000000.r5f is now up
[  388.289130] PM: suspend exit
[  388.289888] virtio_rpmsg_bus virtio2: rpmsg host is online
[  388.299162] virtio_rpmsg_bus virtio0: rpmsg host is online
[  388.299643] virtio_rpmsg_bus virtio0: creating channel ti.ipc4.ping-pong addr 0xd
[  388.312856] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0xe
[  390.845893] am65-cpsw-nuss 8000000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
root@am62axx-evm:/opt/edgeai-gst-apps# modprobe ti_k3_dsp_remoteproc
[  477.819595] k3-dsp-rproc 7e000000.dsp: assigned reserved memory node c7x-dma-memory@99800000
[  477.828879] k3-dsp-rproc 7e000000.dsp: configured DSP for remoteproc mode
[  477.835849] remoteproc remoteproc3: 7e000000.dsp is available
[  477.841894] k3-dsp-rproc 7e000000.dsp: register pm nitifiers in remoteproc mode
[  477.861965] remoteproc remoteproc3: powering up 7e000000.dsp
[  477.867673] remoteproc remoteproc3: Booting fw image am62a-c71_0-fw, size 11534416
[  477.879615] k3-dsp-rproc 7e000000.dsp: mbox_request_channel failed: -16
[  477.886269] remoteproc remoteproc3: can't start rproc 7e000000.dsp: -16

retry to launch inference

root@am62axx-evm:/opt/edgeai-gst-apps# apps_python/app_edgeai.py configs/object_detection.yaml
libtidl_onnxrt_EP loaded 0x1e658890
Final number of subgraphs created are : 1, - Offloaded Nodes - 283, Total Nodes - 283
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
_rpmsg_char_find_ctrldev: could not find the matching rpmsg_ctrl device for virtio2.rpmsg_chrdev.-1.13
_rpmsg_char_find_rproc: 7e000000.dsp does not have any virtio devices!
IPC: ERROR: Unable to create TX channels for CPU [mcu1_0] !!!
IPC: ERROR: Unable to create TX channels for CPU [c7x_1] !!!
IPC: Init ... Done !!!
APP: ERROR: IPC init failed !!!
REMOTE_SERVICE: Init ... !!!
_rpmsg_char_find_ctrldev: could not find the matching rpmsg_ctrl device for virtio2.rpmsg_chrdev.-1.21
_rpmsg_char_find_rproc: 7e000000.dsp does not have any virtio devices!
REMOTE_SERVICE: Init ... Done !!!
  3578.568102 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  3578.571399 s:  VX_ZONE_INIT:Enabled
  3578.571432 s:  VX_ZONE_ERROR:Enabled
  3578.571448 s:  VX_ZONE_WARNING:Enabled
  3578.572840 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-0
  3578.573070 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-1
  3578.573257 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-2
  3578.573424 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-3
  3578.573448 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
  3578.574047 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
  3578.587521 s:  VX_ZONE_ERROR:[ownIpcSendMsg:140] OpenVX send notification failed
  3578.587564 s:  VX_ZONE_ERROR:[ownObjDescSend:307] ownIpcSendMsg failed
  3578.587577 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  3578.587594 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  3578.587608 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  3578.587623 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  3578.587642 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  3578.587654 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
==========[INPUT PIPELINE(S)]==========

[PIPE-0]

multifilesrc location=/opt/edgeai-test-data/videos/video0_1280_768.h264 loop=True ! h264parse ! v4l2h264dec capture-io-mode=5 ! tiovxmemalloc pool-size=12 ! capsfilter caps="video/x-raw, format=(string)NV12;" ! tiovxmultiscaler name=split_01
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)1280, height=(int)720;" ! tiovxdlcolorconvert out-pool-size=4 ! capsfilter caps="video/x-raw, format=(string)RGB;" ! appsink max-buffers=2 drop=True name=sen_0
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)416, height=(int)416;" ! tiovxdlpreproc out-pool-size=4 data-type=3 tensor-format=1 ! capsfilter caps="application/x-tensor-tiovx;" ! appsink max-buffers=2 drop=True name=pre_0


==========[OUTPUT PIPELINE]==========

appsrc do-timestamp=True format=3 block=True name=post_0 ! tiovxdlcolorconvert ! capsfilter caps="video/x-raw, format=(string)NV12, width=(int)1280, height=(int)720;" ! queue ! mosaic_0.sink_0

tiovxmosaic target=1 background=/tmp/background_0 name=mosaic_0 src::pool-size=2
sink_0::startx="<320>" sink_0::starty="<150>" sink_0::widths="<1280>" sink_0::heights="<720>"
! capsfilter caps="video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080;" ! v4l2jpegenc ! multifilesink sync=False location=/opt/edgeai-test-data/output/output_image_%04d.jpg

  3579.091247 s:  VX_ZONE_ERROR:[ownIpcSendMsg:140] OpenVX send notification failed
  3579.091297 s:  VX_ZONE_ERROR:[ownObjDescSend:307] ownIpcSendMsg failed
  3579.091320 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node scaler_node
  3579.091333 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  3579.091345 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  3579.091360 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.hwa.vpac_msc_multi_scale ... failed !!!
  3579.091380 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  3579.091392 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  3579.091587 s:  VX_ZONE_ERROR:[ownReleaseReferenceInt:598] Invalid reference

There is an issue when i reload the k3-dsp-rproc i suppose ? Do you have an idea how to resolve this ?

Yes, I have seen an issue in the past with not building from a fresh yocto install.. there were some cached libraries that were not cross-compatible with a different SDK. These are

root@am62axx-evm:/opt/edgeai-gst-apps# modprobe ti_k3_dsp_remoteproc

Are you reloading this here or was the module/driver not running previously such that this was the first load? I am curious why a reload is necessary.

Are you using the firmware images included in the SDK/targetfs or building them yourself with firmware-builder?

These threads may be relevant since the core problem was from the Yocto build. The first was a problem related to libtivisions_apps and the Yocto recipe that pulled associated libraries. You may need to clear the downloads directory and sstate cache

• https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1341665/sk-am62a-lp-manually-compile-u-boot-for-evm 
• https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1282935/am62a7-cannot-run-example-imx219-pipeline-rpmsg-errors

Can you also run /opt/vx_app_arm_remote_log.out in the background here and include those logs too? Those logs will shown TI OpenVX messages that may not be displaying otherwise.

[  387.547667] PM: suspend entry (deep)

Are you using any power management or sleep modes? Our current software does not have full support for suspend-resume of the C7x MMA. Perhaps this always shows, and I just haven't noticed until observing these logs more critically.

BR,
Reese

Oh yes, it's because i use deep sleep in my application and like noticied here

https://software-dl.ti.com/processor-sdk-linux/esd/AM62AX/09_02_00/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/Power_Management/pm_low_power_modes.html

we need ton unload this module before to go to deep sleep
Then i reload the module after the wake up, but this seems not working.

Reese Grimsley said:

Are you using the firmware images included in the SDK/targetfs or building them yourself with firmware-builder?

Yes i use the firmware provided by default with the sdk.

Reese Grimsley said:

Can you also run /opt/vx_app_arm_remote_log.out in the background here and include those logs too? Those logs will shown TI OpenVX messages that may not be displaying otherwise

root@am62axx-evm:/opt/edgeai-gst-apps# /opt/vx_app_arm_remote_log.out
[MCU1_0]      0.043867 s: CIO: Init ... Done !!!
[MCU1_0]      0.043908 s: APP: Init ... !!!
[MCU1_0]      0.043937 s: MEM: Init ... !!!
[MCU1_0]      0.043953 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ af000000 of size 16777216 bytes !!!
[MCU1_0]      0.043993 s: MEM: Init ... Done !!!
[MCU1_0]      0.044017 s: IPC: Init ... !!!
[MCU1_0]      0.044034 s: IPC: 3 CPUs participating in IPC !!!
[MCU1_0]      0.044337 s: IPC: Waiting for HLOS to be ready ... !!!
[MCU1_0]      0.048689 s: #Sciserver Version: v2023.11.0.0REL.MCUSDK.MM.NN.PP.bb
[MCU1_0]      0.051462 s: ##RM_PM_HAL Version: vMM.NN.PP
[MCU1_0]      0.054409 s: ##Starting Sciserver..... PASSED
[MCU1_0]     12.323335 s: IPC: HLOS is ready !!!
[MCU1_0]     12.323417 s: IPC: Init ... Done !!!
[MCU1_0]     12.323438 s: APP: Syncing with 2 CPUs ... !!!
[MCU1_0]     12.323460 s: APP: Syncing with 2 CPUs ... Done !!!
[MCU1_0]     12.323478 s: REMOTE_SERVICE: Init ... !!!
[MCU1_0]     12.323551 s: REMOTE_SERVICE: Init ... Done !!!
[MCU1_0]     12.323574 s: FVID2: Init ... !!!
[MCU1_0]     12.323602 s: FVID2: Init ... Done !!!
[MCU1_0]     12.323637 s: VHWA: VPAC Init ... !!!
[MCU1_0]     12.323652 s: SCICLIENT: Sciclient_pmSetModuleState module=219 state=2
[MCU1_0]     12.323754 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU1_0]     12.323777 s: VHWA: LDC Init ... !!!
[MCU1_0]     12.323905 s: VHWA: LDC Init ... Done !!!
[MCU1_0]     12.323926 s: VHWA: MSC Init ... !!!
[MCU1_0]     12.324242 s: VHWA: MSC Init ... Done !!!
[MCU1_0]     12.324264 s: VHWA: VISS Init ... !!!
[MCU1_0]     12.324353 s: VHWA: VISS Init ... Done !!!
[MCU1_0]     12.324371 s: VHWA: VPAC Init ... Done !!!
[MCU1_0]     12.324434 s:  VX_ZONE_INIT:Enabled
[MCU1_0]     12.324451 s:  VX_ZONE_ERROR:Enabled
[MCU1_0]     12.324467 s:  VX_ZONE_WARNING:Enabled
[MCU1_0]     12.325531 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MCU1-0
[MCU1_0]     12.325617 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target VPAC_LDC1
[MCU1_0]     12.325684 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target VPAC_MSC1
[MCU1_0]     12.325755 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target VPAC_MSC2
[MCU1_0]     12.325826 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target VPAC_VISS1
[MCU1_0]     12.325857 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
[MCU1_0]     12.325879 s: APP: OpenVX Target kernel init ... !!!
[MCU1_0]     12.329578 s: APP: OpenVX Target kernel init ... Done !!!
[MCU1_0]     12.329604 s: VISS REMOTE SERVICE: Init ... !!!
[MCU1_0]     12.329645 s: VISS REMOTE SERVICE: Init ... Done !!!
[MCU1_0]     12.329663 s: APP: Init ... Done !!!
[MCU1_0]     12.329678 s: APP: Run ... !!!
[MCU1_0]     12.329692 s: IPC: Starting echo test ...
[MCU1_0]     12.329805 s: APP: Run ... Done !!!
[MCU1_0]     12.330761 s: IPC: Echo status: a530-0[.] r5f0-0[s] c75ss0[P]
[C7x_1 ]     12.156306 s: CIO: Init ... Done !!!
[C7x_1 ]     12.156324 s: APP: Init ... !!!
[C7x_1 ]     12.156337 s: SCICLIENT: Init ... !!!
[C7x_1 ]     12.156401 s: SCICLIENT: DMSC FW version [9.2.7--v09.02.07 (Kool Koala)]
[C7x_1 ]     12.156419 s: SCICLIENT: DMSC FW revision 0x9
[C7x_1 ]     12.156434 s: SCICLIENT: DMSC FW ABI revision 3.1
[C7x_1 ]     12.156449 s: SCICLIENT: Init ... Done !!!
[C7x_1 ]     12.156462 s: UDMA: Init ... !!!
[C7x_1 ]     12.156473 s: UDMA: Init ... Done !!!
[C7x_1 ]     12.156485 s: MEM: Init ... !!!
[C7x_1 ]     12.156497 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ b2000000 of size 117440512 bytes !!!
[C7x_1 ]     12.156524 s: MEM: Init ... Done !!!
[C7x_1 ]     12.156536 s: IPC: Init ... !!!
[C7x_1 ]     12.156547 s: IPC: 3 CPUs participating in IPC !!!
[C7x_1 ]     12.156753 s: IPC: Waiting for HLOS to be ready ... !!!
[C7x_1 ]     12.160689 s: IPC: HLOS is ready !!!
[C7x_1 ]     12.160763 s: IPC: Init ... Done !!!
[C7x_1 ]     12.160777 s: APP: Syncing with 2 CPUs ... !!!
[C7x_1 ]     12.323461 s: APP: Syncing with 2 CPUs ... Done !!!
[C7x_1 ]     12.323476 s: REMOTE_SERVICE: Init ... !!!
[C7x_1 ]     12.323685 s: REMOTE_SERVICE: Init ... Done !!!
[C7x_1 ]     12.323707 s:  VX_ZONE_INIT:Enabled
[C7x_1 ]     12.323721 s:  VX_ZONE_ERROR:Enabled
[C7x_1 ]     12.323734 s:  VX_ZONE_WARNING:Enabled
[C7x_1 ]     12.324442 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target DSP_C7-1
[C7x_1 ]     12.324534 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target DSP_C7-1_PRI_2
[C7x_1 ]     12.324633 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target DSP_C7-1_PRI_3
[C7x_1 ]     12.324726 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target DSP_C7-1_PRI_4
[C7x_1 ]     12.324819 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target DSP_C7-1_PRI_5
[C7x_1 ]     12.324910 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target DSP_C7-1_PRI_6
[C7x_1 ]     12.325000 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target DSP_C7-1_PRI_7
[C7x_1 ]     12.325091 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target DSP_C7-1_PRI_8
[C7x_1 ]     12.325117 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
[C7x_1 ]     12.325135 s: APP: OpenVX Target kernel init ... !!!
[C7x_1 ]     12.325334 s: APP: OpenVX Target kernel init ... Done !!!
[C7x_1 ]     12.325349 s: APP: Init ... Done !!!
[C7x_1 ]     12.325361 s: APP: Run ... !!!
[C7x_1 ]     12.325371 s: IPC: Starting echo test ...
[C7x_1 ]     12.325465 s: APP: Run ... Done !!!
[C7x_1 ]     12.330843 s: IPC: Echo status: a530-0[.] r5f0-0[P] c75ss0[s]

Hi Mickael,

Hmm, okay this is a matter of software support for those remote cores (C7x, MCU R5's) in the Edge AI SDK.

Suspend to deep sleep may be supported for those, but the software stacks on those cores do not currently support Resume such that the application can continue using them following a suspend to deep sleep. The foundational SDK (i.e. the SDK that doesn't have ISP/VPAC stack or deep learning stack) supports suspend-resume for baseline IPC functionality between those cores.

The documentation is confusing here, I think. It is saying that the remoteproc modules need to be unloaded before going into sleep state, and this is necessary such that some clocks can be switched off. The document declines to mention support when coming out of sleep state. The power modes are not enforcing a graceful shutdown on the DSP/C7xMMA core.

BR,
Reese

Ok thank you but is there a solution please ?

Hi Mickael, 

I will need to investigate this further to find the solution. Please allow some time here.

There may be a way for the C7x core and remote cores to partially reset such that the software stacks on those act like they are coming from a cold boot. I know this is less than ideal from a resume-latency perspective. Full support for resume with minimal resume-latency on those cores is a larger development item.

BR,

Reese

Hi reese, yes for the moment a reset will be a good solution i mean !

Let me know !

Regards

Hi Reese,

Any update on this subject ?

Hi Mikael,

Yes i can update, but I don't have a specific recommendation for you to try on your side yet. We are still investigating for the right solution here, as there are several complex pieces of software working together for this to operate cleanly.

The C7x firmware in current SDK does not support graceful shutdown at runtime, and needs a firmware level change to implement this function:

• https://software-dl.ti.com/mcu-plus-sdk/esd/AM62AX/09_02_00_38/exports/docs/api_guide_am62ax/GRACEFUL_REMOTECORE_SHUTDOWN.html
  
  • not yet implemented in Edge AI SDK stack for C7x. I'm looking into this further.

Apologies for the delays here. I am frequently out of office this month and haven't been able to do detailed testing here at the firmware level.

BR,
Reese

Hi Reese,

Thank you for your answer. Let me known when you find something that can help please !