AM62A7: Cannot run example IMX219 pipeline; rpmsg errors

Hi Michael,

The media-ctl command looks correct, but I see that there was an initialization error from dmesg. Is this running a custom image / device tree or is it the default one from the SDK? When I see an error like this, my first response is to reboot, especially if the device has been running for a while (not the case here, given dmesg printout).

Does /dev/video2 exist?

Best,
Reese.

Hi Reese,

This device tree is mostly a clone of the SDK, at least for the IMX219 DTO and CSI related nodes.  However it is a custom baseboard so I had to modify a couple of items including the reset GPIO numbers, etc. for our setup.

I have rebooted  and tried "fresh".

It looks like there is an additional device a /dev/video2 since 8.06 (we are using 9.00).

root@am62axx-evm:/opt/edgeai-gst-apps# v4l2-ctl --list-devices
j721e-csi2rx (platform:30102000.ticsi2rx):
 /dev/video3
 /dev/video4
 /dev/video5
 /dev/video6
 /dev/video7
 /dev/video8
 /dev/media0

e5010 (platform:e5010):
 /dev/video2

wave5-dec (platform:wave5-dec):
 /dev/video0

wave5-enc (platform:wave5-enc):
 /dev/video1

root@am62axx-evm:/opt/edgeai-gst-apps#

I have tried switching to /dev/video3.  It's a little better but I am still seeing complaints from the remote proc area:

root@am62axx-evm:/opt/edgeai-gst-apps# !387
media-ctl -V '"imx219 1-0010":0 [fmt:SRGGB10_1X10/1920x1080 field:none]'
root@am62axx-evm:/opt/edgeai-gst-apps# !gst:p
gst-launch-1.0 -v v4l2src device=/dev/video3 io-mode=dmabuf-import ! video/x-bayer, width=1920, height=1080, framerate=30/1, format=rggb10 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_SONY_IMX219_RPI" dcc-isp-file=/opt/imaging/imx219/dcc_viss_10b.bin sink_0::dcc-2a-file=/opt/imaging/imx219/dcc_2a_10b.bin format-msb=9 ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! kmssink driver-name=tidss
root@am62axx-evm:/opt/edgeai-gst-apps# gst-launch-1.0 -v v4l2src device=/dev/video3 io-mode=dmabuf-import ! video/x-bayer, width=1920, height=1080, framerate=30/1, format=rggb10 ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_SONY_IMX219_RPI" dcc-isp-file=/opt/imaging/imx219/dcc_viss_10b.bin sink_0::dcc-2a-file=/opt/imaging/imx219/dcc_2a_10b.bin format-msb=9 ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! kmssink driver-name=tidss
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
_rpmsg_char_find_ctrldev: could not find the matching rpmsg_ctrl device for virtio0.rpmsg_chrdev.-1.13
IPC: ERROR: Unable to create TX channels for CPU [mcu1_0] !!!
IPC: Init ... Done !!!
APP: ERROR: IPC init failed !!!
REMOTE_SERVICE: Init ... !!!
_rpmsg_char_find_ctrldev: could not find the matching rpmsg_ctrl device for virtio0.rpmsg_chrdev.-1.21
_rpmsg_char_find_ctrldev: could not find the matching rpmsg_ctrl device for virtio1.rpmsg_chrdev.-1.21
REMOTE_SERVICE: Init ... Done !!!
261227.218040 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
261227.220140 s: VX_ZONE_INIT:Enabled
261227.220180 s: VX_ZONE_ERROR:Enabled
261227.220189 s: VX_ZONE_WARNING:Enabled
261227.221976 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
261227.223314 s: VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!!
Setting pipeline to PAUSED ...
261227.313879 s: VX_ZONE_ERROR:[ownContextCreateCmdObj:151] context object descriptor [0] allocation failed
261227.313920 s: VX_ZONE_ERROR:[ownContextCreateCmdObj:154] context object descriptor [0] allocation failed
261227.313951 s: VX_ZONE_ERROR:[ownContextCreateCmdObj:155] Exceeded max object descriptors available. Increase TIVX_PLATFORM_MAX_OBJ_DESC_SHM_INST value
261227.313964 s: VX_ZONE_ERROR:[ownContextCreateCmdObj:156] Increase TIVX_PLATFORM_MAX_OBJ_DESC_SHM_INST value in source/platform/psdk_j7/common/soc/tivx_platform_psdk_<soc>.h
261227.313979 s: VX_ZONE_ERROR:[vxCreateContext:970] context objection creation failed
261227.313992 s: VX_ZONE_ERROR:[vxGetStatus:700] Reference is NULL
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstKMSSink:kmssink0: display-width = 1920
/GstPipeline:pipeline0/GstKMSSink:kmssink0: display-height = 1080
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-bayer, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)rggb10, interlace-mode=(string)progressive, colorimetry=(string)2:4:7:1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-bayer, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)rggb10, interlace-mode=(string)progressive, colorimetry=(string)2:4:7:1
/GstPipeline:pipeline0/GstTIOVXISP:tiovxisp0.GstTIOVXIspPad:sink_0: caps = video/x-bayer, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)rggb10, interlace-mode=(string)progressive, colorimetry=(string)2:4:7:1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-bayer, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)rggb10, interlace-mode=(string)progressive, colorimetry=(string)2:4:7:1

** (gst-launch-1.0:1424): CRITICAL **: 17:43:18.311: gst_tiovx_get_exemplar_from_caps: assertion 'context' failed
261227.318866 s: VX_ZONE_ERROR:[vxGetStatus:700] Reference is NULL

** (gst-launch-1.0:1424): CRITICAL **: 17:43:18.311: gst_tiovx_get_size_from_exemplar: assertion 'VX_SUCCESS == vxGetStatus (exemplar)' failed
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: No downstream pool to import from.
Additional debug info:
../gst-plugins-good-1.20.6/sys/v4l2/gstv4l2object.c(5302): gst_v4l2_object_decide_allocation (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
When importing DMABUF or USERPTR, we need a pool to import from
Execution ended after 0:00:00.003693360
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Setting pipeline to NULL ...
Additional debug info:
../gstreamer-1.20.6/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
261227.320766 s: VX_ZONE_ERROR:[vxGetStatus:700] Reference is NULL
ERROR: from element /GstPipeline:pipeline0/GstTIOVXISP:tiovxisp0: Unable to init TIOVX module
Additional debug info:
../git/gst-libs/gst/tiovx/gsttiovxmiso.c(1508): gst_tiovx_miso_negotiated_src_caps (): /GstPipeline:pipeline0/GstTIOVXISP:tiovxisp0
Freeing pipeline ...
261227.328384 s: VX_ZONE_ERROR:[vxGetStatus:700] Reference is NULL
261227.328418 s: VX_ZONE_INIT:[tivxHostDeInitLocal:110] De-Initialization Done for HOST !!!
261227.332971 s: VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
Caught SIGSEGV
#0 0x0000ffff96cc91d8 in wait4 () from /lib/libc.so.6
#1 0x0000ffff96e64358 in g_on_error_stack_trace ()
#2 0x00000000004053a4 in ?? ()
#3 <signal handler called>
#4 0x0000ffff96c91ae4 in ?? () from /lib/libc.so.6
#5 0x0000ffff9546a6bc in appIpcDeleteRpmsgRxTask ()
#6 0x0000ffff95469bf0 in appIpcDeInit ()
#7 0x0000ffff954709f8 in appCommonDeInit ()
#8 0x0000ffff9671950c in ?? () from /usr/lib/libgsttiovx-1.0.so.1
#9 0x0000ffff96ddd120 in g_object_unref () from /usr/lib/libgobject-2.0.so.0
#10 0x0000ffff9671cb94 in ?? () from /usr/lib/libgsttiovx-1.0.so.1
#11 0x0000ffff96ddd120 in g_object_unref () from /usr/lib/libgobject-2.0.so.0
#12 0x0000ffff96fd6f1c in ?? () from /usr/lib/libgstreamer-1.0.so.0
#13 0x0000ffff96fd5f68 in gst_bin_remove () from /usr/lib/libgstreamer-1.0.so.0
#14 0x0000ffff96fd62bc in ?? () from /usr/lib/libgstreamer-1.0.so.0
#15 0x0000ffff96ddd01c in g_object_unref () from /usr/lib/libgobject-2.0.so.0
#16 0x0000000000404244 in ?? ()
#17 0x0000ffff96c3b230 in ?? () from /lib/libc.so.6
#18 0x0000ffff96c3b30c in __libc_start_main () from /lib/libc.so.6
#19 0x00000000004045f0 in _start ()
Spinning. Please run 'gdb gst-launch-1.0 1424' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

Hi Michael,

I see, so by my understanding, your SDK is based on 9.0 (I assume the full Edge AI SDK version, and not a less featured version like the base SDK or thin linux), and you have made changes at device tree level to match your baseboard. Please correct if I'm misunderstanding here.

I notice in your dmesg commands that there was a failure to load program segments for remoteproc1 -- I can replicate this in the stock 9.0 SDK, and I'm aware that this is a bug with the firmware linking. We can ignore that, as remoteproc1 is the MCU R5 that does not get used by default. The device management (DM, aka Wakeup) R5 is what runs the ISP driver. I also see the /dev/video2, which was used in the previous SDK, but just like you saw, that's now being used by another interface. /dev/video3 appears to be correct.

I'm able to run the following command (note I'm using RGGB8, and that the ISP files are under anadditional "linear" directory):

gst-launch-1.0 -v v4l2src device=/dev/video-rpi-cam0 io-mode=dmabuf-import ! video/x-bayer, width=1920, height=1080, framerate=30/1, format=rggb ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_SONY_IMX219_RPI" dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/imx219/linear/dcc_2a.bin format-msb=7 ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! kmssink driver-name=tidss

Has there been any change to the firmware for the DM R5 or the bootloader? That R5 core should populate as remoteproc2, and should also show as attached. The name "mcu1_0" refers to this core (and mcu2_0 is the other R5)

Could you also run the following command in the background? It will print additional message for OpenVX

/opt/vx_app_arm_remote_log.out &

Best,
Reese

Hi Reese,

We have not changed the the firmware from the default SDK.

We have had to rebuild the bootloader code as we are only using 2GB of LPDDR4 memory.  I am wondering if this might be the problem.  Is any of the firmware is attempting to use the upper regions of the 4 GB used by the EVM?  It's not clear from the documentation we have.

Here is the data when the vx_app_arm_remote_log.out is run:

root@am62axx-evm:/opt/edgeai-gst-apps# gst-launch-1.0 -v v4l2src device=/dev/video-rpi-cam0 io-mode=dmabuf-import ! video/x-bayer, width=1920, height=1080, framerate=30/1, format=rggb ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name="SENSOR_SONY_IMX219_RPI" dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/imx219/linear/dcc_2a.bin format-msb=7 ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! kmssink driver-name=tidss
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
_rpmsg_char_find_ctrldev: could not find the matching rpmsg_ctrl device for virtio1.rpmsg_chrdev.-1.13
IPC: ERROR: Unable to create TX channels for CPU [mcu1_0] !!!
IPC: Init ... Done !!!
APP: ERROR: IPC init failed !!!
REMOTE_SERVICE: Init ... !!!
_rpmsg_char_find_ctrldev: could not find the matching rpmsg_ctrl device for virtio1.rpmsg_chrdev.-1.21
_rpmsg_char_find_ctrldev: could not find the matching rpmsg_ctrl device for virtio0.rpmsg_chrdev.-1.21
REMOTE_SERVICE: Init ... Done !!!
  1882.309263 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  1882.311441 s:  VX_ZONE_INIT:Enabled
  1882.311483 s:  VX_ZONE_ERROR:Enabled
  1882.311492 s:  VX_ZONE_WARNING:Enabled
  1882.313049 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  1882.314568 s:  VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!!
Setting pipeline to PAUSED ...
  1882.405251 s:  VX_ZONE_ERROR:[ownContextCreateCmdObj:151] context object descriptor [0] allocation failed
  1882.405295 s:  VX_ZONE_ERROR:[ownContextCreateCmdObj:154] context object descriptor [0] allocation failed
  1882.405324 s:  VX_ZONE_ERROR:[ownContextCreateCmdObj:155] Exceeded max object descriptors available. Increase TIVX_PLATFORM_MAX_OBJ_DESC_SHM_INST value
  1882.405337 s:  VX_ZONE_ERROR:[ownContextCreateCmdObj:156] Increase TIVX_PLATFORM_MAX_OBJ_DESC_SHM_INST value in source/platform/psdk_j7/common/soc/tivx_platform_psdk_<soc>.h
  1882.405352 s:  VX_ZONE_ERROR:[vxCreateContext:970] context objection creation failed
  1882.405365 s:  VX_ZONE_ERROR:[vxGetStatus:700] Reference is NULL
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstKMSSink:kmssink0: display-width = 1920
/GstPipeline:pipeline0/GstKMSSink:kmssink0: display-height = 1080
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-bayer, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)rggb, interlace-mode=(string)progressive, colorimetry=(string)sRGB
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-bayer, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)rggb, interlace-mode=(string)progressive, colorimetry=(string)sRGB
/GstPipeline:pipeline0/GstTIOVXISP:tiovxisp0.GstTIOVXIspPad:sink_0: caps = video/x-bayer, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)rggb, interlace-mode=(string)progressive, colorimetry=(string)sRGB
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-bayer, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)rggb, interlace-mode=(string)progressive, colorimetry=(string)sRGB

** (gst-launch-1.0:1423): CRITICAL **: 17:43:33.091: gst_tiovx_get_exemplar_from_caps: assertion 'context' failed
  1882.409793 s:  VX_ZONE_ERROR:[vxGetStatus:700] Reference is NULL

** (gst-launch-1.0:1423): CRITICAL **: 17:43:33.092: gst_tiovx_get_size_from_exemplar: assertion 'VX_SUCCESS == vxGetStatus (exemplar)' failed
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: No downstream pool to import from.
Additional debug info:
../gst-plugins-good-1.20.6/sys/v4l2/gstv4l2object.c(5302): gst_v4l2_object_decide_allocation (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
When importing DMABUF or USERPTR, we need a pool to import from
Execution ended after 0:00:00.003642350
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Setting pipeline to NULL ...
Additional debug info:
../gstreamer-1.20.6/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
  1882.411585 s:  VX_ZONE_ERROR:[vxGetStatus:700] Reference is NULL
ERROR: from element /GstPipeline:pipeline0/GstTIOVXISP:tiovxisp0: Unable to init TIOVX module
Additional debug info:
../git/gst-libs/gst/tiovx/gsttiovxmiso.c(1508): gst_tiovx_miso_negotiated_src_caps (): /GstPipeline:pipeline0/GstTIOVXISP:tiovxisp0
Freeing pipeline ...
  1882.419046 s:  VX_ZONE_ERROR:[vxGetStatus:700] Reference is NULL
  1882.419084 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:110] De-Initialization Done for HOST !!!
  1882.423604 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
Caught SIGSEGV
#0  0x0000ffff87d491d8 in wait4 () from /lib/libc.so.6
#1  0x0000ffff87ee4358 in g_on_error_stack_trace ()
#2  0x00000000004053a4 in ?? ()
#3  <signal handler called>
#4  0x0000ffff87d11ae4 in ?? () from /lib/libc.so.6
#5  0x0000ffff864ea6bc in appIpcDeleteRpmsgRxTask ()
#6  0x0000ffff864e9bf0 in appIpcDeInit ()
#7  0x0000ffff864f09f8 in appCommonDeInit ()
#8  0x0000ffff8779950c in ?? () from /usr/lib/libgsttiovx-1.0.so.1
#9  0x0000ffff87e5d120 in g_object_unref () from /usr/lib/libgobject-2.0.so.0
#10 0x0000ffff8779cb94 in ?? () from /usr/lib/libgsttiovx-1.0.so.1
#11 0x0000ffff87e5d120 in g_object_unref () from /usr/lib/libgobject-2.0.so.0
#12 0x0000ffff88056f1c in ?? () from /usr/lib/libgstreamer-1.0.so.0
#13 0x0000ffff88055f68 in gst_bin_remove () from /usr/lib/libgstreamer-1.0.so.0
#14 0x0000ffff880562bc in ?? () from /usr/lib/libgstreamer-1.0.so.0
#15 0x0000ffff87e5d01c in g_object_unref () from /usr/lib/libgobject-2.0.so.0
#16 0x0000000000404244 in ?? ()
#17 0x0000ffff87cbb230 in ?? () from /lib/libc.so.6
#18 0x0000ffff87cbb30c in __libc_start_main () from /lib/libc.so.6
#19 0x00000000004045f0 in _start ()
Spinning.  Please run 'gdb gst-launch-1.0 1423' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

Hi Michael,

Thanks for mentioning the change to a smaller DDR module. That's worth investigating. Did you modify any of the DDR configurations for the device tree? I believe some of those may require modification. We have sections of DDR allocated for the various cores on the device.

This page from the SDK docs gives some information on changing the memory map: https://software-dl.ti.com/processor-sdk-linux/esd/AM62AX/latest/exports/docs/linux/Foundational_Components_IPC62ax.html#changing-the-memory-map

There is also a python script within vision_apps that is intended to help resize regions and avoid overlapping: https://git.ti.com/cgit/processor-sdk/vision_apps/tree/platform/am62a/rtos/gen_linker_mem_map.py?h=main

Looking the CMA within linux may also be helpful here - it's listed within /proc/meminfo

Best,
Reese

Hi Reese,

Thanks for the links.  I did check the memory maps and these pages, but defined regions in the SDK are all contained within the 0x80000000 -> 0xFFFFFFFF (2 GB) region that we have available.  I didn't see a need to change the memory map.  Maybe I am missing something?

The python scripts says that the AM62A is using 2 GB (what we have), yet the schematic and EVK appears to include 4 GB.  

I did update the DTS to set the kernel memory side to 2 GB:

/proc/meminfo seems sane, except for the VmallocTotal entry, which makes no sense.

root@am62axx-evm:/opt/edgeai-gst-apps# cat /proc/meminfo
MemTotal:        1404776 kB
MemFree:          959732 kB
MemAvailable:    1149796 kB
Buffers:           10224 kB
Cached:           237708 kB
SwapCached:            0 kB
Active:            77816 kB
Inactive:         261856 kB
Active(anon):        476 kB
Inactive(anon):   101004 kB
Active(file):      77340 kB
Inactive(file):   160852 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         91676 kB
Mapped:            90004 kB
Shmem:              9740 kB
KReclaimable:      27644 kB
Slab:              52464 kB
SReclaimable:      27644 kB
SUnreclaim:        24820 kB
KernelStack:        2416 kB
PageTables:         1664 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      702388 kB
Committed_AS:     420908 kB
VmallocTotal:   133143592960 kB
VmallocUsed:        7712 kB
VmallocChunk:          0 kB
Percpu:             1264 kB
HardwareCorrupted:     0 kB
AnonHugePages:     49152 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:         589824 kB
CmaFree:          477752 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB

Hi Michael,

I see what you mean. All of the defined memory regions appear to be within those 2 GB. They defined the early version of the board / firmware to work with 2 GB and replaced with a comparable 4 GB module later. The SK EVM does includes 4G, and it looks like the upper 2G is allocated in the additional DDR memory region at 0x 8 8000 0000. For what it's worth, I also see the exact same nonsensical VmallocTotal value in meminfo on the stock SDK.

Changes to the memory map at this level are probably not the root cause here. I suppose the memory map file for the DM R5 should also not change, given the existing allocations are all reasonable for 2G.

Let me check back with some other members of our team for this, please give me a day or two on this. Some of our SDK team is on holiday until Wednesday.

Best,
Reese

Quick poke for a little more info, could you run a command like the following?

k3conf dump device

If the DM R5 is not responding at all to IPC, then this should fail. I'm trying to get a baseline of what that core will respond to.

-Reese

Sorry I had left for the day.

Here is the output:

root@am62axx-evm:/opt/edgeai-gst-apps# k3conf dump device

|------------------------------------------------------------------------------|
| VERSION INFO                                                                 |
|------------------------------------------------------------------------------|
| K3CONF | (version v0.1-90-g1dd468d built Tue Jul 11 08:13:20 PM UTC 2023)    |
| SoC    | AM62Ax SR1.0                                                        |
| SYSFW  | ABI: 3.1 (firmware version 0x0009 '9.0.5--v09.00.05 (Kool Koala))') |
|------------------------------------------------------------------------------|

|-----------------------------------------------------------------------|
| Device ID | Device Name                            | Device Status    |
|-----------------------------------------------------------------------|
|     1     | AM62AX_DEV_CMP_EVENT_INTROUTER0        | DEVICE_STATE_ON  |
|     2     | AM62AX_DEV_DBGSUSPENDROUTER0           | DEVICE_STATE_ON  |
|     3     | AM62AX_DEV_MAIN_GPIOMUX_INTROUTER0     | DEVICE_STATE_ON  |
|     5     | AM62AX_DEV_WKUP_MCU_GPIOMUX_INTROUTER0 | DEVICE_STATE_ON  |
|     6     | AM62AX_DEV_TIMESYNC_EVENT_ROUTER0      | DEVICE_STATE_ON  |
|     7     | AM62AX_DEV_MCU_R5FSS0                  | DEVICE_STATE_ON  |
|     9     | AM62AX_DEV_MCU_R5FSS0_CORE0            | DEVICE_STATE_OFF |
|    13     | AM62AX_DEV_CPSW0                       | DEVICE_STATE_ON  |
|    15     | AM62AX_DEV_STM0                        | DEVICE_STATE_ON  |
|    16     | AM62AX_DEV_DCC0                        | DEVICE_STATE_ON  |
|    17     | AM62AX_DEV_DCC1                        | DEVICE_STATE_ON  |
|    18     | AM62AX_DEV_DCC2                        | DEVICE_STATE_ON  |
|    19     | AM62AX_DEV_DCC3                        | DEVICE_STATE_ON  |
|    20     | AM62AX_DEV_DCC4                        | DEVICE_STATE_ON  |
|    21     | AM62AX_DEV_DCC5                        | DEVICE_STATE_ON  |
|    22     | AM62AX_DEV_SMS0                        | DEVICE_STATE_ON  |
|    23     | AM62AX_DEV_MCU_DCC0                    | DEVICE_STATE_ON  |
|    24     | AM62AX_DEV_DEBUGSS_WRAP0               | DEVICE_STATE_ON  |
|    25     | AM62AX_DEV_DMASS0                      | DEVICE_STATE_ON  |
|    26     | AM62AX_DEV_DMASS0_BCDMA_0              | DEVICE_STATE_ON  |
|    27     | AM62AX_DEV_DMASS0_CBASS_0              | DEVICE_STATE_ON  |
|    28     | AM62AX_DEV_DMASS0_INTAGGR_0            | DEVICE_STATE_ON  |
|    29     | AM62AX_DEV_DMASS0_IPCSS_0              | DEVICE_STATE_ON  |
|    30     | AM62AX_DEV_DMASS0_PKTDMA_0             | DEVICE_STATE_ON  |
|    33     | AM62AX_DEV_DMASS0_RINGACC_0            | DEVICE_STATE_ON  |
|    35     | AM62AX_DEV_MCU_TIMER0                  | DEVICE_STATE_ON  |
|    36     | AM62AX_DEV_TIMER0                      | DEVICE_STATE_ON  |
|    37     | AM62AX_DEV_TIMER1                      | DEVICE_STATE_ON  |
|    38     | AM62AX_DEV_TIMER2                      | DEVICE_STATE_ON  |
|    39     | AM62AX_DEV_TIMER3                      | DEVICE_STATE_ON  |
|    40     | AM62AX_DEV_TIMER4                      | DEVICE_STATE_ON  |
|    41     | AM62AX_DEV_TIMER5                      | DEVICE_STATE_ON  |
|    42     | AM62AX_DEV_TIMER6                      | DEVICE_STATE_ON  |
|    43     | AM62AX_DEV_TIMER7                      | DEVICE_STATE_ON  |
|    48     | AM62AX_DEV_MCU_TIMER1                  | DEVICE_STATE_ON  |
|    49     | AM62AX_DEV_MCU_TIMER2                  | DEVICE_STATE_ON  |
|    50     | AM62AX_DEV_MCU_TIMER3                  | DEVICE_STATE_ON  |
|    51     | AM62AX_DEV_ECAP0                       | DEVICE_STATE_ON  |
|    52     | AM62AX_DEV_ECAP1                       | DEVICE_STATE_ON  |
|    53     | AM62AX_DEV_ECAP2                       | DEVICE_STATE_ON  |
|    54     | AM62AX_DEV_ELM0                        | DEVICE_STATE_OFF |
|    55     | AM62AX_DEV_EMIF_DATA_ISO_VD            | DEVICE_STATE_ON  |
|    57     | AM62AX_DEV_MMCSD0                      | DEVICE_STATE_ON  |
|    58     | AM62AX_DEV_MMCSD1                      | DEVICE_STATE_ON  |
|    59     | AM62AX_DEV_EQEP0                       | DEVICE_STATE_ON  |
|    60     | AM62AX_DEV_EQEP1                       | DEVICE_STATE_ON  |
|    61     | AM62AX_DEV_WKUP_GTC0                   | DEVICE_STATE_ON  |
|    62     | AM62AX_DEV_EQEP2                       | DEVICE_STATE_ON  |
|    63     | AM62AX_DEV_ESM0                        | DEVICE_STATE_ON  |
|    64     | AM62AX_DEV_WKUP_ESM0                   | DEVICE_STATE_ON  |
|    73     | AM62AX_DEV_FSS0                        | DEVICE_STATE_ON  |
|    74     | AM62AX_DEV_FSS0_FSAS_0                 | DEVICE_STATE_ON  |
|    75     | AM62AX_DEV_FSS0_OSPI_0                 | DEVICE_STATE_ON  |
|    76     | AM62AX_DEV_GICSS0                      | DEVICE_STATE_ON  |
|    77     | AM62AX_DEV_GPIO0                       | DEVICE_STATE_ON  |
|    78     | AM62AX_DEV_GPIO1                       | DEVICE_STATE_ON  |
|    79     | AM62AX_DEV_MCU_GPIO0                   | DEVICE_STATE_ON  |
|    80     | AM62AX_DEV_GPMC0                       | DEVICE_STATE_OFF |
|    83     | AM62AX_DEV_LED0                        | DEVICE_STATE_ON  |
|    85     | AM62AX_DEV_DDPA0                       | DEVICE_STATE_ON  |
|    86     | AM62AX_DEV_EPWM0                       | DEVICE_STATE_ON  |
|    87     | AM62AX_DEV_EPWM1                       | DEVICE_STATE_ON  |
|    88     | AM62AX_DEV_EPWM2                       | DEVICE_STATE_ON  |
|    95     | AM62AX_DEV_WKUP_VTM0                   | DEVICE_STATE_ON  |
|    96     | AM62AX_DEV_MAILBOX0                    | DEVICE_STATE_ON  |
|    97     | AM62AX_DEV_MAIN2MCU_VD                 | DEVICE_STATE_ON  |
|    98     | AM62AX_DEV_MCAN0                       | DEVICE_STATE_OFF |
|   100     | AM62AX_DEV_MCU_MCRC64_0                | DEVICE_STATE_ON  |
|   102     | AM62AX_DEV_I2C0                        | DEVICE_STATE_ON  |
|   103     | AM62AX_DEV_I2C1                        | DEVICE_STATE_ON  |
|   104     | AM62AX_DEV_I2C2                        | DEVICE_STATE_ON  |
|   105     | AM62AX_DEV_I2C3                        | DEVICE_STATE_ON  |
|   106     | AM62AX_DEV_MCU_I2C0                    | DEVICE_STATE_ON  |
|   107     | AM62AX_DEV_WKUP_I2C0                   | DEVICE_STATE_ON  |
|   110     | AM62AX_DEV_WKUP_TIMER0                 | DEVICE_STATE_ON  |
|   111     | AM62AX_DEV_WKUP_TIMER1                 | DEVICE_STATE_ON  |
|   114     | AM62AX_DEV_WKUP_UART0                  | DEVICE_STATE_ON  |
|   116     | AM62AX_DEV_MCRC64_0                    | DEVICE_STATE_ON  |
|   117     | AM62AX_DEV_WKUP_RTCSS0                 | DEVICE_STATE_ON  |
|   118     | AM62AX_DEV_WKUP_R5FSS0_SS0             | DEVICE_STATE_ON  |
|   119     | AM62AX_DEV_WKUP_R5FSS0                 | DEVICE_STATE_ON  |
|   121     | AM62AX_DEV_WKUP_R5FSS0_CORE0           | DEVICE_STATE_ON  |
|   125     | AM62AX_DEV_RTI0                        | DEVICE_STATE_ON  |
|   126     | AM62AX_DEV_RTI1                        | DEVICE_STATE_ON  |
|   127     | AM62AX_DEV_RTI2                        | DEVICE_STATE_ON  |
|   128     | AM62AX_DEV_RTI3                        | DEVICE_STATE_ON  |
|   131     | AM62AX_DEV_MCU_RTI0                    | DEVICE_STATE_OFF |
|   132     | AM62AX_DEV_WKUP_RTI0                   | DEVICE_STATE_ON  |
|   134     | AM62AX_DEV_COMPUTE_CLUSTER0            | DEVICE_STATE_ON  |
|   135     | AM62AX_DEV_A53SS0_CORE_0               | DEVICE_STATE_ON  |
|   136     | AM62AX_DEV_A53SS0_CORE_1               | DEVICE_STATE_ON  |
|   137     | AM62AX_DEV_A53SS0_CORE_2               | DEVICE_STATE_ON  |
|   138     | AM62AX_DEV_A53SS0_CORE_3               | DEVICE_STATE_ON  |
|   139     | AM62AX_DEV_PSCSS0                      | DEVICE_STATE_ON  |
|   140     | AM62AX_DEV_WKUP_PSC0                   | DEVICE_STATE_ON  |
|   141     | AM62AX_DEV_MCSPI0                      | DEVICE_STATE_ON  |
|   142     | AM62AX_DEV_MCSPI1                      | DEVICE_STATE_ON  |
|   143     | AM62AX_DEV_MCSPI2                      | DEVICE_STATE_ON  |
|   146     | AM62AX_DEV_UART0                       | DEVICE_STATE_ON  |
|   147     | AM62AX_DEV_MCU_MCSPI0                  | DEVICE_STATE_ON  |
|   148     | AM62AX_DEV_MCU_MCSPI1                  | DEVICE_STATE_ON  |
|   149     | AM62AX_DEV_MCU_UART0                   | DEVICE_STATE_ON  |
|   150     | AM62AX_DEV_SPINLOCK0                   | DEVICE_STATE_ON  |
|   152     | AM62AX_DEV_UART1                       | DEVICE_STATE_ON  |
|   153     | AM62AX_DEV_UART2                       | DEVICE_STATE_ON  |
|   154     | AM62AX_DEV_UART3                       | DEVICE_STATE_ON  |
|   155     | AM62AX_DEV_UART4                       | DEVICE_STATE_ON  |
|   156     | AM62AX_DEV_UART5                       | DEVICE_STATE_ON  |
|   157     | AM62AX_DEV_BOARD0                      | DEVICE_STATE_ON  |
|   158     | AM62AX_DEV_UART6                       | DEVICE_STATE_ON  |
|   161     | AM62AX_DEV_USB0                        | DEVICE_STATE_ON  |
|   162     | AM62AX_DEV_USB1                        | DEVICE_STATE_ON  |
|   163     | AM62AX_DEV_PBIST0                      | DEVICE_STATE_ON  |
|   165     | AM62AX_DEV_WKUP_PBIST0                 | DEVICE_STATE_ON  |
|   166     | AM62AX_DEV_A53SS0                      | DEVICE_STATE_ON  |
|   167     | AM62AX_DEV_COMPUTE_CLUSTER0_PBIST_0    | DEVICE_STATE_OFF |
|   168     | AM62AX_DEV_PSC0_FW_0                   | DEVICE_STATE_ON  |
|   169     | AM62AX_DEV_PSC0                        | DEVICE_STATE_ON  |
|   170     | AM62AX_DEV_DDR32SS0                    | DEVICE_STATE_ON  |
|   171     | AM62AX_DEV_DEBUGSS0                    | DEVICE_STATE_ON  |
|   172     | AM62AX_DEV_A53_RS_BW_LIMITER0          | DEVICE_STATE_ON  |
|   173     | AM62AX_DEV_A53_WS_BW_LIMITER1          | DEVICE_STATE_ON  |
|   176     | AM62AX_DEV_WKUP_DEEPSLEEP_SOURCES0     | DEVICE_STATE_ON  |
|   177     | AM62AX_DEV_EMIF_CFG_ISO_VD             | DEVICE_STATE_ON  |
|   178     | AM62AX_DEV_MAIN_USB0_ISO_VD            | DEVICE_STATE_ON  |
|   179     | AM62AX_DEV_MAIN_USB1_ISO_VD            | DEVICE_STATE_ON  |
|   180     | AM62AX_DEV_MCU_MCU_16FF0               | DEVICE_STATE_ON  |
|   182     | AM62AX_DEV_CSI_RX_IF0                  | DEVICE_STATE_ON  |
|   183     | AM62AX_DEV_DCC6                        | DEVICE_STATE_ON  |
|   184     | AM62AX_DEV_MMCSD2                      | DEVICE_STATE_OFF |
|   185     | AM62AX_DEV_DPHY_RX0                    | DEVICE_STATE_ON  |
|   186     | AM62AX_DEV_DSS0                        | DEVICE_STATE_ON  |
|   188     | AM62AX_DEV_MCU_MCAN0                   | DEVICE_STATE_OFF |
|   189     | AM62AX_DEV_MCU_MCAN1                   | DEVICE_STATE_OFF |
|   190     | AM62AX_DEV_MCASP0                      | DEVICE_STATE_OFF |
|   191     | AM62AX_DEV_MCASP1                      | DEVICE_STATE_OFF |
|   192     | AM62AX_DEV_MCASP2                      | DEVICE_STATE_OFF |
|   193     | AM62AX_DEV_CLK_32K_RC_SEL_DEV_VD       | DEVICE_STATE_ON  |
|   194     | AM62AX_DEV_CPT2_AGGR1                  | DEVICE_STATE_ON  |
|   195     | AM62AX_DEV_CPT2_AGGR0                  | DEVICE_STATE_ON  |
|   196     | AM62AX_DEV_MCU_CPT2_AGGR0              | DEVICE_STATE_ON  |
|   197     | AM62AX_DEV_MCU_DCC1                    | DEVICE_STATE_ON  |
|   198     | AM62AX_DEV_DMASS1                      | DEVICE_STATE_ON  |
|   199     | AM62AX_DEV_DMASS1_BCDMA_0              | DEVICE_STATE_ON  |
|   200     | AM62AX_DEV_DMASS1_INTAGGR_0            | DEVICE_STATE_ON  |
|   201     | AM62AX_DEV_JPGENC0                     | DEVICE_STATE_OFF |
|   202     | AM62AX_DEV_WKUP_PBIST1                 | DEVICE_STATE_ON  |
|   203     | AM62AX_DEV_MCU_PBIST0                  | DEVICE_STATE_ON  |
|   204     | AM62AX_DEV_CODEC0                      | DEVICE_STATE_ON  |
|   205     | AM62AX_DEV_RTI4                        | DEVICE_STATE_ON  |
|   206     | AM62AX_DEV_C7XV_RSWS_BS_LIMITER6       | DEVICE_STATE_ON  |
|   207     | AM62AX_DEV_C7X256V0                    | DEVICE_STATE_ON  |
|   208     | AM62AX_DEV_C7X256V0_C7XV_CORE_0        | DEVICE_STATE_ON  |
|   209     | AM62AX_DEV_C7X256V0_CORE0              | DEVICE_STATE_ON  |
|   210     | AM62AX_DEV_C7X256V0_CLEC               | DEVICE_STATE_ON  |
|   211     | AM62AX_DEV_C7X256V0_CLK                | DEVICE_STATE_ON  |
|   212     | AM62AX_DEV_C7X256V0_DEBUG              | DEVICE_STATE_ON  |
|   213     | AM62AX_DEV_C7X256V0_GICSS              | DEVICE_STATE_ON  |
|   214     | AM62AX_DEV_C7X256V0_PBIST              | DEVICE_STATE_OFF |
|   215     | AM62AX_DEV_JPGENC_RS_BW_LIMITER4       | DEVICE_STATE_ON  |
|   216     | AM62AX_DEV_JPGENC_WS_BW_LIMITER5       | DEVICE_STATE_ON  |
|   217     | AM62AX_DEV_VPAC_RSWS_BW_LIMITER8       | DEVICE_STATE_ON  |
|   218     | AM62AX_DEV_VPAC_RSWS_BW_LIMITER7       | DEVICE_STATE_ON  |
|   219     | AM62AX_DEV_VPAC0                       | DEVICE_STATE_OFF |
|   220     | AM62AX_DEV_PBIST3                      | DEVICE_STATE_OFF |
|   221     | AM62AX_DEV_CODEC_RS_BW_LIMITER2        | DEVICE_STATE_ON  |
|   222     | AM62AX_DEV_CODEC_WS_BW_LIMITER3        | DEVICE_STATE_ON  |
|   225     | AM62AX_DEV_HSM0                        | DEVICE_STATE_ON  |
|   226     | AM62AX_DEV_WKUP_CLKOUT_SEL_DEV_VD      | DEVICE_STATE_ON  |
|   227     | AM62AX_DEV_MCU_OBSCLK_MUX_SEL_DEV_VD   | DEVICE_STATE_ON  |
|   228     | AM62AX_DEV_OBSCLK0_MUX_SEL_DEV_VD      | DEVICE_STATE_ON  |
|-----------------------------------------------------------------------|

root@am62axx-evm:/opt/edgeai-gst-apps#

Hi Michael,

Thanks for trying this -- it confirms the DM R5 is responsive in general. It seems to be related to specific RPMSG channels. I understand in some cases, this can happen from DDR size changes due to invalid configuration of VRING and VIRTIO. Are you also using the firmware / tiboot3.bin from 9.0?

The virtio sysfs entries should be under /sys/bus/virtio/devices/virtio{0,1}. Are these populated in your virtio1? That's the one mentioned in the errors above. Contents on stock SDK:

root@am62axx-evm:/sys/bus/virtio/devices/virtio1# ls -l
total 0
-r--r--r-- 1 root root 4096 Oct 24 15:29 device
lrwxrwxrwx 1 root root    0 Oct 24 15:29 driver -> ../../../../../../../../../../bus/virtio/drivers/virtio_rpmsg_bus
-r--r--r-- 1 root root 4096 Oct 24 15:29 features
-r--r--r-- 1 root root 4096 Oct 24 15:29 modalias
drwxr-xr-x 2 root root    0 Oct 24 15:29 power
-r--r--r-- 1 root root 4096 Oct 24 15:29 status
lrwxrwxrwx 1 root root    0 Oct 24 14:54 subsystem -> ../../../../../../../../../../bus/virtio
-rw-r--r-- 1 root root 4096 Oct 24 14:54 uevent
-r--r--r-- 1 root root 4096 Oct 24 15:29 vendor
drwxr-xr-x 4 root root    0 Oct 24 14:54 virtio1.rpmsg_chrdev.-1.13
drwxr-xr-x 4 root root    0 Oct 24 14:54 virtio1.rpmsg_chrdev.-1.21
drwxr-xr-x 4 root root    0 Oct 24 14:54 virtio1.rpmsg_ctrl.0.0
drwxr-xr-x 3 root root    0 Oct 24 14:54 virtio1.rpmsg_ns.53.53
drwxr-xr-x 3 root root    0 Oct 24 14:54 virtio1.ti.ipc4.ping-pong.-1.14

It would be helpful to see what dmesg says for virtio as well:

[    6.046984] rproc-virtio rproc-virtio.6.auto: assigned reserved memory node c7x-dma-memory@99800000
[    6.094844] virtio_rpmsg_bus virtio0: rpmsg host is online
[    6.095569] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0xd
[    6.107214] rproc-virtio rproc-virtio.6.auto: registered virtio0 (type 7)
[    6.442148] rproc-virtio rproc-virtio.7.auto: assigned reserved memory node r5f-dma-memory@9c800000
[    6.458254] virtio_rpmsg_bus virtio1: rpmsg host is online
[    6.459180] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xd
[    6.471894] rproc-virtio rproc-virtio.7.auto: registered virtio1 (type 7)
[    6.477542] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0x15
[    6.486033] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xe
[    6.497467] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0x15
[    6.511902] virtio_rpmsg_bus virtio0: creating channel ti.ipc4.ping-pong addr 0xe

Acommon issue in other threads is with the resource table. You can show that table for the DM R5 with the command:

cat /sys/kernel/debug/remoteproc/remoteproc2/resource_table 

Some other potentially helpful threads:

• https://e2e.ti.com/support/processors-group/processors/f/processors-forum/991477/tda4vm-notify-a72-linux-app-from-dsps
• https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1272994/tda4vm-rpmsg-question-between-mcu2_1-and-mpu1_0

Hopefully what we find some these commands/logs will isolate the root of theproblem.

Best,
Reese

Hi Reese,

Here is what I see.  It is a little different from what you are showing (both dmesg and contents of virtio1).  I don't see the ti-.ipc4.ping-pong information at all on my unit.

root@am62axx-evm:/sys/bus/virtio/devices/virtio1# ls -l
total 0
-r--r--r-- 1 root root 4096 Apr 28 18:30 device
lrwxrwxrwx 1 root root    0 Apr 28 18:30 driver -> ../../../../../../../../bus/virtio/drivers/virtio_rpmsg_bus
-r--r--r-- 1 root root 4096 Apr 28 18:30 features
-r--r--r-- 1 root root 4096 Apr 28 18:30 modalias
drwxr-xr-x 2 root root    0 Apr 28 18:30 power
-r--r--r-- 1 root root 4096 Apr 28 18:30 status
lrwxrwxrwx 1 root root    0 Apr 28 17:42 subsystem -> ../../../../../../../../bus/virtio
-rw-r--r-- 1 root root 4096 Apr 28 17:42 uevent
-r--r--r-- 1 root root 4096 Apr 28 18:30 vendor
drwxr-xr-x 4 root root    0 Apr 28 17:42 virtio1.rpmsg_chrdev.-1.13
drwxr-xr-x 4 root root    0 Apr 28 17:42 virtio1.rpmsg_ctrl.0.0
drwxr-xr-x 3 root root    0 Apr 28 17:42 virtio1.rpmsg_ns.53.53
root@am62axx-evm:/sys/bus/virtio/devices/virtio1# dmesg | grep virtio
[    5.656862] rproc-virtio rproc-virtio.3.auto: assigned reserved memory node r5f-dma-memory@9c800000
[    5.675392] virtio_rpmsg_bus virtio0: rpmsg host is online
[    5.676082] virtio_rpmsg_bus virtio0: creating channel ti.ipc4.ping-pong addr 0xd
[    5.689539] rproc-virtio rproc-virtio.3.auto: registered virtio0 (type 7)
[    5.695835] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0xe
[    5.758780] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node c7x-dma-memory@99800000
[    5.783279] virtio_rpmsg_bus virtio1: rpmsg host is online
[    5.793056] rproc-virtio rproc-virtio.2.auto: registered virtio1 (type 7)
[    5.810425] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xd
root@am62axx-evm:/sys/bus/virtio/devices/virtio1# cat /sys/kernel/debug/remoteproc/remoteproc2/resource_table
Entry 0 is of type vdev
  ID 7
  Notify ID 0
  Device features 0x1
  Guest features 0x1
  Config length 0x0
  Status 0x7
  Number of vrings 2
  Reserved (should be zero) [0][0]

  Vring 0
    Device Address 0x9c800000
    Alignment 4096
    Number of buffers 256
    Notify ID 0
    Physical Address 0x0

  Vring 1
    Device Address 0x9c804000
    Alignment 4096
    Number of buffers 256
    Notify ID 1
    Physical Address 0x0

Entry 1 is of type trace
  Device Address 0x9c900400
  Length 0x1000 Bytes
  Reserved (should be zero) [0]
  Name trace:r5fss0_0

root@am62axx-evm:/sys/bus/virtio/devices/virtio1#

Thanks,

Mike

Hi Mike,

Much appreciated. I see two primary differences:

For the first, the rpmesg channels for 0x15 and 0xe are missing as shown in dmesg - I believe these are linked to the fact that virtio in SYSFS is missing the -1.14 and -1.21 character drivers

For the second one, I had missed including the stock resource table (only including entry 1 as entry 0 is identical):

Entry 1 is of type trace
  Device Address 0x9c900400
  Length 0x80000 Bytes
  Reserved (should be zero) [0]
  Name trace:r5f0  

The length of this entry in the resource table is different -- 512 kB on my side, and 4kB on yours. I also notice the name of the trace is slightly different (r5f0 vs. rf5fss0_0.

We have a good sense of the low-level symptoms of the problem I think. I will work with some folks more knowledgable on this portion of the SoC configuration to understand how we can resolve this - please allow a day or two to so I can collect info on how to move forward.

Thanks,
Reese

Hi Mike,

Could you also try running the following script?

/opt/vision_apps/vision_apps_init.sh

This is a more barebones version of what gstreamer plugins have to run under the hood to start OpenVX. The /opt/vx_app_arm_remote_log.out would print the same thing, so no need to run it in the background during this.

Best,
Reese

Hi Reese, 

This is all I get.  Seems like the Sciserver isn't running (when I compare to EVM output of same command)?

With regards,

Mike

root@am62axx-evm:/opt/edgeai-gst-apps# [C7x_1 ]  99863.305778 s: CIO: Init ... Done !!!
[C7x_1 ]  99863.305800 s: APP: Init ... !!!
[C7x_1 ]  99863.305813 s: SCICLIENT: Init ... !!!
[C7x_1 ]  99863.305886 s: SCICLIENT: DMSC FW version [9.0.5--v09.00.05 (Kool Koala)]
[C7x_1 ]  99863.305907 s: SCICLIENT: DMSC FW revision 0x9
[C7x_1 ]  99863.305923 s: SCICLIENT: DMSC FW ABI revision 3.1
[C7x_1 ]  99863.305938 s: SCICLIENT: Init ... Done !!!
[C7x_1 ]  99863.305950 s: UDMA: Init ... !!!
[C7x_1 ]  99863.305962 s: UDMA: Init ... Done !!!
[C7x_1 ]  99863.305973 s: MEM: Init ... !!!
[C7x_1 ]  99863.305985 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ b2000000 of size 117440512 bytes !!!
[C7x_1 ]  99863.306014 s: MEM: Init ... Done !!!
[C7x_1 ]  99863.306026 s: IPC: Init ... !!!
[C7x_1 ]  99863.306039 s: IPC: 3 CPUs participating in IPC !!!

root@am62axx-evm:/opt/edgeai-gst-apps#

Hi Reese,

Thanks for leading me to check the firmware.  Our build script had an error in it and was pulling the firmware from the git repository instead of from the SDK prebuilt area.  I rebuilt it using the instructions from here, pointing the BINMAN_INDIRS variable off to the SDK instead.

Ihttps://software-dl.ti.com/processor-sdk-linux/esd/AM62X/09_00_00_03/exports/docs/linux/Foundational_Components/U-Boot/UG-General-Info.html

Now the pipelines and the AI demos are working properly.

I am sorry for the trouble, this was operator error.

wIth regards,

Mike

Hi Mike,

No problem! Glad we resolved this. I hadn't seen this error before, so it's good to have a mental model around what this error mode maps to.

Best Regards,
Reese