AM68A: Adjustments for rproc/rpmsg if changing RAM

Hi Frank,

What HLOS are you using and which SDK version are you using? Could you go over the build step you took and share the commands that cause this failure? Providing more complete logs would help me understand the issue better.

Thank you,

Fabiana

Hello Fabiana,

we are using Linux from SDK 10.01.00.04. We build the whole Linux system with another build system than yocto. The processor sdk rtos has been built separately in a virtual machine running a suitable ubuntu. Our target system has only 2GB RAM and it's running pretty good so far.

I solved the issue by using the correct devicetree settings from the processor sdk rtos build. Now the system loads the firmware (vision_apps) for the r5f & c71 cores and seems to successfully starts them. I can observe this with the vx_app_arm_remote_log.out application. By the way: The processor sdk rtos builds vx_app_rtos_linux_mcu2_0.out & vx_app_rtos_linux_mcu2_1.out. These should be the firmware for the mcu2_0 & mcu2_1. Are these the only cores (and of course the c71) used by the vision_apps?

Now that the firmware of the vision_apps is running I start a simple gstreamer pipeline which unfortunately leads to a kernel crash:

[ 157.749691] Unable to handle kernel paging request at virtual address ffff000880000000
[ 157.760304] Mem abort info:
[ 157.763646] ESR = 0x0000000096000145
[ 157.767501] EC = 0x25: DABT (current EL), IL = 32 bits
[ 157.772891] SET = 0, FnV = 0
[ 157.775970] EA = 0, S1PTW = 0
[ 157.779117] FSC = 0x05: level 1 translation fault
[ 157.784080] Data abort info:
[ 157.786962] ISV = 0, ISS = 0x00000145, ISS2 = 0x00000000
[ 157.792435] CM = 1, WnR = 1, TnD = 0, TagAccess = 0
[ 157.797517] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 157.802822] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000082017000
[ 157.809549] [ffff000880000000] pgd=18000000ffff8003, p4d=18000000ffff8003, pud=0000000000000000
[ 157.818243] Internal error: Oops: 0000000096000145 [#1] PREEMPT SMP
[ 157.824493] Modules linked in: can_raw can tag_ksz cfg80211 onboard_usb_hub rpmsg_ctrl rpmsg_char ksz_spi ksz_switch dsa_core hsr cdns_csi2rx bridge stp cdns3 llc cdns_usb_common crct10dif_ce panel_adv7393(O) phy_can_transceiver ti_k6
[ 157.893276] CPU: 0 PID: 407 Comm: videotestsrc0:s Tainted: G O 6.6.44 #1
[ 157.901346] Hardware name: Rosho DirecsAI (DT)
[ 157.905773] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 157.912715] pc : dcache_clean_poc+0x20/0x38
[ 157.916898] lr : arch_sync_dma_for_device+0x24/0x30
[ 157.921761] sp : ffff80008371bbd0
[ 157.925061] x29: ffff80008371bbd0 x28: 0000000900000000 x27: 0000000000039000
[ 157.932180] x26: ffff00000c9976c0 x25: ffff000003b26038 x24: 0000000000000000
[ 157.939299] x23: fffffc0000000000 x22: ffff800080f97ae0 x21: 0000000000000000
[ 157.946417] x20: 0000000000000001 x19: 0000000000000000 x18: 0000000000000000
[ 157.953535] x17: 0000000000000000 x16: 0000000000000000 x15: 0000ffff9283d9d0
[ 157.960654] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 157.967773] x11: 0000000000000000 x10: 0000000000000000 x9 : ffff800081451948
[ 157.974892] x8 : ffff00000c9976e0 x7 : 0000000000000002 x6 : 0000000000000000
[ 157.982011] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 000000000000003f
[ 157.989128] x2 : 0000000000000040 x1 : ffff000880039000 x0 : ffff000880000000
[ 157.996247] Call trace:
[ 157.998682] dcache_clean_poc+0x20/0x38
[ 158.002506] dma_direct_map_sg+0x280/0x2e4
[ 158.006591] __dma_map_sg_attrs+0x28/0x94
[ 158.010586] dma_map_sg_attrs+0x10/0x24
[ 158.014408] dma_heap_map_dma_buf+0x40/0x5c
[ 158.018579] __map_dma_buf+0x2c/0xa0
[ 158.022141] dma_buf_map_attachment+0x9c/0x108
[ 158.026568] rproc_attach_dmabuf+0xcc/0x1b4
[ 158.030739] rproc_device_ioctl+0x11c/0x2ac
[ 158.034909] __arm64_sys_ioctl+0xac/0xf0
[ 158.038819] invoke_syscall+0x48/0x114
[ 158.042556] el0_svc_common.constprop.0+0xc0/0xe0
[ 158.047246] do_el0_svc+0x1c/0x28
[ 158.050548] el0_svc+0x2c/0x84
[ 158.053593] el0t_64_sync_handler+0x120/0x12c
[ 158.057934] el0t_64_sync+0x190/0x194
[ 158.061587] Code: d2800082 9ac32042 d1000443 8a230000 (d50b7a20)
[ 158.067662] ---[ end trace 0000000000000000 ]---

The pipeline causing the crash is:

$> /usr/bin/gst-launch-1.0 ! video/x-raw,format=RGB ! tiovxcolorconvert ! video/x-raw,format=NV12 ! fakesink


APP: Init ... !!!
6052.747019 s: MEM: Init ... !!!
6052.747077 s: MEM: Initialized DMA HEAP (fd=5) !!!
6052.747214 s: MEM: Init ... Done !!!
6052.747235 s: IPC: Init ... !!!
6052.798145 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
6052.805175 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
6052.805308 s: VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
6052.805325 s: VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
6052.805331 s: VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
6052.806106 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-0
6052.806276 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-1
6052.806385 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-2
6052.806497 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-3
6052.806510 s: VX_ZONE_INFO: [tivxInitLocal:126] Initialization Done !!!
6052.806517 s: VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...


Any ideas what we could make wrong or are missing?

With regards,

Frank

Hi Frank,

I am currently out of office on business travel. Please expect a ~1-2 day delay in my response.

Thank you,

Fabiana

Ok, I got it basically working.

Step 1 was to move the 2 memory regions above 2GB down into available DDR RAM.

In step 2 I modified the defines for the physical address of the local heap & scratch buffer in the c7x_1 & c7x_2 main.c files. In the defines the region above 2GB is hard coded.

Now the firmware loads and does not print errors. I can use tiovxcolorconvert (I guess its working) but not tiovxmultiscaler & tiovxmosaic. These last two plugins are loaded but pipeline is not running!

Frank

Hi Frank,

Could you please explain what you are looking to accomplish with this pipeline? GStreamer pipelines require a source to be defined which appears to be missing in the one you shared.

Take a look at some example GStreamer pipelines that use a camera as an input and output to a connected display: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1427869/faq-sk-am69-how-to-stream-from-csi-sensor-and-usb-camera-to-display-using-gstreamer

Thank you,

Fabiana

Oh, sorry! Missed the videotestsrc element at the beginning of the pipe. This pipeline seems to be running now with my modifications.

For the moment it seems to be proven that we can work with the tiovx stuff running in the MCU & C7x cores.

But our real target are the CSITX feature. We need to output images (rendered by the GPU) on the CSI output. When I try to instantiate the CSITX node with tivxCsitxNode I got an error:

VX_ZONE_ERROR: [tivxCreateNodeByKernelName:121] Call to vxGetKernelByName failed; kernel may not be registered

In the log (vx_app_arm_remote_log.out) it seems CSITX is available (as target but not as kernel?):

[MCU2_0] 28.480401 s: CIO: Init ... Done !!!
[MCU2_0] 28.480457 s: ### CPU Frequency = 1000000000 Hz
[MCU2_0] 28.480483 s: CPU is running FreeRTOS
[MCU2_0] 28.480500 s: APP: Init ... !!!
[MCU2_0] 28.480530 s: SCICLIENT: Init ... !!!
[MCU2_0] 28.480651 s: SCICLIENT: DMSC FW version [10.1.6--v10.01.06 (Fiery Fox)]
[MCU2_0] 28.480683 s: SCICLIENT: DMSC FW revision 0xa
[MCU2_0] 28.480704 s: SCICLIENT: DMSC FW ABI revision 4.0
[MCU2_0] 28.480725 s: SCICLIENT: Init ... Done !!!
[MCU2_0] 28.480744 s: UDMA: Init ... !!!
[MCU2_0] 28.481540 s: UDMA: Init ... Done !!!
[MCU2_0] 28.481573 s: UDMA: Init for CSITX/CSIRX ... !!!
[MCU2_0] 28.482002 s: UDMA: Init for CSITX/CSIRX ... Done !!!
[MCU2_0] 28.482033 s: MEM: Init ... !!!
[MCU2_0] 28.482056 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ b9000000 of size 14680064 bytes !!!
[MCU2_0] 28.482100 s: MEM: Created heap (L3_MEM, id=1, flags=0x00000000) @ 60000000 of size 524288 bytes !!!
[MCU2_0] 28.482137 s: MEM: Created heap (DDR_CACHE_WT_MEM, id=7, flags=0x00000000) @ b9e00000 of size 2097152 bytes !!!
[MCU2_0] 28.482174 s: MEM: Init ... Done !!!
[MCU2_0] 28.482191 s: IPC: Init ... !!!
[MCU2_0] 28.482226 s: IPC: 5 CPUs participating in IPC !!!
[MCU2_0] 28.482254 s: IPC: Waiting for HLOS to be ready ... !!!
[MCU2_0] 28.482280 s: IPC: HLOS is ready !!!
[MCU2_0] 28.485131 s: IPC: Init ... Done !!!
[MCU2_0] 28.485173 s: APP: Syncing with 4 CPUs ... !!!
[MCU2_0] 29.963086 s: APP: Syncing with 4 CPUs ... Done !!!
[MCU2_0] 29.963119 s: REMOTE_SERVICE: Init ... !!!
[MCU2_0] 29.963329 s: REMOTE_SERVICE: Init ... Done !!!
[MCU2_0] 29.963367 s: FVID2: Init ... !!!
[MCU2_0] 29.963514 s: FVID2: Init ... Done !!!
[MCU2_0] 29.963557 s: SCICLIENT: Sciclient_pmSetModuleState module=219 state=2
[MCU2_0] 29.963733 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0] 29.963768 s: VHWA: VPAC Init ... !!!
[MCU2_0] 29.963788 s: SCICLIENT: Sciclient_pmSetModuleState module=361 state=2
[MCU2_0] 29.963891 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0] 29.963914 s: VHWA: LDC Init ... !!!
[MCU2_0] 29.967086 s: VHWA: LDC Init ... Done !!!
[MCU2_0] 29.967129 s: VHWA: MSC Init ... !!!
[MCU2_0] 29.974481 s: VHWA: MSC Init ... Done !!!
[MCU2_0] 29.974520 s: VHWA: NF Init ... !!!
[MCU2_0] 29.975443 s: VHWA: NF Init ... Done !!!
[MCU2_0] 29.975480 s: VHWA: VISS Init ... !!!
[MCU2_0] 29.980767 s: VHWA: VISS Init ... Done !!!
[MCU2_0] 29.980810 s: VHWA: VPAC Init ... Done !!!
[MCU2_0] 29.980843 s: VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
[MCU2_0] 29.980869 s: VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
[MCU2_0] 29.980894 s: VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
[MCU2_0] 29.981860 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_sink on target MCU2-0
[MCU2_0] 29.981943 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_source on target MCU2-0
[MCU2_0] 29.982000 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_sink2 on target MCU2-0
[MCU2_0] 29.982054 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_source2 on target MCU2-0
[MCU2_0] 29.982110 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_intermediate on target MCU2-0
[MCU2_0] 29.982166 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.scalar_intermediate_2 on target MCU2-0
[MCU2_0] 29.982226 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.scalar_source_error on target MCU2-0
[MCU2_0] 29.982287 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.scalar_source_obj_array on target MCU2-0
[MCU2_0] 29.982348 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.scalar_sink_obj_array on target MCU2-0
[MCU2_0] 29.982405 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.pyramid_intermediate on target MCU2-0
[MCU2_0] 29.982464 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.pyramid_source on target MCU2-0
[MCU2_0] 29.982521 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.pyramid_sink on target MCU2-0
[MCU2_0] 29.982575 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.test_target on target MCU2-0
[MCU2_0] 29.982630 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.image_intermediate on target MCU2-0
[MCU2_0] 29.982690 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.cmd_timeout_test on target MCU2-0
[MCU2_0] 29.982747 s: VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.tiovx_overhead on target MCU2-0
[MCU2_0] 29.982879 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MCU2-0
[MCU2_0] 29.982963 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target VPAC_NF
[MCU2_0] 29.983053 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target VPAC_LDC1
[MCU2_0] 29.983133 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target VPAC_MSC1
[MCU2_0] 29.983225 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target VPAC_MSC2
[MCU2_0] 29.983366 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target VPAC_VISS1
[MCU2_0] 29.983465 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target CAPTURE1
[MCU2_0] 29.983561 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target CAPTURE2
[MCU2_0] 29.983656 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target CAPTURE3
[MCU2_0] 29.983756 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target CAPTURE4
[MCU2_0] 29.983840 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target CAPTURE5
[MCU2_0] 29.983926 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target CAPTURE6
[MCU2_0] 29.984025 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target CAPTURE7
[MCU2_0] 29.984123 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target CAPTURE8
[MCU2_0] 29.984219 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DISPLAY1
[MCU2_0] 29.984327 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DISPLAY2
[MCU2_0] 29.984418 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target CSITX
[MCU2_0] 29.984498 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target CSITX2
[MCU2_0] 29.984595 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DSS_M2M1
[MCU2_0] 29.984675 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DSS_M2M2
[MCU2_0] 29.984759 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DSS_M2M3
[MCU2_0] 29.984856 s: VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DSS_M2M4
[MCU2_0] 29.984893 s: VX_ZONE_INFO: [tivxInitLocal:126] Initialization Done !!!
[MCU2_0] 29.984921 s: VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
[MCU2_0] 29.984944 s: APP: OpenVX Target kernel init ... !!!
[MCU2_0] 29.992021 s: APP: OpenVX Target kernel init ... Done !!!
[MCU2_0] 29.992053 s: CSI2RX: Init ... !!!
[MCU2_0] 29.992070 s: SCICLIENT: Sciclient_pmSetModuleState module=136 state=2
[MCU2_0] 29.992147 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0] 29.992172 s: SCICLIENT: Sciclient_pmSetModuleState module=38 state=2
[MCU2_0] 29.992263 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0] 29.992286 s: SCICLIENT: Sciclient_pmSetModuleState module=39 state=2
[MCU2_0] 29.992370 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0] 29.992393 s: SCICLIENT: Sciclient_pmSetModuleState module=152 state=2
[MCU2_0] 29.992457 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0] 29.992480 s: SCICLIENT: Sciclient_pmSetModuleState module=153 state=2
[MCU2_0] 29.992537 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0] 29.992688 s: CSI2RX: Init ... Done !!!
[MCU2_0] 29.992715 s: CSI2TX: Init ... !!!
[MCU2_0] 29.992732 s: SCICLIENT: Sciclient_pmSetModuleState module=136 state=2
[MCU2_0] 29.992785 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0] 29.992809 s: SCICLIENT: Sciclient_pmSetModuleState module=40 state=2
[MCU2_0] 29.992880 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0] 29.992901 s: SCICLIENT: Sciclient_pmSetModuleState module=41 state=2
[MCU2_0] 29.992978 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0] 29.993000 s: SCICLIENT: Sciclient_pmSetModuleState module=363 state=2
[MCU2_0] 29.993071 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0] 29.993093 s: SCICLIENT: Sciclient_pmSetModuleState module=364 state=2
[MCU2_0] 29.993157 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0] 29.993223 s: CSI2TX: Init ... Done !!!
[MCU2_0] 29.993246 s: ISS: Init ... !!!
[MCU2_0] 29.993276 s: IssSensor_Init ... Done !!!
[MCU2_0] 29.993332 s: IttRemoteServer_Init ... Done !!!
[MCU2_0] 29.993353 s: ISS: Init ... Done !!!
[MCU2_0] 29.993374 s: VISS REMOTE SERVICE: Init ... !!!
[MCU2_0] 29.993419 s: VISS REMOTE SERVICE: Init ... Done !!!
[MCU2_0] 29.993441 s: UDMA Copy: Init ... !!!
[MCU2_0] 29.994325 s: UDMA Copy: Init ... Done !!!
[MCU2_0] 29.994362 s: APP: Init ... Done !!!
[MCU2_0] 29.994382 s: APP: Run ... !!!
[MCU2_0] 29.994398 s: IPC: Starting echo test ...
[MCU2_0] 29.994517 s: APP: Run ... Done !!!
[MCU2_0] 29.995226 s: IPC: Echo status: mpu1_0[x] mcu2_0[s] mcu2_1[P] C7X_1[.] C7X_2[.]
[MCU2_0] 29.995284 s: IPC: Echo status: mpu1_0[x] mcu2_0[s] mcu2_1[P] C7X_1[P] C7X_2[.]
[MCU2_0] 29.995326 s: IPC: Echo status: mpu1_0[x] mcu2_0[s] mcu2_1[P] C7X_1[P] C7X_2[P]

I have to explain that I deactivate the dispay stuff in processor sdk with BUILD_DISPLAY=no and BUILD_DISPLAY_M2M=no (and additional BUILD_CAPTURE=no) in order to get the firmware running on MCU2_0. With it compiled in the firmware hangs during initialisation of the display.

What could be the reason that I'm unable to get the CSITX node?

With regards,

Frank

I have to correct me: We disabled the display stuff in a header file.

When I call tivxAddKernelCsitx the CSITX node seems to be available. I will check if I can get it working.

Regards,

Frank

Hi Frank,

Please let me know if that resolves your issue!

Thanks,

Fabiana