• State Resolved
  • Locked Locked
  • Replies 6 replies
  • Subscribers 98 subscribers
  • Views 474 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

PROCESSOR-SDK-AM68A: DMA memory problem on HS-SE device

Denys Datsko
Prodigy 20 points
Part Number: PROCESSOR-SDK-AM68A
Other Parts Discussed in Thread: AM68A

Tool/software:

Hello

    I have a custom board with AM68A chip and a custom built Yocto image that is based on the one provided by TI. I am now in the process of movement the build from Processor SDK of version 8.6 to the newest version 10.1. I am using U-boot of version 2023.04 with the ti-linux-firmware of version 10.01.06 (also tried with 10.00.08 and result is the same) and Linux kernel 6.6.32 on an HS-FS device, and am experiencing a problem that when I try to use anything of vision_apps, it mostly doesn't work as expected.

For example, when I run vx_app_arm_mem.out, it just hangs after the following output:
APP: Init ... !!!
  3224.738012 s: MEM: Init ... !!!
  3224.738298 s: MEM: Initialized DMA HEAP (fd=5) !!!
  3224.738465 s: MEM: Init ... Done !!!
  3224.738488 s: IPC: Init ... !!!
  3224.776023 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  3224.784637 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!



When I run it with strace, the last lines are as follows:
write(1, "APP: Init ... Done !!!\n", 23APP: Init ... Done !!!
) = 23
ioctl(5, DMA_HEAP_IOCTL_ALLOC, 0xfffffb2a44a8) = 0
mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_SHARED, 23, 0) = 0xffff70c8d000
openat(AT_FDCWD, "/dev/remoteproc0", O_RDONLY|O_CLOEXEC) = 24
ioctl(24, _IOC(_IOC_READ|_IOC_WRITE, 0xb7, 0, 0x10), 0xfffffb2a4460) = ?


At the same time, the following message is displayed in dmesg:
[ 3248.253908] Unable to handle kernel paging request at virtual address ffff0008800b0000
[ 3248.261899] Mem abort info:
[ 3248.264708]   ESR = 0x0000000096000145
[ 3248.268472]   EC = 0x25: DABT (current EL), IL = 32 bits
[ 3248.273794]   SET = 0, FnV = 0
[ 3248.276853]   EA = 0, S1PTW = 0
[ 3248.279994]   FSC = 0x05: level 1 translation fault
[ 3248.284877] Data abort info:
[ 3248.287752]   ISV = 0, ISS = 0x00000145, ISS2 = 0x00000000
[ 3248.293235]   CM = 1, WnR = 1, TnD = 0, TagAccess = 0
[ 3248.298276]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 3248.303581] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000082ff6000
[ 3248.310271] [ffff0008800b0000] pgd=18000008ffff8003, p4d=18000008ffff8003, pud=0000000000000000
[ 3248.318963] Internal error: Oops: 0000000096000145 [#8] PREEMPT SMP
[ 3248.325214] Modules linked in: xt_conntrack xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_addrtype iptable_filter ip_tables x_tables br_netfilter bridge overlay cfg80211 rfkill 8021q garp mrp stp llc r8153_ecm cdc_ether usbnet r8152 mii rpmsg_ctrl rpmsg_char cdns_csi2rx cdns3 cdns_usb_common crct10dif_ce pwm_fan wave5 j721e_csi2rx at24 k3_j72xx_bandgap arducam(O) videobuf2_dma_contig v4l2_mem2mem videobuf2_memops v4l2_fwnode videobuf2_v4l2 videobuf2_common v4l2_async videodev ti_k3_dsp_remoteproc ti_k3_r5_remoteproc mc sa2ul rtc_rv3028 cdns3_ti pwm_tiehrpwm cdns_dphy_rx rti_wdt fuse drm drm_panel_orientation_quirks backlight ipv6
[ 3248.384701] CPU: 1 PID: 8457 Comm: vx_app_arm_mem. Tainted: G      D W  O       6.6.32-01389-g186e9c935e32 #1
[ 3248.394591] Hardware name: ... (DT)
[ 3248.399021] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 3248.405963] pc : dcache_clean_poc+0x20/0x38
[ 3248.410141] lr : arch_sync_dma_for_device+0x24/0x30
[ 3248.415004] sp : ffff800097e43bd0
[ 3248.418305] x29: ffff800097e43bd0 x28: 00000009000b0000 x27: 0000000000020000
[ 3248.425424] x26: ffff00082f66c5a0 x25: ffff00082da17838 x24: 0000000000000000
[ 3248.432542] x23: fffffc0000000000 x22: ffff800080f843c0 x21: 0000000000000000
[ 3248.439660] x20: 0000000000000001 x19: 0000000000000000 x18: 0000000000000000
[ 3248.446778] x17: 3461326266666666 x16: 667830202c293031 x15: 0000fffffb2a4460
[ 3248.453896] x14: 627830202c455449 x13: 0000000000000090 x12: 0000000000000000
[ 3248.461014] x11: 0000000000000000 x10: 00000000000009a0 x9 : ffff8000813c6968
[ 3248.468132] x8 : ffff00082f66c5c0 x7 : 0000000000000002 x6 : 0000000000000000
[ 3248.475249] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 000000000000003f
[ 3248.482367] x2 : 0000000000000040 x1 : ffff0008800d0000 x0 : ffff0008800b0000
[ 3248.489486] Call trace:
[ 3248.491921]  dcache_clean_poc+0x20/0x38
[ 3248.495745]  dma_direct_map_sg+0x280/0x2e4
[ 3248.499829]  __dma_map_sg_attrs+0x28/0x94
[ 3248.503825]  dma_map_sg_attrs+0x10/0x24
[ 3248.507647]  dma_heap_map_dma_buf+0x40/0x5c
[ 3248.511818]  __map_dma_buf+0x2c/0xa0
[ 3248.515383]  dma_buf_map_attachment+0x9c/0x108
[ 3248.519812]  rproc_attach_dmabuf+0xcc/0x1b4
[ 3248.523984]  rproc_device_ioctl+0x124/0x2b4
[ 3248.528154]  __arm64_sys_ioctl+0xac/0xf0
[ 3248.532065]  invoke_syscall+0x48/0x114
[ 3248.535803]  el0_svc_common.constprop.0+0xc0/0xe0
[ 3248.540492]  do_el0_svc+0x1c/0x28
[ 3248.543794]  el0_svc+0x2c/0x84
[ 3248.546839]  el0t_64_sync_handler+0x120/0x12c
[ 3248.551182]  el0t_64_sync+0x190/0x194
[ 3248.554834] Code: d2800082 9ac32042 d1000443 8a230000 (d50b7a20)
[ 3248.560909] ---[ end trace 0000000000000000 ]---


For a test, I tried using the image for an HS-FS device, provided by TI (the .wic one for j721s2-EVM) and completely replacing the U-boot there with the custom built one, and on that device all the vision_apps work as expected without any errors. I am using the same kernel with modules and devicetree and have verified that all the TI libraries needed by vision_apps are of exactly same version, as well as the firmware in /lib/firmware (I may have missed something important, however). Moreover, the output of vx_app_heap_stats.out  and vx_app_arm_remote_log.out is identical on both devices (up to timestamps), indicating that all the remote processors started successfully.

I would be grateful for any hints on what can be causing the problem on the HS-FS device with custom yocto build and how to fix it

  • 0
    TI__Genius 17805 points

    Hi Denys,

    Between the SDK releases, there are changes between the ti-linux-firmware and TIDL versions. These versions are largely dependent on each other, and if want TIDL to work, you will likely have to migrate to the correct version of the ti-linux-firmware.

    Please let me know if you have any other questions.

    Best,
    Jared

  • 0 in reply to Jared McArthur
    Prodigy 20 points

    Hi Jared,

    Thank you for the quick reply! I understand this dependence, and I am already using the ti-linux-firmware of version 10.01.06 (I also tried 10.00.08 at some point, and the result was the same).

    I basically have two devices: HS-SE with custom rootfs, and HS-FS with rootfs taken from the provided .wic image for j721s2-EVM (with u-boot, Linux kernel and devicetree changed), and on both I have the same ti-linux-firmware (as reported by both u-boot and in ./vx_app_arm_remote_log.out) and identical TIDL libraries. In such configuration, HS-FS device runs all the vision_apps properly.

    My thought is that I have missed some other important part of the system in custom yocto during migration to the newer SDK, but I couldn't find the problem myself, unfortunately

    Regards,

    Denys

  • 0 in reply to Denys Datsko
    TI__Genius 17805 points

    Hi Denys,

    Just reading the trace, it seems that there may be a difference in the memory maps between the two devices.

    Does the HS-FS device with custom rootfs run properly?

    Did the 8.6 version run properly?

    Best,
    Jared

  • 0 in reply to Jared McArthur
    Prodigy 20 points

    Hi Jared,

    I am using the default memory map from TI (defined in k3-j721s2-edgeai-apps.dtbo) on both devices. And yes, the HS-SE device worked well with SDK 8.6. I also completely forgot to mention that my testing HS-FS device has 8GB of RAM, while the HS-SE device has only 4GB, and this seems to be quite important now.

    I was looking into the issue more deeply, and what came to my attention is that vision_apps_shared_region node is located on different address in SDK 10:

    vision_apps_shared_region: vision_apps_shared-memories {
        compatible = "dma-heap-carveout";
        reg = <0x09 0x00000000 0x00 0x20000000>;
    };

    While in SDK 8.6 it was reg = <0x00 0xb8000000 0x00 0x20000000>;

    Do I understand right that with reg = <0x09 0x00000000 0x00 0x20000000>; the region now lies beyond the available 4GB of RAM, or is this something completely different? Anyway, are there any more changes needed to switch to a device with 4GB of RAM (I already have this configured in u-boot, btw), and am I missing something else?

    There also is linux_cma_region node in the devicetree lying beyond the available 4GB, but is it not used by the kernel:
    Reserved memory: bypass linux-cma-buffers@980000000 node, using cmdline CMA params instead

    Regards,
    Denys

  • 0 in reply to Denys Datsko
    TI__Genius 17805 points

    Hi Denys,

    Yes, this is an issue. I've run into the same issue with another custom board, but we haven't resolved it.

    I've filed a bug internally to try and add support for 4GB RAM, but as it stands currently, I don't have a solution to give you.

    You can systematically go through and edit all of the memory mapping, but I assume that would be a large undertaking.

    Best,
    Jared

  • +1 in reply to Jared McArthur
    Prodigy 20 points

    Hi Jared,

    Thanks for the answer. As I mentioned, I have noticed that only the vision_apps_shared-memories node (as I don't use CMA settings from devicetree) had the memory mapped outside of the 4GB, so I tried to change it.

    I have downloaded the RTOS SDK of version 10.01.00.04 and went through the guide from the documentation on how to change the memory map: https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-j721s2/10_01_00_04/exports/docs/psdk_rtos/docs/user_guide/developer_notes_memory_map.html. I have changed the physical address of shared memory by modifying this variable:
    ddr_shared_mem_addr_phys  = 0x8c0000000

    After I updated the memory map in the devicetree and rebuilt and replaced the firmware for remote processors, I can now successfully run apps like vx_app_arm_mem.out, and they finish successfully.

    However, the is another serious problem here: for some reason the firmware for MCU2_0 differs a lot from what was provided in the .wic image by TI.
    This is firstly noticable with file sizes. While generated vx_app_rtos_linux_c7x_1.out and vx_app_rtos_linux_c7x_2.out have exactly same size as old ones in .wic image, and vx_app_rtos_linux_mcu2_1.out size differs by a couple of bytes, the generated vx_app_rtos_linux_mcu2_0.out has the size of 932KB compared to 484KB before.
    But the biggest problem is that this MCU does not start well. For some reason, it starts initializing DSS and fails there. Here's the output from that MCU:


    [MCU2_0]s: CIO: Init ... Done !!!
[MCU2_0]s: ### CPU Frequency = 1000000000 Hz
[MCU2_0]s: CPU is running FreeRTOS
[MCU2_0]s: APP: Init ... !!!
[MCU2_0]s: SCICLIENT: Init ... !!!
[MCU2_0]s: SCICLIENT: DMSC FW version [10.0.8--v10.00.08 (Fiery Fox)]
[MCU2_0]s: SCICLIENT: DMSC FW revision 0xa  
[MCU2_0]s: SCICLIENT: DMSC FW ABI revision 4.0
[MCU2_0]s: SCICLIENT: Init ... Done !!!
[MCU2_0]s: UDMA: Init ... !!!
[MCU2_0]s: UDMA: Init ... Done !!!
[MCU2_0]s: UDMA: Init for CSITX/CSIRX ... !!!
[MCU2_0]s: UDMA: Init for CSITX/CSIRX ... Done !!!
[MCU2_0]s: MEM: Init ... !!!
[MCU2_0]s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ b9000000 of size 14680064 bytes !!!
[MCU2_0]s: MEM: Created heap (L3_MEM, id=1, flags=0x00000000) @ 60000000 of size 524288 bytes !!!
[MCU2_0]s: MEM: Created heap (DDR_CACHE_WT_MEM, id=7, flags=0x00000000) @ b9e00000 of size 2097152 bytes !!!
[MCU2_0]s: MEM: Init ... Done !!!
[MCU2_0]s: IPC: Init ... !!!
[MCU2_0]s: IPC: 5 CPUs participating in IPC !!!
[MCU2_0]s: IPC: Waiting for HLOS to be ready ... !!!
[MCU2_0]s: IPC: HLOS is ready !!!
[MCU2_0]s: IPC: Init ... Done !!!
[MCU2_0]s: APP: Syncing with 4 CPUs ... !!!
[MCU2_0]s: APP: Syncing with 4 CPUs ... Done !!!
[MCU2_0]s: REMOTE_SERVICE: Init ... !!!
[MCU2_0]s: REMOTE_SERVICE: Init ... Done !!!
[MCU2_0]s: FVID2: Init ... !!!
[MCU2_0]s: FVID2: Init ... Done !!!
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState module=219 state=2
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0]s: DSS: Init ... !!!
[MCU2_0]s: DSS: Display type is eDP !!!
[MCU2_0]s: DSS: M2M Path is enabled !!!
[MCU2_0]s: DSS: SoC init ... !!!
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState module=158 state=0
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState module=365 state=2
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState module=156 state=2
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState module=156 state=2
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState module=158 state=0
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleClkFreq module=158 clk=3 freq=148500000
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleClkFreq success
[MCU2_0]s: SCICLIENT: Sciclient_pmModuleClkRequest module=158 clk=3 state=2 flag=2
[MCU2_0]s: SCICLIENT: Sciclient_pmModuleClkRequest success
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState module=158 state=2
[MCU2_0]s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU2_0]s: DSS: SoC init ... Done !!!
[MCU2_0]s: DSS: Board init ... !!!
[MCU2_0]s: DSS: Turning on DP_PWR pin for eDP adapters ... !!!
[MCU2_0]s: DSS: ERROR: Turning on DP_PWR pin for eDP adapters failed !!!
[MCU2_0]s: DSS: Board init ... Done !!!


    And the MCU2_0 status does not get to P in the end:
    [C7x_2 ]s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[P] C7X_1[P] C7X_2[s]

    With this, I cannot include TI plugins (like TIOVXISP) in my gstreamer pipeline (it just hangs). Did I miss some steps to build the same firmware as it is provided prebuilt by TI? Oh, and does setting physical address for DMA memory as 0x8c0000000 looks good to you? It seems to me that the region does not overlap with any other one, but I may have missed something.

    Regards,
    Denys



    UPD: this was again my fault. There is a note in the documentation saying that
    The default setting of these scripts are building the firmware for the <sd-card-rootfs>/lib/firmware/vision_apps_eaik version of the firmware which is designed for use on the small form factor starter
kits, not the larger EVMs. If you are building the firmware for the larger EVM mentioned elsewhere in this user guide, then you will need to make the following changes in the 
${FIRMWARE_BUILDER_INSTALL_PATH}/sdk_builder/makerules/makefile_linux_arm.mak file: Change the following variable in FIRMWARE_VARS list: BUILD_EDGEAI=no

    but this appears to be not true. By default without any modifications it built firmware for vision_apps_evm and I didn't notice that. After changing the TISDK_IMAGE to edgeai in sdk_builder/build_flags.mak the correct firmware was built and MCU2_0 now starts as expected. I have also tested this with some models and they work