• State Resolved
  • Locked Locked
  • Replies 7 replies
  • Subscribers 98 subscribers
  • Views 556 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5728: IPC example not working

Jon Lake
Prodigy 170 points
Part Number: AM5728

Hi, I'm having trouble getting the ex02_messageq example running on any of the non-A15 processors. I've tried DSP1/2 and IPU1/2 and none work, but they fail in different ways depending on the processor type.

I followed the steps in the RTOS Developer Guide to build the DRA7XX_linux_elf examples, and I'm seeing the files I expect in the directory structure (e.g. dsp1/bin/debug/server_dsp1.xe66, ipu1/bin/debug/server_ipu1.xem4, and host/bin/debug/app_host). 

I can successfully create a symbolic link to the file (ln -s /home/root/ipc_example/ex02_messageq/dsp1/bin/debug/server_dsp1.xe66 dra7-dsp1-fw.xe66), but I get some errors when I try to unbind/bind:

root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# echo 40800000.dsp > unbind
[ 114.418808] remoteproc2: releasing 40800000.dspc# echo 40800000.dsp > unbind
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# echo 40800000.dsp > bind
[ 125.098719] omap-rproc 40800000.dsp: assigned reserved memory node dsp1_cma@99000000
[ 125.106555] remoteproc2: 40800000.dsp is available
[ 125.111484] remoteproc2: Note: remoteproc is still under development and considered experimental.
[ 125.120544] remoteproc2: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# [ 125.138657] remoteproc2: powering up 40800000.dsp
[ 125.146967] remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4417084
[ 125.161087] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[ 125.167017] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[ 125.172920] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
[ 125.183777] remoteproc2: bad phdr da 0x800000 mem 0x40
[ 125.189068] remoteproc2: Failed to load program segments: -22
[ 125.195134] remoteproc2: failed to unmap 1048576/0
[ 125.209712] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
[ 125.222691] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[ 125.228594] remoteproc2: rproc_boot() failed -22
[ 125.233357] virtio_rpmsg_bus: probe of virtio1 failed with error -22
[ 125.239772] remoteproc2: registered virtio1 (type 7)

When I run app_host, it hangs after App_create:

root@am57xx-evm:~/ipc_example/ex02_messageq/host/bin/debug# ./app_host DSP1
--> main:
[ 243.618837] NET: Registered protocol family 41
--> Main_main:
--> App_create:

If I try doing the same for IPU1, the unbind/bind output looks healthier:

root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# echo 58820000.ipu > unbind
[ 418.808133] omap_hwmod: mmu_ipu1: _wait_target_disable failed000.ipu > unbind
[ 418.814003] remoteproc0: stopped remote processor 58820000.ipu
[ 418.820136] remoteproc0: releasing 58820000.ipu
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# echo 58820000.ipu > bind
[ 434.198773] omap-rproc 58820000.ipu: assigned reserved memory node ipu1_cma@9d000000
[ 434.206614] remoteproc0: 58820000.ipu is available
[ 434.211797] remoteproc0: Note: remoteproc is still under development and considered experimental.
[ 434.220833] remoteproc0: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# [ 434.433142] remoteproc0: powering up 58820000.ipu
[ 434.438014] remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 4249708
[ 434.445203] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1
[ 434.455896] remoteproc0: remote processor 58820000.ipu is now up
[ 434.463044] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 434.468609] remoteproc0: registered virtio0 (type 7)
[ 434.474324] virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x3d

However, the app_host program now hangs in a different spot:

root@am57xx-evm:~/ipc_example/ex02_messageq/host/bin/debug# ./app_host IPU1
--> main:
--> Main_main:
--> App_create:
App_create: Host is ready
<-- App_create:
--> App_exec:
App_exec: sending message 1
App_exec: sending message 2
App_exec: sending message 3

Am I missing something in my setup? I don't see anything else in the IPC section of the developer guide.

For reference, I'm using Processor SDK 06.03.00.106 on the AM572x EVM, with IPC 3.50.04.08.

Thanks

  • 0
    TI__Guru** 113175 points

    Hi Jon,

    This behavior looks very similar to the behavior being reported on another thread,

    https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1021254/am5728-ipc-ex02messageq-run-error-requested-size-is-too-big

    Can you please try using the pre-built images first? Please follow the same steps as I outlined in that thread.

    regards

    Suman 

  • 0 in reply to Suman Anna
    Prodigy 170 points

    Hi Suman,

    I tried using the pre-built images according to your other posts and still run into problems. Here is my console output:

    modprobe -r omap_remoteproc
    modprobe: FATAL: Module omap_remoteproc is in use.
    root@am57xx-evm:~# lsmod | grep remote
    omap_remoteproc 7509 1
    remoteproc 27139 1 omap_remoteproc
    virtio 7364 2 remoteproc,virtio_rpmsg_bus
    virtio_ring 11848 2 remoteproc,virtio_rpmsg_bus
    root@am57xx-evm:~# modprobe -r virtio_rpmsg_bus
    [ 361.181741] omap_hwmod: mmu_ipu1: _wait_target_disable failed
    [ 361.187583] remoteproc0: stopped remote processor 58820000.ipu
    root@am57xx-evm:~# modprobe -r omap_remoteproc
    [ 370.552366] remoteproc3: releasing 41000000.dsp
    [ 370.557290] remoteproc2: releasing 40800000.dsp
    [ 370.562307] remoteproc1: releasing 55020000.ipu
    [ 370.567141] remoteproc0: releasing 58820000.ipu
    root@am57xx-evm:~# ls -l /lib/firmware | grep lrw
    lrwxrwxrwx 1 root root 49 Jan 28 20:53 am57xx-pru1_0-fw -> /lib/firmware/pru/PRU_RPMsg_Echo_Interrupt1_0.out
    lrwxrwxrwx 1 root root 49 Jan 28 20:53 am57xx-pru1_1-fw -> /lib/firmware/pru/PRU_RPMsg_Echo_Interrupt1_1.out
    lrwxrwxrwx 1 root root 49 Jan 28 20:53 am57xx-pru2_0-fw -> /lib/firmware/pru/PRU_RPMsg_Echo_Interrupt2_0.out
    lrwxrwxrwx 1 root root 49 Jan 28 20:53 am57xx-pru2_1-fw -> /lib/firmware/pru/PRU_RPMsg_Echo_Interrupt2_1.out
    lrwxrwxrwx 1 root root 71 Jan 29 2016 dra7-dsp1-fw.xe66 -> /home/root/ipc_example/pre-built/ex02_messageq/release/server_dsp6
    lrwxrwxrwx 1 root root 71 Jan 29 2016 dra7-dsp2-fw.xe66 -> /home/root/ipc_example/pre-built/ex02_messageq/release/server_dsp6
    lrwxrwxrwx 1 root root 71 Jan 29 2016 dra7-ipu1-fw.xem4 -> /home/root/ipc_example/pre-built/ex02_messageq/release/server_ipu4
    lrwxrwxrwx 1 root root 71 Jan 29 2016 dra7-ipu2-fw.xem4 -> /home/root/ipc_example/pre-built/ex02_messageq/release/server-ipu4
    lrwxrwxrwx 1 root root 46 Jan 28 20:53 ducati-m3-core0.xem3 -> /lib/firmware/ducati-m3-core0.xem3.ipc-test-fw
    lrwxrwxrwx 1 root root 41 Jan 28 20:53 tesla-dsp.xe64T -> /lib/firmware/tesla-dsp.xe64T.ipc-test-fw
    root@am57xx-evm:~# modprobe omap_remoteproc
    [ 428.256655] omap-rproc 58820000.ipu: assigned reserved memory node ipu1_cma@9d000000
    [ 428.264528] remoteproc0: 58820000.ipu is available
    [ 428.269671] remoteproc0: Note: remoteproc is still under development and considered experimental.
    [ 428.278696] remoteproc0: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
    [ 428.293862] remoteproc0: registered virtio0 (type 7)
    [ 428.299208] omap-rproc 55020000.ipu: assigned reserved memory node ipu2_cma@95800000
    [ 428.307045] remoteproc1: 55020000.ipu is available
    [ 428.313032] remoteproc1: Note: remoteproc is still under development and considered experimental.
    [ 428.322090] remoteproc1: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
    [ 428.332569] remoteproc0: powering up 58820000.ipu
    [ 428.332858] remoteproc1: Direct firmware load for dra7-ipu2-fw.xem4 failed with error -2
    [ 428.332862] remoteproc1: Falling back to user helper
    [ 428.333138] omap-rproc 40800000.dsp: assigned reserved memory node dsp1_cma@99000000
    root@am57xx-evm:~# [ 428.333188] remoteproc2: 40800000.dsp is available
    [ 428.333192] remoteproc2: Note: remoteproc is still under development and considered experimental.
    [ 428.333195] remoteproc2: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
    [ 428.339789] omap-rproc 41000000.dsp: assigned reserved memory node dsp2_cma@9f000000
    [ 428.339854] remoteproc3: 41000000.dsp is available
    [ 428.339857] remoteproc3: Note: remoteproc is still under development and considered experimental.
    [ 428.339860] remoteproc3: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
    [ 428.384527] remoteproc2: powering up 40800000.dsp
    [ 428.384537] remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4547276
    [ 428.391624] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
    [ 428.391680] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
    [ 428.391718] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
    [ 428.423338] remoteproc2: bad phdr da 0x800000 mem 0x40
    [ 428.423342] remoteproc2: Failed to load program segments: -22
    [ 428.423587] remoteproc2: failed to unmap 1048576/0
    [ 428.458373] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
    [ 428.465393] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
    [ 428.465452] remoteproc2: rproc_boot() failed -22
    [ 428.473700] virtio_rpmsg_bus: probe of virtio1 failed with error -22
    [ 428.473711] remoteproc2: registered virtio1 (type 7)
    [ 428.489967] remoteproc3: powering up 41000000.dsp
    [ 428.494934] remoteproc1: failed to load dra7-ipu2-fw.xem4
    [ 428.499718] remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 4364400
    [ 428.499839] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1
    [ 428.517069] remoteproc0: remote processor 58820000.ipu is now up
    [ 428.519135] remoteproc3: Booting fw image dra7-dsp2-fw.xe66, size 4418876
    [ 428.525791] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
    [ 428.525828] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
    [ 428.525863] omap-iommu 41502000.mmu: 41502000.mmu: version 3.0
    [ 428.552714] remoteproc3: bad phdr da 0x800000 mem 0x40
    [ 428.557963] remoteproc3: Failed to load program segments: -22
    [ 428.564054] remoteproc3: failed to unmap 1048576/0
    [ 428.577867] omap_hwmod: mmu1_dsp2: _wait_target_disable failed
    [ 428.590346] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
    [ 428.596250] remoteproc3: rproc_boot() failed -22
    [ 428.601034] virtio_rpmsg_bus: probe of virtio2 failed with error -22
    [ 428.607422] remoteproc3: registered virtio2 (type 7)
    [ 428.613743] virtio_rpmsg_bus virtio0: rpmsg host is online
    [ 428.614073] virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x3d
    [ 428.627290] remoteproc2: powering up 40800000.dsp
    [ 428.636513] remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4547276
    [ 428.650233] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
    [ 428.656155] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
    [ 428.662054] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
    [ 428.672518] remoteproc2: bad phdr da 0x800000 mem 0x40
    [ 428.677767] remoteproc2: Failed to load program segments: -22
    [ 428.683953] remoteproc2: failed to unmap 1048576/0
    [ 428.697897] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
    [ 428.710422] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
    [ 428.717144] remoteproc2: rproc_boot() failed -22
    [ 428.721918] virtio_rpmsg_bus: probe of virtio1 failed with error -22
    [ 428.729015] remoteproc3: powering up 41000000.dsp
    [ 428.737362] remoteproc3: Booting fw image dra7-dsp2-fw.xe66, size 4418876
    [ 428.750983] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
    [ 428.756888] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
    [ 428.762779] omap-iommu 41502000.mmu: 41502000.mmu: version 3.0
    [ 428.772267] remoteproc3: bad phdr da 0x800000 mem 0x40
    [ 428.777511] remoteproc3: Failed to load program segments: -22
    [ 428.783570] remoteproc3: failed to unmap 1048576/0
    [ 428.796992] omap_hwmod: mmu1_dsp2: _wait_target_disable failed
    [ 428.809376] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
    [ 428.815740] remoteproc3: rproc_boot() failed -22
    [ 428.820499] virtio_rpmsg_bus: probe of virtio2 failed with error -22
    DSP1root/ipc_example/pre-built/ex02_messageq/debug/app_host
    --> main:
    /home/root/ipc_example/pre-built/ex02_messageq/debug/app_host: symbol lookup error: /usr/lib/libtitransportrpmsg.so.1: undefined sye
    IPU1m57xx-evm:~# /home/root/ipc_example/pre-built/ex02_messageq/debug/app_host
    --> main:
    /home/root/ipc_example/pre-built/ex02_messageq/debug/app_host: symbol lookup error: /usr/lib/libtitransportrpmsg.so.1: undefined sye
    root@am57xx-evm:~#

    I re-ran app_host to show the error since it got cut off in my console. This is what it said:

    /home/root/ipc_example/pre-built/ex02_messageq/debug/app_host: symbol lookup error: /usr/lib/libtitransportrpmsg.so.1: undefined symbol: pthread_create

  • 0 in reply to Jon Lake
    TI__Guru** 113175 points

    Hi Jon,

    You have two issues in your log. First one is the load failure on DSPs (remoteproc3: bad phdr da 0x800000 mem 0x40) and the second one is the undefined symbol error. Very strange to see the first one.

    Please do run "ldconfig" for the second issue if you have built a new executable and copied it to the FS.

    The examples are working fine on my end (I had used the images from the release folder rather than debug), so am not sure what is different between our setups. Please give the exact kernel and U-Boot versions so that I can make sure there are no differences in either of them.

    regards

    Suman

  • 0 in reply to Suman Anna
    Prodigy 170 points

    Hi Suman,

    Here's what I have for the Linux and U-Boot versions:

    Linux version 4.1.13-g8dc6617 (gtbldadm@ubuntu-12)

    U-Boot SPL 2015.07 (Jan 28 2016 - 14:19:33)

    I'm new to using Linux and U-Boot so let me know if these weren't the right numbers.

    For the second issue, I'm using the app_host program that came pre-compiled. If I run the one I compiled myself, it doesn't have the linker issue for finding pthread_create.

    -Jon

  • 0 in reply to Jon Lake
    TI__Guru** 113175 points

    Hi Jon,

    You have mentioned you are using Processor SDK 06.03.00.106 on the AM572x EVM, with IPC 3.50.04.08 in your first message. But the U-Boot and Kernel versions you mentioned above are actually much older than the the 6.3 SDK. I would have expected to see 4.19.x kernel and a 2019.01 U-Boot.

    It does sound like you are also using your built or a different FS. I would have expected the exact opposite behavior. The pre-built binary with the given FS should all have been linked properly.

    These are the versions I expect with 6.3 SDK

    U-Boot 2019.01-g333c3e72d3 (Apr 20 2020 - 04:15:00)

    Linux version 4.19.94-gbe5389fd85

    regards

    Suman

  • +1 in reply to Suman Anna
    TI__Guru** 113175 points

    Hi Jon,

    I confirm that the kernel version "Linux version 4.1.13-g8dc6617 (gtbldadm@ubuntu-12)" that you are using doesn't have the support for loading into DSP internal memories. And if you are trying the latest 6.3 SDK firmwares with this kernel, they won't work. Please use the latest 6.3 SDK and see if you are still running into issues. 

    regards

    Suman

  • 0 in reply to Suman Anna
    Prodigy 170 points

    Hi Suman,

    I downloaded the image from the SDK and put it on the SD card, and the example is working now from my rebuilt executables. The kernel version in the image is 4.19.94-gbe5389fd85, which is many revisions newer than the one that came on the SD card packaged with the EVM.

    Thanks for the help!

    -Jon