• State Resolved
  • Locked Locked
  • Replies 10 replies
  • Answers 2 answers
  • Subscribers 98 subscribers
  • Views 2450 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

RTOS/AM5728: AM57x ARM-DSP IPC communication example crash

db
Intellectual 415 points

Part Number: AM5728

Tool/software: TI-RTOS

Hi,

I am trying to run the ex02_messageq example on a phytec AM57x demo board (AM5728).

I successfully built the examples by following the documentation and produced these binaries:

ipc_3_50_03_04/examples/DRA7XX_linux_elf/ex02_messageq/dsp1/bin/debug/server_dsp1.xe66

ipc_3_50_03_04/examples/DRA7XX_linux_elf/ex02_messageq/host/bin/release/app_host

After that I copyied these binaries to the device /lib/firmware.


# ls -l /lib/firmware/dra7-dsp1-fw.xe66
lrwxrwxrwx 1 root root 32 Mar 12 15:54 /lib/firmware/dra7-dsp1-fw.xe66 -> /home/root/demo/server_dsp1.xe66

# for i in `seq 0 1 3` ; do echo -n "remoteproc$i: " ; cat /sys/class/remoteproc/remoteproc$i/firmware ; done remoteproc0: dra7-ipu1-fw.xem4 remoteproc1: dra7-ipu2-fw.xem4 remoteproc2: dra7-dsp1-fw.xe66 remoteproc3: dra7-dsp2-fw.xe66

I do a DSP1 restart with the following so that it boots the new firmware:

echo stop > /sys/class/remoteproc/remoteproc2/state echo start > /sys/class/remoteproc/remoteproc2/state


And then start the linux app.
Linux app sends 3 messages and then waits for a reply. This reply never comes and remoteproc detects a crash in DSP1 and restarts it.
Now the big question, WHY ?

# ./app_host DSP1
--> main:
[ 26.848432] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
[ 26.854336] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
[ 26.862132] omap-iommu 41502000.mmu: 41502000.mmu: version 3.0
--> Main_main:
--> App_create:
App_create: Host is ready
<-- App_create:
--> App_exec:
App_exec: sending message 1
App_exec: sending message 2
App_exec: sending message 3
[ 32.672900] remoteproc remoteproc2: crash detected in 40800000.dsp: type watchdog
[ 32.680462] remoteproc remoteproc2: handling crash #1 in 40800000.dsp
[ 32.686939] remoteproc remoteproc2: recovering 40800000.dsp
recvfrom failed: Link has been severed (67)
rpmsgThreadFxn: transportGet failed on fd 13, returned -20
<-- App_exec: -20
<-- Main_main:
<-- main:
[ 32.713563] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
[ 32.726490] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
root@am5728-phycore-rdk:~/demo# [ 32.732746] remoteproc remoteproc2: stopped remote processor 40800000.dsp
[ 32.742274] remoteproc remoteproc2: powering up 40800000.dsp
[ 32.754445] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4469652
[ 32.769392] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[ 32.775293] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[ 32.781239] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
[ 32.798622] virtio_rpmsg_bus virtio1: rpmsg host is online
[ 32.804178] remoteproc remoteproc2: registered virtio1 (type 7)
[ 32.810184] remoteproc remoteproc2: remote processor 40800000.dsp is now up
[ 32.817853] virtio_rpmsg_bus virtio1: creating channel rpmsg-proto addr 0x3d



  • 0
    TI__Guru 50170 points
    Hi, db,

    Not sure why you are getting the crash. I suspect it is related to the build. Did you do the high level build in Linux SDK? Which documentation did you follow? Is it the IPC Quick Start Guide, software-dl.ti.com/.../Foundational_Components_IPC.html ?

    I don't think Phytec is the issue, but you can try the ex02_messageq prebuilt images in /usr/bin/ipc/example/ex02_messageq/debug, and see if that works for you. Below are my console logs from TI AM5728 GP EVM using exact the same commands as yours and also server_dsp1.xe66 from debug folder and app_host from release folder:

    root@am57xx-evm:/lib/firmware# ln -sf /usr/bin/ipc/examples/ex02_messageq/debug/server_dsp1.xe66 dra7-dsp1-fw.xe66
    root@am57xx-evm:/lib/firmware# echo stop > /sys/class/remoteproc/remoteproc2/state
    [ 338.508492] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
    [ 338.514393] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
    [ 338.521347] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
    [ 338.538408] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
    [ 338.544336] remoteproc remoteproc2: stopped remote processor 40800000.dsp
    root@am57xx-evm:/lib/firmware# echo start > /sys/class/remoteproc/remoteproc2/state
    [ 348.291039] remoteproc remoteproc2: powering up 40800000.dsp
    [ 348.401183] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4714172
    [ 348.416084] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
    [ 348.421974] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
    [ 348.427915] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
    [ 348.435127] alloc_contig_range: [99004, 99007) PFNs busy
    [ 348.449508] virtio_rpmsg_bus virtio2: rpmsg host is online
    [ 348.452075] virtio_rpmsg_bus virtio2: creating channel rpmsg-proto addr 0x3d
    [ 348.462452] remoteproc remoteproc2: registered virtio2 (type 7)
    [ 348.468559] remoteproc remoteproc2: remote processor 40800000.dsp is now up
    root@am57xx-evm:/lib/firmware#
    root@am57xx-evm:/lib/firmware#
    root@am57xx-evm:/lib/firmware# /usr/bin/ipc/examples/ex02_messageq/release/app_host DSP1
    --> main:
    [ 368.784655] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1
    [ 368.829109] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
    [ 368.835001] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
    [ 368.841058] omap-iommu 41502000.mmu: 41502000.mmu: version 3.0
    --> Main_main:
    --> App_create:
    App_create: Host is ready
    <-- App_create:
    --> App_exec:
    App_exec: sending message 1
    App_exec: sending message 2
    App_exec: sending message 3
    App_exec: message received, sending message 4
    App_exec: message received, sending message 5
    App_exec: message received, sending message 6
    App_exec: message received, sending message 7
    App_exec: message received, sending message 8
    App_exec: message received, sending message 9
    App_exec: message received, sending message 10
    App_exec: message received, sending message 11
    App_exec: message received, sending message 12
    App_exec: message received, sending message 13
    App_exec: message received, sending message 14
    App_exec: message received, sending message 15
    App_exec: message received
    App_exec: message received
    App_exec: message received
    <-- App_exec: 0
    --> App_delete:
    <-- App_delete:
    <-- Main_main:
    <-- main:
    root@am57xx-evm:/lib/firmware# [ 379.359802] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
  • 0 in reply to Rex Chang
    TI__Guru 50170 points

    Just took another look at your logs. Did you use a script file to run the example? Before you ran app_host, had the DSP been up? From the logs, it seems to me that the DSP came up 6 secs afte you ran app_host.

    Rex

  • 0 in reply to Rex Chang
    Intellectual 415 points

    Yes, the documentation was exactly the one that you linked.

    And yes I did the high level build from the SDK by setting environment variables and invoking make:

    export PATH=<LINUX-SDK>/linux-devkit/sysroots/x86_64-arago-linux/usr/bin:$PATH
export TI_RTOS_PATH=<RTOS-SDK>
export IPC_INSTALL_PATH=<RTOS-SDK>/ipc_3_50_03_04
cd <LINUX-SDK>
make ti-ipc-linux
make ti-ipc-linux-examples

    /usr/bin/ipc/example/ex02_messageq/debug prebuilt images was a good idea, I didn't know they are there. By pointing the DSP firmware to the prebuilt one I was able to successfully run the ex02_messageq demo.

    I did this both with prebuilt DSP + prebuild ARM and prebuilt DSP + my own build of ARM and it worked.

    However, when I use my own build of the DSP firmware, it fails. I had no errors while building the DSP image, proof is that it was actually produced.

    I did a binary diff between my own build and the prebuilt images of the DSP firmware, and they differ drastically.

    Not sure what's going on, any ideas :/

  • 0 in reply to Rex Chang
    Intellectual 415 points
    No script file was used. DSP was up, I just didn't include the logs of it starting prior to launching app_host. You see the DSP come up seconds after because it crashes and is restarted.
  • 0 in reply to db
    TI__Guru 50170 points
    Ok. Give it a try with the prebuilt binaries, and let me know which documentation you followed to build the example.
  • 0 in reply to Rex Chang
    Intellectual 415 points
    I think you might've skipped one of my 2 replies :D
  • 0 in reply to db
    TI__Guru 50170 points
    I did miss your first reply for some reason which I didn't get a notification.
    Before I try to build it, I want to be sure that you use both Linux and RTOS SDK of the same release 5.2, correct?
    I'll let you know what I get once I finish.

    Rex
  • 0 in reply to Rex Chang
    Intellectual 415 points
    Yes, I use the following:
    <LINUX-SDK> - ti-processor-sdk-linux-am57xx-evm-05.02.00.10
    <RTOS-SDK> - ti-processor-sdk-rtos-am57xx-evm-05.02.00.10
  • 0 in reply to db
    TI__Guru 50170 points
    Hi, db,

    I don't see the issue. Here are my logs from Linux host and EVM console:

    Linux host building ex02_messageq example:
    2093 echo $PATH
    2094 PATH=~/work/ti-processor-sdk-linux-am57xx-evm-05.02.00.10/linux-devkit/sysroots/x86_64-arago-linux/usr/bin:$PATH
    2095 export TI_RTOS_PATH=~/work/ti-processor-sdk-rtos-am57xx-evm-05.02.00.10
    2096 export IPC_INSTALL_PATH=~/work/ti-processor-sdk-rtos-am57xx-evm-05.02.00.10/ipc_3_50_03_04
    2097 make ti-ipc-linux
    2098 la
    2099 cd ../ti-processor-sdk-linux-am57xx-evm-05.02.00.10/
    2100 make ti-ipc-linux
    2101 cd ../ti-processor-sdk-rtos-am57xx-evm-05.02.00.10/
    2102 export SDK_INSTALL_PATH=~/work/ti-processor-sdk-rtos-am57xx-evm-05.02.00.10
    2103 export TOOLS_INSTALL_PATH=~/work/ti-processor-sdk-rtos-am57xx-evm-05.02.00.10
    2104 cd processor_sdk_rtos_am57xx_5_02_00_10/
    2105 source ./setupenv.sh
    2106 make ipc_bios
    2107 cd ../../ti-processor-sdk-linux-am57xx-evm-05.02.00.10/
    2108 echo $TI_ROTS_PATH
    2109 export TI_RTOS_PATH=~/work/ti-processor-sdk-rtos-am57xx-evm-05.02.00.10
    2110 make ti-ipc-linux-examples
    2111 la ../ti-processor-sdk-rtos-am57xx-evm-05.02.00.10/ipc_3_50_03_04/examples/DRA7XX_linux_elf/ex02_messageq/dsp1/bin/debug/
    2112 cd ../ti-processor-sdk-rtos-am57xx-evm-05.02.00.10/ipc_3_50_03_04/examples/DRA7XX_linux_elf/ex02_messageq/
    2113 scp dsp1/bin/debug/server_dsp1.xe66 root@158.218.117.13:/home/root/ex02_dsp1.xe66
    2114 scp host/bin/debug/app_host root@158.218.117.13:/home/root/ex02_app_host
    2115 history

    EVM console:
    am57xx-evm login: root
    root@am57xx-evm:~# pwd
    /home/root
    root@am57xx-evm:~# ifconfig
    eth0 Link encap:Ethernet HWaddr A0:F6:FD:AA:D2:CC
    inet addr:158.218.117.13 Bcast:158.218.117.255 Mask:255.255.254.0
    inet6 addr: fe80::a2f6:fdff:feaa:d2cc%775872/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:272 errors:0 dropped:14 overruns:0 frame:0
    TX packets:105 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:46309 (45.2 KiB) TX bytes:13342 (13.0 KiB)
    Interrupt:89

    eth1 Link encap:Ethernet HWaddr A0:F6:FD:AA:D2:CD
    UP BROADCAST MULTICAST MTU:1500 Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1%775872/128 Scope:Host
    UP LOOPBACK RUNNING MTU:65536 Metric:1
    RX packets:169 errors:0 dropped:0 overruns:0 frame:0
    TX packets:169 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:12906 (12.6 KiB) TX bytes:12906 (12.6 KiB)

    root@am57xx-evm:~# ls
    test test.cpp tibt
    root@am57xx-evm:~# ls
    ex02_app_host ex02_dsp1.xe66 test test.cpp tibt
    root@am57xx-evm:~# cd /lib/firmware
    root@am57xx-evm:/lib/firmware# ls -l dra7-dsp1-fw.xe66
    lrwxrwxrwx 1 root root 58 Dec 16 03:38 dra7-dsp1-fw.xe66 -> /usr/bin/ipc/examples/ex02_messageq/debug/server_dsp1.xe66
    root@am57xx-evm:/lib/firmware# ln -sf /home/root/ex02_
    ex02_app_host ex02_dsp1.xe66
    root@am57xx-evm:/lib/firmware# ln -sf /home/root/ex02_dsp1.xe66 dra7-dsp1-fw.xe66
    root@am57xx-evm:/lib/firmware# ls -l dra7-dsp1-fw.xe66
    lrwxrwxrwx 1 root root 25 Dec 16 03:37 dra7-dsp1-fw.xe66 -> /home/root/ex02_dsp1.xe66
    root@am57xx-evm:/lib/firmware# echo stop > /sys/class/remoteproc/remoteproc2/state
    [ 305.614397] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
    [ 305.620937] remoteproc remoteproc2: stopped remote processor 40800000.dsp
    root@am57xx-evm:/lib/firmware# echo start > /sys/class/remoteproc/remoteproc2/state
    [ 315.771008] remoteproc remoteproc2: powering up 40800000.dsp
    [ 315.783023] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4447828
    [ 315.797897] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
    [ 315.803788] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
    [ 315.809720] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
    [ 315.824106] virtio_rpmsg_bus virtio3: rpmsg host is online
    [ 315.826681] virtio_rpmsg_bus virtio3: creating channel rpmsg-proto addr 0x3d
    [ 315.837474] remoteproc remoteproc2: registered virtio3 (type 7)
    [ 315.843503] remoteproc remoteproc2: remote processor 40800000.dsp is now up
    root@am57xx-evm:/lib/firmware#
    root@am57xx-evm:/lib/firmware# cd
    root@am57xx-evm:~# ls
    ex02_app_host ex02_dsp1.xe66 test test.cpp tibt
    root@am57xx-evm:~# ls -l ex02_app_host
    -rwxr-xr-x 1 root root 25760 Dec 16 03:37 ex02_app_host
    root@am57xx-evm:~# ./ex02_app_host
    --> main:
    Error: missing procName argument
    Usage:
    app_host [options] procName

    Arguments:
    procName : the name of the remote processor

    Options:
    h : print this help message
    l : list the available remote names

    Examples:
    app_host DSP
    app_host -l
    app_host -h

    <-- main:
    root@am57xx-evm:~# ./ex02_app_host DSP1
    --> main:
    [ 337.177783] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1
    [ 337.224920] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
    [ 337.230812] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
    [ 337.237807] omap-iommu 41502000.mmu: 41502000.mmu: version 3.0
    --> Main_main:
    --> App_create:
    App_create: Host is ready
    <-- App_create:
    --> App_exec:
    App_exec: sending message 1
    App_exec: sending message 2
    App_exec: sending message 3
    App_exec: message received, sending message 4
    App_exec: message received, sending message 5
    App_exec: message received, sending message 6
    App_exec: message received, sending message 7
    App_exec: message received, sending message 8
    App_exec: message received, sending message 9
    App_exec: message received, sending message 10
    App_exec: message received, sending message 11
    App_exec: message received, sending message 12
    App_exec: message received, sending message 13
    App_exec: message received, sending message 14
    App_exec: message received, sending message 15
    App_exec: message received
    App_exec: message received
    App_exec: message received
    <-- App_exec: 0
    --> App_delete:
    <-- App_delete:
    <-- Main_main:
    <-- main:
    root@am57xx-evm:~# [ 348.638113] omap_hwmod: mmu0_dsp2: _wait_target_disable failed

    root@am57xx-evm:~# uname -a
    Linux am57xx-evm 4.14.79-gbde58ab01e #3 SMP PREEMPT Wed Jan 30 14:33:51 EST 2019 armv7l GNU/Linux
    root@am57xx-evm:~#
  • 0 in reply to Rex Chang
    Intellectual 415 points
    Thanks for taking time to build from scratch.

    I followed your history output and I do pretty much the same thing when building.

    The big difference is however the setupenv.sh script.
    This script refuses to run on my Linux distro which is non-Ubuntu.

    I built the binaries again by doing what you showed in your history and omitting setupenv.sh and I still experience the issue.

    It seems setupenv.sh does something that is important in the build process.

    I will mark this as solved although I'm still experiencing the issue but it seems related to my environment.