• State Resolved
  • Locked Locked
  • Replies 13 replies
  • Answers 1 answer
  • Subscribers 97 subscribers
  • Views 1626 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

RTOS/AM5726: DSP crash issue

Prerak Patel
Expert 2230 points

Part Number: AM5726

Tool/software: TI-RTOS

Hello,

We are working on AM5726 custom board.

We are using "4.4.32" Kernel version and "ti-processor-sdk-linux-am57xx-evm-03.02.00.05" SDK version.

We are using DSP to run one application. while using DSP, getting below error out of 20 reboot cycle.

Because of this crash application is not going to start.

Please do the needful. And let me know in case of any other details required.

=========================== Log Start ============================

[   58.924208] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
[   58.936716] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[   58.943788]  remoteproc2: stopped remote processor 40800000.dsp
[   58.950810]  remoteproc2: releasing 40800000.dsp
[   58.956058] omap-rproc 40800000.dsp: assigned reserved memory node dsp1_cma@99000000
[   58.964264]  remoteproc2: 40800000.dsp is available
[   58.969894]  remoteproc2: Note: remoteproc is still under development and considered experimental.
[   58.979036]  remoteproc2: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
[   59.042277] omap_hwmod: mmu1_dsp2: _wait_target_disable failed
[   59.054820] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
[   59.061548]  remoteproc2: powering up 40800000.dsp
[   59.069171]  remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 21998880
[   59.083667] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[   59.089566] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[   59.095539] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
[   59.102011]  remoteproc3: stopped remote processor 41000000.dsp
[   59.110217]  remoteproc3: releasing 41000000.dsp
[   59.115440] omap-rproc 41000000.dsp: assigned reserved memory node dsp2_cma@9f000000
[   59.133065]  remoteproc3: 41000000.dsp is available
[   59.140003]  remoteproc2: remote processor 40800000.dsp is now up
[   59.146223]  remoteproc3: Note: remoteproc is still under development and considered experimental.
[   59.156347] virtio_rpmsg_bus virtio7: rpmsg host is online
[   59.156655] virtio_rpmsg_bus virtio7: creating channel rpmsg-proto addr 0x3d
[   59.169980]  remoteproc3: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
[   59.180737]  remoteproc2: registered virtio7 (type 7)
[   59.221950]  remoteproc3: powering up 41000000.dsp
[   59.226905]  remoteproc3: Booting fw image dra7-dsp2-fw.xe66, size 21998880
[   59.243698] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
[   59.249593] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
[   59.255566] omap-iommu 41502000.mmu: 41502000.mmu: version 3.0
[   59.272858]  remoteproc3: remote processor 41000000.dsp is now up
[   59.279511] virtio_rpmsg_bus virtio6: rpmsg host is online
[   59.279599] virtio_rpmsg_bus virtio6: creating channel rpmsg-proto addr 0x3d
[   59.293387]  remoteproc3: registered virtio6 (type 7)
[   69.155551] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
[   69.169678] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[   70.045811] omap_hwmod: mmu1_dsp2: _wait_target_disable failed
[   70.059599] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
=========================== Log End ============================

Regards,

Prerak

  • 0
    TI__Guru 50170 points
    Hi, Prerak,

    Which error are you referring to, the "omap_hwmod: mmu1_dsp1: _wait_target_disable failed"? It is irrelevant to DSP crash. I assume you are using bind/unbind for reboot. If that is the case, be sure there are at least 15 seconds between bind/unbind, unbind/bind commands. If you still see DSP crash, you may want to see why DSP crashes.

    You may also want to try with latest ProcSDK. I ran IPC messageq example, The message of "_wait_target_disable failed" doesn't affect DSP from running. Your DSP crash should be something else.


    root@am57xx-evm:~# dmesg | grep dsp
    [ 0.000000] OF: reserved mem: initialized node dsp1_cma@99000000, compatible id shared-dma-pool
    [ 0.000000] OF: reserved mem: initialized node dsp2_cma@9f000000, compatible id shared-dma-pool
    [ 5.393322] omap-rproc 40800000.dsp: assigned reserved memory node dsp1_cma@99000000
    [ 5.436171] remoteproc remoteproc2: 40800000.dsp is available
    [ 5.457397] omap-rproc 41000000.dsp: assigned reserved memory node dsp2_cma@9f000000
    [ 5.490316] remoteproc remoteproc3: 41000000.dsp is available
    [ 5.923601] remoteproc remoteproc2: powering up 40800000.dsp
    [ 5.923610] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4663976
    [ 5.930285] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
    [ 5.935574] remoteproc remoteproc3: powering up 41000000.dsp
    [ 5.935581] remoteproc remoteproc3: Booting fw image dra7-dsp2-fw.xe66, size 4510088
    [ 5.942244] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
    [ 6.588888] remoteproc remoteproc2: remote processor 40800000.dsp is now up
    [ 6.744069] remoteproc remoteproc3: remote processor 41000000.dsp is now up
    root@am57xx-evm:~# ls
    ex02_app_host tibt
    root@am57xx-evm:~# ./ex02_app_host DSP1
    --> main:
    --> Main_main:
    --> App_create:
    App_create: Host is ready
    <-- App_create:
    --> App_exec:
    App_exec: sending message 1
    App_exec: sending message 2
    App_exec: sending message 3
    App_exec: message received, sending message 4
    App_exec: message received, sending message 5
    App_exec: message received, sending message 6
    App_exec: message received, sending message 7
    App_exec: message received, sending message 8
    App_exec: message received, sending message 9
    App_exec: message received, sending message 10
    App_exec: message received, sending message 11
    App_exec: message received, sending message 12
    App_exec: message received, sending message 13
    App_exec: message received, sending message 14
    App_exec: message received, sending message 15
    App_exec: message received
    App_exec: message received
    App_exec: message received
    <-- App_exec: 0
    --> App_delete:
    <-- App_delete:
    <-- Main_main:
    <-- main:
  • 0 in reply to Rex Chang
    Expert 2230 points

    Hi Rex,

    I do not restart DSP on my application side. DSP automatically get restarted as the application get started.

    Is there any reason for restarting DSP? This does not happen every time, I got this behaviour out of 20 reboot cycle.

    Regards,

    Prerak

  • 0 in reply to Prerak Patel
    TI__Guru 50170 points

    Hi, Prerak,

    I actually was asking how do you reboot? a script file to reboot from kernel prompt, power cycle. or other means? When you say "DSP automatically get restarted as the application get started", is the application run on ARM/Linux? I am a bit confused. DSP core needs to be up, and DSP application gets downloaded to DSP, then DSP application starts running. If it is a Linux application, does t control the DSP in your code?

    Rex

  • 0 in reply to Rex Chang
    Expert 2230 points

    Hello Rex,

    Sorry for delayed response. I was debugging issue from application side and i found below things.

    I am communicating with DSP using OpenCL library. As i call  clGetPlatformIDs(1, &platform_id, &ret_num_platforms) from my application side, DSP get restarted. This function doesn't return any value and application get stops.

    Answer of Your questions:

    How do you reboot?

    -I am doing a hard reboot of my board.

    Is the application run on ARM/Linux?

    - Yes, the application runs on the arm. and i am using OpenCL library to communicate with DSP.

    Regards,

    Prerak

  • 0 in reply to Prerak Patel
    TI__Guru 50170 points
    Hi, Prerak,

    Let me reconstruct your scenario.

    When the board starts running after hard reboot, the DSP image is downloaded by Linux. Then Linux application using OpenCL starts to run. The linux application is able to run most of the time, but one out of 20 reboots, when running linux application, the DSP will crash. When crash happens, the linux application is making a call to clGetPlatformIDs(). Do I understand it correctly?

    It seems to me that the reboot doesn't contribute to any possible cause to the DSP crash, but the OpenCL calls which sometimes works and sometimes doesn't.

    I need to check with our OpenCL expert to see what impact does the OpenCL API may have.

    Rex
  • 0 in reply to Rex Chang
    TI__Guru 50170 points
    Hi, Prerak,

    By the way, does the Linux application automatically run and the reboots performed through script in a loop in your test? Then after, say, 20 iterations, the DSP crashes when Linux application makes the call to getPlatformIDs()?

    or you hard reboot the board and manually run the linux application. Once done, hard reboot again to retest, and once a while, it crashes the DSP?

    Rex
  • 0 in reply to Rex Chang
    Expert 2230 points

    Hi Rex,

    Your understanding of scenario is perfect. Whether it is soft/hard reboot we are getting always DSP crash one out of 20 reboots.

    I do not start my application manually, it will start through runlevel script which I have put in /etc/init.d /.

    Let me know if you required more details from my side.

    Regards,

    Prerak

  • 0 in reply to Prerak Patel
    TI__Guru 50170 points
    Hi, Prerak,

    Ok. I got a clearer picture. On AM572x, dsp firmware is loaded by remoteproc during the boot process. During the boot process, "ti-mctd" daemon is also started automatically. Any OpenCL application should only be run after remoteproc loads DSP firmware and ti-mctd has started. It wouldn't be a problem if the application is run manually after login, but could have issues if the OpenCL application is run by script automatically run. The script needs to be sure remote proc finishes loading the dsp firmware and ti-mctd has started.

    Rex
  • 0 in reply to Rex Chang
    Expert 2230 points

    Hi Rex,

    I am sure that my "init script" start the application after DSP firmware gets loaded.

    What is ti-mctd? I am unaware of that. How can i check the status of ti-mctd?

    Regards,

    Prerak

  • 0 in reply to Prerak Patel
    TI__Guru 50170 points
    Hi, Prerak,

    Please refer to OpenCL document for details of ti-mctd daemon, downloads.ti.com/.../

    root@am57xx-evm:~# systemctl | grep mct
    ti-mct-daemon.service loaded active running TI MultiCore Tools Daemon
  • 0 in reply to Rex Chang
    Expert 2230 points

    Hi Rex,

    I will go through OpenCL Document.

    when I had applied systemctl | grep mct., I got below error.

    =======================================================================

    root@am57xx-evm:~# systemctl | grep mct
    To show all installed unit files use 'systemctl list-unit-files'.

    =======================================================================

    I had also tried with systemctl list-unit-files | grep mct, and it has nothing regarding mct.

    =======================================================================

    root@am57xx-evm:~# systemctl list-unit-files | grep mct
    root@am57xx-evm:~#

    =======================================================================

    Regards,

    Prerak

     

  • 0 in reply to Prerak Patel
    TI__Guru 50170 points
    Hi, Prerak,

    Did you issue the command on the system which had the issue happened or a healthy system? If it is on the system which had the issue occurred, that may explain it. you can also see if ti-mctd is running using "ps -ef". I am on kernel 4.9.41 (ProcSDK 4.1 release). Your PSDK 3.2 is old, but I am not sure if any diff in term of OpenCL between the 2 releases.

    root@am57xx-evm:~# ps -ef | grep ti-mct
    root 714 1 0 21:42 ? 00:00:00 /usr/bin/ti-mctd
    root 1141 1118 0 21:44 ttyS2 00:00:00 grep ti-mct

    root@am57xx-evm:~# systemctl | grep ti-mct
    ti-mct-daemon.service loaded active running TI MultiCore Tools Daemon

    root@am57xx-evm:~# uname -a
    Linux am57xx-evm 4.9.41-ge3a80a1c5c #2 SMP PREEMPT Tue Sep 26 19:14:57 EDT 2017 armv7l GNU/Linux
    root@am57xx-evm:~#
  • 0 in reply to Rex Chang
    TI__Guru 50170 points
    Hi, Prerak,

    There hasn't been any response on this thread. If you issue is resolved, please click "Resolved". If you have other issue, please submit a new thread. Thanks!

    Rex