Linux/PROCESSOR-SDK-AM57X: Stopping the M4 ipu cores at runtime causes the A15 core to crash

Tavi

Intellectual 410 points

Part Number: PROCESSOR-SDK-AM57X
Other Parts Discussed in Thread: AM5718

Tool/software: Linux

Hello,

I'm having some issues when stopping the M4 ipu cores on an AM5718 custom board.

I'm using yocto to build the kernel, u-boot and rootfs with the arago configuration which came with SDK 05.02.00.10

To test the IPC functionality I use the firmware for ex02_messageq built by the ti-ipc-examples recipe. The communication between the host processor and IPU1/IPU2/DSP1 works fine using this firmware.

The problem appears when I try to reload the IPU firmware at run time. I've tried both the /sys/class/remoteproc/remoteproc0/ and the /sys/bus/platform/drivers/omap-rproc interfaces and the problem is the same. When one of the IPU cores is stopped, the entire systems freezes without any error message on the serial console and needs a hard reboot to bring it back. It might take a few start/stop cycles to reproduce the problem but it always happens for any of the IPU cores. It does not happen for the DSP core though.

I'm using the following script to generate the crash:

while true; do echo stop > /sys/class/remoteproc/remoteproc0/state; sleep 2; echo start > /sys/class/remoteproc/remoteproc0/state; sleep 2; done

One thing I noticed in the ex02_messageq sources is that the IpuAmmu.cfg file for IPU1 and IPU2 are identical and contain some memory mappings which look like they should be core specific:

AMMU.smallPages[2].translatedAddress = 0x55020000;

I don't know if this is a problem, but I wanted to mention it here just in case

I also found another forum question which seems to describe a similar issue:

https://e2e.ti.com/support/processors/f/791/t/755141?Linux-PROCESSOR-SDK-AM57X-starting-and-stopping-the-ipu-core-causes-the-A15-to-crash

Any suggestions on how to debug this further?

--Tavi

over 6 years ago

0 Rex Chang over 6 years ago

TI__Guru 50170 points

Hi, Tavi,

Could you try using unbind/bind? I just tried it back to back and didn't see the issue. Here is my console logs:

am57xx-evm login: root
root@am57xx-evm:~# uname -a
Linux am57xx-evm 4.14.79-gbde58ab01e #1 SMP PREEMPT Thu Dec 20 04:51:24 UTC 2018 armv7l GNU/Linux
root@am57xx-evm:~#
root@am57xx-evm:~#
root@am57xx-evm:~# cat /sys/kernel/debug/remoteproc/remoteproc0/trace0
[0][ 0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][ 0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][ 0.000] 18 Resource entries at 0x3000
[0][ 0.000] [t=0x000eee21] xdc.runtime.Main: --> main:
[0][ 0.000] registering rpmsg-proto:rpmsg-proto service on 61 with HOST
[0][ 0.000] [t=0x00b52335] xdc.runtime.Main: NameMap_sendMessage: HOST 53, port=61
[0][ 0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][ 0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][ 0.000] [t=0x00b9a92f] xdc.runtime.Main: --> smain:
[0][ 0.000] [t=0x00bcb66b] Server: Server_create: server is ready
[0][ 0.000] [t=0x00bd9b6b] Server: <-- Server_create: 0
[0][ 0.000] [t=0x00be564d] Server: --> Server_exec:
root@am57xx-evm:~#
root@am57xx-evm:~#
root@am57xx-evm:~# cd /sys/bus/platform/drivers/omap-rproc/
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# ls
40800000.dsp 55020000.ipu bind uevent
41000000.dsp 58820000.ipu module unbind
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# echo 58820000.ipu > unbind
[ 330.245812] remoteproc remoteproc0: stopped remote processor 58820000.ipu
[ 330.252938] remoteproc remoteproc0: releasing 58820000.ipu
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# cat /sys/kernel/debug/remoteproc/remoteproc0/trace0
cat: can't open '/sys/kernel/debug/remoteproc/remoteproc0/trace0': No such file or directory
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# echo 58820000.ipu > bind
[ 343.591436] omap-rproc 58820000.ipu: assigned reserved memory node ipu1-memory@9d000000
[ 343.599583] remoteproc remoteproc0: 58820000.ipu is available
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# [ 343.616283] remoteproc remoteproc0: powering up 58820000.ipu
[ 343.622178] remoteproc remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 4466748
[ 343.630051] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1
[ 343.646519] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 343.652160] remoteproc remoteproc0: registered virtio0 (type 7)
[ 343.652735] virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x3d
[ 343.668094] remoteproc remoteproc0: remote processor 58820000.ipu is now up
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# cat /sys/kernel/debug/remoteproc/remoteproc0/trace0
[0][ 0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][ 0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][ 0.000] 18 Resource entries at 0x3000
[0][ 0.000] [t=0x000efadb] xdc.runtime.Main: --> main:
[0][ 0.000] registering rpmsg-proto:rpmsg-proto service on 61 with HOST
[0][ 0.000] [t=0x0015b87b] xdc.runtime.Main: NameMap_sendMessage: HOST 53, port=61
[0][ 0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][ 0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][ 0.000] [t=0x001a3d0b] xdc.runtime.Main: --> smain:
[0][ 0.000] [t=0x001d70af] Server: Server_create: server is ready
[0][ 0.000] [t=0x001e5829] Server: <-- Server_create: 0
[0][ 0.000] [t=0x001f1475] Server: --> Server_exec:
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# echo 58820000.ipu > unbind
[ 398.356571] remoteproc remoteproc0: stopped remote processor 58820000.ipu
[ 398.365757] remoteproc remoteproc0: releasing 58820000.ipu
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc#
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# echo 58820000.ipu > bind
[ 404.461538] omap-rproc 58820000.ipu: assigned reserved memory node ipu1-memory@9d000000
[ 404.471040] remoteproc remoteproc0: 58820000.ipu is available
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# [ 404.485803] remoteproc remoteproc0: powering up 58820000.ipu
[ 404.491529] remoteproc remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 4466748
[ 404.499417] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1
[ 404.515115] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 404.520716] remoteproc remoteproc0: registered virtio0 (type 7)
[ 404.526827] remoteproc remoteproc0: remote processor 58820000.ipu is now up
[ 404.537013] virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x3d

0 Tavi over 6 years ago in reply to Rex Chang

Intellectual 410 points

Hi Rex,

Thanks for replying.

The problem happens with bind/unbind as well. I posted below my script and the console output. As you can see it went through a few cycles before it hung after a stop command

cat /sys/kernel/debug/remoteproc/remoteproc0/trace0;
sleep 1
while true
do
    echo "################# Stopping M4 #################"
    echo 58820000.ipu > /sys/bus/platform/drivers/omap-rproc/unbind;
    sleep 4;
    cat /sys/kernel/debug/remoteproc/remoteproc0/trace0;
    sleep 1
    echo "################# Starting M4 #################"
    echo 58820000.ipu > /sys/bus/platform/drivers/omap-rproc/bind;
    sleep 4;
    cat /sys/kernel/debug/remoteproc/remoteproc0/trace0;
    sleep 1
done

~# sh test.sh
[0][      0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][      0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][      0.000] 18 Resource entries at 0x3000
[0][      0.000] [t=0x0008cf81] xdc.runtime.Main: --> main:
[0][      0.000] registering rpmsg-proto:rpmsg-proto service on 61 with HOST
[0][      0.000] [t=0x0046bf4b] xdc.runtime.Main: NameMap_sendMessage: HOST 53, port=61
[0][      0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][      0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][      0.000] [t=0x0049d5f1] xdc.runtime.Main: --> smain:
[0][      0.000] [t=0x004bf483] Server: Server_create: server is ready
[0][      0.000] [t=0x004c9857] Server: <-- Server_create: 0
[0][      0.000] [t=0x004d1849] Server: --> Server_exec:
################# Stopping M4 #################
[   31.869834] remoteproc remoteproc0: stopped remote processor 58820000.ipu
[   31.882376] remoteproc remoteproc0: releasing 58820000.ipu
cat: /sys/kernel/debug/remoteproc/remoteproc0/trace0: No such file or directory
################# Starting M4 #################[   36.901025] omap-rproc 58820000.ipu: assigned reserved memory node ipu1-memory@9d000000

[   36.913814] remoteproc remoteproc0: 58820000.ipu is available
[   36.939135] remoteproc remoteproc0: powering up 58820000.ipu
[   36.945085] remoteproc remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 4216220
[   36.956166] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1
[   36.973759] virtio_rpmsg_bus virtio0: rpmsg host is online
[   36.983700] remoteproc remoteproc0: registered virtio0 (type 7)
[   36.989655] remoteproc remoteproc0: remote processor 58820000.ipu is now up
[   37.004185] virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x3d
[0][      0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][      0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][      0.000] 18 Resource entries at 0x3000
[0][      0.000] [t=0x0008ff03] xdc.runtime.Main: --> main:
[0][      0.000] registering rpmsg-proto:rpmsg-proto service on 61 with HOST
[0][      0.000] [t=0x000c6f07] xdc.runtime.Main: NameMap_sendMessage: HOST 53, port=61
[0][      0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][      0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][      0.000] [t=0x000f8ab1] xdc.runtime.Main: --> smain:
[0][      0.000] [t=0x0011bca3] Server: Server_create: server is ready
[0][      0.000] [t=0x0012825b] Server: <-- Server_create: 0
[0][      0.000] [t=0x00131f25] Server: --> Server_exec:
################# Stopping M4 #################
[   41.953712] remoteproc remoteproc0: stopped remote processor 58820000.ipu
[   41.965370] remoteproc remoteproc0: releasing 58820000.ipu
cat: /sys/kernel/debug/remoteproc/remoteproc0/trace0: No such file or directory
################# Starting M4 #################[   46.983204] omap-rproc 58820000.ipu: assigned reserved memory node ipu1-memory@9d000000

[   46.996117] remoteproc remoteproc0: 58820000.ipu is available
[   47.016450] remoteproc remoteproc0: powering up 58820000.ipu
[   47.024836] remoteproc remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 4216220
[   47.035146] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1
[   47.051541] virtio_rpmsg_bus virtio0: rpmsg host is online
[   47.062427] remoteproc remoteproc0: registered virtio0 (type 7)
[   47.068383] remoteproc remoteproc0: remote processor 58820000.ipu is now up
[   47.081528] virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x3d
[0][      0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][      0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][      0.000] 18 Resource entries at 0x3000
[0][      0.000] [t=0x00090a65] xdc.runtime.Main: --> main:
[0][      0.000] registering rpmsg-proto:rpmsg-proto service on 61 with HOST
[0][      0.000] [t=0x000c6ecd] xdc.runtime.Main: NameMap_sendMessage: HOST 53, port=61
[0][      0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][      0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][      0.000] [t=0x000f8907] xdc.runtime.Main: --> smain:
[0][      0.000] [t=0x0011b809] Server: Server_create: server is ready
[0][      0.000] [t=0x00127fbf] Server: <-- Server_create: 0
[0][      0.000] [t=0x00131c9f] Server: --> Server_exec:
################# Stopping M4 #################
[   52.039423] remoteproc remoteproc0: stopped remote processor 58820000.ipu
[   52.050458] remoteproc remoteproc0: releasing 58820000.ipu
cat: /sys/kernel/debug/remoteproc/remoteproc0/trace0: No such file or directory
################# Starting M4 #################[   57.070317] omap-rproc 58820000.ipu: assigned reserved memory node ipu1-memory@9d000000

[   57.083261] remoteproc remoteproc0: 58820000.ipu is available
[   57.099751] remoteproc remoteproc0: powering up 58820000.ipu
[   57.113948] remoteproc remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 4216220
[   57.124971] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1
[   57.143578] virtio_rpmsg_bus virtio0: rpmsg host is online
[   57.153511] remoteproc remoteproc0: registered virtio0 (type 7)
[   57.159466] remoteproc remoteproc0: remote processor 58820000.ipu is now up
[   57.173772] virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x3d
[0][      0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][      0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][      0.000] 18 Resource entries at 0x3000
[0][      0.000] [t=0x0008fa7f] xdc.runtime.Main: --> main:
[0][      0.000] registering rpmsg-proto:rpmsg-proto service on 61 with HOST
[0][      0.000] [t=0x000c6b1d] xdc.runtime.Main: NameMap_sendMessage: HOST 53, port=61
[0][      0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][      0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][      0.000] [t=0x000f876d] xdc.runtime.Main: --> smain:
[0][      0.000] [t=0x0011b8d9] Server: Server_create: server is ready
[0][      0.000] [t=0x00127dd9] Server: <-- Server_create: 0
[0][      0.000] [t=0x001318b3] Server: --> Server_exec:
################# Stopping M4 #################
[   62.127319] remoteproc remoteproc0: stopped remote processor 58820000.ipu
[   62.138386] remoteproc remoteproc0: releasing 58820000.ipu
cat: /sys/kernel/debug/remoteproc/remoteproc0/trace0: No such file or directory
################# Starting M4 #################[   67.158512] omap-rproc 58820000.ipu: assigned reserved memory node ipu1-memory@9d000000

[   67.173463] remoteproc remoteproc0: 58820000.ipu is available
[   67.189570] remoteproc remoteproc0: powering up 58820000.ipu
[   67.202242] remoteproc remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 4216220
[   67.210115] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1
[   67.230609] virtio_rpmsg_bus virtio0: rpmsg host is online
[   67.241223] remoteproc remoteproc0: registered virtio0 (type 7)
[   67.247178] remoteproc remoteproc0: remote processor 58820000.ipu is now up
[   67.263272] virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x3d
[0][      0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][      0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][      0.000] 18 Resource entries at 0x3000
[0][      0.000] [t=0x00090ab1] xdc.runtime.Main: --> main:
[0][      0.000] registering rpmsg-proto:rpmsg-proto service on 61 with HOST
[0][      0.000] [t=0x000c7b73] xdc.runtime.Main: NameMap_sendMessage: HOST 53, port=61
[0][      0.000] Watchdog disabled: TimerBase = 0x68824000 ClkCtrl = 0x6a005568
[0][      0.000] Watchdog disabled: TimerBase = 0x68826000 ClkCtrl = 0x6a005570
[0][      0.000] [t=0x000f9a89] xdc.runtime.Main: --> smain:
[0][      0.000] [t=0x0011d135] Server: Server_create: server is ready
[0][      0.000] [t=0x0012976f] Server: <-- Server_create: 0
[0][      0.000] [t=0x0013338d] Server: --> Server_exec:
################# Stopping M4 #################

0 Tavi over 6 years ago in reply to Tavi

Intellectual 410 points

Hi Rex,

Were you able to reproduce the problem on your side?

0 Rex Chang over 6 years ago in reply to Tavi

TI__Guru 50170 points

Hi, Tavi,

The IPC examples are meant to show the functionality how they work. The DSP/IPU firmware are meant to be loaded during the system comes up and it is not expected to unload and reload the firmware frequently during operation. It is a nice tool to reload the DSP during development for different projects. We don't see any reason to stress test the reloading.

Could you provide further info on the application which requires the DSP/IPU firmware be reloaded multiple times instead of coming up during boot up time and stay running?

Rex

0 Tavi over 6 years ago in reply to Rex Chang

Intellectual 410 points

Hi Rex,

Like you said, runtime reloading of the firmware is a nice feature to have during development and testing. I was hopping to use it in order to avoid repeated reboots while working on the code, but we don't need it for production.

I did raise the issue because I do find it concerning that the A15 core completely freezes when stopping the IPU cores. I was worried that it might be something wrong with my software or hardware setup.

Can you confirm that you see the issue on your side and that it will not pose a problem if the feature (stopping IPU cores at runtime) is not used?

--Tavi

0 Rex Chang over 6 years ago in reply to Tavi

TI__Guru 50170 points

Hi, Tavi,

I see it intermittently. It isn't a problem if the feature is not used. However, I'll discuss the hung issue internally. If it can be improved or be more robust, we'll include it in the future release.

Rex

0 Rex Chang over 6 years ago in reply to Rex Chang

TI__Guru 50170 points

Hi, Tavi,

By the way, could you try using the IPC test binaries for unbind/bind test? It is located in /lib/firmware/ipc/ti_platforms_evmDRA7XX_ipuX/messageq_single.xem4

This binary behaves better than ex02_messageq example.

Rex

0 Tavi over 6 years ago in reply to Rex Chang

Intellectual 410 points

Hi Rex,

The binaries you list above are definitely better. The bind/unbind used to fail after 3-5 loops at the most (< 1 minute), and with the new firmware the loop has been running for more than 30 minutes and hasn't failed yet.

--Tavi

0 Rex Chang over 6 years ago in reply to Tavi

TI__Guru 50170 points

Hi, Tavi,

It seems that ex02_messageq is not built with PM or DEH, so it is not really compliant with the kernel remoteproc driver.
If your issue is resolved, please click "Resolved".

Rex

0 Tavi over 6 years ago in reply to Rex Chang

Intellectual 410 points

Hi Rex,

What are PM and DEH?

I was planing to use the ex02_messageq example as a starting point for my custom app. Is there a better choice for this? Or are there any changes to the build configuration I should make to ensure that my code is compliant with the remoteproc driver?

--Tavi

0 Rex Chang over 6 years ago in reply to Tavi

TI__Guru 50170 points

Hi, Tavi,

PM is the Power Managemnet, and DEH is the Device Exception Module. You may want to compare DSP vs IPU code in ex02_messageq example to see if they are configured differently. MessageQBench seems to behave better which you can use it as the example. It is based on messageq but adding measurement for round trip time.

Rex

0 Tavi over 6 years ago in reply to Rex Chang

Intellectual 410 points

Thank you

Processors

Processors forum

Linux/PROCESSOR-SDK-AM57X: Stopping the M4 ipu cores at runtime causes the A15 core to crash