This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5716: DSP1 program launched by remoteproc crashes

Part Number: AM5716
Other Parts Discussed in Thread: SYSBIOS

We are developing an application for DSP1 based on the IPC example ex02_messageq

http://software-dl.ti.com/processor-sdk-linux/esd/docs/latest/linux/Foundational_Components_IPC.html#ipc-for-am57xx

The DSP program crashes even if we have an empty "main" : return 0. The console output is the following :

root@am5716-phycore-rdk:/sys/bus/platform/drivers/omap-rproc# [  103.535346] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4454120
[  103.556917] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[  103.562817] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[  103.568776] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
[  103.588962] ------------[ cut here ]------------
[  103.593618] WARNING: CPU: 0 PID: 129 at /media/jenkins1/builddata1/yocto_ti_pd18.2.0/build_am5716/arago-tmp-external-linaro-toolchain/work-shared/am5716-phycore-rdk/kernel-source/drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x25c/0x36c
[  103.615089] 44000000.ocp:L3 Custom Error: MASTER DSP1_MDMA TARGET L4_PER3_P3 (Idle): Data Access in User mode during Functional access
[  103.627223] Modules linked in: cbc xfrm_user xfrm4_tunnel ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo xhci_plat_hcd xhci_hcd usbcore dwc3 udc_core usb_common rpmsg_rpc ecb sha512_generic sha512_arm sha256_generic hmac md5 sha1_generic sha1_arm_neon sha1_arm omap_aes_driver omap_sham omap_wdt ahci_platform libahci_platform libahci libata scsi_mod ti_vpe ti_sc ti_csc ti_vpdma dwc3_omap rtc_omap extcon_palmas extcon_core rtc_palmas omap_des des_generic crypto_engine omap_rng rng_core omap_remoteproc virtio_rpmsg_bus rpmsg_core remoteproc virtio virtio_ring sch_fq_codel uio_module_drv(O) uio gdbserverproxy(O) cryptodev(O) cmemk(O)
[  103.683560] CPU: 0 PID: 129 Comm: kworker/0:3 Tainted: G        W  O    4.9.41-g61a65e1-BSP-Yocto-TISDK-AM57xx-PD18.2.0 #1
[  103.694650] Hardware name: Generic DRA72X (Flattened Device Tree)
[  103.700777] Workqueue: events request_firmware_work_func
[  103.706117] Backtrace:
[  103.708591] [<c020b208>] (dump_backtrace) from [<c020b4c4>] (show_stack+0x18/0x1c)
[  103.716195]  r7:00000009 r6:600f0193 r5:00000000 r4:c102f8d0
[  103.721886] [<c020b4ac>] (show_stack) from [<c04f4940>] (dump_stack+0x8c/0xa0)
[  103.729147] [<c04f48b4>] (dump_stack) from [<c022d030>] (__warn+0xec/0x104)
[  103.736140]  r7:00000009 r6:c0bdb598 r5:00000000 r4:ee5b3ab0
[  103.741828] [<c022cf44>] (__warn) from [<c022d088>] (warn_slowpath_fmt+0x40/0x48)
[  103.749344]  r9:0000001b r8:ef1ca950 r7:c0bdb918 r6:00000000 r5:c0bdb4c4 r4:c0bdb568
[  103.757124] [<c022d04c>] (warn_slowpath_fmt) from [<c0524510>] (l3_interrupt_handler+0x25c/0x36c)
[  103.766032]  r3:ef1ca7c0 r2:c0bdb568
[  103.769621]  r4:80080003
[  103.772168] [<c05242b4>] (l3_interrupt_handler) from [<c027de80>] (__handle_irq_event_percpu+0xb4/0x138)
[  103.781690]  r10:c101a36f r9:ef1cf000 r8:00000017 r7:ee5b3bd4 r6:00000000 r5:ef1cf000
[  103.789551]  r4:ef1cacc0
[  103.792097] [<c027ddcc>] (__handle_irq_event_percpu) from [<c027df28>] (handle_irq_event_percpu+0x24/0x60)
[  103.801793]  r10:00000000 r9:ee5b2000 r8:ef006000 r7:00000000 r6:c1008b74 r5:ef1cf000
[  103.809655]  r4:ef1cf000
[  103.812201] [<c027df04>] (handle_irq_event_percpu) from [<c027dfa4>] (handle_irq_event+0x40/0x64)
[  103.821109]  r5:ef1cf060 r4:ef1cf000
[  103.824701] [<c027df64>] (handle_irq_event) from [<c0281650>] (handle_fasteoi_irq+0xc0/0x190)
[  103.833262]  r7:00000000 r6:c1008b74 r5:ef1cf060 r4:ef1cf000
[  103.838949] [<c0281590>] (handle_fasteoi_irq) from [<c027d0a8>] (generic_handle_irq+0x2c/0x3c)
[  103.847598]  r7:00000000 r6:00000000 r5:00000017 r4:c0e55da0
[  103.853284] [<c027d07c>] (generic_handle_irq) from [<c027d630>] (__handle_domain_irq+0x64/0xbc)
[  103.862022] [<c027d5cc>] (__handle_domain_irq) from [<c02014a0>] (gic_handle_irq+0x40/0x7c)
[  103.870410]  r9:ee5b2000 r8:fa213000 r7:fa212000 r6:ee5b3c90 r5:fa21200c r4:c1003420
[  103.878188] [<c0201460>] (gic_handle_irq) from [<c020c078>] (__irq_svc+0x58/0x8c)
[  103.885701] Exception stack(0xee5b3c90 to 0xee5b3cd8)
[  103.890774] 3c80:                                     00000000 00000000 00000000 edf92038
[  103.898987] 3ca0: edf92038 c1092c14 00000000 00000001 bf0588f0 00000032 00000000 ee5b3d04
[  103.907198] 3cc0: ee5b3cc8 ee5b3ce0 c06210a8 c0624a90 600f0013 ffffffff
[  103.913841]  r9:ee5b2000 r8:bf0588f0 r7:ee5b3cc4 r6:ffffffff r5:600f0013 r4:c0624a90
[  103.921622] [<c0624a08>] (driver_probe_device) from [<c0624e3c>] (__device_attach_driver+0xa0/0xd4)
[  103.930707]  r9:00000000 r8:c1092bf0 r7:00000001 r6:edf92038 r5:ee5b3d50 r4:bf0588f0
[  103.938487] [<c0624d9c>] (__device_attach_driver) from [<c0622fd0>] (bus_for_each_drv+0x68/0x9c)
[  103.947309]  r7:00000001 r6:c0624d9c r5:ee5b3d50 r4:00000000
[  103.952994] [<c0622f68>] (bus_for_each_drv) from [<c06248e4>] (__device_attach+0xb8/0x11c)
[  103.961294]  r6:bf03dde4 r5:edf9206c r4:edf92038
[  103.965934] [<c062482c>] (__device_attach) from [<c0624ebc>] (device_initial_probe+0x14/0x18)
[  103.974494]  r7:00000000 r6:bf03dde4 r5:edf92038 r4:edf92040
[  103.980179] [<c0624ea8>] (device_initial_probe) from [<c0623ee4>] (bus_probe_device+0x8c/0x94)
[  103.988830] [<c0623e58>] (bus_probe_device) from [<c0622054>] (device_add+0x3fc/0x588)
[  103.996780]  r7:00000000 r6:edf92038 r5:eebc2820 r4:edf92040
[  104.002465] [<c0621c58>] (device_add) from [<c06221fc>] (device_register+0x1c/0x20)
[  104.010154]  r10:eebc2a1c r9:edf92004 r8:eebc2820 r7:edf92028 r6:edf92038 r5:bf0459e8
[  104.018015]  r4:edf92038
[  104.020568] [<c06221e0>] (device_register) from [<bf03d494>] (register_virtio_device+0xb4/0xf4 [virtio])
[  104.030086]  r5:bf0459e8 r4:edf92028
[  104.033698] [<bf03d3e0>] (register_virtio_device [virtio]) from [<bf045f40>] (rproc_add_virtio_dev+0x78/0xfc [remoteproc])
[  104.044788]  r7:edf92028 r6:00000007 r5:eebc2820 r4:edf92000
[  104.050494] [<bf045ec8>] (rproc_add_virtio_dev [remoteproc]) from [<bf0432f4>] (rproc_vdev_do_probe+0x18/0x1c [remoteproc])
[  104.061671]  r7:00000000 r6:eebc29cc r5:eebc29c4 r4:eebc2800
[  104.067376] [<bf0432dc>] (rproc_vdev_do_probe [remoteproc]) from [<bf044fa4>] (__rproc_boot+0x4bc/0x5bc [remoteproc])
[  104.078052] [<bf044ae8>] (__rproc_boot [remoteproc]) from [<bf0450d0>] (rproc_auto_boot_callback+0x18/0x24 [remoteproc])
[  104.088969]  r10:ee5ac500 r9:00000000 r8:ef64dc00 r7:00000000 r6:ef64a4c0 r5:ee5ac500
[  104.096829]  r4:eddb35c0
[  104.099386] [<bf0450b8>] (rproc_auto_boot_callback [remoteproc]) from [<c063aa10>] (request_firmware_work_func+0x44/0x6c)
[  104.110386]  r5:ee5ac500 r4:eddb3ac0
[  104.113983] [<c063a9cc>] (request_firmware_work_func) from [<c02439cc>] (process_one_work+0x1dc/0x3f8)
[  104.123326]  r4:eddb3ac0
[  104.125873] [<c02437f0>] (process_one_work) from [<c0244624>] (worker_thread+0x58/0x574)
[  104.133999]  r10:ee5ac500 r9:ee5b2000 r8:ef64a4d8 r7:c1002d00 r6:00000008 r5:ee5ac518
[  104.141860]  r4:ef64a4c0
[  104.144408] [<c02445cc>] (worker_thread) from [<c024995c>] (kthread+0x100/0x118)
[  104.151837]  r10:00000000 r9:00000000 r8:c02445cc r7:ee5ac500 r6:ee5b2000 r5:ee5a0140
[  104.159697]  r4:00000000
[  104.162243] [<c024985c>] (kthread) from [<c0207c08>] (ret_from_fork+0x14/0x2c)
[  104.169497]  r8:00000000 r7:00000000 r6:00000000 r5:c024985c r4:ee5a0140
[  104.176225] ---[ end trace d3395628396b0b84 ]---
[  104.244024] virtio_rpmsg_bus virtio2: rpmsg host is online
[  104.249565] remoteproc remoteproc2: registered virtio2 (type 7)
[  104.257994] remoteproc remoteproc2: remote processor 40800000.dsp is now up
[  113.188861] remoteproc remoteproc2: crash detected in 40800000.dsp: type watchdog
[  113.197799] remoteproc remoteproc2: handling crash #1 in 40800000.dsp
[  113.204335] remoteproc remoteproc2: recovering 40800000.dsp
[  113.226676] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
[  113.239755] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[  113.249832] remoteproc remoteproc2: stopped remote processor 40800000.dsp
[  113.256729] remoteproc remoteproc2: powering up 40800000.dsp
[  113.270163] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4454120
[  113.286591] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[  113.292491] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[  113.298457] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
[  113.318707] ------------[ cut here ]------------
[  113.323366] WARNING: CPU: 0 PID: 129 at /media/jenkins1/builddata1/yocto_ti_pd18.2.0/build_am5716/arago-tmp-external-linaro-toolchain/work-shared/am5716-phycore-rdk/kernel-source/drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x25c/0x36c
[  113.344837] 44000000.ocp:L3 Custom Error: MASTER DSP1_MDMA TARGET L4_PER3_P3 (Idle): Data Access in User mode during Functional access
[  113.356972] Modules linked in: cbc xfrm_user xfrm4_tunnel ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo xhci_plat_hcd xhci_hcd usbcore dwc3 udc_core usb_common rpmsg_rpc ecb sha512_generic sha512_arm sha256_generic hmac md5 sha1_generic sha1_arm_neon sha1_arm omap_aes_driver omap_sham omap_wdt ahci_platform libahci_platform libahci libata scsi_mod ti_vpe ti_sc ti_csc ti_vpdma dwc3_omap rtc_omap extcon_palmas extcon_core rtc_palmas omap_des des_generic crypto_engine omap_rng rng_core omap_remoteproc virtio_rpmsg_bus rpmsg_core remoteproc virtio virtio_ring sch_fq_codel uio_module_drv(O) uio gdbserverproxy(O) cryptodev(O) cmemk(O)
[  113.413304] CPU: 0 PID: 129 Comm: kworker/0:3 Tainted: G        W  O    4.9.41-g61a65e1-BSP-Yocto-TISDK-AM57xx-PD18.2.0 #1
[  113.424394] Hardware name: Generic DRA72X (Flattened Device Tree)
[  113.430533] Workqueue: events rproc_crash_handler_work [remoteproc]
[  113.436833] Backtrace:
[  113.439307] [<c020b208>] (dump_backtrace) from [<c020b4c4>] (show_stack+0x18/0x1c)
[  113.446911]  r7:00000009 r6:600f0193 r5:00000000 r4:c102f8d0
[  113.452602] [<c020b4ac>] (show_stack) from [<c04f4940>] (dump_stack+0x8c/0xa0)
[  113.459862] [<c04f48b4>] (dump_stack) from [<c022d030>] (__warn+0xec/0x104)
[  113.466854]  r7:00000009 r6:c0bdb598 r5:00000000 r4:ee5b3af0
[  113.472543] [<c022cf44>] (__warn) from [<c022d088>] (warn_slowpath_fmt+0x40/0x48)
[  113.480058]  r9:0000001b r8:ef1ca950 r7:c0bdb918 r6:00000000 r5:c0bdb4c4 r4:c0bdb568
[  113.487839] [<c022d04c>] (warn_slowpath_fmt) from [<c0524510>] (l3_interrupt_handler+0x25c/0x36c)
[  113.496748]  r3:ef1ca7c0 r2:c0bdb568
[  113.500335]  r4:80080003
[  113.502881] [<c05242b4>] (l3_interrupt_handler) from [<c027de80>] (__handle_irq_event_percpu+0xb4/0x138)
[  113.512403]  r10:c101a36f r9:ef1cf000 r8:00000017 r7:ee5b3c14 r6:00000000 r5:ef1cf000
[  113.520264]  r4:ef1cacc0
[  113.522810] [<c027ddcc>] (__handle_irq_event_percpu) from [<c027df28>] (handle_irq_event_percpu+0x24/0x60)
[  113.532506]  r10:00000000 r9:ee5b2000 r8:ef006000 r7:00000000 r6:c1008b74 r5:ef1cf000
[  113.540366]  r4:ef1cf000
[  113.542912] [<c027df04>] (handle_irq_event_percpu) from [<c027dfa4>] (handle_irq_event+0x40/0x64)
[  113.551820]  r5:ef1cf060 r4:ef1cf000
[  113.555412] [<c027df64>] (handle_irq_event) from [<c0281650>] (handle_fasteoi_irq+0xc0/0x190)
[  113.563972]  r7:00000000 r6:c1008b74 r5:ef1cf060 r4:ef1cf000
[  113.569659] [<c0281590>] (handle_fasteoi_irq) from [<c027d0a8>] (generic_handle_irq+0x2c/0x3c)
[  113.578307]  r7:00000000 r6:00000000 r5:00000017 r4:c0e55da0
[  113.583994] [<c027d07c>] (generic_handle_irq) from [<c027d630>] (__handle_domain_irq+0x64/0xbc)
[  113.592733] [<c027d5cc>] (__handle_domain_irq) from [<c02014a0>] (gic_handle_irq+0x40/0x7c)
[  113.601121]  r9:ee5b2000 r8:fa213000 r7:fa212000 r6:ee5b3cd0 r5:fa21200c r4:c1003420
[  113.608898] [<c0201460>] (gic_handle_irq) from [<c020c078>] (__irq_svc+0x58/0x8c)
[  113.616412] Exception stack(0xee5b3cd0 to 0xee5b3d18)
[  113.621484] 3cc0:                                     eebc2898 600f0013 00000081 00000005
[  113.629698] 3ce0: fffffff3 eebc2898 00000004 600f0013 c1092bf0 00000000 00000000 ee5b3d2c
[  113.637910] 3d00: ee5b3d30 ee5b3d20 c062f4e4 c08b155c 600f0013 ffffffff
[  113.644553]  r9:ee5b2000 r8:c1092bf0 r7:ee5b3d04 r6:ffffffff r5:600f0013 r4:c08b155c
[  113.652338] [<c08b1534>] (_raw_spin_unlock_irqrestore) from [<c062f4e4>] (__pm_runtime_resume+0x64/0x6c)
[  113.661862] [<c062f480>] (__pm_runtime_resume) from [<c06248cc>] (__device_attach+0xa0/0x11c)
[  113.670423]  r7:00000001 r6:bf03dde4 r5:edff346c r4:edff3438
[  113.676109] [<c062482c>] (__device_attach) from [<c0624ebc>] (device_initial_probe+0x14/0x18)
[  113.684669]  r7:00000000 r6:bf03dde4 r5:edff3438 r4:edff3440
[  113.690356] [<c0624ea8>] (device_initial_probe) from [<c0623ee4>] (bus_probe_device+0x8c/0x94)
[  113.699007] [<c0623e58>] (bus_probe_device) from [<c0622054>] (device_add+0x3fc/0x588)
[  113.706958]  r7:00000000 r6:edff3438 r5:eebc2820 r4:edff3440
[  113.712643] [<c0621c58>] (device_add) from [<c06221fc>] (device_register+0x1c/0x20)
[  113.720333]  r10:eebc2a1c r9:edff3404 r8:eebc2820 r7:edff3428 r6:edff3438 r5:bf0459e8
[  113.728193]  r4:edff3438
[  113.730744] [<c06221e0>] (device_register) from [<bf03d494>] (register_virtio_device+0xb4/0xf4 [virtio])
[  113.740264]  r5:bf0459e8 r4:edff3428
[  113.743872] [<bf03d3e0>] (register_virtio_device [virtio]) from [<bf045f40>] (rproc_add_virtio_dev+0x78/0xfc [remoteproc])
[  113.754961]  r7:edff3428 r6:00000007 r5:eebc2820 r4:edff3400
[  113.760667] [<bf045ec8>] (rproc_add_virtio_dev [remoteproc]) from [<bf0432f4>] (rproc_vdev_do_probe+0x18/0x1c [remoteproc])
[  113.771843]  r7:00000000 r6:eebc29cc r5:eebc29c4 r4:eebc2800
[  113.777548] [<bf0432dc>] (rproc_vdev_do_probe [remoteproc]) from [<bf044fa4>] (__rproc_boot+0x4bc/0x5bc [remoteproc])
[  113.788225] [<bf044ae8>] (__rproc_boot [remoteproc]) from [<bf04536c>] (rproc_trigger_recovery+0x5c/0x68 [remoteproc])
[  113.798967]  r10:ee5ac500 r9:00000000 r8:ef64dc00 r7:00000000 r6:00000000 r5:eebc2a58
[  113.806828]  r4:eebc2800
[  113.809394] [<bf045310>] (rproc_trigger_recovery [remoteproc]) from [<bf0453ec>] (rproc_crash_handler_work+0x74/0x84 [remoteproc])
[  113.821182]  r7:00000000 r6:eebc2800 r5:eebc29cc r4:eebc2a44
[  113.826882] [<bf045378>] (rproc_crash_handler_work [remoteproc]) from [<c02439cc>] (process_one_work+0x1dc/0x3f8)
[  113.837187]  r7:00000000 r6:ef64a4c0 r5:ee5ac500 r4:eebc2a44
[  113.842876] [<c02437f0>] (process_one_work) from [<c0244624>] (worker_thread+0x58/0x574)
[  113.851001]  r10:ee5ac500 r9:ee5b2000 r8:ef64a4d8 r7:c1002d00 r6:00000008 r5:ee5ac518
[  113.858862]  r4:ef64a4c0
[  113.861411] [<c02445cc>] (worker_thread) from [<c024995c>] (kthread+0x100/0x118)
[  113.868840]  r10:00000000 r9:00000000 r8:c02445cc r7:ee5ac500 r6:ee5b2000 r5:ee5a0140
[  113.876702]  r4:00000000
[  113.879249] [<c024985c>] (kthread) from [<c0207c08>] (ret_from_fork+0x14/0x2c)
[  113.886502]  r8:00000000 r7:00000000 r6:00000000 r5:c024985c r4:ee5a0140
[  113.893230] ---[ end trace d3395628396b0b85 ]---
[  113.965828] virtio_rpmsg_bus virtio2: rpmsg host is online
[  113.971549] remoteproc remoteproc2: registered virtio2 (type 7)
[  113.977499] remoteproc remoteproc2: remote processor 40800000.dsp is now up

We thought that this would indicate that the addressing space related to L4_PER3_P3 was missing in
the resource table. But that was not the case.

Any suggestion ?

Thank you in advance for your answer.

  • We managed to work around this issue in selecting in the Dsp1.cfg file a specific timerId  for the clock: Clock.timerId = 5.

    In this case, the content of the file /sys/kernel/debug/remoteproc/remoteproc2/trace0 is :

    [      0.000] [t=0x00018f9f] ti.sysbios.family.c64p.Hwi: ERROR: line 194: E_alre
    adyDefined: Hwi already defined: intr# 15
    [      0.000] ti.sysbios.family.c64p.Hwi: line 194: E_alreadyDefined: Hwi alread
    y defined: intr# 15
    [      0.000] xdc.runtime.Error.raise: terminating execution

    Is it possible to select another interrupt number for the timer ?

    Thank you in advance for your answer.

  • Hi Gilbert,

    Thank you for updating the thread with an answer to your first issue.

    Regarding your question about remapping the IRQ mapping, yes, it is possible.

    The IRQ crossbar can be used for this.

    Below is a section copied from the TRM that talks about what sections you can reference to learn about this functionality:

    The “Default Mapping” column in Table 22-4 GP Timers Hardware Requests shows the
    default mapping of module IRQ source signals. These IRQ source signals can also be
    mapped to other lines of each device Interrupt controller through the IRQ_CROSSBAR
    module.

    For more information about the IRQ_CROSSBAR module, see Section 18.4.6.4,
    IRQ_CROSSBAR Module Functional Description, in Chapter 18, Control Module.
    For more information about the device interrupt controllers, see Chapter 17, Interrupt
    Controllers.

    For the description of the interrupt source, see Section 22.2.4.5, GP Timer Interrupt.

    Regards,
    Mike