• State
  • Locked Locked
  • Replies 10 replies
  • Subscribers 103 subscribers
  • Views 2796 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/AM5728: DSP Unload Failure on Processor SDK 04.00.00.04

Gerard
Expert 1290 points
Part Number: AM5728
Other Parts Discussed in Thread: BQ40Z60

Tool/software: Linux

I completed the upgrade to Processor SDK 04.00.00.04 and am immediately seeing an issue that was not present with Processor SDK 03.03.00.04. Any time I unload a DSP I get a kernel error; see below. At that point I get kicked back to the login prompt (odd) and can no longer use the DSP until I reboot. I'm running no custom software in the example below - it's simple interaction with kernel module loading and the unbind node.

Any help would be appreciated as this makes Processor SDK 04.00.00.04 unusable for my system.

Thanks!

<system boots>

-bash-4.3# modprobe omap_remoteproc

-bash-4.3# modprobe rpmsg_proto

<wait 30 seconds for DSP1 and DSP2 to get automatically started as a result of loading the omap_remoteproc driver>

-bash-4.3# echo 40800000.dsp > /sys/bus/platform/drivers/omap-rproc/unbind
[ 127.973491] Unable to handle kernel paging request at virtual address e510607c
[ 127.980821] pgd = dcc55a80
[ 127.983583] [e510607c] *pgd=80000080007003, *pmd=9d496003, *pte=00000000
[ 127.990580] Internal error: Oops: a07 [#1] PREEMPT SMP ARM
[ 127.996091] Modules linked in: rpmsg_proto virtio_rpmsg_bus rpmsg_core omap_remoteproc remoteproc virtio_ring virtio fpga_config hci_uart btbcm bnep rfcomm bluetooth usb_f_ecm g_ether usb_f_rndis libcomposite u_ether extcon_palmas dwc3 udc_core usb_common uio_pdrv_genirq uio phy_omap_usb2 dwc3_omap bridge stp llc xt_tcpudp ipv6 iptable_filter ip_tables x_tables hw_info bq40z60_battery mpu9250(C) extcon_usb_gpio extcon_core rtc_isl1208
[ 128.037832] CPU: 0 PID: 425 Comm: bash Tainted: G WC 4.9.28-geed43d1050 #1
[ 128.045782] Hardware name: Generic DRA74X (Flattened Device Tree)
[ 128.051905] task: dd72b700 task.stack: dd73a000
[ 128.056474] PC is at rproc_free_vring+0x10c/0x164 [remoteproc]
[ 128.062331] LR is at 0x0
[ 128.064875] pc : [<bf204294>] lr : [<00000000>] psr: 60020013
[ 128.064875] sp : dd73bd00 ip : dcc5a9e0 fp : dd73bd3c
[ 128.076404] r10: e2fe6000 r9 : c1207b8c r8 : de293a10
[ 128.081650] r7 : ddbeb400 r6 : 00003000 r5 : e5106064 r4 : ddbebdd8
[ 128.088206] r3 : e5106064 r2 : ffffffff r1 : 00000000 r0 : 00000064
[ 128.094763] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 128.101929] Control: 30c5387d Table: 9cc55a80 DAC: fffffffd
[ 128.107701] Process bash (pid: 425, stack limit = 0xdd73a218)
[ 128.113473] Stack: (0xdd73bd00 to 0xdd73c000)
[ 128.117851] bd00: 00000000 dd73a018 dd73bd54 e0000000 c0326ddc ddbebc00 ddbeb5ec ddbeb400
[ 128.126066] bd20: de293a10 00000000 c1207b8c 00100000 dd73bd54 dd73bd40 bf204310 bf204194
[ 128.134283] bd40: 00000000 ddbeb5d4 dd73bdac dd73bd58 bf2047e0 bf2042f8 00000000 c0642004
[ 128.142498] bd60: de293a10 c1265ac8 e2e00000 ddbeb5d8 ddbeb658 ddbeb420 dd73bdac dd73bd88
[ 128.150711] bd80: bf211ba0 ddbeb5f4 ddbeb400 ddbeb5f4 ddbeb59c ddbeb5a4 ddbeb420 dcc70d0c
[ 128.158924] bda0: dd73bdd4 dd73bdb0 bf204f3c bf204370 de293a00 ddbeb400 0000000d bf21341c
[ 128.167138] bdc0: dd73bf80 ddbc6000 dd73bdf4 dd73bdd8 bf2050c4 bf204eac bf2119b4 de293a00
[ 128.175351] bde0: ddbeb400 0000000d dd73be0c dd73bdf8 bf2119d0 bf205000 de293a10 de293a10
[ 128.183566] be00: dd73be24 dd73be10 c0639a74 bf2119c0 de293a10 bf21341c dd73be3c dd73be28
[ 128.191778] be20: c0638258 c0639a54 de293a44 de293a10 dd73be54 dd73be40 c0638308 c06381d8
[ 128.199992] be40: c1233530 de293a10 dd73be74 dd73be58 c0636904 c06382f0 ddbc6000 dcc70d00
[ 128.208204] be60: 00000000 00000000 dd73be84 dd73be78 c0635ea8 c0636890 dd73be9c dd73be88
[ 128.216416] be80: c03ab340 c0635e8c dd67b8c4 0000000d dd73bedc dd73bea0 c03aab0c c03ab300
[ 128.224630] bea0: 00000000 00000000 de0d53d0 00000000 697302a5 dcc5c540 dd73bf80 00148408
[ 128.232846] bec0: dd73bf80 0000000d dd73a000 00000000 dd73bf4c dd73bee0 c0336fd0 c03aaa3c
[ 128.241059] bee0: 00000000 0000000a dd4e8d00 dcc55b00 00060003 00000000 0000000d 00000000
[ 128.249276] bf00: dcc5c540 00000001 dd73bf4c dd73bf18 c0337b94 c04c3664 00000000 0000040f
[ 128.257488] bf20: 00000001 dd4e8d00 0000000d 0000000d dcc5c540 00148408 dd73bf80 c0207ca4
[ 128.265700] bf40: dd73bf7c dd73bf50 c0337e00 c0336fac c0335534 c033975c dcc5c540 dcc5c540
[ 128.273912] bf60: 0000000d 00148408 c0207ca4 dd73a000 dd73bfa4 dd73bf80 c0338bfc c0337d60
[ 128.282124] bf80: 00000000 00000000 0000000d 00148408 b6eeed60 00000004 00000000 dd73bfa8
[ 128.290335] bfa0: c0207b00 c0338bbc 0000000d 00148408 00000001 00148408 0000000d 00000000
[ 128.298549] bfc0: 0000000d 00148408 b6eeed60 00000004 00000000 00000000 b6f4a000 bec33b84
[ 128.306762] bfe0: 00000000 bec33adc b6e2646c b6e7e39c 60020010 00000001 00000000 00000000
[ 128.314973] Backtrace:
[ 128.317464] [<bf204188>] (rproc_free_vring [remoteproc]) from [<bf204310>] (rproc_vdev_release+0x24/0x78 [remoteproc])
[ 128.328210] r10:00100000 r9:c1207b8c r8:00000000 r7:de293a10 r6:ddbeb400 r5:ddbeb5ec
[ 128.336075] r4:ddbebc00
[ 128.338643] [<bf2042ec>] (rproc_vdev_release [remoteproc]) from [<bf2047e0>] (rproc_resource_cleanup+0x47c/0x53c [remoteproc])
[ 128.350082] r4:ddbeb5d4 r3:00000000
[ 128.353697] [<bf204364>] (rproc_resource_cleanup [remoteproc]) from [<bf204f3c>] (rproc_shutdown+0x9c/0x154 [remoteproc])
[ 128.364704] r10:dcc70d0c r9:ddbeb420 r8:ddbeb5a4 r7:ddbeb59c r6:ddbeb5f4 r5:ddbeb400
[ 128.372568] r4:ddbeb5f4
[ 128.375136] [<bf204ea0>] (rproc_shutdown [remoteproc]) from [<bf2050c4>] (rproc_del+0xd0/0xdc [remoteproc])
[ 128.384921] r9:ddbc6000 r8:dd73bf80 r7:bf21341c r6:0000000d r5:ddbeb400 r4:de293a00
[ 128.392717] [<bf204ff4>] (rproc_del [remoteproc]) from [<bf2119d0>] (omap_rproc_remove+0x1c/0x34 [omap_remoteproc])
[ 128.403197] r6:0000000d r5:ddbeb400 r4:de293a00 r3:bf2119b4
[ 128.408893] [<bf2119b4>] (omap_rproc_remove [omap_remoteproc]) from [<c0639a74>] (platform_drv_remove+0x2c/0x44)
[ 128.419112] r5:de293a10 r4:de293a10
[ 128.422707] [<c0639a48>] (platform_drv_remove) from [<c0638258>] (__device_release_driver+0x8c/0x118)
[ 128.431964] r5:bf21341c r4:de293a10
[ 128.435556] [<c06381cc>] (__device_release_driver) from [<c0638308>] (device_release_driver+0x24/0x30)
[ 128.444902] r5:de293a10 r4:de293a44
[ 128.448496] [<c06382e4>] (device_release_driver) from [<c0636904>] (unbind_store+0x80/0x100)
[ 128.456970] r5:de293a10 r4:c1233530
[ 128.460566] [<c0636884>] (unbind_store) from [<c0635ea8>] (drv_attr_store+0x28/0x34)
[ 128.468343] r7:00000000 r6:00000000 r5:dcc70d00 r4:ddbc6000
[ 128.474035] [<c0635e80>] (drv_attr_store) from [<c03ab340>] (sysfs_kf_write+0x4c/0x50)
[ 128.481992] [<c03ab2f4>] (sysfs_kf_write) from [<c03aab0c>] (kernfs_fop_write+0xdc/0x1e4)
[ 128.490203] r4:0000000d r3:dd67b8c4
[ 128.493799] [<c03aaa30>] (kernfs_fop_write) from [<c0336fd0>] (__vfs_write+0x30/0x120)
[ 128.501755] r10:00000000 r9:dd73a000 r8:0000000d r7:dd73bf80 r6:00148408 r5:dd73bf80
[ 128.509616] r4:dcc5c540
[ 128.512162] [<c0336fa0>] (__vfs_write) from [<c0337e00>] (vfs_write+0xac/0x16c)
[ 128.519504] r8:c0207ca4 r7:dd73bf80 r6:00148408 r5:dcc5c540 r4:0000000d
[ 128.526236] [<c0337d54>] (vfs_write) from [<c0338bfc>] (SyS_write+0x4c/0xa8)
[ 128.533316] r9:dd73a000 r8:c0207ca4 r7:00148408 r6:0000000d r5:dcc5c540 r4:dcc5c540
[ 128.541100] [<c0338bb0>] (SyS_write) from [<c0207b00>] (ret_fast_syscall+0x0/0x34)
[ 128.548702] r7:00000004 r6:b6eeed60 r5:00148408 r4:0000000d
[ 128.554387] Code: e3e02000 e5900218 e0833000 e0835105 (e5c51018)
[ 128.562695] ---[ end trace 97f93c9087233425 ]---

  • 0
    TI__Guru**** 161600 points
    Hi,

    Are you using AM572x GP EVM? Or is this a custom board?

    Best Regards,
    Yordan
  • 0 in reply to Yordan Kovachev
    Expert 1290 points

    Yordan Kovachev said:
     

    Are you using AM572x GP EVM? Or is this a custom board?

    This is on our custom board.

  • 0 in reply to Gerard
    TI__Guru**** 161600 points
    Hi,

    How much ddr do you have on your board? Have yo modified the am57xx-evm-cmem.dtsi?
    Also how did you define your memory in the dts:
    memory@0 {
    device_type = "memory";
    reg = <0x0 0x80000000 0x0 0x80000000>;
    };

    reserved-memory {
    #address-cells = <2>;
    #size-cells = <2>;
    ranges;

    ipu2_cma_pool: ipu2_cma@95800000 {
    compatible = "shared-dma-pool";
    reg = <0x0 0x95800000 0x0 0x3800000>;
    reusable;
    status = "okay";
    };

    dsp1_cma_pool: dsp1_cma@99000000 {
    compatible = "shared-dma-pool";
    reg = <0x0 0x99000000 0x0 0x4000000>;
    reusable;
    status = "okay";
    };

    ipu1_cma_pool: ipu1_cma@9d000000 {
    compatible = "shared-dma-pool";
    reg = <0x0 0x9d000000 0x0 0x2000000>;
    reusable;
    status = "okay";
    };

    dsp2_cma_pool: dsp2_cma@9f000000 {
    compatible = "shared-dma-pool";
    reg = <0x0 0x9f000000 0x0 0x800000>;
    reusable;
    status = "okay";
    };
    };

    Best Regards,
    Yordan
  • 0 in reply to Yordan Kovachev
    TI__Guru 50170 points

    Hi, Gerard,

    Do you boot kernel from NAND? The symptom looks like when the ubi image it too big to fit in NAND. 

    Rex

  • 0 in reply to Rex Chang
    Expert 1290 points

    Rex Chang said:

    Do you boot kernel from NAND? The symptom looks like when the ubi image it too big to fit in NAND. 

    Nope, we boot the kernel from eMMC. 


  • 0 in reply to Gerard
    TI__Guru 50170 points
    Gerard,

    Do you mind attach the console logs with printenv info and full kernel boot logs. The kernel didn't boot to login prompt but showing "bash-3.4#" which we saw similar symptom but on other platform using NAND boot. I'd like to get your info and see if we can narrow it down to either a kernel issue or different from the other NAND boot.

    Rex
  • 0 in reply to Yordan Kovachev
    Expert 1290 points

    Yordan Kovachev said:

    How much ddr do you have on your board? Have yo modified the am57xx-evm-cmem.dtsi?

    We have 4GB of DDR on our board. I have modified our custom *cmem.dtsi file:

    / {
    reserved-memory {
    #address-cells = <2>;
    #size-cells = <2>;
    ranges;

    /* 0xf000_0000 - 0xf0ffffff */
    cmem_block_mem_0: cmem_block_mem@f0000000 {
    reg = <0x0 0xf0000000 0x0 0x01000000>;
    no-map;
    status = "okay";
    };

    /* 0xf100_0000 - 0xf1ffffff */
    cmem_block_mem_1: cmem_block_mem@f1000000 {
    reg = <0x0 0xf1000000 0x0 0x01000000>;
    no-map;
    status = "okay";
    };

    };

    cmem {
    compatible = "ti,cmem";
    #address-cells = <1>;
    #size-cells = <0>;

    #pool-size-cells = <2>;

    status = "okay";

    /* Pool 0 - DSP1's FPGA Buffer */
    cmem_block_0: cmem_block@0 {
    reg = <0>;
    memory-region = <&cmem_block_mem_0>;
    cmem-buf-pools = <1 0x0 0x01000000>;
    };

    /* Pool 1 - DSP2's FPGA Buffer */
    cmem_block_1: cmem_block@1 {
    reg = <1>;
    memory-region = <&cmem_block_mem_1>;
    cmem-buf-pools = <1 0x0 0x01000000>;
    };

    };
    };

    Yordan Kovachev said:

    Also how did you define your memory in the dts:
    memory@0 {
    device_type = "memory";
    reg = <0x0 0x80000000 0x0 0x80000000>;
    };

    reserved-memory {
    #address-cells = <2>;
    #size-cells = <2>;
    ranges;

    ipu2_cma_pool: ipu2_cma@95800000 {
    compatible = "shared-dma-pool";
    reg = <0x0 0x95800000 0x0 0x3800000>;
    reusable;
    status = "okay";
    };

    dsp1_cma_pool: dsp1_cma@99000000 {
    compatible = "shared-dma-pool";
    reg = <0x0 0x99000000 0x0 0x4000000>;
    reusable;
    status = "okay";
    };

    ipu1_cma_pool: ipu1_cma@9d000000 {
    compatible = "shared-dma-pool";
    reg = <0x0 0x9d000000 0x0 0x2000000>;
    reusable;
    status = "okay";
    };

    dsp2_cma_pool: dsp2_cma@9f000000 {
    compatible = "shared-dma-pool";
    reg = <0x0 0x9f000000 0x0 0x800000>;
    reusable;
    status = "okay";
    };
    };

    Here's the memory definition:

    reserved-memory {
    #address-cells = <2>;
    #size-cells = <2>;
    ranges;

    /* 0xe000 0000 - 0xe7ff ffff */
    dsp1_cma_pool: dsp1_cma@e0000000 {
    compatible = "shared-dma-pool";
    reg = <0x0 0xe0000000 0x0 0x8000000>;
    reusable;
    status = "okay";
    };

    /* 0xe800 0000 - 0xefff ffff */
    dsp2_cma_pool: dsp2_cma@e8000000 {
    compatible = "shared-dma-pool";
    reg = <0x0 0xe8000000 0x0 0x8000000>;
    reusable;
    status = "okay";
    };

    ipu1_cma_pool: ipu1_cma@df000000 {
    compatible = "shared-dma-pool";
    reg = <0x0 0xdf000000 0x0 0x800000>;
    reusable;
    status = "okay";
    };

    ipu2_cma_pool: ipu2_cma@df800000 {
    compatible = "shared-dma-pool";
    reg = <0x0 0xdf800000 0x0 0x800000>;
    reusable;
    status = "okay";
    };
    };

    Thanks

  • 0 in reply to Gerard
    TI__Guru 50170 points
    Gerard,

    If I read it correctly, you use cmem block 0 for DSP1's FPGA buffer and block 1 for DSP2's FPGA buffer. Please refer to the post on July 6 in this thread, e2e.ti.com/.../604490. The first 2 cmem blocks should be avoid.

    Rex
  • 0 in reply to Rex Chang
    Expert 1290 points

    Rex Chang said:

    If I read it correctly, you use cmem block 0 for DSP1's FPGA buffer and block 1 for DSP2's FPGA buffer. Please refer to the post on July 6 in this thread, e2e.ti.com/.../604490. The first 2 cmem blocks should be avoid. 

    We don't use OpenCL. We've been using the cmem configuration posted previously for all Processor SDK prior; is this some new requirement?

    Also, I see that there is now a 04.01 Processor SDK - would you expect better behavior for my issue with that version?

    Thanks

  • 0 in reply to Gerard
    TI__Guru 50170 points
    Hi, Gerard,

    ti-mctd daemon was introduced in ProcSDK 3.3 and always reserves the first 2 CMEM blocks. It starts at boot time and starting in ProcSDK 4.0, it reserves the cmem blocks according to /etc/ti-mctd/ti_mctd_config.json. Could you check if ti-mctd is running? If it is not running, yes, you can use the first 2 blocks.

    Rex