RTOS/AM5716: DSP firmware load error

Rong Wang2

Part Number: AM5716

Tool/software: TI-RTOS

Hello,

in my application, I need to have a chunk of memory which is not cacheable. I firstly tried to set EXT_DATA as non-cacheable by setting MAR149 and it worked when I used Jtag to load dsp binary by CCS, however, it did not work when DSP build is loaded by Linux kernel. I notice that this memory is defined in original custom resources table. so I try to use another memory range by adding DSP_DATA in linker.cmd as below, and I also add it in resources table by adding following codes as well updated the number of element, etc.

#define DSP_NON_CACHE_DATA 0x9A000000

#define DSP_NON_CACHE_DATA_SIZE (SZ_1M)

{
TYPE_DEVMEM,
DSP_NON_CACHE_DATA, 0,
DSP_NON_CACHE_DATA_SIZE, 0, 0, "DSP_NON_CACHE_DATA",
},

the memory define in linker.cmd:

MEMORY
{

L2SRAM (RWX) : org = 0x800000, len = 0x40000
OCMC_RAM1 (RWX) : org = 0x40300000, len = 0x80000
OCMC_RAM2 (RWX) : org = 0x40400000, len = 0x100000
OCMC_RAM3 (RWX) : org = 0x40500000, len = 0x100000
EXT_CODE (RWX) : org = 0x95000000, len = 0x100000
EXT_DATA (RW) : org = 0x95100000, len = 0x100000
EXT_HEAP (RW) : org = 0x95200000, len = 0x300000
EXC_DATA (RW) : org = 0x9f060000, len = 0x10000
PM_DATA (RWX) : org = 0x9f070000, len = 0x20000
DSP_DATA (RW) : org = 0x9A000000, len = 0x10000

}

SECTIONS
{
.text: load >> EXT_CODE
.ti.decompress: load > EXT_CODE
.stack: load > EXT_DATA
GROUP: load > EXT_DATA
{
.bss:
.neardata:
.rodata:
}

.cinit: load > EXT_DATA
.pinit: load >> EXT_DATA
.init_array: load > EXT_DATA
.const: load >> EXT_DATA
.data: load >> EXT_DATA
.fardata: load >> EXT_DATA
.switch: load >> EXT_DATA
.sysmem: load > EXT_DATA
.far: load >> EXT_DATA
.args: load > EXT_DATA align = 0x4, fill = 0 {_argsize = 0x64; }
.cio: load >> EXT_DATA
.ti.handler_table: load > EXT_DATA
.c6xabi.exidx: load > EXT_DATA
.c6xabi.extab: load >> EXT_DATA
.tracebuf: load > EXT_DATA
.resource_table: load > 0x95000000, type = NOINIT
.vecs: load > EXT_CODE
.no_cache > EXT_DATA //DSP_DATA
xdc.meta: load > EXT_DATA, type = COPY

DPR
{
MemoirePartagee.obj
}> DSP_DATA

}

however, the linux failed to load DSP binary with following error message( in red):

root@am57xx-evm:~# ./unbinddsp
[ 4593.580589] remoteproc remoteproc2: failed to unmap 4194304/0
[ 4593.599912] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
[ 4593.613815] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[ 4593.619722] remoteproc remoteproc2: stopped remote processor 40800000.dsp
[ 4593.629584] remoteproc remoteproc2: releasing 40800000.dsp
root@am57xx-evm:~#
root@am57xx-evm:~# cp AIM_G3_DSP35.xe66 /lib/firmware/dra7-dsp1-fw.xe66
root@am57xx-evm:~# ./binddsp
[ 4621.364874] omap-rproc 40800000.dsp: assigned reserved memory node dsp1_cma@99000000
[ 4621.374002] remoteproc remoteproc2: 40800000.dsp is available
[ 4621.386669] remoteproc remoteproc2: powering up 40800000.dsp
root@am57xx-evm:~# [ 4621.398787] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4074844
[ 4621.417946] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[ 4621.423855] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[ 4621.429754] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
[ 4621.445240] remoteproc remoteproc2: bad phdr da 0x9a000000 mem 0x1f3c
[ 4621.451718] remoteproc remoteproc2: Failed to load program segments: -22
[ 4621.460462] remoteproc remoteproc2: failed to unmap 4194304/0
[ 4621.478906] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
[ 4621.492698] omap_hwmod: mmu0_dsp1: _wait_target_disable failed

Did I miss something or gone wrong?

Thanks in advance!

Ron

over 7 years ago

0 Biser Gatchev-XID over 7 years ago

TI__Guru**** 393215 points

The RTOS team have been notified. They will respond here.

0 Sahin Okur over 7 years ago

TI__Mastermind 27355 points

Hi Ron,

In your resource table, DSP_NON_CACHE_DATA is allocated 1 MB, but in your linker cmd file it's given 64 KB. Add a zero to make it 1 MB so that they match and give it another try.

0 Rong Wang2 over 7 years ago in reply to Sahin Okur

Expert 1420 points

Hi Sahin,

I made the change, but I still got the same error as below. the resources table file is attached for your reference.2235.rsc_table_dsp.h

[ 9417.957731] remoteproc remoteproc2: failed to unmap 4194304/0
[ 9417.977019] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
[ 9417.989910] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[ 9417.997298] remoteproc remoteproc2: stopped remote processor 40800000.dsp
[ 9418.006523] remoteproc remoteproc2: releasing 40800000.dsp
root@am57xx-evm:~# cp AIM_G3_DSP37.xe66 /lib/firmware/dra7-dsp1-fw.xe66
root@am57xx-evm:~# ./binddsp
[ 9445.204003] omap-rproc 40800000.dsp: assigned reserved memory node dsp1_cma@99000000
[ 9445.213139] remoteproc remoteproc2: 40800000.dsp is available
[ 9445.226578] remoteproc remoteproc2: powering up 40800000.dsp
root@am57xx-evm:~# [ 9445.237799] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4074788
[ 9445.257079] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[ 9445.262982] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[ 9445.268881] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
[ 9445.284422] remoteproc remoteproc2: bad phdr da 0x9a000000 mem 0x1f3c
[ 9445.290914] remoteproc remoteproc2: Failed to load program segments: -22
[ 9445.299672] remoteproc remoteproc2: failed to unmap 4194304/0
[ 9445.318110] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
[ 9445.331895] omap_hwmod: mmu0_dsp1: _wait_target_disable failed

if you need more information, please let me know.

thanks,

Rong

0 Sahin Okur over 7 years ago in reply to Rong Wang2

TI__Mastermind 27355 points

Hi Rong,

Can you upload your map file and dts file?

You may need to define a TYPE_CARVEOUT section for the address as mentioned here:

0 Rong Wang2 over 7 years ago in reply to Sahin Okur

Expert 1420 points

am571x-aim.rarHi Sahin,

thank you so much for your quick response. attached is the dts file. this is my first time to modifiy resource table, not sure what dts file is for. do you mean if I modified rsc_table_dsp.h , I will need to update dts file accordingly? what should I added in dts file?

thank you!

Ron

0 Sahin Okur over 7 years ago in reply to Rong Wang2

TI__Mastermind 27355 points

Hi Ron,

Yes potentially the dts file may need to be adjusted when changing memory. I haven't had a chance to look at your dts file yet, but here are some resources that may help you with this in the meantime:

0 Rong Wang2 over 7 years ago in reply to Sahin Okur

Expert 1420 points

Hi Sahin,

thanks for your information.

I had read the articles from the link. I see following code in my dts file:

dsp1_cma_pool: dsp1_cma@99000000 {
compatible = "shared-dma-pool";
reg = <0x0 0x99000000 0x0 0x4000000>;
reusable;
status = "okay";

To my understanding, the memory range from 0x99000000~ 0x9D00000 is already reserved for dsp1 and I should be able to use it. so I modified my linker.cmd as following( the changes in red):

MEMORY
{

}

SECTIONS
{
.text: load >> EXT_CODE
.ti.decompress: load > EXT_CODE
.stack: load > EXT_DATA
GROUP: load > EXT_DATA
{
.bss:
.neardata:
.rodata:
}

.cinit: load > EXT_DATA
.pinit: load >> EXT_DATA
.init_array: load > EXT_DATA
.const: load >> EXT_DATA
.data: load >> EXT_DATA
.fardata: load >> EXT_DATA
.switch: load >> EXT_DATA
.sysmem: load > EXT_DATA
.far: load >> EXT_DATA
.args: load > EXT_DATA align = 0x4, fill = 0 {_argsize = 0x64; }
.cio: load >> EXT_DATA
.ti.handler_table: load > EXT_DATA
.c6xabi.exidx: load > EXT_DATA
.c6xabi.extab: load >> EXT_DATA
.tracebuf: load > EXT_DATA
.resource_table: load > 0x95000000, type = NOINIT
.vecs: load > EXT_CODE
.no_cache > DSP_NON_CACHE_DATA
xdc.meta: load > EXT_DATA, type = COPY

DPR
{
MemoirePartagee.obj
}> DSP_NON_CACHE_DATA

}

I set MAR153=0x0000000c; to make DSP_NON_CACHE_DATA uncacheable. however, I still got the same error when load dsp. error meesage are following:

root@am57xx-evm:~# ./binddsp
[83328.843957] omap-rproc 40800000.dsp: assigned reserved memory node dsp1_cma@99000000
[83328.853434] remoteproc remoteproc2: 40800000.dsp is available
[83328.866857] remoteproc remoteproc2: powering up 40800000.dsp
root@am57xx-evm:~# [83328.878157] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4074756
[83328.897384] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[83328.903301] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[83328.909202] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
[83328.925552] remoteproc remoteproc2: bad phdr da 0x99000000 mem 0x1f3c
[83328.932031] remoteproc remoteproc2: Failed to load program segments: -22
[83328.943264] remoteproc remoteproc2: failed to unmap 4194304/0
[83328.959373] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
[83328.973141] omap_hwmod: mmu0_dsp1: _wait_target_disable failed

what gone wrong? my map file is also attached.

AIM_G3_DSP37.rar

Thanks,

Ron

0 Sahin Okur over 7 years ago in reply to Rong Wang2

TI__Mastermind 27355 points

Hi Ron,

The address 0x99000000 is reserved for the vring buffers so we're better off using the 0x9a000000 region. The following should work:

In your resource table,

Add the region and size definitions:

#define DSP_NONCACHE_DATA 0x9A000000
#define DSP_NONCACHE_DATA_SIZE SZ_1M

Add the struct in my_resource_table, so something like "struct_fw_carveout dspnoncachedata_cout." Update the offset to account for this addition.

Add an entry to the offsets to entries for the new region and increment the number of entries to reflect this.

Add the carveout, like so:

 {
        TYPE_CARVEOUT,
        DSP_NONCACHE_DATA, 0,
        DSP_NONCACHE_DATA_SIZE, 0, 0, "DSP_NONCACHE_DATA",
    },

I'm not sure if the carveouts need to be grouped together, but it might be good to add these new entries after the last carveout just to be safe.

Add an entry in your linker command file for region 0x9A000000 and size 1MB.

I would try this both with cache enabled/disabled.

Are you disabling cache using Cache.setMarMeta?

0 Rong Wang2 over 7 years ago in reply to Sahin Okur

Expert 1420 points

Hi Sahin,

Thank you so much for your help!

I had updated my rsc_table_dsp.h as you told. now I did not see " bad address..." as before. but I still got some errors. see message below. Do I need to modify dts file as well? or something still missing or wrong in my resource table? attached are my updated rsc_table_dsp.h, original dts file and map file from test build. I also copy the message from the build without resources table updated at end.

//------------- message from new build---------------------------------//

root@am57xx-evm:~# ./loaddsp
[ 153.574236] omap-rproc 58820000.ipu: assigned reserved memory node ipu1_cma@9d000000
[ 153.586254] remoteproc remoteproc0: 58820000.ipu is available
[ 153.598470] remoteproc remoteproc0: Direct firmware load for dra7-ipu1-fw.xem4 failed with error -2
[ 153.608862] remoteproc remoteproc0: powering up 58820000.ipu
root@am57xx-evm:~# [ 153.612405] omap-rproc 55020000.ipu: assigned reserved memory node ipu2_cma@95800000
[ 153.613542] remoteproc remoteproc1: 55020000.ipu is available
[ 153.613851] omap-rproc 40800000.dsp: assigned reserved memory node dsp1_cma@99000000
[ 153.614893] remoteproc remoteproc2: 40800000.dsp is available
[ 153.660337] remoteproc remoteproc0: Direct firmware load for dra7-ipu1-fw.xem4 failed with error -2
[ 153.675240] remoteproc remoteproc0: request_firmware failed: -2
[ 153.683902] remoteproc remoteproc1: Direct firmware load for dra7-ipu2-fw.xem4 failed with error -2
[ 153.694337] remoteproc remoteproc1: powering up 55020000.ipu
[ 153.700059] remoteproc remoteproc1: Direct firmware load for dra7-ipu2-fw.xem4 failed with error -2
[ 153.711257] remoteproc remoteproc1: request_firmware failed: -2
[ 153.723810] remoteproc remoteproc2: powering up 40800000.dsp
[ 153.729514] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4074724
[ 153.747167] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[ 153.753070] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[ 153.758969] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
[ 153.769314] remoteproc remoteproc2: unsupported resource 6656
[ 153.776269] remoteproc remoteproc2: carveout rsc has non zero reserved bytes
[ 153.784476] remoteproc remoteproc2: Failed to process resources: -22
[ 153.799383] omap_hwmod: mmu1_dsp1: _wait_target_disable failed
[ 153.813141] omap_hwmod: mmu0_dsp1: _wait_target_disable failed

root@am57xx-evm:~# ./dsplog
cat: can't open '/sys/kernel/debug/remoteproc/remoteproc2/trace0': No such file or directory
root@am57xx-evm:~#

//--------------------------- message from old build----------------------------------//

root@am57xx-evm:~# ./binddsp
[ 1234.564767] omap-rproc 40800000.dsp: assigned reserved memory node dsp1_cma@99000000
[ 1234.573915] remoteproc remoteproc2: 40800000.dsp is available
[ 1234.587148] remoteproc remoteproc2: powering up 40800000.dsp
root@am57xx-evm:~# [ 1234.599153] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 4074540
[ 1234.618121] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
[ 1234.624783] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
[ 1234.630685] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
[ 1234.649378] virtio_rpmsg_bus virtio0: rpmsg host is online
[ 1234.655195] remoteproc remoteproc2: registered virtio0 (type 7)
[ 1234.665070] remoteproc remoteproc2: remote processor 40800000.dsp is now up

thanks,

Ron

AIM_G3_DSP35.rar 0827.rsc_table_dsp.h am571x-aim-dts.rar

0 Rong Wang2 over 7 years ago in reply to Rong Wang2

Expert 1420 points

Hi Sahin,

forgot to send you my linker.cmd file. attached here.

thanks,

Ron

0 Sahin Okur over 7 years ago in reply to Rong Wang2

TI__Mastermind 27355 points

Hi Ron,

Please see the attached resource table. I updated the offset to 26 to match the number of entries in the table and I grouped the carveouts together. The description of the function "rproc_handle_carveout" from the remoteproc source suggests that the carveouts should come first in the resource table, see snippet below.

* This function will handle firmware requests for allocation of physically
* contiguous memory regions.
*
* These request entries should come first in the firmware's resource table,
* as other firmware entries might request placing other data objects inside
* these memory regions (e.g. data/code segments, trace resource entries, ...).
*
* Allocating memory this way helps utilizing the reserved physical memory
* (e.g. CMA) more efficiently, and also minimizes the number of TLB entries
* needed to map it (in case @rproc is using an IOMMU). Reducing the TLB
* pressure is important; it may have a substantial impact on performance.
*/

/cfs-file/__key/communityserver-discussions-components-files/791/3073.0827.rsc_5F00_table_5F00_dsp.h

I don't think any changes need to be made to the dts file as long as we're using the CMA region that's already been allocated (0x99000000).

0 Rong Wang2 over 7 years ago in reply to Rong Wang2

Expert 1420 points

Hi Sahin,

I found a error in my rsc_table_dsp.h(index should be 26 instead of 2). after corrected this error, my dsp binary was loaded successfully. However, it seems my program has not access to this new memory region. just for example, if I just write a code , such as int test, test = 1; when I read back this variable from my console, it is something else, and the whole area looks like garbage.

root@am57xx-evm:~# ./omapconf dump 0x9a001d70 0x9a001df0
|----------------------------|
| Address (hex) | Data (hex) |
|----------------------------|
| 0x9A001D70 | 0xFFD7FB7F |
| 0x9A001D74 | 0x27FF5BED |
| 0x9A001D78 | 0x7FFFFFDF |
| 0x9A001D7C | 0x75FF3F47 |
| 0x9A001D80 | 0xFFB89DF1 |
| 0x9A001D84 | 0xFFFDF4BE |
| 0x9A001D88 | 0xFFEEDFDF |
| 0x9A001D8C | 0xFD56F67D |
| 0x9A001D90 | 0x7F2FFFF7 |
| 0x9A001D94 | 0xBEFFFCDD |
| 0x9A001D98 | 0x3DFFBFDB |
| 0x9A001D9C | 0xFBDDF757 |
| 0x9A001DA0 | 0xDFDE77F7 |
| 0x9A001DA4 | 0xDFEFFEDB |
| 0x9A001DA8 | 0xDF5FF63F |
| 0x9A001DAC | 0xDDEB4FD9 |
| 0x9A001DB0 | 0xCFB3DDBF |
| 0x9A001DB4 | 0xBFFE57F7 |
| 0x9A001DB8 | 0xFDFFD77F |
| 0x9A001DBC | 0xB571F7EC |
| 0x9A001DC0 | 0x6A1FEE7D |
| 0x9A001DC4 | 0xC51BFD7F |
| 0x9A001DC8 | 0x17F1FC71 |
| 0x9A001DCC | 0xF77DB3BC |
| 0x9A001DD0 | 0x877F5367 |
| 0x9A001DD4 | 0xDE8FD5F7 |
| 0x9A001DD8 | 0x4EFF7965 |
| 0x9A001DDC | 0x93AEFB77 |
| 0x9A001DE0 | 0xBCFD7DFF |
| 0x9A001DE4 | 0x75FFD6F4 |
| 0x9A001DE8 | 0xB7DCD354 |
| 0x9A001DEC | 0x3F6EDDFF |
| 0x9A001DF0 | 0xDF7FE67D |
|----------------------------|

no matter what my code write to this area, it did not work. is there anything missing in my resources table? or any update needed in dts file? is there any visural address mapping?

Thanks,

Ron

0 Rong Wang2 over 7 years ago

Expert 1420 points

Hi Sahin,

another issue I found is that if my ping pong buffers are in memory from CMA pool, EDMA did not load ping pong buffer address correctly. what could be the issue?

Thanks,

Ron

0 Rong Wang2 over 7 years ago in reply to Sahin Okur

Expert 1420 points

Hi Sahin,

I had tried your resources table, and it is ok for linux to load DSP binary. but I found EDMA did not load ping pong buffer address correctly. the right address of ping pong buffer should be erither 0x9A001f00(ping) or 0x9A001f20(pong). but what I see is that EDMA loaded address other than those, such as 0x9A001f04, 0x9A001f08, 0x9A001f24, etc depending the time I read it. Is this issue related to using memory from CMA?

do you mean I should use region allocated at 0x99000000 instead of 0x9500000 as in my linker.cmd as below:
also, I used MAR154= 0x0000000c; to disable cache for location 0x9A000000~ 0x9AFFFFFF instead of using Cache.setMarMeta. It works when I did not use memory from CMA pool and load DSP binary using emulator, but it seems to not working when EDMA ping pong buffer is in memory from CMA pool and loaded by linux.

Is this issue related to memory from CMA or MMU0, MMU1 enabled?

MEMORY
{

L2SRAM (RWX) : org = 0x800000, len = 0x40000
OCMC_RAM1 (RWX) : org = 0x40300000, len = 0x80000
OCMC_RAM2 (RWX) : org = 0x40400000, len = 0x100000
OCMC_RAM3 (RWX) : org = 0x40500000, len = 0x100000
EXT_CODE (RWX) : org = 0x95000000, len = 0x100000
EXT_DATA (RW) : org = 0x95100000, len = 0x100000
EXT_HEAP (RW) : org = 0x95200000, len = 0x300000
EXC_DATA (RW) : org = 0x9f060000, len = 0x10000
PM_DATA (RWX) : org = 0x9f070000, len = 0x20000
DSP_NON_CACHE_DATA (RW) : org = 0x9A000000, len = 0x100000

}

SECTIONS
{
.text: load >> EXT_CODE
.ti.decompress: load > EXT_CODE
.stack: load > EXT_DATA
GROUP: load > EXT_DATA
{
.bss:
.neardata:
.rodata:
}

.cinit: load > EXT_DATA
.pinit: load >> EXT_DATA
.init_array: load > EXT_DATA
.const: load >> EXT_DATA
.data: load >> EXT_DATA
.fardata: load >> EXT_DATA
.switch: load >> EXT_DATA
.sysmem: load > EXT_DATA
.far: load >> EXT_DATA
.args: load > EXT_DATA align = 0x4, fill = 0 {_argsize = 0x64; }
.cio: load >> EXT_DATA
.ti.handler_table: load > EXT_DATA
.c6xabi.exidx: load > EXT_DATA
.c6xabi.extab: load >> EXT_DATA
.tracebuf: load > EXT_DATA
.resource_table: load > 0x95000000, type = NOINIT
.vecs: load > EXT_CODE
.no_cache > EXT_DATA
xdc.meta: load > EXT_DATA, type = COPY

DPR
{
MemoirePartagee.obj
}> DSP_NON_CACHE_DATA

}

thanks for your help!

Ron

0 Sahin Okur over 7 years ago in reply to Rong Wang2

TI__Mastermind 27355 points

Discussion continued in this thread:

Processors

Processors forum

RTOS/AM5716: DSP firmware load error