OMAPL138 How to specify shared memory regions between Linux and DSP

Douglas Pepelko

Other Parts Discussed in Thread: OMAPL138, DA8XX, OMAP-L138

I currently have a OMAPL138 LCDK board and I am trying to get IPC communications working.
System setup on ARM is Linux with 3.14.18 with the patches for remote proc and rpmsg.
On the DSP side I am using CCS 6.1. BIOS 6.42.x IPC 3.36.x XDCtools version 3.30.6.67_core

Using the example MessageQ I have created a project with a custom rsc_table

#define DATA_DA 0xc3000000

#define RPMSG_VRING0_DA 0xc2000000
#define RPMSG_VRING1_DA 0xc2004000

I have also specified shared memory regions for the communication in DSP.cfg

var SR_0 = {
name: "SR_0", space: "data", access: "RWX",
base: 0xC2000000, len: 0x10000,
comment: "SR#0 Memory (64 KB)"
};
var SR0Mem = Program.cpu.memoryMap["SR_0"];
SharedRegion.setEntryMeta(0,
new SharedRegion.Entry({
name: "SR0",
base: SR0Mem.base,
len: SR0Mem.len,
ownerProcId: 0, // host
cacheEnable: false,
isValid: true
})
);

My boot command line:
bootargs=console=ttyS2,115200n8 ip=off root=/dev/ram0 rw rootfstype=ramfs
mem=32M@0xc0000000 mem=64M@0xc4000000 init=/sbin/init rproc_mem=32M@0xC2000000

Initially I was unable to load the DSP code and found this solution on the forums:

e2e.ti.com/.../274182

It specifies changing out "dma_alloc_coherent" for a "request_mem_region" and "ioremap_nocache"
in the handle_carveout code. I have done this and now can load the code correctly and can it on the DSP.

I am now trying to get MessageQ to communicate. I think the issue is that my VRING DA specifications are being
discarded in the remoteproc_core.c function "rproc_handle_vdev". Later in "rproc_alloc_vring" the function
"dma_alloc_coherent" is called again. *** It does not take any input from our DSP resource table and returns a
DMA address outside of our rproc_mem specification and outside of our resource table specification (see above). Later
that DMA address is returned to the remote proc as the DA.

This is the output of the parsing of the resource table

Jan 1 00:02:18 buildroot kern.debug kernel: ->rproc_parse_vring: vdev rsc: vring0: da c2000000, qsz 256, align 4096
Jan 1 00:02:18 buildroot kern.debug kernel: <-rproc_parse_vring len 256 aling 4096 rvdev c60f9400
Jan 1 00:02:18 buildroot kern.debug kernel: ->rproc_parse_vring: vdev rsc: vring1: da c2004000, qsz 256, align 4096
Jan 1 00:02:18 buildroot kern.debug kernel: <-rproc_parse_vring len 256 aling 4096 rvdev c60f9400

And here is the output of the allocation of one of those vring buffers (with dma_alloc_coherent)

Jan 1 00:02:19 buildroot kern.debug kernel: ->rproc_virtio_find_vqs nvqs 2
Jan 1 00:02:19 buildroot kern.debug kernel: vring1: va c8836000 dma c6080000 size 3000 idr 1
Jan 1 00:02:19 buildroot kern.debug kernel: <-rproc_alloc_vring
Jan 1 00:02:19 buildroot kern.debug kernel: vring1: va c8836000 qsz 256 notifyid 1
Jan 1 00:02:19 buildroot kern.debug kernel: <-rp_find_vq id 1 name [output]

I have tried to specify to Linux where physical memory is in two attempts:
1) In the resource table. This clearly does not work as the code just drops the DA.
2) On the boot command line rproc_mem=.... This does not appear to work either.

How does one tell Linux where our physical memory is?

-Doug

over 8 years ago

0 Robert Tivy over 8 years ago

TI__Mastermind 18260 points

Douglas Pepelko said:
var SR_0 = {
name: "SR_0", space: "data", access: "RWX",
base: 0xC2000000, len: 0x10000,
comment: "SR#0 Memory (64 KB)"
};

For Linux <-> DSP IPC SharedRegions aren't used. The remoteproc module will allocate the memory for the shared comm from CMA. A SharedRegion is needed only for DSP <-> DSP. Having this SR at 0xc2000000 could well be causing runtime problems. If you actually do need an SR for DSP/DSP comm, it needs to be somewhere else.

Douglas Pepelko said:
mem=32M@0xc0000000 mem=64M@0xc4000000 ... rproc_mem=32M@0xC2000000

This is the correct setting for rproc_mem and will cause a CMA area to be created at that location, from which remoteproc will allocate. However, CMA needs to be given memory that belongs to the kernel, and your 2 mem= entries cause 0xc2000000 to be *hidden* from the kernel. Please try again without the hole (maybe mem=128M@0xc0000000?)

Douglas Pepelko said:

Initially I was unable to load the DSP code and found this solution on the forums:

e2e.ti.com/.../274182

It specifies changing out "dma_alloc_coherent" for a "request_mem_region" and "ioremap_nocache"
in the handle_carveout code. I have done this and now can load the code correctly and can it on the DSP.

That Forum thread related specifically to internel L1 memory (on which you can't call dma_alloc_coherent()). While it may have appeared to work, it's not the right solution.

I get the feeling that the memory at 0xc2000000 is not being used with CMA. There's a file named <linux>/arch/arm/mach-davinci/devices-da8xx.c that contains a function da8xx_rproc_reserve_cma() to do this, but is it even called in the Linux 3.14 environment?

In that function is this line:
pr_info("%s: reserving 0x%lx @ 0x%lx...\n",
__func__, rproc_size, (unsigned long)rproc_base);
Does that get printed during early Linux boot?

I ask because for another architecture, the DRA7XX, all the remoteproc stuff is driven by the DT file in 3.14. OMAP-L138's remoteproc was developed long ago, well before use of the DT file for remoteproc.

Also, make sure the following message is *not* printed during early boot:
pr_err("%s: dma_declare_contiguous failed %d\n", __func__, ret);

Douglas Pepelko said:

I have tried to specify to Linux where physical memory is in two attempts:
1) In the resource table. This clearly does not work as the code just drops the DA.
2) On the boot command line rproc_mem=.... This does not appear to work either.

How does one tell Linux where our physical memory is?

Your two attempts should be your *one* attempt - both the resource table and remoteproc configured address need to be present and match exactly.

If you're still having issues, please post your complete resource table file.

Regards,

- Rob

0 Douglas Pepelko over 8 years ago in reply to Robert Tivy

Prodigy 130 points

Ok so I have set my bootargs to have these memory declarations:

mem=128M@0xC0000000 rproc_mem=32M@0xC2000000

I get the error from dma_declare_contiguous

da8xx_rproc_reserve_cma: reserving 0x2000000 @ 0xc2000000...
da8xx_rproc_reserve_cma: dma_declare_contiguous failed -38

Here is my rproc_table_omapl138.h

#define DATA_DA                 0xc3000000

#ifndef DATA_SIZE
//#  define DATA_SIZE  (SZ_1M * 15)
#  define DATA_SIZE  (SZ_1M * 8)
#endif

#define RPMSG_VRING0_DA         0xc3000000
#define RPMSG_VRING1_DA         0xc3004000

#define CONSOLE_VRING0_DA       0xc3008000
#define CONSOLE_VRING1_DA       0xc300C000

#define BUFS0_DA                0xc3040000
#define BUFS1_DA                0xc3080000

/*
 * sizes of the virtqueues (expressed in number of buffers supported,
 * and must be power of 2)
 */
#define RPMSG_VQ0_SIZE          256
#define RPMSG_VQ1_SIZE          256

#define CONSOLE_VQ0_SIZE        256
#define CONSOLE_VQ1_SIZE        256

/* flip up bits whose indices represent features we support */
#define RPMSG_IPU_C0_FEATURES         1

struct my_resource_table {
    struct resource_table base;

    UInt32 offset[13];

    /* data carveout entry */
    struct fw_rsc_carveout data_cout;

    /* rpmsg vdev entry */
    struct fw_rsc_vdev rpmsg_vdev;
    struct fw_rsc_vdev_vring rpmsg_vring0;
    struct fw_rsc_vdev_vring rpmsg_vring1;


    /* trace entry */
    struct fw_rsc_trace trace;
};

#define TRACEBUFADDR (UInt32)&xdc_runtime_SysMin_Module_State_0_outbuf__A
#define TRACEBUFSIZE 0x8000

#pragma DATA_SECTION(ti_ipc_remoteproc_ResourceTable, ".resource_table")
#pragma DATA_ALIGN(ti_ipc_remoteproc_ResourceTable, 4096)

struct my_resource_table ti_ipc_remoteproc_ResourceTable = {
	0x1, /* we're the first version that implements this */
    3, /* number of entries in the table */
	//2,   // number of entries in the table
    0, 0, /* reserved, must be zero */
    /* offsets to entries */
    {
        offsetof(struct my_resource_table, data_cout),
        offsetof(struct my_resource_table, rpmsg_vdev),
        offsetof(struct my_resource_table, trace),
    },

    {
    		TYPE_CARVEOUT, DATA_DA, DATA_DA, DATA_SIZE, 0, 0, "IPU_MEM_DATA",
    },

    /* rpmsg vdev entry */
    {
        TYPE_VDEV, VIRTIO_ID_RPMSG, 0,
        RPMSG_IPU_C0_FEATURES, 0, 0, 0, 2, { 0, 0 },
        /* no config data */
    },
    /* the two vrings */
    { RPMSG_VRING0_DA, 4096, RPMSG_VQ0_SIZE, 1, 0 },
    { RPMSG_VRING1_DA, 4096, RPMSG_VQ1_SIZE, 2, 0 },


    {
        TYPE_TRACE, TRACEBUFADDR, TRACEBUFSIZE, 0, "trace:dsp",
    },
};

0 Robert Tivy over 8 years ago in reply to Douglas Pepelko

TI__Mastermind 18260 points

Douglas Pepelko said:

da8xx_rproc_reserve_cma: reserving 0x2000000 @ 0xc2000000...
da8xx_rproc_reserve_cma: dma_declare_contiguous failed -38

The only way I can see -38 (ENOSYS) returned from that function is if you don't have CONFIG_DMA_CMA configured. Can you check that?

Douglas Pepelko said:
#define RPMSG_VRING0_DA 0xc3000000 #define RPMSG_VRING1_DA 0xc3004000

If the CMA area is at 0xc2000000 then you need to also have the VRING DA there (it needs to be set to the base addr of your CMA area).

Also, your CARVEOUT can't come from the same address as the VRING:

#define DATA_DA 0xc3000000

I'm not sure, but remoteproc will allocate your CARVEOUT from the CMA area and you will get that address overwritten in the da field of the CARVEOUT.

Regards,

- Rob

Processors

Processors forum

OMAPL138 How to specify shared memory regions between Linux and DSP