Linux/PROCESSOR-SDK-AM57X: Remoteproc resource table question

Sorry for the delay. We will try to get you some information soon.

RonB,

Thanks in advance!

Tim,

I did some work using M4: there is a resource table built into M4 SYSBIOS side; Linux downloads M4 code to IPU1 via remoteproc; the M4 uses OCMC memory. I think it also applies to your case with DSP.

Attached is the original and modified resource table. I followed this wiki:

You can see that an entry devmem14 was added with "TYPE_CARVEOUT", not "TYPE_DEVMEM".

Then, I have a config.bld which defines the memory map and code placement I modified to keep code/data into OCMC. I used evmDRA7XX_ExtMemMapIpu1, for you would be evmDRA7XX_ExtMemMapDsp. The files are also attached.

Finally I rebuilt the M4 SYSBIOS application and copied into /lib/firmware. The Linux was able to download it and run.

Regards, Eric

Common.zip

Eric,

  thank you for the effort - sorry for the delay I haven't been at the office the recent days.

The resource tables you are providing are very similar to the ones I already got from you. Unfortunately these won't help either. Are you sure the mapped memory at 0x60400000 from IPUs pov is really OCMC? The remoteproc code that handles carveouts doesn't care for the provided physical address. I doubt it will map anything else than standard Memory from the CMA region defined in the linux device tree.

see the remoteproc code excerpt from linux-rt-sdk 3.1.0.6, driver/remoteproc/remoteproc_core.c:701 ff.

/**
 * rproc_handle_carveout() - handle phys contig memory allocation requests
 * @rproc: rproc handle
 * @rsc: the resource entry
 * @avail: size of available data (for image validation)
 *
 * This function will handle firmware requests for allocation of physically
 * contiguous memory regions.
 *
 * These request entries should come first in the firmware's resource table,
 * as other firmware entries might request placing other data objects inside
 * these memory regions (e.g. data/code segments, trace resource entries, ...).
 *
 * Allocating memory this way helps utilizing the reserved physical memory
 * (e.g. CMA) more efficiently, and also minimizes the number of TLB entries
 * needed to map it (in case @rproc is using an IOMMU). Reducing the TLB
 * pressure is important; it may have a substantial impact on performance.
 */
static int rproc_handle_carveout(struct rproc *rproc,
				 struct fw_rsc_carveout *rsc,
				 int offset, int avail)
{
	struct rproc_mem_entry *carveout, *mapping;
	struct device *dev = &rproc->dev;
	dma_addr_t dma;
	void *va;
	int ret;

	if (sizeof(*rsc) > avail) {
		dev_err(dev, "carveout rsc is truncated\n");
		return -EINVAL;
	}

	/* make sure reserved bytes are zeroes */
	if (rsc->reserved) {
		dev_err(dev, "carveout rsc has non zero reserved bytes\n");
		return -EINVAL;
	}

	dev_dbg(dev, "carveout rsc: da %x, pa %x, len %x, flags %x\n",
		rsc->da, rsc->pa, rsc->len, rsc->flags);

	carveout = kzalloc(sizeof(*carveout), GFP_KERNEL);
	if (!carveout)
		return -ENOMEM;

	va = dma_alloc_coherent(dev->parent, rsc->len, &dma, GFP_KERNEL);
	if (!va) {
		dev_err(dev->parent, "dma_alloc_coherent err: %d\n", rsc->len);
		ret = -ENOMEM;
		goto free_carv;
	}

	dev_dbg(dev, "carveout va %p, dma %pad, len 0x%x\n", va,
		&dma, rsc->len);

	/*
	 * Ok, this is non-standard.
	 *
	 * Sometimes we can't rely on the generic iommu-based DMA API
	 * to dynamically allocate the device address and then set the IOMMU
	 * tables accordingly, because some remote processors might
	 * _require_ us to use hard coded device addresses that their
	 * firmware was compiled with.
	 *
	 * In this case, we must use the IOMMU API directly and map
	 * the memory to the device address as expected by the remote
	 * processor.
	 *
	 * Obviously such remote processor devices should not be configured
	 * to use the iommu-based DMA API: we expect 'dma' to contain the
	 * physical address in this case.
	 */
	if (rproc->domain) {
		mapping = kzalloc(sizeof(*mapping), GFP_KERNEL);
		if (!mapping) {
			dev_err(dev, "kzalloc mapping failed\n");
			ret = -ENOMEM;
			goto dma_free;
		}

		ret = iommu_map(rproc->domain, rsc->da, dma, rsc->len,
				rsc->flags);
		if (ret) {
			dev_err(dev, "iommu_map failed: %d\n", ret);
			goto free_mapping;
		}

		/*
		 * We'll need this info later when we'll want to unmap
		 * everything (e.g. on shutdown).
		 *
		 * We can't trust the remote processor not to change the
		 * resource table, so we must maintain this info independently.
		 */
		mapping->da = rsc->da;
		mapping->len = rsc->len;
		list_add_tail(&mapping->node, &rproc->mappings);

		dev_dbg(dev, "carveout mapped 0x%x to %pad\n",
			rsc->da, &dma);
	}

	/*
	 * Some remote processors might need to know the pa
	 * even though they are behind an IOMMU. E.g., OMAP4's
	 * remote M3 processor needs this so it can control
	 * on-chip hardware accelerators that are not behind
	 * the IOMMU, and therefor must know the pa.
	 *
	 * Generally we don't want to expose physical addresses
	 * if we don't have to (remote processors are generally
	 * _not_ trusted), so we might want to do this only for
	 * remote processor that _must_ have this (e.g. OMAP4's
	 * dual M3 subsystem).
	 *
	 * Non-IOMMU processors might also want to have this info.
	 * In this case, the device address and the physical address
	 * are the same.
	 */
	rsc->pa = dma;

	carveout->va = va;
	carveout->len = rsc->len;
	carveout->dma = dma;
	carveout->da = rsc->da;

	list_add_tail(&carveout->node, &rproc->carveouts);

	return 0;

free_mapping:
	kfree(mapping);
dma_free:
	dma_free_coherent(dev->parent, rsc->len, va, dma);
free_carv:
	kfree(carveout);
	return ret;
}

rsc->pa is the structure member that you initialized in the resource table with L3_OCMC_RAM (0x40400000). It won't ever get read but overwritten in the end with the physical address of the CMA region remoteproc is mapping instead.

Still no solution for this problem. I think this is mainly a linux problem - remoteproc simply does not support carveouts from several distinct memory regions - I think it was never intended to be controlled from the slave CPU regarding physical memory addresses. Despite that it is a valid use case and I'm pretty sure I'm not the only one wanting to carve out locations in different memory types/regions.

Best,

      Tim

PS.: following the resource table howto placing a carveout after the devmem sections is bad habit and putting a carveout into the devmem sub-structure is a bit misleading...  just my 2cents