AM623: AM623: uisng GPMC to interface with FPGA

Hello,

Many customers are already using the GPMC interface to communicate with an FPGA.

In synchronous communication, data is transferred between the SoC and FPGA with reference to a clock, whereas in asynchronous communication, data is transferred without a clock between the SoC and FPGA.

The above requirement is possible using GPMC. You can interface devices such as PSRAM or NOR a direct connection to the SoC side. These devices are treated as memory-mapped interfaces.

Once the GPMC is initialized, reading and writing data using the chip-select (CS) base address is possible.

I am not sure which memory interfaces are supported on the Linux side. Hence, I am routing your query to a Linux expert for further clarification.

Regards,
Anil

Ok, thanks Anil.

I think an application notes on GPMC interface will be helpful to us.

We are still concerning about whether it is able to support the required throughput for our application, not just about the GPMC raw speed, but whether Linux is able to cope with that throughput or not as a complete data chain.

So, we are thinking to get a hardware evaluation kit to do a proof of concept. The best is with schematic available for our hardware design reference.

Probably we will have a Linux application to continuously pull the samples out from the FIFO into the memory, make sure it is fast enough and not overflow the FIFO.

Hi Bin,

Our processor board currently only routes the OSPI lines to external connector that can be used to interface with FPGA. If OSPI is able to work, then we do not need to go to GPMC interface which requires hardware board change.

Understand from the e2e thread, it is possible to set DMA up once and then fire and forget. Then the DMA will continue to run in CYCLIC mode and it can be external triggered by side band event generated by FPGA to an AM62x GPIO pin. At each transfer complete, that will be a callback.

For our case, it will be one callback per microsecond. We need that callback to increment a counter so that we know the current DMA progress in the memory. The counter shall be incremented for each sample transferred by DMA.

The whole sample transfer process does not involve CPU, may be just the callback to increment the counter which I believe is in DMA ISR context. 

I think above is pretty much what we need. Great that it works with GPMC interface!

To get the above to work, a patch is needed to apply to current Linux kernel DMA driver for AM62x. It will be good if you can provide the final patch for the implementation details in that e2e thread because the discussion is a bit too long for me to follow.

Also the discussion in that e2e thread seems to be based on GPMC interface, I still don't see how does it work with OSPI interface? I understand we still need CPU to setup and start each OSPI transfer. This is the whole reason we start to consider the GPMC interface.

rgds,

kc Wong

Hi KC,

Kiung Chung Wong said:

Understand from the e2e thread, it is possible to set DMA up once and then fire and forget. Then the DMA will continue to run in CYCLIC mode and it can be external triggered by side band event generated by FPGA to an AM62x GPIO pin. At each transfer complete, that will be a callback.

Correct.

Kiung Chung Wong said:

For our case, it will be one callback per microsecond. We need that callback to increment a counter so that we know the current DMA progress in the memory. The counter shall be incremented for each sample transferred by DMA.

But this won't work in Linux. The callback is handled by CPU but A53 in Linux won't be able to switch the context to handle a ISR within a microsecond.

If your FPGA has to be notified in reading every sample/FIFO in Linux, I think you would have to change your FPGA design, the current 8-bytes FIFO design doesn't fit in Linux.

Similar to the FPGA design in the referred e2e thread, you probably design your FPGA interface in memory mode, something like PSRAM or NOR as Anil mentioned above. The memory size, for example, is 2048 bytes / 256 samples, so that the DMA completion callback will be handled by CPU in every 256 microseconds. Your FPGA internally tracks the counter for every 256 samples.

However, I am not familiar the OSPI interface, and not sure how to handle/read memory mode from the OSPI interface.

Hi Bin,

In that case, we just focus on the GPMC interface here. Yes, we definitely want the FPGA registers to be memory mapped into the A53 Linux address space.

Ok, so sample by sample read won't work for Linux. We will try to read in block of 256-sample then.

As we need to modify our current processor board before we can try this. Is there hardware evaluation kit available for us to do proof of concept? The best is with schematic available for our hardware design reference.

rgds,

kc Wong

Hi KC,

Kiung Chung Wong said:

Yes, we definitely want the FPGA registers to be memory mapped into the A53 Linux address space.

No only the FPGA registers, the entire data memory region has to be available to A53 too, so that A53 can 'memcpy' the whole 256*8 bytes. You might also consider ping-pong buffers scheme inside the FPGA, so that the incoming data are still captured in FPGA while A53/DMA are reading them.

Kiung Chung Wong said:

As we need to modify our current processor board before we can try this. Is there hardware evaluation kit available for us to do proof of concept? The best is with schematic available for our hardware design reference.

Unfortunately we don't have an AM62 EVM which has the full GPMC interface exposed. But I still think if the scenario can be implemented on the AM62x OSPI interface...

Hi Bin,

Just clarification on the understanding ...

The scheme discussed previously does not require A53 to do anything, the DMA will be external triggered and samples will be transferred to memory by DMA through GPMC interface.

But, if the entire data memory region is mapped available to A53, the A53 needs to call 'memcpy'? Can you elaborate more if the scheme is different from what has been discussed previously.

Any TI expert we can talk to use your DMA patch with AM62x OSPI interface?

Our understanding the DMA only available in direct mode for OSPI interface. And that requires FPGA to emulate as flash, which unnecessarily complicated the FPGA design.

Also the direct mode is for XIP (execute in place), which means read-only memory region. But, we need to read/write the FPGA registers.

As it is a read-only memory region, the controller may perform speculative prefetch. But, this is not static data, it is dynamic live samples from the front end.

Hope some TI expert can help us on above doubts/questions ...

rgds,

kc Wong

Hi KC,

Kiung Chung Wong said:

But, if the entire data memory region is mapped available to A53, the A53 needs to call 'memcpy'? Can you elaborate more if the scheme is different from what has been discussed previously.

Sorry for the confusion with the reference to 'memcpy'. When DMA does the data transfer, A53 doesn't do so, A53 only needs to handle the DMA completion ISR. I mentioned memcpy only meant the DMA channel will be programmed to access the entire 256*8-bytes memory region, not only the 8-bytes of the FIFO interface.

Kiung Chung Wong said:

Our understanding the DMA only available in direct mode for OSPI interface. And that requires FPGA to emulate as flash, which unnecessarily complicated the FPGA design.

Yes it seems to me the FPGA should emulate as flash. But if this complicates the FPGA design, we can stick with using GPMC interface. I was thinking if you don't need to redesign your board.

Ok, thanks for the clarification. Then, I think we just need a hardware to try the GPMC interface.

As for OSPI interface, even we can emulate the FPGA as flash, how about the read-only XIP mode and the speculative prefetch by controller?

I am not sure are they are real issues, or they are some workaround for them?

Yes, we pretty much want to stay with OSPI interface if possible, so that we don't need to redesign the processor board.

I barely touch the OSPI interface on AM6xx devices, and am not sure if XIP has to be used in flash mode, but I know serial NAND/NOR devices can be attached to the AM62x OSPI interface, and they take read/write access.

Ya, we actually engage the design partner for this work.

From the start, the design partner engineer, Maneesh had told us DMA won't work for OSPI interface based on the discussion in below e2e thread.

e2e.ti.com/.../am623-am623-dma

Let us know if the understanding is not correct so that we can start to look again using DMA for OSPI interface.

Hi KC,

Kiung Chung Wong said:

From the start, the design partner engineer, Maneesh had told us DMA won't work for OSPI interface based on the discussion in below e2e thread.

e2e.ti.com/.../am623-am623-dma

Thanks for bringing up the previous thread. The main point of my comments in that thread is that kernel DMA driver doesn't support DMA_DEV_TO_MEM transfer type in the OSPI device driver, which would be needed for this FPGA use case.

For this FPGA data streaming use case, kernel DMA driver would have to support DMA_DEV_TO_MEM transfer in CYCLIC mode, which is not available in current kernel. This was the reason I implemented it in the e2e thread I mentioned above in my first response. This custom implementation technically could be used on OSPI interface too, however the OSPI device driver would have to be modified to adopt DMA CYCLIC mode due to the design in kernel SPI framework.

So comparing the two options for your FPGA use case - using OSPI and GPMC, I would recommend to pick GPMC. We already had several customers used my custom DMA driver mentioned above on their GPMC-FPGA design, but I am not aware of any customer used FPGA on OSPI interface, regardless using DMA or not.

Hi KC,

It seems at the moment you are trying to find all the possible solutions for your usecase. Please understand all the pros/cons of each solution and pick the one fit your project the best.

In terms of the DMA solution on GPMC discussed here, even with my reference implementation mentioned in my first response above, there is still quite amount of software development/integration work you would have to do, not to mention my DMA reference implementation was done on kernel 6.1, and the K3 DMA driver already has structural change in kernel 6.12. So you would have to be comfortable on kernel DMAEngine framework and AM62x DMA controller to do the work.

Ok, Bin. Understand that both solution also need custom implemention to patch the current kernel for our use case.

Else, do we have other simpler choice that doesn't require a lot of work on AM62x? Or there is other SOC better suit for this use case?

By the way, don't see GPMC mentioned in the Linux SDK documentation? Is GPMC actually supported on AM62x?

https://software-dl.ti.com/processor-sdk-linux/esd/AM62X/latest/exports/docs/linux/Foundational_Components_Kernel.html

OSPI interface would be our more preferred option for now because that doesn't require hardware change. Of course, we have to make the hardware change if eventually proven that OSPI option is not really feasible.

Understand that after modifying kernel DMA driver to support DMA_DEV_TO_MEM transfer in CYCLIC mode, that's possible to use the OSPI interface for our use case.

But, how does the scheme work? Does the DMA still be external triggered?

Or the external trigger is used to trigger the OSPI transfer. Then, let the OSPI to trigger the DMA transfer.

If that's the case, I believe A53 is required to setup and start OSPI transfer on each external trigger. That seems more A53 involvement as compared to GPMC option.

Then ultimate question is still can A53 Linux able to catch up with the required throughput?

And also not forget we need the same OSPI interface for the other FPGA registers read and write.

Is my understanding correct?

Or is there a way to setup OSPI once and then start and forget?

Hi KC,

Kiung Chung Wong said:

Else, do we have other simpler choice that doesn't require a lot of work on AM62x? Or there is other SOC better suit for this use case?

I think you have another thread discussing PRU. I guess it would be an option, but I don't know PRU deep enough to tell if it can meet your timing requirement, not to mention you would have implement the PRU program to read the FPGA and communicate with A53/Linux.

All the Sitara processors have similar common peripherals, I don't see any other SoC can do it differently.

Kiung Chung Wong said:

By the way, don't see GPMC mentioned in the Linux SDK documentation? Is GPMC actually supported on AM62x?

It is not documented in the SDK doc, only because the AM62x SK EVM doesn't expose the GPMC interface for software validation automation. But yes, AM62x GPMC is supported.

Kiung Chung Wong said:

But, how does the scheme work? Does the DMA still be external triggered?

Yes, your usecase requires DMA in cyclic mode, and cyclic mode requires external trigger regardless the interface which the FPGA is attached too.

Kiung Chung Wong said:

Or the external trigger is used to trigger the OSPI transfer. Then, let the OSPI to trigger the DMA transfer.

In the current kernel SPI framework, yes this should be the scheme. However, this likely won't meet the timing requirement in your use case. This is why I said the kernel SPI framework has to be patch to use DMA cyclic mode. But I am not aware of that similar work has ever been done, so I am really sure the exact scope of the work, kind of risky. So I recommend GPMC over OSPI for your usecase.

Hi Bin,

Ya, I am also exploring the other cores on the AM623 SOC. But, I understand there is no SDK support for OSPI interface on both M4F and PRU. The SDK is only supported on R5F. But, I am kind of hesitating using R5F because it is running the DM task.

 AM623: using DM R5F to continuously read data from FPGA 

I know it is kind of risky, but since our current processor board is already with OSPI interface, we are still recommending our design partner to give it a try on the OSPI-FPGA design with your custom DMA driver integrated. Hope you can help us on that work.

We will see how far the OSPI-FPGA design can go. If eventually proven that this design is not feasible, at least we have substantial data to convince the management for the hardware change to go for GPMC-FPGA design.

By the way, which hardware are you using when you are helping the other customer developing the custom DMA driver for the GPMC-FPGA design?

I came across the below 2 eval HW, any recommendation which one to use for trying the GPMC-FPGA design? Or you have other recommndation?
(a) https://www.phytec.com/product/phycore-am62x/
(b) https://www.tq-group.com/en/products/tq-embedded/cpu-families/am62x/

I am out of office for the holidays, my response will be delayed for a few days. 

Ok, thanks for letting me know.

Hi KC,

Kiung Chung Wong said:

But, I am kind of hesitating using R5F because it is running the DM task.

Agreed, DM R5F would be the last option for me to run any application on it.

Kiung Chung Wong said:

but since our current processor board is already with OSPI interface, we are still recommending our design partner to give it a try on the OSPI-FPGA design with your custom DMA driver integrated. Hope you can help us on that work.

I am able to help in DMA driver customization, but as I mentioned previously, the kernel OSPI framework would have be customized too to use DMA cyclic mode, I am unable to help in this regards.

Kiung Chung Wong said:

We will see how far the OSPI-FPGA design can go. If eventually proven that this design is not feasible, at least we have substantial data to convince the management for the hardware change to go for GPMC-FPGA design.

Sounds good.

Kiung Chung Wong said:

By the way, which hardware are you using when you are helping the other customer developing the custom DMA driver for the GPMC-FPGA design?

I didn't use any particular GPMC-FPGA design (because we don't have one for AM62x platform), instead, I created a virtual GPMC-FPGA kernel driver which feeds data to a predefined memory region, so that my DMA driver reads data from it.

Hi Bin,

Bin Liu said:

I am able to help in DMA driver customization, but as I mentioned previously, the kernel OSPI framework would have be customized too to use DMA cyclic mode, I am unable to help in this regards.

OK. In that case, we will need to maintain both the custom DMA driver and also the custom kernel OSPI framework for OSPI-FPGA design.

Bin Liu said:

I didn't use any particular GPMC-FPGA design (because we don't have one for AM62x platform), instead, I created a virtual GPMC-FPGA kernel driver which feeds data to a predefined memory region, so that my DMA driver reads data from it.

This is interesting. Can you share how you do it? We might need that if we decide to go for GPMC-FPGA design.

Attached below is my dma test driver.

/*
 *  DMA DEV_TO_MEM testing
 */
#include <linux/dma-mapping.h>
#include <linux/dmaengine.h>
#include <linux/miscdevice.h>
#include <linux/platform_device.h>
#include <linux/fs.h>
#include <linux/of.h>

int dma_mode = 0;
module_param_named(mode, dma_mode, uint, 0644);
MODULE_PARM_DESC(mode, "dma mode: 0 - cyclic, 1 - sg");

int bufsz = 2;
module_param_named(buf, bufsz, uint, 0644);
MODULE_PARM_DESC(buf, "tgt buffer size in times of src fifo (default 2)");

#define FIFO_SIZE	4	/* in bytes, must be less than PAGE_SIZE/2 */

struct dmatest_info {
	struct platform_device *pdev;
	struct dma_chan	       *dma;
	dma_cookie_t cookie;
	char* buf;
	dma_addr_t buf_dma;
	int buf_size;
	char *fifo;
	dma_addr_t fifo_dma;
	int fifo_size;
	struct dma_async_tx_descriptor *tx;
	int cnt;
	int tdowned;
};

static void dmatest_rx_callback(void *param)
{
	struct dmatest_info *info = param;

	dev_info(&info->pdev->dev, "in dma callback(%d)\n", info->cnt++);
	print_hex_dump(KERN_INFO, "", DUMP_PREFIX_OFFSET, 16, 1,
			info->buf, 32, true);

	if (info->cnt > 2) {
		dev_info(&info->pdev->dev, "=== term dma\n");
		dmaengine_terminate_async(info->dma);
		info->tdowned = 1;
	} else {
		memset(info->buf, 0, 32);
	}
}

static int config_dma(struct dmatest_info *info)
{
        struct device *dev = &info->pdev->dev;
        struct dma_slave_config conf;
        struct dma_async_tx_descriptor *tx;
        int ret;

        memset(&conf, 0, sizeof(conf));
        conf.direction = DMA_DEV_TO_MEM;
        conf.src_addr_width = 2;
        conf.src_maxburst = info->fifo_size / conf.src_addr_width;
        conf.src_addr = info->fifo_dma;
        ret = dmaengine_slave_config(info->dma, &conf);
        if (ret < 0) {
		dev_err(dev, "dma slave config failed (%d)\n", ret);
                return ret;
	}

	if (dma_mode) {
		tx = dmaengine_prep_slave_single(info->dma, info->buf_dma,
				info->buf_size, DMA_DEV_TO_MEM,
				DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
		dev_info(dev, "slave single dma preparation passed\n");
	} else {
		tx = dmaengine_prep_dma_cyclic(info->dma, info->buf_dma,
				info->buf_size, conf.src_maxburst,
				DMA_DEV_TO_MEM, DMA_PREP_INTERRUPT);
		dev_info(dev, "cyclic dma preparation passed\n");
	}

        if (!tx) {
                dev_err(dev, "dma preparation failed\n");
                return -ENODEV;
        }

	info->tx = tx;
        tx->callback = dmatest_rx_callback;
        tx->callback_param = info;
        info->cookie = dmaengine_submit(tx);

        return 0;
}

static int __init dev_probe(struct platform_device *pdev)
{
	struct device *dev = &pdev->dev;
	struct dmatest_info *info;
	dma_cap_mask_t mask;
	int ret;

	dev_info(dev, "dev_probe()");
	info = devm_kzalloc(dev, sizeof(struct dmatest_info), GFP_KERNEL);

	if (!info)
		return -ENOMEM;

	info->pdev = pdev;

	/* Allocate a page. The first half is the buffer. The second half
	 * is the FIFO, but only use the FIFO_SIZE */
	info->buf = dma_alloc_coherent(dev, PAGE_SIZE, &info->buf_dma,
			GFP_KERNEL);
	if (!info->buf) {
		dev_err(dev, "dma buffer allocation failed\n");
		return -ENOMEM;
	}
	info->buf_size = FIFO_SIZE * bufsz;
	info->fifo = info->buf + PAGE_SIZE/2;
	info->fifo_dma = info->buf_dma + PAGE_SIZE/2 + 2;
	info->fifo_size = FIFO_SIZE;

	dma_cap_zero(mask);
	dma_cap_set(DMA_SLAVE, mask);
	dma_cap_set(DMA_CYCLIC, mask);

	info->dma = dma_request_chan(dev, "rx");
	if (IS_ERR(info->dma)) {
		dev_err(dev, "dma channel request failed (%ld)\n",
				PTR_ERR(info->dma));
		return PTR_ERR(info->dma);
	}

	ret = config_dma(info);
	if (ret) {
		dev_err(dev, "config dma failed (%d)\n", ret);
		goto err_dma;
	}

	platform_set_drvdata(pdev, info);

	memcpy(info->fifo, "1234567890abcdef", 16);
	print_hex_dump(KERN_INFO, "", DUMP_PREFIX_OFFSET, 16, 1,
			info->fifo, 32, true);
	print_hex_dump(KERN_INFO, "", DUMP_PREFIX_OFFSET, 16, 1,
			info->buf, 32, true);

	dma_async_issue_pending(info->dma);
	dev_info(dev, "dma pending queue activated\n");
	return 0;

err_dma:
	dma_release_channel(info->dma);
	return ret;
}

static int __exit dev_remove(struct platform_device *pdev)
{
	struct dmatest_info *info = platform_get_drvdata(pdev);

	dev_info(&pdev->dev, "dev_remove()");
	if (info->dma) {
		if (info->tdowned)
			dmaengine_synchronize(info->dma);
		else
			dmaengine_terminate_sync(info->dma);
		dma_release_channel(info->dma);
	}
	if (info->buf)
		dma_free_coherent(&pdev->dev, PAGE_SIZE, info->buf, info->buf_dma);

	return 0;
}

static const struct of_device_id of_ids[] = {
	{ .compatible = "ti,dma_test" },
	{},
};
MODULE_DEVICE_TABLE(of, of_ids);

static struct platform_driver dmatest_driver = {
	.probe = dev_probe,
	.remove = dev_remove,
	.driver = {
		.name = "dmatest",
		.of_match_table = of_ids,
		.owner = THIS_MODULE,
	}
};

module_platform_driver(dmatest_driver);

MODULE_LICENSE("GPL v2");

Thanks for sharing  ...