Real-world DDR performance

Hi Carsten,

Windows Embedded is not supported by TI, and this forum supports Linux only. I can give you the Linux benchmarks for comparison: http://processors.wiki.ti.com/index.php/Processor_SDK_Linux_Kernel_Performance_Guide

The LMBench metrics table provides this information. Different benchmarks are explained here: http://www.bitmover.com/lmbench/man_lmbench.html

Carsten,

This question comes up very regularly across all platforms (ARMs, DSPs, etc.).  So frequently in fact that I wrote a wiki page to try and explain it clearly:

http://processors.wiki.ti.com/index.php/Common_Issue_Resulting_in_Slow_External_Memory_Performance

As a very high level data point, without the cache enabled I typically see performance deltas on the order of 100x.  So I'm not at all surprised by the performance you're seeing.  It's inherent in non-cached accesses.  On a related note, if you configure the memory as "device" (i.e. buffered but non-cached) you can substantially increase your write performance since that will allow the ARM to make use of the write buffer, including write-merging, etc.

Brad

DO NOT use uncached memory for DMA! The access speed of the CPU into this type of memory is slow.

Instead, use normal cached memory:

memptr = kmalloc(SIZE_XXX, GFP_DMA);

phys_ptr = dma_map_single(NULL, memptr, SIZE_XXX, DMA_xxx_DEVICE);

and then,  inside your CPU function:

dma_sync_single_for_cpu(NULL, phys_ptr, SIZE_XXX, DMA_xxx_DEVICE);

do_something(memptr);

dma_sync_single_for_device(NULL, phys_ptr, SIZE_XXX, DMA_xxx_DEVICE);

Carsten Hansen said:

If I understand you correctly, DDR bandwidth of 40-50 MB/s is not surprising on the platforms we use.

You have misinterpreted me here...  The AM335x DDR bandwidth is excellent.  There is no issue there.  I am not surprised with your measurement due to the fact that you're using strongly ordered accesses.

Carsten Hansen said:

The reason we are so interested in uncached rather than cached memory bandwidth is because of the requirements of our application on the AM335x, which are:
* Receive at least 10 (ideally, 20) MBytes/s on one LAN interface (that's DMA from EMAC to DDR3)
* Process data on the CPU (read data back in, none of which are in cache)
* Send all the data out on the other LAN interface (so invalidate cache + DMA from DDR3 to EMAC)

I agree with where you started, i.e. that when EMAC puts data into DDR3 that it will not be in the cache (yet).  However, the main point I'm trying to make is that when you go to read that data, it will be roughly 100x faster if the area is marked as cacheable.

Bottom line, I see no reason in the example above for you to be configuring the memory to be non-cacheable (strongly ordered).  This is decreasing your throughput substantially.  Before your second step (reading the data), you should invalidate that memory region so that you're sure you are getting "fresh" data.  And once you're done making any modifications you should perform a writeback of the data to make sure it's sitting in the DDR3 ready for the EMAC.

Carsten Hansen said:

Going back to the wiki page, I'm not sure CPU pipelining would be an issue. The test code uses a series of 100 ldm/stm instructions, then repeats in a loop, it's difficult to imagine this would affect the pipeline much.

Please mark the memory as cacheable and try it again.

Carsten Hansen said:

Cached performance, as mentioned previously, is completely satisfactory, so we have little doubt that the MMU has been set up correctly.

I think we are having a disconnect in terminology.  Let's please try to use the following terms to discuss:

• Cacheable: MMU page marked so as to allow its contents to be allocated into the cache
• Non-cacheable (or "strongly ordered" using ARM terminology): MMU page marked so as to NOT allow its contents to be allocated into the cache
• Cache hit:  Data already in the cache (extremely fast access)
• Cache miss: Data non in the cache, its corresponding cache line is fetched from external memory and allocated in the cache

What I suspect might be the case, is that you're trying to quantify the penalty of a cache miss by doing a non-cacheable memory test.  If that's the case, I don't recommend that...  You would be better off invalidating a big chunk of memory and then reading it (while still marked cacheable).

Good to hear!

Some final tips:

Don't pay too much attention to the Architecture Reference Manual v7. It is overly broad and general, while the true behaviour of the Cortex-A8 is much simpler and more restrictive. If possible, locate revision B (2011) instead of the latest to avoid having to read around all the stuff about virtualization. Keep in mind the A8 is the very first ARMv7-A processor, and unlike later members still executes in-order. It never performs speculative data accesses (only instruction prefetching).

If you want more insight into the A8's behaviour, this paper is a good read. Also get the A8 TRM of course, but don't trust it too much.

If you're processing a dataset that doesn't fit in L2 cache, the A8 has a "prefetch engine" which can load data into or evict data out of L2 cache in the background. Unfortunately its completion irq is not physically connected to anything afaik in TI's instantiation of the A8.