AM6442: About access priority to DDR.

There might be a few reasons for the behavior you are seeing:

-are you running with cache enabled?  The data may already be cached and thus the access to the data doesn't need to go all the way out to the memory

-i'm not sure what you are using to time your accesses.  Maybe there is not enough granularity in the timer to perceive a difference with just 4 bytes

-the DDR controller could be reordering the commands to optimize accesses  

-Not sure how you coding the accesses to the DDR.  It will make a difference if you are using single cycle accesses, loops, different element sizes, DMAs, etc.

Regards,

James

Hello James,
Thank you for your reply.

-are you running with cache enabled? The data may already be cached and thus the access to the data doesn't need to go all the way out to the memory
Cache is enabled, but I set the DDR area that performs memory access to non-cache with MMU/MPU.

-i'm not sure what you are using to time your accesses. Maybe there is not enough granularity in the timer to perceive a difference with just 4 bytes
I am using a Performance Monitor Unit (PMU) cycle counter.

-the DDR controller could be reordering the commands to optimize accesses
Is the order changed even when memory accesses are made to the same address?

-Not sure how you coding the accesses to the DDR. It will make a difference if you are using single cycle accesses, loops, different element sizes, DMAs, etc.
R5F core:
  Loop 4Bytes write (STR instruction 100 times) to the same address
A53 core:
  Loop 4Bytes read (LDR instruction once) to the address being written by the R5F core (256 times).
  In the loop, we read the PMU's cycle counter and check the average time it takes to read the DDR.

I have attached the CCS project to my first post, so please see "empty.c" for details.

Regards.

DDR CoS is intended to prioritize some initiator over others. So there needs to be interfering traffic from some other initiator to see an effect. The A53 is designed to operate for high memory throughput through cached memory, specifically the shared L2 cache, not for non-cached to DDR from each individual core. The non-cached access from A53 to DDR is not performance optimized in the A53 core, the CoS should be set for the cluster (L2 cache) and the benchmark to use would be some throughput or bandwidth oriented one. 

What route ID are you using for the A53? I noticed it is not that clear in the TRM, but 0 and 1 should be the non-cacheable strongly ordered access from each of the 2 cores. 4 is the shared L2 cache. For any normal SW only the shared cache should matter.

tomitama said:

R5F core:
  Loop 4Bytes write (STR instruction 100 times) to the same address
A53 core:
  Loop 4Bytes read (LDR instruction once) to the address being written by the R5F core (256 times).
  In the loop, we read the PMU's cycle counter and check the average time it takes to read the DDR.

I'm not clear what this is intended to measure. The sequence is shared memory, so coherency related features will come into play in the DDR controller. CoS is intended for relative prioritization of independent access streams. R5 working on memory at one address, A53 at another. CoS can be used to improve one of these over the other. Depending on what is the intention I would think R5 and A53 should work on completely different DDR locations to see relative priority.

As a summary CoS is a tool to say no matter what this other core or DMA is doing, I want to to prioritize this core.

I am using 0 and 16 as Route IDs for A53, which are IDs corresponding to COMPUTE_CLUSTER0 described in TRM (Rev.G).
Please let me know what is not clear in RouteID's TRM.

My aim is to check about two things:
  1.DDR CoS priority change has given priority to A53 core over R5 core.
  2.Difference in processing time required when DDR CoS priority is changed and when the priority is the same.
Please let me know if there are any specific steps to confirm the above.

I changed the memory access address to a different address for each core, but I could not see any difference due to the DDR CoS priority change.
Using the following DDR address:
  A53:0x86000000
  R5_0_0:0x90000000
  R5_0_1:0xA0000000
  R5_1_0:0xB0000000
  R5_1_1:0xC0000000

Greetings Tomitama,

tomitama said:

  1.DDR CoS priority change has given priority to A53 core over R5 core.
  2.Difference in processing time required when DDR CoS priority is changed and when the priority is the same.

Understand you're looking to verify features, but can you elaborate on what are your goal(s) for using CoS? Are you trying to accelerate some application or ensure some kind of time deadline? This will help us understand and possibly suggest other ideas for your overall goal(s). 

Pekka is correct as stated above, the practical usecase for the cores is for high throughput applications that have DDR set as cacheable (in the MMU for A53 and MPU for R5) so that it will send cacheline fetches (64B for A53, 32B for R5) to DDR. 

When a lot of high throughput initiators (core/DMA/other) saturate the bandwidth of the DDR at once this could create starvation on some of their threads, as a result they could see severely lowered throughput. CoS is one mechanism to mitigate that, but may not show a noticeable difference unless there is a high amount of traffic. The available bandwidth of DDR is dependent on the speed of it's operation (16-bit DDR at 1600MT/s is a bit lower that 3200MB/s) so you may not even come close to using up available bandwidth for your overall usecase (DDR speed, IPs being used, access pattern, etc.) 

Sincerely,

Lucas

From the A53 the standard C-library memcpy() (or memset()) will utilize optimized memory access instructions that with be able to generate almost 70% of the theoretical wire rate at the DDR interface. From Linux this gets called with for example bw_mem -P 2 8M bcopy this will generate maximum number of outstanding cache line operations from the 2 A53s at the memory controller. This will be much higher load than the inline assembly that I can see in the attached project. Without a flood of interfering memory reads/writes you will not see any difference in access latency. Similar with R5. Would suggest to modify the background load to be a memset() or memcpy(), measure typical latency from the other core while the interfering core is doing memset() or memcpy().

I'm expecting the following behavior by changing the CoS priority:
  For example, if there are three cores with the same priority and they access the same memory address, the access time will be about the same.
  If only one core has a high priority and three cores access the same memory address, the access time of the core with a high priority will be faster.

In other words, I expect that the higher priority core's access will be prioritized when accessing the same memory address.

Can the above behavior be achieved by using CoS?

Adding a further detail here. The TRM does not have the detailed topology, instead just a simplified "CBASS0" for all of the interconnect. This is the more exact way the interconnect is done, A53 to LPDDR4/DDR4 on one block, then the rest of the main initiators on another.