AM6442: Cache coherency with DMA masters and other processors (R5f, PRUs)

Dominic Rath

Part Number: AM6442

Dear TI team,

we've finally realized that on the AM65x the A53's caches are not coherent with accesses from the R5f (see related thread).

We're working on several designs spanning multiple TI processors, including the AM64x.

In that thread Pekka Varis explained that the AM64x is cache coherent via its ACP port, unlike the AM65x, where coherence is achieved via the MSMC and the northbridge (NB).

Unfortunately the AM64x TRM is still rather sparse on details, and apparently doesn't cover this topic at all. As far as I understand the ACP provides a means for an uncached master (e.g. DMA) to access memory via the ACP, so that the accesses pass by the A53's L2 memory system, thus ensuring coherence. How is this implemented on the AM64x?

Do all accesses by uncached masters go through the ACP?

How can we control which accesses go through the ACP?

Regards,

Dominic

over 3 years ago

0 Pekka Varis over 3 years ago

TI__Mastermind 25230 points

Dominic,

Thanks for working through the issues with AM65x and coherency, unfortunately resulting in the clarification the R5 access will not be coherent with A53 caches. On AM64x the approach to coherency is much simpler and is controlled by the initiator/master setting a bus value called asel (address selector) to decide if the read/write is non-coherent, or if it gets routed through the A53 ACP port for coherency. The asel values are:

Write transaction with asel=14 ,cause L2 cache allocation: for cache warming feature

Write transactions with asel=15, does not cause L2 cache allocation

Read transactions with asel=14 and 15, does not cause L2 cache allocation

asel = 0 is a non-coherent transaction. This feature is used in Linux DMA driver and is measured to result in 10-25% performance lift at the iperf3 or netperf level benchmarks with Ethernet (cpsw3g). Asel can be set per DMA channel and per initiator like R5. Unfortunately it looks like the TRM section is lacking these details. I will add some of the details to this post and file some documentation bugs to get the TRM updated but wanted to get this initial response first. If you search the TRM for asel you'll find couple DMA channel setups for this, without the bits defined the four bits are as above (legal values 0, 14, 15).

Pekka

0 Dominic Rath over 3 years ago in reply to Pekka Varis

Mastermind 6905 points

Hello Pekka,

thanks for this explanation, that really helps.

I already saw the few places where ASEL is mentioned, but the TRM definitely could use an update in that area (among others... ). For now it would be great if you could post the details here.

Regards,

Dominic

+1 Pekka Varis over 3 years ago in reply to Dominic Rath

TI__Mastermind 25230 points

I've found the details and they will be in the next TRM revision. But while waiting for that here is a compressed version. The register QoS_MAP to modify looks like:

Bit	Name	Type	Reset	Description
31:30	reserved	r	0x0	Reserved
29:28	atype	rw	0x0	atype signal for channel N. 0 = physical. 1 = intermediate. 2 = virtual. 3 = physical with coherence.
27:16	virtid	rw	0x0	virtid signal for channel N.
15	reserved	r	0x0	Reserved
14:12	epriority	rw	0x7	epriority signal for channel N.
11:8	asel	rw	0x0	asel signal for channel N. 0 = SOC address. 14 for ACP cache allocate, 15 for ACP do not allocate.
7:4	orderid	rw	0x0	orderid signal for channel N.
3	reserved	r	0x0	Reserved
2:0	qos	rw	0x0	qos signal for channel N.

These start at base address 0x45D0 0000. For each R5 there is a separate read and write port for a total of 8 of these registers (will be in Table 3-14 in TRM). Formula for the base address is 0x45d0 0000+QoS_MAP_index*0x400+0x100. The QoS_MAP_index 's are: below. R5 core 0 read port is is 0x45d00000+(522*0x400)+0x100 = 0x45d82900. I tested a couple and the values look like reset value of 0x00007000. And all of these should work with ACP.

Port	QoS index	Address
R5_core0.rmst0	522	0x45d82900
R5_core0.wmst0	523	0x45d82d00
R5_core1.wmst1	527
R5_core1.rmst1	528
R5_core2.rmst0	532
R5_core2.wmst0	533
R5_core3.wmst1	537
R5_core3.rmst1	538
PCIe read	544
PCIe write	545

Pekka

Processors

Processors forum

AM6442: Cache coherency with DMA masters and other processors (R5f, PRUs)