AM6548: AM6548: Accessing NAVSS proxy registers via PCIe

Dominic Rath

Part Number: AM6548

Dear TI team,

we're currently trying to access the PRU-ICSS dual emac firmware directly from a x86 processor to which the AM65x is connected as a PCIe EP (x86 is RC).

For this we need to use the MCU NAVSS proxy to access a ring via the RA that is used in message mode.

Unfortunately the proxy generates an error when we're trying to access it from the x86 via PCIe.

We're using proxy #16 at base address 0x2a510000. The proxy is configured for tail access mode with an elment size of 0x8 (we need it to write 64-bit pointers to the RA).

With this configuration we need to write our data to address 0x2a5103f8...0x2a5103ff. Writing the last byte should trigger the proxy write to the destination. Write outside of that address range should generate an error.

When we write using a 64-bit write (or a 32-bit, or a 8-bit write) to 0x2a5103f8, we immediately see an error signaled in the PROXY_STATUS register @ 0x2a510004.

In order to debug this we've configured the firewall in front of the proxy to deny accesses via PCIe. This resulted in a firewall exception being logged that saw an access to address 0x00002A5103F0 with a length of 0x10. This is the same problem that we saw exactly a year ago when we tried to access the DRU via PCIe, see the referenced thread. Here's the full content of the firewall registers:

0x45b04000	01 71 00 66 00 00 00 00 00 00 00 00 00 00 00 00
0x45b04010	00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x45b04020	00 00 00 00 00 74 18 01 00 00 07 00 F0 03 51 2A
0x45b04030	00 00 00 00 B3 20 63 01 10 00 00 00 00 00 00 00

Back then we verified that the accesses on the PCIe bus appear with exactly the right properties (correct address, correct size, correct 1st/last byte enables).

We're sure that this is not an issue of any firewalls blocking the access, as we don't see any firewall exceptions being logged, except when we deliberately deny access via PCIe for debugging purposes.

To me it looks like the PCIe AXI2VBUSM bridge turning every transaction to an aligned 128-bit transaction (the documentation says the bridge is 128 bits wide), and that some peripherals (at least DRU and proxy) can't handle these transactions. I would consider that a bug ("undocumented constraint") in the hardware, unless there's some configuration option that I'm missing.

I just noticed that jian35385 came back on the original thread (which is locked without a solution) regarding the mapping to LP/HP ports. Not sure what exactly I should be looking for, but the VMAP_HP_CTRL_j and VMAP_LP_CTRL_j registers are all 0, i.e. the mapping is not enabled.

Regards,

Dominic

18 days ago

0 jian35385 7 days ago

TI__Expert 6270 points

Dominic,

Thanks for confirming the firewall setting. I assume you already did, but could you still confirm:

1. Were you able to see the same behavior with Main_NAV proxy?

2. You WERE able to write to the MCU_NAV proxies from A53 core, at address 0x2a5103f8?

Reason for the second question was that I saw the GPMC0_DATA occupies a 1GB space from 0x20000000, since PCIe only understand physical address, I want to make sure there are no aliasing going on.

Topology-wise, MCU_NAV is sitting on a 64-bit CBASS that get bridged with the Main domain, but I would expect all transactions handled correctly including non-aligned access. I did verified non-aligned (memory) access based on a 2-EVM setup, there were no issues. So the two questions above was trying to understand whether the issue is related to decoding.

Regards

Jian

0 Dominic Rath 4 days ago in reply to jian35385

Genius 3340 points

Hello Jian,

I'm on vacation this week, so I'm unable to perform any new tests right now.

No, we haven't tested this with the Main_NAV proxy, since we're porting code from TI that works only on the R5f (and thus uses MCU DMA) to be triggered from the x86.

I guess we can rule out aliasing effects, since we can see that the access via PCIe triggers an error in the proxy. We can also see that the proxy's firewall triggers on the access, if we program it to deny access from PCIe.

jian35385 said:
I did verified non-aligned (memory) access based on a 2-EVM setup, there were no issues. So the two questions above was trying to understand whether the issue is related to decoding.

I don't think there is a general issue with "unaligned" accesses, and more importantly, the accesses that we're performing are all naturally aligned, i.e. a 8-byte write to an 8-byte aligned address. I've verified that accesses to memory (MCU SRAM and DDR) have no undesired side-effects (see https://e2e.ti.com/support/processors/f/processors-forum/891037/am6548-accessing-dru-configuration-registers-via-pcie/3297886#3297886).

I'm pretty sure the problem is related to the transactions generated by the PCIe EP (or maybe the AXI2VBUSM bridge), and how these transactions are handled by some peripherals (at least DRU and MCU NAVSS Proxy). We've seen several hints that the PCIe EP generates transactions with a 16-byte aligned address even though I'm only accessing 4 or 8 bytes at an 8-byte aligned address. That's what we saw in SoC transaction logs and in the firewall logs.

Unfortunately the CBA is completely undocumented, but I'm guessing there are things like byte-enables or byte-mask signals. Would it be possible that the PCIe EP somehow changes my 8-byte write to 0x...8 to a 16-byte write to 0x0...0, but with the lower 8 bytes masked, i.e, would it be possible to describe such a transaction to the CBA?

This has low priority for us, since we're moving from AM65x to AM64x for the applications where PCIe is used as an EP, and we'll have to re-evaluate how the AM64x with its different PCIe IP core behaves in that regard. Any way, I'm pretty sure there is a hardware issue in the AM65x that isn't understood yet, and that could affect other customers as well.

Regards,

Dominic

Processors

Processors forum

AM6548: AM6548: Accessing NAVSS proxy registers via PCIe