AM5716: Disabled BAR settings

Hi Darren,

Erratas for AM571x can be found here: https://www.ti.com/lit/er/sprz436g/sprz436g.pdf

One errata that might be causing the behavior the customer is seeing could be errata i870 PCIe Unaligned Read Access Issue. I think observed behavior *would be that some portions of PCIe config space can get corrupted when an unaligned write happens.

The workaround is mentioned in the errata documentation, so an experiment they can do is to see if issuing read/writes in 4 bytes causes issues.

Regards,

Takuma

Hi Takuma,

Thank you for the response.  I'll take a look at the errata.  It looks like one of your sentences was cut off though.  Did you have more to say?

Thanks,
Darren

Hi Darren,

Apologies and thanks for the notice!

I edited the response with the cut-off portion.

Regards,

Takuma

Thank you Takuma.

Here is the customer's response.

i870 seems unlikely to relate for several reasons:

- We have the MMU configured such that this address region is "strongly ordered device memory" which will generate data exception for unaligned access, and send the processor into a Data Abort handler. That doesn't help against byte or half-word accesses but does reduce the chances of hitting something like i870. Additionally we have checked our disassembly, and the processor is exclusively using load/store word (32-bit) instructions for this. We are running bare metal, so we know nothing else is touching this register space.

- Additionally, that errata refers to the generation of TLPs based on Slave AXI transactions. We're not doing that here, since the A15 is only configuring the Config registers, it is not generating TLPs on the PCIe bus, but merely configuring the internal registers. The host comes and reads this asynchronously later, using CFG reads over PCIe. Those reads would follow the dashed blue line read path in Figure 24-169, which doesn't use the AXI Slave port.

Thanks,
Darren

Hi Darren,

I have not seen this issue before. I think closest issue I have seen would be this E2E: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1291922/am5728-about-pci-initialization-tlp-packet-has-uninitialized-value-when-write-1-or-2-byte-register/4918962?tisearch=e2e-sitesearch&keymatch=pcie%201%20byte%20write#4918962

Regards,

Takuma

Hi Takuma,

From the customer:

The posted E2E does not seem all that related however, here are three examples we found of people seeing the exact same issue as us on various TI processors that use the Synopsys PCIe IP. Unfortunately, none of these had a resolution.

 <https://e2e.ti.com/support/processors-group/processors/f/processors-forum/241828/pci-bar-setup>

<https://e2e.ti.com/support/processors-group/processors/f/processors-forum/924530/am5749-pcie-endpoint-bar4-containing-bad-value>

<https://e2e.ti.com/support/processors-group/processors/f/processors-forum/204910/c6678-pcie-boot-with-wrong-bar-configuration>

There may be more examples as well. We could be wrong (and please explain why if we are!) but I don't see how this can related to TLPs (like the suggested errata or e2e post).

So that someone at TI can reproduce this, the issue can be seen in two ways, one of which doesn't even require a PCIe link being online and could be reproduced on an eval board with no special hardware:

After bringing all the various clocks online etc, as following the TRM's "Low Level Programming Model", program the DBICS_BAR2 and DBICS2_BAR2MASKs to both be 0x0:

From the A15 CPU, use a STR to do a 32-bit store: i.e. "str 0 [0x51000018]" and "str 0 [0x51001018]". This should disable the BAR and sets the prefetch bit to 0.

Once you've done that:

1/ From the A15, read the register with "ldr r3 [0x51000018]".

- According to Table 24-1034 in the TRM, this should result in 0x0 in R3.

- In reality, the contents of R3 are now 0x8. This means that the prefetchable bit is set, despite the BAR being disabled. This is not a valid state for this register to be in.

2/ If you have a PCIe host connected to the EP, read the BAR2 configuration register (0x18 in the PCIe header) 

- According to the PCIe spec, and the TRM 24.9.4.8.3.2, "BARn is disabled: register is read-only and return 0x0 over PCIe"

- In reality, when read over PCIe, the bar returns 0x8. This means that the prefetchable bit is set, despite the BAR being disabled. This is not a valid state for this register to be in.

Thanks,

Darren

Hi Darren,

We can try this out on our end, but please set expectations that it will take us more than a month before we can get initial test results. Apologies for the delay and thank you for your patience.

Regards,

Takuma

Hi Darren,

This may be related to TLP, mainly because the nature of that errata is when a message that is shorter in length than 4 bytes is sent, the message automatically gets aligned to 4 bytes. Since the receiving end expects only 1 byte but receives 4 bytes, depending on how the message was automatically padded, it can corrupt neighboring memory space. Since BAR registers are all neighboring, I was thinking that there could be some corruption causing some of the BAR registers that are supposed to be 0 to have a non-zero number.

Otherwise, if the above was not the issue, is it possible for them to do the workaround that is mentioned in this E2E where they allocate a small region for BAR4: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/924530/am5749-pcie-endpoint-bar4-containing-bad-value

Regards,

Takuma