AM6441: How to support PCIe bus enumeration in endpoint mode

Hi Grant,

Do you use TI Processor SDK on TMDS64EVM? Is it Linux or MCU+ SDK? Which version?

My apologies...

I should clarify that we're planning on running our PCIe application code on the R5 core, not on the A53, and the example apps I mentioned above are from the MCU+ SDK.

I've found the following in the MCU+ SDK  API Guide:

Is that stating that the MCU+ SDK doesn't support bus enumeration (even though the PCIe controller does)? 

Or is it stating the the PCIe controller itself doesn't support bus enumeration?

I'm using mcu_plus_sdk_am64x_08_05_00_24.

We've also been reviewing the EVM schematic, it looks like we might need to move some 0-ohm resistors on p30 to connect the refclk from the PCIe connector to the serdes refclk pins W16,W17?

Hi Grant,

Thanks for the update. I am routing your query to our MCU+ SDK expert for comments.

I've been looking at both the MCU+ PCIe drivers and the UBoot PCIe drivers, and there seems to be a lot of PCIe registers that are used by both but are missing from "AM64x /AM243x Processors Silicon Revision 2.0 Texas Instruments Families of Products Technical Reference Manual".

For example there appears to be a 352 register block containing EP ATU configuration addr0/1 values at offset 0x00400840 through 0x00400DBC. That is missing from the TRM. The register map in the TRM ends at 0x00400824 on p6427. 

All of the CSL_PCIE_EP_CORE_*  registers defined in cslr_pcie_ep.h also seem to be missing from the TRM.

Is there a somewhere a good document describing the PCIe hardware?

The TRM doesn't appear to be usable for EP mode.

Hi,

I will check on this and get back to you by next week.

Best Regards

Ashwani

Hi Grant,

Thanks for your time and patience.

As per my discussion with dev and test team,

• We have test setup as AM64x(RC) <=> AM64x(EP) only 
• Means, Board to Board testing only
• We do not have test setup having Windows / Linux (RC) <=> AM64x (EP) or vice-versa.
• I took it as a test gap and reported to test team.

Grant Edwards said:

For example there appears to be a 352 register block containing EP ATU configuration addr0/1 values at offset 0x00400840 through 0x00400DBC. That is missing from the TRM. The register map in the TRM ends at 0x00400824 on p6427. 

I reported this to TRM team as well. Thanks for your kind feedback and helping us to improve.

Now,

I am going to tell what I quickly try on my side is 

Setup: Linux (Ubuntu 18.04) (working as RC) <=> AM64x (EP).

Steps:

• I locally build application (C:\mcu_plus_sdk\examples\drivers\pcie\pcie_buf_transfer\pcie_buf_transfer_ep) on windows PC 
• Load it to AM64x-evm board
• Put a break point in application (to make sure PCIe should not enter into de-init).
• Then connect PCIe cable to linux-PC
• run lspci command to scan PCIe devices. But, It was not detected.
  • Expected as Hot Plug feature is not enabled. TRM section 12.2.2.1.2 PCIe Subsystem. 
  • 
• power cycle / Restart Linux-PC
• run lspci command again
• Device got detected as Cadence Design Systems
  • 

Best Regards

Ashwani

Ashwani Goel said:

or example there appears to be a 352 register block containing EP ATU configuration addr0/1 values at offset 0x00400840 through 0x00400DBC. That is missing from the TRM. The register map in the TRM ends at 0x00400824 on p6427. 

I reported this to TRM team as well. Thanks for your kind feedback and helping us to improve.

There are also several thousand other EP registers defined/used by the example apps that are missing from the TRM. Please see all of the CSL_PCIE_EP_CORE_* registers defined in cslr_pcie_ep.h. None of them are listed in the TRM.

Ashwani Goel said:

Device got detected as Cadence Design Systems

Very good, that's what I've been hoping to see. Thank you for testing an EP example with a PC.

When the buffer transfer EP application is connected to a PC, is the EP incoming address translation unit using the PCIe memory address range assigned by the PC host during bus enumeration?

When the buffer transfer EP application is connected to a PC, do PCIe accesses by the PC work as expected?

I see that you're using a different EP example than I was. I'll try the buffer transfer EP application instead of the benchmark EP. With the benchmark EP application we never get a good link status when connected to the PC: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1214891/tmds64evm-pcie-link-failure-connected-to-pc-mainboard/.

Can you provide any information on how you connected the EVM board to the PC?

• Are you using a TX/RX crossover cable similar to that used to connect two EVM boards?
• How is the PCIe Reference Clock provided to the EP?  Are you using separate clocks or is refclk provided from the PC via the cable?
• How is the EVM being powered?
• Are 12V and 3.3V power connections in the PCIe cable disconnected or not?

Thanks again...

Grant Edwards said:

Very good, that's what I've been hoping to see. Thank you for testing an EP example with a PC

Thanks you Grant. Happy to help you

Grant Edwards said:

When the buffer transfer EP application is connected to a PC, do PCIe accesses by the PC work as expected?

As I mentioned earlier, we do not have any test case for PC <=> Board. The given test case is also applicable for Board (RC) <=> Board (EP).

Please find the details of test case here:

https://software-dl.ti.com/mcu-plus-sdk/esd/AM64X/08_05_00_24/exports/docs/api_guide_am64x/EXAMPLES_DRIVERS_PCIE_BUF_TRANSFER_EP.html

Grant Edwards said:

Can you provide any information on how you connected the EVM board to the PC?

Sorry, I dismantle the setup. Otherwise I could share the pictures.

Grant Edwards said:

How is the EVM being powered?

12 volt as mentioned here: software-dl.ti.com/.../EVM_SETUP_PAGE.html

Best Regards

Ashwani

Ashwani Goel said:

Can you provide any information on how you connected the EVM board to the PC?

Sorry, I dismantle the setup. Otherwise I could share the pictures.

Thanks, but I don't really care about photos.

What I would like to know are details about the connection between the EVM and PC:

1. Are rx/tx pairs swapped in the cable?
2. Are the 12V lines connected by the cable between the PC and EVM? If yes, how did you prevent conflict between the two 12V supplies?
3. Are the 3.3V lines connected by the cable between the PC and EVM? If yes, how did you prevent conflict between the two 3.3V supplies?
4. How is the PCIe reference clock being provided? Are you using separate clocks or is the refclk being provided by the PC?
5. Is the PCIe reset line connected to the am64 processor? When the PC is rebooted, does the AM64 get reset?
6. What position are the jumpers on J34 and J35?

Did you have to power cycle the PC for the EVM to be recognized? Or was it recognized after a warm start (using the "reboot" command from the Linux command prompt)?

Ashwani Goel said:

We have test setup as AM64x(RC) <=> AM64x(EP) only 

We've been wondering about the reasoning behind that decision.

Is that the most common use case for your customers?

You intend customers to use PCie to communicate from one Sitara processor to another rather than as an EP connected to a PC or as an RC with some other off-the-shelf hardware as the EP?

Hello Grant

As of now this is our test setup. This is easiest for us from a perspective of test automation and regression and ensuring good coverage for EP/RC drivers for the device family. 

From a use-case perspective both SOC to SOC and PC to SOC are supported and intended use-cases. 

Regards

Mukul

Grant Edwards said:

Are the 12V lines connected by the cable between the PC and EVM? If yes, how did you prevent conflict between the two 12V supplies?

No there is no external connection for power supply from PC <=> Board

Below is connector I used to connect PC <=> Board

Below is PCIe slot (RED Circled) on AM64x I connected to this cable

Below is PCIe slot (RED Circled) on PC I used for connection

Best Regards

Ashwani

Grant Edwards said:

When the PC is rebooted, does the AM64 get reset?

No.

Grant Edwards said:

• Are the 12V lines connected by the cable between the PC and EVM? If yes, how did you prevent conflict between the two 12V supplies?
• Are the 3.3V lines connected by the cable between the PC and EVM? If yes, how did you prevent conflict between the two 3.3V supplies?

No

Grant Edwards said:

What position are the jumpers on J34 and J35?

I will check tomorrow when I will be in office and update on this.

Best Regards

Ashwani

Grant Edwards said:

Did you have to power cycle the PC for the EVM to be recognized?

Yes.

Ashwani Goel said:

Grant Edwards said:

Did you have to power cycle the PC for the EVM to be recognized?

Yes.

Just to be clear:

You are stating that the EVM was not recognized if you just did a "reboot" from the Linux command prompt after starting the EVM endpoint application?

You must power down the PC and power it up again (while the EVM is powered the entire time) before the EVM is recognized?

[I've been reluctant to power down the PC with the EVM powered because applying powered bus signals to a non-powered mainboard is generally regarded as a bad idea.]

Are the rx/tx pairs swapped in the cable?

Is it the same cable that's used to connect two EVM boards?

Have any of the EVM clock selection resistors (circled on p30 of the EVM schematic) been moved/changed?

Grant Edwards said:

You are stating that the EVM was not recognized if you just did a "reboot" from the Linux command prompt after starting the EVM endpoint application?

No. I did not try that..

Grant Edwards said:

You must power down the PC and power it up again (while the EVM is powered the entire time) before the EVM is recognized?

I tried this only.

I just connected PCIe cable Linux-PC <=> Board (application running on it)

==> Board is not recognized. As hot plug feature is not supported.

Then, power cycle Linux-PC (No change from board side)

=> Board is recognized.

Best Regards

Ashwani

Grant Edwards said:

Is it the same cable that's used to connect two EVM boards?

I will check and update you.

Grant Edwards said:

Have any of the EVM clock selection resistors (circled on p30 of the EVM schematic) been moved/changed?

I will check and update you.

Best Regards

Ashwani

Grant Edwards said:

What position are the jumpers on J34 and J35?

J34 and J35 positions.

Best Regards

Ashwani

Grant Edwards said:

Is it the same cable that's used to connect two EVM boards?

Yes

Grant Edwards said:

Have any of the EVM clock selection resistors (circled on p30 of the EVM schematic) been moved/changed?

Not getting what you are looking for here ?

Best Regards

Ashwani