TMDS64EVM: PCIe: Boot from PCIe is not working

Hi Paulo,

What SDK version do you use on TMDS64EVM? DId you modify the kernel device tree to configure the PCIe to Endpoint mode?

I also have a question of the PCIe cable you use, wondering if it is valid. Our PCIe hw expert is out of office for the US holidays. I will let him to comment on this once he is back.

Hi Bin Liu,

Well, we're trying to boot the board! At this point there is no kernel involved...
The issue is we cannot see the TMDS64EVM as EP on Jacinto side in other to load the image.

Regarding the cable, we've been using for other PCIe devices, and we can enumerate them through the bus!
We also used other cables and same issue! But we're open to suggestions!

Regards

Hi Paulo,

The another thing that I can think of would the SYSBOOT switch settings, however since PCIe boot is not supported in Processor SDK, I won't be able to provide further comments.

Hi Liu,

Thanks for the tip!

Regarding your statement about "PCIe boot not being supported in Processor SDK", then I'm bit confused....
On AM64x TRM we have this,

and my understanding is whenever the PCIe boot option is selected then "somehow" the device gets enumerated ("seen") on the bus. That should be a really low level behavior not depending (I supposed) on any software. So in the end the issue boils down to hardware... and even if there is no support of the boot process of this SoC on SDK, based on TRM we should be able enumerate the device.

I do admit my understanding could be wrong, so in the end I would like to clarify if my understanding/expectation is reasonable.

Thanks,

Paulo 

Hi Paulo,

Yes I agree with you, even if the second boot loader (u-boot) doesn't support PCIe boot, if the hardware setup is correct, the PCIe RC should detect and enumerate AM64x as the EP.

I am routing your query to our HW expert for comments.

Hi Liu,

Thanks for the feedback!

Looking forward for the next steps.

PCIe EP boot example is planned for 9.1 MCU+ SDK on AM64x with a arbitrary RC. While PCIe boot should work already (as described in TRM for PCIe boot) to check that your HW setup works have you tried booting the AM6442 EVM using for example SD card as stated in the endpoint steps in https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/latest/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/PCIe/PCIe_End_Point.html ? So before directly trying PCIe boot check that PCIe is working by running the steps described for EP and RC on AM64x described on that page.

  Pekka

Hi,
We can test a later stage (Linux PCIe EP), but if the boot does not work, it has a deep impact in our design.

Either way, I'm still confusion with yours statement.
- If you say we need a SDK (whatever R5 boot code), it means we'll need some flash to support that PCI boot,

- When my interpretation from TRM is as soon as one do selects the PCI boot mode the EP will be seen by hardware. (If you want something like DFU). So, no auxiliary boot code. And we should see the EP out of the box!

I'm ok if it is like that, I just need to know before hand to access the impact of such operation.

Paulo

Paulo Machado said:

We can test a later stage (Linux PCIe EP), but if the boot does not work, it has a deep impact in our design.

There is at least one customer of AM64x using PCIe boot successfully today with at least hundred boards built and in testing at scale. It is Windows host, their own bootloader based on our MCU+ bootloader. There is another design that uses OSPI boot to load R5 based bootloader and gets Linux image from PCIe.

Paulo Machado said:

- If you say we need a SDK (whatever R5 boot code), it means we'll need some flash to support that PCI boot,

HW setup as you have. Boot the AM64x EVM using for example SD card with the configturation as in https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/latest/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/PCIe/PCIe_End_Point.html?highlight=pcie . Use this to validate the PCIe connection is ok.

Paulo Machado said:

- When my interpretation from TRM is as soon as one do selects the PCI boot mode the EP will be seen by hardware. (If you want something like DFU). So, no auxiliary boot code. And we should see the EP out of the box!

Yes I agree "lspci" should show the powered up AM64x PCIe EP, with the vendor ID showing TI, just with the AM64x EVM set to PCIe boot.

Ok! Thanks for the clarification!
I'll review the setup then!

Hi Paulo,

Good to see you on E2E again, and thank you for your patience. 

Is it possible to share boot logs from the AM64x? Also, if you are following this guide from Pekka, could you share the logs from running the commands in "Starting the EP device" section: https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/latest/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/PCIe/PCIe_End_Point.html

With these logs, I would like to confirm if EP can be set up properly.

Regards,

Takuma

Hi Paulo,

After we look at EP setup, we should probably look at the cable as well. 

If the SDK example is followed, then it needs a cable with clock and reset removed. Clock needs to be removed mainly because clock is generated at both EP/RC ends independently, and reset needs to be removed because resetting the EP will remove any command line configuration done for setting up the EP-side in the SDK example.

If the EP is being set up properly and we see that after RC initialization the settings are reset, then my suspicion would be this reset signal being propagated from RC to EP which is clearing the EP configuration.

Regards,

Takuma

Hi Takuma,
Glad to talk with you again!

I'm currently on vacations, and I will return to work in 2 weeks from now. As soon as I'm back I'll review all the steps and discuss them with you.

Thanks for your support,

Paulo

Hello Takuma,

On behalf of Paulo Machado, that is not available at the moment.
I have took this thread and I carried on with a couple of tests.

Currently (just to make it clear), our setup is composed by a J784s4 (J784S4XG01EVM) as Root Complex connected through PCIe to AM64x (TMDS64EVM) as End Point, and for our tests we followed the described procedure of Sitara's SDK 09_00_00_03[1].

We performed the following steps (that is described in [1]):
1) physical changes as already described in the first entry of this thread
2) the Linux driver configuration for the HOST and the DEVICE
3) the complete sequence to start the endpoint, the Sitara, before start the root complex, the Jacinto

As you can see on log output, in the end of this entry, we are getting the following message

[ 38.273312] pci_epf_test pci_epf_test.0: Failed to get private DMA rx channel. Falling back to generic one

ln -s functions/pci_epf_test/func1 controllers/f102000.pcie-ep/

The Kernel versions of our boards are:
* SITARA
Linux am64xx-evm 6.1.26-g1ada48a6cf
* JACINTO
Linux j784s4-evm 6.1.46-g3ec3755134

Summary:
1) We have tried to setup the endpoint on Sitara side, apparently only that issue regarding "DMA Rx channel" pops up. Apart from that it seems to be ok.
2) We have tried to do a rescan on Jacinto side. However, We are not to see the device enumeration on host side
3) Our hardware team tried to remove the host's reset and we were able to restart Jacinto without losing the Sitara configuration. That also didn't work.

References:
[1] https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/09_00_00_03/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/PCIe/PCIe_End_Point.html

Logs:

=======================================================================
SITARA LOG
=======================================================================
root@am64xx-evm:~# ./setup_endpoint.sh
[INFO] KERNEL VAR: CONFIG_PCI_ENDPOINT
CONFIG_PCI_ENDPOINT=y
[INFO] KERNEL VAR: CONFIG_PCI_ENDPOINT_CONFIGFS
CONFIG_PCI_ENDPOINT_CONFIGFS=y
[INFO] KERNEL VAR: CONFIG_PCI_EPF_TEST
CONFIG_PCI_EPF_TEST=y
[INFO] KERNEL VAR: CONFIG_PCI_J721E
CONFIG_PCI_J721E=y
[INFO] KERNEL VAR: CONFIG_PCIE_CADENCE_EP
CONFIG_PCIE_CADENCE_EP=y
[INFO] KERNEL VAR: CONFIG_PCI
CONFIG_PCI=y
[INFO] KERNEL CONFIG_PCI_ENDPOINT_TEST
CONFIG_PCI_ENDPOINT_TEST=m
[INFO] KERNEL VAR CONFIG_PCIE_CADENCE_HOST

Press any key to continue...
[INFO] Check list of endpoint controller devices in the system
[INFO] FILE: f102000.pcie-ep
[INFO] Found: f102000.pcie-ep !
[INFO] Check list of endpoint function devices in the system
[INFO] FILE: pci_epf_ntb
[INFO] Found: pci_epf_ntb
[INFO] FILE: pci_epf_test
[INFO] Found: pci_epf_test

[INFO] To create pci-epf-test device
Press any key to continue...
[INFO] Creating pci-epf-test device
mount: /sys/kernel/config: none already mounted or mount point busy.
[INFO]
baseclass_code
cache_line_size
deviceid
interrupt_pin
msi_interrupts
msix_interrupts
primary
progif_code
revid
secondary
subclass_code
subsys_id
subsys_vendor_id
vendorid
[INFO] CONFIGURE VENDOR ID:
[DEBUG] 0xffff
[INFO] 0x104c
[INFO] CONFIGURE DEVICE ID:
[DEBUG] 0xffff
[INFO] 0xb010
[INFO] CONFIGURE THE NUMBER OF INTERRUPTS FOR MSI AND MSI-X WITH NUMBER 2
[INFO] BINDING PCI-EPF-TEST DEVICE TO A EP CONTROLLER
[   38.273312] pci_epf_test pci_epf_test.0: Failed to get private DMA rx channel. Falling back to generic one
[INFO] STARTING EP DEVICE

=======================================================================
JACINTO LOG
=======================================================================
####
Reboot
####
root@j784s4-evm:~# echo 1 > /sys/devices/platform/bus@100000/2910000.pcie/pci0001\  :00/pci_bus/0001\:00/rescan
root@j784s4-evm:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Device b00d
0001:00:00.0 PCI bridge: Texas Instruments Device b013
root@j784s4-evm:~# echo 1 > /sys/devices/platform/bus@100000/2900000.pcie/pci0000\  :00/0000\:00\:00.0/rescan
root@j784s4-evm:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Device b00d
0001:00:00.0 PCI bridge: Texas Instruments Device b013
(failed reverse-i-search)`resc': echo 1 > /sys/devices/platform/bus@100000/2900000  .pcie/pci0000\:00/0000\:00\:00.0/^Cscan
root@j784s4-evm:~# echo 1 > /sys/bus/pci/rescan
root@j784s4-evm:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Device b00d
0001:00:00.0 PCI bridge: Texas Instruments Device b013
root@j784s4-evm:~#

Any support or guidance will be appreciated.

Kind regards,
Joao Lima