J722SXH01EVM: u-boot PCIe support

Matthew Fatheree

Part Number: J722SXH01EVM

Tool/software:

Hello,

I am working with a yocto build that uses the git.ti.com/.../ti-u-boot.git ( branch ti-u-boot-2025.01 ), and I am trying to get pci devices enumerated in u-boot using the 'pci enum' command, but I am running into the error: "pcie_cdns_ti pcie@f102000: failed to bring up link" error.

I have verified that the device exists and is showing up in the 'dm tree' command, and have setup the serdes lane configuration in my device tree file, but I still can not enumerate devices on the PCIe port.

Here are the relevant things in my device tree:

&wkup_conf {

pcie0_ctrl: pcie0-ctrl@4070 {

compatible = "syscon";

reg = <0x4070 0x4>;

#syscon-cells = <0>;

};

&main_conf {

serdes_ln_ctrl: mux-controller@4080 {

compatible = "reg-mux";

reg = <0x4080 0x14>;

#mux-control-cells = <1>;

mux-reg-masks = <0x0 0x3>, /* SERDES0 L0 */

<0x10 0x3>; /* SERDES1 L0 */

idle-states = <0>, <2>;

//bootph-all;

bootph-pre-ram;

};

/* Optional but helpful so SPL/U-Boot pre-reloc can see them */

usb_phy_ctrl0: syscon@4008 {

compatible = "ti,am62-usb-phy-ctrl", "syscon";

reg = <0x4008 0x4>;

bootph-all;

};

usb_phy_ctrl1: syscon@4018 {

compatible = "ti,am62-usb-phy-ctrl", "syscon";

reg = <0x4018 0x4>;

bootph-all;

};

/* 100MHz fixed refclk for SerDes */

&serdes_refclk {

compatible = "fixed-clock";

#clock-cells = <0>;

clock-frequency = <100000000>;

};

/* WIZ wrapper: reset + TI-SCI power/clock */

&serdes_wiz1 {

compatible = "ti,am64-wiz-10g";

#reset-cells = <1>;

#clock-cells = <1>;

num-lanes = <1>;

ti,sci = <&dmsc>;

ti,sci-dev-id = <280>;

//power-domains = <&k3_pds 280 1>;

power-domains = <&k3_pds 280 TI_SCI_PD_EXCLUSIVE>; // for testing

clocks = <&k3_clks 280 0>, <&k3_clks 280 1>, <&serdes_refclk>;

clock-names = "fck", "core_ref_clk", "ext_ref_clk";

status = "okay";

/* for testing */

assigned-clocks = <&k3_clks 280 1>;

assigned-clock-parents = <&k3_clks 280 5>;

/* for testing */

serdes@f010000 {

compatible = "ti,j721e-serdes-10g";

reg = <0x0f010000 0x10000>;

reg-names = "torrent_phy";

clocks = <&serdes_refclk>;

clock-names = "refclk";

//resets = <&serdes_wiz1 0>;

reset-names = "torrent_reset";

#address-cells = <1>;

#size-cells = <0>;

status = "okay";

serdes1_lane0: phy@0 {

reg = <0>;

cdns,num-lanes = <1>;

cdns,phy-type = <PHY_TYPE_PCIE>;

mux-controls = <&serdes_ln_ctrl 1>;

status = "okay";

};

&pcie0_rc {

compatible = "ti,j722s-pcie-host", "ti,j721e-pcie-host", "ti,j7200-pcie-host";

reg = <0x0 0x0f102000 0x0 0x00001000>, /* intd_cfg */

<0x0 0x0f100000 0x0 0x00000400>, /* user_cfg */

<0x0 0x0d000000 0x0 0x00800000>, /* RC DBI (core) */

<0x0 0x68000000 0x0 0x00001000>; /* CFG-space window */

reg-names = "intd_cfg", "user_cfg", "reg", "cfg";

num-lanes = <1>;

/* IO 64KB, MEM ~1GB-17KB */

ranges = <0x01000000 0x0 0x68001000 0x0 0x68001000 0x0 0x00010000>,

<0x02000000 0x0 0x68011000 0x0 0x68011000 0x0 0x03fef000>;

dma-ranges = <0x02000000 0x0 0x0 0x0 0x0 0x10000 0x0>;

max-link-speed = <3>;

power-domains = <&k3_pds 259 1>; /* 1 = TI_SCI_PD_EXCLUSIVE */

clocks = <&k3_clks 259 0>;

clock-names = "fck";

ti,syscon-pcie-ctrl = <&wkup_conf 0x4070>;

phys = <&serdes1_lane0>;

phy-names = "pcie-phy";

reset-gpios = <&exp1 18 GPIO_ACTIVE_HIGH>;

status = "okay";

};

from poking around the registers with the md.l command I see the following:

md.l 0x43004090 1 "Synchronous Abort" handler, esr 0x96000010, far 0x43004090

elr: 00000000808c5da0 lr : 00000000808c5cf0 (reloc)

elr: 00000000fff77da0 lr : 00000000fff77cf0

x0 : 0000000000000009 x1 : 00000000fde592d8

x2 : 00000000fffffffe x3 : 0000000000000020

x4 : 0000000000000000 x5 : 00000000fde592d0

x6 : 0000000000000030 x7 : 000000000000000f

x8 : 00000000fde59220 x9 : 00000000ffffffd8

x10: 0000000000000010 x11: 000000000001869f

x12: 00000000fde59548 x13: 00000000fde596b0

x14: 0000000000000008 x15: 00000000fde592d0

x16: 00000000ffecdc28 x17: 0000000000000000

x18: 00000000fde71df0 x19: 0000000000000004

x20: 0000000000000004 x21: 0000000000000001

x22: 0000000043004090 x23: 00000000fde592d9

x24: 0000000000000000 x25: 00000000fde59288

x26: 00000000fffa4b0c x27: 0000000000000008

x28: 0000000000000004 x29: 00000000fde59220

Code: 2a0403f3 17ffffcb 7100129f 54000181 (b94002c3) Resetting CPU ...

which seems to indicate the MAIN CTRL_MMR window is being blocked from reading.

How can I debug and fix this issue so u-boot is able to enumerate PCI devices?

thanks

23 days ago

0 Takuma Fujiwara 21 days ago

TI__Guru 50873 points

Hi Matthew,

I have not tested out the PCIe functionality within U-Boot, so let me check on this.

Otherwise, as a quick comment, the CTRL_MMR registers are locked by default, and a unlock value needs to be written into two LOCK KICK registers before it can be written to. U-Boot should have a function that gets called that unlocks all of the CTRL_MMR registers, so it could be that the registers are not being unlocked when the register access happens.

Regards,

Takuma

0 Matthew Fatheree 21 days ago in reply to Takuma Fujiwara

Prodigy 40 points

Thanks,

That aligns with what I have found.

If you can point me towards what u-boot function should unlock the CTRL_MMR registers, and what value needs to be written to "LOCK KICK" those registers it would be greatly appreciated.

Thank you for your quick response.

0 Takuma Fujiwara 20 days ago in reply to Matthew Fatheree

TI__Guru 50873 points

Hi Matthew,

There is a function in U-Boot arch/arm/mach-k3/j722s/j722s_init.c called crtl_mmr_unlock which should unlock all the CTRL MMR registers.

Unlock values are KICK0 0x68ef3490 and KICK1 0xd172bc5a. Register addresses, please reference the register excel sheet that is zipped in with the TRM: https://www.ti.com/lit/zip/sprujb3

Regards,

Takuma

0 Matthew Fatheree 20 days ago in reply to Takuma Fujiwara

Prodigy 40 points

As a quick test I used u-boot to write those values to those registers using:
mw.l 0x00104008 0x68ef3490
mw.l 0x0010400c 0xd172bc5a

I then tried to access the PCIe DBI registers via:

md.l 0x0D000000 1

And I get an asynchronous abort in u-boot, but even prior to that the PCIe link doesn't train with the 'pci enum' command.

Can you verify that the PCIe device should be able to work, and let me know anything I may have missed in my devicetree configuration or otherwise?

Thank you

0 Takuma Fujiwara 19 days ago in reply to Matthew Fatheree

TI__Guru 50873 points

Hi Matthew,

Matthew Fatheree said:
And I get an asynchronous abort in u-boot, but even prior to that the PCIe link doesn't train with the 'pci enum' command.

The lock kick unlocks only the CTRL MMR registers. The PCIe DBI registers are not CTRL MMR registers and are instead part of the PCIe IP block. So, if the PCIe is not initialized (i.e., powered on) then it will cause a bus error. Most likely this is the reason why the asynchronous abort is happening when accessing that register.

Matthew Fatheree said:
Can you verify that the PCIe device should be able to work, and let me know anything I may have missed in my devicetree configuration or otherwise?

Will try out sometime today. So far, not finding any support added specifically for J722S. Although, I do see support added for other processors in the same family (AM64, J7200).

Regards,

Takuma

0 Takuma Fujiwara 17 days ago in reply to Takuma Fujiwara

TI__Guru 50873 points

Hi Matthew,

I tried out on my end for J722S. Looks like there is no current support within U-Boot. Support for PCIe starts at the Linux kernel for J722S.

Regards,

Takuma

0 Matthew Fatheree 17 days ago in reply to Takuma Fujiwara

Prodigy 40 points

Hi Takuma,

This creates a problem for us, because our design requires that we load the Linux Kernel and device tree from an NVME disk attached using the PCIe slot.

I have the PCIe as well as NVME working in linux, but support in u-boot would be required to achieve the goals for our product.

Is there a plan to add PCIe support in u-boot for the J722s, or could u-boot be patched in some way to support it?

Thank you for your help and verification so far.

0 Takuma Fujiwara 17 days ago in reply to Matthew Fatheree

TI__Guru 50873 points

Hi Matthew,

As of now, no plans that I can find in our roadmap. Historically, there has been requests periodically for booting from NVMe disk attached through PCIe, but this feature has not been supported on the J7x devices. Main usecases reported by developers is to use the J7x device kind of like a PC that boots off of NVMe.

For development purposes, micro SD card is recommended. And for production, booting from OSPI or eMMC is the general recommendation due to their faster boot times. NVMe is recommended mainly for storing data.

Regards,

Takuma

0 Matthew Fatheree 10 days ago in reply to Takuma Fujiwara

Prodigy 40 points

Can you provide any additional information about what would need to be done if we wanted to add PCIE and NVME support to u-boot for the J722s-evm devices?

Would it require us to build sysfw file that can be used?

From our testing, the NVME is much faster than the MMC, and from the u-boot standpoint, the linux kernel, and device tree is just data.

0 Takuma Fujiwara 10 days ago in reply to Matthew Fatheree

TI__Guru 50873 points

Hi Matthew,

It should be mainly changes to U-Boot. However, no customers have used the PCIe interface within U-Boot for J722S to my knowledge, and validation for PCIe is done at Linux level, and not at U-Boot. There may be need for changes in the driver, but since this is uncharted territory, cannot say with confidence.

Is NVMe an necessity for your requirements, or can it be substituted for a different storage media to meet requirements?

Regards,

Takuma

0 Matthew Fatheree 10 days ago in reply to Takuma Fujiwara

Prodigy 40 points

Hi Takuma,

For the current project we are working on NVME is a requirement, and from testing with the fio utility comparing NVME and MMC read and write speeds the NVME offers considerably better performance than the MMC. The product we are creating is an upgraded design of a previous product that used SSD cards for storage, and additional storage, and more features could be added by using a larger SSD card.

We plan to do the same thing with the NVME disks. In production we will write the entire disk image, containing both kernel, device tree, and rootfs to those SSD cards. This is because the kernel is often tightly bound to the rootfs, and there are some instances where the boot process will fail if the kernel doesn't "match" the rootfs that it was built for. During a firmware update, we will often write a disk image out to the disk, to ensure the kernel and rootfs are upgraded together. We understand that we can break that into parts, but it will require a significant amount of changes to our upgrade, and our production process. There are times when we will send an entire new disk to a customer if theirs has been damaged, and in those cases it could cause the unit not to be able to boot correctly.

In the patch set for u-boot to support the J722s-evm chip, PCIe is mentioned as one of the supported devices:
https://lists.denx.de/pipermail/u-boot/2024-June/556071.html

We are willing to look into 3rd party resources to add support for PCIe, or even to try to add support for it in house, but to make sure we are doing it correctly, we would really require a bit more guidance from TI.

You have said there is support for this in other similar products like AM64, and J7200, would there be a way to look at those implementations, and guide us to what might be missing, or needed to be fixed in the J722 device?

Thanks

0 Takuma Fujiwara 9 days ago in reply to Matthew Fatheree

TI__Guru 50873 points

Hi Matthew,

Matthew Fatheree said:
You have said there is support for this in other similar products like AM64, and J7200, would there be a way to look at those implementations, and guide us to what might be missing, or needed to be fixed in the J722 device?

Essentially, the same thing done in Linux driver for PCIe/NVMe needs to be done for U-Boot.

Matthew Fatheree said:
In the patch set for u-boot to support the J722s-evm chip, PCIe is mentioned as one of the supported devices:

Looking through U-Boot a little more, it looks like k3-j722s-r5-evm.dts includes k3-j722s-evm.dts that has some definition for PCIe. However, PCIe interface is not tested in U-Boot by our SDK teams. Furthermore we do not enable PCIe in our defconfig files.

In any case, I did an experiment on J721S2 (since my J722S is currently lent out to a colleague), to see if I can enable PCIe. I'm able to bring up PCIe with some additions to the configuration file, but have not been able to read/write to the NVMe device. May have something to do with the SSD card being enumerated 32 times.

Fullscreen U-Boot_NVME.txt Download

In j721s2_evm_a72_defconfig:
CONFIG_CMD_PCI=y
CONFIG_PCIE_CDNS_TI=y
CONFIG_PCI=y
CONFIG_NVME=y
CONFIG_NVME_PCI=y
CONFIG_CMD_NVME=y

In U-Boot:
=> pci enum
pcie_cdns_ti pcie@2910000: link up
=> nvme scan
=> nvme info
Device 0: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 1: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 2: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 3: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 4: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 5: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 6: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 7: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 8: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 9: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 10: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 11: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 12: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 13: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 14: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 15: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 16: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 17: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 18: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 19: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 20: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 21: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 22: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 23: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 24: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 25: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 26: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 27: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 28: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 29: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 30: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
Device 31: Vendor: 0x2646 Rev: CRTP2324 Prod: 50026B7686ABE27C
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
=>

So, huge disclaimer to expect slow support on this topic if PCIe support in U-Boot is desired... recommendation will be to go down an approach similar to RaspberryPi or our EVM boards where micro SD card is provided for booting and storing the filesystem, and customers can bring their own SSD cards to have larger storage. Or the Beagle approach where the filesystem is flashed onto eMMC and customers can bring their own SSD.

Regards,

Takuma

0 Matthew Fatheree 4 days ago in reply to Takuma Fujiwara

Prodigy 40 points

Takuma,

Would it be possible for you to share the device tree, and the version of u-boot you were using to get the device enumerated on your J721S2 board?

With that I might be able to get a better idea for the scope of the changes that need to be made in the J722S-evm board.

Thanks

0 Takuma Fujiwara 4 days ago in reply to Matthew Fatheree

TI__Guru 50873 points

Hi Matthew,

For my previous experiment, it was using default devicetree for J721S2. However, I had to add below to U-Boot kernel configuration file (j721s2_evm_a72_defconfig):

CONFIG_CMD_PCI=y
CONFIG_PCIE_CDNS_TI=y
CONFIG_PCI=y
CONFIG_NVME=y
CONFIG_NVME_PCI=y
CONFIG_CMD_NVME=y

Regards,

Takuma

Processors

Processors forum

J722SXH01EVM: u-boot PCIe support