J784S4XEVM: Booting Jacinto 7 through PCIe from another SoC

Hi KR,

Due to US holiday today, please expect a delay of 1-2 days in responses.

Regards,

Josiitaa

Hi Cedric,

Hardware supports PCIe boot, but no software support for PCIe boot for J784S4/J7x devices.

Regards,

Takuma

Hello Takuma,

Is there any clue on how to get started? I mean, I guess we need to configure BAR, then write into those to initiate boot, but I have no clue on which file I need to write (maybe tiboot3.bin?)

Thanks

Hello Ajins,

Thanks for the details. Takuma, who is handling this thread, is traveling today and must be able to look at this on Monday.

Thanks.

Hi Cedric,

This requires a PCIe driver in U-Boot to be written and/or PCIe driver for SBL boot flow to be written. PCIe driver for TDA4VH/J784S4 is only available for Linux, so it will be hard to get PCIe boot functional.

The recommendation is to use an onboard storage like eMMC to boot the board and then start PCIe link via Linux, instead of PCIe boot.

Regards,

Takuma

Hello Takuma,

We don't necessarily want to start in PCIe from U-Boot, we can do it from Linux. What we want to acheive is to validate the mechanism to ensure that the Jacinto will properly start in PCIe.

First of all, we don't understand why there are 6 memory regions, and what their purpose is?

Also, following J784S4 AM69A TRM :

What's the mapping between the memory regions displayed after enumeration on the root complex side, and the corresponding end point memory locations mentioned in the documentation?

Is there any doc related to it?

If we want to write on the address location 0x41CF3FE0 as mentioned, what address we should write into on the Root Complex side?

KR,

Cedric

Hi Cedric,

Let me clarify that even non U-boot, we unfortunately do not have support for PCIe boot for J784S4. ROM has the ability to read from PCIe, but it will only read the initial boot loader, tiboot3.bin, through PCIe. However, tiboot3.bin will not fetch additional boot binaries from PCIe, so boot will stall.

So, what I am recommending is to boot the J784S4 EP from a different boot media to set it up as EP, then you may access the J784S4 EP from a RC like TMDS64EVM.

Regards,

Takuma

Hi Takuma,

I am working with Cedric and I  would like to follow up on your previous answer. We really want to boot from PCIe, not from a different boot media. This is then very important for us to understand how it works.

We understand what you are saying and that from a software perspective, the binary tiboot3.bin, which needs to be initially transferred via PCIe, does not handle the continuation of the boot process via PCIe. This indeed requires modifying the code to handle PCIe and to pursue the boot phase.

However, we want to understand how to transfer the tiboot3.bin.

If I refer to a portion of a screenshot that Cédric already provided to you:

When we are looking at this capture, we clearly observe that 6 regions of sizes 256, 512, 1K, 16K, 128K, 1M have been requested by the endpoint.

Could you explicitly tell us more about the mapping of these 6 regions on the Jacinto SoC side? Are these regions mapped into the shared memory or any where else? Could you provide us the mapped addresses in the Jacinto mapping in order to be able to transfer the tiboot3.bin file?

Thank you for your assistance.

Kind regards,

Kiêt

Hi Kiêt,

L.Kiet said:

Could you explicitly tell us more about the mapping of these 6 regions on the Jacinto SoC side?

Long story short, this maps to some region of contiguous memory that is defined in Linux devicetree as CMA.

For a detailed response...

I assume this is the default EP/RC example from Linux, meaning the EP is running the pci-epf-test.c driver here: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/drivers/pci/endpoint/functions/pci-epf-test.c?h=ti-linux-6.6.y. This defines some arbitrary bar sizes of increasing sizes to demo the fact that different sizes can be allocated.

If the code is followed, then eventually a call to dma_alloc_coherent to allocate memory: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/drivers/pci/endpoint/pci-epf-core.c?h=ti-linux-6.6.y#n285 

dma_alloc_coherent allocates memory from the contiguous CMA memory pool to allow DMA to be used smoothly.

L.Kiet said:

However, we want to understand how to transfer the tiboot3.bin.

For this query, what I just explained above is not useful. This is because the 6 BARs that are created under each PCIe function is only created when the pci-epf-test Linux driver is ran. And, since tiboot3.bin is required before Linux, U-Boot, or any other bootloader is ran, the pci-epf-test would not have been ran when tiboot3.bin is needed.

Even though I understand that you want PCIe boot, I still do not recommend PCIe boot. It will add large complexity to your system.

To explain the complexity of a full PCIe boot, the bootflow looks like:

1. ROM code initializes PCIe and waits for tiboot3.bin
2. tiboot3.bin initializes PCIe and waits for tispl.bin
3. tispl.bin initializes PCIe and waits for uboot.img
4. uboot.img initializes PCIe and waits for Linux kernel and devicetree
5. Linux kernel initializes PCIe and mounts the filesystem from PCIe

We do not have steps 2, 3, and 4, so you will have to develop a PCIe driver for 3 different codebases. On top of that, each driver needs to gracefully give control of the PCIe hardware to the next step of the boot, and RC must re-enumerate the EP at each stage, and synchronize the sending of the binary from RC to EP at each stage.

Instead, if you do eMMC or OSPI boot that is supported in the SDK with PCIe initialized only in Linux, the bootflow would be:

1. ROM reads from eMMC or OSPI the tiboot3.bin, which reads tispl.bin, then uboot.img, Linux kernel, devicetree, and filesystem.
2. Linux initializes PCIe to do PCIe transfer.

For eMMC or OSPI boot, all steps are supported in SDK. Boot time will be significantly faster than PCIe boot. And at the end, you will still have PCIe to transfer data.

Regards,

Takuma

Hi Takuma,

Thank you for your reply. Indeed, I am sorry, I went too fast on what was written by Cédric. The screenshot is indeed between two Linux systems.

Cédric tested again and provided me a screenshot from the endpoint showing the BAR assignment.

It looks better and shows two BARs (both with a size of 32 MB).

I repeat the same question about these newly mapped regions: could you provide us with the mapped addresses in the Jacinto mapping in order to transfer the tiboot3.bin file? Where should I write the tiboot3.bin file? In BAR0 or BAR1, and at offset 0 or another address?

Locally on the endpoint side, what are the addresses of the 2 mapped regions?

We are designing our custom board and we need to connect one SoC to another SoC via PCIe. We can't use either eMMC or QSPI for the second SoC.

I saw the steps you described. However, once the tiboot3.bin is transferred and booted, I think we could follow these steps:

1. ROM code initializes PCIe and waits for tiboot3.bin.
2. tiboot3.bin initializes PCIe and waits for tispl.bin (in MSM?).
3. tispl.bin initializes PCIe and LPDDR4 and waits for uboot.img, device tree, Linux Kernel, and RAM rootfs (in LPDDR4).

Thank you in advance.

Best Regards,

Kiêt

Hi Kiêt,

This is a warning that this is a path not taken by others. I worry that you will go down this path and hit a roadblock, and we will not be able to help you. Support from TI will practically be none for implementing booting through PCIe after step 1 of loading tiboot3.bin.

On the other hand, the alternative solution I suggested has customers that have went down this eMMC/OSPI boot path for dual SoC solutions that primarily use PCIe to communicate with each other. So, this is a known working design. I understand you want PCIe boot (probably for less BoM cost), but please understand that there are alternative boot solutions for a custom board that has two SoCs connected via PCIe.

Now, with the warning in mind, getting back to your questions...

There should be two more BARs for a total of 4 BARs like below:

01:00.0 Non-VGA unclassified device: Texas Instruments Device b010
        Flags: fast devsel, IRQ 536
        Memory at 68100000 (32-bit, non-prefetchable) [virtual] [size=1M]
        Memory at 68200000 (32-bit, prefetchable) [virtual] [size=2M]
        Memory at 6a000000 (64-bit, prefetchable) [virtual] [size=32M]
        Memory at 6c000000 (64-bit, prefetchable) [virtual] [size=32M]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [160] Power Budgeting <?>
        Capabilities: [180] Resizable BAR <?>
        Capabilities: [1b8] Latency Tolerance Reporting
        Capabilities: [1c0] Dynamic Power Allocation <?>
        Capabilities: [300] Secondary PCI Express
        Capabilities: [400] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [440] Process Address Space ID (PASID)
        Capabilities: [4c0] Virtual Channel
        Capabilities: [900] L1 PM Substates
        Capabilities: [a20] Precision Time Measurement
        Kernel driver in use: pci-endpoint-test
        Kernel modules: pci_endpoint_test

Maybe you are using a PC that requires sudo permissions to see the extra two BARs. 

In my logs, 0x68200000 is BAR1, which is mapped to 0x41C00000. TRM should have the register that should be written to for start address location and location to write magic word to indicate transfer is complete.

Regards,

Takuma

Hi Takuma,

Yes, I understood that you are not supporting it and could not provide help for the next steps of the boot flow. We have some reasons to boot from PCIe, and not from flash, and it is not about the BOM cost. One of them is the space to route a new flash on the board since we are designing complex boards.

I do not know why we do not see the BAR of 1 MB. We will check it and come back to you later if needed.

Indeed, we saw in the TRM the needed information to continue the boot once tiboot3 is transferred.

Thank you for your support.

Regards,

Kiêt

Hi Kiêt,

Best of luck in your endeavors!

Regards,

Takuma