TMDSEVM6678: PCIe boot loader issue with 64-bit ARM host

Some additional details which might be helpful:

The boot loader configures 5 BARs of non-prefetchable memory on the device:
- BAR0 (32-bit, 4 kB) hardwired for "application registers" (PCIe config)
- BAR1 (32-bit, 32 kB) used to access device/peripheral registers
- BAR2 (32-bit, 512 kB) used to access LL2 memory for each of the eight cores
- BAR3 (32-bit, 4 MB) used to access MSM memory
- BAR4:5 (64-bit, 256 MB) used to access DDR memory

The kernel driver creates 11 character devices, for which the loader tool expects to find device files in "/dev/atvdsp":
- "control" for accessing the device/peripheral registers -- IOCTL is used to change the base address
- "ll2_core<0-7>" for each of the eight LL2 memories
- "msm" for the MSM memory
- "ddr" for the DDR memory

I have noticed that on the desktop PC, `lspci -vv` reports that the PCI device has a 64-byte cache line.
But on the 64-bit ARM host, it reports that the PCI device has a 32-byte cache line.
I don't know the significance of this, but it seems strange to me as I believe it is a fixed device-capability.

If I read any of the memories on the 64-bit ARM host, I get junk content.

The junk content changes if I reboot the system, but does not change between reads.
If I write to the memory, the content changes accordingly and I can read it back.
So I am apparently accessing a memory somewhere, but I don't understand where.

Apparently, the junk does also have some pattern in it.
If I group the bytes together 8 and 8, the even groups tend to be 0xff, while the odd groups tend to be a bit more random.
But I have found that the odd groups are not entirely random, as I can find byte sequences which repeat every 64 kB.

This all has me very confused...

Hi Martin,

I've forwarded this to the software experts. Their feedback should be posted here.

BR
Tsvetolin Shulev

Hi,

I would like to know some details:

1) The boot loader is based on the PCIe driver example from the PDK v2.0.4, located in packages/ti/drv/pcie/example/sample/src/. ===> do you use the 6678 EVM in PCIE boot mode? In the EVM, FPGA is programed to force the DSP boot from I2C address 0x51 (EEPROM) in PCIE boot mode. In EEPROM, we have the IBL code flashed there which intialized the PCIE perihphral. So I wonder how you use the PCIE example code as the PCIE boot loader code?

2) I have been able to run the regular PCIe boot demo with the 64-bit ARM host =====> is this a 64-bit Linux PC? Do you use 32-bit BAR or 64-BAR? If you used 64-bit BAR, demo works meant you have no problem in R/W DSP memory? Then why it is problematic when you porting to another 64-bit host NXP LS1043A-RDB?

3) From your description, you have PCIE enumerated but don't have the correct mapping. If I read any of the memories on the 64-bit ARM host, I get junk content.=====> If you use BAR1, 2, 3 which are 32-bit BAR, do you get junk R/W?

4) Are you able to locate the junk data by searching the whole DDR range, when using 64-bit BAR4/5? I think EVM only have 512MB DDR.

Regards, Eric

Hi Eric,

lding said:

1) The boot loader is based on the PCIe driver example from the PDK v2.0.4, located in packages/ti/drv/pcie/example/sample/src/. ===> do you use the 6678 EVM in PCIE boot mode? In the EVM, FPGA is programed to force the DSP boot from I2C address 0x51 (EEPROM) in PCIE boot mode. In EEPROM, we have the IBL code flashed there which intialized the PCIE perihphral. So I wonder how you use the PCIE example code as the PCIE boot loader code?

We boot the 6678 EVM in ROM SPI boot mode. The `spiboot_pcieinit/` from the ZIP I attached has been programmed to the flash using the NOR writer application.

Does the FPGA force the DSP to run other code before booting from the NOR flash?
Then I suppose the IBL and/or RBL for PCIe boot mode may be doing something as part of its initialization which I also need to do.

lding said:

2) I have been able to run the regular PCIe boot demo with the 64-bit ARM host =====> is this a 64-bit Linux PC? Do you use 32-bit BAR or 64-BAR? If you used 64-bit BAR, demo works meant you have no problem in R/W DSP memory? Then why it is problematic when you porting to another 64-bit host NXP LS1043A-RDB?

Yes, the 64-bit ARM host is a Linux system, and I simply cross-compile the kernel module and loader tool for the ARM platform.
It is operating in little-endian mode, same as the desktop PC, by the way.

Sorry, I should have been more specific with what I meant with "the regular PCIe demo", because this does not use our boot loader, kernel module or loader tool.
I meant booting the EVM in ROM PCIe boot mode and using the linux host loader in `pdk_c667x_2_0_4/packages/ti/boot/examples/pcie/` to load the helloworld application. The ROM PCIe boot mode sets up only 32-bit BARs, I believe.

Our boot loader sets up the largest BAR, for DDR, as 64-bit, and the four others as 32-bit.
And the ARM host is configured to allocate 32-bit PCI addresses, by the way.

lding said:

3) From your description, you have PCIE enumerated but don't have the correct mapping. If I read any of the memories on the 64-bit ARM host, I get junk content.=====> If you use BAR1, 2, 3 which are 32-bit BAR, do you get junk R/W?

Well, I haven't quite concluded that the problem is with the inbound mapping yet. I'm not an expert on PCIe, so for all I know, there could be some sort of clock/timing or signalling issue due to missing initialization steps that could cause this.

I'll try to modify the boot loader to periodically print out the inbound mapping and perhaps some other data. That should hopefully help narrow down the issue.
Thanks for the idea!

Yes, the memories which BARs 1, 2 and 3 map to read as junk. But writes to them do not become junk.
If I write to one of the memories, I can read back what I have written, and the junk does not change between reads.

lding said:

4) Are you able to locate the junk data by searching the whole DDR range, when using 64-bit BAR4/5? I think EVM only have 512MB DDR.

I have not attempted this, but I can try.

By the way, another thing I noticed is that the junk looks _similar_ (but not the exact same!) in all memories.
It is as if there is some common junk data, 64 kB in size, that has noise added, and the noise depends on which memory you're reading.
The exceptions are the LL2 memories which apparently have the same noise.

Thank you for your input!

Best regards,
Martin Etnestad

Good to know the issue resolved and thanks for sharing the feedback!

Regards, Eric

Hi Martin,
Could you please share the code (after correction). We are also trying to boot C6678 from 66AK2H12 (K2) processor thro PCIe.
My mail Id is sivanantham.subramoniam@gmail.com

Thanks in advance.

Regards,
S.Sivanantham