Part Number: AM5728
Other Parts Discussed in Thread: SYSBIOS
Tool/software: TI-RTOS
Hello,
We have an AM5728 custom board. The ARM is the main controller and runs under Linux. The DSP runs TI_RTOS and is connected to an FPGA via GPMC for loading and PCIe for communication. The overall process works like this:
- FPGA Image is stored on eMMC.
- Linux reads the image into memory shared by the ARM and the DSP.
- Linux sends the DSP a message via MessageQ that the image is loaded in memory.
- The DSP writes the image to the FPGA via GPMC.
- The FPGA responds with a hardware signal saying it was loaded successfully.
This process generally works just fine, but almost always fails after a power cycle of the board or when the image file to be loaded changes.
In debugging the issue, we noticed that occasionally the DSP would read the image memory and it would have the value 0x55555555 which is not present anywhere in the image file. Any time the DSP saw a 0x555555555 in the data the FPGA programming would fail. If we did nothing else but repeat the process above (ie no power cycle or reboot) the image would then load successfully.
At this point we suspected some kind of cache issue, though not sure which side it is on, ARM or DSP. On the ARM side an attempt was made to sync the transfer to force the data transfer from eMMC to memory to complete before telling the DSP to start the download. However, from our Linux developer,
-
I’m currently using a mechanism that’s documented to do no caching on the Linux side of things. In fact, the memory driver doesn’t even implement the fsync capability because it’s documented to go right to physical memory – Linux doesn’t do anything in between.
So, that seemed like a dead end.
On the DSP side, I attempted to make the entire shared memory area un-cached via the .cfg file like this:
var Cache = xdc.useModule('ti.sysbios.family.c66.Cache');
Cache.setMarMeta(0x8CA80000,0x01000000,0);
0x8CA80000 is the DSP side memory address. The Linux side is defined via the resource table at 0xF0000000.
That did not seem to help matters at all. So, then I tired to invalidate the cache when I read the data from memory like this:
if ((i % 32) == 0)
{
Cache_inv(&fpgaDownload[i],128,Cache_Type_ALL,true);
}
"i" is a count of the 32 bit words, so "i" mod 32 is every 128 bytes, which I think is the cache line size.
Again, no luck, behavior is the same. Keep in mind, this does occasionally work, and almost always works if the ARM writes the same image to memory twice.
I then did an interesting debug log in the DSP. Before I attempted to load the FPGA I scanned the memory for 0x555555555 in a loop. If I detected a 0x55555555 I logged an error and the offset into
the memory, slept for 1 second and I started over again, continuing to loop until the load was successful. To my surprise it worked, but took 30 seconds! Notice how the address that 0x55555555 was
detected moves down in the file, sometimes repeating over and over, over a matter of seconds. It is like the ARM thinks the memory transfer from the eMMC is complete, but it really is not.
Fpga: Loading FPGA image of size 5159580 bytes
Fpga: Scanning FPGA image found fives at 1744960 bytes
Fpga: Scanning FPGA image found fives at 1753152 bytes
Fpga: Scanning FPGA image found fives at 1753152 bytes
Fpga: Scanning FPGA image found fives at 1753152 bytes
Fpga: Scanning FPGA image found fives at 1753152 bytes
Fpga: Scanning FPGA image found fives at 1753152 bytes
Fpga: Scanning FPGA image found fives at 1753152 bytes
Fpga: Scanning FPGA image found fives at 1753152 bytes
Fpga: Scanning FPGA image found fives at 1753152 bytes
Fpga: Scanning FPGA image found fives at 1753152 bytes
Fpga: Scanning FPGA image found fives at 3223680 bytes
Fpga: Scanning FPGA image found fives at 3223680 bytes
Fpga: Scanning FPGA image found fives at 3223680 bytes
Fpga: Scanning FPGA image found fives at 3223680 bytes
Fpga: Scanning FPGA image found fives at 3981696 bytes
Fpga: Scanning FPGA image found fives at 4387072 bytes
Fpga: Scanning FPGA image found fives at 4387072 bytes
Fpga: Scanning FPGA image found fives at 4387072 bytes
Fpga: Scanning FPGA image found fives at 5030080 bytes
Fpga: Scanning FPGA image found fives at 5030080 bytes
Fpga: Scanning FPGA image found fives at 5030080 bytes
Fpga: Scanning FPGA image found fives at 5030080 bytes
Fpga: Scanning FPGA image found fives at 5030080 bytes
Fpga: write image time: 30261 msec 1289895 32-bit words
Fpga: Done went high after 0 msec!
(Each line above is 1 second interval)
So, any additional ideas on how to troubleshoot this further or how to make sure the file is completely loaded before letting the DSP use it?
Thanks,
Chris
