RTOS/PROCESSOR-SDK-AM65X: MSI Interrupts with TI-RTOS

Edge triggered SPI can be used for message interrupts by mapping GICD_SETSPI_NSR in place of GITS_TRANSLATER.  Is there any reason from your use case why LPI (vs edge triggered SPI) must be used to service MSI?

Also is there any reason legacy interrupt cannot be used for this use case?

Thank you very much for the updated example John.

Unfortunately, we haven´t been able to make the MSI interrupts work. This is the environment that we have used:

- Root Complex: AM65x IDK (TMDX654IDKEVM)
- End point: a FPGA card.

We have adapted our code with the updates included in the example, but still, the Root Complex does not receive the MSI interrupts generated by the FPGA.

In order to discard any error that we might have introduced, we decided to use exactly the example provided by TI. Therefore, we have setup 2 AM65x IDK boards (RC and EP) connected by a PCIe cable, taken into account the considerations mentioned here:

http://processors.wiki.ti.com/index.php/Processor_SDK_RTOS_PCIe

http://processors.wiki.ti.com/index.php/PCIe_CableMod

After a lot of tries, we also haven´t managed to make it work. It seems that the Link cannot be configured, the program waits forever in the function “pcieWaitLinkUp”.

Any idea about what are we doing wrong? Could you give us some hint about how to continue?

Thanks a lot for your support.

Best regards,
Ismael Pérez

I'm going to re-run based on publicly released installer (from http://www.ti.com/tool/processor-sdk-am65x) and post my logs.  We can then follow up as apropriate. 

The example from the release works.  I installed fresh install on linux from above link, built the pcie example with:

cd ~/ti/pdk_am65xx_1_0_4/packages

source pdksetupenv.sh

make pcie

Its not easily possible to create one workspace with one target configuration that connects to two evms, because of the what dmsc (system controller is loaded).

Best way to use CCS is to use two separate workspaces with two separate target configurations.  I called them workspace_v8 and workspace_v8b.  I also created two target configurations following the instructions in http://software-dl.ti.com/processor-sdk-rtos/esd/docs/05_03_00_07/rtos/How_to_Guides.html#am65x-debug-software-setup.  The first step is to just create them following the link.  If you only have one IDK connected to your PC then either of the two target configs should work and connect.  The final setup step is to enable them both to work at the same time.  

Identify the serial numbers of your IDKs as follows (yes I switched to a windows machine for ccs, z:\ is mapped to that linux machine where I built).

C:\ti_8_3_0\ccsv8\ccs_base\common\uscif\xds110>xdsdfu -e
USB Device Firmware Upgrade Utility
Copyright (c) 2008-2018 Texas Instruments Incorporated.  All rights reserved.

Scanning USB buses for supported XDS110 devices...


<<<< Device 0 >>>>

VID: 0x0451    PID: 0xbef3
Device Name:   XDS110 Embed with CMSIS-DAP
Version:       2.3.0.17
Manufacturer:  Texas Instruments
Serial Num:    30180106
Mode:          Runtime
Configuration: Standard

<<<< Device 1 >>>>

VID: 0x0451    PID: 0xbef3
Device Name:   XDS110 Embed with CMSIS-DAP
Version:       2.3.0.17
Manufacturer:  Texas Instruments
Serial Num:    32180127
Mode:          Runtime
Configuration: Standard

Found 2 devices.

For each of the two working target configurations, edit the two *.ccxml (double click from Target Configurations), switch to the "Advanced" tab, highlight "Texas Instruments XDS110 USB Debug Probe_0", and you will see "Debug Probe Selction" on right.  Change that to "Select by serial number" and enter one of your two serial numbers in the box that appears below.  After clicking "Save", you can use the "Test Connection" button to see if it finds the EVM.  Repeat with the second *.ccxml, using the second serial number.

Now launch each of the *.ccxml from the two separate instances of CCS with separate workspaces.  The "launch_am65xx.js" should automagically run when you launch, it should dump you into the R5 and show the following on the Console:

DMSC Board Configuration with Debug enable 
 DMSC Firmware Version 19.2.0-v2019.02 (Curious Crow)
 Firmware revision 0x13
 ABI revision 2.5
 DDR PLL set to 400.000000 MHz 

The IO/interaction with pcie example occurs on the UART.  Each EVM has 4 UARTs starting at COMbase to COMbase+3.  The interaction will occur on the first uart for each board.  Find COMbase from device manager.  You may have to unplug/plug USB to figure out which board is which.  The speed is 115200/8/n/1

Connect to the CortexA53_0_0 for each of the two boards, load Z:\ti\pdk_am65xx_1_0_4\packages\ti\binary\PCIE_sample_ExampleProject\bin\am65xx_idk\PCIE_sample__am65xx_idk_mpu1_0Example_Project_release.xa53fg on both, and run.  You should see the following on the UART:

**********************************************
*             PCIe Test Start                *
Enter: E for Endpoint or R for Root Complex

On one EVM enter "E" and on the other enter "R" (no quotes).

When it works you should get:

**********************************************
*             PCIe Test Start                *
Enter: E for Endpoint or R for Root Complex
r
*                RC mode                     *
*                RC mode                     *
**********************************************

Version #: 0x02030003; string PCIE LLD Revision: 02.03.00.03:Apr 23 2019:09:16:33

PCIe Power Up.
SYSFW  ver 19.2.0-v2019.02 (Curious Crow) running
Serdes Init Complete
PLL configured.
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Set lanes from 3 to 1
Starting link training...
Link is up.
Root Complex received data.
RC waiting for 10 of each of 2 types of interrupts
RC got all 20 interrupts
UDMA memcpy application started...
Speed: 4410 Mbps
UDMA memcpy using TR15 block copy Passed!!
Test passed.

**********************************************
*             PCIe Test Start                *
Enter: E for Endpoint or R for Root Complex
e
*                EP mode                     *
*                EP mode                     *
**********************************************

Version #: 0x02030003; string PCIE LLD Revision: 02.03.00.03:Apr 23 2019:09:16:33

PCIe Power Up.
SYSFW  ver 19.2.0-v2019.02 (Curious Crow) running
Serdes Init Complete
PLL configured.
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Set lanes from 3 to 1
Starting link training...
Link is up.
End Point received data.
End Point sent data to Root Complex, completing the loopback.
EP sending interrupts to RC
End of Test.
UDMA memcpy application started...
Speed: 4376 Mbps
UDMA memcpy using TR15 block copy Passed!!
Test passed.

I attached my executable for a53 in case it helps with debug.  However, don't think this is a build issue since the example is built with a makefile (vs a project).

A couple things that may lead to training forever.  The two things that lead to training forever that i've noticed are both sides configured as EP, and loading the GP EVM binary on IDK or visa versa.

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/PCIE_5F00_sample_5F005F00_am65xx_5F00_idk_5F00_mpu1_5F00_0Example_5F00_Project_5F00_release.xa53fg

Thanks for your detailed explanation John.

Unfortunately, we are still not able to make the example work, the program keeps training forever.

We have followed the steps you described, and we have even used the executable that you provided, but the result is always the same, the programs stop at “Starting link training...”:

**********************************************
*             PCIe Test Start                *
Enter: E for Endpoint or R for Root Complex
e
*                EP mode                     *
*                EP mode                     *
**********************************************

Version #: 0x02030003; string PCIE LLD Revision: 02.03.00.03:Apr 23 2019:09:16:33

PCIe Power Up.
SYSFW  ver 19.2.0-v2019.02 (Curious Crow) running
Serdes Init Complete
PLL configured.
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Set lanes from 3 to 1
Starting link training...


**********************************************
*             PCIe Test Start                *
Enter: E for Endpoint or R for Root Complex
r
*                RC mode                     *
*                RC mode                     *
**********************************************

Version #: 0x02030003; string PCIE LLD Revision: 02.03.00.03:Apr 23 2019:09:16:33

PCIe Power Up.
SYSFW  ver 19.2.0-v2019.02 (Curious Crow) running
Serdes Init Complete
PLL configured.
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Set lanes from 3 to 1
Starting link training...

We have also tried CCS8 and CCS9, but it makes no difference.

    - Do we need to configure some jumper or switch in the AM65x IDK board that will run as EndPoint in order to set this board in EndPoint mode?

    - According to the instructions described in http://processors.wiki.ti.com/index.php/PCIe_CableMod, we have modified the PCIe cable, cutting the traces to isolate the reference clocks. Is that ok?

    - Have you also run the example using two AM65x IDK (TMDX654IDKEVM) boards?

    - Have you made any adaptation to the GEL files or to the launch_am65xx.js file?

If you have any other hint or suggestion, we will appreciate it.

Thanks and best regards,
Ismael Pérez

I used the IDK version of the board.  No jumpers/switches were modified to enable PCIe.

I currently have a PE-FLEX1-G2-MMCX-12-TI1 (probably the one used to take the photos on processor.wiki.ti.com).

I used the GEL files that came with CCS 8.3

We can completely eliminate all the gel files/ccs by booting it from sd card.  If you need to create a new SD card use the directions in http://software-dl.ti.com/processor-sdk-rtos/esd/docs/latest/rtos/index_overview.html#tools-create-sd-card-linux-label and burn ~/ti/processor_sdk_rtos_am65xx_5_03_00_07/prebuilt-sdcards/am65xx_idk/ onto the card.  Once you have a working card, back up and replace tiboot3.bin and app (just copy the files over to the SD using any linux or windows pc).  The file tiboot3.bin and sysfw.bin comes from ~/ti/processor_sdk_rtos_am65xx_5_03_00_07/prebuilt-sdcards/am65xx_idk/sd_card_files/tiboot3.bin and I attached a copy here to avoid confusion.  I converted the *.out attached in prior post into app using the instructions in ~/ti/pdk_am65xx_1_0_4/packages/ti/boot/sbl/tools/scripts/K3ImageGen.sh into "app" and attached.

Update: forum seems to only take one attachment per post so zipped the 3 files (app, sysfw.bin, tiboot3.bin) that go on sdcard.

If this works then shows the problem is with ccs/gel/javascript.

If it doesn't then its related to hw setup.

sdcard.zip

Hello John,

following your instructions, we have tried to boot both boards from SD card, but we still have the same issue (the link training lasts forever). Therefore, as you have pointed out, it seems that the problem has to do with the hardware. We have checked our HW setup several times, but everything seems to be ok, we are run out of ideas... I send you some pictures of our setup (“HW setup.zip”), maybe you can see something.

Thanks and best regards,
Ismael Pérez

HW setup.zip

Hello John,

We have found the source of the problem: the cable we were using was not a cross cable. Now, with a cross cable, the pcie_sample runs without problems and the test is passed.

We are very sorry for the inconveniences we have caused, we really thought that the cables we ordered were cross cables...

Thanks and best regards,
Ismael Pérez