PCIe transaction between two DSP C6678

I think in the PCIe LLD example, both of the RC and EP are configuring themselves locally, which means there is no remote configuration from RC to EP.

Please correct me if I miss anything, but if it is the case, we probably do not need to update the example test case when adding the PCIe switch between them.

The RC will not do the enumeration in the example since it is designed to peer-to-peer communication. And the LLD provides some pre-defined APIs for the users to configure the PCIe more efficiently. You can program the registers directly as well but they (LLD vs other programming) are the same from the configuration point of view.

So I guess there is probably something missing in the PCIe switch part. Do you have any document along with the switch please?

I am wondering there is some configuration required for the switch besides the physical connection of RC and EP.

Sincerely,

Steven

thanks Steven,

on the PCIe switch, i configured the ports :

- upstream port which connected to the RC (DSP1).

-dowstream port which connected to the EP (DSP2).

Also, i configuerd routing tables ("the switch routes the paquets from DSP1 to DSP2");

but i would like to know if i need to change somthing in the example, or change the state of SW6[4:3] ("PCIe sybsystem selection") on the EVM,

by default it is set to the value "00" (PCIe enpoind mode).

when i  run the example in RC mode, do I have to change the state of the SW[4:3]  to "10" ?

because i need that the DSP host (RC) enumerates my system ("switch + DSP2")

thank you,

The boot SW on the EVM are used to configure the boot mode. The PCIe LLD example could be loaded in no boot mode and it will configure the PCIe module again in the test case, so the boot SW setup should not matter.

May I ask if you could connect the two EVMs directly via PCIe link (bypass or without the PCIe switch) and complete the PCIe loopback transfer between the two EVMs successfully please?

We need to confirm if the test case you compiled are running properly on EVMs first without the PCIe switch. Thanks.

Sincerely,

Steven

it's what I try to test for now, i.e. transfer data from a DSP to another DSP through the PCIe switch.

the programs are launched for both DSPs ( RC and EP ). but the EP did not receive the Data sent from the RC !!

It is better to confirm the data transfer from one DSP to another DSP WITHOUT the PCIe switch is working in your setup first.

We can verify that if the EVM hardware and the PCIe LLD example you compiled are working correctly.

Could you confirm this please? Thanks.

Sincerely,

Steven

hello Steven,

i don't have possiblity to connect both DSPs directly, the only possiblity that i have, is using the PCIe switch.

i use the PCIe_sample program, then i configured both of the DSPs as Endpoints.

the first DSP (BAR=7000_0000) sent data to the second DSP (BAR=9000_0000), so I configured the switch registers so that the given packet address  is included in the range of the base / limit register of receiver port.

the program is executed on the DSP correctly , but data transfer between two DSPs has not been established.!!

any suggestions please !!

1. First of all, the PCIe LLD example is designed to be run on one RC and one EP, not two EPs.

The RC sends data to EP first and then waits for data back from EP. The EP waits for the data from RC first and then sends data to RC.

If you configure both of them as EPs, both of them will wait for the data and no transfer will happen.

2. If you configure one DSP as RC and one DSP as EP (as you mentioned before), do you see the message from any DSP showing "Link is up." please?

I thought the link up is not established when you add the switch, but I would like to confirm that. You can also check the LTSSM_STATE field in DEBUG0 register (0x21801728). LTSSM_STATE=0x11 means the link is up, otherwise we have issue to establish the link and the data transfer will not happen.

3. The key point for the data transfer is to make the PCIe address sent from one PCIe can be accepted by the other. It means the PCIe address after the outbound transaction in RC should be in the range of the BAR register of EP and vice versa.

In the PCIe LLD example, the default setup of RC outbound transaction is "PCIE_OB_LO_ADDR_M" (0x7000_0000) and the EP BAR register setup is "PCIE_IB_LO_ADDR_S" which is 0x7000_0000 as well. That will make sure the PCIe packets from RC will be accepted by EP.

And it is same for EP to RC transfer, that EP outbound transaction is the same as RC BAR register (0x9000_0000).

Added: so you may need to include 0x7xxx_xxxx ~ 0x9xxx_xxxx to the PCIe switch support range.

So please configure one DSP as RC and one DSP as EP first. Load the original example test to EP compiled with "PcieModeGbl=pcie_EP_MODE" and load the other test case to RC with"PcieModeGbl=pcie_RC_MODE". And debug in the CCS on both DSPs to see at which stage the program is blocked. Thanks.

thank you very much Steven for your replay,

i will explain again what  i'm looking for.

I’d like to make transactions between two PCIe DSPs through a switch.

For that there are two possibilities:

1-      Use DSP1 as RC and DSP2 as an EP in one partition of the switch.
I tried this option, the message "link is up" is displayed, "Data sent from RC» too.
But I need to know if the DSP (RC) do the process of enumeration (bus number, device number…) of the entire system including the switch and the other DPS that is an endpoint.
So the 2nd DSP (EP) receives nothing even if I set up the Switch registers with address ranges (base / limit registers) corresponding to the RC outbound and the EP inbound.

2- Use DSP1 and DSP2 as two EPs in the same partition on the Switch that is connected to the computer which is the RC in this case.
For this option, I adapted the PCIe LLD program,so a first DSP (EP1) sends data to another DSP (EP2) through the switch . this later is already configured in such a way to route packets from DSP1 to DSP2 respecting any address ranges in outbound and inbound.

i hope that you understand my situation !!

Hello Abdel,

I also work on the exchange of data between two EVMB c6678 via PCI-Express switch (PEX8648). One EVMB c6678 - Root Complex, another EVMB c6678 - End Point.

I modified the example provided by TI (PCIe_sample, that dedicated to direct communication between two DSP Shannon). Rework for enumerate PCI bus. The implementation enumerate PCI bus using only PCIe LLD.
I have everything working. Two EVMB c6678 exchange data via PCI-Express switch (PEX8648).

If you still need this rework, I can share it.

Hi Dmitry, 

Is it possible to share this code you modified with me?  My email is solson@robotics-redefined.com

I have one c6678 plugged in to PEX8606 RDK and then a USB-> PCIe bridge I am trying to talk to, but I need help with the enumeration procedure.  Thanks!

Hi Scott,

I sent you the code to the email.

May I ask for the copy too? rrlagic at gmail dot com

Hi Dmitry, it didn't seem to come through, can you try changing the extension if it is a .zip file to something else, like .txt?

Sent ........

Thanks Dmitry, that came through.  Much appreciated.

Hi Dmitry, I am curious about the following code snippet in pcie_bus.c

if(inDevNode->priBusNumber == 0)
{
devCountMax = 1;
currentRequestType = pcieReqestType0;
}

Can you explain the significance of priBusNumber?  When I am attempting to enumerate, I always get back only 1 node and it is VID and PID of 0; I am curious if I am doing something wrong.

-Scott

Hi Scott

Answers with explanations to send by e-mail.

Dear Dmitry,

  I have the same problem of Abdel Abdou, could you send me, if possible, your code version of the MMC for the EVMB c6678 please? Maybe I have the wrong version.

  In my case, the two EVMB c6678 are plugged in a MICROTCA backplane where a switch is present. I configure one EVMB as RC and the other as EP. Both PCIe link are up, but no packet are sent between the EVMB. I think that there is to configure in the MMC code.

Regards, 

 Massimiliano Bitossi

Dear Dmitry,

  I forgot, not only the MMC code but also your PCIe_sample software that you changed that is my main problem.

Dear Massimiliano,

а)To work "PCIe_sample" something changed in the MMC (Module Management Controller.  If you talk about it.) code is not necessary. MMC code only to activate  (power supply only) the module EVMB c6678 in the MTCA chassis.

b) Implementation of IPMI on EVMB c6678 (in MMC) is not very successful. IPMI implementation is not complete, in accordance with the specifications of PICMG.
I worked with different MCH. In each case, something was not right.
I've successfully used to activate (for power supply only) the module EVMB c6678 in the chassis MTCA with SMCH-100 or NAT-MCH. I used the  "EVM_ipm430_v1.32" (for MMC without any code modifications) for this MCHs.

с)  Where to send the modified "PCIe_sample"?

Dear Dmitry, 

my mail is massimiliano.bitossi@pi.infn.it

Thanks a lot for you suggestions. It will be very useful. I do not need the code right now since Scott sent me your code yestarday. It works!

Now we are trying to modify it to execute measurements on the latency and on the speed of the PCIe in our test stand (two EVMc6678 in a uTCA backplane with a NAT-MCH switch).

Regards

  Massimiliano

Dear Massimiliano,

Best of luck!

Regards

  Dmitry

Hello, Dmitry.

I need your PCI Bus enumeration source code too.

Please....

E-mail : sclee@dawonsys.com

Thank you.

Best regards.

Hi Sang Cheol,

I sent you the code to the email.

Regards

Dmitry

Dear Dmitry and all,

I am using the PCIe example code under "C:\ti\pdk_C6678_1_1_2_6\packages\ti\drv\pcie\example\sample" and  "pcie_sample.c" to understand how to interface C6678 (RC)  to our SPARTAN6 FPGA (EP)  using the PCIe. In the example, they are using C6678 EVM and address translation is enabled. in my case, we instantiated a 4K DPRAM in the FPGA where we store data into and want to read the contents into the C6678 Multicore shared memory or DDR3 so all cores can access the same data. Any advice on how to change the code to achieve this specially the address translation part?. please if you have any code to share, my email is

mqahwash@devry.edu

Regards,

Murad

Murad, you're right.

The only way to get a high data rate of exchange is used EDMA3 in c66x or CDMA in FPGA (Spartan-6). What to choose as a basis (EDMA3 or CDMA) depends on the system you are building.
I have used and EDMA3 (c6678, PCIe x2 2.0) and CDMA (Virtex-6, PCIe x2 2.0).
The maximum baud rate was approximately 680 MByte/s. The maximum baud rate depends on your model of data exchange between two devices and implementation in FPGA the hardware platform.

Regards,

Dmitry.

Hello again Dmitry,
Sorry to bother you again, but can you look at my thread below. I am having a problem getting the EDMA3 to work with PCIe...it works with local buffer and the PCIe works without EDMA but not EDMA for PCIe!!??

http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/387481

Hello Dmitry,

I’m writing to get information about your work which uses PCIe switch and two c6678 EVMs. My aim is to transmit and receive data packets between these two c6678 EVMs via PCIe switch(from IDT). I have configured one EVM as Root Complex and another as End Point. Firstly, I've achieved loopback data transfer between the two EVMs successfully without PCIe switch.

Then I've connected Root Complex to upstream port of PCIe switch and End Point to the downstream port. I think that I couldn't achieve bus enumeration done by Root Complex. Could you suggest a way to achieve bus enumeration procedure using one c6678 EVM as Root Complex? Is it possible to share your modified code with me?

I look forward to hearing from you

Best regards

Alican

Hello Dmitry,

I need your PCIe Bus enumeration source code too. (Your work which uses PCIe switch and two c6678 EVMs.) 
 
E-mail : aeslek@aselsan.com.tr

Thank you.

Best regards.

Dear Dmitry,

Could you please share the example project and PCIe Bus enumeration source code . (Your work which uses PCIe switch and two c6678 EVMs.).
 
E-mail : sivananthams@bel.co.in

Thanks in advance.

Regards,

S.Sivanantham

Hi Alican,

I received the project files. Thank you very much.

Regards,

S.Sivanantham