C6657 PCIE Problem

jin fu

Hello

I try to build a x1 PCIE with C6657EVM(RC) and FPGA(EP).

I modified the example project ,config the dsp outbound through:

if ((retVal = pcieObTransCfg (handle, 0x90000000, 0, 0)) != pcie_RET_OK)
{
System_printf ("Failed to configure Outbound Address Translation (%d)\n", (int)retVal);
exit(1);

}

config FPGA's BAR0 through:

barCfg.location = pcie_LOCATION_REMOTE;
barCfg.mode = pcie_EP_MODE;
barCfg.base = 0x90000000;
barCfg.prefetch = pcie_BAR_NON_PREF;
barCfg.type = pcie_BAR_TYPE32;
barCfg.memSpace = pcie_BAR_MEM_MEM;
barCfg.idx = 0;

and I dump the data from 0x21802000 ,it shows exactly the same with code above.

so I guess if i want to write data to FPGA's BAR0,I should do something like:

for (i=0; i<PCIE_BUFSIZE_APP; i++)
{
*((volatile uint32_t *)pcieBase + i) = 0x11223344;}

But FPGA can't recive anything.The link is always up.I don't know whether I miss something.

I doubt I can't write data into 0x60000000 ,because when I run:

for (i=0; i<PCIE_BUFSIZE_APP; i++)
{
*((volatile uint32_t *)pcieBase + i) = 0x11223344;
uint32_t j= *((volatile uint32_t *)pcieBase + i);
System_printf ("output=%x\n",j);
}

I get the output like:

[C66xx_0] output=60000000
[C66xx_0] output=1
[C66xx_0] output=60000000
[C66xx_0] output=3
[C66xx_0] output=60000000
[C66xx_0] output=5
[C66xx_0] output=60000000
[C66xx_0] output=7
[C66xx_0] output=60000000
[C66xx_0] output=9
[C66xx_0] output=60000000
[C66xx_0] output=b
[C66xx_0] output=60000000
[C66xx_0] output=d
[C66xx_0] output=60000000
[C66xx_0] output=f
[C66xx_0] output=60000000
[C66xx_0] output=11
[C66xx_0] output=60000000
[C66xx_0] output=13
[C66xx_0] output=60000000
[C66xx_0] output=15

And the memory browser shows all zeros.

Thank you in advice

over 10 years ago

0 Ganapathi Dhandapani95 over 10 years ago

TI__Mastermind 28085 points

Hi,

The Default MCSDK PCIe sample example support x1 mode. In this example, two DSP EVMs are used to test the PCIe driver. DSP 1 is configured as a Root Complex and DSP 2 is configured as End Point. Each DSP configure the PCIe BAR configuration. Once the PCIe link is established, the following sequence of actions will happen:

1. DSP 1 sends data to DSP 2 - DSP 2 waits to receive all the data

2. DSP 2 sends the data back to DSP 1

3. DSP 1 waits to receive all the data

4. DSP 1 verifies if the received data matches the sent data and declares test pass or fail.

Have you follow the same step to configure the FPGA PCIe BAR's?

In the PCIe example, RC and EP use the BAR index PCIe as 1. Better to use the same on FPGA side instead of barCfg.idx = 0;

/* BAR Index PCie*/
#define PCIE_BAR_IDX_M 1
#define PCIE_BAR_IDX_S 1

Thanks,

0 Victor Kazmirenko over 10 years ago

Guru 13202 points

Hello,

I am working on C6670, but I guess their PCIe is same. Because 0x9000_0000 address is already used in my system for IB translation, I have set OB Lo, Hi, Region as 0xA000_0000, 0x0000_0000, 0. FPGA's BAR0 is set to 0xA000_0000. Then write to 0x6000_0008 does trigger PCIe receive procedure on FPGA side and I see TLP address was translated to 0xA000_0008. I monitor that using ChipScope.

So I'd like to ask, how do you judge, that FPGA does not receive your writes. I suggest you not rely on Memory Browser to see remote endpoint memory space. While its still OK to see endpoint config space, don't do that for memory. Be sure, I also see all zeros there, though actual read and writes complete with correct values. I could speculate here, as I know nothing about memory browser design. Perhaps, it uses some larger read sizes, than your endpoint could handle.

I used a loop to write/read to 0x6000_0000 + offset and receive expected values. In my design there is register file matched to BAR0 of FPGA. What is your FPGA user logic? Are you sure there are proper TX, RX engines and memories for such accesses?

So once again, I see no trouble in your setup of OB translation. If pcieBase value corresponds to PCIe Data space, which is 0x6000_0000 in C6670, but not sure what is yours, then read and write transfers to that address with reasonable offsets would produce PCIe transactions, and I do confirm that in my setup. Please be more specific about your configuration and observed trouble, perhaps I could help.

0 jin fu over 10 years ago in reply to Ganapathi Dhandapani95

Prodigy 110 points

Thanks to you fast reply.

I think I configured the FPGA PCIE BAR successfully.

And I'm sure the FPGA Artix-7 can use BAR0 to memory read/write.

Here I print all of the config data about Rc and Ep:

[C66xx_0] ******************App Register:0x21800200****************************
[C66xx_0] offset 0x0000: 90000001 00000000 00000000 00000000
[C66xx_0] offset 0x0010: 00000000 00000000 00000000 00000000
[C66xx_0] offset 0x0020: 00000000 00000000 00000000 00000000
[C66xx_0] offset 0x0030: 00000000 00000000 00000000 00000000
[C66xx_0] offset 0x0040: 00000000 00000000 00000000 00000000
[C66xx_0] ******************local config:0x21801000****************************
[C66xx_0] offset 0x0000: 8888104c 00100146 00000001 00010000
[C66xx_0] offset 0x0010: 00000000 70000000 00000000 00000000
[C66xx_0] offset 0x0020: 00000000 00000000 00000000 00000000
[C66xx_0] offset 0x0030: 00000000 00000040 00000000 000001ff
[C66xx_0] offset 0x0040: 00035001 00000000 00000000 00000000
[C66xx_0] ******************remote config:0x21802000****************************
[C66xx_0] offset: 0x0 : 701110ee 00100002 05800000 00000000
[C66xx_0] offset: 0x10 : 90000000 00000000 00000000 00000000
[C66xx_0] offset: 0x20 : 00000000 00000000 00000000 000710ee
[C66xx_0] offset: 0x30 : 00000000 00000040 00000000 000001ff
[C66xx_0] offset: 0x40 : 78034801 00000008 00806005 00000000
[C66xx_0] offset: 0x50 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x60 : 00020010 00648e02 00002810 0003f411
[C66xx_0] offset: 0x70 : 10110000 00000000 00000000 00000000
[C66xx_0] offset: 0x80 : 00000000 00000002 00000000 00000000
[C66xx_0] offset: 0x90 : 00010000 00000000 00000000 00000000
[C66xx_0] offset: 0xa0 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0xb0 : 00000000 00000000 00000000 00000000
[C66xx_0] ******************DEBUG0:0x21801728****************************
[C66xx_0] offset:0x000 : 03b2e211

Is there something wrong?

Regards,

Jin

0 Victor Kazmirenko over 10 years ago in reply to jin fu

Guru 13202 points

As far as I can see, there is 0x9000_0000 at location 0x10. That is BAR0 and that is correct. Ganapathi's note about using BAR1 instead of BAR0 is irrelevant, it does not apply to your situation. I do confirm successful operation of my design against remote BAR0.
If you have a chance to use ChipScope, try to see, whether endpoint receives transactions.

0 jin fu over 10 years ago in reply to Victor Kazmirenko

Prodigy 110 points

Thx for your reply
I used ChipScope to capture rx_bar_hit[0] which shoulde be asserted when TLP transfered to BAR0, But I got nothing.
regards,
jin

0 Victor Kazmirenko over 10 years ago in reply to jin fu

Guru 13202 points

If so, I would suggest to use trn_rsof signal as a trigger and see whether you receive any transaction at all. In my case there are some debug ports on PCIE block. So I have pulled them through the hierarchy to the top module and observe as well. At least, if something wrong with your BARs, there is a dbg_no_bar_hot signal to check.
Finally, you may disable OB translation at all, or set OB low address as 0x00000000, set remote BAR to 0, and then try to write to 0x6000_0000. That should land to 0 address. Just in case, I have tried similar setup before. I have posted long reply before, but seems it still waits for premoderation. In my case I tried 0xA0000000 as a translation base, because 0x90000000 was already used in my system. Be sure, that did work. So I don't see any problem with your setup.

0 jin fu over 10 years ago in reply to Victor Kazmirenko

Prodigy 110 points

Thx again,
I followed your suggestion ,use trn_rsof as a trigger,but It's strange I still saw nothing when I tried to read and write my config data to remote
config space.From the output I think I successfully generated the config transactions.I don't know why I can't capture the signal.
I rechecked DSP's ob config ,I think I did it right. ob low address is 0x90000000 ,hi add is 0,and obXltEN=1 .
regards
Jin

0 Victor Kazmirenko over 10 years ago in reply to jin fu

Guru 13202 points

No-no, when you make reads and writes to config space, you'll see nothing on trn_* interface. In PCIe endpoint accesses to config space would be handled by integrated block itself, and you'll see only values of config registers. In my device there is cfg_rd_wr_done signal, but it is triggered only on some accesses, not on all.
To see something on trn_* you have to make read or write to data space. I have checked the data sheet for your DSP, it also has PCIe data space at 0x60000000. So I suggest to turn off OB translation, set BAR0 to 0, and make a write to 0x6000_0000. Then youl have to see some activity on trn_*.

0 jin fu over 10 years ago in reply to Victor Kazmirenko

Prodigy 110 points

hello rrlagic,
you mean disable ob traslation and set FPGA's BAR0 to 0?
But still can't capture the trigger signal.Does Fpga cause the problem?
In my system there's a PCIE switch I don't know is there something wrong with my upstream or downstream ports' config.
I printed the config data:
[C66xx_0] ******************upstream remote config:0x21802000****************************
[C66xx_0] offset: 0x0 : 874810b5 00100046 060400ba 00010000
[C66xx_0] offset: 0x10 : 00000000 00000000 00050000 000001f1
[C66xx_0] offset: 0x20 : 0000fff0 0001fff1 00000000 00000000
[C66xx_0] offset: 0x30 : 00000000 00000040 00000000 00000100
[C66xx_0] offset: 0x40 : c8034801 00000008 01866805 00000000
[C66xx_0] offset: 0x50 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x60 : 00000000 00000000 0052a410 00008002
[C66xx_0] offset: 0x70 : 000b0810 02415c43 00220000 00000000
[C66xx_0] ******************downstream remote config:0x21802000****************************
[C66xx_0] offset: 0x0 : 874810b5 00100046 060400ba 00010000
[C66xx_0] offset: 0x10 : 00000000 00000000 00050000 000001f1
[C66xx_0] offset: 0x20 : 0000fff0 0001fff1 00000000 00000000
[C66xx_0] offset: 0x30 : 00000000 00000040 00000000 00000100
[C66xx_0] offset: 0x40 : c8034801 00000008 01866805 00000000
[C66xx_0] offset: 0x50 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x60 : 00000000 00000000 0052a410 00008002
[C66xx_0] offset: 0x70 : 000b0810 02415c43 00220000 00000000
regards,
Jin

0 Victor Kazmirenko over 10 years ago in reply to jin fu

Guru 13202 points

Jin, Just minor correction. When OB translation is off, BARs have to be ar 0x60000000. If you keep OB translation on and set lower addres to zero, then yor remote BARs will match at 0x00000000. This is preferred, because FPGA will start with all BARs zeroed. Having a switch between dsp and fpga makes things more complex. I have no ready recipe.

0 Raja over 10 years ago in reply to Victor Kazmirenko

TI__Guru* 81335 points

Hi rrlagic,
Thank you for your dedicated support on this.

0 Victor Kazmirenko over 10 years ago in reply to jin fu

Guru 13202 points

Jin,

Don't you mind to inform about your progress or findings?

I had no experience with switches, and my superficial search suggests, that some work should be done before switch becomes operational. I saw some references that multiple downstream ports are recognized at root complex by their function numbers. Perhaps that might give you some idea.

And just in case, don't you see that above printing of upstream/downstream configs are just the same? I'm afraid you print not what you want.

0 jin fu over 10 years ago in reply to Victor Kazmirenko

Prodigy 110 points

rrlagic,Thank you for your advice!

I tried to disable downstream ports unused ,and finally it worked well both read and write. I guess I made some mistakes on my switch configures.

I will recheck that.

regards,

Jin

Processors

Processors forum

C6657 PCIE Problem