RTOS/AM5728: PCIe EP communication with PC

yu chen

Expert 1475 points

Part Number: AM5728

Tool/software: TI-RTOS

Hi,all

HW: I am using a am5728 board which is based on am5728-evm board.

SW: PCIE_idkAM572x_wSoCFile_armExampleProject

We notice that the am5728-evm board is designed as RC,but we want to change it as EP,So we modify the hardware design and let PC provides the PCIE reference clock (spread spectrum) to EP.

e2e.ti.com/.../2236952

we overwrite the code at the PlatformPCIESS1PllConfig()

    HW_WR_FIELD32(SOC_SEC_EFUSE_REGISTERS_BASE + CSL_CONTROL_CORE_SEC_SMA_SW_6,
                  CSL_CONTROL_CORE_SEC_SMA_SW_6_PCIE_TX_RX_CONTROL, 0x02U);
    HW_SET_FIELD(regVal, CM_CLKMODE_APLL_PCIE_REFSEL,
                 CM_CLKMODE_APLL_PCIE_REFSEL_CLKREF_ACSPCIE);
Then we test two am5728-evm board communicate with pcie,the RC do not detect the EP at this time.(It is success at CM_CLKMODE_APLL_PCIE_REFSEL_CLKREF_ADPLL),So I think the APLL ref clock is change successfully.
But when we put the am5728-evm(EP) to PC PCIe slot,and run the program.the program is always at "Starting link training..".
I have doubts about the 100Mhz hardware design,but at sometime,the link is up.....but PC do not detect the EP board.

I have some questions want to confirm:

(1) from what I have done now,Can I sure the 100Mhz to ljcb_clkp is right now?(Because sometime the link is up)

(2)The PERST signal,we are ignore this signal at this solution,does it matter?

(3)When I put the board at PC PCIe slot and run Linux at the EP mode,the LTSSM_EN bit is always clear to 0,even thougn I use devmem2 to set it,which pin will affect this bit always 0?

(4)AM5728 have pcie boot code just like DM8168 or DM8148?

(5)our PC do not support the hot-plug,So I can't sure if the EP power up before RC

Any help will be appreaciate!

Thanks!

over 8 years ago

0 Biser Gatchev-XID over 8 years ago

TI__Guru**** 393215 points

The RTOS team have been notified. They will respond here.

0 lding over 8 years ago in reply to Biser Gatchev-XID

TI__Guru* 95265 points

Hi,

If sometimes you saw the linkup from the AM572x EP side, can you check the register 0x51002104 bit 2:7 for LTSSM state, is it 0x11 for L0 status and keep there? If not, then it came to L0 state then dropped so your host PC can't detect it.

PERST# Signal is de-asserted to indicate when the system power sources are within their specified voltage tolerance and are stable. PERST# should be used to initialize the card functions once power sources stabilize. PERST# is asserted when power is switched off and also can be used by the system to force a hardware reset on the card.

LTSSM_EN when set to 1, you should see it is "1" and the LTSSM status at least toggle bewteen 0x0 and 0x1. It is not related to any pin.

We don't have PCIE boot example code for AM57x as the EVM was designed as PCIE RC, we can't test it in EP mode without modifying the board.

Regards, Eric

0 yu chen over 8 years ago in reply to lding

Expert 1475 points

Hi,Eric

Thanks for you reply.

I can detect the am5728-evm on PC now.All the problem is about the pc.

the am5728-evm as EP,we can detect it and use windriver to create a windows driver.but windows device allocates insufficient resources.

I think it is the am5728-evm EP BAR is too big,I want to know the default BAR size.(I am running Linux at am5728-evm now,so I use

)

I want to know how can I set the BAR size by myself.I add __raw_writel(0x000FFFFF, 0x51001010); at dra7xx_pcie_probe();

The system is hung at start_kernel...

BTY, how can I map the DDR address to BAR address? just like DM8168,I will set the IB_START1_LO and IB_OFFSET1 register. Then I can check the BAR1 address to check the DDR address.which register will i overwrite to get it?

Thanks.

CY.

0 lding over 8 years ago in reply to yu chen

TI__Guru* 95265 points

Hi,

Good to know you have the PC detection of your AM572x EP card. Can you elaborate more so it will be helpful for other users who have similar problem? Thanks!

If you use Linux on EP side, and want to know how to setup BAR size and DDR mapping, I will ask my colleague in Linux side. For the PCIE IP, it is different from DM8168.

Regards, Eric

0 yu chen over 8 years ago in reply to lding

Expert 1475 points

Hi,Eric
Yes,I will elaborate the whole thing when i finish this issue.Because i think that will be more clear.
I am looking forward for the answer,Thanks.

0 yu chen over 8 years ago in reply to lding

Expert 1475 points

Hi,Eric

I think I found how to set the bar size.at drivers/pci/endpoint/functions/pci-epf-test.c,static int bar_size[] = { 512, 512, 1024, 16384, 131072, 1048576 };

it is so small,and I use two am5728 to check the bar configure.

But I use windriver to create a driver,but can install it normally. The device can not find enough resources to use. (Code 12)

The bar size is no too large,dm8148 and dm8168 can set the BAR as 16M.and the IRQ is not conflict.

So,the am5728-evm EP configuration is not right yet.Can you help me to confirm it?about the EP configuration at am5728-evm.

0 lding over 8 years ago in reply to yu chen

TI__Guru* 95265 points

Hi,

If you use TI Processor SDK RTOS PCIE sample code to setup the BAR mask, you can refer to pcie_sample.c:

/* Configure BAR Masks */
/* First need to enable writing on BAR mask registers */
if ((retVal = pcieCfgDbi (handle, 1)) != pcie_RET_OK)
{
return retVal;
}

/* Configure Masks*/
memset (&setRegs, 0, sizeof(setRegs));
memset (&getRegs, 0, sizeof(getRegs));

type1Bar32bitIdx.reg.reg32 = PCIE_BAR_MASK;
setRegs.type1BarMask32bitIdx = &type1Bar32bitIdx;

/* BAR 0 */
type1Bar32bitIdx.idx = 0; /* configure BAR 0*/
if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
{
PCIE_logPrintf ("SET BAR MASK register failed!\n");
return retVal;
}

/* BAR 1 */
type1Bar32bitIdx.idx = 1; /* configure BAR 1*/
if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
{
PCIE_logPrintf ("SET BAR MASK register failed!\n");
return retVal;
}

/* Disable writing on BAR Masks */
if ((retVal = pcieCfgDbi (handle, 0)) != pcie_RET_OK)
{
return retVal;
}

You need to change the value of PCIE_BAR_MASK (it is #define PCIE_BAR_MASK 0x0FFFFFFF) for different BARs on your EP side. Then the Linux host or Windriver PCIE RC can allocate the resource when they read the BAR mask of EP side.

Regards, Eric

0 user3910344 over 8 years ago in reply to lding

Prodigy 130 points

Hi,Eric
I use the am5728-evm linux as PCIE ep mode. I use the kernel verison 4.4.32 .I change the configuration in /drivers/pci/endpoint/function/pci-epf-test.c file as I just use BAR0 and BAR1 ,but I can not read data from the dma space I malloc in pci-epf-test.c file. I send data from the PC to BAR0 , but can not read it from the index in follow program:

static int pci_epf_test_alloc_space(struct pci_epf *)
{
struct pci_epf_test *epf_test = epf_get_drvdata(epf);
struct device *dev = &epf->dev;
void *bade;
int bar;

for(bar = BAR_0; bar <= BAR_1; bar++)
{
base = pci_epf_alloc_space(epf, bar_size[bar], bar);
if(!base)
{
dev_err(dev, "file to allocate space for BAR %d\n", bar);
}
epf_test->reg[bar] = base;
}
return 0;
}

I can not read data from BAR0 index "base" address with program follow:

printk("BAR 0 addr 0 data : 0x%x \n", *(unsigned int *)base);

Can you give me some advices about how can I read data from BAR0 in application program?

0 yu chen over 8 years ago in reply to user3910344

Expert 1475 points

Hi，

sounds like you finish the PC driver.

can you tell me how to do？

what is your windows/linux version？

0 user3910344 over 8 years ago in reply to yu chen

Prodigy 130 points

I make two changes as follows:
(1)The am5728-evm should powerup before the PC powerup.
(2)I change the pci-epf-test.c configuration:
a: I just use BAR0 and BAR1 , this should be the same as the PC driver.
b: I change the size of BAR0 and BAR1 , this should be the same as the PC driver too.
(3) I install a pciscope software to see the PC pcie information ,after I install it , the driver on PC install normal.

0 user3910344 over 8 years ago in reply to user3910344

Prodigy 130 points

But I do not know how to communicate with the PC in my applacation program when access the DMA space .

0 yu chen over 8 years ago in reply to user3910344

Expert 1475 points

HI，
thanks for your reply！
yes，i use windriver and pciscope！
what do you mean the bar size as same as pc driver？on my opinion,the pc driver will allocate resorces to the device
what is the size do you set？ bar0 and bar1.

on my test，when i use windriver to test BAR0，i can see the data at 0x20000000

0 user3910344 over 8 years ago in reply to yu chen

Prodigy 130 points

My PC driver is used on other project. I use the PC driver without any chang. So I have to make my am5728 driver match my PC driver. How do you access the 0x20000000 address? I can not access the address with "mmap" the "/dev/mem" ? Can you give me some advice or some example code? My email address is w13630570960@126.com, we can also talk about it in email.

0 yu chen over 8 years ago in reply to user3910344

Expert 1475 points

Hi,
I get the data from pcie.
at pci_epf_test_set_bar() function. epf_bar->phys_addr this address will used to set the inbound function.
I can use devmem2 tools to check the data.
You can try this address.
Thanks for you set TWO bars advices.

0 yu chen over 8 years ago in reply to user3910344

Expert 1475 points

HI,
0x5100 0918,you should check this register.

0 user3910344 over 8 years ago in reply to yu chen

Prodigy 130 points

Hi CY;

Sorry to reply so late.

Thank you for your advice about the devmem2 tools.

But I can not read data from the address epf_bar->phys_addr at pci_epf_test_set_bar() function, and I can not read data from the 0x51000918 address too.

Thank you all the same. I will try in other way.

0 yu chen over 8 years ago in reply to user3910344

Expert 1475 points

Hi,

from your picture log.I do not think your PC driver is right at here.

If the PC driver is right,and the 0x51001010 is not 0x0,it will be the PC PCI address.

Secondly,I can read the data at the epf_bar->phys_addr,is allocated by pci_epf_alloc_space() function.or you can specify a fixed address to the epf_bar->phys_addr variable in kernel.

Also,you can use devmem2 tools or windriver to set the address map.It will let the BAR1 to 0x80000000 DDR address.but you must set the BAR0 map to 0x51000000 first.

0 user3910344 over 8 years ago in reply to yu chen

Prodigy 130 points

Hi,CY;

Thank you for your advices.

I will try later. It is helpful for me.

Thank you again!

Best regard!

0 user3910344 over 8 years ago in reply to user3910344

Prodigy 130 points

Hi,CY;
Thank you for your " windriver at PC send data " advice.
I have get data from the epf_bar->phys_addr addr when PC send data to PCIE EP.
But I do not know the PC how to tell the EP when it send data to EP. Commonly ，PCIE EP works as slave mode, after it successfully gets data , it will raise a interrupt to PC. But how can ep know it has data to read from the PC.

Processors

Processors forum

RTOS/AM5728: PCIe EP communication with PC