AM572x PCIe example bug in PDK v1.0.4

Tim Dylla

Other Parts Discussed in Thread: AM5728

Hi,

I try to get the PCIe example code from the pdk_am57xx_1_0_4 to run on DSP1 of an am5728 on a beagleboard X15.

Beside the hint from here to use EDMA instance #1 instead of #0 (PCIeEDMAselector.c:69), I have problems to configure the BAR registers.

In pcie_example.c:1666ff. the funtion Pcie_cfgBar() is called for both use cases RC and EP. In both cases it returns with pcie_RET_RO_CHANGED.

I dug a bit deeper...

To configure a BAR register a configuration struct of type pcieBarReg_t is filled with the raw actual register value and several other members each representing a bitfield of the target register. The function pciev1_write_type1Bar_reg (pciev1_rc.c:130) that actually writes to the register takes the raw value from pcieBarReg_t, masks out the writable bitfields and then ORs the content of the bitfield members of pcieBarReg_t to the appropriate positions. To ensure to never change read-only bits it finally compares the read-only part of the result with the read-only part of the register content just before writing it to the register. Unfortunately in the example code the raw value member of the struct is never set to the actual register value and since there are some readonly bits set the final comparison always fails and the function returns with pcie_RET_RO_CHANGED.

Two solutions come to my mind:

initializing the raw value member of the struct by reading the register prior to writing it. The function Pciev1_writeRegs wich is used here has a complement Pciev1_readRegs that does exactly what is necessary - but this seems a bit of an overhead (not that it seems anybody would care...)
never even using the raw value member since the register is already read once in the process and can be used not only for validating the result but also for computing the RO-part of it.

I have found nothing about this problem in the forum - am I the only one trying this, or have I done something completely unusual and the example is failing only for me?

Best,

Tim

over 7 years ago

0 Biser Gatchev-XID over 7 years ago

TI__Guru**** 393215 points

The RTOS team have been notified. They will respond here.

0 Tim Dylla over 7 years ago in reply to Biser Gatchev-XID

Expert 1070 points

Ok, I see it's even a bit weirder than I thought.

Even initializing the raw member of pcieBarReg_t doesn't help: though it is initially used to compute the result register value it gets completely overwritten in the process of adding the bitfield members.

The Macro pcie_setbits does a masking (~&) and an OR. Using the Mask PCIE_RC_BAR_BASE_FULL here to add the base bitfield leads to zeroing the RO-bits even with an initialized raw member. This won't fail if all registers RO bits are starting as zeros, assuming the base member is correct. But if the register has any RO bit set the function will fail.

0 lding over 7 years ago in reply to Tim Dylla

TI__Guru* 95265 points

Tim,

Thanks for your debug of this issue! We didn't see this before and I re-tested the pdk_am57xx_1_0_4 PCIE project on our EVM to make sure the function Pcie_cfgBar() worked properly this morning.

As you traced, this finally calls into pciev1_write_type1Bar_reg() when I use RC mode as an example:

In below assmbly code:

8003e094:   E2011003            and        r1, r1, #3
8003e098:   E18C1081            orr        r1, r12, r1, lsl #1
146         pcie_setbits(new_val, CSL_RCCFGDBICS_BAR_PREFETCHABLE,    swReg->prefetch);
8003e09c:   E2033001            and        r3, r3, #1
8003e0a0:   E3C11008            bic        r1, r1, #8
150         pcie_getbits(origReg, CSL_RCCFGDBICS_BAR_BASE_ADDR_RO,    origRoBase);
8003e0a4:   E7EF0250            ubfx       r0, r0, #4, #0x10
146         pcie_setbits(new_val, CSL_RCCFGDBICS_BAR_PREFETCHABLE,    swReg->prefetch);
8003e0a8:   E1813183            orr        r3, r1, r3, lsl #3
147         pcie_setbits(new_val, PCIE_RC_BAR_BASE_FULL,              swReg->base);
8003e0ac:   E203300F            and        r3, r3, #0xf
8003e0b0:   E183320E            orr        r3, r3, lr, lsl #4
152         pcie_getbits(new_val, CSL_RCCFGDBICS_BAR_BASE_ADDR_RO,    newRoBase);
8003e0b4:   E7EF1253            ubfx       r1, r3, #4, #0x10
154         if (origRoBase != newRoBase)
8003e0b8:   E1500001            cmp        r0, r1

In my debug, both r0 and r1 is zero so the comparison doesn't fail. The bit field/mask is:

#define CSL_RCCFGDBICS_BAR_BASE_ADDR_RO_MASK (0x000FFFF0U)
#define CSL_RCCFGDBICS_BAR_BASE_ADDR_RO_SHIFT (4U)

If you open a CCS memory window and look at 0x51000014 (this is BAR1 register they tried to wrtite to), before write it is 0x8 and after write is 0x9000_0008. So the readonly bit field are zero. In your failure case what is that register before and after write?

Regards, Eric

0 Tim Dylla over 7 years ago in reply to lding

Expert 1070 points

Eric,

thanks for having a look in it!

As I stated in my appended reply, the function will fail if one or more of the registers RO bits are set at call time. This is the case in my setup.

The BAR registers initial state is 0x20020000 - bit17 is set und will be masked out by codeline pciev1_rc.c:147

pcie_setbits(new_val, PCIE_RC_BAR_BASE_FULL, swReg->base);

pcie_setbits will mask out the RW AND the RO bits of the base adress bitfield and ORs the content of swReg->base where RO bits MUST NOT be set.

The result is a base adress bitfield with all RO bits set to zero.

When the RO bits of the result is compared to the RO bits of the actual register content the difference will make the function fail and the register won't be written:

/* Extract origin read only bits */
pcie_getbits(origReg, CSL_RCCFGDBICS_BAR_BASE_ADDR_RO, origRoBase);
/* Extract new read only bits */
pcie_getbits(new_val, CSL_RCCFGDBICS_BAR_BASE_ADDR_RO, newRoBase);
/* Puke if user tried to change RO bits */
if (origRoBase != newRoBase)
{
ret_val = pcie_RET_RO_CHANGED;
}

Please confirm that

pciev1_rc.c:147 masks out all RO bits
pciev1_rc.c:152 will therefore always return 0
pciev1_rc.c:137 the initialization of newVal = swReg->raw is useless since all bits in newVal will be overwritten.
even if the RO bits of newVal wouldn't be overwritten the swREG->raw value was never set to the raw register value and the function would still fail.

IMHO the bug is to use the PCIE_RC_BAR_BASE_FULL mask instead of using CSL_RCCFGDBICS_BAR_BASE_ADDR_RW for setting the base address bitfield in pciev1_rc.c:147 and the missing initialisation of swREG->raw with the original register content.

Thanks in advance,

Tim

0 lding over 7 years ago in reply to Tim Dylla

TI__Guru* 95265 points

Tim,

I agreed to your analysis except that I don't understand why "the BAR registers initial state is 0x20020000", the bit 17 falls into read-only field and by reset the default value is 0.

I tried to modify that 0x51000014 from the default (when PCIE PRCM enabled) 0x8 to 0x90020008 in CCS memory window (I tried to set some read-only bit), then I ran the pciev1_write_type1Bar_reg function, the code aborted as R0=0x00002000 and R1=0.

We didn't see this issue in the past because we don't have those RO bit set, can you explain in which scenarios you came into that, if you know? Anyway, our driver code can't handle this, I will discuss with our driver developer.

Regards, Eric

0 Tim Dylla over 7 years ago in reply to lding

Expert 1070 points

Eric,

we have an am5728 beagleboard X15. There is nothing attached to the pcie port. We use the minimal arago linux rootfs and the linux rt kernel from processor sdk 3.1.0.6 along with the am57xx-beagle-x15.dtb. The linux system goes up and I honestly don't care what the linux pcie subsystem possibly does in terms of configuration work. I don't intend to use it. The DSP firmware is supposed to be loaded via remoteproc to take control over the pcie subsystem.

I could strip the pcie part from the kernel and the dtb to find out if it changes anything, if that is of any help. But to be honest the initialization code should be fixed, it is a terrible mess right now. :) - no bashing intended.

That brings me to another question:
Is there some PCIe documentation similar to sprugs6d for Keystone except for Sitara devices? It's really kind of a lottery without further detailed information, especially when it comes to implementing a PHY loopback test which isn't covered by the pcie example from the pdk and is not described in the TRM.

Best,
Tim

0 Chris Pendleton over 7 years ago in reply to Tim Dylla

Intellectual 410 points

Hi Tim.

I had some recent experience that may be relevant. I started with the X15, but moved to the IDK for several reasons. One of which is that it has a nice PCIe x1 connector and there was a TI-RTOS PCIe example targeted to the board. In any case, the software environments are very similar.

I am using arago linux with the linux-rt kernel from processor-sdk 3.00.00.04. I am using remoteproc to bring up the DSP/TI-RTOS which controls the PCIe subsystem. It seems to be working well now, as I am reliably communicating with an external Xilinx board. I did remove all PCIe references from the dtb before doing this to avoid any clashing with the TI-RTOS PCIe example initialization.

Hope this encourages you. I did not find it to be that painful.

Regards,

Chris

0 lding over 7 years ago in reply to Tim Dylla

TI__Guru* 95265 points

Tim,

It looks like the Linux enables PCIE and writes to BAR1 registers RO field. RTOS successively runs on DSP to do PCIE intialization again and failed in BAR cfg. I will discuss with our driver developer for the RTOS code fix. It takes time and it would be good for you to remove the PCIE dts in the Linux as a workaround.

For the PCIE document on Sitara, it is a unified AM 572xTRM in Chapter 24.9 PCIe Controller. Unfortunatedly there is little talk about PCIE loopback and no PDK SW implemented for that.

Good news is that we tested the AM572x GP EVM with several generic USB or NIC PCIE cards (AM572x is RC, running Processor SDK RTOS software, GP EVM has a LCD screen with PCIE x1 connector on the back, connection is GEN1/2 x 1 lane), the enumeration always worked.

Regards, Eric

0 Tim Dylla over 7 years ago in reply to lding

Expert 1070 points

Eric,

thank you for your efforts. I've been away from the board for some days but now I'm on it again.

I've stripped the pcie part off the dtb as you proposed. Including both axi subtrees and all related clock definitions.

Then I put on the touch-tft expansion on the beagle board and grinded a JTAG-20pin adaptor to fit between beagleboard and expansion, enjoying being away from the office for some manual work. ;)

Then I attached a jmb360 based esata pcie card to the minipcie slot via a pcie 1lane slot adaptor.

Since I'm dealing with a regularily attached pcie card now I resurrected the GPIO code which is part of the example for the IDK_AM572x and mapped it to the GPIO that is IMHO used for pcie reset on the beagle board: GPIO2_8.

Unfortunately that seems to interfere with the linux that runs on the A15. Every time I try to write to any GPIO ctrl register I get the following kernel messages:

[14057.159376] WARNING: CPU: 1 PID: 28 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x25c/0x368()
[14057.168813] 44000000.ocp:L3 Custom Error: MASTER DSP1_MDMA TARGET L4_PER1_P3 (Read): Data Access in User mode during Functional access
[14057.180949] Modules linked in: rpmsg_proto virtio_rpmsg_bus xhci_plat_hcd xhci_hcd usbcore dwc3 udc_core pru_rproc pruss_intc ecb snd_soc_simple_card sha512_generic sha512_arm sha256_generic snd_soc_omap_hdmi_audio hmac md5 sha1_generic sha1_arm_neon sha1_arm omap_aes_driver pruss omap_sham omap_wdt ahci_platform libahci_platform libahci libata scsi_mod ti_vpe ti_sc ti_vpdma dwc3_omap rtc_omap extcon_palmas extcon rtc_palmas rtc_ds1307 snd_soc_tlv320aic3x omap_des omap_rng rng_core omap_remoteproc remoteproc virtio virtio_ring sch_fq_codel cryptodev(O) cmemk(O)
[14057.230931] CPU: 1 PID: 28 Comm: irq/23-l3-app-i Tainted: G W O 4.4.19-rt25-gf572d285f0 #1
[14057.230934] Hardware name: Generic DRA74X (Flattened Device Tree)
[14057.230938] Backtrace:
[14057.230955] [<c0012ec4>] (dump_backtrace) from [<c00130c0>] (show_stack+0x18/0x1c)
[14057.230964] r7:c02e35a4 r6:20000013 r5:00000000 r4:c09339c4
[14057.230976] [<c00130a8>] (show_stack) from [<c02b7464>] (dump_stack+0x8c/0xa0)
[14057.230988] [<c02b73d8>] (dump_stack) from [<c0034064>] (warn_slowpath_common+0x88/0xb8)
[14057.230995] r7:c02e35a4 r6:00000093 r5:00000009 r4:ee651e28
[14057.231004] [<c0033fdc>] (warn_slowpath_common) from [<c00340cc>] (warn_slowpath_fmt+0x38/0x40)
[14057.231013] r8:00000017 r7:c081d3f0 r6:00000002 r5:c081d028 r4:c081d0cc
[14057.231022] [<c0034098>] (warn_slowpath_fmt) from [<c02e35a4>] (l3_interrupt_handler+0x25c/0x368)
[14057.231026] r3:ee602e80 r2:c081d0cc
[14057.231029] r4:80080003
[14057.231039] [<c02e3348>] (l3_interrupt_handler) from [<c008183c>] (irq_forced_thread_fn+0x28/0x7c)
[14057.231048] r10:c0081814 r9:ee608400 r8:ee5f9900 r7:00000001 r6:00000000 r5:ee5f9900
[14057.231051] r4:ee608400
[14057.231058] [<c0081814>] (irq_forced_thread_fn) from [<c0081ba4>] (irq_thread+0x124/0x1f8)
[14057.231065] r7:00000001 r6:00000000 r5:ee650000 r4:ee608424
[14057.231072] [<c0081a80>] (irq_thread) from [<c004ec24>] (kthread+0xe4/0xfc)
[14057.231081] r10:00000000 r9:00000000 r8:00000000 r7:c0081a80 r6:ee608400 r5:ee608440
[14057.231083] r4:00000000
[14057.231090] [<c004eb40>] (kthread) from [<c000fa90>] (ret_from_fork+0x14/0x24)
[14057.231096] r7:00000000 r6:00000000 r5:c004eb40 r4:ee608440

Anyway, regardless if I try to use the GPIO or not the pcie card seem to answer in a basic fashion. It seems the GPIO is not necessary but I'd like to understand how and why it interferes with the kernel. Sooner or later on our own hardware I'll need it.

Staying on topic, the link training detects something on the lane but stalls in LTSSM_POLL_COMPLIANCE state.

Following the complete link training state sequence I have been able to read out:

**********************************************
* PCIe Test Start *
* RC mode *
**********************************************

Version #: 0x02020006; string PCIE LLD Revision: 02.02.00.06:Oct 4 2016:15:15:27

PCIe Power Up.
PLL configured.
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Starting link training...
LTSSM state: LTSSM_DETECT_QUIET (0)
LTSSM state: LTSSM_DETECT_ACT (1)
LTSSM state: LTSSM_DETECT_WAIT (6)
LTSSM state: LTSSM_DETECT_ACT (1)
LTSSM state: LTSSM_POLL_ACTIVE (2)
LTSSM state: LTSSM_POLL_COMPLIANCE (3)
LTSSM state: LTSSM_POLL_COMPLIANCE (3)
LTSSM state: LTSSM_POLL_COMPLIANCE (3)

...

I've compiled the example code with following preprocessor defines:

am5728, EVM_AM572x, SOC_AM572x, _TMS320C6X

The configuration is 2.5Gbps (#undef GEN2) on one lane (#undef PCIESS1_X2) in root complex mode and I changed the xdc.loadPackage('ti.board').Settings.boardName in the app.cfg to "evmAM572x".

Since the enter compliance bit isn't set in the pcie port link ctrl register, I assume it enters LTSSM_POLL_COMPLIANCE because one of the participants hasn't received anything meaningful and drops by default to test mode.

So, still out of luck. Any suggestions?

Thanks again,

Tim

0 Chris Pendleton over 7 years ago in reply to Tim Dylla

Intellectual 410 points

Hi Tim.

I ran into some of these problems over the past 6 months when working with the X15 / IDK + Linux + DSP + PCIe. Regarding the DSP GPIO module access causing the Linux exception, you can look at 2 things:

1) Did you make custom resource table header for the DSP as described here: http://processors.wiki.ti.com/index.php/IPC_Resource_customTable ? The contents of this header will instruct Linux / IPC to create DSP MMU entries that will allow the DSP to access various system resources, including the GPIO modules. I found I needed to create the custom header to access most things. I forget whether I needed to add an entry for the GPIO but I definitely needed to add one to access the PCIe space. It caused a linux mmu fault exception without it. I can send you my custom rsc_table_vayu_dsp.h if it would help.

2) I also found that accessing "random" GPIO with the DSP would cause Linux exceptions. I was able to work around this by adding an entry to am57xx-beagle-x15-common.dtsi in the "leds {" section. Add a another led entry like:

led@4 {
label = "beagle-x15:usr4";
gpios = <&gpio8 15 GPIO_ACTIVE_HIGH>;
linux,default-trigger = "chris-test";
default-state = "off";
};

This seemed to allow the DSP to manipulate this GPIO without Linux complaining. It is a bit of a hack and I am sure there is better way to do it, but it might get you further along.

-Chris

0 Tim Dylla over 7 years ago in reply to Chris Pendleton

Expert 1070 points

Hi Chris,

thanks for offering help, really appreciated - I'm running out of ideas right now.
Regarding the custom resource table: I'm already working with one and the complaining mmu was my first thought too. But that's not the issue here, my resource table already contains all necessary gpio mmrs.
The dtsi hack is very intresting - I'd really like to understand the background. I'll give it a try.

thanks again,

Tim

0 Chris Pendleton over 7 years ago in reply to Tim Dylla

Intellectual 410 points

Sure thing. I might be transitioning back to the X15 in my project, which means getting PCIe working through the DSP with Linux on the A15s. I wasn't successful in my initial attempt, but I didn't spend much time on it before moving to the IDK board.

0 lding over 7 years ago in reply to Tim Dylla

TI__Guru* 95265 points

Tim,

For testing PCIE on the EVM, can you try with the below .out file on C66x core, I just tested on my AM572x GP EVM (same as your X15+LCD) with a small Intel WiFI card (112BNHMW) inserted to mini-PCIE slot in the back of LCD. Just load and run, I got below:

[C66xx_DSP1] **********************************************

* PCIe Test Start *

* RC mode *

**********************************************

Version #: 0x02020002; string PCIE LLD Revision: 02.02.00.02:Oct 7 2015:14:42:32

PlatformPCIESS1 ClockEnable

PlatformPCIESS2 ClockEnable

L3INIT CLKSTCTRL is :: [ 518702]

PCIeSS CLKSTCTRL is :: [ 3f02]

PCIeSS1 CLKCTRL is :: [ 40702]

PCIeSS2 CLKCTRL is :: [ 40702]

PlatformPCIESS/21PllConfig

PlatformPCIESS1CtrlConfig

PlatformPCIESS2CtrlConfig

PlatformPCIESS1Reset

PlatformPCIESS2Reset

PlatformPCIESS1PhyConfig

PlatformPCIESS2PhyConfig

PCIe Power Up.

PLL configured.

Successfully configured Inbound Translation!

Successfully configured Outbound Translation!

Starting link training...

Link is up.

RC side 0x51002104 = 0x45 and 0x51000080 = 0x30110008

EP sending interrupts to RC

So the PCIE link is coming up. Note I don't have Linux running on A15 (I am from RTOS application team, don't use Linux), can you try this out file and a few different PCIE EP cards to see if it works for you?

6014.PCIE_evmAM572x_wSoCFile_C66BiosExampleProject.out

Regards, Eric

0 Tim Dylla over 7 years ago in reply to lding

Expert 1070 points

Eric,

your executable does the trick:

[C66xx_DSP1] **********************************************
* PCIe Test Start *
* RC mode *
**********************************************

Version #: 0x02020002; string PCIE LLD Revision: 02.02.00.02:Oct 7 2015:14:42:32

PlatformPCIESS1 ClockEnable
PlatformPCIESS2 ClockEnable
L3INIT CLKSTCTRL is :: [ 1700f02]
PCIeSS CLKSTCTRL is :: [ 3f02]
PCIeSS1 CLKCTRL is :: [ 40702]
PCIeSS2 CLKCTRL is :: [ 40702]
PlatformPCIESS/21PllConfig
PlatformPCIESS1CtrlConfig
PlatformPCIESS2CtrlConfig
PlatformPCIESS1Reset
PlatformPCIESS2Reset
PlatformPCIESS1PhyConfig
PlatformPCIESS2PhyConfig
PCIe Power Up.
PLL configured.
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Starting link training...
Link is up.
RC side 0x51002104 = 0x45 and 0x51000080 = 0x30110008
EP sending interrupts to RC

Great, that shows there are at least no electrical or linux related problems.

But now I wonder what is wrong with the example code here... I didn't change anything of the configuration details. With what set of predefined Symbols did you compile?

I could dig into the configuration register differences. But - questions are free - would you be so kind to provide the source code of your executable?

Best,

Tim

0 Chris Pendleton over 7 years ago in reply to lding

Intellectual 410 points

Eric - I second that request on the source code. Could you zip up your project and make it available please?

Thanks, Chris

0 lding over 7 years ago in reply to Chris Pendleton

TI__Guru* 95265 points

Chris and Tim,

Great to know the DSP out file worked while Linux running on A15! It was built last year with very old P-SDK release, I can't find the project. Let me try a fresh new from latest P-SDK 3.1 (pdk_am57xx_1_0_4), tested on my setup then send to you today or tomorrow.

Regards, Eric

0 Tim Dylla over 7 years ago in reply to Chris Pendleton

Expert 1070 points

Ok,

I got it. One has to configure the second lane regardless if it will be used or not. That's the one-sentence explanation.

In more detail:

There is a Symbol PCIESS1_X2 that can be defined if one wants to use the first pcie controller with 2 lanes. If it is not set following initialization functions remain empty:

PlatformPCIESS2ClockEnable

PlatformPCIESS2Reset

PlatformPCIESS2CtrlConfig

PlatformPCIESS2PhyConfig

All these functions are called from pcieSerdesCfg but if the symbol isn't defined they return without doing anything.

So in conclusion the Symbol does things partly wrong, one or the other way:

If I don't define PCIESS1_X2 the link training fails.
If I define PCIESS1_X2, the link training succeeds but the example fails because it expects 2 lanes but only one is active.
If I change the 4 mentioned functions that they contain the initialization even though the Symbol isn't defined, the link training and the expectation test succeed.

Again, a really nasty one.

When I started working with the example code I expected some kind of reference what to do in which order. To be honest: it leaves me with more questions than I had before.

But in an optimistic view this only means I now know a lot more about the subject than before. :)

Best,

Tim

0 Chris Pendleton over 7 years ago in reply to lding

Intellectual 410 points

Hi Eric.

Thanks very much. I look forward to your project/code.

After upgrading to PROCESSOR-SDK-RTOS-AM57X 03_01_00_06, I ran the pdkProjectCreate on pdk_am57xx_1_0_4. I still only see PCIE_idkAM572x_wSoCFile_C66BiosExampleProject for the IDK. Why doesn't the EVM have an example project? Your binary's name seems to indicate that there is one floating around at TI.

Thanks,

Chris

0 Chris Pendleton over 7 years ago in reply to Tim Dylla

Intellectual 410 points

Hi Tim.

Could you share your *working* project? Perhaps you could zip up the entire project folder and email it or post it to a cloud drive?

Thanks,
Chris

0 lding over 7 years ago in reply to Chris Pendleton

TI__Guru* 95265 points

Chris,

The Processor SDK package only tests the PCIE connection between TI EVMs, so there are projects for IDK EVMs, where they can be connected via PCIE x1 connector. On the other hand, GP EVM has mini-PCIE connector and can't be directly connected between EVMs (at least need some adapator), we don't provide PCIE test project for GP EVM.

To test GP EVM, please put the attached two files under C:\ti\pdk_am57xx_1_0_4\packages\ti\drv\pcie\example\sample\am572x\c66\bios, then run"pdkProjectCreate.bat AM572x evmAM572x little pcie dsp" to create the project, build and run.

For source code, if you want to define output to CCS console, please #define IO_CONSOLE in pcie_sample.h file.

In pcie_sample.c, you need to change pcieMode_e PcieModeGbl = pcie_RC_MODE;

From the thread, looks you run Linux in A15, that means PINMUX already configured, so in the pcie_sample.c, you need to comment out BOARD_INIT_PINMUX_CONFIG flag.

Board_initCfg boardCfg;

boardCfg = BOARD_INIT_UNLOCK_MMR

#ifndef IO_CONSOLE

| BOARD_INIT_UART_STDIO

| BOARD_INIT_MODULE_CLOCK

#endif

#if !defined(DEVICE_K2E) && !defined(SOC_K2E)

/* | BOARD_INIT_PINMUX_CONFIG */

#endif

;

Board_init(boardCfg);

Let me know if this work for you or not. pcie_sample.c

pcie_sample.h

7026.PCIE_evmAM572x_wSoCFile_C66BiosExampleProject.txt

-ccs.linkFile "PDK_INSTALL_PATH/ti/drv/pcie/soc/am572x/src/pcie_soc.c"
-ccs.linkFile "PDK_INSTALL_PATH/ti/drv/pcie/example/sample/src/pcie_sample.c"
-ccs.linkFile "PDK_INSTALL_PATH/ti/drv/pcie/example/sample/am57x/src/pcie_sample_board.c"
-ccs.linkFile "PDK_INSTALL_PATH/ti/drv/pcie/example/sample/am572x/c66/bios/pcie_sample_wSocFile_evm.cfg"
-ccs.linkFile "PDK_INSTALL_PATH/ti/drv/pcie/example/EDMA/commonEDMA.c" 
-ccs.linkFile "PDK_INSTALL_PATH/ti/drv/pcie/example/EDMA/PCIeEDMA.h" 
-ccs.linkFile "PDK_INSTALL_PATH/ti/drv/pcie/example/EDMA/PCIeDMA.c" 
-ccs.linkFile "PDK_INSTALL_PATH/ti/drv/pcie/example/EDMA/PCIeEDMAselector.c" 
-ccs.setCompilerOptions " -mv6600 -g -DSOC_AM572x --diag_warning=225 -I${PDK_INSTALL_PATH}/ti/drv/pcie/example/sample/src -I${PDK_INSTALL_PATH}/ti/drv/pcie/example/sample/am57x/src --entry_parm=address --exit_hook=ti_utils_exit --exit_parm=address --entry_hook=ti_utils_entry " -rtsc.enableRtsc

6378.pcie_sample_wSoCFile_evm.cfg

Regards, Eric

0 lding over 7 years ago in reply to Tim Dylla

TI__Guru* 95265 points

Tim,

Good to know you get that PCIE working on C66x. However, enumeration still works for me without define PCIESS1_X2, the purpose of this flag is to test on IDK571x EVM where is PCIEx2 connector on board. For IDK 572 and GP572 EVMs, there is only x1 lane connector, so in the project we don't set this flag, so the code only expect/check one lane condition. In pcie_sample.h there is a #define GEN2, if you use a GEN1 card for testing, the code will check and abort as only GEN1 is enumerated but GEN2 is expected. Anyway, glad to know that PCIE works on C66x with A15 runs Linux.

Regards, Eric

0 Tim Dylla over 7 years ago in reply to lding

Expert 1070 points

Eric,

so then please explain why is the initialization of SS2 in your executable active even though it is supposed to be a 1-lane example? I have disassembly-debugged your executable and the only difference I found between it and the actual example code is that the functions PlatformPCIESS2ClockEnable, PlatformPCIESS2Reset, PlatformPCIESS2CtrlConfig and PlatformPCIESS2PhyConfig are completely executed.

To be clear: the solution for me is not to define PCIESS1_X2, if I would, the example would fail just later after the link training. The only thing I did to make it run was to comment out the #ifdef PCIESS1_X2 lines in the mentioned initialization functions for SS2.

If you could further delimit what is encessary of the SS2 init to make lane 1 run I'd be happy!

@Chris: I don't have a project that would work out of the box - for example the resource table I use is part of another firmware project which executable I load at boot time from linux. So I don't think it will be of any use for you. But all the information I have and you need to put a project together on your own is in this thread. The only thing's missing is the list of files I took from the pdk source directory:

PDK_INSTALL_PATH/ti/drv/pcie/soc/am572x/src/pcie_soc.c

PDK_INSTALL_PATH/ti/drv/pcie/example/sample/src/pcie_sample.c

PDK_INSTALL_PATH/ti/drv/pcie/example/sample/src/pcie_sample.h

PDK_INSTALL_PATH/ti/drv/pcie/example/sample/am57x/src/pcie_sample_board.c

PDK_INSTALL_PATH/ti/drv/pcie/example/sample/am57x/src/pcie_sample_board.h

PDK_INSTALL_PATH/ti/drv/pcie/example/EDMA/commonEDMA.c

PDK_INSTALL_PATH/ti/drv/pcie/example/EDMA/PCIeEDMA.h

PDK_INSTALL_PATH/ti/drv/pcie/example/EDMA/PCIeDMA.c

PDK_INSTALL_PATH/ti/drv/pcie/example/EDMA/PCIeEDMAselector.c

PDK_INSTALL_PATH/ti/drv/pcie/example/sample/am572x/c66/bios/pcie_sample_wSocFile.cfg"

instead of the last you can possibly use the one from eric.

Make a RTSC project from scratch and add these files. You'll have to change the memory layout depending on your setup since you'll have to share resources with your linux install. According to this you have to build a custom resource table. If you then follow the information Eric and I provided in this thread you should be fine!

All the best,

Tim

0 Tim Dylla over 7 years ago in reply to Tim Dylla

Expert 1070 points

Eric,

My apologies, the PCIESS1_X2 thing was a false trail. Even though your provided executable does a SS2 initialization it is not crucial for a functioning link training on SS1. I shot myself in the knee changing two things at the same time while testing. Not the reenabled ss2 intialization did the trick, but the reset logic seems to work inverted in comparison to the IDK example. Actually my GPIO reset code held the PCIe card in reset... m(

So with the inverted reset logic the gen1 1-lane setup works now. That's great! :)

Thank you again, Eric

Best,

Tim

0 Chris Pendleton over 7 years ago in reply to Tim Dylla

Intellectual 410 points

Hi Tim.

I am trying to get this working on the X15 as we speak.

Regarding the reset, do you mean:

8 GPIO2_8 GPIO2[8] PCIe_RESET PCI-Express Mini Card RESET (active high)

So that needs to be held low to bring the PCIe card out of reset?

Did you control this in Linux/A15 or RTOS/DSP?

Thanks,
Chris

0 Tim Dylla over 7 years ago in reply to Chris Pendleton

Expert 1070 points

Chris,

exactly, GPIO2_8. You have to clear it, means write (0x01 << 8) to 0x48055190 which is the GPIO_CLEARDATAOUT register for GPIO port 2.

At least here in Berlin :)

Hope that helps,

Tim

0 Chris Pendleton over 7 years ago in reply to lding

Intellectual 410 points

Hi Eric and Tim.

I am now communicating with my external Xilinx board over DSP/TI-RTOS/PCIe using the X15/EVM with Linux running on the A15s. Thanks very much for your help.

Eric - I am using an adapter as you described. It is very cheap ($13) and seems to work well: https://www.amazon.com/gp/product/B017R1T2Z8 . I would definitely encourage your team to make the PCIe example available to X15/EVM users with instructions to use one of these adapters. It would have helped me a lot about 6 months ago.

I ended up purchasing 2 x IDKs to run the PCIe example as suggested. That worked well. I wanted to share my experience getting the right custom cable for the IDK test. The team at Adex Electronics (http://www.adexelec.com/) was very helpful. The part number I needed was PE-FLEX1-G2-MMCX-12-TI1 (12 inch Male-Male Crossover with Clock and Power disconnected). This cable works out of the box for the IDK test, so no modifications or cutting traces. I would suggest pointing people to this very specific cable/part number in the example documentation: http://processors.wiki.ti.com/index.php/Processor_SDK_RTOS_PCIe. You could probably eliminate this page: http://processors.wiki.ti.com/index.php/PCIe_CableMod.

Thanks!

-Chris

0 lding over 7 years ago in reply to Chris Pendleton

TI__Guru* 95265 points

Chris,

Thanks for the feedback and great to know the PCIE example works well!

Regards, Eric

0 Christopher Peters over 7 years ago in reply to lding

Genius 3370 points

I seem to be having this exact problem myself and I am not sure I followed what was changed to fix the issue. We are running Linux on the A14 and TIRTOS on the DSP. I am connecting to an FPGA via PCIe. I have had it working for a bit now, but then this problem with pcie_RET_RO_CHANGED has come back. I have rebooted several times and power cycled with no luck. I am using DSP core 1 but PCIe SS2 with one lane to the FPGA. Like I said this was working great and unexplainably started to fail. We have not removed PCIe from the Linux device tree, so maybe that is the "fix". If someone could summarize what fixed the pcie_RET_RO_CHANGED problem I'd really appreciate it.

0 Tim Dylla over 7 years ago in reply to Christopher Peters

Expert 1070 points

Hi Christopher,

it seems the PCIe hw init of the linux driver sets some RO-bits of the BAR config registers. The RTOS-pcie-lld is buggy and acts as if your init code would try to alter the RO portion and complains. You have 3 possible ways to get rid of this:

1. strip the pcie part off the linux device tree (maybe only the ss2 part - I don't know if that alone helps).

2. ignore the set RO bits of the BAR register and OR these into your base address to configure. In a brute force manner it would look like this:

/* Configure Address Translation */

barCfg.location = pcie_LOCATION_LOCAL;
barCfg.mode = pcie_RC_MODE;
barCfg.base = PCIE_IB_LO_ADDR_RC | (CSL_RCCFGDBICS_BAR_BASE_ADDR_RO_MASK & *((unsigned int *) (0x51000010U + 4 * PCIE_BAR_IDX_RC)));
barCfg.prefetch = pcie_BAR_NON_PREF;
barCfg.type = pcie_BAR_TYPE32;
barCfg.memSpace = pcie_BAR_MEM_MEM;
barCfg.idx = PCIE_BAR_IDX_RC;

3. set the RO bits back to zero and then do the config:

/* Configure Address Translation */

uint32_t * bar = (uint32_t *) (0x51000010U + 4 * PCIE_BAR_IDX_RC);
*bar &= ~CSL_RCCFGDBICS_BAR_BASE_ADDR_RO_MASK;

barCfg.location = pcie_LOCATION_LOCAL;
barCfg.mode = pcie_RC_MODE;
barCfg.base = PCIE_IB_LO_ADDR_RC;
barCfg.prefetch = pcie_BAR_NON_PREF;
barCfg.type = pcie_BAR_TYPE32;
barCfg.memSpace = pcie_BAR_MEM_MEM;
barCfg.idx = PCIE_BAR_IDX_RC;

0x51000010 is the base address for the RC BAR registers.

Hope that helps,

Tim

0 Christopher Peters over 7 years ago in reply to Tim Dylla

Genius 3370 points

Tried #2 and #3. Seemed to get rid of the error, but PCIe still not working intermittently. I can see values in the PCIe registers, at 0x51000000 that are different from the defaults, so Linux must be setting up the PCIe before the DSP can. So, we are going to try #1 to see if that helps.

0 David Beaulieu over 7 years ago in reply to Christopher Peters

Expert 1280 points

Hi Tim Dylla,

Are you pushing the binary to the DSP using the Linux remoteproc / rpmsg drivers?
I am trying to perform pretty much the same thing but on the M4 instead.

Chris Pendleton, I have made an attempt to create and use a custom resource table but can't seem to get access to the PCIe resources from the M4 processor. I am almost positive I got the mapping wrong. Would you mind sharing your resource table that allowed you to configure and access the PCIe module on the DSP? It might help me to figure out what I did wrong on the M4 resource definition.

FYI, I have removed the PCIe components from the Linux device tree. I even went as far as completely removing the PCI bus support in the linux kernel.

Best regards,
- David

0 Chris Pendleton over 7 years ago in reply to David Beaulieu

Intellectual 410 points

Hi David.

Here is my DSP custom resource table: /cfs-file/__key/communityserver-discussions-components-files/791/rsc_5F00_table_5F00_vayu_5F00_dsp.h. Do you know about the bit-banding feature in the M4? I has this problem awhile back with the M4: https://e2e.ti.com/support/arm/sitara_arm/f/791/p/532104/1937953. I didn't check to see whether PCIe registers fall into this area.

Good luck. I was able to push the binary to the DSP with remoteproc.

Regards,

Chris

0 Tim Dylla over 7 years ago in reply to David Beaulieu

Expert 1070 points

Hi David,

yes, I used remoteproc, too. It was a no-brainer and comes with fancy features like debug log pipe via debugfs etc.

If you have already managed to build a project with custom resource table you only need to put the resulting binary into /lib/firmware/...

Best regards,

Tim

0 David Beaulieu over 7 years ago in reply to Chris Pendleton

Expert 1280 points

Thanks Chris,

I was vaguely aware of the bit-banding feature without actually knowning how it was callled. I had to adapt some of the PCIe init code that were referencing addresses in the 0x4A00 0000 range in the same manner you had to for the GPIO example.

Now, looking at the documentation provided in your thread, it seems that all addresses between 0x4000 0000 and 0x6000 0000 are offsetted to 0x6000 0000 to 0x8000 0000 on the IPU.

And, almost all the PCIe registers are in the space (SS1: 0x5100 0000, SS2: 0x5180 0000, PCIe reserved: 0x5200 0000). That might explain why my code wasn't working last week :).

Thanks for letting me know. I'll give it a shot.
Regards,
- David

0 David Beaulieu over 7 years ago in reply to David Beaulieu

Expert 1280 points

Ok, I got a lot further today.
I got the PCIe link-up working on the M4 while Linux is running. Like you pointed out, I had to offset the PCIe register access. The good news was that the PCIe LLD driver already supported this so it was as simple as changing the base addresses used to initialize the LLD.

But I am now stuck at enumerating. The AM572x is running in RC mode and I have an Altera FPGA connected to it. From the TRM, I gathered the enumeration is done using the ATU outbound region 0. However, I don't understand how to do it. From linux, I know that my Cyclone V FPGA is located on Bus 1, Device 0, function 0 on the PCIe bus. But I can't seem to figure out how to use the ATU region to enumerate...

How did you guys manage it?
So far, I haven't managed to figure it out using the TRM...

Thanks,
- David

0 Chris Pendleton over 7 years ago in reply to David Beaulieu

Intellectual 410 points

Hi David.

This problem was challenging. It requires a good understanding of the PCIe standard, the TI implementation of its PCIe modules, and the EP board/application on the other end. I am running with a Xilinx board, so it is very similar. The keys are the pcieIbTransCfg and pcieObTransCfg functions, and understanding how they map the L3 space to PCIe space. I attached a few files that might help. Look at function pcie2() in pcie_samples.c and cross-reference the translations as shown in the diagram. FYI I had to manually configure the BAR on my Xilinx EP board (XAPP1171 application). Good luck.

8228.pcie_sample.c

XAPP1171.pdf

0 Aidin Haddadi over 7 years ago in reply to Chris Pendleton

Expert 1085 points

Hi Chris

I am trying to run the example and I have build an adapter myself. But there is one problem: IN EVM board GPIO2_8 is dedicated for PCIe PERSTn signal. GPIO2_8 is passed through a uni directional buffer (inverter) which is fine in Root Complex Mode. But for running the test with 2 EVM boards, for End Point board the PERSTn should be input which is impossible because of the uni directional buffer in its way.
So how physically it is possible to run the test?
From dts file It seems the Linux kernel uses this signal in both RC and EP mode.
Am I missing something?

Thanks.

Processors

Processors forum

AM572x PCIe example bug in PDK v1.0.4