Forums

Joel,

The soft reset will reset most of the PCIe memory map registers (MMRs) except the sticky bits in those registers. Unfortunately, the BAR registers or other configuration registers are not sticky and will be reset to default values (PCIe link will be lost as you have seen on the Host side).

This change (sticky bits in PCIe MMR, not all PCIe MMR for soft reset) will be addressed in the next releases of C6678 data manual and PCIe user's guide.

We will work on the IBL reboot issue and get back to you.

Eventually we need to re-configure the BAR or other related registers in PCIe after the soft reset and reboot. RC may need to re-enumerate and re-train the PCIe link again after soft reset, just like the initial start up.

I am not sure if the local reset of DSP CorePac will be good for your application since it will not reset the PCIe MMRs.

Sincerely,

Steven

Hi Joel,

In our board design,we have C6678 and DM8148.

        The c6678 DSP acts as a 'number-crunching' pcie peripheral controlled by a TI DM8148 SOC RC running linux(it is same as yours).

        PCIe memory mapping and pcie-based booting are working wrong for me.

        The following is the kernel printing：

   ti81xx_pcie: Invoking PCI BIOS...

ti81xx_pcie: Setting up Host Controller...
ti81xx_pcie: Register base mapped @0xc6820000
ti81xx_pcie: INTx enabled legacy IRQ48
ti81xx_pcie: MSI info enabled msi_irq=49 msi_irq_num=32
ti81xx_pcie: Starting PCI scan...
PCI: bus0: Fast back to back transfers disabled
PCI: bus1: Fast back to back transfers disabled
pci 0000:00:00.0: BAR 9: assigned [mem 0x20000000-0x28ffffff pref]
pci 0000:00:00.0: BAR 8: assigned [mem 0x29000000-0x290fffff]
pci 0000:00:00.0: PCI bridge to [bus 01-01]
pci 0000:00:00.0:   bridge window [io  disabled]
pci 0000:00:00.0:   bridge window [mem 0x29000000-0x290fffff]
pci 0000:00:00.0:   bridge window [mem 0x20000000-0x28ffffff pref]
PCI: enabling device 0000:00:00.0 (0140 -> 0143)

Finding the device....
Found TI device
TI device: vendor=0x0000104c, dev=0x0000b005, drv=0x00000000, irq=0x0030, bus=0xc2c7c200
Reading the BAR areas....
BAR0 start:0x0
BAR0 len:0x1000
BAR0 flag:0x40200
BAR1 start:0x10000000
BAR1 len:0x1000000
BAR1 flag:0x42208
BAR2 start:0x10000000
BAR2 len:0x2000000
BAR2 flag:0x42208
BAR3 start:0x12000000
BAR3 len:0x2000000
BAR3 flag:0x42208
Enabling the device....
pci 0000:01:00.0: Refused to change power state, currently in D3
PCI: Device 0000:01:00.0 not available because of resource collisions
Access PCIE application register ....

Did you have the same problem?How can you save it?

Thanks,

zengxingyuan

Sorry zengxingyuan, I have not seen that error.  

Hi Joel,

Did you use IBL,I did not use IBL,so we can not read the C6678's PCI_CLASS_REVISION.

  Now,the problem solved,but a new error happened.

ti81xx_pcie: Invoking PCI BIOS...
ti81xx_pcie: Setting up Host Controller...
ti81xx_pcie: Register base mapped @0xc6820000
ti81xx_pcie: INTx enabled legacy IRQ48
ti81xx_pcie: MSI info enabled msi_irq=49 msi_irq_num=32
ti81xx_pcie: Starting PCI scan...
@@@@yuan class:6040001
PCI: bus0: Fast back to back transfers disabled
@@@@yuan class:4800001
PCI: bus1: Fast back to back transfers disabled
pci 0000:00:00.0: BAR 9: assigned [mem 0x20000000-0x28ffffff pref]
pci 0000:00:00.0: BAR 8: assigned [mem 0x29000000-0x290fffff]
pci 0000:01:00.0: BAR 4: assigned [mem 0x20000000-0x23ffffff pref]
pci 0000:01:00.0: BAR 4: set to [mem 0x20000000-0x23ffffff pref] (PCI address [0x20000000-0x23ffffff])
pci 0000:01:00.0: BAR 2: assigned [mem 0x24000000-0x25ffffff pref]
pci 0000:01:00.0: BAR 2: set to [mem 0x24000000-0x25ffffff pref] (PCI address [0x24000000-0x25ffffff])
pci 0000:01:00.0: BAR 3: assigned [mem 0x26000000-0x27ffffff pref]
pci 0000:01:00.0: BAR 3: set to [mem 0x26000000-0x27ffffff pref] (PCI address [0x26000000-0x27ffffff])
pci 0000:01:00.0: BAR 1: assigned [mem 0x28000000-0x28ffffff pref]
pci 0000:01:00.0: BAR 1: set to [mem 0x28000000-0x28ffffff pref] (PCI address [0x28000000-0x28ffffff])
pci 0000:01:00.0: BAR 0: assigned [mem 0x29000000-0x29000fff]
pci 0000:01:00.0: BAR 0: set to [mem 0x29000000-0x29000fff] (PCI address [0x29000000-0x29000fff])
pci 0000:00:00.0: PCI bridge to [bus 01-01]
pci 0000:00:00.0:   bridge window [io  disabled]
pci 0000:00:00.0:   bridge window [mem 0x29000000-0x290fffff]
pci 0000:00:00.0:   bridge window [mem 0x20000000-0x28ffffff pref]
PCI: enabling device 0000:00:00.0 (0146 -> 0147)

After the kernel register the pci bus,when I want to boot the C6678,it says BUS error.

<4>Enabling the device....
<7>ti81xx_pcie: Reading config[44] for device 0001:00:00..
<7>ti81xx_pcie: Data abort: address = 0xc6821728 fsr = 0x1008 PC = 0xc005ccc0 LR = 0xc005ce60
<7>ti81xx_pcie: Handled PCIe abort
<7>failed. No link/device.
<7>ti81xx_pcie: Reading config[44] for device 0001:00:00..
<7>ti81xx_pcie: Data abort: address = 0xc6821728 fsr = 0x1008 PC = 0xc005ccc0 LR = 0xc005ce60
<7>ti81xx_pcie: Handled PCIe abort
<7>failed. No link/device.
<7>ti81xx_pcie: Writing config[44] = 0 for device 0001:00:00 ...
<7>ti81xx_pcie: Data abort: address = 0xc6821728 fsr = 0x1008 PC = 0xc005ccc0 LR = 0xc005cd78
<7>ti81xx_pcie: Handled PCIe abort
<7>failed. No link/device.
<7>ti81xx_pcie: Reading config[44] for device 0001:00:00..
<7>ti81xx_pcie: Data abort: address = 0xc6821728 fsr = 0x1008 PC = 0xc005ccc0 LR = 0xc005ce60
<7>ti81xx_pcie: Handled PCIe abort
<7>failed. No link/device.
<6>pci 0000:01:00.0: Refused to change power state, currently in D3
<7>ti81xx_pcie: Reading config[4] for device 0001:00:00..
<7>ti81xx_pcie: Data abort: address = 0xc6821728 fsr = 0x1008 PC = 0xc005ccc0 LR = 0xc005ce60
<7>ti81xx_pcie: Handled PCIe abort
<7>failed. No link/device.
<7>ti81xx_pcie: Reading config[4] for device 0001:00:00..
<7>ti81xx_pcie: Data abort: address = 0xc6821728 fsr = 0x1008 PC = 0xc005ccc0 LR = 0xc005ce60
<7>ti81xx_pcie: Handled PCIe abort
<7>failed. No link/device.
<7>ti81xx_pcie: Writing config[d] = 80 for device 0001:00:00 ...
<7>ti81xx_pcie: Data abort: address = 0xc6821728 fsr = 0x1008 PC = 0xc005ccc0 LR = 0xc005cd78
<7>ti81xx_pcie: Handled PCIe abort
<7>failed. No link/device.
<7>ti81xx_pcie: Reading config[c] for device 0001:00:00..
<7>ti81xx_pcie: Data abort: address = 0xc6821728 fsr = 0x1008 PC = 0xc005ccc0 LR = 0xc005ce60
<7>ti81xx_pcie: Handled PCIe abort
<7>failed. No link/device.
<7>ti81xx_pcie: Reading config[4] for device 0001:00:00..
<7>ti81xx_pcie: Data abort: address = 0xc6821728 fsr = 0x1008 PC = 0xc005ccc0 LR = 0xc005ce60
<7>ti81xx_pcie: Handled PCIe abort
<7>failed. No link/device.
<7>ti81xx_pcie: Reading config[4] for device 0001:00:00..
<7>ti81xx_pcie: Data abort: address = 0xc6821728 fsr = 0x1008 PC = 0xc005ccc0 LR = 0xc005ce60
<7>ti81xx_pcie: Handled PCIe abort
<7>failed. No link/device.
<7>ti81xx_pcie: Writing config[4] = fbff for device 0001:00:00 ...
<7>ti81xx_pcie: Data abort: address = 0xc6821728 fsr = 0x1008 PC = 0xc005ccc0 LR = 0xc005cd78
<7>ti81xx_pcie: Handled PCIe abort
<7>failed. No link/device

Joel,

I'm trying to configure our C6678 evaluation board in the same way as you described...a pcie peripheral board for doing the heavy lifting.  I've created a module and it insmods fine, and I can make simple changes to the card.  But the means to transfer arbitrary data between the card (and accessible by the cores on the board) and the host Linux system has eluded me so far.  I can't find an example and the documentation is just not making it clear enough to me. Do you have some guidance on how to set that up, or what documentation helped you do it?  Thanks.

Hi Rob,

The two things that helped me most are the PCIe bootloader example which comes with the MCSDK, at /opt/ti/mcsdk_2_00_05_17/tools/boot_loader/examples/pcie on my system, and the PCIE Peripheral User Guide (sprugs6a.pdf).  The most important thing to understand is the way that the PCIe peripheral does address translations.  You mention that you are able to make 'simple changes' to the card.  what do you mean by that?

Joel

Hi Steven,  

You mention that the "RC may need to re-enumerate and re-train the PCIe link again after soft reset".  Do you know how I can accomplish this when using Linux on a DM8148 as the RC?

Thanks,

Joel

 

Joel,

It is just like the initial setup of PCIe link training, writing 0x1 to LTSSM_EN field (bit 0) in CMD_STATUS register (offset 0x4) in PCIe module of both RC and EP.

Then both RC and EP could start the link negotiation. The link training will be done when LTSSM_STATE field (bit 4:0) is read as 0x11 in DEBUG0 register (offset 0x728) and the PCIe link between RC and EP is ready for the communication.

Sincerely,

Steven

Well, I may have overstated what I was able to do.  I didn't actually attempt to make changes to the card configuration, but at some point I was able to read PID register, so I assumed I'd be able to write to configuration registers if needed.

Here is what I have done.  I took the boot loader example program and inserted parts of it into a simple framework for a driver module.  Now when I insmod, it finds the card and executes the actions the boot loader example did before (loading Hello World on the cores and executing them).  I then added the char device registration, and created the /dev/device.  So now I have a user-mode program that takes the /dev/device file,opens it, and then executes a read.

For the read program, I am just taking various hard-coded memory addresses (DDR_START, LL2_START, etc) and using DSPRead (from the boot loader example) and returning them to the user program.  Doing this I've been able to get back 0xBABEFACE and also the coded binary of the hello world example.

I've read the sprugs manual, and I agree it all boils down to the memory translation, but so far it's just not sinking in what that practically means.  I copied in the part of the bootloader that sets the inbound address translations, so that's a part of my driver module...but I'm still not exactly sure what that is doing, or what it means to my user-mode program trying to access the DSP board's memory.

My ultimate goal is to have an area of memory that I can read and write to, and that the cores (or a single core) running on the board can also read/write from so that we can communicate data back and forth.  I'm sure EDMA is involved in this somehow, but that is an added layer of complexity that I don't need right now if I can avoid it.  If I can write a value from a usermode program to this board using my driver module (either through an ioctl or a write), have the board change that value somehow, and then read it back....I'll declare complete victory.

I appreciate any help, thanks Joel.

Steven,

The issue is that on the Linux side (RC) I am not writing the device driver for the PCIe peripheral.  I'm using the RC driver which comes with the EZSDK for the DM8148.  I *could* write to the CMD_STATUS register of the RC, but I wouldn't want to do that from my EP's device driver would I?   I think I want to request that the PCIe subsystem reinitialize itself.   I think I essentially need to re-do what is done in ti81xx_pcie_setup() in pcie-ti81xx.c, correct?  Is there a way (perhaps through sysfs or something) that I can make the RC re-initialize, or do I need to modify the RC pcie driver (pcie-ti81xx.c)? 

Thanks,

Joel

Joel,

I am not familiar with the RC driver on DM8148 side. I think you probably have already seen the following page or PSP driver document for DM8148:

http://processors.wiki.ti.com/index.php/DM81xx_AM38xx_PCI_Express_Root_Complex_Driver_User_Guide#RC_Driver_Source_Files

In the register level, the link training in the PCIe modules of the DM8148 and C66x should be the same as what I described in previous reply.

And the PCIe link training should be included in the function of the driver you mentioned before and you could give a try on it before modifying the RC driver.

The DM81xx forum may have more info on the RC driver related question as well.

Sincerely,

Steven

Rob,

It sounds like you are quite close.   If you look at the bootloader example in the init_module() function, you will see something like this:

		//Configure IB_BAR1 to BAR1 for LL2 for core0
		iowrite32(1, ptrReg+IB_BAR1/4);     
		iowrite32((int)PCIE_DEV->resource[1].start, ptrReg+IB_START1_LO/4);    
		iowrite32(0, ptrReg+IB_START1_HI/4);     
		iowrite32(LL2_START + (1 << 28), ptrReg+IB_OFFSET1/4);    

Basically, if you understand what these 4 lines are doing, I think you'll get it.  From the PCIe User guide:

However, address translation for Address Space 1 requires the use of one of the four
regions (Regions[0-3]) to map accepted TLPs to internal memory. Four memory
mapped registers that are region specific are used by the inbound address translator.
1. Inbound Translation Bar Match Register (IB_BARn [n=0-3]) — This field
indicates which BAR the inbound transaction must be targeted to for the
translation rule specified to activate.
2. Inbound Translation Start Address High Register (IB_STARTn_HI [n=0-3])
— This field indicates the starting address bits [63:32] as seen in PCIe address.
Typically, this field will match the BAR value and is used as the reference for this
address translation region in 64-bit addressing. This register is required to be
programmed with a Zero value if using 32-bit addressing.
3. Inbound Translation Start Address Low Register (IB_STARTn_LO [n=0-3])
— This field indicates the starting address bits [31:0] as seen in PCIe address.
Typically, this field will match the BAR value and is used as the reference for this
address translation region in both of 32-bit and 64-bit addressing.
4. Inbound Translation Address Offset Register (IB_OFFSETn [n=0-3]) — This
field indicates the internal bus address that will be the starting point of the
mapped or translated PCIE address region.

What this means is that after configuring the device's inbound address translation unit as above, your device driver can do read/write operations to/from addresses starting at

PCIE_DEV->resource[1].start and these reads/writes will target the DSP's memory starting at LL2_START + (1 << 28).   PCIE_DEV->resource[1].start is the physical address that Linux

assigned to BAR1 of your device.  To actually read/write this address range from your driver, you will want to map the memory via ioremap(), so that you have a kernel virtual address to work with.

Hope this helps,

Joel

Okay,  Thanks Steven.  I'll try posting to the DM81xx forum when I get back to working on this.  

Joel, Yes, that is beginning to make some kind of sense. The pci_readBar function from the boot_loader seems to do the required ioremap. So that leaves me with a regVirt (virtual address for app registers), memVirt (Local L2 virtual address), msmcVirt (dunno), and ddrVirt (DDR virtual address). So then, using these memory pointers, I can read/write to those various memory locations from the driver. I guess the remaining questions I have is: 1) What is the difference between all of those memory regions? I'm pretty sure the app registers is for changing board registers (and thus configuration). Local L2...that's memory that is specific to a core? MSMC....don't know what that is, but comments seem to indicate "shared L2". So, L2 memory all cores can access? and DDR....also seems shared. So what's the difference then between MSMC and DDR? 2) So, in the code executing on the core itself (say, core 4), then for me to access DDR, I'd just read/write to address 0x80000000? So in theory, if I use core 4 to write 0xFAFAFAFA to 0x80000000, and then use the Linux driver to read from ddrVirt + 0, I should get that value? Seems like something worth trying... Thanks for the help. I knew I was close, but had mental hurdles to get over.

It worked! I used core 4 to write 0xFAFAFAF4 to 0x8000000, and then in my read function of the driver, I just read from ddrVirt...and got the value. Neato. Now to see if I can write from the user area into DDR via the driver, and have one of the cores see it and react. That'd be awesome. Other than understanding why I'd use one memory area over the other, I think I'm beginning to understand this. Thanks Joel! It just too a nudge.