Tool/software: TI-RTOS
Hi there,
I'm currently in the process of writing a Linux device driver to talk to a K2H demo board via PCIe. The K2H has the SDK PCIe SYS/BIOS demo example running on it, and it is running in End Point mode. The project was generated via the SDK tools, and is called PCIE_K2H_wSoCFile_C66BiosExampleProject in Code Composer Studio. I have manually hardcoded pcieMode_e PcieModeGbl = pcie_EP_MODE; in order to force End Point mode.
On the other side, I have a Linux device driver that I have from an engineer on a different project. I have lightly modified his code in an attempt to get it to work. It is currently running on an x86 Linux machine, and the DSP is slotted into a PCIe slot on the device's motherboard.
The sequence of events that I run is this:
- Power off x86 machine. DSP is powered off because it draws power from the PCIe slot.
- Plug in the power supply to the DSP, and connect usb to the on-board JTAG emulator to a separate PC (not the same x86 machine - although ultimately I don't think it matters).
- Initialize/connect to DSP Core 0 via a GEL file in code composer studio. Load the .out file generated from the SDK PCIe example onto the DSP board and run it. It will be in LTSSM State 2 because it cannot perform link training on any connected device - because the x86 machine is still off.
- Power on the x86 machine. The print statements in code composer studio indicate that the "Link is up" - therefore link training has succeeded.
- On the Linux x86 machine, do an $ lspci . Notice that I can see device: 0000:01:00.0 Non-VGA unclassified device: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)
- This means that we've been able to enumerate the device. So far so good.
- Move kernel module, built for the correct architecture, onto the x86 Linux machine.
- Do an $ insmod pcie_axi.ko
- Check the kernel log via $ dmesg | tail -n 100
- Interesting output is as follows:
- [ 48.388563] Some attributes of the pci_dev *pci_dev object
[ 48.394090] int cfg_size, Size of configuration space, is 4096
[ 48.399923] unsigned short vendor = 104c
[ 48.403926] unsigned short device = 8888
[ 48.407953] unsigned short subsystem_vendor = 0
[ 48.412564] unsigned short subsystem_device = 1
[ 48.417184] pci_dev->current_state = 0
[ 48.421016] PCI_ROM_RESOURCE = 6
[ 48.424237] PCI_BRIDGE_RESOURCES = d
[ 48.427811] DEVICE_COUNT_RESOURCE = 11
[ 48.431553] Beginning of pci_probe
[ 48.434964] pcie_dsp 0000:01:00.0: can't enable device: BAR 0 [mem 0x9040000000-0x904fffffff] not claimed
[ 48.445304] Error: Not able to enable pci device.
- [ 48.388563] Some attributes of the pci_dev *pci_dev object
- This error is coming about when the device driver attempts to call the method pci_enable_device during a PCIe "probe"
-
/// \Name: pci_probe
/// \Description: Probes PCI bus for devices with matching VENDOR_ID and DEVICE_ID (found in id param)
/// so this driver can be associated with the device(s)
/// \Arguments:
/// -struct pci_dev *pci_dev: struct describing device
/// -const struct pci_device_id *id: struct with VENDOR_ID and DEVICE_ID values that this driver is compatible with
/// \Returns:
/// -int: 0 if successful, nonzero for error
static int pci_probe(struct pci_dev *pci_dev, const struct pci_device_id *id)
{
int result;
int index = -1; // Which number in the list is the current device? Used for assigning char minor number
int i; // Loop variable
/* PRINT STATEMENTS FOR DEBUG */
printk(KERN_CRIT "Some attributes of the pci_dev *pci_dev object\n");
printk(KERN_CRIT "int cfg_size, Size of configuration space, is %d\n",pci_dev->cfg_size);
printk(KERN_CRIT "unsigned short vendor = %x \n", pci_dev->vendor);
printk(KERN_CRIT "unsigned short device = %x \n", pci_dev->device);
printk(KERN_CRIT "unsigned short subsystem_vendor = %x \n", pci_dev->subsystem_vendor);
printk(KERN_CRIT "unsigned short subsystem_device = %x \n", pci_dev->subsystem_device);
if (pci_dev->pm_cap) {
u16 pmcsr;
pci_read_config_word(pci_dev, pci_dev->pm_cap + PCI_PM_CTRL, &pmcsr);
printk(KERN_CRIT "pci_dev->current_state = %x \n",(pmcsr & PCI_PM_CTRL_STATE_MASK) );
}
else{printk(KERN_CRIT "elsed on the pci_dev->pm_cap\n");
}printk(KERN_CRIT "PCI_ROM_RESOURCE = %x\n", PCI_ROM_RESOURCE);
printk(KERN_CRIT "PCI_BRIDGE_RESOURCES = %x\n", PCI_BRIDGE_RESOURCES);
printk(KERN_CRIT "DEVICE_COUNT_RESOURCE = %x\n", DEVICE_COUNT_RESOURCE);
/* END ANDRE PRINT STATEMENTS FOR DEBUG */
#if DEBUG_MODE
printk(KERN_CRIT "Beginning of pci_probe\n");
#endif // #if DEBUG_MODE/* Enable PCI device. This takes care of the following:
* - Wakes up device if in suspended state
* - Allocates I/O and memory regions (if not done by BIOS)
* - Assigns an IRQ to the device (if not done by BIOS) */
result = pci_enable_device(pci_dev);
if(result)
{
printk(KERN_ERR "Error: Not able to enable pci device.\n");
return result;
} // if
else
{
printk(KERN_CRIT "Successfully enabled device\n\r");
} // else}
- Interesting output is as follows:
- Check lspci for more detailed information via $ lspci -xvv
- Relevant output is below:
0000:01:00.0 Non-VGA unclassified device: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 366
Region 0: Memory at 9040000000 (32-bit, non-prefetchable) [size=256M]
Region 1: Memory at 9050000000 (32-bit, non-prefetchable) [size=256M]
Region 2: Memory at 9060000000 (32-bit, prefetchable) [size=1M]
Region 3: Memory at 9060100000 (32-bit, prefetchable) [size=64K]
Region 4: Memory at 9060110000 (32-bit, prefetchable) [size=4K]
Region 5: Memory at 9060120000 (32-bit, prefetchable) [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 15.000W
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 256 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s, Exit Latency L0s <2us, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn+ ChkCap+ ChkEn+
00: 4c 10 88 88 46 01 10 00 01 00 00 00 08 00 00 00
10: 00 00 00 40 00 00 00 50 08 00 00 60 08 00 10 60
20: 08 00 11 60 08 00 12 60 00 00 00 00 00 00 01 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 6e 01 00 00
Do you see any reason why I can't connect? My current suspicion is that there's an issue with the inbound address translation settings on the DSP side. I notice from reading the pcie_sample.c and .h files that PCIE_IB_LO_ADDR_EP (the inbound lower address for the endpoint) is set to 0x70000000. And then the outbound lower address for the endpoint is set to 0x90000000.
Which is confusing to me because the lspci -xvv command shows me: Region 0: Memory at 9040000000 (32-bit, non-prefetchable) [size=256M]
And also the kernel driver error message says: [ 48.434964] pcie_dsp 0000:01:00.0: can't enable device: BAR 0 [mem 0x9040000000-0x904fffffff] not claimed
This corresponds to the outbound lower address for the DSP endpoint, which is weird because my understanding is that the Linux driver (the Root Complex) should be trying to access the INBOUND lower address for the DSP endpoint - 0x70000000 + some offset. Please correct me if I'm mistaken, I'm getting most of my information from Chapter 2.7 of the PCI Express (PCIe) for KeyStone Devices User's Guide (Rev. D) here: http://www.ti.com/lit/ug/sprugs6d/sprugs6d.pdf
Please let me know what help you can give me in figuring out this issue.
Thanks very much!
-Andre
P.S. There are some macros in the pcie_sample.c program that define PCIE_REV2_HW and PCIE_REV1_HW. Right now neither is defined - so do either of these need to be forcefully defined? Right now the only define I have is SOC_K2H - which seems as though it enables other macros as needed at the top of these files. It's just that when I do an lspci on the x86 Linux machine I see the following output:
0000:01:00.0 Non-VGA unclassified device: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01)
Which makes me think that I may need to enable the PCIE_REV1_HW macro? Please let me know.
