Linux/AM5718: PCIe EP allocated memory is only 2MB

@font-face{ font-family:"Times New Roman"; } @font-face{ font-family:"宋体"; } @font-face{ font-family:"SimSun"; } @font-face{ font-family:"Calibri"; } @list l0:level1{ mso-level-start-at:2; mso-level-number-format:decimal; mso-level-suffix:space; mso-level-text:"%1."; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:0.0000pt; text-indent:0.0000pt; font-family:'Times New Roman';} p.MsoNormal{ mso-style-name:Normal; mso-style-parent:""; margin:0pt; margin-bottom:.0001pt; mso-pagination:none; text-align:justify; text-justify:inter-ideograph; font-family:Calibri; mso-fareast-font-family:SimSun; mso-bidi-font-family:'Times New Roman'; font-size:10.5000pt; mso-font-kerning:1.0000pt; } span.msoIns{ mso-style-type:export-only; mso-style-name:""; text-decoration:underline; text-underline:single; color:blue; } span.msoDel{ mso-style-type:export-only; mso-style-name:""; text-decoration:line-through; color:red; } @page{mso-page-border-surround-header:no; mso-page-border-surround-footer:no;}@page Section0{ margin-top:72.0000pt; margin-bottom:72.0000pt; margin-left:90.0000pt; margin-right:90.0000pt; size:595.3000pt 841.9000pt; layout-grid:15.6000pt; } div.Section0{page:Section0;}

Dear Rex,

Yes. Without bridge if we connect the Endpoint, able to allocate 32MB for it. But with the bridge we are getting only 2MB.

During drivers code walk through, identified the corresponding macros in  ./drivers/pci/pci.c.

#define DEFAULT_CARDBUS_IO_SIZE (256)
#define DEFAULT_CARDBUS_MEM_SIZE (64*1024*1024)
/* pci=cbmemsize=nnM,cbiosize=nn can override this */
unsigned long pci_cardbus_io_size = DEFAULT_CARDBUS_IO_SIZE;
unsigned long pci_cardbus_mem_size = DEFAULT_CARDBUS_MEM_SIZE;

#define DEFAULT_HOTPLUG_IO_SIZE (256)
#define DEFAULT_HOTPLUG_MEM_SIZE (2*1024*1024)
/* pci=hpmemsize=nnM,hpiosize=nn can override this */
unsigned long pci_hotplug_io_size = DEFAULT_HOTPLUG_IO_SIZE;
unsigned long pci_hotplug_mem_size = DEFAULT_HOTPLUG_MEM_SIZE;

We have modified the following macro 

#define DEFAULT_HOTPLUG_MEM_SIZE (8*1024*1024) in drivers/pci/pci.c

after that we could see the Memory behind bridge: 8MB 

Our Endpoint which is requesting 8MB is not getting Enumerated. Then we increased the size to 16MB. Later it started getting enumerated

#define DEFAULT_HOTPLUG_MEM_SIZE (16*1024*1024) in drivers/pci/pci.c

For one more Endpoint we require 64MB and accordingly increased the macro size to 

#define DEFAULT_HOTPLUG_MEM_SIZE (64*1024*1024)

In this case “Memory behind bridge” for PCIe switch down port is not allocating memory space for any of the bridge ports. We suspect two reasons for this: 

1. For each Bridge port we are observing two memory spaces as shown below. You can also observe the same at the end, which we are providing lspci output.

Memory behind bridge: 20200000-211fffff

     Prefetchable memory behind bridge: 0000000024200000-00000000251fffff

2. In Device trees we are requesting 256 MB which is by default in the SDK and also as per the information in AM5718 TRM: 

Table 2-1. L3_MAIN Memory Map.

/* Default device tree */

 pcie1_rc: pcie@51000000 {

compatible = "ti,dra7-pcie";

reg = <0x51000000 0x2000>, <0x51002000 0x14c>, <0x1000 0x2000>;

reg-names = "rc_dbics", "ti_conf", "config";

interrupts = <0 232 0x4>, <0 233 0x4>;

#address-cells = <3>; 

#size-cells = <2>; 

device_type = "pci";

ranges = <0x81000000 0 0          0x03000 0 0x00010000

0x82000000 0 0x20013000 0x13000 0 0xffed000>; 

Using the linux lspci command, we are observing the PCIe switch down Port memory space as shown below

(i.e when we set the macro value to 16MB 

#define DEFAULT_HOTPLUG_MEM_SIZE (16*1024*1024) in drivers/pci/pci.c).

 Memory behind bridge: 20200000-211fffff

     Prefetchable memory behind bridge: 0000000024200000-00000000251fffff

00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01) (prog-if 00 [Normal decode])

        Flags: bus master, fast devsel, latency 0, IRQ 439

        Memory at 20100000 (32-bit, non-prefetchable) [size=1M]

        Memory at 20020000 (32-bit, non-prefetchable) [size=64K]

        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0

        I/O behind bridge: 00001000-00004fff

        Memory behind bridge: 20200000-241fffff

        Prefetchable memory behind bridge: 24200000-281fffff

        Capabilities: [40] Power Management version 3

        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+

        Capabilities: [70] Express Root Port (Slot-), MSI 00

        Capabilities: [100] Advanced Error Reporting

        Kernel driver in use: pcieport

01:00.0 PCI bridge: Pericom Semiconductor Device 2608 (prog-if 00 [Normal decode])

        Flags: bus master, fast devsel, latency 0

        Bus: primary=01, secondary=02, subordinate=06, sec-latency=0

        I/O behind bridge: 00001000-00004fff

        Memory behind bridge: 20200000-241fffff

        Prefetchable memory behind bridge: 0000000024200000-00000000281fffff

        Capabilities: [40] Power Management version 3

        Capabilities: [5c] Vital Product Data

        Capabilities: [64] Vendor Specific Information: Len=34 <?>

        Capabilities: [b0] Subsystem: Device 0000:0000

        Capabilities: [c0] Express Upstream Port, MSI 00

        Capabilities: [100] Advanced Error Reporting

        Capabilities: [140] Virtual Channel

        Capabilities: [20c] Power Budgeting <?>

        Capabilities: [230] Latency Tolerance Reporting

        Capabilities: [240] L1 PM Substates

        Kernel driver in use: pcieport

02:01.0 PCI bridge: Pericom Semiconductor Device 2608 (prog-if 00 [Normal decode])

        Flags: bus master, fast devsel, latency 0, IRQ 440

        Bus: primary=02, secondary=03, subordinate=03, sec-latency=0

        I/O behind bridge: 00001000-00001fff

        Memory behind bridge: 20200000-211fffff

        Prefetchable memory behind bridge: 0000000024200000-00000000251fffff

        Capabilities: [40] Power Management version 3

        Capabilities: [4c] MSI: Enable+ Count=1/1 Maskable- 64bit+

        Capabilities: [64] Vendor Specific Information: Len=34 <?>

        Capabilities: [b0] Subsystem: Device 0000:0000

        Capabilities: [c0] Express Downstream Port (Slot+), MSI 00

        Capabilities: [100] Advanced Error Reporting

        Capabilities: [140] Virtual Channel

        Capabilities: [20c] Power Budgeting <?>

        Capabilities: [220] Access Control Services

        Capabilities: [240] L1 PM Substates

        Kernel driver in use: pcieport

02:02.0 PCI bridge: Pericom Semiconductor Device 2608 (prog-if 00 [Normal decode])

        Flags: bus master, fast devsel, latency 0, IRQ 441

        Bus: primary=02, secondary=04, subordinate=04, sec-latency=0

        I/O behind bridge: 00002000-00002fff

        Memory behind bridge: 21200000-221fffff

        Prefetchable memory behind bridge: 0000000025200000-00000000261fffff

        Capabilities: [40] Power Management version 3

        Capabilities: [4c] MSI: Enable+ Count=1/1 Maskable- 64bit+

        Capabilities: [64] Vendor Specific Information: Len=34 <?>

        Capabilities: [b0] Subsystem: Device 0000:0000

        Capabilities: [c0] Express Downstream Port (Slot+), MSI 00

        Capabilities: [100] Advanced Error Reporting

        Capabilities: [140] Virtual Channel

        Capabilities: [20c] Power Budgeting <?>

        Capabilities: [220] Access Control Services

        Capabilities: [240] L1 PM Substates

        Kernel driver in use: pcieport

02:03.0 PCI bridge: Pericom Semiconductor Device 2608 (prog-if 00 [Normal decode])

        Flags: bus master, fast devsel, latency 0, IRQ 442

        Bus: primary=02, secondary=05, subordinate=05, sec-latency=0

        I/O behind bridge: 00003000-00003fff

        Memory behind bridge: 22200000-231fffff

        Prefetchable memory behind bridge: 0000000026200000-00000000271fffff

        Capabilities: [40] Power Management version 3

        Capabilities: [4c] MSI: Enable+ Count=1/1 Maskable- 64bit+

        Capabilities: [64] Vendor Specific Information: Len=34 <?>

        Capabilities: [b0] Subsystem: Device 0000:0000

        Capabilities: [c0] Express Downstream Port (Slot+), MSI 00

        Capabilities: [100] Advanced Error Reporting

        Capabilities: [140] Virtual Channel

        Capabilities: [20c] Power Budgeting <?>

        Capabilities: [220] Access Control Services

        Capabilities: [240] L1 PM Substates

        Kernel driver in use: pcieport

02:04.0 PCI bridge: Pericom Semiconductor Device 2608 (prog-if 00 [Normal decode])

        Flags: bus master, fast devsel, latency 0, IRQ 443

        Bus: primary=02, secondary=06, subordinate=06, sec-latency=0

        I/O behind bridge: 00004000-00004fff

        Memory behind bridge: 23200000-241fffff

        Prefetchable memory behind bridge: 0000000027200000-00000000281fffff

        Capabilities: [40] Power Management version 3

        Capabilities: [4c] MSI: Enable+ Count=1/1 Maskable- 64bit+

        Capabilities: [64] Vendor Specific Information: Len=34 <?>

        Capabilities: [b0] Subsystem: Device 0000:0000

        Capabilities: [c0] Express Downstream Port (Slot+), MSI 00

        Capabilities: [100] Advanced Error Reporting

        Capabilities: [140] Virtual Channel

        Capabilities: [20c] Power Budgeting <?>

        Capabilities: [220] Access Control Services

        Capabilities: [240] L1 PM Substates

        Kernel driver in use: pcieport

Thanks,

Suryaprakash Rao.

Rex,

Any update?

Tom

Suryaprakesh,

Please post an update to this thread.

Tom

Rex,

The team has been able to get the accessible memory up to 16MB.  The system appears to allocate 16MB for accessible memory and 16MB for prefetch memory for a total of 32MB per Endpoint.  Even with the need for the prefetch block, they believe that the memory should double to 32MB.  However, the system refuses to provide more than 16MB.

There appears to be access issues for the development team.  I will work to get that resolved.

Tom

Please keep this open until customer posts updates that improved operation.

Still waiting for customer to post solution implemented.

Tom