• TI Thinks Resolved

RTOS/AM5728: PCIe - How to Retrieve EP BAR

Part Number: AM5728

Tool/software: TI-RTOS

I'm running TIRTOS on AM5728 SOM with a Microsemi Igloo2 FPGA PCIe device plugged into the PCIe slot.  I have a successful linked up and can access the EP vendor and device info.   However, I'm having difficulty with figuring how to access the EP BAR addresses so i can turn on/off the its LEDs.  Base on the Linux device driver for this device, i need to use the BAR0 + LED_OFFSET.  Can I use the same method?  It is so, how can i request the BAR addresses from the FPGA?

Thanks.

  • In reply to lding:

    The vendor/device id and GEN2 data are correct based on the information my vendor provided.

    I can set  EP BAR with idx=1 and can see 0x40000 in 0x2000_1011 but not at 0x2000_1010.  However, when i set 0xFFFF_FFFFF to address 0x2000_1010, it changes to 0xFFF0_0000, but set it to 0x4_0000, reset it back to 0x0.

    When i change 0x4_0000 at  0x2000_1011 to 0xFFFFFFFF, CCS refreshes the value to 0xFFFF0000.   If I set it to 0x4_0001 or 0x4_0002/3/4, CCS shows 0x0004_0000

    This is the default BAR mask in pcie_sample.h

    #define PCIE_BAR_MASK        0x0FFFFFFF

    These are the information that my vendor provided to be used with their GUI on WIndows/Linux.  This description on paragraph 2.5.1 in the attached file.

    – Select Window 0 and configure following settings:
    • Size: Select as 1 MB from the drop-down list
    • PCIe BAR: Select as Bar0 from the drop-down list
    • Local Address: Enter values as 0x40000 to translate the BAR0 address space to CoreGPIO
    address (0x4000_0000)
    – Select Window 1 and configure following settings:
    • Size: Select as 64 KB from the drop-down list
    • PCIe BAR: Select as Bar1 from the drop-down list
    • Local Address: Enter values as 0x20000 to translate the BAR1 address space to
    COREAHBLSRAM address (0x2000_0000)

    Microsemi_SmartFusion2_PCIe_Control_Plane_Liberov11p8_Sp1_Tutorial_TU0456_V9.pdf

  • Guru 50180 points

    In reply to Jodat Vu:

    Hi,

    I interpret 0x2000_1011 as 0x2000_1014, because this is 32-bit address and BAR1 is at 0x2000_1014.

    For your trial of writing 0xFFFF_FFFF to BAR0 and got 0xFFF0_0000, this confirmed the BAR size is 0x10_0000. That is 1MB, matched your document excerpt. Similarly, BAR1 is 64KB from your trial. This also matched the document excerpt.

    For a BAR0 mask of 1MB, it masks bit 0-19. So you wrote a 0x4_0000 and got 0x0000_0000. It is expected. I don't understand "Local Address: Enter values as 0x40000 to translate the BAR0 address space to CoreGPIO address (0x4000_0000)" on the FPGA side." Maybe you can check the FPGA vendor for this.

    Some thing we can try on AM57x side,

    1) 

    barCfg.location = pcie_LOCATION_REMOTE;

     barCfg.mode     = pcie_EP_MODE;

     barCfg.base     = 0x40000; =========> change this to 0, as it has no effects

     barCfg.prefetch = pcie_BAR_NON_PREF;

     barCfg.type     = pcie_BAR_TYPE32;

     barCfg.memSpace = pcie_BAR_MEM_MEM;

     barCfg.idx      = 0;

     if ((retVal = Pcie_cfgBar(handle, &barCfg)) != pcie_RET_OK)

     {

       PCIE_logPrintf ("Failed to configure FPGA BAR0 (%d)\n", (int)retVal);

       exit(1);

     }

    2) In the pcie_sample.c, there is a function called pcieRet_e pcieObTransCfg(Pcie_Handle handle, uint32_t obAddrLo, uint32_t obAddrHi, uint8_t region)

    You see there is an additional region configure for RC mode

    if(PcieModeGbl == pcie_RC_MODE)
    {
    /*Configure OB region for remote configuration access space*/
    regionParams.regionDir = PCIE_ATU_REGION_DIR_OUTBOUND;
    regionParams.tlpType = PCIE_TLP_TYPE_CFG;
    regionParams.enableRegion = 1;

    regionParams.lowerBaseAddr = PCIE_WINDOW_CFG_BASE + resSize;
    regionParams.upperBaseAddr = 0; /* only 32 bits needed given data area size */
    regionParams.regionWindowSize = PCIE_WINDOW_CFG_MASK;

    regionParams.lowerTargetAddr = 0U;
    regionParams.upperTargetAddr = 0U;

    if ( (retVal = Pcie_atuRegionConfig(
    handle,
    pcie_LOCATION_LOCAL,
    (uint32_t) 0U,
    &regionParams)) != pcie_RET_OK)
    {
    return retVal;
    }
    }

    Where #define PCIE_WINDOW_CFG_MASK 0x00000FFFU

    This is a configuration translation to access the FPGA side PCIE registers with a 0x1000 window. That is why from AM572x side (0x2000_1000 to 0x2000_2000) you can look into FPGA side BAR0 for 0x1000 memory range (e.g, device/vendor ID, BAR, etc...).

    Please update the PCIE_WINDOW_CFG_MASK to a big number like  0x1F_FFFF (that is 2 MB window). Then from AM57 side, make sure the increase doesn't break your access of FPGA by looking into 0x2000_1000 region. Then try to look at 0x2004_1000 or 0x2004_0000 (just do it via CCS memory window) to see if the FPGA can map this 0x4_0000 address into your GPIO region.

    If still no luck, please try to check the FPGA vendor how to setup the access of BAR0 with address 0x4_0000. If this is a CFG access or MEM access? 

    Regards, Eric

  • In reply to lding:

    hi Eric,

    Using the driver and a demo program from the board vendor, we are able to see the following settings on Windows.   Is it possible to use these information to configure TI RTOS so it can R/W to the FPGA?

    If is possible, could you help with the changes?  Thanks.

  • Guru 50180 points

    In reply to Jodat Vu:

    Hi,

    When you use a PCIE RC with an operating system, like Windows or Linux machine, it worked differently than using the AM572x running RTOS code. The former has OS and does the enumeration, it reads in the BAR0/1 mask of EP and determine the memory address allocated and writes into the BAR0/1 of EP side. When you use the latter, it doesn't have enumeration process and the configuration of both RC and EP side is static.

    In your Windows PC test case, are you able to toggle the LED? How? E.g. accessing 0xFE80_0000 + 0x4_0000 + OFFSET?

    Then you can do the same way in RTOS like Windows PC case:
    1) in the CCS memory view, change the BAR0 of EP (address 0x2000_1010 = 0xFE80_0000).
    2) in the AM572x RC side, when configure the OB translation, use 0xFE80_0000 as the outbound address (instead of 0x0000_0000 or 0x40000 or 0x7000_0000).

    Regards, Eric
  • In reply to lding:

    hi Eric,

    With the completion of step #1 and #2,  I can see BAR0 set to 0xFE80_0000 at 0x2000_1010, and from CCS, I can view and set values at 0xFE80_0000 + <offset>.   However, I still can't  control the LEDs with numerous offsets provided by my FPGA vendor.

    Is there any other ways within CCS that would help me verifying if EP BAR0 is set properly and if there is any outbound address translation going on that could mess up the address mapping?

    Thanks.