This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

PCIe enumeration problems

Other Parts Discussed in Thread: TMDXEVMPCI

Hello,

    We are using the EVMC6657L and TMDXEVMPCI adaptor on a linux PC. But the PC does not recognize the EVM card very well and the inbound ATU is not working.

    I heard that the combination of EVM+TMDXEVMPCI can not be enumerated on every PC, especially thoes with SSC enabled in BIOS, so I hunted a Gigabyte GA-G41MT motherboard with is declared to be OK to work with the 'delicate' EVM+TMDXEVMPCI, and simply turn the SSC functionality off. However, it is not satisfying enough either:

    (1)  With EVM powered up firstly, when PC is powered on , the LTSSM_STATE turns to 0x11 quickly, which seems to be right, but the PC would not recognize the board ('lspci' is used here) except that I keep retrying link-training after link up for several times. Emphasize that I'm retrying after a sucessful link-up.

    (2) With PC started up firstly (The evm loads slowly on SPI boot), the EVM never get linked up until I restart the PC with the RESET button on PC. And more unfortunately, C6657 on the EVM will hang after several link-training retrys (see(1)), even an on-board emulator would not help now.

    So the question is why do I have to retry link-training after a sucessful link-up ? And if I have to do this to make the PC recognize the EVM correctly, how many times should I retry? Note that I'm using the PCIe LLD example provided with mcsdk (2.1.2.6) and the PC has ubuntu 12.04 installed.

 

   And the problems did not end up here. After the linux PC recognizes the EVM successfully, I get this message with a 'lspci -vv' command:

01:00.0 Non-VGA unclassified device: Texas Instruments Device 8888 (rev 01)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Step                                                                                        ping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-                                                                                         <MAbort- >SERR+ <PERR- INTx-
        Latency: 0, Cache Line Size: 4 bytes
        Interrupt: pin A routed to IRQ 12
        Region 0: Memory at fddf8000 (32-bit, non-prefetchable) [size=16K]
        Region 1: Memory at fddff000 (32-bit, non-prefetchable) [size=4K]
        Region 2: Memory at fc000000 (32-bit, prefetchable) [size=16M]
        Region 4: Memory at <unassigned> (32-bit, prefetchable)
        Capabilities: <access denied>

    Something is strange, I configured the BARs as follows:

  {

#define PCIE_BAR0_MASK        0x00003FFF    // 16KB
#define PCIE_BAR1_MASK        0x00000FFF    // 4KB
#define PCIE_BAR2_MASK        0x00FFFFFF    // 16MB
#define PCIE_BAR3_MASK        0x00000000    // DISABLE
#define PCIE_BAR4_MASK        0x00000000    // DISABLE
#define PCIE_BAR5_MASK        0x00000000    // DISABLE


      static const uint32_t barMaskTab[6] =
      {
          PCIE_BAR0_MASK,
          PCIE_BAR1_MASK,
          PCIE_BAR2_MASK,
          PCIE_BAR3_MASK,
          PCIE_BAR4_MASK,
          PCIE_BAR5_MASK,
      };
     
      uint32_t i = 0;

      for (i = 0; i < sizeof(barMaskTab) / sizeof(barMaskTab[0]); i++)
      {
          type0Bar32bitIdx.reg.reg32 = barMaskTab[i];
          type0Bar32bitIdx.idx       = i;

          memset (&setRegs, 0, sizeof(setRegs));
          setRegs.type0Bar32bitIdx   = &type0Bar32bitIdx;

          if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
          {
              DBG("SET BAR%d MASK register failed!\n", i);
              return retVal;
          }
      }
  }

    As you can see, BAR4 is not behaving as I expected in the code. I'm ignoring this problem currently because it does not affect my work now. And I continued to configure the inbound ATU to redirect the 16kB BAR1 space to DDR with the following configuration:

CMD_STATUS = 0x0000000f
IB_BAR1 = 0x00000001
IB_START1_LO = 0xfddff000
IB_START1_HI = 0x00000000
IB_OFFSET1 = 0x80000000
CFG_STATUS_CMD = 0x40100146
DEBUG0 = 0x00008d11
SERDES_PLL_LOCK = 0x00000001

    I am quite sure the above is configured successfully because I just copied the text from a COM-debug-terminal connected to the EVM and it is printed by my program. In fact, IB_BAR1 is configured by the linux PC via BAR0, just like what the linux_boot_loader example does. However, the inbound ATU is not working as I expected either. I tried to read the memory out:

root@ubuntu:/home/dsp# ./lookat -l 4 0xfddff000
  ADDRESS  |   VALUE
-----------+-----------
0xfddff000 | 0xffffffff
0xfddff004 | 0xffffffff
0xfddff008 | 0xffffffff
0xfddff00c | 0xffffffff

    Note that 'lookat' is a small util to read/write physical memory via shared memory in linux. I tried to write the BAR1 and read the data back but failed. To verify 'lookat' is working correctly, I 'looked at ' the BAR0 memory space:

root@ubuntu:/home/dsp# ./lookat -l 4 0xfddf8000
  ADDRESS  |   VALUE
-----------+-----------
0xfddf8000 | 0x4e301101
0xfddf8004 | 0x0000000f
0xfddf8008 | 0x00000000
0xfddf800c | 0x00000000

    You can see that I read out the data in BAR0 successfully, and you can see that I enabled the IB_XLT_EN bit in CMD_STATUS register(0xfddf8004 | 0x0000000f).

    So the question is that, what else do I need to do to get the inbound ATU to work?

    Any guidance, advices or even hints would be of great help to me. Thanks!

    Best  regards.

  • Hi Hawk,

    If you connect the EVM to PC via AMC-to-PCIe (TMDXEVMPCI) adapter card means the EVM get power from PC. No need to connect external power supply to EVM.

    I think the some power sequence problem on initial power on time. So that only your EVM is not enumerated properly on without the restart. Please try to boot EVM card via PCIe boot mode (Refer \ti\mcsdk_2_01_02_06\tools\boot_loader\examples\pcie\docs\README.pdf)

    Which version of PCIe adapter card you using? There was some PC compatibility issue with old version of AMC-to-PCIe adapter card. But the issue should be fixed in the new version of adapter card, labeled with “PCB REV: 17-00107-03; PCA REV: 18-00107-03”.

    Please check the following link for the adapter info:

    http://www.ti.com/tool/tmdxevmpci#1

    Refer section 3.1.1 PCIe Outbound Address Translation Examples and 3.2.3 PCIe Inbound Address Translation Examples (examples 1,2,3,4) in document sprabk8 (http://www.ti.com/lit/sprabk8). To avoid those confusions of inbound/outbound terms, it good to understand the how PCIE read/write works and how address is translated.

    Take a look at below links:

    http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/p/279249/976090.aspx#976090

    http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/304761.aspx

    Thanks,

  • Hi Ganapathi,

        Thank you for your reply!

        First I should apologize for mixing the two problems together, and it's a little crude for me to push the second problem (ATU) here without enough test and examination. I am working on it today, and some new decoveries have been made, I'll expain it  below in details.

        I have to emphasize the only way to make to PC recognize the EVM is to connect external power supply to the EVM and let it start first, the EVM will be stuck in a while loop which is detecting the LOCK bit of PCIE_SERDES_PLL register. And till now, it's safe to power on the PC and let BIOS do the enumeration work. No DSP core hang, no device un-recognition, they just work fine.

        I am using the '-03' version of TMDXEVMPCI which is OK. However, PCIe boot is unavailable in our target system because there is no EEPROM on our custom board and thus the IBL which has been declared to have a "PCIe Workaround" for the RBL( Have it been fixed now?) is unavailable. And I am just using the EVM for debug purporse because our target system is somehow more complicated with a PCIe switch(IDT,12NT12) connected between X86 and DSPs, and I do not want to be trapped by the 'mysterious' switch chip.

        Power sequences are very important  because we are controlling the power sequence of the x86, switch, DSPs and even FPGAs by ourselves, so could you please provide some more info on the "some power sequence problem on initial power on time" ? Thanks.

        And finally I can explain the ATU problem here. I am still working on it, but a big mistake I've made is that  I  trusted the "simple" util "lookat" too much, it seems that linux should do something to the pcie driver object before using the BAR memories, I used the linux_bootloader example and it's starting to get to work now. I'll tell you if I've made any progress (today or tomorrow). 

        Regards,

        Hawk

       

  • Ganapathi,

        I am quite confused after a whole day of work. I guess that the weird behavior of the inbound ATU and the ghost BAR4 is the result of some kind of problem that happens at the link-up stage, and the behavior  is  acturally 'random'. With a lot of tries, I found that the linux PC sometimes writes BAR1 successfully, and sometimes gets 0xFFFFFFFF and then cracks down( 'sometimes' again), with exactly the same code both on PC and EVM DSP.

        Considering that the two problems I met may both be caused by bad power sequence, I downloaded the FPGA_V2.bit to the EVM and turned back to the basic IBL PCIe boot example. I must admit that the EVM works well with the linux PC using the combination of FPGA_V2 + IBL if I power up the EVM on the motherboard without any external power supply.  For consistency, I also tried the power sequences below:

    (1) If I power the EVM up first with an external power, the linux PC would not recognize the EVM (just like my code with single try of sucessful link-training).

    (2) When the external power is removed, if I hold the FULL_RESET button and power up the PC, and then release FULL_RESET after several seconds to fake the post-power-up condition, the PC would not recognize the EVM either.

    So I guess the EVM requires X86 to power up exactly "at the same time" with the C6657 DSP...Am I correct ? If so, how exactly should the time slot be ? I have to complain it's too strict for my system which boot from SPI with an image much bigger than the IBL in the EEPROM( it takes several seconds as usual).

    Power sequence is very important in my target system as I said early this morning, so I have to make it clear. Can you explain the IBL's behavior above to me in details first? I think that public demos is a good beginning for me to study on this. Thanks a lot for your pacience!

        Regards,

        Hawk

     

  • Hi Ganapathi,

        Are you still on this thread?

        I tested the effect of the power sequence with the IblPCIeWorkAround in my own test program today (SPI boot, even smaller than the IBL). And I find that there is a time slot that lasts for several seconds from the moment at which PC is powered up( the moment at which BIOS code is loaded in fact, I guess) to the moment at which BIOS is exiting and the OS is loaded. If my program is executing IblPCIeWorkAround in this time slot, the EVM is enumerated successfully. Otherwise, the EVM can not be recognized by the PC. If the EVM is powered on first by an external power supply, it would be recognized sometimes, but lots of weird things would happen, including unexpected link-down, unaccessible BARs, etc.

        Although I can control the power sequence and increase the probability that the EVM hits the timeslot, I do NOT think this is a good programming model for safety-critical systems. I am expecting two sync methods:

        (1) The EVM is powered up first, it initializes the PCIe module and wait for the PC to be powered up. In this case , it's easy for me to build a close-looped controlling logic with a hardware assert message posted to PC after the PCIe in DSP is initialized.

        (2) With some techniques which I don't know yet, re-activate the enumeration procedure in BIOS code and rescan the PCI bus at hardware level. I didn't have too much information on this idea but I accept this as a safe plan because I can rescan the bus and pick up the device from OS (not BIOS) after it is left by the BIOS code just because it is 'late'.

        In one word, the most annoying condition is that the DSP and PC is like racing on the time axis, enumeration fails if either of the two device is too slow or too fast.

        You can test the above with IBL+FPGA_V2. And I'm hoping that I can get some more information to help me solve the enumeration problem by providing  more 'loose' power sequence constraints. 

        Thank you for your help !

        Regards,

        Hawk

  • Hi Hawk,

    Could you please do below steps once to resolve this issue?

    1) Remove power cable of PC

    2) Insert EVM in PC using TMDXEVMPCI card. (Sometimes due to improper insertion PC can not detect EVM)

    3) Power on PC. Go to BIOS, wait for 1-3 seconds, then press F10(Which is key for save and exit option)

    4) Check whether the EVM is detected or not.

    Please let me know your result.

    Thanks & Regards,

    Rakesh Modi

  • Hi Rakesh,

        Thank you for your reply!

        I tried step by step exactly as what you told me to do on my PC and EVM once I got seated in my office. And the EVM is still not being detected. I also tried for a delay of longer time in BIOS, and pressing F11 instead of F10,  the result is the same.

        I'm sorry but acturally I don't understand why you what to test the above, could you describe your ideas to me and maybe I could help you do some experiments although I don't think there is anything special for my workbench(EVM+TMDXEVMPCI+GA-G41MT-S2P).

       In fact, the BIOS code is reloaded after F10 or F11 is pressed. In another word, the BIOS is not waiting for the 'slow' EVM, but soft-restarting for another enumeration cycle instead. I think it is the same as getting the EVM powered first and waiting for the PC to be powered up.

       Not important but curious, are you from einfochips? I have to say that I've tested PCIe on our target board without the EVM and with a PCIe switch connected between x86 and the DSP, and the result is just the same. I have to hit the time slot of the BIOS (no more than 3s) to make the DSP detected by x86. And to be more exact,  the time slot ends before BIOS exits, it ends when BIOS is starting to grab display devices (the LPC debug card on motherboard shows code A3).

        Regards,

        Hawk

  • Hi Hawk

    My understanding of the power up sequence is as follows:

    1. PCIe slot on PC host provides power and ref clock to PCIe module on EVM
    2. PCIe boot code on EVM initializes C66x PCIe module and ready for link up
    3. PCIe root complex in PC host is powered up and link up is established between PCIe root complex in host and PCIe end point in EVM
    4. PC host enumeration (BIOS) starts to scan the PCIe bus
    5. PCIe end point in EVM is enumerated and registered in PC host OS

    If step 4 happens between step 2 or 3, then the enumeration will fail.

    But you mentioned in previous thread“ I must admit that the EVM works well with the linux PC using the combination of FPGA_V2 + IBL if I power up the EVM on the motherboard without any external power supply”

    I do not know why you could not stick with the updated FPGA/IBL and working power sequence.

    Thanks,

  • Hi Ganapathi,

        I do NOT think step 4 will happend before step 3, because the so-called "RC" is actrually in the x86 CPU, and it is impossible to "scan the PCIe bus" without power supply to the x86 CPU. So, "step 4 happens between step 2 or 3" is an impossible case.

    Ganapathi Dhandapani said:
    1. PCIe slot on PC host provides power and ref clock to PCIe module on EVM
    2. PCIe boot code on EVM initializes C66x PCIe module and ready for link up
    3. PCIe root complex in PC host is powered up and link up is established between PCIe root complex in host and PCIe end point in EVM
    4. PC host enumeration (BIOS) starts to scan the PCIe bus
    5. PCIe end point in EVM is enumerated and registered in PC host OS

    I also mentioned in previous thread there is no EEPROM on our target board, so IBL would be unavailable, I can not "stick with" the FPGA+IBL because I have to boot from SPI flash. 

    Ganapathi Dhandapani said:

    But you mentioned in previous thread“ I must admit that the EVM works well with the linux PC using the combination of FPGA_V2 + IBL if I power up the EVM on the motherboard without any external power supply”

    I do not know why you could not stick with the updated FPGA/IBL and working power sequence.

     

        And by the way, are you meaning that the "FPGA_V2" is the updated version of the FPGA code on EVM? I get FPGA_V3 on my EVM when I buy the EVM board and I get the FPGA_V2 from einfochips. Note that FPGA_V03 is the "production version" of FPGA firmware on EVM, however, it would not redirect from RBL to IBL and thus would fail in PCIe enumeration.

        Please have a look at the following link:

        http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/322907.aspx

        Regards,

        Hawk