This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

DS125BR401: PCIe Link training issues

Part Number: DS125BR401

Tool/software:

Dear TI-Team,

as a follow up to the initial thread we currently see a not well understood behaivior in the link training phase and the resulting PCIe link status.

Description of the setup:

Intel Xeon CPU (Rootcomplex) --> Slot --> Custom PCIe PCB with 2x DS125BR401 for x8 gen. 2 link and iPass connector --> 50 cm iPass cable --> Custom PCB with Stratix 10 FPGA

TX-Caps are placed accordingly.

Description of the behavior:

Once the link is trained for x8 gen. 2 it stays at these parameters. The tricky part is link training.

The custom PCIe PCB with 2x DS125BR401 was added to the setup as we assumed signal integrity issues due to the long cable link and the resulting eye being too closed. The initial tuning of INEQ and OUTEQ of the DS125BR401 was done according to the Datasheet with relatively high levels. First successful link training was achived with all INEQ and OUTEQ on level 1 in both directions. The mentioned 20K resistor on the "MODE" pin was not placed and the pin was left floating. This setup is some sorf of a baseline for further tests. This setup did perform well in some statistical analysys in different setups but not achieve 100 % gen. 2 (sometimes fallback to gen. 1 after link training, always x8).

Our next step was changing the levels of INEQ and OUTEQ to improve the eyediagram. Results show a pretty open eye:

Upstream direction towards the CPU

Downstream direction towards FPGA

The INEQ levels are both 1 for upstream and downstream direction. The OUTEQ level for the upstream is 4, for downstream it is level 8.

Now comes the caveat: The link training fails more often resulting in a gen 1 link with the "better" eye levels set at the redriver as with the level 1 tunning where we get a 1:8 failure rate. With the "good" tuning we almost never train a gen. 2 link.

As mentioned in the other thread,we placed a 20 K resistor to GND at the MODE pin switching it to "Gen 3 - no de-emphasis". This made gen 2 link training even possible and we are currently generating statistics but so far there are rare cases of gen 1 link training and high rates of gen 2.

Now we need to understand what is happening:

  • Why is link training failing more often with "better" eyes?
  • What is the mode pin exactly doing and why does it improve link training dramatically for us?
  • Why is the mode "Gen 1, Gen 2" not improving even though that is the desired datarate?

Thanks

Marvin

  • Hi Marvin,

    I don't believe Gen1/2 mode and Gen3 mode are very fitting names because the redriver does not actually care about the data rate being transmitted, regardless of the mode.

    Tying 1k to GND places the redriver in limiting mode. In limiting mode, the redriver applies de-emphasis on the transmitter side to help with Tx equalization. However, de-emphasis prevents input-to-output waveform linearity.

    Tying 20k to GND places the redriver in linear mode. De-emphasis is disabled and the input-to-output waveform linearity is better maintained. Linearity is important to preserve training signals and ensure they are properly received by the RC and the EP. This is why tying 20k to GND greatly improves link training.

    I have a few ideas you can try to prevent Gen1 link training cases.

    1. I believe you are currently using auto RX-Detect mode (RXDET pin is floating). Can you try using manual RX-Detect mode, where input termination is fixed at 50 ohm? In case the RX-Detect circuitry is incorrectly not recognizing the RC and EP receivers, this will prevent the redriver Rx termination from remaining high impedance. There are 2 ways you can select manual RX-Detect mode:
      1. Tie 1k to VDD on the RXDET pin.
      2. If you are using SMBus mode, you can write the following register bits:
        • 0x08[3]=1
        • 0x0E[3:2]=11
        • 0x15[3:2]=11
        • 0x1C[3:2]=11
        • 0x23[3:2]=11
        • 0x2B[3:2]=11
        • 0x32[3:2]=11
        • 0x39[3:2]=11
        • 0x40[3:2]=11
    2. Have you tuned EQ settings on a channel-by-channel basis? It may be possible that a different EQ setting is necessary on each channel to yield optimal eye opening. If you haven't already, I recommend tuning each channel EQ individually.

    Best,

    Lucas

  • Hi Lucas, thank you for your reply,

    we do not use SMBus mode.

    1. I have tested tieing the RX-Detect Mode Pin to VDD with no success.
    2. No we have not tuned the EQ settings channel-by-channel. But we have done the eye diagram measurements on every single channel to check if they all behave the same (wich they do). We even tried the different EQ settings on every channel and they did not differ.

    Can you clarify the questions I have asked in my post above?

    Thank you very much

    Marvin

  • Hi Marvin,

    Thank you for trying to use manual RX-Detect mode. I understand this did not resolve the issue and that you are using pin mode.

    Regarding question 1, it's unclear to me at this time why link training is failing more often with better eyes. However my understanding is that tying 20k to GND on the MODE pin greatly improved the success rate of link training bringing the link to Gen2. Therefore you have the best Gen2 link training success rate with optimized EQ settings and linear (Gen3) mode selection. Is this correct?

    I wanted to confirm the RX-Detect circuitry is functioning properly and that your EQ settings are optimized for every channel. I suspected improvements in these areas could improve link training to a 100% Gen2 success rate. This is why I made the recommendations that I did. Since these did not resolve the issue, I'd like to examine all of your redriver settings to see if any improvements can be made. Can you share schematics for both units DS125BR401 for review?

    I believe I already answered questions 2 and 3 in my previous reply. Please let me know if you still have questions about limiting versus linear mode.

    Best,

    Lucas

  • Hi Lucas,

    yes it is correct that it did not resolve the issue.

    We have seen the following cases regarding the 20k:

    • All levels on 1 shows Gen.1 linktrainings on some systems sporadically
    • All levels on the best eye diagram settings shows almost 100 % Gen.1 linktraining on all systems
    • All levels on the best eye diagram settings AND 20k to GND on MODE pin greatly improves Gen. 2 success rate on some systems and shows no improvement on other systems

    It completely baffles me, that some combinations run very well, some combinations get better/worse. As if there is some sort of statistical process that rolls the dice while linktraining.

    I have an additional question regarding the De-emphasis settings of the rootcomplex and the endpoint. are both -3.5 and -6 db supported by the redriver?

    I can not share the schematics in this public space. I hope we can find another way for examination.

    Thanks,

    Marvin

  • Hi Marvin,

    I'm surprised to here that the combination of tuned EQ levels and linear mode (20k to GND) greatly improves Gen2 success rate in some systems, but shows no improvement in other systems. Are there any differences between systems with improvement and without improvement?

    I have an additional question regarding the De-emphasis settings of the rootcomplex and the endpoint. are both -3.5 and -6 db supported by the redriver?

    I believe -3.5 dB and -6 dB are the fixed de-emphasis levels which can be selected at the RC or the EP by Gen2 link training. Is your question if the redriver is able to pass through these de-emphasis levels between the RC and the EP? The answer is yes, especially when the redriver is placed in linear mode (20k to GND) and input-to-output waveform linearity is better maintained. If the redriver is placed in limiting mode, the device linearity isn't as good and these de-emphasis settings may not pass through as transparently.

    If you're asking if the redriver is capable of applying -3.5 dB or -6 dB of de-emphasis, the answer is yes when using limiting mode. However, using de-emphasis on the redriver disrupts the device linearity and isn't recommended when link training pass through is needed. This is why I am recommending that you place the redriver in linear mode (20k to GND) and do not use de-emphasis on the redriver.

    Note that the redriver does not actively participate in link training, it transparently passes training signals between the RC and EP. This is why device linearity is important.

    I understand that you do not want to share schematics on this public forum. I sent you a friend request on E2E. After you accept, you can share your schematics with me via private message, and I'll share my feedback privately as well. Does this work for you?

    Best,

    Lucas

  • Hi Lucas,

    we have not found any differences between the systems. Our guess is that they all have a certain tolerance range (probably in PCB impedance?).

    Best regards,

    Marvin

  • Hi Marvin,

    I understand. Allow me to review your schematic and share feedback privately.

    Best,

    Lucas