AM6412: How to configure PCIe clock outputs

Hello Kouji Nishigata

Thank you for the query.

We have the below registers for PCIe

12.2.2.5 PCIe Subsystem Registers
This section describes the PCIe Subsystem registers.
12.2.2.5.1 PCIE Registers

SPRUIM2G – MAY 2020 – REVISED JULY 2023

Could you please elaborate your requirement.

Regards,

Sreenivasa

Hi, Sreenivasa

Thanks for your response.

I am having a problem with their custom boards where PCIe devices are sometimes not recognised immediately after boot. We are gathering information on this issue, with the intention of setting up a separate tree.
The customer's investigation on this matter shows that the timing of the PCIe clock start is different between the TMDS64GPEVM and the custom board, and they are trying to see if they can make the timing similar in the custom board settings.
They say they cannot find any documentation or sources on how to configure PCIe or SERDES settings for this.
I will contact the customer about the TRM description, but I assume they are looking, is there any information for the Clock output?
Also, TRM has been updated to SPRUIM2H in the last week or so. There does not appear to be any changes as far as this time, but please confirm.

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

Thank you.

Would customer be able to share some additional information on the customer board to review or compare with the EVM?

Is there any difference in the circuit implementation.

Based on your explanation, i assume customer is using the SoC serdes clock output.

Can you check with customer if they have looked at the errata related to the Serdes clock output.

Regards,

Sreenivasa

こんにちは、スリーニバサ

顧客は正誤表を認識しています。ただし、現在のシリコン エディションでこれが解決されているかどうかについては質問されています。
また、カスタム ボードの SERDES_REFCLK0 ライン上の直列のダンピング抵抗 (R296、R297) を 0Ω から 33Ω に変更した後、PCIe デバイスの非検出現象は発生しなくなります (添付の回路図の右上を参照)。

2425.schematic.pdf
以前は10回程度でこの現象が発生していましたが、すでに1000回程度発生していないので、対策が講じられているのは確かかと思われます。
しかし、顧客は、出力ピンのダンピング抵抗を変更すると現象が変わることに納得していません。
クロックラインの反射が何らかの問題を引き起こした可能性がありますが、お客様を納得させる理由を提供していただけますか?

よろしくお願いします、

西潟宏治

Hello Kouji Nishigata

Thank you for the inputs and good to hear that the customer has been able to resolve the issue.

user1201164 said:

It is possible that the clock line reflections may have caused some problems, but can you provide any reason to convince the customer?

The Signal integrity related performance has layout dependency. Adding series resistors helps resolve some of these issues.

I will check with the PCIe expert if he has some additional thoughts.

Regards,

Sreenivasa

Hello Kouji Nishigata

Please refer inputs i received from the PCIe expert

There is nothing board specific in the “clock on” setting as far as I am aware of. I’m looping in Bin for his comment. 

My suggestion would be that they first ensure that all the PCIe power sequencing specs are being met. I’ve highlighted the three parameters I think most pertinent to this discussion.

 Regards,

Sreenivasa

Hello Kouji Nishigata

Pls refer below additional inputs i received:

It’s certainly possible that these R’s are dampening reflection caused by the connector or the add-in card itself. I’d call it probable. It’s also possible that they are helping to mitigate an impedance discontinuity imposed by the traces themselves, but  I think this is unlikely in the absence of a board fab error as the schematic does state that these traces should be impedance-controlled @ 95 Ohms.

Suggestions 

If 33 Ohms was just the first value customer tried, it would be recommended to try lower values to “tune” the solution.

Regards,

Sreenivasa

Hi, Sreenivasa

Thanks for your response.

Unfortunately, the phenomenon has been reproduced.
When they tested with Micron SSDs, the PCIe device is not recognised about 1 in 20 times.
We have not yet heard the results of the earlier case of changing the resistor value, so the circuitry is still the same as before.
At this time, an error is reported in the U-Boot boot log.

[    3.073976] phy phy-f000000.serdes.2: phy poweron failed --> -110
[    3.080101] j721e-pcie-host f102000.pcie: Failed to init phy
[    3.085821] j721e-pcie-host: probe of f102000.pcie failed with error -110

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

user1201164 said:

We have not yet heard the results of the earlier case of changing the resistor value, so the circuitry is still the same as before.

Does this mean series resistor is not added.

Can you please check if the require control impedance related routing guidelines have been followed on the PCB.

Regards,

Sreenivasa

Hello Kouji Nishigata

An additional question

Is customer seeing change in performance based on the add-on card?

Regards,

Sreenivasa

Hi, Sreenivasa

Thanks for your response.

You were vague about the current circuit configuration.
The board on which the phenomenon recurred is the one with the series damping resistor on the ERDES_REFCLK0 line set to 33ohm.
The SSD they were using when they changed the resistor stopped the phenomenon, but when they tested it with another manufacturer's SSD, the phenomenon reappeared.
They have not yet tried changing the resistance.

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

user1201164 said:

You were vague about the current circuit configuration.

Thank you for the inputs.

user1201164 said:

The SSD they were using when they changed the resistor stopped the phenomenon, but when they tested it with another manufacturer's SSD, the phenomenon reappeared.

Noted.

user1201164 said:

They have not yet tried changing the resistance.

Please update me when customer has. results.

Additional queries

Can you please check if the required control impedance related routing guidelines have been followed on the PCB.

Regards,

Sreenivasa

Hi,.Sreenivasa

I had the customer test the resistance on the signal by changing it to 10ohm and they did not see any change.
They think that although Signal Integrity is being questioned in this thread, they are not.
On the basis of,
1.
The phenomenon is not reproduced when the system reboot, meaning warm reset, is repeated after the PCIe device is successfully recognized from Cold start.
If there is a problem with the SI, there is no difference in operation between cold start and warm reset, and we believe that the same random non-detection may occur.
2.
After the PCIe device is successfully recognized from cold start, even when the PCIe device is operated continuously for 72 hours, no errors or other problems occur on the PCIe bus and the device is still operating normally.
Furthermore, the PCIe device is operating normally even after temperature test (-20C~60C) under this condition. If there is a signal integrity problem, PCIe bus error or CPU hung-up, etc. should occur.

so they think it may not be an SI problem but in the process of PCIe device detection from Cold start.

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

user1201164 said:

1.
The phenomenon is not reproduced when the system reboot, meaning warm reset, is repeated after the PCIe device is successfully recognized from Cold start.
If there is a problem with the SI, there is no difference in operation between cold start and warm reset, and we believe that the same random non-detection may occur.

Thank you.

Let me try to understand the failure phenomenon.

The failure is observed when the power is cycled with the PCIE device mounted.

No failure is observed with repeated warm reset. 

Does this issue happen on multiple board or is this a specific board. I am not sure if you answered this before. 

Regards,

Sreenivasa

Hi,Sreenivasa

The direct problem is that sometimes PCIe devices are not detected during cold start. The frequency is about 1 in 20 times. They have two custom boards and both have the same condition.
They changed the series damping resistor on the SERDES_REFCLK0 line from 0ohm to 33ohm. They set the resistor to 10ohm this time, but the phenomenon is still the same.
After the PCIe device was detected at cold start, they did not see any abnormality in continuous operation, and they could detect the PCIe device even after repeated warm reset.
I have already uploaded the schematic and the logs of the errors at startup in the thread, but I will upload them again if you can't find them.

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

user1201164 said:

The direct problem is that sometimes PCIe devices are not detected during cold start. The frequency is about 1 in 20 times. They have two custom boards and both have the same condition.

Is this issue seen when the whole system including AM64x board and the PCIe device is powered off and powered-up - cold start?

Is the issue observed on a specific PCIE device or any PCIe device customer uses.

user1201164 said:

I have already uploaded the schematic and the logs of the errors at startup in the thread, but I will upload them again if you can't find them.

i have access to the PCIe interface schematics page that you shared.

Regards,

Sreenivasa

Hi,Sreenivasa

Yes, the customer is testing the power off and on. Sometimes it happens that PCIe devices are not detected at this time.
Initially, it seemed to occur with all of their available devices, but after changing the resistor values, the undetected error does not seem to occur with some devices. However, on the devices where it does occur, it occurs about once every 20 times.

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

user1201164 said:

Yes, the customer is testing the power off and on. Sometimes it happens that PCIe devices are not detected at this time.

This seems to be a system level issue. 

I am not sure on the overall implementation.

I assume the PCie device is powered by the AM64x board. 

Could you review the reset implementation as below 

The PCIe reset remains asserted till the SoC GPIO releases the reset.

Regards,

Sreenivasa

Hi,Sreenivasa

I do not know what this confirmation means.

If you are concerned about the timing between the reset release and the clock, we have heard the following from our customers.
"Since it follows the PCIe standard, the clock should be output at least 100us before the reset release. Our board is clocked 226us before the reset release, so it meets the standard."
Also,
The PCIe reset (PERST_N) is controlled by GPIOs as on the evaluation board, and the timing waveform image from the start of PCIe clock (REFCLK0) output to reset release is attached (REFCLK0_to_PERST_N.png). From clock output to reset release, there is 214.88us in the attached figure, which is more than 100us, so it meets PCIe standard."

Let me know if you need confirmation for some other intention.

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

Thank you for the inputs and my question was around the reset timing.

Would it be possible for customer to increase the time by another 50-100 us and do a check.

Thios is based on the input that the issue is observed during power-up and not happening always.

Regards,

Sreenivasa

Hi,Sreenivasa

I can ask the customer to do this, but would a long screenshot after the reset be ok?
Also, do you need a shot of when the phenomenon occurred?
What is to be confirmed? Could you please be more specific about what the customer will be asked to work on?

Best Regards,

Hello Kouji Nishigata

Here is the issue observed

However, on the devices where it does occur, it occurs about once every 20 times.

user1201164 said:

What is to be confirmed? Could you please be more specific about what the customer will be asked to work on?

I am not sure if there is a way to be specific since this is a system level issue and would need step by step analysis. We can offer limited support due to familiarity of the system implementation and the use case.

Possible causes could be software related or hardware related including power, reset sequences or any other issue.

user1201164 said:

I can ask the customer to do this, but would a long screenshot after the reset be ok?

Can you please elaborate your inputs.

user1201164 said:

Also, do you need a shot of when the phenomenon occurred?

Yes this helps but the question would be what shot are required for analysis and can be provided - reset, clock, power , signals ...

Since this is a power-up issue i was asking about reset being on of the possible issue.

Regards,

Sreenivasa

Hi,Sreenivasa

We received three diagrams from the customer.
The 1st one is when the REFCLK0 - PERST_N delay is extended to 616us. In this case, the phenomenon (undetected PCIe device) still occurs.

The 2nd picture shows the timing waveform of PCIe Power 3.3V - PERST_N. The delay time is long enough to meet the PCIe standard of 100ms or more.

The 3rd waveform is the waveform of REFCLK0 - PERST_N when PCIe device non-detection occurs. However, the Reset time is 200us because it was before the delay time was extended.

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

Thank you for the inputs.

user1201164 said:

We received three diagrams from the customer.
The 1st one is when the REFCLK0 - PERST_N delay is extended to 616us. In this case, the phenomenon (undetected PCIe device) still occurs.

Is the frequency of occurrence same. 

Without the additional delay, is there a change is reset timing or clock behavior or power supply when the PCIE is detected vs not detected.

Regards,

Sreenivasa

Hi,Sreenivasa

We received a response from a customer.
When they extended the waiting time for the reset, it did not make any difference in the occurrences of the phenomenon.
Also, there is no change in reset timing, clock operation, or power supply when PCIe is detected or not detected.
(Never mind that the 3rd figure only intentionally sets the reset delay to 200us)

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

Thank you.

Can you pls check if customer tried to reduce the 47 uF + 47 uF buclk cap to 47 uF and did a test on the VCC_3V3_MINIPCIE.

Regards,

Sreenivasa

Hii, Sreenivasa

Do I ask the customer if they have removed and tested C313 in the diagram?
Or do you want them to remove it (if they have not done so) and perform the test?

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

Thank you.

user1201164 said:

Do I ask the customer if they have removed and tested C313 in the diagram?

Yes please.

user1201164 said:

Or do you want them to remove it (if they have not done so) and perform the test?

If the above test is not done, we would want customer to do a test.

Regards,

Sreenivasa

Hi, Sreenivasa

The 47uF capacitor C313 was removed and tested with no change.
No difference in PCIe power, reset, clock and frequency of non-detected events.

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

Thank you.

After the device is not detected, help me understand how this is recovered - is it by only power-off.

Has customer tested the same functionality on the EVM ?

With the available information, i cant think of any other solution for now.

Regards,

Sreenivasa

Hi, Sreenivasa

After the PCIe device failure occurred, they said it could be fixed by simply turning the power off and on. They have never seen the phenomenon occur in a row, although they also think it is a stochastic thing.

The customer tested with the EVM and did not see the phenomenon.
However, the EVM is externally PCIe clocked as an errata measure. On the other hand, the customer's board uses the PCIe clock output of the CPU. The errata should be fixed in the current silicon release, so I don't think there is a problem, but I am a little concerned.

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

Thank you.

user1201164 said:

After the PCIe device failure occurred, they said it could be fixed by simply turning the power off and on. They have never seen the phenomenon occur in a row, although they also think it is a stochastic thing.

Has customer tried to recover by resetting the PCIe device and the AM64x SERDES0 module without powering off the complete board.

Regards,

Sreenivasa

Hello Kouji Nishigata

Could you please confirm the required filters, bulk caps and decaps are provided for the serdes0 supplies on the customer board.

Customer could compare the EVM for the values.

Regards,

Sreenivasa

Hi, Sreenivasa

The customer turns the power off and on to the entire board.
And I don't think they tried to reset only the PCIe devices and SERDES0, leaving the power to the board itself intact.
In this case, should I have them try this?

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

user1201164 said:

And I don't think they tried to reset only the PCIe devices and SERDES0, leaving the power to the board itself intact.
In this case, should I have them try this?

Yes please.

Also please check the filtering query i had in the above thread.

Regards,

Sreenivasa

Hi, Sreenivasa

> Could you please confirm the required filters, bulk caps and decaps are provided for the serdes0 supplies on the customer board.

The serdes0 and CPU power supplies on their board are basically the same circuit and components as the 64GPEVM.
And they have added the necessary filters to the VDDA_0P85_SERDES0 and VDDA_1P8_SERDES0 power pins.
I attach a schematic of the surrounding area just in case.

> Has customer tried to recover by resetting the PCIe device and the AM64x SERDES0 module without powering off the complete board.

The customer has tested the following patterns.

(1) With PCIe device not detected, the customer rebooted the system after powering off->on and resetting the PCIe module only. As a result, the PCIe device non-detection status remains unchanged.
(This result suggests that the problem is not on the PCIe device side.)

(2) Reboot the system with PCIe device not detected. As a result, the status of PCIe device not detected remains unchanged.
(This result suggests that the PCIe host controller is in some kind of abnormal state.)

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

Thank you.

Let me review the inputs and comeback.

Regards,

Sreenivasa

Hello Kouji Nishigata

I checked with another expert and received the below patch.

https://patchwork.kernel.org/project/linux-pci/patch/20230707095119.447952-1-a-verma1@ti.com/

Is this something customer could try?

Regards,

Sreenivasa

Hi, Sreenivasa

Thanks for the responce.

Does this mean that the REFCLK0 - PERST_N delay is extended beyond 100msec?
For "PCIe Power 3.3V - PERST_N" the town has already included more than 100msec.
The phenomenon still occurs.

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

Thank you.

user1201164 said:

Does this mean that the REFCLK0 - PERST_N delay is extended beyond 100msec?
For "PCIe Power 3.3V - PERST_N" the town has already included more than 100msec.

Not sure i understand your inputs. Did you say customer already delayed reset (and the clock output from SoC) by 100 ms and tested.

Dis customer try the patch already?

Regards,

Sreenivasa

Hi, Sreenivasa

I am looking at the patch you gave us and it reads PCIe Power 3.3V - PERST_N delay to 100msec or more.
This was already implemented by the customer.

There does not seem to be any specific mention of the REFCLK0 - PERST_N delay, but I assume you mean to make this more than 100msec? Since it is now 200 -> 600usec, should I ask the customer to make this change?

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

Thank you.

Yes please.

Regards,

Sreenivasa

Hi, Sreenivasa

The customer had the REFCLK0 - PERST_N delay changed to 105.4 ms, but the situation has not changed and does not appear to have changed in terms of playback frequency.

What concerns me about this issue is the i2236 errata. The current silicon revision has been addressed, but the EVM is externally powered. Is there an example of using the SERDES reference clock as a PCIe reference clock as our customer does?

Best Regards,

Kouji Nishigata

Hello Kouji Nishigata

Thank you for the inputs.

Let me check internally and update you.

Regards,

Sreenivasa