DP83869HM: DP83869HM SGMII inconsistent auto negotiation

Hi Magnus,

We have a SGMII troubleshooting guide that compiles our steps for addressing SGMII issues. Can you read through this document and see if it helps the problem?

Best,

Shane

Thank you for your reply. Yes, we have seen and gone through this trouble shooting guide already. Unfortunately there is no explanation about our issue in this guide as we see it. 

Hi Magnus,

Here are a few ideas I have for looking into this:

1. Have you tried disabling SGMII auto negotiation on both the PHY and FPGA? Does this fix the issue?

2. Have you done any loopback testing as described in section 1.4 of the troubleshooting guide?

3. Does restarting SGMII auto-negotiation via 0xC00[9] have any effect?

4. Is 0x37[1] set in the working and failing cases?

5. Is the issue present on all of your boards or do certain boards show this behavior?

6. Ensure the PHY reference clock is within the datasheet specs, particularly for ppm.

Best,

Shane

Hi Magnus,

1. The register write to disable SGMII auto negotiation is 0x14[7]. If you are already linked up a reset may be necessary.

2. There is no functional difference between auto negotiation restarted manually or through a link up event.

I am still curious if 0x37[1] is set in the PHY in failing cases vs working cases. Our app note suggests this bit reflects the MAC's acknowledgement of SGMII control information.

Best,

Shane

1. 0x14[7] = 0 Is what we have tried but did not result in a working link. Which of the various resets would you advice to utilize after such an operation? We have tried 0x1F[14]=1 - does that register bit need to be cleared after writing?

2. SGMII_AUTO_NEG_STATUS Register (0x37) has the following values:
* Good Case: 0x37 = 0x3 -> 0x37[0] = 1, 0x37[1] = 1
* Failure Case: 0x37 = 0x2 -> 0x37[0] = 0, 0x37[1] = 1
So it would seem the MAC/FPGA is indeed responding with an acknowledge even in the bad case, please advice.

We have sampled the SGMII auto-negotiation Clause 37 state machine of the MAC from "AN_ENABLE" to "LINK_OK", for both successful and failed scenarios. What we find interesting is the following:

1. Every copper Link Up event results in 2 back to back SGMII autonegotiation sequences. Is this expected?
2. In the success case, AN A.2 and A.3 have different behaviour. Could this be a clue?
3. In the failure case, the state machine appears to be stuck after B.2 and doesn't proceed.
   
   

Please take a look at the attached file in combination with our other response regarding the SGMII_AUTO_NEG_STATUS values.

File: 

AN troubleshooting.pdf

Hi Magnus,

Thank you for the details, to answer your questions:

1. I'd try a software reset in this case (0x001F = 4000). Doing a hardware reset would revert the register value so it would re-enable autonegotiation.

2. Its interesting that the PHY shows an acknowledgement when there seems to be no acknowledgement on the MAC side in scenario B.2. In this scenario it seems the negotiation breaks down in the AN_RESTART stage. What value do you see in 0xC01[2] in both working and failing cases? I believe this would be the SGMII link status on the PHY.

• Furthermore, does your MAC show what causes the link to stop at the AN_RESTART stage? From what I can tell the only requirement to exit this stage is for the link_timer to finish

3. The two SGMII autoneg sequences (A.2/A.3) look identical. I believe you do not need to negotiate twice, although I don't see an issue with doing so if the link is brought up.

4. The difference I see between A.1 and A.2/A.3 is that the Link_ok begins in a low state and AN_EN is slightly longer (perhaps due to power up circuitry).

Best,

Shane

Thank you for your feedback!

1. This is what we have tried already disabling both MAC and PHY, but resulted in a non working link. However we have now tried disabling AN only at the PHY 0x14[7] and then performing a reset 0x1F=0x4000. We see that the PHY still performs / responds to SGMII AN. Does this really turn off AN?

2. The FX_STS bit STTS_LINK_STATUS bit seems to be asserted in booth cases indicating link up. 

Hi Magnus,

1. Yes this bit disables SGMII auto negotiation on the PHY.

2. Here are my thoughts on your other questions.

• I suspect the double SGMII AN may be caused by a renegotiation request from either the MAC or PHY. From what I see in Section 37.2.4.1 in the spec, renegotiation can be requested from either side. This looks to correspond to a 'mr_restart_an' variable in the state machine.
• The expectation is for the PHY to send updated control information to the MAC and to have the MAC acknowledge.
• From your waveforms of the state machine, is seems the link_timer_done flag is not getting set. In the clause 37 spec, link_timer_done must be set in order to transition from AN_RESTART to ABILITY_DETECT. The SGMII standard reduces the link_timer from 10ms to 1.6ms, which is clear in your A.2/A.3 cases. In your B.2 case the RESTART state lasts long after the 1.6ms link_timer should have completed.
• Looking through E2E and our released documents I do not see a way to determine if the PHY is requesting the AN renegotiation. If there is a way to track this on the MAC side, it would give us more insight. 

Typically when the SGMII data path is routed correctly (loopback shows no errors) auto negotiation is not an issue. Whether this is a PHY or a MAC discrepancy is unclear so far, but I do have a few more questions:

• You mentioned seeing this on all boards but with varying degree. What is the variation among these boards? Do some boards show this less than others?
  • If you swap PHYs between boards, does the behavior track to the PHY or the board it is connected to? 
• Are all boards tested with one FPGA, or is there a dependency on the FPGA used?
• How often does the problem occur? (1/10 links, 1/100,...)
• Are there any dependencies on ethernet cable length or link partner? If you swap cables or link partners does the behavior change?
• Are you able to share the layout of the SGMII traces for review?

Best,

Shane

1.We seem to have an issue then because as we have noted, the PHY does NOT actually disable AN when doing this register write. The note on forced speed mode, can that or something else affect the behavior of the PHY according to CFG2 SGMII_AUTONEG_EN?

2. The expectation is then that there should be exclusively "1" SGMII AN event triggered by the PHY.
* Could the fact that we have two help explain the problem? Perhaps one of the devices is requesting a re-negotiation because the process is not carried out in a fully compatible way. In the successful cases we see that the first re-negotiation (A2) differs from A3 slightly even though both result in LINK_OK from the MAC perspective.
* You mentioned the 1.6ms link timer; the PHY has a default timer of 2us while the MAC has the standard 1.6ms. How does that play a role here? We tried extending the MAC timer to 10ms and we saw a different behavior in this process but still error prone.
* We have tried changing the SGMII link timer on the PHY and not seen much effect, does a reset need to be performed after changing this value? Or is it sufficient to change it before the next re-negotiation?
* Unfortunately, we cannot track more than we have already on the MAC.

3. Regarding the SGMII data path:
* The problem occurs with varying degree depending on the board, however this is a "qualitative" assessment because the behavior is not consistent. The error rate can be anything between 1/10 to 1/1000 or potentially even lower so swapping PHYs is perhaps not so helpful.
* All boards utilize exactly the same type of FPGA and PHY.
* Changing copper link partner and cable or cable length does not seem to have an effect.
* As we have mentioned before, whenever the SGMII link is established, we have full-duplex 1G data flowing between the MAC and a link partner with up to jumbo sized ethernet frames completely error-free. We think this is a more relevant test than any loopback for our current situation. Is it your opinion that there might still be an issue in the SGMII layout or traces despite this?

Regards,
Magnus

1.We seem to have an issue then because as we have noted, the PHY does NOT actually disable AN when doing this register write. The note on forced speed mode, can that or something else affect the behavior of the PHY according to CFG2 SGMII_AUTONEG_EN?

2. The expectation is then that there should be exclusively "1" SGMII AN event triggered by the PHY.
* Could the fact that we have two help explain the problem? Perhaps one of the devices is requesting a re-negotiation because the process is not carried out in a fully compatible way. In the successful cases we see that the first re-negotiation (A2) differs from A3 slightly even though both result in LINK_OK from the MAC perspective.
* You mentioned the 1.6ms link timer; the PHY has a default timer of 2us while the MAC has the standard 1.6ms. How does that play a role here? We tried extending the MAC timer to 10ms and we saw a different behavior in this process but still error prone.
* We have tried changing the SGMII link timer on the PHY and not seen much effect, does a reset need to be performed after changing this value? Or is it sufficient to change it before the next re-negotiation?
* Unfortunately, we cannot track more than we have already on the MAC.

3. Regarding the SGMII data path:
* The problem occurs with varying degree depending on the board, however this is a "qualitative" assessment because the behavior is not consistent. The error rate can be anything between 1/10 to 1/1000 or potentially even lower so swapping PHYs is perhaps not so helpful.
* All boards utilize exactly the same type of FPGA and PHY.
* Changing copper link partner and cable or cable length does not seem to have an effect.
* As we have mentioned before, whenever the SGMII link is established, we have full-duplex 1G data flowing between the MAC and a link partner with up to jumbo sized ethernet frames completely error-free. We think this is a more relevant test than any loopback for our current situation. Is it your opinion that there might still be an issue in the SGMII layout or traces despite this?

Regards,
Magnus

Hi Magnus,

1. The SGMII speed should be forced to the PHY's resolved MDI speed when auto negotiation is disabled. When auto negotiation is disabled on the PHY, it should also be disabled on the MAC.

2. These are good questions, but it is difficult to say whether the behaviors you notice are causing this problem. For instance there is a double auto negotiation in the working condition and this is able to transmit frames without issues. If so, then the double auto negotiation (at a higher level) is the expected behavior in your system correct? From the PHY standpoint this behavior looks ok.

• The same applies for the auto-negotiation timer. Since changing the PHY autoneg timer seems to have no effect I do not believe this is an issue with the PHY's timer. What is the behavior difference you see when changing this on the MAC side?
• Changing the link timer on the PHY should only require a new link up on the MDI, not a reset. 

3. So on a single board you may see the issue from 1/10 to 1/1000 link ups correct? Or do some boards show 1/10 and some boards show 1/1000? If the second case is true, it would help to see an ABA swap of the PHY between two boards to see whether the performance follows the PHY or the board. 

• I agree that passing real data error free is evidence that your layout is good, but it never hurts to have a second look . Layout reviews are common practice on E2E so If you would like me to take a look I can do that.

One question I do have: Does restarting the SGMII autoneg process via the 0xC00 register fix the issue when it occurs? I believe this would be equivalent to a new link up on the SGMII side.

Best,

Shane

1. What is the difference between 0x14[7] SGMII_AN_EN and 0xC00[12] CTRL_ANEG_EN? How do these relate to each other? As we mentioned, disabling 0x14[7] does not stop the AN process for us everything still works. Do both need to be disabled?
2. We do not know if this is expected behaviour, but at least we can conclude that it is not standard and a double negotiation should not be needed.Changing the timer on the MAC indeed affects the time axis of the state machine tracking but the observable behaviour is still the same.
3. No the same board can vary between all those values.
4. Once the issue occurs, writing 0xC00[1] does not resolve the issue, link stays down. We tried the register write in a link-established case and saw that it did indeed restart SGMII AN just to make sure. Interestingly, unplugging the copper cable immediately caused the SGMII link to re-negotiate successfully, so the two processes are perhaps not identical somehow.

Regards,

Magnus

Hi Magnus,

1. 0x14[7] is the SGMII autonegotiation control in SGMII to copper applications. 0xC00[12] is for bridge modes such as SGMII to RGMII or RGMII to SGMII. You only need to disable 0x14[7] to disable auto negotiation in SGMII to Copper mode, but auto negotiation should be disabled on the MAC as well.

4. Just to confirm, you're writing 0xC00[9] to '1' correct? 0xC00[1] is reserved so I'm unsure what this would do to the autoneg process.

I can try to retrofit one of our DP83869 EVMs to see if I can get SGMII to link up with another EVM's SGMII interface (Such as DP83867-S-EVM). This would be a PHY <--> PHY back to back SGMII connection and should at least validate the DP83869 SGMII auto negotiation process is ok. It will take time to order the SMA connectors to populate on the EVM.

• Without a second PHY or MAC type to correlate with it is difficult to discern whether the MAC or PHY is causing this condition. Doing a back to back test with the EVM is one way I can think of to test with a different PHY platform.

Best,

Shane

1. OK, just want to note that 0xC00[12] is set per default when we read it.
2. You are correct, that was a typo, I meant 0xC00[9]. I understand that the SGMII AN needs to be disabled on both devices, but when we did that we did not have a functional link. We would like to achieve a working state without AN over SGMII.
   1. For debugging it would be interesting to know what should happen if we only disable SGMII AN on the PHY and not the MAC via 0x14[7]? What we observe is that after doing this and resetting, AN from the MAC perspective still completes and reaches LINK_OK. We find this strange as it should be OFF but somehow the MAC can complete AN.
   2. What is the correct way to check SGMII link status on the PHY is AN is disabled? 

Regards,

Magnus

Hi Magnus,

I'd expect that when the PHY has autoneg disabled it will not wait for any acknowledgement from the MAC before sending data. I don't see a section in the spec describing disabled autoneg so its likely the implementation is manufacturer dependent.

We may be sending a config page at the beginning of SGMII communication, however I believe we do not wait for acknowledgement before sending data. This may allow the MAC to go through auto negotiation even with our negotiation disabled.

For the link status when AN is disabled I'll see if we have a register that does this. 

Best,

Shane 

Hi Magnus,

For SGMII link status, you can use 0xC01[5].

Best,

Shane