DP83867CR: RX_CTRL Signal Not Sampled or Shifted Correctly on FPGA

Part Number: DP83867CR

Greetings,

I'm interfacing DP83867CR in RGMII mode with Xilinx Spartan7 FPGA. RGMII traces on PBC are length/impedance matched without added clock delay. Inside FPGA RGMII is converted into GMII using IDDR primitives.

I'm expiriencing a verry strange issue, where when sending ethernet frames with even number of bytes everything works fine. In contrast when sending odd sized frames the RX_CTRL signal is behaving strangely. It can be seen in pictures below.

Even sized frames.

Picture above shows Xilinx logic analyzer with even sized frames, inter frame gap is 12 and RX_CTRL is sampled correctly.

Two pictures abowe show the same situation, just with odd sized frames. As shown gmii enable signal is deasserted before packet end, and the error signal is asserted.

Also inter frame gap is 11 cycles and 13 cycles repetitevly, never 12.

The RX clock delay is implemented with FPGA primitives and I've tried also setting various RX_CLK delays using PHY with MDIO. Also I tried delaying data and control while leaving clock in place.

Nothing really worked.

I have the same board with simmilar PHY from alternative manufacturer where I don't get those errors. 

Please ask for any additional info if needed.

Thank you.

Asmir A

  • Hi Asmir,

    We are looking into this issue and will provide additional feedback by Tuesday of next week.

    Is this phenomenon causing any packet errors/lost packets? Is communication still clean?

    Thank you,

    Nikhil

    All information in this correspondence and in any related correspondence is provided “as is” and “with all faults”, and it is subject to TI’s Important Notice (http://www.ti.com/corp/docs/legal/important-notice.shtml).

     

  • Hi Nikhil,

    Thank you for your response.

    Yes my packet generator reports a lot of FCS errors with higher link utilization. When the link utilization drops there are no errors.

    Also as said if all frames have even number of bytes there is no problems even with 100% link utilization.

    I can provide actual numbers (frame drop rate with different utilization and frame sizes) if needed.

    Asmir

  • Hi Asmir,

    Can you provide a register dump of the following registers for the case where an even number of bytes is used at 100% utilization vs an odd number of bytes at 100% utilization

    • 0x0 - 0x1F
    • 0x2D - 0x2E
    • 0x6E - 0x6F 
    • 0x31 - 0x33
    • 0x86
    • 0x170

    Thank you,

    Nikhil

    All information in this correspondence and in any related correspondence is provided “as is” and “with all faults”, and it is subject to TI’s Important Notice (http://www.ti.com/corp/docs/legal/important-notice.shtml).

  • Hi Nikhil, 

    Thanks for reaching out.

    I've dumped the required registers, here is a link to the sheet, I hope the format is acceptable.

    reg_dump_spredsheet

    I've checked output for sensibility but feel free to request additional tests with other conditions if needed.

    Asmir

  • Hi Asmir,

    Unfortunately I am being blocked from the link attached. Are you able to upload register dump as an excel file?

    What is the format of the link attached (dropbox link, google doc link, etc.)?

    Thank you,

    Nikhil

  • Oh, It was google docs, I even made sure it's publicly availabe, but who knows, anyway here is another link, I hope this one works.

    dp83867_reg

    Thanks for your patience.

    Asmir

  • Hi Asmir,

    I have sent you a request to connect and have reached out private message. Please check your inbox and I look forward to your reply!

    Thank you,

    Nikhil

  • dp83867cr_reg_dump_xls.xls

    Message attachment in case of firewall blocking external sites.

  • Hi Asmir,

    Thank you for providing the register values. I will review spreadsheet and provide additional feedback tomorrow.

    Thank you,

    Nikhil 

  • Hi Asmir,

    When you say you are using an odd size frame, does this mean packet is not byte boundary aligned?

    Can you try toggling the odd nibble detect bit in register 0x43?

    Thank you,

    Nikhil

  • Hi Nikhil,

    I think packets are byte boundary aligned, just their size is different. For example when I generate packets with sizes of 64, 68, 1518, 4000... bytes - everything works.

    But with packet sizes of 65, 71, 1519 - I get errors pictured in first post.

    I'll try toggling the register field you mentioned and post the results.

    Thank you,

    Asmir A

  • Hi Asmir,

    Please let me know the status after adjusting the odd nibble detect register value.

    Thank you,

    Nikhil

    All information in this correspondence and in any related correspondence is provided “as is” and “with all faults”, and it is subject to TI’s Important Notice (http://www.ti.com/corp/docs/legal/important-notice.shtml).

  • Hi Nikhil,

    I didn't notice any change after seting odd nibble detect register field. As I can not find any affect the register change has please inform me if you need any register dump after the change.

    Thank you.

    Asmir A

  • Hi Asmir,

    Thanks for the update. I will look into this issue and provide an update by Friday of this week.

    Thank you,

    Nikhil

  • Hi Asmir,

    Quick question: What is value of register 0x32 in both cases odd and even byte size?

    From a PHY perspective, for 1G data, as long as the packets are byte aligned, PHY should be immune to odd vs even byte length. An IPG of at least 12 bytes (~96ns) is needed for clock recovery in 1G mode. 

    What is the link partner used during these tests? I understand and FPGA and DP83867 is used on one side, what is on the other end of the cable? Is it another DP83867 and FPGA setup, and if so, is this second FPGA the one counting errors?

    Can the PHY be placed in MII or digital loopback such that the FPGA sends data to the PHY via RGMII, and the PHY loops data back to the FPGA? Is communication clean in this test case?

    Thank you,

    Nikhil

    All information in this correspondence and in any related correspondence is provided “as is” and “with all faults”, and it is subject to TI’s Important Notice (http://www.ti.com/corp/docs/legal/important-notice.shtml).

  • Hi Nihkil,

    I'll get the 0x32 redings ASAP.

    To give a more detailed overview, I'm building an ethernet tap with aggregation support. There are four DP83867 phys connected to the FPGA.

    But the issue is on the passthrough mode given by the block diagram below, so I'll focus on it.

    Traffic from first port is sent to the second and vice versa.

    The link partner to the all DP83867 PHYs is Xena Valkyrie traffic generator and analysis platform. So I'm using traffic generator to send data to one port and, when working correctly, should receive the same data from second port.

    But when I send packets with odd number of bytes to the first port I receive a lot of FCS errors from the second port when the gigabit link utilization is about 80% and more. Error rates increase with link speed utilization.

    I've isolated the issue to be on the receiving end of the FPGA, i.e. packets sent from FPGA to traffic generator/analyzer via DP83867 are all fine, regarding of frame size and link speed.

    In contrast packets with uneven number of bytes sent from packet generator to the FPGA are not received correctly and such damaged packets forwarded to the second port give FCS errors on the network traffic analyzer.

    What I did more is to isolate one port and attach in-chip logic analyzer to it, so packet generator was sending data to the FPGA and I was sampling the same packets using logic analyzer, where I saw the issue presented on the pictures in the first post.

    Simplified by ignorring rest of the system, the issue is receiving data from PHY chip to the FPGA.

    I've tested the traffic generator by simply connecting both ports with a cable - there was no issue. Alternetive PHYs in same configuration also don't have any issue.

    I'll also try to put PHY into loopback mode and post results (it will take a bit) - thanks for the suggestion.

    Best regards, Asmir

  • Hi Asmir,

    So the way I understand this is the Xena traffic generator sends data to the RJ45 of one DP83867 -> Xilinx FPGA -> second DP83867 -> back to Xena. If we can make the below configuration, set the first DP83867 for reverse loopback, such that the Xena sends data to the first DP83867, and the first DP83867 sends it right back to the Xena, can the Xena count packet errors? Are the errors still present? 

    Please let me know if you can't view the picture pasted above. Please let me know if further clarification is needed for this setup. 

    What is the cable length between Xena and DP83867?

    Thank you,

    Nikhil Menon

    All information in this correspondence and in any related correspondence is provided “as is” and “with all faults”, and it is subject to TI’s Important Notice (http://www.ti.com/corp/docs/legal/important-notice.shtml).

  • Hi Nikhil,

    Yes, you understood right, I'll create suggested configuration shortly and post results.

    Regarding MDIO register 0x32, the value I got is 0xD3. Though I've tried various delay setings using 0x32 and 0x86 registers before with 0.25ns steps - results were same or much worse depending on the delay amount. I've also tried with internal FPGA delay with ~80ps resolution - it did not mitigate the issue.

    Thank you for your sugestion, I'll post results shortly.

    PS: The cable length is no more than 3m, and I tested the Xena Valkyrie by connecting the both ports i.e just sending data from one port to the other without my device between.

    Asmir A

  • Hi, Nikhil

    So I've put the PHY into the Far-End (Reverse) loopback mode and disabled FPGA passthrough logic, I was using one port at the time and l think loopback was configured correctly as I got packets back to the generator.

    Again with even frames there was no error, but with odd number of bytes I got errors.

    Testing was done at 1Gbit, and the latency was around 300ns (if I substracted packet generator latency correctly).

    Let me know if you need more data or testing with different loopback mode.

    Thank you,

    Asmir A

  • Hi Asmir,

    Does the FPGA have the capability to generate packets? Can we try an experiment where we have the FPGA generate packets to PHY 1 via RGMII, connect PHY 1 to PHY 2 so packets are returned back to the FPGA? Will the FPGA be able to check for errors? This setup removes the Xena device from the system, so we can check the PHY and FPGA.

    Thank you,

    Nikhil

    All information in this correspondence and in any related correspondence is provided “as is” and “with all faults”, and it is subject to TI’s Important Notice (http://www.ti.com/corp/docs/legal/important-notice.shtml).