This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM3358: KSZ9031 RGMII problem

Part Number: AM3358
Other Parts Discussed in Thread: AM3352

I am working on a design that is based around the AM3358.  We have been trying for some time to get gigabit Ethernet working but haven’t had any luck.  We are using Microchip’s/Micrel’s KSZ9031RNX PHY, and it talks to the AM3358 over RGMII.  When we do throughput testing with a 1 Gb link, we often only see ~1 Mbps.  With a 10/100 link, we see speeds in line with what we’d expect.  Our thought is that something is off with our skew values that is causing issues at gigabit speeds.

 Though it would have been helpful to find this in the datasheet prior to laying out and assembling boards, we found a TI forum post that recommends 1.8 ns of delay on the receive and transmit clocks; this post uses an AM3352 (instead of our AM3358) but the same PHY as us. 

https://e2e.ti.com/support/arm/sitara_arm/f/791/p/334206/1166129

With that target in mind, we modified the PHY skew registers to get as close as we could, taking into account our actual trace lengths on the board (see the spreadsheet below).  On the receive side (inputs from the perspective of the processor), the PHY already has 1.2 ns default delay on the receive clock pad.  So, we added 0.6 ns delay (via the registers) to reach the recommended 1.8 ns.  On the transmit side, there isn’t a default delay, so we had to adjust the skew for the clock, data, and control lines (+ skew for clock and – for data and control).  With the max delays in both directions, we could only get to 1380 ps with the registers, so we had to add ~420 ps to our clock signal on the board.  This took the form of a 3.1” wire that was added to the board as shown below.

Even with these changes, we are still seeing many retries in the transmit direction.  The receive path doesn’t have this problem. 

Could an apps engineer:

  1. Confirm 1.8 ns delay is the recommended delay.
  2. Suggest next steps for debugging this problem.

Thanks!

Matt

 

  • The Ethernet experts have been notified. They will respond here.
  • Hello Matt.

    Section 14.1.2 of the TRM states that RGMII Internal Delay Mode is not supported. The usage notes for AM335x RGMII, including a workaround for the absence of Internal Delay Mode support is in section 1.0.10 of the device errata (SPRZ360i). Please keep in mind that this 1.8ns bus/chip delay is a RGMII specification requirement, not a AM335x requirement.

    1) The RGMII v2.0+ Specification lists TskewR (Data to Clock input Skew at receiver) at MIN: 1ns and MAX: 2.6ns. 1.8ns is the middle of this window so it's a good target.

    2) I'd suggest instrumenting the interface (at the receiver) with a multi-channel scope or logic analyzer to ensure that the RGMII receiver timing specifications are indeed being met. The RGMII timing information is reproduced in Table 7-1 of the Microchip data manual if you don't have access to the specification itself. It also appears that you are running (at least) GTX_CLK very close to the signal bus. It's possible you could be experiencing cross-talk on the interface as a result. Ideally one would want a "keep out" for high-speed period signals to prevent cross-talk from occurring. Please refer to SPRAAR7 for some background on this.

  • DK,

    Thank you for the response.  I went and checked the transmit signals with an MSO5104, both with the logic analyzer section (to capture CLK, TXD[3:0], and CTL) and the analog portion (just CLK, TXD[1:0], and CTL).  Surprisingly, when I had all analyzer leads connected, the board averaged almost 370 Mb/s.  I then started removing one lead at a time, and the speed got progressively worse.

    With the analog channels, I didn't see anything very ugly on the data lines.  There's some overshoot, but nothing I see to cause issues.  I suspect that the ~4 pF of capacitance that's added each time I add a scope probe is rounding off edges enough to improve the speed.  I don't think this is related to the AM3358 anymore.

    CH1 = TCLK, CH2 = TXD0, CH3 = TXD1

    All probe leads attached

    iperf Done.

    root@jester:~# iperf3 -c 192.168.1.102

    Connecting to host 192.168.1.102, port 5201

    [  4] local 192.168.1.105 port 57410 connected to 192.168.1.102 port 5201

    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

    [  4]   0.00-1.01   sec  39.4 MBytes   328 Mbits/sec  137   26.9 KBytes

    [  4]   1.01-2.02   sec  44.5 MBytes   369 Mbits/sec  117   42.4 KBytes

    [  4]   2.02-3.00   sec  43.9 MBytes   374 Mbits/sec   76   33.9 KBytes

    [  4]   3.00-4.02   sec  44.5 MBytes   368 Mbits/sec  106   25.5 KBytes

    [  4]   4.02-5.01   sec  44.1 MBytes   372 Mbits/sec  100   38.2 KBytes

    [  4]   5.01-6.01   sec  44.4 MBytes   372 Mbits/sec  115   38.2 KBytes

    [  4]   6.01-7.01   sec  45.5 MBytes   381 Mbits/sec   92   24.0 KBytes

    [  4]   7.01-8.01   sec  43.6 MBytes   367 Mbits/sec   89   29.7 KBytes

    [  4]   8.01-9.02   sec  45.0 MBytes   373 Mbits/sec   90   50.9 KBytes

    [  4]   9.02-10.03  sec  45.4 MBytes   377 Mbits/sec   99   67.9 KBytes

    - - - - - - - - - - - - - - - - - - - - - - - - -

    [ ID] Interval           Transfer     Bandwidth       Retr

    [  4]   0.00-10.03  sec   440 MBytes   368 Mbits/sec  1021             sender

    [  4]   0.00-10.03  sec   440 MBytes   368 Mbits/sec                  receiver

    iperf Done.

    root@jester:~#

    Removed logic analyzer lead from TXD0

    root@jester:~# iperf3 -c 192.168.1.102

    Connecting to host 192.168.1.102, port 5201

    [  4] local 192.168.1.105 port 57414 connected to 192.168.1.102 port 5201

    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

    [  4]   0.00-1.03   sec  18.7 MBytes   153 Mbits/sec  183   15.6 KBytes

    [  4]   1.03-2.00   sec  29.1 MBytes   251 Mbits/sec  204   17.0 KBytes

    [  4]   2.00-3.01   sec  33.0 MBytes   274 Mbits/sec  134   41.0 KBytes

    [  4]   3.01-4.02   sec  47.3 MBytes   395 Mbits/sec  152   33.9 KBytes

    [  4]   4.02-5.00   sec  45.9 MBytes   391 Mbits/sec  152   29.7 KBytes

    [  4]   5.00-6.02   sec  46.3 MBytes   379 Mbits/sec  179   28.3 KBytes

    [  4]   6.02-7.01   sec  44.6 MBytes   380 Mbits/sec  166   59.4 KBytes

    [  4]   7.01-8.03   sec  45.8 MBytes   378 Mbits/sec  172   55.1 KBytes

    [  4]   8.03-9.02   sec  45.6 MBytes   386 Mbits/sec  154   36.8 KBytes

    [  4]   9.02-10.01  sec  46.1 MBytes   389 Mbits/sec  158   31.1 KBytes

    - - - - - - - - - - - - - - - - - - - - - - - - -

    [ ID] Interval           Transfer     Bandwidth       Retr

    [  4]   0.00-10.01  sec   402 MBytes   337 Mbits/sec  1654             sender

    [  4]   0.00-10.01  sec   402 MBytes   337 Mbits/sec                  receiver

     

    iperf Done.

    root@jester:~#

    Logic analyzer leads removed from TXD0, TXD1

    root@jester:~# iperf3 -c 192.168.1.102

    Connecting to host 192.168.1.102, port 5201

    [  4] local 192.168.1.105 port 57418 connected to 192.168.1.102 port 5201

    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

    [  4]   0.00-1.00   sec  25.4 MBytes   213 Mbits/sec  300   21.2 KBytes

    [  4]   1.00-2.01   sec  20.0 MBytes   167 Mbits/sec  268   26.9 KBytes

    [  4]   2.01-3.01   sec  42.5 MBytes   356 Mbits/sec  385   21.2 KBytes

    [  4]   3.01-4.02   sec  43.5 MBytes   360 Mbits/sec  347   26.9 KBytes

    [  4]   4.02-5.00   sec  43.1 MBytes   366 Mbits/sec  376   38.2 KBytes

    [  4]   5.00-6.01   sec  45.3 MBytes   377 Mbits/sec  322   26.9 KBytes

    [  4]   6.01-7.02   sec  43.5 MBytes   362 Mbits/sec  389   22.6 KBytes

    [  4]   7.02-8.00   sec  42.7 MBytes   364 Mbits/sec  343   12.7 KBytes

    [  4]   8.00-9.00   sec  42.8 MBytes   358 Mbits/sec  360   26.9 KBytes

    [  4]   9.00-10.02  sec  43.6 MBytes   361 Mbits/sec  382   21.2 KBytes

    - - - - - - - - - - - - - - - - - - - - - - - - -

    [ ID] Interval           Transfer     Bandwidth       Retr

    [  4]   0.00-10.02  sec   392 MBytes   328 Mbits/sec  3472             sender

    [  4]   0.00-10.02  sec   392 MBytes   328 Mbits/sec                  receiver

     

    Logic analyzer leads removed from TXD0, TXD1, TXD2

    iperf Done.

    root@jester:~# iperf3 -c 192.168.1.102

    Connecting to host 192.168.1.102, port 5201

    [  4] local 192.168.1.105 port 57422 connected to 192.168.1.102 port 5201

    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

    [  4]   0.00-1.02   sec  10.4 MBytes  85.5 Mbits/sec  307   26.9 KBytes

    [  4]   1.02-2.00   sec  16.0 MBytes   137 Mbits/sec  380   4.24 KBytes

    [  4]   2.00-3.03   sec  15.8 MBytes   128 Mbits/sec  479   21.2 KBytes

    [  4]   3.03-4.00   sec  16.8 MBytes   146 Mbits/sec  495   14.1 KBytes

    [  4]   4.00-5.00   sec  17.4 MBytes   146 Mbits/sec  463   4.24 KBytes

    [  4]   5.00-6.00   sec  1.21 MBytes  10.1 Mbits/sec   64   1.41 KBytes

    [  4]   6.00-7.01   sec  19.7 MBytes   164 Mbits/sec  505   28.3 KBytes

    [  4]   7.01-8.00   sec  11.5 MBytes  97.5 Mbits/sec  237   2.83 KBytes

    [  4]   8.00-9.00   sec   225 KBytes  1.84 Mbits/sec   16   1.41 KBytes

    [  4]   9.00-10.00  sec  5.55 MBytes  46.5 Mbits/sec  175   4.24 KBytes

    - - - - - - - - - - - - - - - - - - - - - - - - -

    [ ID] Interval           Transfer     Bandwidth       Retr

    [  4]   0.00-10.00  sec   115 MBytes  96.1 Mbits/sec  3121             sender

    [  4]   0.00-10.00  sec   114 MBytes  96.0 Mbits/sec                  receiver

     

     

    Logic analyzer leads removed from TXD0, TXD1, TXD2, TXD3

    iperf Done.

    root@jester:~# iperf3 -c 192.168.1.102

    Connecting to host 192.168.1.102, port 5201

    [  4] local 192.168.1.105 port 57426 connected to 192.168.1.102 port 5201

    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

    [  4]   0.00-1.00   sec  10.6 MBytes  88.6 Mbits/sec  508   4.24 KBytes

    [  4]   1.00-2.00   sec  3.93 MBytes  33.0 Mbits/sec  241   1.41 KBytes

    [  4]   2.00-3.00   sec  9.51 MBytes  79.8 Mbits/sec  477   5.66 KBytes

    [  4]   3.00-4.00   sec  2.80 MBytes  23.5 Mbits/sec  125   2.83 KBytes

    [  4]   4.00-5.00   sec   526 KBytes  4.31 Mbits/sec   42   2.83 KBytes

    [  4]   5.00-6.00   sec  2.15 MBytes  18.0 Mbits/sec  115   5.66 KBytes

    [  4]   6.00-7.00   sec  8.98 MBytes  75.4 Mbits/sec  451   2.83 KBytes

    [  4]   7.00-8.00   sec  2.41 MBytes  20.2 Mbits/sec  152   2.83 KBytes

    [  4]   8.00-9.00   sec  3.47 MBytes  29.1 Mbits/sec  180   2.83 KBytes

    [  4]   9.00-10.00  sec  3.71 MBytes  31.1 Mbits/sec  205   2.83 KBytes

    - - - - - - - - - - - - - - - - - - - - - - - - -

    [ ID] Interval           Transfer     Bandwidth       Retr

    [  4]   0.00-10.00  sec  48.0 MBytes  40.3 Mbits/sec  2496             sender

    [  4]   0.00-10.00  sec  47.9 MBytes  40.2 Mbits/sec                  receiver

     iperf Done.

    Logic analyzer leads removed from TXD0, TXD1, TXD2, TXD3, TCTL

    root@jester:~# iperf3 -c 192.168.1.102

    Connecting to host 192.168.1.102, port 5201

    [  4] local 192.168.1.105 port 57430 connected to 192.168.1.102 port 5201

    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

    [  4]   0.00-1.00   sec   325 KBytes  2.66 Mbits/sec   36   2.83 KBytes

    [  4]   1.00-2.00   sec   246 KBytes  2.02 Mbits/sec   34   2.83 KBytes

    [  4]   2.00-3.00   sec  43.8 KBytes   359 Kbits/sec    8   1.41 KBytes

    [  4]   3.00-4.00   sec  74.9 KBytes   614 Kbits/sec   12   2.83 KBytes

    [  4]   4.00-5.00   sec   221 KBytes  1.81 Mbits/sec   36   2.83 KBytes

    [  4]   5.00-6.00   sec   202 KBytes  1.66 Mbits/sec   32   2.83 KBytes

    [  4]   6.00-7.00   sec   164 KBytes  1.34 Mbits/sec   20   2.83 KBytes

    [  4]   7.00-8.00   sec   369 KBytes  3.02 Mbits/sec   44   11.3 KBytes

    [  4]   8.00-9.00   sec   110 KBytes   903 Kbits/sec   13   2.83 KBytes

    [  4]   9.00-10.00  sec   202 KBytes  1.66 Mbits/sec   21   4.24 KBytes

    - - - - - - - - - - - - - - - - - - - - - - - - -

    [ ID] Interval           Transfer     Bandwidth       Retr

    [  4]   0.00-10.00  sec  1.91 MBytes  1.60 Mbits/sec  256             sender

    [  4]   0.00-10.00  sec  1.87 MBytes  1.57 Mbits/sec                  receiver

     iperf Done.

     

    Logic analyzer leads removed from TXD0, TXD1, TXD2, TXD3, TCTL, TCLK

    root@jester:~# iperf3 -c 192.168.1.102

    Connecting to host 192.168.1.102, port 5201

    [  4] local 192.168.1.105 port 57434 connected to 192.168.1.102 port 5201

    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

    [  4]   0.00-1.00   sec   409 KBytes  3.35 Mbits/sec   48   1.41 KBytes

    [  4]   1.00-2.00   sec   269 KBytes  2.20 Mbits/sec   38   4.24 KBytes

    [  4]   2.00-3.00   sec   120 KBytes   984 Kbits/sec   23   2.83 KBytes

    [  4]   3.00-4.00   sec  18.4 KBytes   151 Kbits/sec    9   2.83 KBytes

    [  4]   4.00-5.00   sec  72.1 KBytes   591 Kbits/sec   15   2.83 KBytes

    [  4]   5.00-6.00   sec  60.8 KBytes   498 Kbits/sec   12   1.41 KBytes

    [  4]   6.00-7.00   sec   102 KBytes   834 Kbits/sec   15   1.41 KBytes

    [  4]   7.00-8.00   sec  59.4 KBytes   486 Kbits/sec   13   1.41 KBytes

    [  4]   8.00-9.00   sec  84.8 KBytes   695 Kbits/sec   11   2.83 KBytes

    [  4]   9.00-10.00  sec   567 KBytes  4.65 Mbits/sec   58   1.41 KBytes

    - - - - - - - - - - - - - - - - - - - - - - - - -

    [ ID] Interval           Transfer     Bandwidth       Retr

    [  4]   0.00-10.00  sec  1.72 MBytes  1.44 Mbits/sec  242             sender

    [  4]   0.00-10.00  sec  1.68 MBytes  1.41 Mbits/sec                  receiver

    Thanks,

    Matt

  • Interesting.

    What values are you using for your RGMII series resistors?