XFIMDIO not supported in 66AK2E05??

Also please let me know where I can download the userguide for XFIMDIO for its register details and so. On product page I don't see any relevant info for XFI/XGE/10G subsystem.

Thanks,
Kushal

Hi,
Please refer to the user guide.
www.ti.com/lit/ug/spruhj5/spruhj5.pdf

Can you please tell us the reason for not supporting XFI MDIO? Is is because of HW or Linux support issue?

Why bit banging is the only way in kernel? This can cause a complete design re-spin as it is nowhere mentioned in ERRATA as well as on web.

Thanks,

Kushal

Hi, Kushal,

The PHY selected earlier on the Breakout Card did not support clause 45, so bit bang was used to get the functionality working. In the newer version of the BoC, retimer is used which does not require MDIO.

Rex

Rex,

Does this mean that TI have not validated XFI MDIO? We are using phy which supports clause 45 and connected over XFI MDIO.
If TI has validated and working then it would be great help if can identify the changes in kernel too.

The only major concern is this SHOULD be mentioned somewhere! We have board in our hands already!

Thanks,
Kushal B. Patel

Kushal,

Your understanding is correct. TI has not been able to validate XFI MDIO due to what is supported on the BoC.

Rex

Rex, I have following questions.

1> What can be done in case of we have already connected XFI_MDIO/MDC line with 10G PHY which supports Clause 45 register read/write only?
 Do you have a driver XGE_MDIO driver ready? Will it be ever ready?

2> If hardware modification is required, then what can be the best new approach? I am sure that you should have some way to communicate with the 10G phy that supports the C45 only.

Thanks & regards,

Kushal B. Patel

Hi, Patel,

I don't think we have a platform using PHY, but retimer. I can forward your question to the hardware team to see if they can be any help.

Rex

Rex,

We have found some workaround. I found existing MDIO module in Keystone 2 supporting both Clause 45 and Clause 22 PHYs. We have disconnected XFI_MDIO/MDC lines and connected MDIO/MDC lines instead (Of-course with wire jumpers!!! :-( ). Some changes in Device tree and davinci_mdio driver did read and write into PHY.

Let us keep this thread open, and see what best solution we can find for this issue. Meanwhile can you confirm if TI has intention to add support for XFI_MDIO  driver in next release? Is someone aware and working on this?

Thanks & regards,

Kushal B. Patel

Rex,

We are facing one problem related to 10G XGMII. We have Cortina 10G PHY with verified hardware.

Cortina says, one of the status register will indicate the Rx_lock and TX_lock. Both are getting locked with 10G network switch.

But we are not able to see serdes link up in K2E side. Also we are not able to see any transmit counter (Tx counter) incrementing if we enter the ifconfig command for eth8 and eth9 interfaces. But each time we get Rx counter increased. We have done this with 10G network switch connected for SFP+ copper.

Tx counter is not getting incremented so I believe that serdes link may be an issue.

Would you please let us know that how did you test the XFI interface (with re-timer, we are using PHY.) Do you perform external loopback?

In that case do you see the serdes link up? What can be the reason for not getting serdes link up?

Below is the our kernel log.

[   47.074265] davinci_mdio 24200f00.mdio: phy[0]: device 24200f00.mdio:00, driver Marvell 88E1510
[   47.082955] davinci_mdio 24200f00.mdio: phy[1]: device 24200f00.mdio:01, driver Marvell 88E1510
[   47.091637] davinci_mdio 24200f00.mdio: phy[2]: device 24200f00.mdio:02, driver unknown
[   47.099631] davinci_mdio 24200f00.mdio: phy[4]: device 24200f00.mdio:04, driver Marvell 88E1543
[   47.108318] davinci_mdio 24200f00.mdio: phy[5]: device 24200f00.mdio:05, driver Marvell 88E1543
[   47.117006] davinci_mdio 24200f00.mdio: phy[6]: device 24200f00.mdio:06, driver Marvell 88E1543
[   47.125694] davinci_mdio 24200f00.mdio: phy[7]: device 24200f00.mdio:07, driver Marvell 88E1543
[   47.134381] davinci_mdio 24200f00.mdio: phy[8]: device 24200f00.mdio:08, driver Cortina cs4315
[   47.142981] davinci_mdio 24200f00.mdio: phy[9]: device 24200f00.mdio:09, driver Cortina cs4315
[   47.151832] keystone-netcp 24000000.netcp: Missing cpts_clock_mult property in the DT.
[   47.159746] keystone-netcp 24000000.netcp: Missing cpts_clock_shift property in the DT.
[   47.167742] keystone-netcp 24000000.netcp: Missing cpts_clock_div property in the DT.
[   47.176440] keystone-netcp 24000000.netcp: Created interface "eth0"
[   47.182712] keystone-netcp 24000000.netcp: dma_chan_name nettx0
[   47.189341] keystone-netcp 24000000.netcp: Created interface "eth1"
[   47.195607] keystone-netcp 24000000.netcp: dma_chan_name nettx1
[   47.202270] keystone-netcp 24000000.netcp: Created interface "eth2"
[   47.208529] keystone-netcp 24000000.netcp: dma_chan_name nettx2
[   47.215189] keystone-netcp 24000000.netcp: Created interface "eth3"
[   47.221448] keystone-netcp 24000000.netcp: dma_chan_name nettx3
[   47.228066] keystone-netcp 24000000.netcp: Created interface "eth4"
[   47.234337] keystone-netcp 24000000.netcp: dma_chan_name nettx4
[   47.240956] keystone-netcp 24000000.netcp: Created interface "eth5"
[   47.247226] keystone-netcp 24000000.netcp: dma_chan_name nettx5
[   47.253870] keystone-netcp 24000000.netcp: Created interface "eth6"
[   47.260132] keystone-netcp 24000000.netcp: dma_chan_name nettx6
[   47.266764] keystone-netcp 24000000.netcp: Created interface "eth7"
[   47.273032] keystone-netcp 24000000.netcp: dma_chan_name nettx7
[   47.279206] XGE serdes config:
[   47.282259]   ref_clk=156.25MHz, link_rate=10.3125G, lanes=2
[   47.287905]   c1=2, c2=0, cm=2, tx_att=12, tx_vreg=4
[   47.292860]   eq flags: vreg=1, cdfe=1, offset=1
[   47.297467] XGE: serdes reset
[   48.529380] XGE: timeout waiting for serdes link up
[   48.534651] keystone-netcp 2f00000.netcp: Created interface "eth8"
[   48.540824] keystone-netcp 2f00000.netcp: dma_chan_name xgetx0
[   48.546661] ******* [CS4315] cs4315_config_init called for PHY_ID : 0x13e51002
[   48.555223] ******* [CS4315] cs4315_upload_firmware called *******
[   49.324056] Num of lines written  : 9494
[   49.328044] ************** CS4315 Firmware upload complete with checksum 0x0 **************
[   49.561712] XGE: serdes reset
[   50.793635] XGE: timeout waiting for serdes link up
[   50.798899] keystone-netcp 2f00000.netcp: Created interface "eth9"
[   50.805080] keystone-netcp 2f00000.netcp: dma_chan_name xgetx1
[   50.810911] ******* [CS4315] cs4315_config_init called for PHY_ID : 0x13e51002
[   50.819483] ******* [CS4315] cs4315_upload_firmware called *******
[   51.588341] Num of lines written  : 9494
[   51.592331] ************** CS4315 Firmware upload complete with checksum 0x0 **************
[   51.826197] XGE: serdes reset
[   53.058235] XGE: timeout waiting for serdes link up
[   53.063155] keystone-netcp 24000000.netcp: pdsp 0 firmware: keystone/pa_in0_pdsp0.fw
[   53.070890] keystone-netcp 24000000.netcp: pdsp 1 firmware: keystone/pa_in0_pdsp1.fw


Thank you,
Kushal B. Patel

Hi, Kushal,

The driver stops waiting for SerDes to come up after timeout so the boot can continue without wasting time waiting. In TI EVM case, SerDes comes up later after timeout and XGE can be used. Below are the logs from TI K2E kernel boot. I boot the EVM from ubi without ethernet connection on eth0 so no other connection than through eth8 on the RTM-BoC which connects to a Linux PC through SPF+ cable.

[   24.401187] keystone-netcp 24000000.netcp: dma_chan_name nettx7
[   24.407359] XGE serdes config:
[   24.410402]   ref_clk=156.25MHz, link_rate=10.3125G, lanes=2
[   24.416054]   c1=2, c2=0, cm=2, tx_att=12, tx_vreg=4
[   24.421008]   eq flags: vreg=1, cdfe=1, offset=1
[   24.425615] XGE: serdes reset
[   25.556822] XGE: timeout waiting for serdes link up
[   25.562051] keystone-netcp 2f00000.netcp: Created interface "eth8"
[   25.568221] keystone-netcp 2f00000.netcp: dma_chan_name xgetx0
[   25.574053] XGE: serdes reset
[   26.705210] XGE: timeout waiting for serdes link up
[   26.710417] keystone-netcp 2f00000.netcp: Created interface "eth9"
[   26.716587] keystone-netcp 2f00000.netcp: dma_chan_name xgetx1
[   26.722420] XGE: serdes reset
[   27.853504] XGE: timeout waiting for serdes link up
[   27.858417] keystone-netcp 24000000.netcp: pdsp 0 firmware: keystone/pa_in0_pdsp0.fw
[   27.866151] keystone-netcp 24000000.netcp: pdsp 1 firmware: keystone/pa_in0_pdsp1.fw
[   27.873884] keystone-netcp 24000000.netcp: pdsp 2 firmware: keystone/pa_in1_pdsp0.fw
[   27.881617] keystone-netcp 24000000.netcp: pdsp 3 firmware: keystone/pa_in1_pdsp1.fw
[   27.889350] keystone-netcp 24000000.netcp: pdsp 4 firmware: keystone/pa_in2_pdsp0.fw

Arago 2013.12 k2e-evm ttyS0

k2e-evm login: root
root@k2e-evm:~# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 08:00:28:32:BB:65
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth1      Link encap:Ethernet  HWaddr 02:18:31:7E:3E:6F
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth2      Link encap:Ethernet  HWaddr 66:82:AD:AB:DF:45
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth3      Link encap:Ethernet  HWaddr 2A:99:07:01:CF:84
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth4      Link encap:Ethernet  HWaddr BE:16:5E:26:3A:F0
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth5      Link encap:Ethernet  HWaddr 16:1B:A3:5E:7D:EA
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth6      Link encap:Ethernet  HWaddr DA:D7:70:26:A7:2B
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth7      Link encap:Ethernet  HWaddr 9A:16:9B:B6:AE:0B
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth8      Link encap:Ethernet  HWaddr 02:18:31:7E:3E:5E
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth9      Link encap:Ethernet  HWaddr 02:18:31:7E:3E:5F
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

gre0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          NOARP  MTU:1476  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

gretap0   Link encap:Ethernet  HWaddr 00:00:00:00:00:00
          BROADCAST MULTICAST  MTU:1476  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ip6tnl0   Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          NOARP  MTU:1452  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:268 (268.0 B)  TX bytes:268 (268.0 B)

tunl0     Link encap:UNSPEC  HWaddr 00-00-00-00-30-30-30-30-00-00-00-00-00-00-00-00
          NOARP  MTU:0  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@k2e-evm:~# ifconfig eth8 192.168.1.88 up
[   59.170520] keystone-netcp 2f00000.netcp: initialize cpsw version 1.1 (0) CPSW identification value 0x4ee3
[   59.180347] keystone-netcp 2f00000.netcp: Created a cpsw ale engine
[   59.218412] net eth8: netcp device eth8 opened
[   59.224130] IPv6: ADDRCONF(NETDEV_UP): eth8: link is not ready
[   59.229961] 8021q: adding VLAN 0 to HW filter on device eth8
[   59.235609] net eth8: adding rx vlan id: 0
root@k2e-evm:~# [   59.278058] IPv6: ADDRCONF(NETDEV_CHANGE): eth8: link becomes ready

root@k2e-evm:~# ping 192.168.1.22
PING 192.168.1.22 (192.168.1.22): 56 data bytes
64 bytes from 192.168.1.22: seq=0 ttl=64 time=0.185 ms
64 bytes from 192.168.1.22: seq=1 ttl=64 time=0.070 ms
64 bytes from 192.168.1.22: seq=2 ttl=64 time=0.065 ms
^C
--- 192.168.1.22 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.065/0.106/0.185 ms
root@k2e-evm:~#

 

 

Hello Rex,

We have tried with connecting two of boards back to back with SFP+ Direct Attached copper Cable and see if assigning static ip addresses to two of them works or not.

I found that interface (eth9 in our case) link does not become ready. There is no 10G network switch in between two boards.

Will our setup work? Setup is like:

(Board1 -> Cortina_PHY_1 -> SFP+ copper connector of Board1 -> Cable -> SFP+ copper connector of Board2 - > Cortina_PHY_2 -> Board2).

On 1st board:
root@k2e-evm:~# ifconfig eth9 192.168.2.2 up
[  331.563295] keystone-netcp 2f00000.netcp: initialize cpsw version 1.1 (0) CPSW identification value 0x4ee3
[  331.573133] keystone-netcp 2f00000.netcp: Created a cpsw ale engine
[  331.579799] ******* [CS4315] cs4315_config_init called for PHY_ID : 0x13e51002
[  331.813948] keystone-netcp 2f00000.netcp: phy found: id is: 24200f00.mdio:09, drv: Cortina cs4315
[  331.871191] net eth9: netcp device eth9 opened
[  331.876492] IPv6: ADDRCONF(NETDEV_UP): eth9: link is not ready
[  331.882323] 8021q: adding VLAN 0 to HW filter on device eth9
[  331.887972] net eth9: adding rx vlan id: 0
root@k2e-evm:~# ping 192.168.2.3
PING 192.168.2.3 (192.168.2.3): 56 data bytes

On 2nd board:

root@k2e-evm:~# ifconfig eth9 192.168.2.3 up
[  343.255178] keystone-netcp 2f00000.netcp: initialize cpsw version 1.1 (0) CPSW identification value 0x4ee3
[  343.265010] keystone-netcp 2f00000.netcp: Created a cpsw ale engine
[  343.271676] ******* [CS4315] cs4315_config_init called for PHY_ID : 0x13e51002
[  343.505728] keystone-netcp 2f00000.netcp: phy found: id is: 24200f00.mdio:09, drv: Cortina cs4315
[  343.553036] net eth9: netcp device eth9 opened
[  343.558338] IPv6: ADDRCONF(NETDEV_UP): eth9: link is not ready
[  343.564169] 8021q: adding VLAN 0 to HW filter on device eth9
[  343.569817] net eth9: adding rx vlan id: 0
root@k2e-evm:~# ping 192.168.2.2
PING 192.168.2.2 (192.168.2.2): 56 data bytes
^C
--- 192.168.2.2 ping statistics ---
28 packets transmitted, 0 packets received, 100% packet loss
root@k2e-evm:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.2.0     *               255.255.255.0   U     0      0        0 eth9
root@k2e-evm:~# 

So would you please let us know what is your test setup?
I understand that your setup includes K2E EVM which is connected to 10G Network switch through RTM-BOC and SFP+ Direct Attched Copper Cable.
And same time the switch has also a linux host directly connected to the another port of 10G switch.

OR 

You may have directly connected K2E EVM with RTM BOC to PC without network switch being involved at all with the single DAC cable.

Please correct our understanding. We are also under impression that you are using copper cable. Not Fiber.

Thansk & regards,
Kushal B. Patel

Hi, Kushal,

Our sytstem test lab uses a 10GbE switch and both K2E EVM+RTM BoC and Linux PC connected to the switch with SPF+ cables. However, my setup is direct connect between K2E EVM+RTM BoC and Linux PC using a SPF+ cable. One of the my coworkers connected 2 K2E EVMs using a back-to-back RTM connector (no BoC and SPF+ cable involved). All 3 setups have 10GbE up and ping successfully.

Rex

Rex,

I think this piece of info is enough. Thanks.

One Last question and clarification.
Did you test 10G SFP+ fiber module with the EVM? Our PHY supports two standards. SFP+ Fiber and Copper.

Still we are waiting for Fiber cable to come and to test with the switch but in advance, can you please confirm,

- If you have tested 10G on Fiber module.

- Does RTM BOC supports Fiber 10G SFP+ optics cable?

- With no changes in Linux kernel will it also work with Optical cable if Cortina PHY is properly configured for 10GBASE-LR/LRM mode?

Thanks & regards,

Kushal B. Patel

HI, Kushal,

My understanding is that fiber optical mode has to do with the transceiver which is compatible with the cable itself. It is the transceiver plugged into the cage. However, we have not tested with any fiber cable. All are copper.

Rex

Rex,

Thanks for your support so far. We are able to get 10G Copper SFP+  working.

Tested with board- to- board and with a 10G network Switch. :-)

I did try with the iperf utility to measure throughput. In even board-to-board (back to back connected with SFP+ cable) we are getting BW 1Gbps.

Did you run throughput test? Can you please share the method of that?

Kushal B. Patel

Ok. Thanks for the support so far.

Regards,

Kushal B. Patel

iperf version 2.0.5 (08 Jul 2010) pthreads
I try the command ,board-to-pc (back to back connected with SFP+ cable) only 484 MBytes/sec . I use TCP mode ,only 128MB/s.
Can you post more detail about 10Gpbs have only 1Gpbs .
and iperf version