10G XFI communication slow / missing documentation on SERDES and netcpx

Olaf,Thanks for using our products.
Please check the technical documents section: http://www.ti.com/product/66AK2E02/technicaldocuments
there is one document about Clocks and other about SERDES. Please let us know if you need anything else.

Hi Rogerio,

we already found and read those documents (including the one on the 10G subsystem). Unfortunately my answers from my previous post are not answered in there. Please find them here again:

• We setup our core clock at 1,25GHz to match the device's spec. From that, we derived the netcp clock (350Mhz) via PASS_PLL. Is that clock automatically applied to the 10G complex, too?
• At what clock rate should the 10G subsystem run?
• And how is the SERDES config clock domain and SERDES clock derived?

Can you please answer those answers?

Best regards,

Olaf

Hello Olaf,

The NETCP clock is only applicable for GbE module and not for 10GbE module.

For your other questions, you can refer section 2.1 Clock Control in 10 Gigabit Ethernet Subsystem User Guide.

http://www.ti.com/lit/ug/spruhj5/spruhj5.pdf

Regards,

Senthil

Dear Senthil,

unfortunately the clocks section in the document you referred to is only a single page with rough information.

In the document http://www.ti.com/lit/ug/spruho3/spruho3.pdf setting up the SERDES clock is described. Can you please provide information on how this is mapped to the Linux Kernel device tree? (https://git.ti.com/keystone-linux/linux/blobs/master/arch/arm/boot/dts/k2e-net.dtsi)

Best regards,

Olaf

Hi Olaf,

In DTS file, the '3' is the property value of SERDES reference clock.

It is just a number only, if 3 is chosen and SERDES would call & setup the required registers to achieve the 156.25MHz ref clock for SERDES peripheral.

If you want to run SERDES at 100MHz then set the property value to "0"

    cpsw: cpsw@24200000 {
        cpsw-ss-reg = <0x24200000 0x40000>;
        label = "keystone-cpsw2";

        /* serdes configuration registers */
        serdes_reg = <0x0232a000 0x2000
                  0x02324000 0x2000>;
        serdes_at_probe = <1>;

        /* Number of SerDes */
        num-serdes = <2>;
        /* Max number of lanes per each SerDes */
        serdes-lanes = <4>;
        /* SerDes reference clock 156.25 MHz */
/*         serdes-ref-clk = <3>; */
        /* SerDes baud rate 5GBaud */

        /* SerDes reference clock 100 MHz */
         serdes-ref-clk = <0>;

        serdes-baud-rate = <1>;
        /* SerDes quarter rate mode */
        serdes-rate-mode = <2>;
        /* SerDes loopback */
        serdes-loopback = <0>;
        /* SerDes PHY interface: SGMII */
        serdes-phy-intf = <0>;

Please refer to the following files.

<keystone linux source>/drivers/net/ethernet/ti/keystone_serdes.h

/* SERDES Reference clock */

enum SERDES_CLOCK {

SERDES_CLOCK_100M, /* 100 MHz */

SERDES_CLOCK_122P88M, /* 122.88 MHz */

SERDES_CLOCK_125M, /* 125 MHz */

SERDES_CLOCK_156P25M, /* 156.25 MHz */                      ----> 3

SERDES_CLOCK_312P5M, /* 312.5 MHz */

};

drivers/net/ethernet/ti/keystone_ethss2.c

ret = of_property_read_u32(node, "serdes-ref-clk",

  &cpsw_dev->serdes.clk);

if (ret < 0) {

dev_err(dev, "missing serdes-ref-clk parameter\n");

cpsw_dev->serdes.clk = SERDES_CLOCK_156P25M;

}

dev_dbg(dev, "serdes-ref-clk %u\n", cpsw_dev->serdes.clk);

 

./drivers/net/ethernet/ti/keystone_serdes.c

int serdes_init(void __iomem *serdes_regs, struct serdes *serdes, u32 num_lanes)
{
    int i, ret = 0;

    /* The driver currently supports 5GBaud rate with ref clock 156.25MHz */
    if (serdes->clk == SERDES_CLOCK_156P25M)
        if (serdes->rate == SERDES_RATE_5G)
            ret = serdes_init_156p25m_5g(serdes_regs, num_lanes);
        else if (serdes->rate == SERDES_RATE_10p3125g)
            ret = serdes_init_156p25m_10p3125g(serdes_regs,
                               num_lanes);
        else
            return -EINVAL;

I hope this helps.

Hi,

you described the SERDES path for 1G, but I was asking for 10G. Can you please provide the correct configuration and pointers into the code for the "cpswx" device tree entry?

Thanks in advance,

Olaf

Hi,

MCSDK3.0 have 10Ge CSL drive support, Please take a look at "csl_serdes_10ge" file on mcsdk package.
MCSDK Path: \ti\pdk_keystone2_3_01_01_04\packages\ti\csl\src\ip\serdes_sb\V0\

Thanks,

Sorry for the misinterpretation.

10G ethernet drivers are available in keystone linux.

drivers/net/ethernet/ti/keystone_xgess.c

By default it was disabled. If you want to enable it, do the following change.

Comment it out the "status" entry in net dts.

arch/arm/boot/dts/k2e-evm.dts

netcpx: netcp@2f00000 {
#address-cells = <1>;
#size-cells = <1>;
reg = <0x2f00000 0xa0600>;
compatible = "ti,keystone-netcp";
dma-coherent;
/* status = "disabled"; Modified */

drivers/net/ethernet/ti/keystone_xgepcsr.c

int xge_serdes_init(struct device_node *node)
------
if (of_property_read_u32(node, "ref_clock",
&hw->ref_clock_rate))
hw->ref_clock_rate = K2SERDES_REF_CLOCK_156P25M;

if (of_property_read_u32(node, "link_rate",
&hw->link_rate))
hw->link_rate = K2SERDES_LINK_RATE_10P3125G;

Now re-build the dts file and boot the board.

Please refer to 10GbE driver in User's Guide, http://processors.wiki.ti.com/index.php/MCSDK_UG_Chapter_Exploring#10Gig_Ethernet_Driver . You may want to get the latest kernel code.

Rex

Hello,

we still need support to bring up the 10G XFP-Module for our custom board.

The connection is always slow.

I debugged the serializer and get following information:

The LN0_10G_Linked Bit in the LANExCTL_STS register which indicates that the 10G MAC/PCS layer has link is zero.

Is that right?

Can you help us, how can we debug the connection (for example can we get information via the registers).

Best regards,

Sebastian

Hi, Olaf,

PASS_PLL only provides clock to Packet Accelerator SubSystem, not to XFI. For the SerDes, it is 156.25MHz and from clock generator which is CDCM6208 on TI K2E EVM. Please refer to TI K2E EVM schematics. For NetCP, it is 100MHz using refclkmain. To debug it, you should be able to read the corresponding SGMII status register. I am not sure what port numbers they are on your custom board, but on TI EVM, they are port 9 and 10 (eth8 and eth9). LOCK (bit 4) should be PLL locked and LINK (bit 0) should be up.

Rex

Hello Rex,

why you talk about the SGMII register. The SGMII register are only for 1G support or?

I talk about the serdes config register. I get Link up and PLL locked nevertheless the connection is slow.

How can I debug the interface to get information what is wrong?

Best regards,

Sebastian

 

Sebastian,
Our factory apps team is looking into this issue here...sorry for the delay.

Hi all,

we now have the 10g xfi to copper phy (mv88x3120) up and running.
In case of speed we have nearly the same experience as with the xfp module.
We have testes three transfer types:
In (PC => Keystone2), Out(Keystone2 => PC), In+Out(Keystone=>PC + PC=>Keystone at the same time).

We got the following results:
In: 1.62GBit/s
Out: 1.62GBit/s
In+Out: 1.62GBit/s (sum of in + out)

We find it very strange, that the sum of in + out also equals 1.62GBit/s. If XFI is the limmiting factor, shouldn't we get 1.62GBit/s per direction (a sum of 3.24GBits/s)?

Can you please point us to the right chapters in the documentation, that shows the whole clock network including all plls and all post dividers including the plls and dividers that may exist in the peripherals. We are missing the nessesary information in the documentation to understand how the clock for the 10GbE Subsystem is derived (see spruhj5: 2.1.1 10GbE Subsystem Clock & SerDes Configuration Clock).

Best regards,
Michael

Sebastian,

Our factory apps are working on this issue and will provide answer on E2E as soon as we can. Sorry for the delay.

Hi, Michael,

I meant to get the clock and PLL divider info if we have them before replying to this thread, but I have not get hold of any info yet.

Our experience on 10GbE is that the jumbo packet needs to be enabled to get throughput higher. We needed to configure the MTU  up to around 8500 and iperf sending packets of similar size. You may want to tune them so that no fragmentation happens.

Rex

HI, Michael,
A PLL calculator tool is being created for K2E. It will be ready next month. Currently, there isn't a list of all available PLL clock dividers for all peripheral.
Rex

Hi Rex,

Thank you for your effort, but the calculator tool does not help us to understand and debug the clock paths.

We really do need a documentation about the clock network. This is much more important to us than having a PLL calculator tool.

Michael

Hi, Michael,

If you have not looked at the K2E data sheet,  table 10-7 which lists the clock domains, and the PLL User's Guide in  66Ak2E05 page, http://www.ti.com/product/66AK2E05/technicaldocuments, you may find some info. If you have looked into these docs already, they will be the ones I know.

Rex

Hi Rex,

We have looked at that table many times. As you mentioned, it is only a list of the clock domains.

What we need is a schematic like overview that shows us all clock inputs into the Keystone2, how they are connected to the PLL's and the peripherals, how the clock outputs of the PLL's are connected to the Peripherals and dividers inside the peripherals, for those peripherals that have their own clock dividers.

We would like to have one overview diagram that shows all clocks.

Figure 10-7 from "SPRS865B - January 2014" is a good start but wee need that for all components involved in the clock network as well as all interconnects.

Michael

Michael,

As Rex mentioned the clock tree spreadsheet will help. We are working hard to make it available in the next couple of weeks. Our apps team is confident this tool will help your team to better understand how it works..