J784S4XEVM: How to configurate the clock to use for 10G XFI

Part Number: J784S4XEVM

Tool/software:

Hello,

I am trying to use 10G on XFI, for that I want to use a clock frequency of 156.25MHz. The idea is to use the external clock and to use lane 2 of serdes 2 as XFI. If it is also possible to get 156.25MHz from an internal clock that would be good to know.

Here is what I have done so far : 

  • In k3-j784s4-exp1-exp2-usxgmii.dts, I have changed the speed from 5000 to 10000 :
    • &main_cpsw0_port1 {
      status = "okay";
      phy-mode = "usxgmii";
      mac-address = [00 00 00 00 00 00];
      phys = <&phy_gmii_sel_cpsw0 1>, <&serdes2_usxgmii_link>;
      phy-names = "portmode", "serdes-phy";
      fixed-link {
      speed = <10000>;
      full-duplex;
      };
      };
  • Still in k3-j784s4-exp1-exp2-usxgmii.dts, I checked that serdes2 lane2 was assigned, and as I see it is assigned by default : 
    • &serdes_ln_ctrl {
      idle-states = <J784S4_SERDES0_LANE0_PCIE1_LANE0>, <J784S4_SERDES0_LANE1_PCIE1_LANE1>,
      <J784S4_SERDES0_LANE2_IP3_UNUSED>, <J784S4_SERDES0_LANE3_USB>,
      <J784S4_SERDES1_LANE0_PCIE0_LANE0>, <J784S4_SERDES1_LANE1_PCIE0_LANE1>,
      <J784S4_SERDES1_LANE2_PCIE0_LANE2>, <J784S4_SERDES1_LANE3_PCIE0_LANE3>,
      <J784S4_SERDES2_LANE0_IP2_UNUSED>, <J784S4_SERDES2_LANE1_IP2_UNUSED>,
      <J784S4_SERDES2_LANE2_QSGMII_LANE1>, <J784S4_SERDES2_LANE3_QSGMII_LANE2>;
      };
  • In k3-j784s4-evm.dts, I added a node for serdes2 containing the frequency I want to use (156.25MHz) : 
    • &serdes2_refclk {
      clock-frequency = <156250000>;
      };*
  • In k3-j784s4-main.dtsi, I copied what existed with serdes for serdes2 : 
    •  / {
      serdes2_refclk: serdes-refclk {
      #clock-cells = <0>;
      compatible = "fixed-clock";
      };
      };
  • Still in k3-j784s4-main.dtsi, I replaced &serdes_refclk by &serdes2_refclk and switched the status to "okay" : 
    • serdes_wiz2: wiz@5020000 {
      compatible = "ti,j784s4-wiz-10g";
      #address-cells = <1>;
      #size-cells = <1>;
      power-domains = <&k3_pds 406 TI_SCI_PD_EXCLUSIVE>;
      clocks = <&k3_clks 406 2>, <&k3_clks 406 6>, <&k3_clks 406 5>, <&serdes2_refclk>;
      clock-names = "fck", "core_ref_clk", "core_ref1_clk", "ext_ref_clk";
      num-lanes = <4>;
      #reset-cells = <1>;
      #clock-cells = <1>;
      ranges = <0x05020000 0x0 0x05020000 0x00010000>;

      assigned-clocks = <&k3_clks 406 6>;
      assigned-clock-parents = <&k3_clks 406 10>;

      status = "okay";

  • Still in k3-j784s4-main.dtsiI also understood that I need to use &serdes2_refclk in assigned-clock-parents but I don't understand which PLL to use :
    • serdes2: serdes@5020000 {
      compatible = "ti,j721e-serdes-10g";
      reg = <0x05020000 0x00010000>;
      reg-names = "torrent_phy";
      resets = <&serdes_wiz2 0>;
      reset-names = "torrent_reset";
      clocks = <&serdes_wiz2 TI_WIZ_PLL0_REFCLK>,
      <&serdes_wiz2 TI_WIZ_PHY_EN_REFCLK>;
      clock-names = "refclk", "phy_en_refclk";
      assigned-clocks = <&serdes_wiz2 TI_WIZ_PLL0_REFCLK>,
      <&serdes_wiz2 TI_WIZ_PLL1_REFCLK>,
      <&serdes_wiz2 TI_WIZ_REFCLK_DIG>;
      assigned-clock-parents = <&k3_clks 406 6>,
      <&k3_clks 406 6>,
      <&k3_clks 406 6>;
      #address-cells = <1>;
      #size-cells = <0>;
      #clock-cells = <1>;

      status = "disabled";
      };
      };

Here are my questions : 

  • Following the method I just exposed, the remaining task is to assign &serdes2_refclk in the assigned-clock-parents field in red, is it the right way to do ?
  • Do we need to activate the external clock separately ? I imagine it should be activated in the u-boot source code, do you have an idea on its activation process ?

Thank you, 

Kind regards,

Wissam

  • Hi Wissam,

    First of all, please go through the serdes documentation here: https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-j784s4/10_00_00_08/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/SERDES/SERDES.html

    What you have done is modified the external clock where as the 156.25MHz clock is an internal clock.

    You would want to set "<&k3_clks 406 6>" to 156.25 by changing its parent to "<&k3_clks 406 9>" which can be configured to 156.25MHz like following;

    Still in k3-j784s4-main.dtsi, I replaced &serdes_refclk by &serdes2_refclk and switched the status to "okay" : 
    serdes_wiz2: wiz@5020000 {
    compatible = "ti,j784s4-wiz-10g";
    #address-cells = <1>;
    #size-cells = <1>;
    power-domains = <&k3_pds 406 TI_SCI_PD_EXCLUSIVE>;
    clocks = <&k3_clks 406 2>, <&k3_clks 406 6>, <&k3_clks 406 5>, <&serdes_refclk>;
    clock-names = "fck", "core_ref_clk", "core_ref1_clk", "ext_ref_clk";
    num-lanes = <4>;
    #reset-cells = <1>;
    #clock-cells = <1>;
    ranges = <0x05020000 0x0 0x05020000 0x00010000>;
    
    assigned-clocks = <&k3_clks 406 6>;
    assigned-clock-parents = <&k3_clks 406 9>;
    
    status = "okay";

    This should work for and set the parent of one of the ref clock to 156.25MHz.

    Please verify this by running "k3conf dump clock 406" and seeing that clock ID 9 is indeed 156.25MHz and clock ID 5 is 100MHz

    Regards,
    Tanmay

  • Hi Tanmey,

    Thank you for your reply, I have set assigned-clock-parents = <&k3_clks 406 9> like you mentioned above, but it is still not working. When I run "k3conf dump clock 406", clock ID 5 is indeed 100MHz but I don't find the value 156.25MHz for clock ID 9 : 

    Is there another configuration to assign 156.25MHz to clock ID 9 ?

    Thank you,

     

    Kind regards,

    Wissam

  • Hello,

    I had the occasion to work on it, and for our case of using 10GB with XFI I was confused on how to initialize the clocks and the switch port.

    I understand that the ethernet interface should be initialised in k3-j784s4-exp1-exp2-usxgmii.dts, but we are also using ethfw in which core R5F MCU2_0 is working as server. I saw that the CPSW9G switch, the serdes clocks and PHY were configured by the core R5F MCU2_0 thanks to this guide :Ethernet Firmware: Porting Guide.

    In this case, should I do the clocks/switch configuration on the R5F side ? I guess that it can create an issue if it is configured at 2 different locations.

    Thank you,

    Kind regards,

    Wissam

  • Hi Wissam,

    If you are using using ethfw, "k3-j784s4-exp1-exp2-usxgmii.dts" this is not the device-tree to use. You should use the "arch/arm64/boot/dts/ti/k3-j784s4-evm-virt-mac-client.dtso" for using CPSW with ethfw.

    In this case, the configuration for the clocks and serdes should happen from Ethfw.

    Regards,
    Tanmay

  • Hello Tanmay,

    I tried to do the configuration within ethfw. For that I tried to use Ethernet Firmware: Porting Guide. As I understood, I need to follow these steps : 

    1. Ethernet port configuration : In Board_j784s4_evm.c, I attributed ports 1 and 2 to USXGMII 10G and called gEthFw_macPortCfg in findportcfg

    In pdk/packages/ti/board/src/j784s4_evm/board_ethernet_config.c :

    2. Serdes configuration : In Board_j784s4_evm.c, I called the function Board_serdesCfgUsxgmii instead of Board_serdesCfgQsgmii, and I set the clock of serdes 2 to 156.25MHz :

    In pdk/packages/ti/board/src/j784s4_evm/board_serdes_cfg.c, I selected the internal clock of 156.25MHz and a linkrate of 10p3125G

    After those configurations, it seems there is still a problem, as the core R5F is not able to boot correctly anymore, these are the only logs I get from A72 UART for R5F : 

    An error also appears from R5F UART :

     

    I noticed that this error appeared especially after I initialized the serdes to USXGMII.

    Based on that, my question is how to initialize the serdes 2 for XFI/USXGMII 10G.

    Thank you,

    Kind regards,

    Wissam

  • Hi,

    The changes made in Board_j784s4_evm.c are fine for
    1. Enabling Ports in XGMII mode
    2. Configuring SerDes clock to 156.25MHz clock.
    3. Invoking SerDes USXGMII configuration.


    Also, Changes made in  pdk/packages/ti/board/src/j784s4_evm/board_serdes_cfg.c are also fine for enabling UXSGMII in 10Gbps.
    By default Lane 2 only enabled in SerDes configuration.


    Also, changes in pdk/packages/ti/board/src/j784s4_evm/board_ethernet_config.c file also fine for enabling CPSW Ports in USXGMII Mode & SGMII Mode.
    But, by default port-2 is not enabled from application main.c file under "
    gEthAppPorts" array.

    An error also appears from R5F UART :

     

    I noticed that this error appeared especially after I initialized the serdes to USXGMII.

    By default ETHFW will wait for Link Up of ports enabled for SGMII_FORCEDLINK.
    As it is waiting for Ports Link Up, remote services were not initialized so connection to client will not establish.

    Please make sure that USXGMII PHY connected to SerDes also powered and Link up with link partner side at 10Gbps speed.

    Best Regards,
    Sudheer

  • Hello Sudheer, 

    Thank you very much for your reply. Here is my update:

    For information : From the TRM, we saw that only Serdes2 lane 2 and lane 3 are compatible with XFI 10G. So the hardware team decided to connect a 10G cable directly on the serdes 2 lane 2 at the bottom of the EVM, which means that we removed the Qenet daughter card.

    • Knowing that we want to use Serdes 2 lane 2, I understood from the TRM that Serdes 2 lane 2 is related to Enet port 1, so based on that I succeeded to intialize the port 1. We can see that the register's value is 5, which is the expected value for USXGMII/XFI : 

    • Then, in board_serdes_cfg.c, I initialised laneNum and laneMask like this. I gave laneNum = BOARD_SERDES_SGMII_ENET1_LANE_NUM and laneMask = BOARD_SERDES_SGMII_ENET1_LANE_MASK, the value BOARD_SERDES_SGMII_ENET1_LANE_NUM being defined to 2U in board_serdes_cfg.h. Is it the right way to do it ? Here is the screenshot 

     Then I would like to check from the serdes registers, which one is selected and ideally the clock frequency related. I tried to see                         SERDES2_LN2_CTRL_LANE_FUNC_SEL at address 0x001040A8 but the value is always 0 as shown above, and I didn't find a way to check the frequency.

    • I also investigated on why the R5F is not booting correctly, and indeed it seems like the R5F is stuck in a while loop trying to find the port linked : 

    Based on this issue, it seems to USXGMII PHY is not found like you mentioned. I guess it can also come from the Serdes configuration, because it is possible that we expect USXGMII PHY on the wrong Serdes if the intialization is not correct. That's why it would be useful to check the Serdes configuration and registers as well.

    Thank you,

    Kind regards,

    Wissam

  • Hi,

    Knowing that we want to use Serdes 2 lane 2, I understood from the TRM that Serdes 2 lane 2 is related to Enet port 1, so based on that I succeeded to intialize the port 1. We can see that the register's value is 5, which is the expected value for USXGMII/XFI : 

    Yes, Port-1 is XFI/USXGMII.

    Then, in board_serdes_cfg.c, I initialised laneNum and laneMask like this. I gave laneNum = BOARD_SERDES_SGMII_ENET1_LANE_NUM and laneMask = BOARD_SERDES_SGMII_ENET1_LANE_MASK, the value BOARD_SERDES_SGMII_ENET1_LANE_NUM being defined to 2U in board_serdes_cfg.h. Is it the right way to do it ? Here is the screenshot 

    This is fine to enable Lane2 of SerDes.

    Then I would like to check from the serdes registers, which one is selected and ideally the clock frequency related. I tried to see                         SERDES2_LN2_CTRL_LANE_FUNC_SEL at address 0x001040A8 but the value is always 0 as shown above, and I didn't find a way to check the frequency.

    SerDes Lane CTRL doesn't provide information about SerDes clock.
    It provides the IP selected for SerDes Lane. It should be 1 for SERDES2_LN2
    Can you check the value of "BOARD_SERDES_LANE_SELECT_CPSW" is 1 or not to select IP2 on SEDES2 LN2 i.e. Port-1. If it is 0 means IP1 i.e. Port-7.

    I also investigated on why the R5F is not booting correctly, and indeed it seems like the R5F is stuck in a while loop trying to find the port linked

    Yes, It will till Link Up of Port, it will be resolved by negotiating PHY & MAC.
    The Nophy mode is based on MAC Port configuration with PHY ADDR as Invalid i.e. kind of MAC2MAC connection. PHY is boot strapped and enabled for USXGMII Mode.

    Please check your PHY supports Auto-negotiation or not? Also, SerDes clock configuration as 156.25MHz.

    Best Regards,
    Sudheer

  • Hello,

    For the Serdes confguration, I checked the value of BOARD_SERDES_LANE_SELECT_CPSW, and indeed it is set to 1, and the structure parameter serdesLaneEnableParams.phyInstanceNum is set to 1 like expected.

    The program apparently takes into account the serdes frequency of 156.25MHz, 

    But I don't see the register SERDES2_LN2_CTRL_LANE_FUNC_SEL to 1 like expected, so I guess that Serdes 2 lane 2 is not selected.

     

    Is there another change I can do to select Serdes2 lane2 ?

    Thank you,

    Kind regards,

    Wissam

  • Hello again, I had a feedback from hardware team regarding the auto-negociation of the PHY connected to the Serdes, here is the answer : 

    We don’t have a PHY component (as in base-T phy), we have an SFP module supporting XFI. There is therefore no negotiation; it will convert electrical XFI to optical as long as the bitrate is between 9.95 Gb/s and 10.5 Gb/s.

     

    The blackbox module’s reference is LGC220A.

    Kind regards,

    Wissam

  • Hi,

    SerDes configuration invokes CSL_serdesIPSelect() API for SerDes lane selection. It will configure SerDes Lane control.

    Also, make sure that SerDes2 is not configured from u-boot & Linux.

    Also, share details of SDK version you are using?

    Best Regards,
    Sudheer

  • Hello, 

    I tried to comment the loop in the following screenshot where the code was stuck trying to find a PHY :

    As a result, the code completes the initialization and the scheduler starts, and we get in the idle task, but I have several failed : 

    Also, I don't understand why MAC port 1 is searching for a PHY even if we attrbuted .phyaddress = ENETPHY_INVALID_PHYADRRESS

    I can double check if the code goes to CSL_serdesIPSelect(), because SERDES2_LN2_CTRL_LANE_FUNC_SEL is still 0.

    I am using CCS' debugger to go through the code after the boot is complete, so I think the ethfw's Serdes configuration should erase the one from u-boot, but I am not sure. I can still deactivate it from u-boot, can you confirm from which file I can do it ?

    I'm using SDK RTOS ti-processor-sdk-rtos-j784s4-evm-09_01_00_06 and SDK Linux ti-processor-sdk-linux-j784s4-evm-08_06_01_02.

    Kind regards,

    Wissam

  • Hi,

    I'm using SDK RTOS ti-processor-sdk-rtos-j784s4-evm-09_01_00_06 and SDK Linux ti-processor-sdk-linux-j784s4-evm-08_06_01_02.

    There was an issue in SDK 9.1 for SerDes2 Lane configuration for XFI/USXGMII. Please make following change in pdk/packages/ti/csl/src/ip/serdes_cd/v3/csl_serdes3.c file and rebuild pdk libs.



     Note:
    1. 9.1 SDK ETHFW is not compatible with 8.6 Linux ETHFW client. Please migrate Linux SDK also 9.1.
    2. Above Lane selection issue addressed in 9.2 SDK.

    Best Regards,
    Sudheer

  • Hello,

    Thank you for your reply,

    • would you suggest to upgrade the SDK RTOS to ti-processor-sdk-rtos-j784s4-evm-10_00_00_05, and SDK Linux to ti-processor-sdk-linux-adas-j784s4-evm-10_00_00_08 ?
    • Was XFI 10G tested with these 2 SDK 10 ?

    If you judge that SDK RTOS ti-processor-sdk-rtos-j784s4-evm-09_02_00_05 and SDK Linux ti-processor-sdk-linux-adas-j784s4-evm-09_01_00_06 are more accurate then we can use it. Also, do you suggest to install equivalent SDK versions, for example SDK RTOS 9.2 and SDK Linux 9.2 or SDK RTOS 9.1 and SDK Linux 9.1 ?

    Also, it is not clear for me why SDK Linux needs to be upgraded, is it because in previous SDK Linux the Serdes are selected from u-boot ?

    Thank you,

    Kind regards,
    Wissam

  • Hi,

    • would you suggest to upgrade the SDK RTOS to ti-processor-sdk-rtos-j784s4-evm-10_00_00_05, and SDK Linux to ti-processor-sdk-linux-adas-j784s4-evm-10_00_00_08 ?
    • Was XFI 10G tested with these 2 SDK 10 ?

    Yes, it was tested in 10.0

    If you judge that SDK RTOS ti-processor-sdk-rtos-j784s4-evm-09_02_00_05 and SDK Linux ti-processor-sdk-linux-adas-j784s4-evm-09_01_00_06 are more accurate then we can use it

    9.1 has above issue due to which we can't validate XFI from SerDes2, but we can validate from SerDes1.
    Also, with above change you can validate on SerDes2 as well.

    All SDKs work for XFI/USXGMII, we need make SerDes configuration & clock as needed. 

    Also, do you suggest to install equivalent SDK versions, for example SDK RTOS 9.2 and SDK Linux 9.2 or SDK RTOS 9.1 and SDK Linux 9.1 ?

    Yes, this how TI is evaluating SDKs on TI EVMs.

    Also, it is not clear for me why SDK Linux needs to be upgraded, is it because in previous SDK Linux the Serdes are selected from u-boot ?

    Not about SerDes configuration, Some enhancements to drivers and bug fixes happen in every SDK.
    Also, when you want to use CPSW9G with ETHFW Server on MCU2_0 core and Linux A72 client, then you need to migrate Linux SDK also to latest for the client & server communication as the protocol has modified from 8.6 to 9.1.

    Best Regards,
    Sudheer

  • Hello,

    Thank you, I upgraded my SDK Linux, now this is my setup : 

    • ti-processor-sdk-rtos-j784s4-evm-09_01_00_06 with your change

    • ti-processor-sdk-linux-adas-j784s4-evm-09_01_00_06

    Thanks to that, I can see from register that Serdes2 lane2 is selected : 

    Note that this register value changed first without applying the above change, then I still applied it to be synchronised.

    I also checked the frequency of the clock I'm using for Serdes2 lane2 (TISCI_DEV_SERDES_10G2_CORE_REF_CLK), and I have 156250000

    I also see that the port 1 10G is up : 

    Now we have : 

    • CTRL_MMR_CFG0_CPSW2_ENET1_CTRL_PROXY  = 0x5 which means port 1 is set for USXGMII/XFI
    • CTRL_MMR_CFG0_SERDES2_LN2_CTRL = 0x1 which means Serdes2 lane2 is selected
    • TISCI_DEV_SERDES_10G2_CORE_REF_CLK is set to 156250000 based on information from Linux

    But ping is still not a success, as I'm trying to ping the R5F from an external 10G pc.

    Do I need to do/check something else to be able to ping?

    I also have a question from hardware team : 

    Can we configure the SERDES to generate and check a test pattern (such as PRBS31), so we can exclude any hardware/connection issue ?

    We can hook up a BER tester facing the SERDES to monitor any error.

    Thank you,

    Kind regards,

    Wissam

  • HI,

    Do I need to do/check something else to be able to ping?

    Can you check by dumping CPSW statistics any Tx or Rx good frames from 10G Port.
    Please refer to FAQ, and integrate the patch to dump CPSW statistics.
    https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1220277/faq-tda4vm-how-to-get-cpsw-ethernet-statistics-on-tda4-dra8-devices

    I also have a question from hardware team : 

    Can we configure the SERDES to generate and check a test pattern (such as PRBS31), so we can exclude any hardware/connection issue ?

    We can hook up a BER tester facing the SERDES to monitor any error.

    Can you post a new query for this, will be addressed by H/W expert.

    Best Regards,
    Sudheer