This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

EVMK2H: Trouble using SGMII 2 (eth2) in Linux

Part Number: EVMK2H

I’m having an eerily simmilar problem to the one described here:
https://e2e.ti.com/support/processors/f/791/t/728803?tisearch=e2e-quicksearch&keymatch=Linux/EVMK2H:%20receive%20doesn%27t%20work%20on%20eth2%20and%20eth


I have a custom board modeled on the EVMK2H and I’m trying to update to a newer kernel version using Yocto. The board has three Ethernet interfaces (eth0, eth1, and eth2). Prior to updating, everything worked fine. Since updating, Linux does not recognize eth3. However U-Boot still works fine with all three interfaces.

Linux can no longer receive packets on eth2 (SGMII 2). Running Wireshark, I have determined that the board is transmitting packets as expected over eth2.

Also, the netcp seems to be getting packets. Examining the ALE table shows an entry for “a0:ce:c8:02:7a:f5” which is the MAC of the laptop running Wireshark.

root@xyz:~# cat /sys/devices/platform/soc@0/2620110.netcp/ale_table
index 0, raw: 0000007c d000ffff ffffffff, type: addr(1), addr: ff:ff:ff:ff:ff:ff, mcstate: f(3), port mask: 1f, no super
index 1, raw: 00000000 10004c24 9814e3b6, type: addr(1), addr: 4c:24:98:14:e3:b6, uctype: persistent(0), port: 0
index 2, raw: 0000007c d0003333 00000001, type: addr(1), addr: 33:33:00:00:00:01, mcstate: f(3), port mask: 1f, no super
index 3, raw: 0000007c d0000100 5e000001, type: addr(1), addr: 01:00:5e:00:00:01, mcstate: f(3), port mask: 1f, no super
index 4, raw: 0000007c d0003333 ff14e3b6, type: addr(1), addr: 33:33:ff:14:e3:b6, mcstate: f(3), port mask: 1f, no super
index 5, raw: 00000008 d00000e0 4c0004ee, type: addr(1), addr: 00:e0:4c:00:04:ee, uctype: touched(3), port: 2
index 6, raw: 0000007c d0003333 00000202, type: addr(1), addr: 33:33:00:00:02:02, mcstate: f(3), port mask: 1f, no super
index 7, raw: 0000007c d0003333 00010003, type: addr(1), addr: 33:33:00:01:00:03, mcstate: f(3), port mask: 1f, no super
index 8, raw: 0000007c d0000100 5e0000fc, type: addr(1), addr: 01:00:5e:00:00:fc, mcstate: f(3), port mask: 1f, no super
index 9, raw: 00000000 10000218 317e3e6f, type: addr(1), addr: 02:18:31:7e:3e:6f, uctype: persistent(0), port: 0
index 10, raw: 0000007c d0003333 ff7e3e6f, type: addr(1), addr: 33:33:ff:7e:3e:6f, mcstate: f(3), port mask: 1f, no super
index 16, raw: 00000008 d00000e0 4c0006bd, type: addr(1), addr: 00:e0:4c:00:06:bd, uctype: touched(3), port: 2
index 17, raw: 00000008 d00054b2 0397cda7, type: addr(1), addr: 54:b2:03:97:cd:a7, uctype: touched(3), port: 2
index 18, raw: 0000000c d000a0ce c8027af5, type: addr(1), addr: a0:ce:c8:02:7a:f5, uctype: touched(3), port: 3
[0..1023]: 14 entries


Looking at the qmss debug subsystem, it seems that the RX completion queue is not receiving packets from eth2.

root@xyz:~# cat /sys/kernel/debug/qmss
soc@0:qmss@2a40000: 0-16383
        queue id 8705 (rx-compl-eth1)
                handle 213a52e4: pushes        0, pops    17868, count        0, notifies     7644, push errors        0, pop errors        0
        queue id 8707 (tx-compl-eth1)
                handle bbfaf08b: pushes        0, pops    11785, count        0, notifies    10168, push errors        0, pop errors        0
        queue id 8708 (rx-compl-eth2)
                handle 5e5236e5: pushes        0, pops        0, count        0, notifies        0, push errors        0, pop errors        0
        queue id 8709 (tx-compl-eth2)
                handle 8aa539ad: pushes        0, pops       21, count        0, notifies       20, push errors        0, pop errors        0
        queue id 4000 (rx-pool-eth1)
                handle 8bc7bac0: pushes    18892, pops    18124, count      768, notifies        0, push errors        0, pop errors        0
        queue id 4001 (tx-pool-eth1)
                handle 8895959d: pushes    20234, pops    19210, count     1024, notifies        0, push errors        0, pop errors        0
        queue id 4002 (rx-fdq-eth1-0)
                handle a7fa8a4d: pushes    17996, pops        0, count      128, notifies        0, push errors        0, pop errors        0
        queue id 4003 (rx-fdq-eth1-1)
                handle e5acb7db: pushes      128, pops        0, count      128, notifies        0, push errors        0, pop errors        0
        queue id 4004 (rx-pool-eth2)
                handle 6e82bca4: pushes     1024, pops      256, count      768, notifies        0, push errors        0, pop errors        0
        queue id 4005 (tx-pool-eth2)
                handle c2e4629c: pushes     1045, pops       21, count     1024, notifies        0, push errors        0, pop errors        0
        queue id 4006 (rx-fdq-eth2-0)
                handle 0de08cbe: pushes      128, pops        0, count      128, notifies        0, push errors        0, pop errors        0
        queue id 4007 (rx-fdq-eth2-1)
                handle 332e61a5: pushes      128, pops        0, count      128, notifies        0, push errors        0, pop errors        0
        queue id 648 (tx-pipe-2620110)
                handle 3e1e17e6: pushes    11806, pops        0, count        0, notifies        0, push errors        0, pop errors        0

root@xyz:~# cat /sys/kernel/debug/knav_dma
dma_gbe : max_tx_chan: (9), max_rx_flows: (32)
        tx chan 8:      einfo - 0, pswords - 0, priority - 2
        rx flow 23:     einfo - 1, psinfo - 1, desc_type - 0
                        dst_q: [8705], thresh: 0 fdq: [4002][4003][4003][4003]
        rx flow 24:     einfo - 1, psinfo - 1, desc_type - 0
                        dst_q: [8708], thresh: 0 fdq: [4006][4007][4007][4007]

Here is my netcp device tree file:

/*
 * Device Tree Source for Keystone 2 Hawking Netcp driver
 *
 * Copyright 2015 Texas Instruments, Inc.
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License version 2 as
 * published by the Free Software Foundation.
 */

qmss: qmss@2a40000 {
	compatible = "ti,keystone-navigator-qmss";
	dma-coherent;
	#address-cells = <1>;
	#size-cells = <1>;
	clocks = <&chipclk13>;
	ranges;
	queue-range	= <0 0x4000>;
	linkram0	= <0x100000 0x8000>;
	linkram1	= <0x0 0x10000>;

	qmgrs {
		#address-cells = <1>;
		#size-cells = <1>;
		ranges;
		qmgr0 {
			managed-queues = <0 0x2000>;
			reg = <0x2a40000 0x20000>,
			      <0x2a06000 0x400>,
			      <0x2a02000 0x1000>,
			      <0x2a03000 0x1000>,
			      <0x23a80000 0x20000>,
			      <0x2a80000 0x20000>;
			reg-names = "peek", "status", "config",
				    "region", "push", "pop";
		};

		qmgr1 {
			managed-queues = <0x2000 0x2000>;
			reg = <0x2a60000 0x20000>,
			      <0x2a06400 0x400>,
			      <0x2a04000 0x1000>,
			      <0x2a05000 0x1000>,
			      <0x23aa0000 0x20000>,
			      <0x2aa0000 0x20000>;
			reg-names = "peek", "status", "config",
				    "region", "push", "pop";
		};
	};

	queue-pools {
		qpend {
			qpend-0 {
				qrange = <658 8>;
				interrupts =<0 40 0xf04 0 41 0xf04 0 42 0xf04
					     0 43 0xf04 0 44 0xf04 0 45 0xf04
					     0 46 0xf04 0 47 0xf04>;
			};
			qpend-1 {
				qrange = <8704 16>;
				interrupts = <0 48 0xf04 0 49 0xf04 0 50 0xf04
					      0 51 0xf04 0 52 0xf04 0 53 0xf04
					      0 54 0xf04 0 55 0xf04 0 56 0xf04
					      0 57 0xf04 0 58 0xf04 0 59 0xf04
					      0 60 0xf04 0 61 0xf04 0 62 0xf04
					      0 63 0xf04>;
				qalloc-by-id;
			};
			qpend-2 {
				qrange = <8720 16>;
				interrupts = <0 64 0xf04 0 65 0xf04 0 66 0xf04
					      0 59 0xf04 0 68 0xf04 0 69 0xf04
					      0 70 0xf04 0 71 0xf04 0 72 0xf04
					      0 73 0xf04 0 74 0xf04 0 75 0xf04
					      0 76 0xf04 0 77 0xf04 0 78 0xf04
					      0 79 0xf04>;
			};
		};
		general-purpose {
			gp-0 {
				qrange = <4000 64>;
			};
			netcp-tx {
				qrange = <640 9>;
				qalloc-by-id;
			};
			netcpx-tx {
				qrange = <8752 8>;
				qalloc-by-id;
			};
		};
		accumulator {
			acc-low-0 {
				qrange = <480 32>;
				accumulator = <0 47 16 2 50>;
				interrupts = <0 226 0xf01>;
				multi-queue;
				qalloc-by-id;
			};
		};
	};

	descriptor-regions {
		#address-cells = <1>;
		#size-cells = <1>;
		ranges;
		region-12 {
			id = <12>;
			region-spec = <8192 128>;	/* num_desc desc_size */
			link-index = <0x4000>;
		};
	};

	pdsps {
		#address-cells = <1>;
		#size-cells = <1>;
		ranges;
		pdsp0@0x2a10000 {
			reg = <0x2a10000 0x1000    /*iram */
			       0x2a0f000 0x100     /*reg*/
			       0x2a0c000 0x3c8	   /*intd */
			       0x2a20000 0x4000>;  /*cmd*/
			id = <0>;
		};
	};
}; /* qmss */

knav_dmas: knav_dmas@0 {
	compatible = "ti,keystone-navigator-dma";
	clocks = <&papllclk>;
	#address-cells = <1>;
	#size-cells = <1>;
	ranges;
	ti,navigator-cloud-address = <0x23a80000 0x23a90000
				   0x23aa0000 0x23ab0000>;

	dma_gbe: dma_gbe@0 {
		reg = <0x2004000 0x100>,
			  <0x2004400 0x120>,
			  <0x2004800 0x300>,
			  <0x2004c00 0x120>,
			  <0x2005000 0x400>;
		reg-names = "global", "txchan", "rxchan",
				"txsched", "rxflow";
		ti,enable-all;
	};
};

gbe_serdes: phy@232a000 {
	compatible		= "ti,keystone-serdes-gbe";
	reg			= <0x0232a000 0x2000>;
	status			= "ok";
	link-rate-kbps		= <1250000>;
	num-lanes		= <4>;
	#address-cells	= <1>;
	#size-cells	= <0>;

	serdes_lane0: lane@0 {
		#phy-cells	= <0>;
		reg		= <0>;
		status		= "ok";
		control-rate	= <2>;
		rx-start	= <7 5>;
		rx-force	= <1 1>;
		tx-coeff	= <0 0 0 12 4>;
	};
	serdes_lane1: lane@1 {
		#phy-cells	= <0>;
		reg		= <1>;
		status		= "ok";
		control-rate	= <2>;
		rx-start	= <7 5>;
		rx-force	= <1 1>;
		tx-coeff	= <0 0 0 12 4>;
	};
	serdes_lane2: lane@2 {
		#phy-cells	= <0>;
		reg		= <2>;
		status		= "ok";
		control-rate	= <2>;
		rx-start	= <7 5>;
		rx-force	= <1 1>;
		tx-coeff	= <0 0 0 12 4>;
	};
	serdes_lane3: lane@3 {
		#phy-cells	= <0>;
		reg		= <3>;
		status		= "disabled";
		control-rate	= <2>;
		rx-start	= <7 5>;
		rx-force	= <1 1>;
		tx-coeff	= <0 0 0 12 4>;
	};
};

netcp: netcp@2000000 {
	reg = <0x2620110 0x8>;
	reg-names = "efuse";
	compatible = "ti,netcp-1.0";
	#address-cells = <1>;
	#size-cells = <1>;

	/* NetCP address range */
	ranges  = <0 0x2000000 0x100000>;

	clocks = <&clkpa>, <&clkcpgmac>, <&chipclk12>;
	clock-names = "pa_clk", "ethss_clk", "cpts";
	dma-coherent;

	ti,navigator-dmas =
			<&dma_gbe 22>, /* rx flow for CPSW port 0 */
			<&dma_gbe 23>, /* rx flow for CPSW port 1 */
			<&dma_gbe 8>,  /* CPSW tx channel */
			<&dma_gbe 24>; /* rx flow for CPSW port 2 */
	ti,navigator-dma-names = "netrx0", "netrx1", "nettx", "netrx2";

	netcp-devices {
		ranges;
		#address-cells = <1>;
		#size-cells = <1>;
		gbe@90000 { /* ETHSS */
			#address-cells = <1>;
			#size-cells = <1>;
			label = "netcp-gbe";
			compatible = "ti,netcp-gbe";
			reg = <0x90000 0x300>, <0x90400 0x400>, <0x90800 0x700>;
			tx-queue = <648>;
			tx-channel = "nettx";

			interfaces {
				gbe0: interface-0 {
					phys = <&serdes_lane0>;
					slave-port = <0>;
					link-interface = <2>;
				};
				gbe1: interface-1 {
					phys = <&serdes_lane1>;
					slave-port = <1>;
					link-interface = <2>;
				};
				gbe2: interface-2 {
					phys = <&serdes_lane2>;
					slave-port = <2>;
					link-interface = <1>;
					phy-handle = <&ethphy0>;
				};
			};
		};
	};

	netcp-interfaces {
		interface-0 {
			rx-channel = "netrx0";
			rx-pool = <1024 12>;
			tx-pool = <1024 12>;
			rx-queue-depth = <128 128 0 0>;
			rx-buffer-size = <1518 4096 0 0>;
			rx-queue = <8704>;
			tx-completion-queue = <8707>;
			efuse-mac = <0>;
			netcp-gbe = <&gbe0>;
		};
		interface-1 {
			rx-channel = "netrx1";
			rx-pool = <1024 12>;
			tx-pool = <1024 12>;
			rx-queue-depth = <128 128 0 0>;
			rx-buffer-size = <1518 4096 0 0>;
			rx-queue = <8705>;
			tx-completion-queue = <8708>;
			efuse-mac = <1>;
			netcp-gbe = <&gbe1>;
		};
		interface-2 {
			rx-channel = "netrx2";
			rx-pool = <1024 12>;
			tx-pool = <1024 12>;
			rx-queue-depth = <128 128 0 0>;
			rx-buffer-size = <1518 4096 0 0>;
			rx-queue = <8706>;
			tx-completion-queue = <8709>;
			efuse-mac = <0>;
			local-mac-address = [02 18 31 7e 3e 6f];
			netcp-gbe = <&gbe2>;
		};
	};
};

sa_subsys: subsys@20c0000 {
	compatible = "simple-bus";
	#address-cells = <1>;
	#size-cells = <1>;
	ranges = <0 0x20c0000 0x40000>;

	sa_config: subsys@0 {
		compatible = "syscon";
		reg = <0x0 0x100>;
	};

	rng@24000 {
		compatible = "ti,keystone-rng";
		reg = <0x24000 0x1000>;
		ti,syscon-sa-cfg = <&sa_config>;
		clocks = <&clksa>;
		clock-names = "fck";
	};
};



I would greatly appreciate any guidance of where to dig deeper. Thanks for the help!

  • Hi, Bill,

    The upgrade is from which release and to which kernel releaes?

    Rex

  • Hi Rex, the old kernel version was 4.4.12

    I am using the kernel recipe from meta-TI (https://git.yoctoproject.org/cgit/cgit.cgi/meta-ti/tree/recipes-kernel/linux?h=rocko)

    First I tried using the "rocko" branch which is 4.14.

    After encountering the ethernet issue, I updated to the version in the "zeus" branch which is 5.4 (although upgrading did not fix the problem).

  • Hi, Bill,

    Are you using the kernel source from TI's Processor Linux SDK? Do you only need to build kernel? Have you tried with the build instruction in PLSDK User's Guide?

    http://software-dl.ti.com/processor-sdk-linux/esd/docs/latest/linux/Foundational_Components_Kernel_Users_Guide.html

    If you only need to build kernel, you can use the build instruction described in the user guide which is faster and easier, IMO.

    There are some patches in PLSDK which are not upstreamed. If you are not using PLSDK for K2H, I strongly suggest you use it. The current Kernel version in PLSDK 6.x is 4.19.

    Rex

  • Hi Rex,

    I am trying to use Yocto to build all of the system components (u-boot, kernel, device tree, rootfs, etc). I included the meta-ti Yocto layer in my build: git.yoctoproject.org/.../linux

    so my kernel is being built from the TI sources located here: git.ti.com/.../

    The specific kernel recipe git.yoctoproject.org/.../linux-ti-staging-rt_5.4.bb indicates the branch "ti-rt-linux-5.4.y" and SHA hash "c3dd64420d144641f0a803523185dffa26250f8b"

    Please correct me if I'm wrong, but I believe that PLSDK itself is based on Yocto, so I *think* that I am still using the proper kernel sources. Does that look correct?

    As to my specific problem, I just found a hint. Looking at the netcp PKTDMA registers, it appears that rx flow #24 was never initialized by Linux.

    Reading address 0x2004b00 (netcp pktdma, RX channel 24 global configuration register) gave 0x40000000, so the RX_TEARDOWN bit was active. I manually wrote the register with a value of 0x80000000 (RX_ENABLE), and eth2 started working (being able to both send and receive packets). I will dig further into the kernel driver.

  • Hi, Bill,

    Good to hear you made progress and found what may be the cause. If it is what you said about rx flow not set up for eth2, would it be miss configuration in dts file? I am not sure the following info would help you or not because you seem to pass what the info can provide. In the PLSDK NetCP User's Guide, there is a note on how to enable extra ethernet interfaces for KS2 family. When it was written, it is based on older kernel, but I can't remember which version, may be 4.4. With newer kernel, you may need to port the dts file.

    http://software-dl.ti.com/processor-sdk-linux/esd/docs/latest/linux/Foundational_Components/Kernel/Kernel_Drivers/Network/NetCP.html

    Rex

  • Turns out that it was a device tree issue. The default EVMK2H device tree has this snippet for the netcp DMA:

    knav_dmas: knav_dmas@0 {
    compatible = "ti,keystone-navigator-dma";
    clocks = <&papllclk>;
    #address-cells = <1>;
    #size-cells = <1>;
    ranges;
    ti,navigator-cloud-address = <0x23a80000 0x23a90000
    0x23aa0000 0x23ab0000>;

    dma_gbe: dma_gbe@0 {
    reg = <0x2004000 0x100>,
    <0x2004400 0x120>,
    <0x2004800 0x300>,
    <0x2004c00 0x120>,
    <0x2005000 0x400>;
    reg-names = "global", "txchan", "rxchan",
    "txsched", "rxflow";
    ti,enable-all;
    };
    };

    The third line in "reg" <0x2004800 0x300> specifies the RX DMA channels. The range 0x300 covers up through channel #23. I made it <0x2004800 0x320>, to additionally cover channels #23 and #24, and things started to work!

    This information was not anywhere in the users guide. Hopefully it can be helpful for others, because I was definitely not the first to have this problem.