AM3358: After kernel upgrade - CPSW configuration AR8035 - Destination Host Unreachable

Hi Arno,

TI doesn't release any Debian Linux for AM335x, please ask the software vendor for support.

The major issue is that we have troubles using the RGMII interface of the AM3358. We want to develop a proprietary board with the AM3358 in combination with the TI DP83822 PHY for Optical Fiber Ethernet. Also here we do not get the RGMII working and see similar behavior.

Nowhere there is a clear explanation how to use and debug the RGMII interface and how to configure the TI CPSW kernel Ethernet driver. We don't ask for support on Debian Linux but on the kernel drivers for the AM335x.

We started debugging with the OSD3358 EVM but already encounter problems upgrading the kernel without regression.

Can one of your Ethernet experts not provide support?

And isn't the CPSW kernel driver is developed by TI?

Hi Arno,

When you run into an issue in Linux, the issue could be in kernel or userspace, even if it is in kernel, it could be due to improper kernel config, so we might not know what is the root cause is in the kernel driver itself or elsewhere.

We will be providing support If you can reproduce the issue using TI Processor SDK Linux release v6.3.0.106, which you can download from ti.com.

Hi,

> I downloaded image am335x-evm-linux-06.03.00.106.img containing the PROCESSOR-SDK-LINUX-AM335X v6.3.0.108 and TI kernel 4.19.94.

The OSD3358-SM-RED board does boot up with this image on a SD card.

> Hereafter I adapted the am335x-evm.dts file so it recognizes the AR8035 PHY - WHICH HAS ADDRESS 4:

&davinci_mdio {
pinctrl-names = "default", "sleep";
pinctrl-0 = <&davinci_mdio_default>;
pinctrl-1 = <&davinci_mdio_sleep>;
status = "okay";

ethphy4: ethernet-phy@4 {
reg = <4>;
};
};

&cpsw_emac0 {
phy-handle = <&ethphy4>;
phy-mode = "rgmii-txid";
};

Compiled the device tree and put the new am335x-evm.dtb in the /boot folder.

>After reboot the AR8035 PHY (ADDRESS 4) is recognized

root@am335x-evm:~# dmesg | grep davinci_mdio
[ 1.026113] davinci_mdio 4a101000.mdio: davinci mdio revision 1.6, bus freq 000000
[ 1.033815] davinci_mdio 4a101000.mdio: detected phy mask ffffffef
[ 1.045119] davinci_mdio 4a101000.mdio: phy[4]: device 4a101000.mdio:04, drier Atheros 8035 ethernet

>However the debug log also shows that the kernel tries to access a PHY with address 0 for eth0

root@am335x-evm:~# dmesg | grep mdio
[ 0.953049] mdio_bus fixed-0: GPIO lookup for consumer reset
[ 0.953063] mdio_bus fixed-0: using lookup tables for GPIO lookup
[ 0.953070] mdio_bus fixed-0: No GPIO consumer reset found
[ 0.973054] mdio_bus 4a101000.mdio: GPIO lookup for consumer reset
[ 0.973065] mdio_bus 4a101000.mdio: using device tree for GPIO lookup
[ 0.973086] of_get_named_gpiod_flags: can't parse 'reset-gpios' property of node '/ocp/ethernet@4a100000/mdio@4a101000[0]'
[ 0.973100] of_get_named_gpiod_flags: can't parse 'reset-gpio' property of node '/ocp/ethernet@4a100000/mdio@4a101000[0]'
[ 0.973109] mdio_bus 4a101000.mdio: using lookup tables for GPIO lookup
[ 0.973116] mdio_bus 4a101000.mdio: No GPIO consumer reset found
[ 1.026113] davinci_mdio 4a101000.mdio: davinci mdio revision 1.6, bus freq 1000000
[ 1.033815] davinci_mdio 4a101000.mdio: detected phy mask ffffffef
[ 1.041075] libphy: 4a101000.mdio: probed
[ 1.045119] davinci_mdio 4a101000.mdio: phy[4]: device 4a101000.mdio:04, driver Atheros 8035 ethernet
[ 12.067818] libphy: PHY 4a101000.mdio:00 not found
[ 12.072663] net eth0: phy "4a101000.mdio:00" not found on slave 0, err -19

This might be a reason why we don't get 'eth0' alive with the other kernel.

How can I link 'net eth0' with "4a101000.mdio:04"? 

Is U-boot interfering here? I do also see that the phy-mode is reset to "mii", probably by U-boot

--------

I appreciate if I can get support on this eth0, PHY address <> 0 problem, utilizing TI Processor SDK Linux release v6.3.0.106

Hi Arno,

Appreciate your effort migrating to TI Processor SDK Linux. I am routing this query to our Ethernet expert. The response will be posted here soon.

Hi,

If the ethtool eth0 command output from the first post is correct then the PHY is indicating it has a link partner. Having a link is the first and most important step. It is concerning that you are seeing a no such device error for the PHY 0. Let's verify eth0 path for a moment. Could you please attach the output of ifconfig -a please?

The next steps are to verify the packets at each step of their journey from your board to the link partner and then back. For example this would be checking to make sure the Ping or ARP packet leaves your board and is received, processed and returned. You are showing no received packets with the ifconfig command so the path is broken somewhere. It has to be proven that the interface is sending the ARP packets to resolve the MAC address of the link partner. 

Look at the MAC level statistics and verify that the MAC sent the broadcast ARP request for the MAC address of the link partner. Take a look at ethtool -S eth0 and look for tx frame counts and check to make sure the count is incrementing due the packets of the ARP request. If the tx count is incrementing this means that in the MAC is at least sending the packets. Now check the MAC RX frame statistics of the link partner to see if the packets from your board are being received. I would strongly recommend using wireshark here on the link partner to see the actual packets and possibly response. I would also recommend directly connecting the two platforms without any intervening switches, hubs, bridges etc. This could require setting static ip addresses for both link partners.

If the link partner is responding with an ARP reply indicating the MAC address needed for the ping packets then check to see if the MAC statistics on your board are showing received packets.

If the packets are not showing up on the link partner receiver MAC statistics then double check your tx pin mux. This would be the same for checking the rx pin mux if the packets responses are not showing up on the rx frames of your board MAC statistics. 

Since you are using a reference board to me that implies a known good starting point so there should not be a hardware issue.

Best Regards,

Schuyler

Hi Schuyler,

Thank you for helping us.

In the very first report the reference board with Debian Linux kernel 5.9.70-ti-r19 has link between the PHY and my laptop. However in the setup with 'PROCESSOR-SDK-LINUX-AM335X v6.3.0.108 and TI kernel 4.19.94' I did not succeed getting the link up. This because I could not select PHY address 4 for the MAC.

So as you suggested let's continue with the first setup (Debian Linux kernel 5.9.70-ti-r19):

The Ethernet monitoring with Wireshark I already did and on my laptop (link partner) I do not see any packets from the OSD3358. Firewalls are down and the OSD3358 has a trusted IP for my laptop.

debian@beaglebone:~$ ifconfig -a
can0: flags=128<NOARP> mtu 16
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 10 (UNSPEC)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 54

can1: flags=128<NOARP> mtu 16
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 10 (UNSPEC)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 55

eth0: flags=-28605<UP,BROADCAST,RUNNING,MULTICAST,DYNAMIC> mtu 1500
inet 192.168.100.2 netmask 255.255.255.0 broadcast 192.168.100.255
inet6 fe80::6264:5ff:fe23:8040 prefixlen 64 scopeid 0x20<link>
ether 60:64:05:23:80:40 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 61 bytes 9864 (9.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 3520 bytes 236960 (231.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3520 bytes 236960 (231.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

usb0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.7.2 netmask 255.255.255.0 broadcast 192.168.7.255
ether 1c:ba:8c:a2:ed:6a txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

usb1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.6.2 netmask 255.255.255.0 broadcast 192.168.6.255
ether 1c:ba:8c:a2:ed:6e txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

debian@beaglebone:~$ ethtool -S eth0
NIC statistics:
Good Rx Frames: 0
Broadcast Rx Frames: 0
Multicast Rx Frames: 0
Pause Rx Frames: 0
Rx CRC Errors: 0
Rx Align/Code Errors: 0
Oversize Rx Frames: 0
Rx Jabbers: 0
Undersize (Short) Rx Frames: 0
Rx Fragments: 0
Rx Octets: 0
Good Tx Frames: 51
Broadcast Tx Frames: 0
Multicast Tx Frames: 51
Pause Tx Frames: 0
Deferred Tx Frames: 0
Collisions: 0
Single Collision Tx Frames: 0
Multiple Collision Tx Frames: 0
Excessive Collisions: 0
Late Collisions: 0
Tx Underrun: 0
Carrier Sense Errors: 11

debian@beaglebone:~$ ping 192.168.100.3
PING 192.168.100.3 (192.168.100.3) 56(84) bytes of data.
From 192.168.100.2 icmp_seq=1 Destination Host Unreachable
From 192.168.100.2 icmp_seq=2 Destination Host Unreachable
From 192.168.100.2 icmp_seq=3 Destination Host Unreachable
From 192.168.100.2 icmp_seq=4 Destination Host Unreachable
From 192.168.100.2 icmp_seq=5 Destination Host Unreachable
From 192.168.100.2 icmp_seq=6 Destination Host Unreachable
From 192.168.100.2 icmp_seq=7 Destination Host Unreachable
From 192.168.100.2 icmp_seq=8 Destination Host Unreachable
From 192.168.100.2 icmp_seq=9 Destination Host Unreachable
^C
--- 192.168.100.3 ping statistics ---
11 packets transmitted, 0 received, +9 errors, 100% packet loss, time 231ms
pipe 4

debian@beaglebone:~$ ethtool -S eth0
NIC statistics:
Good Rx Frames: 0
Broadcast Rx Frames: 0
Multicast Rx Frames: 0
Pause Rx Frames: 0
Rx CRC Errors: 0
Rx Align/Code Errors: 0
Oversize Rx Frames: 0
Rx Jabbers: 0
Undersize (Short) Rx Frames: 0
Rx Fragments: 0
Rx Octets: 0
Good Tx Frames: 63
Broadcast Tx Frames: 12
Multicast Tx Frames: 51
Pause Tx Frames: 0
Deferred Tx Frames: 0
Collisions: 0
Single Collision Tx Frames: 0
Multiple Collision Tx Frames: 0
Excessive Collisions: 0
Late Collisions: 0
Tx Underrun: 0
Carrier Sense Errors: 11

"If the packets are not showing up on the link partner receiver MAC statistics then double check your tx pin mux. This would be the same for checking the rx pin mux if the packets responses are not showing up on the rx frames of your board MAC statistics. "

> No packets are showing on the link partner. Checked tx/rx pinmux configuration and all seems fine:

cpsw_default {
pinctrl-single,pins = <
/* Slave 1 */
AM33XX_IOPAD(0x914, PIN_OUTPUT_PULLDOWN | MUX_MODE2) /* mii1_txen.rgmii1_tctl */
AM33XX_IOPAD(0x918, PIN_INPUT_PULLDOWN | MUX_MODE2) /* mii1_rxdv.rgmii1_rctl */
AM33XX_IOPAD(0x91c, PIN_OUTPUT_PULLDOWN | MUX_MODE2) /* mii1_txd3.rgmii1_txd3 */
AM33XX_IOPAD(0x920, PIN_OUTPUT_PULLDOWN | MUX_MODE2) /* mii1_txd2.rgmii1_txd2 */
AM33XX_IOPAD(0x924, PIN_OUTPUT_PULLDOWN | MUX_MODE2) /* mii1_txd1.rgmii1_txd1 */
AM33XX_IOPAD(0x928, PIN_OUTPUT_PULLDOWN | MUX_MODE2) /* mii1_txd0.rgmii1_txd0 */
AM33XX_IOPAD(0x92c, PIN_OUTPUT_PULLDOWN | MUX_MODE2) /* mii1_txclk.rgmii1_txclk */
AM33XX_IOPAD(0x930, PIN_INPUT_PULLDOWN | MUX_MODE2) /* mii1_rxclk.rgmii1_rxclk */
AM33XX_IOPAD(0x934, PIN_INPUT_PULLDOWN | MUX_MODE2) /* mii1_rxd3.rgmii1_rxd3 */
AM33XX_IOPAD(0x938, PIN_INPUT_PULLDOWN | MUX_MODE2) /* mii1_rxd2.rgmii1_rxd2 */
AM33XX_IOPAD(0x93c, PIN_INPUT_PULLDOWN | MUX_MODE2) /* mii1_rxd1.rgmii1_rxd1 */
AM33XX_IOPAD(0x940, PIN_INPUT_PULLDOWN | MUX_MODE2) /* mii1_rxd0.rgmii1_rxd0 */
>;
};

fragment@3 {
target = <&mac>;
__overlay__ {
status = "okay";
slaves = <1>;
};
};

fragment@4 {
target = <&cpsw_emac0>;
__overlay__ {
status = "okay";
phy-handle = <&ethphy0_4>;
phy-mode = "rgmii-txid";
};
};

fragment@5 {
target = <&davinci_mdio>;
__overlay__ {
status = "okay";

ethernet-phy@0 {
status = "disabled";
};

ethphy0_4: ethernet-phy@4 {
reg = <4>;
};
};
};

root@beaglebone:/sys/kernel/debug/pinctrl/44e10800.pinmux-pinctrl-single# cat pins

pin 69 (PIN69) 44e10914 00000002 pinctrl-single
pin 70 (PIN70) 44e10918 00000022 pinctrl-single
pin 71 (PIN71) 44e1091c 00000002 pinctrl-single
pin 72 (PIN72) 44e10920 00000002 pinctrl-single
pin 73 (PIN73) 44e10924 00000002 pinctrl-single
pin 74 (PIN74) 44e10928 00000002 pinctrl-single
pin 75 (PIN75) 44e1092c 00000002 pinctrl-single
pin 76 (PIN76) 44e10930 00000022 pinctrl-single
pin 77 (PIN77) 44e10934 00000022 pinctrl-single
pin 78 (PIN78) 44e10938 00000022 pinctrl-single
pin 79 (PIN79) 44e1093c 00000022 pinctrl-single
pin 80 (PIN80) 44e10940 00000022 pinctrl-single

-----------------------------

I doubt the AR8035 PHY is properly configured on rgmii clock delays. If I am right, the AM335x has NO internal clock delays. However, in the device tree the phy mode is set to "rgmii-txid" (M-BB-OSD3358-SM-RED-00A0.dtbo) ? This suggests an rgmii tx clock delay that could never be due to the AM335x, right?

Do you have more things to try?

Hi Schuyler,

After trying several things at the end I was able to resolve the RGMII communication with the AR8035.

1) It appeared that the AR8035 enters sometimes a power hibernate state.

2) And as I was suspected the AR8035 PHY has to add a delay on BOTH the TX clock and RX clock.

To resolve 1, I used the tool 'phyreg' to configure power hibernate of the AR8035 PHY over the MDIO bus.

To resolve 2, I changed the device-tree overlay:

fragment@4 {

target = <&cpsw_emac0>;
__overlay__ {
status = "okay";
phy-handle = <&ethphy0_4>;
phy-mode = "rgmii-id";
};
};

fragment@5 {
target = <&davinci_mdio>;
__overlay__ {
status = "okay";

ethernet-phy@0 {
status = "disabled";
};

ethphy0_4: ethernet-phy@4 {
reg = <4>;
};
};
};

-------------------

The next step is to get the OSD3358 with DP83822 PHY prototype working, now that the original OSD3358-SM-RED with AR8035 PHY and kernel 5.4.70 is working.

Hi,

I need to discuss with a colleague due the requirement of tx-id on an AM335x RGMII PHY connection. How are you adding timing delays to the AR8035 PHY? I usually see these as DTS node properties.

Best Regards,

Schuyler

Hi Schuyler,

The AR8035 PHY has the ability to switch on/off adding a delay on either the TX and/or RX clock. In the DTS you can set this with the property 'phy-mode'.

The kernel driver for the PHY uses this to set mdio register bits of the AR8035. The delay time value itself can be changed as well in the AR8035 but was in my case not needed since the default preset delay is 2.4ns.

I had to change the DTS property RGMII 'phy-mode' from 'txid' (enable tx clock delay in PHY) to 'id' (enable both tx and rx clock delay in the PHY).

The AM3358 can not add delays to the clocks, however the RGMII timing protocol requires delays on the clocks. Another way to create clock delay is to increase the length of your clock trace. On the OSD3358-SM-RED board, the PHY is placed close to the OSD3358, so the delays must be created in a different way.

The strange thing is that the 'older' kernel only requires a TX clock delay in the PHY. Where the newer kernel versions require both the TX and RX clocks to have a delay. The later versions of the CPSW Ethernet kernel driver may follow the RGMII timing protocol more strictly?

Best regards,

Arno