AM62A7: CPSW3G network failure

When I configure the value of register 00104044 to 0x00000012, execute the following command:

./devmem2 0x00104044h w 0x00000012

Then, the network under Linux was normal.

Additionally, in the /driver/net/ti/am65-cpsw-nuss.c/am65_cpsw_gmii_sel_k3() under U-Boot, there is the following code that configures the 0x00104044h register:

However, in the Linux environment, this register seems not to have been configured, and the default value is used instead. This has led to this problem.

1. Under U-boot, the value 0x00000012 is written to the register at 0x00104044h. Bit 4 is set to 1, but this bit is reserved in the register manual and we cannot determine what it means.

2. Is this issue an unaddressed bug in the SDK?

Hello, 

Apologies for the delayed response as I was out of office last week. 

jie.liu9 said:

2. When the Ethernet is not initialized in U-boot, after entering the Linux system, the network becomes abnormal and there is no ping connectivity between the PC and the AM62A7. 
When U-boot fails to initialize the Ethernet, we have disabled the following code in uboot/common/board_r.c/init_sequence_r:

Just to clarify, are you modifying the U-boot code to purposely disable Ethernet initialization in U-boot? If so, may I ask why are you purposely modifying the default code to disable Ethernet initialization in U-boot? I'm assuming the code disablement in init_sequence_r is what you are using to disable U-boot Ethernet initialization? 

jie.liu9 said:

1. Under U-boot, the value 0x00000012 is written to the register at 0x00104044h. Bit 4 is set to 1, but this bit is reserved in the register manual and we cannot determine what it means.

I'll try to look into what bit 4 is describing but this may take some time as I will need to check internally and due to the holidays several team members are out of office.

In the meantime, since the value 0x00000012 is from the "reg", which is obtained from "mode" based on the U-boot code, can you also let me know what "mode" is configured as before this line in the U-boot code? https://git.ti.com/cgit/ti-u-boot/ti-u-boot/tree/drivers/net/ti/am65-cpsw-nuss.c?h=ti-u-boot-2025.01#n302 I'm hoping this will help give us an indication to why the value 0x00000012 is necessary.

jie.liu9 said:

2. Is this issue an unaddressed bug in the SDK?

Currently to my knowledge there isn't a reported bug related to this type of issue mainly because this issue is only introduced when a custom modification on top of the default SDK is made.

-Daolin

1、To optimize the boot time, we disabled the initialization process of the Ethernet in U-Boot and commented out the following code:
uboot/common/board_r.c/init_sequence_r/initr_net

2. "reg is obtained from 'mode' based on the U-boot code", since phy_mode is PHY_INTERFACE_MODE_RGMII_ID, the mode here is AM65_GMII_SEL_MODE_RGMII, as shown below:

3. Finally, we discovered that, based on the current SDK, the bit 4 of this register was not configured on the Linux platform. After modifying the following code, the Ethernet function on Linux worked properly.

linux/driver/phy/ti/phy-gmii-sel.c：

The specific configuration process is as follows:

Hello Jie,

jie.liu9 said:

2. "reg is obtained from 'mode' based on the U-boot code", since phy_mode is PHY_INTERFACE_MODE_RGMII_ID, the mode here is AM65_GMII_SEL_MODE_RGMII, as shown below:

Just to clarify, this issue is only observed when you are using RGMII_ID mode? If, for example, you had used "rgmii_rxid" mode as specified in the AM62A TI EVM DTS: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/arch/arm64/boot/dts/ti/k3-am62a7-sk.dts?h=ti-linux-6.12.y-cicd#n792, do you also see the same problem when U-boot initialization is disabled? 

May I ask if there was particular reason for using rgmii_id mode (as you have also indicated in your DTS snippet) as opposed to rgmii_rxid mode?

jie.liu9 said:

3. Finally, we discovered that, based on the current SDK, the bit 4 of this register was not configured on the Linux platform. After modifying the following code, the Ethernet function on Linux worked properly.

linux/driver/phy/ti/phy-gmii-sel.c：

Good to know that you have found a way to address your issue with this modification. When you use this solution, if U-boot Ethernet initialization was kept enabled, does Ethernet also still function correctly in Linux?

I cannot currently comment on whether this solution is the proper fix to the problem. There are currently two unknowns/factors influencing this issue:

1. I still need to consult experts internally on what bit 4 of register 0x00104044 is modifying, I'm hoping to get some more answers next week.

2. When rgmii_id mode is used, it configures the PHY to enable both RX and TX delays and the MAC/SoC should neither RX nor TX delay in this case. My understanding is that the SoC adds TX delay by default and we do not support disabling this. So if rgmii_id mode is used, it would end up with both the MAC and PHY adding TX delay which might break Ethernet TX traffic on 1000Mbps links. This may be a reason why our example DTS only uses rgmii_rxid mode and may be a reason for you to use rgmii_rxid mode as well.

In your screencapture, you mentioned "experiences network ... when operating in 100Mbps mode" which might why you haven't seen a problem with rgmii_id mode when you applied your code changes. Just to confirm, are you testing only for 100Mbps link speed or will you also be testing 1000Mbps?

-Daolin

Hello.Daolin
    We have found that bit 4 of register 00104044 is used to control the transmission delay of the MAC on the SoC. A value of 1 indicates that the transmission delay is disabled, while a value of 0 indicates that it is enabled. Currently, the transmission delay is enabled by default. Therefore, in this issue, there are three main registers that affect Ethernet communication: register 00104044 on the AM62A, the RGMII_TXC_DELAY_CONFIG register on the TJA1103, and the RGMII_RXC_DELAY_CONFIG register. They are as follows:

Therefore, when the device tree is configured in different modes such as rgmii-id, rgmii-txid, and rgmii-rxid, it will affect the delay configuration of TJA1103 and am62a. Thus, when configured as rgmii-id in uboot, the values of these three registers should be: 0x001044: 0x12 (disable MAC transmit delay), 0xAFCC: 0x8012 (enable PHY transmit delay), and 0xAFCD: 0x8012 (enable PHY receive delay). Only with this combination can Ethernet communication function properly. However, when configured as rgmii-rxid in uboot, the values of these three registers should be: 0x001044: 0x02 (enable MAC transmit delay), 0xAFCC: 0x8012 (enable PHY transmit delay), and 0xAFCD: 0x0012 (disable PHY receive delay). With this combination, Ethernet communication cannot function properly. The relevant code configuration snippet is as follows:
uboot/drivers/net/ti/am65-cpsw-nuss.c:

uboot/drivers/net/phy/nxp-c45-tja11xx.c:

Additionally, under Linux, the default state is 0x001044:0x02 (enabling MAC transmission delay), so the device tree under Linux can only be configured as rgmii-txid.

My question is:

Why is it that in the code under U-boot, the bit 4 of 0x001044 can be configured when the RGMIi-ID/RGMIi-TXID protocol is supported? However, in the Linux code, it is by default not supported to configure the bit 4 of 0x001044?

uboot:

linux：

Hi Jie, 

jie.liu9 said:

We have found that bit 4 of register 00104044 is used to control the transmission delay of the MAC on the SoC. A value of 1 indicates that the transmission delay is disabled, while a value of 0 indicates that it is enabled.

May I ask where you have discovered this information? Is this determined from experimentation and by measuring the TX delay when bit 4 of 00104044 is 1 vs 0? 

jie.liu9 said:

However, when configured as rgmii-rxid in uboot, the values of these three registers should be: 0x001044: 0x02 (enable MAC transmit delay), 0xAFCC: 0x8012 (enable PHY transmit delay), and 0xAFCD: 0x0012 (disable PHY receive delay). With this combination, Ethernet communication cannot function properly

jie.liu9 said:

Additionally, under Linux, the default state is 0x001044:0x02 (enabling MAC transmission delay), so the device tree under Linux can only be configured as rgmii-txid.

I think there is a mismatch in what is meant by rgmii-rxid vs rgmii-txid between you and I. When rgmii-rxid mode is used, this should be configuring the PHY to enable receive delay and the MAC will not be adding any receive delay. My understanding is that in this case, the MAC is responsible for the transmit delay. When rgmii-txid mode is used, this configures the PHY to enable transmit delay and the MAC should not add any transmit delay. My understanding of this case is that the MAC is responsible for the receive delay in this case. 

Daolin Qiu said:

My understanding is that the SoC adds TX delay by default and we do not support disabling this. So if rgmii_id mode is used, it would end up with both the MAC and PHY adding TX delay which might break Ethernet TX traffic on 1000Mbps links.

As I had mentioned, due to SoC adding TX delay be default and we don't support disabling this (most likely reason for why bit 4 of register 00104044 is reserved), we encourage use of rgmii-rxid mode. In this mode, the transmit delay is provided by the MAC (as you have observed 0x001044:0x02 by default which enables MAC transmission delay), and receive delay is provided by the PHY. This, of course, means that you should ensure that the PHY disables transmit delay. The MAC does not support receive delay since the CPSW module on the SoC does not support it. This is the mode we use for AM62A TI EVM by default in the DTS. As a sanity check, I would double check testing rgmii_rxid mode, ensuring you have disabled transmit delay on the PHY side. 

On an AM62A TI EVM, I had tested

1. rgmii_rxid mode with initr_net commented at to disable U-boot Ethernet initialization --> ping communication successful

2. rgmii_id mode without initr_net commented to keep enabled U-boot Ethernet initialization --> eth0 interface does not show up at all with ifconfig (let alone have ping functioning correctly)

3 rgmii_id mode with init_net commented to disable U-boot Ethernet initialization --> same as #2

jie.liu9 said:

Why is it that in the code under U-boot, the bit 4 of 0x001044 can be configured when the RGMIi-ID/RGMIi-TXID protocol is supported? However, in the Linux code, it is by default not supported to configure the bit 4 of 0x001044?

In general, because by default we have always tested with Ethernet initialization enabled in U-boot and the state of the PHY set in U-boot stage carries over to the Linux environment, meaning bit 4 being set on 0x00104044 gets carried over to the Linux environment, the need for configuring bit 4 in the Linux code was never caught. As the problem you have observed only gets exposed when U-boot Ethernet initialization is disabled.

-Daolin

Hello.Daolin：

    Regarding the situation when configured as rgmii-rxid, the MAC is set to 0x02 and the transmission delay is enabled. However, in uboot/drivers/net/phy/nxp-c45-tja11ax.c, the 0XAFCC register of the PHY is configured to 1, which enables TX_DELAY as follows:

From this perspective, when configured as rgmii-rxid, the tx-delay is enabled on the PHY side.
Also, in U-Boot, when configured as rgmii-id/rgmii-txid, the tx-delay on the MAC side is disabled. Why is this?

Hi Jie, 

jie.liu9 said:

Also, in U-Boot, when configured as rgmii-id/rgmii-txid, the tx-delay on the MAC side is disabled. Why is this?

In rgmii_id mode, the PHY is expected to add both the RX and TX delays. The MAC should not be responsible for either. 

Please see the explanation for rgmii_txid mode from my previous response below.

Daolin Qiu said:

When rgmii-txid mode is used, this configures the PHY to enable transmit delay and the MAC should not add any transmit delay. My understanding of this case is that the MAC is responsible for the receive delay in this case. 

jie.liu9 said:

From this perspective, when configured as rgmii-rxid, the tx-delay is enabled on the PHY side.

My understanding from the defintion of rgmii-rxid mode from the DTS point of view is that the PHY should be enabling only RX delay in rgmii_rxid mode and its implied that it should not enable TX delay in this mode. The driver you referenced "uboot/drivers/net/phy/nxp-c45-tja11ax.c" is not a TI part so I cannot make a full recommendation but if it is indeed enabling TX Delay in rgmii_rxid mode, then my understanding is that it is not compatible with the definition of rgmii-rxid mode from the DTS point of view. It might be a good idea to double check with the vendor for this PHY.

Daolin Qiu said:

When rgmii-rxid mode is used, this should be configuring the PHY to enable receive delay and the MAC will not be adding any receive delay. My understanding is that in this case, the MAC is responsible for the transmit delay.

-Daolin

Hello. Daolin:
1. The issue with the phy driver is currently being addressed by our supplier.
2. My confusion lies in the inconsistency of the code logic between Uboot and Kernel. When the device tree is configured as rgmii-txid/rgmii-id, in Uboot, the bit 4 of 0x001044 is set to 1. However, when the device tree is configured as rgrgmii-txid/rgmii-id under Linux, it does not support setting the bit 4 of 0x001044 to 1. From this, it seems that the code logic between Uboot and Linux is inconsistent when configuring rgrgmii-txid/rgmii-id. Why is this so?

Hi Jie, 

jie.liu9 said:

2. My confusion lies in the inconsistency of the code logic between Uboot and Kernel. When the device tree is configured as rgmii-txid/rgmii-id, in Uboot, the bit 4 of 0x001044 is set to 1. However, when the device tree is configured as rgrgmii-txid/rgmii-id under Linux, it does not support setting the bit 4 of 0x001044 to 1. From this, it seems that the code logic between Uboot and Linux is inconsistent when configuring rgrgmii-txid/rgmii-id. Why is this so?

bit 4 set (U-boot) --> MAC no TX delay --> rgmii_txid/rgmii_id --> proper functionality

bit 4 cleared (Linux)  --> MAC TX delay --> rgmii_rxid, if also rgmii_txid/rgmii_id --> NOT proper functionality

Under normal operations/logic (not TI AM6x SoCs), bit 4 should be set for rgmii_txid/rgmii_id mode. This means for non-TI AM6x SoCs, the bug would be in the Linux code where bit 4 is not set. However, TI AM6x SoCs should always enable MAC TX delay by default (bit 4 is cleared). We reserve bit 4 because we do not recommend/support modifying this bit. By extension, this means we recommend using rgmii_rxid mode only. If rgmii_txid/rgmii_id mode is used, it is expected that bit 4 will still stay cleared, therefore likely causing issues with Ethernet communication. My understanding is that this means there is a bug in U-boot where it is modifying bit 4 but it should not be due to this restriction on TI AM64x SoCs.

I hope this explanation helps, let me know if there is any additional confusion.

-Daolin