AM5716: CPSW-3G after enable the port, change MAC address, will cause traffic broken

Hello Semon,

Just want to double check you are using AM57x, I'm assuming the following is a typo?

Semon Zhang said:

sdk 6.3, A53 Linux

AM57x has an A15 core.

Another question, is 

Semon Zhang said:

during bring up, if send 100Mbps traffic to the eth0

Is this statement in reference to Linux boot?

Can you share the full boot logs please?

please use the Insert->Image/video/File and click on the Upload word.

-Josue

Josue Zamitiz-Ayala said:

Is this statement in reference to Linux boot?

Can you share the full boot logs please?

please use the Insert->Image/video/File and click on the Upload word.

env3_start_2024-08-19_19-01-49.log

please check the log from customer Sungrow

the problem is that during Linux boot up, send full speed traffic to the ethernet port as the background traffic, ifconfig up/ifconfig down/change MAC address/ifconfig up/, this operation procedure will cause network port 

abnormal, even remove the background traffic after port linkup, network can not recover

Regards

   Semon 

Semon,

Can you check if the IP address changes after the MAC address changes? If you reassign the same IP address to the port will it work?
What is the board connected to? Is it a DHCP server?

-Josue

Josue Zamitiz-Ayala said:

Can you check if the IP address changes after the MAC address changes? If you reassign the same IP address to the port will it work?
What is the board connected to? Is it a DHCP server?

Hello Josue

     IP address is configured after change MAC address, at the beginning, IP adress is not configured

    with following script added to /etc/network/interfaces, send full speed traffic to the port, and reboot the board,

after Linux bring up, remove the network storm traffic, ping this port from PC, ping failed,

capture at the eth0 of the board, frame can be sent out to the PC, but can't receive traffic from outside,

Regards

   Semon

Hello Semon, 

Is the IP address the same one set for the first MAC address? 

Have they tried changing the MAC address without bringing the interface down?

-Josue

Hi Semon,

My theory is that the quick change in MAC address is causing confusion with your DHCP server, since MAC address is usually fixed. My suggestion would be to either

1. not change MAC address at run time after an IP address is assigned, or
2. if the MAC address change at runtime is necessary, then log into the router or whatever device is handling the DHCP server to observe if IP address to MAC address association is happening correctly. My video below is an example of this experiment

For some reasoning:

As an experiment, I used a J784S4/AM69 device (which is the next generation of device from AM57x) and conducted a similar experiment done on your end for changing MAC address. Attached below is a video.

Some points of interest are:

1. I start with MAC address 00:55:22:33:44:FF and IP address 192.168.1.3
2. I then change MAC address to 00:66:22:33:44:FF without switching IP address
3. There is a brief moment when DHCP server on router sees two MAC addresses both associated with 192.168.1.3
4. New IP address is assigned to 00:66:22:33:44:FF of 192.168.1.8, and the old MAC address and IP address is forgotten
5. Ping stops because the IP address is no longer the original 192.168.1.3

Event 3 and 4 looks to be a potential issue, and depending on how smart your router is, and how frequently the list of addresses are updated, this behavior could be different on your end.

As a disclaimer, AM57x and networking is not my area of expertise, so please consider my theory with caution.

Regards,

Takuma

My theory is that the quick change in MAC address is causing confusion with your DHCP server, since MAC address is usually fixed. My suggestion would be to either

-----------------------

Hi Takuma

     in customer system, they did not use DHCP, they configure the IP address statically, so there should be no confusion about IP and MAC address

------------------------------------------------------------------

according to new test result, only do following configuration steps and send 1Gbps traffic to the eth0 port at the same time, then stop the traffic, 

then ping this port, ping will fail

------------------------------------------

    ifconfig eth0 up

    ifconfig eth0 down

    ifconfig eth0 up

-------------------------------------------

Regards

   Semon

Hello Josue, Takuma

     according to customer test, sdk6.3 has issue, but SDK9.2 seems not be able to re-produce it,

     please help review the code change between these two sdk version

Thanks

   Semon

------------------------------------------

    ifconfig eth0 up

    ifconfig eth0 down

    ifconfig eth0 up

-------------------------------------------

Hello Josue, Takuma

      I verified these steps on TDA2 EVM board, it's Linux 4.19 kernel, same as AM5716, the above can re-produce the customer issue,

      please help verify it on AM5716 EVM, it should be an issue,

      I also compare the sdk6.3 and sdk9.2 CPSW-3G driver, in sdk9.2 it adds XDP feature on Ethernet driver, 

      can this difference fix the packet loss issue with XDP feture?

      please give some explanation

Thanks

    Semon 

Hi Semon,

The issue that I have shown you in the video will happen with statically allocated IP address. Again, the recommendation is to not change the MAC address at runtime, or as an alternative, you will need a good understanding of how your local network/router is set up so that you can change your usage such that each IP address has a unique MAC address.

As for reasoning, using statically allocated IP addresses will guarantee that this issue will happen. This is because the issue is not due to the dynamic allocation of IP address, but instead the issue is due to the DHCP server keeping track of IP to MAC address association getting two MAC addresses with the same IP address. Essentially, step 3 in my previous response. When manually allocating the same static IP addresses, you are guaranteeing failure since there will be two MAC addresses of the same IP address and there is no dynamic reallocation of IP address, so the network stack cannot determine where to send data. As a possible fix, you could assign a different unused IP address to the second MAC address, or somehow force the router or whatever device is keeping track of MAC to IP address association to forget the old MAC address.

Enabling XDP fixing the issue makes sense, because what XDP does is it bypasses the network stack that does some IP address check like below diagram I found online:

Regards,

Takuma

Hi Semon,

Please share more details about the customer requirements and setup?

Why are they bringing the port down and back up?

Why is the MAC address being changed?

Even if the IP address is static, is there still a DCHP server on the network?

-Josue

Josue Zamitiz-Ayala said:

Please share more details about the customer requirements and setup?

Why are they bringing the port down and back up?

Why is the MAC address being changed?

Even if the IP address is static, is there still a DCHP server on the network?

Hello Josue

    bringing port down and up, 

        customer application will down the port and configure the MAC address and IP address, then bring port up

    in customer design, MAC address is configured by customer application, not set in the kernel boot script

   there is no DHCP server in the network

   now the issue is that even not change the MAC address after Linux boot, just ifconfig down/up eth0 port, the issue can be re-produced,

   and if the driver changes in sdk 9.2 can be ported to sdk6.3?

Regards

   Semon

Hello Semon, 

TI will not do back porting for 6.03 since there is no new development nor updates being done for it, this SDK is in as-is state. Customer would have to do this on their own or reach out to a 3rd party for porting.

I think I might have replicated the issue on AM57 board. Will need to do more testing in the following days.

Best,

Josue

Josue Zamitiz-Ayala said:

I think I might have replicated the issue on AM57 board. Will need to do more testing in the following days.

Hello Josue

   in my test,  I connect a Linux PC to the eth0 port, run packeth to generate the full speed traffic to eth0 on the EVM, then ifconfig down/up the eth0,

   stop packeth, then try to ping eth0, at this time, ping fail, not too difficult to re-produce it

Hello Josue

   First of all, The issue has replicated in my AM5716 baord, just as semon said,  generate the full speed traffic to eth0 and ifconfig down/up, stop packeth,the eth0  will ping faild. 

  The same problem has been repeated in the semon EVM board, also using the 6.3 SDK ,So you don't think it's a bug ??? 

  The hardware and software are all yours. If there is a problem, why do you need third-party support? Please explain ?

Semon,

Semon Zhang said:

I connect a Linux PC to the eth0 port, run packeth to generate the full speed traffic to eth0 on the EVM, then ifconfig down/up the eth0,

This is exactly what I have done. I am  consulting with someone on Dev team to interpret results.

I am not sure that a packet storm was a use-case that was supported on CPSW-3G for this release, but as you can tell from the other thread that was open here, this was a use-case with software support on the PRU.

Zmg,

This SDK release is very old and no longer maintained. As tested by customer, this is no longer a bug in the newer SDK, therefore our recommendation would be to move to newer SDK. 

Best,

Josue

Hello Josue：

          Although this is an older version, it is also the version you released, and this issue triggered a large area of the field we use ，so this is a bug for us.

Even if you don't maintain it in 6.3, you still need to explain and fix the problem in this versoin. 

• For a field problem, the clock is ticking

Hello Zmg,

I understand. I will discuss this internally and I will get back to you.

Best,

Josue

Hi Semon and Zmg,

While you wait for Josue's response, could you try out this experiment? Depending on the result of this experiment you would be able to fix this issue easily.

1. Log in to the router that is handling the network that your EVM is on
2. There should be a table to see the IP address to MAC address associations as shown in my video from last week
3. Do the experiment with the "ifconfig eth0 up", "ifconfig eth0 down", ifconfig eth0 up". Please also note the output from "ifconfig" after each command because we need to make sure IP address is assigned to eth0 when the network is up.
4. Record a video of how this IP address to MAC address table on the router is reacting to these commands

The same can be done with all of the experiments that have been tried so far, including the packet storm scenario, by replacing step 3. If we see that the table on the router side looks good and there are no duplicate entries for IP address, then we can close this theory. 

My theory is still that the experiments that are being tried out is messing up this IP address to MAC address table on the router. This is because if my theory is correct the issue will happen regardless of using static IP address, not using a DHCP server, or if the network simply goes down (you mentioned there was no DHCP server, so an IP address must manually be assigned after the network goes down, and it was not clear whether this step was done).

The XDP feature fixing the issue further supports my theory, since that allows bypassing the network stack as a whole, so it bypasses the IP and UDP/TCP layers as well.

Regards,

Takuma

Takuma Fujiwara said:

The same can be done with all of the experiments that have been tried so far, including the packet storm scenario, by replacing step 3. If we see that the table on the router side looks good and there are no duplicate entries for IP address, then we can close this theory. 

My theory is still that the experiments that are being tried out is messing up this IP address to MAC address table on the router. This is because if my theory is correct the issue will happen regardless of using static IP address, not using a DHCP server, or if the network simply goes down (you mentioned there was no DHCP server, so an IP address must manually be assigned after the network goes down, and it was not clear whether this step was done).

The XDP feature fixing the issue further supports my theory, since that allows bypassing the network stack as a whole, so it bypasses the IP and UDP/TCP layers as well.

Regards,

Hi Takuma

     now we can re-produce this issue without changing MAC address, while send traffic to eth0 port with full speed traffic, just ifconfig eth0 down/ifconfig eth0 up, can re-produce it on AM5716, 

     and this issue can not be re-produce it on SDK 9.2, there should have trace in the JIRA, and from the code, I guess sdk8.2 should fix this issue too

     please help check internally to see which change can fix this issue\

Thanks

   Semon

Hello Semon,

Could you please check if the following commit fixes the issue?
https://github.com/torvalds/linux/commit/51302f77bedab8768b761ed1899c08f89af9e4e2

Regards,
Siddharth.

To Siddharth Vadapalli :

      This change is synchronized in our code. can you have another sugges

Best.

On toggling the interface, do you see any errors in Linux before the traffic is no longer received? Could you also share the commit hash or tag that your Linux source is based on?

Hi Siddharth

     I check the Linux source, following is the tag info

--------------------------------------------

(base) semon@semon-u18:~/tda2/ti-processor-sdk-linux-am57xx-evm-06.03.00.106/board-support/linux-4.19.94+gitAUTOINC+be5389fd85-gbe5389fd85$ git log
commit 070d88d712b9e7f3a5e4e78078b496bc51665541 (HEAD -> processor-sdk-local, processor-sdk-linux-4.19.y)
Author: Texas Instruments SDK Builder <>
Date: Sun Apr 19 02:36:19 2020 +0000

Create local branch

The below commit is used for this local branch and is the one used by
this sdk:
be5389fd85b69250aeb1ba477447879fb392152f.

To reduce the size of the SDK installer only a portion of the git commit
history has been preserved. Also links to remote branches and tags do
not exist in this copy of the repository.
To fix this please run the unshallow-repositories.sh
script located in the sdk's bin directory

The script may take several minutes but you will then have a full copy of
the git repository including its entire git history.

commit be5389fd85b69250aeb1ba477447879fb392152f
Merge: 58f4443a8c f8e155c279
Author: Muralidharan Karicheri <m-karicheri2@ti.com>
Date: Tue Apr 14 06:12:18 2020 -0500

Merge pull request #223 in PROCESSOR-SDK/processor-sdk-linux from PTEST-2081 to processor-sdk-linux-4.19.y

* commit 'f8e155c2798d11903af1931568b543f49bfc3a92':
net: ethernet: prueth: fix emac_ndo_open() error path

commit f8e155c2798d11903af1931568b543f49bfc3a92
Author: Aaron Kramer <a-kramer@ti.com>

Siddharth Vadapalli said:

On toggling the interface, do you see any errors in Linux before the traffic is no longer received? Could you also share the commit hash or tag that

there is no error log after executing ifconfig down/up command, 

Regards

   Semon

Semon,

Updating Thread as it was followed up off-line in an email.

"Older SDKs can only be supported in a limited manner.
Customers are expected to migrate to the newer SDKs
since they not only introduce more features, but also
fix issues which are present in older SDKs (as observed
in this case). Given that customers may not be willing
to migrate to the newer SDKs, and, given that the issue
has been fixed at least since SDK 8.6, I will suggest bisecting
the issue by testing with other SDK releases as is already
being done. As you have indicated in the E2E that there
are no error logs associated with the issue and it seems
to be a silent failure, it is difficult to determine where to
start looking at. If I have to guess, I would say that the
RX DMA Channel teardown is timing out due to the large
traffic being received which has most likely filled up all
the RX buffers. You could try increasing the timeout
and see if that fixes the issue." - Siddharth

-Josue

Josue Zamitiz-Ayala said:

If I have to guess, I would say that the
RX DMA Channel teardown is timing out due to the large
traffic being received which has most likely filled up all
the RX buffers. You could try increasing the timeout
and see if that fixes the issue." - Siddharth

Hello Siddharth/Josue

     if there are too much packets causing RXDMA time-out, can the port be disabled to not receive packet during the down time?

Regards

   Semon

Semon Zhang said:

 if there are too much packets causing RXDMA time-out, can the port be disabled to not receive packet during the down time?

RX DMA Channel teardown is performed on "ndo_stop" which is when the interface is brought down.
Even with high traffic load, as long as the interface is up, you don't seem to face any issues.
It is only when the interface is brought down and then brought up that the issue is seen.
Therefore I guessed that the RX DMA Channel teardown which is being performed when the interface
is brought down, is timing out as there are a lot of packets to be flushed out.

To check if that's the case, you could reduce the number of Descriptors and see if that makes a difference:

diff --git a/drivers/net/ethernet/ti/cpsw.c b/drivers/net/ethernet/ti/cpsw.c
index 0b7a3eb06a65..b7ae2d77f45e 100644
--- a/drivers/net/ethernet/ti/cpsw.c
+++ b/drivers/net/ethernet/ti/cpsw.c
@@ -155,7 +155,7 @@ do {                                                                     \
                cpsw->data.active_slave)
 #define IRQ_NUM                        2
 #define CPSW_MAX_QUEUES                8
-#define CPSW_CPDMA_DESCS_POOL_SIZE_DEFAULT 256
+#define CPSW_CPDMA_DESCS_POOL_SIZE_DEFAULT 64
 #define CPSW_FIFO_QUEUE_TYPE_SHIFT     16
 #define CPSW_FIFO_SHAPE_EN_SHIFT       16
 #define CPSW_FIFO_RATE_EN_SHIFT                20

Siddharth Vadapalli said:

RX DMA Channel teardown is performed on "ndo_stop" which is when the interface is brought down.
Even with high traffic load, as long as the interface is up, you don't seem to face any issues.
It is only when the interface is brought down and then brought up that the issue is seen.
Therefore I guessed that the RX DMA Channel teardown which is being performed when the interface
is brought down, is timing out as there are a lot of packets to be flushed out.

To check if that's the case, you could reduce the number of Descriptors and see if that makes a difference:

Hi Siddharth

    I tried to reduce the DESC number to 64, but the issue is not fixed, 

    if it's caused by too many packets in buffer when interface down, disable this port should be ok, but I do not find the register to disable the port

    where to change to time-out value?

Regards

   Semon

Semon,

Since there are two interfaces: eth0 and eth1, of which only one is being brought down, the DMA Channel teardown will not occur.
Could you try bringing down eth0 and eth1 and then bring both of them up and check if the issue persists?

Regards,
Siddharth.

Siddharth Vadapalli said:

Since there are two interfaces: eth0 and eth1, of which only one is being brought down, the DMA Channel teardown will not occur.
Could you try bringing down eth0 and eth1 and then bring both of them up and check if the issue persists?

Hi Siddharth

     when eth0 and eth1 both are down then both are up, this issue will not happen

Regards

   Semon

Siddharth Vadapalli said:

1. Bring down both interfaces and bring them up
2. Migrate to the cpsw_new.c driver (which is not present in SDK 6.3)

Hi Siddharth

      these two options maybe hard to be accepted by customer,

      I have one question, how to disable the down port to receive frame from the network, I can't find such register?

Regards

    Semon

Semon,

Could you test the following diff?

diff --git a/drivers/net/ethernet/ti/cpsw.c b/drivers/net/ethernet/ti/cpsw.c
index 0b7a3eb06a65..b7cfb29e2aa7 100644
--- a/drivers/net/ethernet/ti/cpsw.c
+++ b/drivers/net/ethernet/ti/cpsw.c
@@ -1610,6 +1610,7 @@ static int cpsw_fill_rx_channels(struct cpsw_priv *priv)
 static void cpsw_slave_stop(struct cpsw_slave *slave, struct cpsw_common *cpsw)
 {
        u32 slave_port;
+       u32 mac_control;

        slave_port = cpsw_get_slave_port(slave->slave_num);

@@ -1620,6 +1621,14 @@ static void cpsw_slave_stop(struct cpsw_slave *slave, struct cpsw_common *cpsw)
        slave->phy = NULL;
        cpsw_ale_control_set(cpsw->ale, slave_port,
                             ALE_PORT_STATE, ALE_PORT_STATE_DISABLE);
+       /* Command Idle */
+       mac_control = readl_relaxed(&slave->sliver->mac_control);
+       mac_control |= BIT(11);
+       writel_relaxed(mac_control, &slave->sliver->mac_control);
+       msleep(500);
+       mac_control = readl_relaxed(&slave->sliver->mac_control);
+       mac_control &= ~BIT(11);
+       writel_relaxed(mac_control, &slave->sliver->mac_control);
        soft_reset_slave(slave);
 }

Siddharth Vadapalli said:

Could you test the following diff?

this diff can't work,

if following register can be used to disable the port?, but I do not know which can be used for the port? there are 6 PORTCTL register

port state can be set to disabled when down the port?

Regards

   Semon

Semon Zhang said:

port state can be set to disabled when down the port?

Hi Siddharth

    I tried to test with following steps, traffic can recover after down/up operation:

    maybe these setting can be added to the driver for correct operation

------------------------------

ifconfig eth0 down
devmem2 0x48484d40 w 0

ifconfig eth0 up
devmem2 0x48484d40 w 3

-------------------------

Regards

   Semon

Semon Zhang said:

    I tried to test with following steps, traffic can recover after down/up operation:

    maybe these setting can be added to the driver for correct operation

The driver already implements this:
cpsw_ndo_stop()
=> for_each_slave(priv, cpsw_slave_stop, cpsw)
==> cpsw_ale_control_set(cpsw->ale, slave_port, ALE_PORT_STATE, ALE_PORT_STATE_DISABLE);

So it appears to be the case that this path is not executed in the customer's setup.
"for_each_slave" is defined as:

#define for_each_slave(priv, func, arg...)				\
	do {								\
		struct cpsw_slave *slave;				\
		struct cpsw_common *cpsw = (priv)->cpsw;		\
		int n;							\
		if (cpsw->data.dual_emac)				\
			(func)((cpsw)->slaves + priv->emac_port, ##arg);\
		else							\
			for (n = cpsw->data.slaves,			\
					slave = cpsw->slaves;		\
					n; n--)				\
				(func)(slave++, ##arg);			\
	} while (0)

Siddharth Vadapalli said:

Depending on whether "dual_emac" is set in the device-tree, the macro expands differently.
That could be making a difference here, resulting in this issue.

Hi Siddhart

    after executing 

------------------

Semon, 

Can you share the DTS setup from the customer?

-Josue

The dts in our code is same as what you posted sdk 6.3 , and "dual_emac" is also setted in the device-tree

HI Siddharth

    in my test on TDA2P EVM, if send traffic less than 480Mbps, then after ifconfig eth0 down/up operation, eth0 can recover,  but if traffic is big than 500Mbps,

    eth0 can not recover, can this be related with DMA buffer issue?

    sorry I make a mistake previously, the port1 has already been disabled when eth0 is shutdown, at this time, traffic can not input to CPSW-3G,

    the issue should be the DMA buffer manager issue, or Linux stack buffer issue

Regards

   Semon

Hello Semon,

Does enabling interrupt pacing make a difference?

ethtool -C eth0 rx-usecs 250

Regards,
Siddharth.

Semon,

Please check if the following commit is present in the Linux source:
github.com/.../871e846585919adf727f21398f433fd424b6f0e1

Regards,
Siddharth.

Siddharth Vadapalli said:

Please check if the following commit is present in the Linux source:
github.com/.../871e846585919adf727f21398f433fd424b6f0e1

Siddharth

     I tried it, but it doesn't work

Regards

   Semon

I have marked Siddharth's answer that very well summarizes the  status of this thread. 

With the following options as the pertinent actions:

Siddharth Vadapalli said:

The options therefore are:
1. Bring down both interfaces and bring them up
2. Migrate to the cpsw_new.c driver (which is not present in SDK 6.3)

Best,

Josue