AM6442: ICSSG1 eth port receive length is increased 1532

Hi  Expert,

More issue description for your reference, 

1) The loopback sending packet length if fixed, it is 1024 bytes, and sending frequency is one time per seconds

2) When issue occurred, the receive packet length in receive descriptor show that it is 1532, the superfluous 508 bytes (1532-1024) is "0"

3) The MAC statisc module show that the external receive packet length is right based on the calculation mentioned in above ticket.

@pengdong zhang, per talked, please upload the descriptor filed information and more related log for reference.

-Thomas

Can they program MII_G_RT_RX_STAT_BKT5_SIZE and MII_G_RT_TX_STAT_BKT5_SIZE and then monitor via MII_G_RT_RX_STAT_BKT5 and MII_G_RT_TX_STAT_BKT5 to identify the source of error. Looking at the description, it looks to be TX side issue, padding is possible if length field in descriptor is corrupted...

The Phy inner ring sends 1024 bytes, and the period is 1ms.

The MII_G statistics read indicates that the data sent each time is 1064 bytes
The data received by MII_G is 1056 bytes.

the receive thread is bind to cpu 0, The CPU usage is high.,The data cannot be received in time.

We set the length of the descriptor to 2048, there is no overwriting case

this is one of the logs

rx_good_frames = 34758

rx_broadcast_frames = 34760

rx_multicast_frames = 34767

rx_crc_error_frames = 0

rx_mii_error_frames = 0

rx_odd_nibble_frames = 0

rx_frame_max_size = 2000

rx_max_size_error_frames = 0

rx_frame_min_size = 64

rx_min_size_error_frames = 0

rx_overrun_frames = 0

rx_class0_hits = 34824

rx_class1_hits = 0

rx_class2_hits = 0

rx_class3_hits = 0

rx_class4_hits = 0

rx_class5_hits = 0

rx_class6_hits = 0

rx_class7_hits = 0

rx_class8_hits = 34850

rx_class9_hits = 34857

rx_class10_hits = 0

rx_class11_hits = 0

rx_class12_hits = 0

rx_class13_hits = 0

rx_class14_hits = 0

rx_class15_hits = 0

rx_smd_frags = 0

rx_bucket1_size = 64

rx_bucket2_size = 128

rx_bucket3_size = 256

rx_bucket4_size = 512

rx_64B_frames = 0

rx_bucket1_frames = 0

rx_bucket2_frames = 0

rx_bucket3_frames = 0

rx_bucket4_frames = 0

rx_bucket5_frames = 34886

rx_total_bytes = 36839616

rx_tx_total_bytes = 73958320

tx_good_frames = 34886

tx_broadcast_frames = 34886

tx_multicast_frames = 34886

tx_odd_nibble_frames = 0

tx_underflow_errors = 0

tx_frame_max_size = 2000

tx_max_size_error_frames = 0

tx_frame_min_size = 64

tx_min_size_error_frames = 0

tx_bucket1_size = 64

tx_bucket2_size = 128

tx_bucket3_size = 256

tx_bucket4_size = 512

tx_64B_frames = 0

tx_bucket1_frames = 0

tx_bucket2_frames = 0

tx_bucket3_frames = 0

tx_bucket4_frames = 0

tx_bucket5_frames = 34886

tx_total_bytes = 37118704

__________________________________

from the above date ,wo can get 

tx_total_bytes  =  37118704

37118704/1064 = 34,886

tx_good_frames = 34886

tx_bucket5_frames = 34886

From the above data, it can be seen that the calculated value of the total number of packets sent is equal to the statistical value, and there is no problem in sending

rx_total_bytes = 36839616

 36839616/1056 = 34866

rx_bucket5_frames = 34886

rx_good_frames = 34758

The problem seems to be reception.

I read the descriptor filed data, the receive length also is  1536 (include crc 4 Bytes).

We then changed the CPU bound to the receiving thread. This core only sends and receives tasks for network ports.

In the current test, there is no packet loss and no abnormal length when the speed is 70MB/s.

The question is, if the cpu is heavily loaded, why is the length abnormal

Hi Pengdong zhang,

We believe this issue could be because of PRU firmware cycle budget. I suggest you to run the PRU ICSSG at 333Mhz.

I would also recommend you to move to the latest SDK (pick the PRU firmware from latest SDK), as 09.00 is pretty old.

BR
JC

Unfortunately, in the long time test of 70MB/S (48 hours) (both the receiving and sending threads are bound to core 1, and core 1 has only these two threads), the length is getting longer again.

Our hardware boards have icssg ports and cpsw ports, both of which use udma and ringacc.

In the subsequent cpsw network port test, the length of the problem also appeared.
The length of sending data is 1024, the period is 1ms, the receiving thread is bound to the core 0, the core 0 cpu load is heavy, and it can reach 100% when receiving network data.
When the problem occurs, the receive length becomes 1536, and the length data in the descriptor is also 1536. After testing the driver of ringacc, no code problems were found.

Based on the above, I feel that the problem may be in the process of pis-l->dma.
I was testing the driver of icssg eth, how can I locate the problem.

Hi JC

Based on the communication with pengdong, I got the feedback as below:

1) On the high CPU loading as in below 2 cases, core 1 run receiving and sending task together, OR core 0 run receiving only but core 0 under high loading, the abnormal length issue will occur.

2) If the CPU loading is low, pendong didn't test this case, so it is hard to say whether the issue is caused by high CPU loading or not, I think your last reply to try to make PRU_ICSSG run as 333MHz is based on the issue is  caused by high CPU loading.

@pengdong zhang, please clarify and update if any new test is finished.

3) As the issue is not only related PRU_ICSSG, but also occured in the CPSW, so PISL- DMA path may have some issues there, I provided some suggestion to pengdong, but still want to get more guidance from BU expert on the suggestion, thanks!

-Thomas

This is the record of cpsw network port driver test performance.

The sending thread of board A binds core 1, and the receiving thread of board B binds core 1.

There are no other tasks in the core 1 of the two boards, and there are still length anomalies in the long-term test.

When pkt_len is detected in udma_receive as 1536, the descriptor information is printed as shown in the figure above.
The length in the log display descriptor is 1536(0x600).

In this case, the cpsw receiving port number is 0, which is normally 1 or 2.

pengdong zhang said:

When pkt_len is detected in udma_receive as 1536, the descriptor information is printed as shown in the figure above.
The length in the log display descriptor is 1536(0x600).

In this case, the cpsw receiving port number is 0, which is normally 1 or 2.

Can you check the 1536 bytes length packet contents here ? Is the padded test packet (of 1024 bytes length) or some other packet ...

Hello Pengdong,

Please also confirm which core or cores you are using to control the network interface, and what OS is running on that core (including OS version).

Regards,

Nick

Assuming the above means most frames are fine, just occasional 0x600 length. What is the ratio, how frequent are the frames with incorrect length. Can you capture entire frame in this 0x600 length case? Was Ethernet CRC still correct? Was the TCP or UDP checksum correct?

I would focus on the CPSW case just to remove one variable (firmware version, ICSSG in general). IS the error with CPSW the same, lenght is 0x600 bytes with 508 last bytes being 0 (and CRC correct)?

pengdong zhang said:

test method: phy internal loopback

What is the PHY used?

  Pekka

One more thing to check. 0x600 in the Ethernet frame length location signifies Ethertype not length. https://networkengineering.stackexchange.com/questions/51467/ethernets-frame-format-length-or-ethertype and https://www.iana.org/assignments/ieee-802-numbers/ieee-802-numbers.xhtml . Maybe it is a coincidence but exactly 0x600 length as an error in a dual purpose field occasionally seems too much of a coincidence. The previous comment still stands, repeat with CPSW, but please capture entire frame with Ethernet headers.

  Pekka

It is the padded packet. The contents of the package are correct, but there are extra zeros at the end.

It is the padded packet. The contents of the package are correct, but there are extra zeros at the end.

cppi5_hdesc_get_pktlen() return 1536.

Below is the tail of the previous test data:

b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5 c6

c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6

d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5 e6

e7 e8 e9 ea eb ec ed ee ef f0 f1 76 c9 ec 4d  0

 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

 0  0  0  0  0  0  0  0  0  0  0

both the receiving and sending threads are bound to core 1, 

Our company's own RTOS.   

The driver is based on the SDK9.0 u-boot, include udma ringacc cpsw icssg .

Where are DMA descriptors placed DDR or OCRAM? Is this possible to move them to OCRAM and test?

Pekka Varis said:

pengdong zhang said:

test method: phy internal loopback

What is the PHY used?

Can you answer this? Do you see 0x600 length fields in any other Ethernet test?

Pekka Varis said:

What is the ratio, how frequent are the frames with incorrect length. Can you capture entire frame in this 0x600 length case? Was Ethernet CRC still correct? Was the TCP or UDP checksum correct?

Especially this ratio, how often do you see these 0x600 length padded with zeroes frames?

  Pekka

1、 Was Ethernet CRC still correct?  ---> I didn't notice a CRC error, this one needs to look at which register?

2、Was the TCP or UDP checksum correct? --> We are using our own Ethernet frame, not the tcp/ip protocol stack. Except for the 0 in the tail, the previous data content is correct.

3  CPSW also has this problem

Below is the test data.

recv-0 len 1532

ff ff ff ff ff ff 2b 2c 2d 2e 2f 30 aa 99 33 34 35

36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45

46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55

56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65

66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75

76 77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85

86 87 88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95

96 97 98 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5

a6 a7 a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5

b6 b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5

c6 c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5

d6 d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5

e6 e7 e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5

f6 f7 f8 f9 fa fb fc fd fe ff  0  1  2  3  4  5

 6  7  8  9  a  b  c  d  e  f 10 11 12 13 14 15

16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25

26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35

36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45

46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55

56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65

66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75

76 77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85

86 87 88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95

96 97 98 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5

a6 a7 a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5

b6 b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5

c6 c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5

d6 d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5

e6 e7 e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5

f6 f7 f8 f9 fa fb fc fd fe ff  0  1  2  3  4  5

 6  7  8  9  a  b  c  d  e  f 10 11 12 13 14 15

16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25

26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35

36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45

46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55

56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65

66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75

76 77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85

86 87 88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95

96 97 98 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5

a6 a7 a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5

b6 b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5

c6 c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5

d6 d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5

e6 e7 e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5

f6 f7 f8 f9 fa fb fc fd fe ff  0  1  2  3  4  5

 6  7  8  9  a  b  c  d  e  f 10 11 12 13 14 15

16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25

26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35

36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45

46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55

56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65

66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75

76 77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85

86 87 88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95

96 97 98 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5

a6 a7 a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5

b6 b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5

c6 c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5

d6 d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5

e6 e7 e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5

f6 f7 f8 f9 fa fb fc fd fe ff  0  1  2  3  4  5

 6  7  8  9  a  b  c  d  e  f 10 11 12 13 14 15

16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25

26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35

36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45

46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55

56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65

66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75

76 77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85

86 87 88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95

96 97 98 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5

a6 a7 a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5

b6 b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5

c6 c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5

d6 d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5

e6 e7 e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5

f6 f7 f8 f9 fa fb fc fd fe ff  0  1  2  3  4  5

 6  7  8  9  a  b  c  d  e  f 10 11 12 13 14 15

16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25

26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35

36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45

46 47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55

56 57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65

66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75

76 77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85

86 87 88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95

96 97 98 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5

a6 a7 a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5

b6 b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5

c6 c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5

d6 d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5

e6 e7 e8 e9 ea eb ec ed ee ef f0 96 71 b2 64  0

 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

 0  0  0  0  0  0  0  0  0  0  0

recv-0 len 1532

ff ff ff ff ff ff 2c 2d 2e 2f 30 31 aa 99 34 35 36

37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46

47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56

57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66

67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76

77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85 86

87 88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96

97 98 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5 a6

a7 a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6

b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5 c6

c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6

d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5 e6

e7 e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5 f6

f7 f8 f9 fa fb fc fd fe ff  0  1  2  3  4  5  6

 7  8  9  a  b  c  d  e  f 10 11 12 13 14 15 16

17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26

27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36

37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46

47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56

57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66

67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76

77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85 86

87 88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96

97 98 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5 a6

a7 a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6

b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5 c6

c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6

d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5 e6

e7 e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5 f6

f7 f8 f9 fa fb fc fd fe ff  0  1  2  3  4  5  6

 7  8  9  a  b  c  d  e  f 10 11 12 13 14 15 16

17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26

27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36

37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46

47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56

57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66

67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76

77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85 86

87 88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96

97 98 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5 a6

a7 a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6

b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5 c6

c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6

d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5 e6

e7 e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5 f6

f7 f8 f9 fa fb fc fd fe ff  0  1  2  3  4  5  6

 7  8  9  a  b  c  d  e  f 10 11 12 13 14 15 16

17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26

27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36

37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46

47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56

57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66

67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76

77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85 86

87 88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96

97 98 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5 a6

a7 a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6

b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5 c6

c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6

d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5 e6

e7 e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5 f6

f7 f8 f9 fa fb fc fd fe ff  0  1  2  3  4  5  6

 7  8  9  a  b  c  d  e  f 10 11 12 13 14 15 16

17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26

27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36

37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46

47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56

57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66

67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76

77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85 86

87 88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96

97 98 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5 a6

a7 a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6

b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5 c6

c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6

d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5 e6

e7 e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5 f6

f7 f8 f9 fa fb fc fd fe ff  0  1  2  3  4  5  6

 7  8  9  a  b  c  d  e  f 10 11 12 13 14 15 16

17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26

27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36

37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46

47 48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56

57 58 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66

67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76

77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85 86

87 88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96

97 98 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5 a6

a7 a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6

b7 b8 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5 c6

c7 c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6

d7 d8 d9 da db dc dd de df e0 e1 e2 e3 e4 e5 e6

e7 e8 e9 ea eb ec ed ee ef f0 f1 76 c9 ec 4d  0

 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

 0  0  0  0  0  0  0  0  0  0  0

phy is YT8531

Other tests also have abnormal lengths

The appearance of this phenomenon is uncertain, sometimes minutes, sometimes hours.

The above picture shows the contents of the descriptor printed in udma_receive during normal communication。

/**
* Descriptor header, present in all types of descriptors
*/
struct cppi5_desc_hdr_t
{
u32 pkt_info0; /* Packet info word 0 (n/a in Buffer desc) */
u32 pkt_info1; /* Packet info word 1 (n/a in Buffer desc) */
u32 pkt_info2; /* Packet info word 2 Buffer reclamation info */
u32 src_dst_tag; /* Packet info word 3 (n/a in Buffer desc) */
} __packed;

pkt_info1， pkt_info2 ，src_dst_tag  is not zero。

When the receive length is abnormal, these values are 0。

pengdong zhang said:

Below is the test data.

recv-0 len 1532

ff ff ff ff ff ff 2b 2c 2d 2e 2f 30 aa 99 33 34 35

36 37 38 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45

This is all from your own driver code so lets try to confirm what you are printing out. Assuming this above is the start of the Ethernet frame? Please correct me if this does not seem correct:

destination MAC address is 0xFF FF FF FF FF FF, which is broadcast 

source MAC is 0x2B 2C 2D 2E 2F 30, looks like a prefilled pattern, not overwritten with actual MAC address

Ethertype/length field is 0x AA 99, is this some Ethertype you intentionally use?

Is this as you expected?

pengdong zhang said:

phy is YT8531

Other tests also have abnormal lengths

The appearance of this phenomenon is uncertain, sometimes minutes, sometimes hours

Is there a datasheet for this PHY? I can find https://www.motor-comm.com/Public/Uploads/uploadfile/files/20240513/YT8531C-CA_YT8531H-CA_YT8531DC-CA_YT8531DH-CA_YT8531P-CAProductBrief_v0.1.pdf but a datasheet does not seem to be available?

To rule out any HW or PHY issue, can you run the same test on the TI EVM?

Abnormal lengths, so not always 0x600 ?

Have you tried something other the PHY loobpack, do you also observe length of 0x600 (or something else corrupted)?

I don't think this has anything to do with PHY.
I'm testing two boards against each other now.

No loopback is used

yes，the schematic diagram is   [BOARD A  cpsw---phy]--- [phy----cpsw BOARD B]

the ethernet frame like the below figure

this is  our udma receive funciton 

In today's test, the descriptor's cppi5_desc_hdr_t message is the same as the one I loaded. This descriptor just pops up when it's pushed?

Which memory are descriptors stored? Can you move to OCRAM and check? Is HW cache coherency enabled in your RTOS/setup?

The descriptors stored in DDR。

It is diffculty to move descriptors to OCRAM.

The memory where descriptors reside has cache enabled

How to enable hardware consistency for am64x.

After descriptor pop from ringacc, I invalidate the descriptor cache.

pengdong zhang said:

test method: phy internal loopback

pengdong zhang said:

No loopback is used

So original issue is a loopback using the PHY loopback feature of Motor-comm YT8531 PHY. Which apparently has a loopback feature (but there is no datasheet for the PHY). But now you moved to try 2 boards. Which is a good datapoint. 

And not directly stated but it seems you are using your own RTOS on the A53's. For cache coherency and setup what is the MMU page table settings for where the descriptors and buffers are? Other settings that might be interesting or relevant, could you share the compiler flags and compiler version you are using?

Can you run this RTOS on the TI EVM? Do you see the same issue with Ethernet frames? Or can you run TI MCU+ on your board to see if the issue is present with the tested TI drivers as well (of course the PHY is not directly supported in the TI SDK).

  Pekka

Yes, There are two types of tests, loopback tests performed by one board and performance tests performed by connecting two boards.

Both of these tests have this problem.

Phy loop test was used a month ago, and two boards connected test was used a week ago.

There are now three tests:

1. In a loopback test, the board receives bound core 1, sends bound core 1, and sends 124 packets (1488bytes per packet) with a delay of 1ms. The packets are sent according to the bandwidth of 1 Gb /s.

2.
A board sends, sends thread bound core 1, sends 124 packets, each packet 1488bytes, delay 1ms, is sent according to the bandwidth of 1Gb/s.
B board receives, receives thread bound  core 1, and checks the packet serial number and length after receiving

3. The loopback test was carried out with the development board of Tronlong ( same as TI EVM).

All three tests have the problem of abnormal length

Later, I will add the Settings of MMU and cache。

In udma_receive(), 

invalidate_dcache_range((ulong)desc_rx, (ulong)(desc_rx + ucc->hdesc_size));  

There's something wrong with this?

The range of parentheses in the code is wrong ?

It should be invalidate_dcache_range((ulong)desc_rx, (ulong)(desc_rx ) + ucc->hdesc_size);  

But I cannot understand the root cause.

Why does too much invalidate cache cause this problem.

Is there any similar issue with flush_dcache_range in the code?

No.

Below is the code in udma_send().

flush_dcache_range((unsigned long)desc_tx, ALIGN((unsigned long)desc_tx + uc->config.hdesc_size, ARCH_CACHE_MINALIGN));

There is similar issue with u-boot code.

Exact memory settings (MMU page tables) and what else happens to be cached can lead to weird behavior if you are randomly flushing caches for areas unrelated to what you are intending to do. I would suggest to be extremely disciplined in your MMU setup and privileges of the SW you are running.

This is especially true if your driver is running in EL1 exception level.

For general stress testing of your SW cache management I'd recommend running a multimegabyte memcpy() in parallel as a cache trasher, for example instead of idle to uncover race conditions.

  Pekka