AM623: TX checksum offloading not working

Hello MK,

First a couple of questions

1. Are you testing on a custom AM6234 board or a TI AM62x SKEVM?

M K said:

In Wireshark I can see that the checksum is untouched and still the partial checksum is transmitted.

2. Can you share the Wireshark capture where you are seeing this?

M K said:

After disabling TX checksumming (ethtool --offload eth2 tx off)

3. The fact that you are using "eth2" interface indicates to me that perhaps you are testing with a custom designed board? AM62x can only support two CPSW Ethernet interfaces, which on the TI EVM is by default "eth0" and "eth1".

4. Can you share the results of "ethtool -k ethX" where ethX is the interface you are testing with? (Please share the result with Insert>Code)

M K said:

I use the Kernel sources from ti-rt-linux-6.1.y branch

From my understanding, TX checksum offload should be enabled by default on the TI SDK. I would suggest using the RT-Linux Kernel used on the latest TI SDK if possible (Kernel 6.6).

-Daolin

Hello,

Daolin Qiu said:

Are you testing on a custom AM6234 board or a TI AM62x SKEVM?

I am using a TQMa62xxL SoM on a custom board.

Daolin Qiu said:

Can you share the Wireshark capture where you are seeing this?

Sure: 2480.dump.zip

172.17.114.97 is my host, 172.17.114.98 is the board. Packet no. 22 is received, but still contains the partial checksum.

Daolin Qiu said:

The fact that you are using "eth2" interface indicates to me that perhaps you are testing with a custom designed board? AM62x can only support two CPSW Ethernet interfaces, which on the TI EVM is by default "eth0" and "eth1".

eth0 is a virtual interface, the CPSW interfaces are eth1 and eth2. On eth1 there is a switch attached on the board and only VLAN traffic going through it. I do not experience the same issue there, but maybe checksum offloading with VLANs is disabled by default?

Daolin Qiu said:

Can you share the results of "ethtool -k ethX"

~ # ethtool -k eth2
Features for eth2:
rx-checksumming: on
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
	tx-tcp-segmentation: off [fixed]
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp-mangleid-segmentation: off [fixed]
	tx-tcp6-segmentation: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: on
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

Daolin Qiu said:

I would suggest using the RT-Linux Kernel used on the latest TI SDK if possible (Kernel 6.6).

This is not possible because the base system is on Kernel 6.1

Hello MK, 

M K said:

172.17.114.97 is my host, 172.17.114.98 is the board. Packet no. 22 is received, but still contains the partial checksum.

It looks like from the Wireshark capture you are sending TCP packets. Can you share the exact Linux commands that you used to send packets from your AM62x custom board to your host PC? I want to see if I can replicate your results on my setup using a TI AM62x SKEVM. Please note, I will be using Kernel 6.6 in my test.

Additionally, is it possible for you try running the same test on the eth1 interface (sending the TCP traffic) and see if you observe the same issue?

-Daolin

Hello,

I am using iperf for testing: On the board "iperf -s" and then on the host "iperf -c 172.17.114.98". The issue is also visible vice versa.

Yes, I can observe the same issue on eth1 if I use the same traffic (without VLAN). But as I explained earlier, this interface usually only has VLAN traffic on it.

Thanks.

Hello MK,

Thanks for clarifying. Can you please point me to where exactly in the Wireshark dump that you see the partial checksum? I'm looking at packet 22 checksum but I'm not seeing exactly where it indicates that there is a partial checksum. 

When I tried to replicate your test, I'm not seeing where it is indicating a partial checksum in my Wireshark dump, perhaps I'm looking in the wrong part of the dump?

Ideally, if you can also share the expected result of the working case when tx checksum offload is turned off and that would help.

-Daolin

Hello,

you need to enable checksum parsing in Wireshark by right clicking the TCP entry and then selecting "Protocol settings → Validate the TCP checksum if possible". Once enabled the 0x3d15 is shown as partial checksum.

Here is a dump of the working case: iperf_working.zip 
In this case Wireshark can verify the checksum as "correct" and iperf also accepts it.

Hi MK, 

M K said:

you need to enable checksum parsing in Wireshark by right clicking the TCP entry and then selecting "Protocol settings → Validate the TCP checksum if possible". Once enabled the 0x3d15 is shown as partial checksum.

Thanks for sharing this step. I see the following from your previous Wireshark capture after enabling "Validate the TCP checksum if possible" on packet 22. 

I'm trying to replicate this behavior on my AM62x SKEVM (Kernel 6.6), however I see a different behavior in that many TCP packets have an incorrect checksum rather than a partial checksum. This behavior occurs whether the tx checksum offload is enabled or not. A similar behavior is observed when I tested with Kernel 6.1.

I will need some time to work with the internal team to investigate this problem. I plan to give an update by Thursday.

In the meantime, I would need to be able to observe this partial checksum issue on a TI EVM. Do you have access to an AM62x SKEVM? Are you able to see the same behavior on the TI EVM?

-Daolin

Hello,

no, I do not have access to an AM62x SKEVM. We started development on the STKa62xx Starter Kit from TQ. I can setup a test on this one if it helps.

Hello MK,

M K said:

We started development on the STKa62xx Starter Kit from TQ. I can setup a test on this one if it helps.

Thanks for clarifying this. I'm not as familiar with the starter kit from TQ and do not have one on my end. However, if you can check on the TQ starter kit and see if you see the same partial checksum issue, it may be one additional data point to help with investigating this issue. 

Daolin Qiu said:

I will need some time to work with the internal team to investigate this problem. I plan to give an update by Thursday.

I've created an internal bug to track this issue of the TCP checksum errors I'm seeing on AM62x SKEVM. My guess is that we didn't see this issue before because we have only checked if iperf3 log results in little to zero retries and haven't specifically grabbed a Wireshark log and enabled the "Validate TCP checksum if possible" setting. 

Can you share what your iperf3 log looks like? Please share both the client log and server log (i.e. the iperf test that resulted in the partial checksum).

-Daolin

Hello,

I tested with iperf3 now and observe exactly the same results. Here are the dumps from the board and the host: iperf3.zip

Also on the TQ starter kit the behavior is the same, which is expected, because the SoM is the same on both boards.

If you have any news on this please let me know.

Thanks.

Hello MK, 

I did little more research on this "partial checksum" shown by Wireshark when you enable the "Validate the TCP checksum if possible" setting and came across this: https://www.wireshark.org/docs/wsug_html_chunked/ChAdvChecksums.html#:~:text=Partial%20Checksums,it%20in%20the%20checksum%20field. 

I'm interpreting this information on how Wireshark displays TCP checksum as the partial checksum is expected when checksum is offloaded. However, please note, since this is on how Wireshark displays packet information, I am not in a position to confirm if my understanding is correct or not as a TI representative.

Further supporting my interpretation is the fact that your previously shared working Wireshark dump indicated that the Host PC transmit to DUT path also produced "partial checksum", which is indicative that perhaps you have TX checksum offload enabled in the Host PC and you would still see the "partial checksum". If this is true, then the "partial checksum" doesn't seem to be unique to the AM623 you are using. I would suggest checking with Wireshark directly if this partial checksum is expected when checksum is offloaded.

-Daolin

Hello,

Yes, partial checksum is expected when offloading, but only on the sender side. The receiver should get the complete checksum.
Please have a look at this side-by-side comparison:

Left: Target, Right: Host

In the packet from Host to Target, partial checksum is shown on the Host, then the hardware correctly inserts it and it is received as complete correct checksum on the Target.

In the packet from Target to Host, partial checksum is shown both on the Target and the Host, the hardware did not insert the complete checksum.

On a second look I noticed, that on the Target a VLAN header is present. I am not sure whether this comes from my networking setup or from the Target itself.

Regards.

Hello MK, 

M K said:

Yes, partial checksum is expected when offloading, but only on the sender side. The receiver should get the complete checksum.
Please have a look at this side-by-side comparison:

Thanks for clarifying this. I am checking internally with our CPSW experts to see this behavior is expected from our TX checksum offload or not. I hope to respond by Tuesday next week. If you don't receive a response by then, please kindly ping this thread.

As a note, I will also be out of office next week so my responses any new communication here may be delayed.

-Daolin

Update:

Can you please check if the RX_CHECKSUM_EN bit in the CPSW_P0_CONTROL_REG register is set? You can use the "devmem2" utility to check the contents of the CPSW_P0_CONTROL_REG by specifying the register address. 

More details in 12.3.1.4.8.6 CPPI Receive Checksum Offload of the AM62x TRM. 

Also, I had previously asked this: Can you share what your iperf3 log looks like (this is NOT the wireshark capture, simply the output of iperf3)? Please share both the client log and server log (i.e. the iperf test that resulted in the partial checksum).

-Daolin

Hello Daolin,

Do you have any news from the tests on your hardware?

To answer your last questions:

RX_CHECKSUM_EN bit is set (CPSW_P0_CONTROL_REG = 0x1).

There is no helpful iperf3 log output with the failure:

Board $ iperf3 --verbose --debug -c 172.17.114.1
iperf 3.12
Linux (none) 6.1.112 #9 SMP Wed Apr  2 16:31:58 CEST 2025 aarch64

Server $ ./iperf3 --verbose --debug -s
Linux (none) 5.10.0-34-amd64 #1 SMP Debian 5.10.234-1 (2025-02-24) x86_64
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

With checksum disabled the log is:

Board $ iperf3 --verbose --debug -c 172.17.114.1
iperf 3.12
Linux (none) 6.1.112 #9 SMP Wed Apr  2 16:31:58 CEST 2025 aarch64
Control connection MSS 1448
Time: Fri, 28 Mar 2025 09:23:28 GMT
Connecting to host 172.17.114.1, port 5201

Cookie: doloqapokdq4wxip556y7idj2fmpr2jhgkno
TCP MSS: 1448 (default)
[  5] local 172.17.114.3 port 47304 connected to 172.17.114.1 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  12.0 MBytes   100 Mbits/sec    0    259 KBytes
[  5]   1.00-2.00   sec  10.9 MBytes  91.2 Mbits/sec    0    259 KBytes
[  5]   2.00-3.00   sec  11.2 MBytes  93.8 Mbits/sec    0    259 KBytes
[  5]   3.00-4.00   sec  11.2 MBytes  93.8 Mbits/sec    0    259 KBytes
[  5]   4.00-5.00   sec  11.2 MBytes  93.8 Mbits/sec    0    259 KBytes
[  5]   5.00-6.00   sec  11.2 MBytes  93.8 Mbits/sec    0    259 KBytes
[  5]   6.00-7.00   sec  11.2 MBytes  93.8 Mbits/sec    0    259 KBytes
[  5]   7.00-8.00   sec  11.2 MBytes  93.8 Mbits/sec    0    259 KBytes
[  5]   8.00-9.00   sec  11.2 MBytes  93.8 Mbits/sec    0    259 KBytes
[  5]   9.00-10.00  sec  11.2 MBytes  93.8 Mbits/sec    0    259 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   112 MBytes  94.2 Mbits/sec    0             sender
[  5]   0.00-10.07  sec   112 MBytes  93.3 Mbits/sec                  receiver
CPU Utilization: local/sender 3.0% (0.3%u/2.7%s), remote/receiver 1.0% (0.1%u/0.9%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

Server $ ./iperf3 --verbose --debug -s
Linux (none) 5.10.0-34-amd64 #1 SMP Debian 5.10.234-1 (2025-02-24) x86_64
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Time: Wed, 02 Apr 2025 14:35:55 GMT
Accepted connection from 172.17.114.3, port 47300
Cookie: doloqapokdq4wxip556y7idj2fmpr2jhgkno
TCP MSS: 0 (default)
[  5] local 172.17.114.1 port 5201 connected to 172.17.114.3 port 47304
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  10.8 MBytes  89.1 Mbits/sec
[  5]   1.01-2.01   sec  11.1 MBytes  93.1 Mbits/sec
[  5]   2.01-3.02   sec  11.2 MBytes  93.9 Mbits/sec
[  5]   3.02-4.01   sec  11.1 MBytes  93.8 Mbits/sec
[  5]   4.01-5.02   sec  11.2 MBytes  93.8 Mbits/sec
[  5]   5.02-6.01   sec  11.1 MBytes  93.8 Mbits/sec
[  5]   6.01-7.02   sec  11.2 MBytes  93.8 Mbits/sec
[  5]   7.02-8.01   sec  11.1 MBytes  93.8 Mbits/sec
[  5]   8.01-9.02   sec  11.2 MBytes  93.8 Mbits/sec
[  5]   9.02-10.02  sec  11.1 MBytes  93.8 Mbits/sec
[  5]  10.02-10.07  sec   640 KBytes  94.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate
[  5] (sender statistics not available)
[  5]   0.00-10.07  sec   112 MBytes  93.3 Mbits/sec                  receiver
rcv_tcp_congestion cubic

Regards

Hello MK, 

M K said:

To answer your last questions:

RX_CHECKSUM_EN bit is set (CPSW_P0_CONTROL_REG = 0x1).

There is no helpful iperf3 log output with the failure:

Thanks for sharing this. From past tests I've ran with iperf3 with the default setting (which is when checksum offloading is enabled), I was able to see iperf3 log output showing the Interval, Transfer, Bitrate, etc that you see when the checksum offload is disabled. There clearly is some different behavior on my tests with the AM62x TI EVM and the TQ AM62x SOM you are using since I'm unable to replicate the same issue you are seeing.

Just to clarify, you didn't make any changes to the am65-cpsw-nuss.c driver?

M K said:

On a second look I noticed, that on the Target a VLAN header is present. I am not sure whether this comes from my networking setup or from the Target itself.

May I ask, where are you observing the VLAN header present on the target? Was this something that was showing in your Wireshark?

-Daolin

Hello Daolin,

Daolin Qiu said:

Just to clarify, you didn't make any changes to the am65-cpsw-nuss.c driver?

I have modified the driver to allocate buffer descriptors from specific physical pool, but no register configuration is changed.

Daolin Qiu said:

May I ask, where are you observing the VLAN header present on the target? Was this something that was showing in your Wireshark?

You can see it in the first Wireshark screen shout (host to target): 

On the left side after the MAC addresses (... 2d) there is a VLAN header 81 00 00 00, which is not visible on the right (sender) side.

Regards

Hi MK,

I was finally able to observe some "partial checksum" frames. I simply had an older version of Wireshark (I'm assuming the Wireshark version you were using was 4.2 or above). However, in my setup where host is 172.168.1.1 and target (AM62x SKEVM) is 172.168.1.170, when target is sending to host, the checksum is correct and there is no partial checksum, when Wireshark is captured on the host side. The only partial checksum that shows up is when looking at the host to target frame, again when Wireshark is captured on the host side, which is expected based on the Wireshark documentation.

Target to host, Wireshark captured on host side:

Host to target, Wireshark captured on host side:

Based on these results, I still am not able to observe your findings (target to host resulting in partial checksum). 

Since the correct behavior is observed when TX checksum offload is disabled, would disabling the checksum offload be a possible workaround for your use case? 

-Daolin

Hello Daolin

Daolin Qiu said:

Since the correct behavior is observed when TX checksum offload is disabled, would disabling the checksum offload be a possible workaround for your use case? 

yes, this is a possible workaround which we currently use. However, it would be nice to identify the cause of the problem, if it is a hardware or driver issue.

I have another cots base board available and will check the behavior there.

Regards.

Hi MK, 

M K said:

However, it would be nice to identify the cause of the problem, if it is a hardware or driver issue.

As I shared in my findings from my last response, I was unable to observe the issue you found. On my setup, it appears that the TX checksum offload does work since I do not see partial checksum results when checking the packets received on the host from the target. At the moment, I do not see any particular indication that it is a driver issue.

M K said:

I have another cots base board available and will check the behavior there.

If it's not a driver issue, it may be a hardware issue. You can also take a look at the schematic and layout files for the SK-AM62x EVM to see if there are any differences in your board and the SKEVM. https://www.ti.com/lit/zip/spar001

Please let me know if you have follow-up questions/findings.

-Daolin

Hello Daolin,

for our system configuration we want separate interrupts for RX and TX. So we changed MAX_EVENTS_PER_VINT in driver/irqchip/irq-ti-sci-inta.c from 64 to 1. While everything seemed to work fine, now I found that this change causes the TX checksum to fail, if I change it back to 64 iperf runs successfully. Do you have any idea why this change has an effect? I do not see a connection between the interrupt configuration and the CPSW hardware checksum feature.

Not working configuration:

ti-sci 44043000.system-controller: stub) TI_SCI_MSG_SET_IRQ - i-src_id:28 i-src_index:0x1213 i-adev:28 i-vint:5 i-gevt:13 i-bit:0
ti-sci 44043000.system-controller: stub) TI_SCI_MSG_SET_IRQ - i-src_id:28 i-src_index:0x1613 i-adev:28 i-vint:6 i-gevt:14 i-bit:0

Working configuration:

ti-sci 44043000.system-controller: stub) TI_SCI_MSG_SET_IRQ - i-src_id:28 i-src_index:0x1213 i-adev:28 i-vint:5 i-gevt:13 i-bit:0
ti-sci 44043000.system-controller: stub) TI_SCI_MSG_SET_IRQ - i-src_id:28 i-src_index:0x1613 i-adev:28 i-vint:5 i-gevt:14 i-bit:1

Thanks and regards.

Hi MK, 

M K said:

So we changed MAX_EVENTS_PER_VINT in driver/irqchip/irq-ti-sci-inta.c from 64 to 1. While everything seemed to work fine, now I found that this change causes the TX checksum to fail, if I change it back to 64 iperf runs successfully. Do you have any idea why this change has an effect? I do not see a connection between the interrupt configuration and the CPSW hardware checksum feature.

When it comes to making custom modifications to non-TI specific Linux drivers, it is harder for us to interpret what the effects of those modifications will be. In other words, we won't be able to offer specific support or give a specific conclusion to why changing the interrupt configuration will impact the CPSW hardware checksum. 

That being said, I'm not fully understanding how changing the MAX_EVENTS_PER_VINT to 1 helps separate the interrupts for RX and TX? I'm assuming in this case you are referring to the RX and TX on the Ethernet interface (e.g. RX and TX on RGMII interface)? From my understanding the CPSW Ethernet interface has only 1 RX queue and 8 TX queues, perhaps these are related to the RX and TX interrupts you are referencing?

-Daolin