Ethernet FCS errors on ACK packets

Hi David,

Which board are using?

Have you checked these two posts: post1 and post2. The posts discuss that these FCS errors may be false alarms from Wireshark and discusses an option to confirm the FCS by running ifconfig (if your PC is running on Linux). The ifconfig program can give you statistics about how many packets were errors or dropped that were recieved on your PC.

Vikram

Vikram,

Yes, I saw those, but they seem somewhat different (FCS = 0) and I'm not convinced. I routinely use wireshark and no other device has this problem. Why would it single out the DSP (I'm using the C6670 EVM and PDK)?

One odd thing is the length of the packet. Most other ACK packets are 54 bytes (14 MAC header + 20 IP + 20 TCP + 0 bytes data). But the ones from the DSP are 64 bytes. There are 10 bytes after the end of the TCP. They are always zeros of 0xFF. I can't seem to upload a capture file, but see the example below.

Thanks,
Dave

Note: I was mistaken about them always being 0 of 0xFF. They usually are, but here is a different case. Always 64 bytes total with 10 bytes after the TCP.

Other hints. I'm on Windows 7 and running netstat -e shows 0 errors.

Also, the problem is not sporadic; if I run it for a while, every single len=0 packet from the DSP shows the problem but none from other nodes.

I'm going to try sending a packet with 1 byte of data and see if the C6670 is possibly forcing it to be at least 64 bytes.

Sorry for the piecemeal posting, but it does appear to be padding to 64 bytes total. Here I sent "ABCD". The packet from the PC to DSP is 58 bytes as expected. The return packet is 64 bytes.

It looks like the issue stops at len=6. If I send "ABCDE", it pads to 64 bytes and shows an FCS error. If I send 6 or more bytes, it does not pad. For example, at "ABCDEF" (ie len = 6), the bytes on wire is 60 as expected. After that, it is fine.

I guess it could be a wireshark issue, but again, not sure why it would single out the C6670 EVM board.

Hi David,

I was trying to figure this problem out last fall for a different device (OMAPL138) and I believe it's due to both a bug in the Ethernet driver, in which the driver adds a 4 byte padding for CRC when it should not, as well as a misreading in Wireshark (details on that later in this post).  The former issue is due to code like the following:

    /* Make sure the driver does not transmit packet less than min. as per the
     * Ethernet standards. */
    if( PBM_getValidLen(hPkt) < 60 )
        PBM_setValidLen (hPkt, 64 );

I believe this is unnecessary if the hardware is configured to append the FCS.  In this case, the driver should only need to pad up to 60 bytes.  This hypothesis is also in agreement with what the customer (from the original post) saw (the errors went away when he/she removed the above code).

It seems that the above code to add padding should only happen when the driver is configured NOT to add the FCS.  So, I think the above should be surrounded by a check of the hardware configuration for FCS (the value of EMAC_DSC_FLAG_PASSCRC for OMAPL138).  However, the code is still incomplete in this case, because it would also need to actually compute the frame check in the event that the h/w is not doing it.

So to summarize, for the OMAPL138 case, the driver code is adding an extra 4 byte FCS pad, and also configuring the h/w to compute the FCS, resulting in an additional 4 bytes to be added to the frames (that are < 60 bytes).  I suspect the 6670 Ethernet driver may have a similar issue.

But, it is also due to Wireshark misreading those extra 4 bytes of padding as a FCS.

I actually talked to someone from the Wireshark team on their forums about this (at least from what I recall ... now I get an error when clicking on "cmaynard's" profile, but pretty sure it showed he was a Wireshark employee last year).  You can find the post here (please make sure to click the button that says "showing 5 of 9 show 4 more comments" in order to reveal all of the posts, including mine.

In particular, the following excerpt is of interest:

"[...]by the time Wireshark gets the packet, the FCS has already been discarded by the NIC. In fact, if the FCS truly was bad, Wireshark would never even see the packet. Those last 4 bytes are part of the padding, being misinterpreted by Wireshark as an FCS due to the sending device adding an extra, completely unnecessary 4 bytes of padding. [...]"

Please have a look and let us know what you think.

Cheers,

Steve

teve,

Thanks for the info. I think you're right that Wireshark is confused, but the question is why? It has no problem with any other packets from any other device. It looks to me like the DSP is not creating a proper Ethernet frame. Not just padding, but the frame as a whole. For example, below is a typical ACK packet. It reports 54 bytes captured which makes sense (14 header + 20 IP + 20 TCP). It does not even show the padding or FCS.

B

But then for the frame from the DSP, it shows all 64 bytes of the "minimum" packet including padding and FCS.

Hi David,

Based on what I've read, it would seem that these frames with less than 60 bytes (in WS) are improperly formatted frames, which are not conforming to the standard's minimum requirement of 64 bytes.  The following from Wikipedia talks about this:

"A runt frame is an Ethernet frame that is less than the IEEE 802.3's minimum length of 64 octets. Runt frames are most commonly caused by collisions; other possible causes are underruns, a bad network card, or software bugs."

Additionally, from what cmaynard (WS employee on linked thread) said, the 4 byte CRC should be stripped off by the NIC and should not show in WS.

This would lead me to believe a "minimum" frame (whose [header + payload] length is less than 60 bytes) should show in WS as a frame with length 60 bytes, and also showing padding to make up for the header + payload size being less than 60 (i.e. it should show in WS as a frame with length 60 and format [header + payload + padding]).  Based on this, I would also expect to not see any 64 byte size "minimum" frames, as the CRC should be gone by the time WS gets the frame. (the fact that we do see the CRC in the "bad frames" that are the original topic of this forum question, I believe is explained by cmaynard's response that WS is misreading extra bytes due to extra CRC being added by the driver.  But, I don't have an answer as to why WS should interpret such a frame the way it does).

I see an example of "minimum" (ARP) frames in WS coming from my DLINK switch frames.  These frames have the format [header + payload + padding] since the ARP payload plus Ethernet header plus CRC is less than the 64 byte minimum.  From what I can tell, this frame with the padding is correct, meeting the minimum length requirement of 64 bytes (and showing in WS as having length 60 bytes, since the 4 byte CRC is stripped by the h/w and not viewable in WS):

I also see the same "minimum" frame coming out on the network from the PC's NIC - the same kind of ARP request frame.  But here, the total length in WS is 42 bytes, and there is no padding to make it meet the minimum length of 64 bytes.  This frame is in the format [header + payload] and it looks to me like this is incorrectly formatted, fitting the definition of "runt frame" that I saw in the Wikipedia page:

It seems that this frame should have padding added, so that it meets the minimum requirement; but as we can see it does not.

What do you think?  I'm open to discussion and or your thoughts on this topic.

Steve

Hi Steve,

I really appreciate your looking at this. It seems to me we can eliminate many possible causes. If hardware was bad (on the PC or DSP), it would not single out ACK packets from the DSP. I would expect to see all sorts of errors if the PC NIC was bad, and errors on things other than ACK packets on ones from the DSP.

I have the PC and DSP on a separate network, and there should be no collisions when using switches in full duplex mode. It's point to point.

Most ebmedded EMAC drivers I've used have an option to include the FCS if desired. I have no idea how to do this on Windows though.  

It seems to me this is beyond what Wireshark can handle and a full Ethernet network analyzer is required. I'd like to see all the bits on the wire including preamble, FCS, and gap. I had access to such in a previous life, but not now.

I really think it is a driver issue.

Thanks,
Dave 

Steve,

As to where else I see them, in a previous post in this thread:

It looks like the issue stops at len=6. If I send "ABCDE", it pads to 64 bytes and shows an FCS error. If I send 6 or more bytes, it does not pad. For example, at "ABCDEF" (ie len = 6), the bytes on wire is 60 as expected. After that, it is fine.

I guess it could be a wireshark issue, but again, not sure why it would single out the C6670 EVM board.

This seems to agree perfectly with the driver bug you mention. At 6 bytes of data, the total length is 14 + 20 + 20 + 6 = 60, so it stops inserting padding.

Is there some reason TI does not fix this bug? I really don't want to have to edit drivers. I had been using an FPGA Soc with an ARM. One of the main reasons I was looking at TI was the vendors drivers were buggy.

Thanks,
Dave

Dave,

I'll bring this to the attention of the driver team.

Steve

Steve,

I'm just going to go back to using the FPGA Soc, but I'm sure a lot of people would appreciate this being fixed.

Thanks for the extensive help,

Dave

Dave,

I talked with the C66x driver team, they are having someone take a look at this.

I also filed a bug for the evmOMAPL138, for which we (at my site) have control over the driver sources.  I'll ask the C66x team to file a bug or duplicate this one for their driver:

SDOCM00112837 - OMAPL138 driver adds redundant 4 byte FCS padding causing errors to show in Wireshark

Steve

Dave,

I understand that, you are sending  the packet through TCP socket from DSP to PC using NDK.

I like to reproduce FCS issue on C6670 EVM. Are you using TI test code for this or your own code for TCP socket? 

Pubesh,

I just used send() and recv() with flags set to 0.  The sockets were created as per the TI examples. I made the DSP the server, it listened for a connection, and simply exchanged some packets. Any packet with len < 6 sent from the DSP to the PC will show the FCS error.

I'm afraid I've uninstalled the TI tools, so I don't have the exact reference to the example I based it on.

Thanks,

Dave

David,

If possible share your TCP server code for test and reproduce the FCS issue. 

Pubesh,

I was just using the TI examples. I added them to a workspace but used the "do not copy" option (since they don't compile if you do copy them) and they were all deleted when I uninstalled the TI tools.

It was whichever one showed how to create a listen socket, accept a connection, and send and receive data. It's easy to reproduce. Any len < 6 packet from the DSP will show the problem.

Thanks,
Dave

Dave,

I have filed the IR(SDOCM00112902) this issue for C66x(C6670).

Whenever you see Ethernet frames that are less than the minimum frame size (64 bytes including FCS, or 60 bytes without FCS), it's almost certainly due to the fact that you are capturing packets on the same machine sending those seemingly short frames, before the data is handed to the NIC and padding and FCS are added.  If you were to capture the packets using an external capture device, you would see the properly sized minimum frames.

So in this case, Wireshark is not confused.  It simply does not see the padding because it gets handed a copy of the outgoing data before the padding and FCS are added.

On the other hand, when drivers/NICS improperly add excessive padding, Wireshark can and does get confused because it assumes that the 4 extra bytes are the FCS when they're not.

In general, unless you have special capture hardware or a NIC/driver that allows you to actually receive invalid frames (such as bad FCS, etc.), you can ignore Wireshark's misleading indication of bad Ethernet frame check sequences.  In fact, you might even choose to disable Wireshark's option to calculate those frame check sequences to begin with (Wireshark: Edit -> Preferences -> Protocols -> Ethernet -> Validate the Ethernet checksum if possible: uncheck).

By the way, in the future, if you have Wireshark-specific questions, you might want to search/post them here.

- Chris

P.S. I am not a Wireshark employee.  Wireshark is an open source project and I am merely a voluntary contributor to that project as my time and abilities allow.

Hi Chris,

Good to hear from you again, interesting that you found this thread!

Thanks again for your insights into this, it's greatly appreciated. And thanks for clarifying on your WS employment status ;)

Steve