MSP432E401Y: TCP/IP stack issue - ack and sequence numbers

Hi Sai,

Thanks for you response.

Here's how I'm interpreting frames 149, 152, 153:

Frame 149: configuration SW sends a TCP MODBUS request. No answer is received. The answer in fact is sent by the MSP432, but it is not received by the configuration SW, due to packet loss on the network. So, according to TCP rules, frame 149 is sent again.

Frame 152: TCP on PC side forces a re-send: the same message sent in frame 149 is sent again. 

Frame 153: this is the ACK to packet 152, received by PC immediately after sending frame 152. In my opinion here we see the issue: sequence number (82) of frame 153 is not the sequence number expected by configuration SW. Expected sequence number would be 73, that is the ACK number specified in request 152. I think Wireshark signals this misbehavior with the comment "TCP Previous segment not captured". 

As you point out, here we receive only the ACK, not the response, possibly, again, due to packet loss in the network. Anyway from now on the ACKs and responses from MSP432 have "wrong" sequence numbers. Configuration SW keeps on asking an ACK number = 71 (see frames 160, 169, 178). Instead sequence numbers in the responses by MSP432 keep on incrementing.
My impression is that the sequence number keeps on incrementing without considering the expected ACK number in the request. So, when a packet is lost, ACK number in request and sequence number in the response, that should be aligned, are different.
We never recover from this situation, messages keep on following the same sequence as the last messages you can see in included logs. They go on forever.

I would expect sequence number on MSP432 to align to ACK number expected by configuration SW on PC, so responses would be considered valid by PC.

Thanks,

Luca

Hi Brandon,

Here are the files where MODBUS and TCP are implemented.

Please let me know if you need further info.

Thanks,

LucaTCP_files.7z

Hi Brandon,

Thanks.

No news, the problem keeps on occurring in the aforementioned conditions.

Hi Brandon,

Thanks for your support, I really appreciate it.

Before asking your support on this forum, I tried to figure out what happens in tcpin.c and tcpout.c files, but they're quite complicated and I didn't notice any obvious bug. I just recompiled my project including them and stepped in with the debugger.

But going back to these files suggested me a way to reproduce my "bug". You should be able to reproduce it consistently, too.

This "bug" is generated on my real device by the loss of a transmitted packet in the network. I simulate the loss of a transmitted packet (transmitted by MSP432) with the following modification in file tcpout.c:

In label send:, around line 750, I force not to call "IPTxPacket()" once every 100 times. So, 99 times TCP stack responds as expected, the 100th time (and following iterations) the stack creates the TCP response, but it does not send it. Here's how I did it:

TcpCounter++;

if (TcpCounter%100 == 0)
{

   // here's where we are not sending the response
   error = 0;
}
else
{
   error = IPTxPacket( pPkt, SockGetOptionFlags(pt->hSock) & FLG_IPTX_SOSUPPORTED );
}

My assumption is that this modification reproduces a loss of a transmitted packet. The stack performs all the needed calculations (seq, ack...), creates the response packet and no error is generated. The stack is not immediately aware that the packet has not been sent out, as it happens when a packet is lost in the network (in my case due to electromagnetic noise).

This way I can reproduce exactly the same bug that I see in case of loss of packet in the network, with the same problem of seq numbers and acks: MSP432 receives a ACK number, but responds with a different sequence number. The response sequence number keeps on increasing, whereas the request ACK number remains the same. The problem is the same that I mentioned in my previous posts.

I can reproduce the bug with our custom MODBUS client. But I can also reproduce it with a third party MODBUS client (QModMaster). Modbus is a very simple master/slave protocol, where a request is always followed by a response.

I can reproduce the bug also with HTTP traffic, using the webserver that is running on our MSP432 based device. The HTTP server sits on top of the same TCP/IP stack. Wireshark logs highlight the same problem.

I think that in general any kind of communication based on TCP/IP stack is affected by this problem.

By the way, looking into the code, I noticed that, in ipout.c file, a very similar debug code already exist, that is commented out.

At line 55, inside function IPTxPacket, I uncommented the debug code:

/*///// Simulate Trouble //////// */
/* static int foo = 0; */
/* if( !(foo++ % 13) ){ PktFree(hPkt); return(IPTX_SUCCESS); } */
/*/////////////////////////////// */

I think this code simulates exactly what I did forcing not to call the IPTxPacket tout court.

Uncommenting this debug code generates the same error scenario.

Could you please reproduce these modifications on your side? 

Thanks,
Luca

Hi Luca,

Thanks a lot for filling me in on what you've tried and found. With the modification of forcing the tcp stack to drop a packet every 100 packets I wasn't able to reproduce your exact issue, but I did hit another issue that on the surface seems unrelated to yours, but I am also chasing it down. If I force my ndk to drop every 100th tcp packet, my device retransmits the dropped packet within receiving three duplicate acks - usually after receiving one duplicate ack. The wireshark capture provided doesn't go on long enough for me to see, but from your comments it sounds like yours does not retransmit whatsoever?

Regards,

Brandon

Hi Brandon,

I also see the duplicate ACKs when the error scenario occurs, but they seem to be a side effect of the issue.

The issue can be reproduced consistently on my side, dropping the transmitted packets, as we did with a code modification.

I wil refer to the logs I attached before, to make it clear.

Message num 146, REQ, ack num = 60   -> Message num 147, RESP, seq num = 60. REQ ack and RESP seq are equal, that's ok, communication is fine.

------ Here we drop the packet   ------- from now on, we see the error:

Message num 157, REQ, ack num = 71  -> Message num 158,  RESP, seq num = 82. REQ ack and RESP seq are NOT equal, that's wrong.

From now, on the REQ ack number is always 71 (since response is not considered valid) and and RESP seq number keeps on incrementing, as you can see in the following messages in the log.

When I use a web browser to communicate with the web server of the MSP432, the wireshark log changes a little (after some time the web browser stops the requests), but the issue I get is exactly the same:

the mismatch between request ACK number and response SEQUENCE number. 

The issue occurs consistently, even though the stack seems to react correctly to the first packet dropped. From the second packet dropped on, the TPC IP stack seems to respond with an increasing seq num, despite the request ACK number remains always the same.

When the error scenario occurs, the client (our proprietary Modbus client, a generic Modbus client or the web browser) considers the response wrong and the communication does not work anymore.

I have here a MSP-EXP432E401Y demo board. Do you think it is useful if I try to reproduce the issue on the demo board, with the demo code?

Regards,

Luca

Hi Brandon,

Now I can reproduce consistently the packet loss issue, uncommenting the lines you suggested in ipout.c file. I tried to reproduce the error scenario on the demo board, with the httpserver demo, and I can't, it works properly. I see now the TCP retransmission after some DUP ACKs. So I ported the httpserver demo on our board, and again everything works properly, the TCP stack reacts correctly. 

I'll review our application implementation, there is definitively a bug on our side.

Thanks for guiding me through this debug!

Regards,

Luca