NDK server ack delay on first packet

I would also like to add that it takes almost half a second to send the 64k of data.  That seems very slow.  I have all my code in DD3.  Could this be part of the problem?

Will

Is anyone looking into this?

Thanks,

Will

Will,

Can you describe the setup for us to duplicate your issue?

- What is the NDK version or MCSDK version you ran on 6657? Is it a TCP server or UDP server?

- On the PC side, what is the application to send a 64KB data? How can the data size to be 2920 bytes? You meant you configured the data size as 2920 in your sending application, but it is up to the socket to break it down to smaller fragment?

Regards, Eric

Hello Will, sorry for the delay.

I've got a setup here that I will try to duplicate and diagnose.

Do you have any wireshark captures you can share?  I'm curious what is going on the wire, if anything, during the delays/ packet size negociation.

Mike

@ Eric,

I am using NDK 2_21_01_38 and MCSDK 2_01_02_06.  I used the cfgDemo as a basline and created a TCP server application.  The PC is running a java application that is sending 64024 bytes.  The 2920 packet size is what is seen on wireshark.  Neither the java nor the processor code has specified a size.  I have set up the Receive buffer size to be 65535 so I know the receive buffer is large enough to handle the packet.

@ Mike,

Attached is a wireshark capture.  I have to send the 64024 byte packets with a 3 second delay before sending a second 64024 byte packet.  You can see the Syn, Syn/ack, and then the final ack. Then you can see the multiple retries to send the first packet of data.  First at a size of 2920, then 2 different times at 1460, and then finally at 536 which is when the 6657 finally responds with an ack.  Once the first packet is successful, the next 10 packets seem to transfer slowly.  From the first acked packet to the final packet sent seems to take about .5 seconds which seems extremely slow for a 64024 byte payload.

8255.6657 Ethernet Capture.zip

Will,

I've looked at the wireshark trace, and there is some interesting stuff going on..  First, the initial traffic pattern is the same on all three transfers.  Next, I notice the last ~50K is transferred in ~3K chunks in about 0.007 seconds, so the traffic does ramp up. 

Maybe you could try a direct PC-to-EVM connection with static IP addresses and see if you get the same traffic pattern- this test would isolate the problem to the network?

Finally, if you would be willing to zip your code and attach it here, that would help (you can omit your client if you wish).

Mike

Hi Mike,

  Currently I am directly connected from PC to the ethernet connection on the 6657 evm.  Attached is my program I am currently using.  enetComms.c is the file with the server function "enetDiagToolConnTask".

Thanks,

Will

 1602.6657 Source.zip

Will,

Are you explicitly setting the Don't Fragment [DF] flag?  If so, try turning that off.

Also, have you tried breaking up your client write's into smaller packets, say 1000 bytes?

I'd be interested to see if the wireshark traces look any better. 

Mike

Mike,

  I am not setting the don't frangment flag.  Where is that even done?   Also, I am now using JPerf to test the performance of the server and I see the same timing issue.

Will

Mike,

  Just tested sending the packets in 1000 byte chunks and the software acts the same.  Attached is the wireshark capture.

4382.6657 Ethernet Capture(1000byte).zip

Mike,

  I just tested the client example using static ip of 192.168.1.2 and JPerf as the pc end.  I sent 64K bytes of data and the wireshark capture looks very similar.  Still takes over 2 seconds to ack on the first packet and then the packet transmits fine, just like in my project.  Is there something fundamentally wrong with the examples or ethernet stack?

Will

Will,

What is the name of the example project you tested- the only client example I can find relates to hosting HTTP clients.  I would like to run that on my setup and compare with your results to determine if this is an EVM/ network issue, or this is a stack issue, though I am not aware of any stack defects along the lines of what you're seeing.

Mike

Hi Mike,

The MCSDK example for NDK Client.  If you look at client.c, the example opens Daemons for tcp_echo, udp_echo, tcp_datasrv, tcp_nullsrv, and tcp_oobsrv.  I ran JPerf on the tcp_datasrv.  I set the localIPAddr to 192.168.1.2.  The server port is 1000.  I sent 64k bytes of data using JPerf to the server task and saw the same behavior as with my program.

Will

Will,

Thanks for that.

I am able to reproduce your problem as described.  I see the same ~2 second delay after the first ACK, then the remaining packets are blasted across.

I'll investigate and get back to you when I determine the resolution process.

Mike

Will,

I wanted to let you know we have not forgotten about you and are still working on the issue.  I am working with one of the stack authors to find the cause and a fix. 

Mike

Hi Mike,

I appreciate the update.

Will

Hi Mike,

  Any updates?

Will

Will,

We should be able to take a deeper look today.  I've been doing some experiments until I get time with the developer that may be of some help.

First, breaking up your packets into < MTU size, and making sure the bytes have actually been sent before sending more is key (don't allow the underlying host OS to buffer your send requests).  Using the client demo on the EVM and a simple python client, I found 192 bytes was a good size.  There was still some buffering occurring, despite my best efforts to purge before sending, short of sleeping for a millisecond between writes.  This achieved throughput of about 3MB/s, which is not terribly impressive.

Disabling the Nagle algorithm (enable TCP_NODELAY) allowed me to get around the buffering and send 1400 byte packets, resulting in ~15-20MB/s throughput.  The bulk of the throughput increase came from not having to verify I could write before writing.

Let me know if you need some help implementing this in C.

Mike

I seem to have a very similar problem. Besides the first ACK, other TCP ACKs are sometimes delayed. The host has to timeout and retransmit multiple times. Similar to OP, my network is also not congested.

My stack also doesn't ack 1514 bytes packets.I tried to enable TCP_NODELAY with 192 bytes. My 512k byte data was first sent in a burst and then since the host doesn't receive all the ACKs, it goes back to retransmit first unack'ed packet. Retransmission would fall back to 1514 bytes a time.

My questions is, what could cause the Ethernet stack not ACK in time? I have enough recv buffer space.

Thanks,

KC

KC,

I do not know the intimate details of the stack (yet), but my own tests have shown that packets > 1500 bytes (default MTU of Keystone stack) will cause packet delays/ re-transmissions.  

I assume you're running wireshark and can watch this as it happens- try breaking your payload up into 1000 byte chunks, work your way up to 1400-1500 to find the threshold.  

I should also note after each write/ send from my client, I call select to make sure I can write to the socket- whether or not this is really doing what I think it is doing, or just providing enough delay for the target to digest the packet, I cannot say.  Using select and NO_DELAY seemed to be redundant, but these were just quick bench tests.

Mike

Hi Mike,

  Its been a few months.  Any updates

Will

Will,

I've been working on a few unrelated NDK tasks and have seen packet size of 1514 hardcoded in a few places (1500 byte Ethernet packet + 14 byte Ethernet header).  You'll want to stay below that value, usually on the order of 1460 or 1472 bytes of payload, depending on how many headers your protocol adds to the payload.

Mike

Hi Mike,

  Are you talking about my packet to send or the packet size that is segmented my the stack?  Some of my packet sizes are greater than 1514.  Is this what causes the retries?  Is there a way around this issue because I am receiving packet sizes up to 64k and sending packets around 2.5k.

Will