Compiler/66AK2G12: NDK Recceive Problem in TCP socket programming program

Hi,

Can you explain what the example you used for this test? Is it a TI NIMU example or one you wrote? I don't see particular reason the program stopped at 1024 bytes, but it is possible that it break at 1500 bytes due to the MTU size.

I would suggest you try a K2G EVM with a PC to determine at what packet size it break? Tx/Rx of 6500 bytes needs jumbo packet supports. Before enabling that, I want to make sure regular 1500 bytes TCP/IP working as expected.

Regards, Eric  

Hi Eric,

Im replyig on behalf of selva.

Actually we coudn't find TI NIMU example code for TCP, but there was a FTP example NIMU_FtpExample_evmK2G_armExampleproject. By refereing this project we wrote TCP server and client.

When we send 1350 bytes of a packet from the client, it received properly as a single packet in server and we could able to send more packets continuosly from client and the same is received in server without any issue.

But when we send 1355 bytes of data from client, it gets splitted and received as two packets in server and further no packets are received in server when we try to send more packets from client.

We even tried K2G EVM with a linux machine and K2G EVM as server and linux machine as client. Here also we observed same behaviour i.e there is no issue with packet size less than 1350 but we are facing issue when we send packet size of 1355.

Please help us to solve this issue ASAP.

Thanks 

Hi,

Thanks! So you wrote your own TCP/IP socket program based on the NIMU FTP example for K2G. I recall that FTP example used 1500 bytes packet size for transfer so I don't see any restriction on 1350 bytes.

Please see http://software-dl.ti.com/processor-sdk-rtos/esd/docs/latest/rtos/index_Foundational_Components.html#ndk 4.10.2.5.1.2. NDK Example Description

There are TCP/IP server example on other processors (called client program on C665x, C667x and AM57x), the source code is available under the K2G installation \pdk_k2g_1_0_ xx\packages\ti\transport\ndk\nimu\example\client\src as well. Are you able to port this code to K2G and see the same issue?

Regards, Eric 

Thanks Eric.

I looked the code in the path you have mentioned. In the client code of function NIMU_testNetworkOpen, they are calling variuos daemon services for tcp and udp. But i couldn't find the function definition for dtask_tcp_echo.

Is there any working example CCS project for TCP ? can you please share it with us.

The FTP example project should work for packet size greater than 1500 also , right?

Hi Eric,

Thanks for your response but it didn't resolve our issue.

We are facing multiple compilation issue while porting especially from nimu_cppi_qmss_iface.c file. As we are already running out of time, can you please help us with TCP example specific to K2G device?

Do we have to raise any special service request ticket for this?

Even though we faced multiple errors while porting client.c completely , we tried creating TCP socket using DaemonNew as in client.c (as FTP example didn't use DaemonNew for socket creation). But this time also no luck, the same 1350 bytes of packets only we are receiving using recvnc function and if we send 1400 bytes of packet from client, we are not able to receive that packet using  recvnc function, it always returns with -1 as no. of bytes received.

Hi Eric,

Thanks for your reply.

We tried changing NDK PBM buffer size but after changing the size ping is not working for less packet size also.

Were you able to change at your side and is it working for you? 

And one more thing what we are trying modify either buffer size or enabling JUMBO packet will affect L2 layer.

But we are facing issue at application level itself i.e send() and recv() functions.

Even if we send greater than jumbo packet size i.e 6500, we should not face issue at application level, right?  The packet should have assembled/fragmented in the middle layers according to MTU size, right?

When we communicate with two linux machines, we are not facing any issue even when we send and recv packets which of size 64K at application layer. 

Hi,

I tried to change the A15: open_cfg.max_pkt_size = 1514 and got exception, this needs further debug. Please use C66x core for the network interface if possible.   

Regards, Eric

HI Eric,

Have you tried in C66x,? is it working for you?

We have tried NIMU_BasicExample_evmK2G_c66xExampleProject and NIMU_BasicExample_iceK2G_c66xExampleProject. In both the cases till NetworkOpen is called but dtask_udp_hello is not called.

Hi,

I am travelling and will be back Thursday. I tried ping test using NIMU_BasicExample_evmK2G_c66xExampleProject on K2G EVM, it worked for packet size 1514 bytes on C66x. Ping is an ICMP test will not call dtask_udp_hello().

And you are developing TCP/IP, what you need is  

hEcho = DaemonNew( SOCK_STREAMNC, 0, 7, dtask_tcp_echo,
OS_TASKPRINORM, OS_TASKSTKNORM, 0, 3 );

Such call is inside client example, http://software-dl.ti.com/processor-sdk-rtos/esd/docs/latest/rtos/index_Foundational_Components.html#ndk, you need to port client example to K2G EVM. Test TCP/IP you can use tools: Some Windows host test applications are available under ndk_3_xx_xx_xx\packages\ti\ndk\winapps. I expect TCP/IP work on C66x with 1514 bytes.

Regards, Eric

Hi Eric,

I tried TCP in C66, at the maximum, i could able to receive 1454 bytes of packet using recvncfrom() function. When i send a packet more than that size, im not able to receive it and apllication blocked in recvncfrom() function.

It seems like there is issue in C66 also.

Why such hang is there in both arm and c66?

While communicating with two linux machines we could able to receive 64KB of data using send() and recv() functions without any issue. We want such feature in 66AK2G12 arm core also. Have you checked it?. This become blocker for us proceeding further.

Hi Eric,

A gentle remainder. Is there any update on this?

Hi,

I rebuilt the NDK with jumbo support enabled, I didn't see any issue to build it. My ndk.mak and build log are attached.

I firstly built the NDK without .bld changes, to make sure the build process worked. Then I merged your two .bld files, the build is still worked. 
ndk_build.log

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/81/ndk.mak

Regards, Eric

Hi Eric,

We are able to compile ndk now.

Next step for nimu modification,  I check v5/nimu_eth.c file.  In nimu_alloc_pkt(), I have modified PBM_alloc(10236).  For compiling nimu, in terminal we using command gmake nimu. I don't where should I add macro _INCLUDE_JUMBOFRAME_SUPPORT. Because in that link for compiling jumbo packet , they compiled nimu as ccs project but we are compiling it through gmake command.

Have you enabled jumbo packets? After that can you able to send jumbo packets? Can you please share the steps

Hi,

Per today's meeting, you don't need to have jumbo packet support enabled. I looked at several other TI devices and tested with a PC to see how big the packet can be passing through. I found another issue, inside pdk_k2g_1_0_15\packages\ti\transport\ndk\nimu\src\v5\nimu_eth.c

open_cfg.max_pkt_size = 1514; =======> This should be 1518. After this change, I can ping with 1400, 1510, 1514, 1528 (will be segmented into two packets), working cases as below capture. This is the same behavior of a Linux PC to PC, you will see the 1514 bytes packet on Wireshark. Bigger than 1514 will be segmented. A working case capture like below (K2G and PC test)

However, the K2G crashed after a while, still in debug.

Regards, Eric

Thanks Eric.

We too observed the same here. Please update us when there is any improvement

Hi Eric,

From the above experiement, hope you able to clearly reproduce the issue from your end and progressing on debug. Please let us know if you want us to try/test any experiement/workaround from our end aswell. 

Regards,

Balaji A R 

Balaji,

I tested the K2G running Linux, then I tried to ping it with 1514 bytes packets for several minutes, it worked. So this is a pure RTOS/NDK software issue on K2G, and can be fixed. This is not hardware issue on the K2G and shouldn't block your board design.

Attached is the wireshark capture.

• 158.218.117.176 is K2G running Linux

[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.14.79-gbde58ab01e (oe-user@oe-host) (gcc version 7.2.1 20171011 (Linaro GCC 7.2-2017.11)) #1 SMP PREEMPT Thu Dec 20 03:57:44 UTC 2018

• 158.218.112.223 is a PC.

Regards, Eric  https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/81/ping_5F00_1514.pcapng

Hi,

I hope I have good news for you. We debugged the issue in the past several days and root caused it.

At this moment, you should be able to run your TCP server application on A15 core with a PC to support 1514 bytes at layer 2 for dozens of packets then the application crashed. You should also be seen the same issue using simple ICMP ping with syntax: ping <IP> -l 1472 -n 1000.

If you run the same application on C66x, you should see the TCP server didn't work at all.  ping <IP> -l 1470 -n 1000 should work, this is 1512 bytes packets, 2 bytes short of 1514.

Regards, Eric

Hi,

I am attaching an engineering build, can you verify at your end? The test is done with K2G 6.1 release (pdk_k2g_1_0_15) and NDK ndk_3_61_01_01, on TI K2G general purpose EVM.

Instructions:

0. Use the original NDK package, no change

1. Use the attached NIMU lib to replace pdk_k2g_1_0_15\packages\ti\transport\ndk\nimu\lib, this is for A15 and C66x

2. Use the attached pdk_k2g_1_0_15\packages\ti\transport\ndk\nimu\example\helloWorld, see there are 3 files changed. 

• pdk_k2g_1_0_15\packages\ti\transport\ndk\nimu\example\helloWorld\k2g\armv7\bios\helloWorld.cfg
• pdk_k2g_1_0_15\packages\ti\transport\ndk\nimu\example\helloWorld\k2g\c66\bios\helloWorld.cfg
• pdk_k2g_1_0_15\packages\ti\transport\ndk\nimu\example\helloWorld\src\helloWorld_k2g.c

Highlight the change in .cfg file:

Tcp.transmitBufSize = 16384;
Tcp.receiveBufSize = 65536;
Tcp.receiveBufLimit = 65536;
Global.pktNumFrameBufs=384;
Global.pktSizeFrameBuf=1664;

Global.stackInitHook = "&stackInitHook";
Global.networkOpenHook = "&my_network_OPEN";
Global.networkCloseHook = "&my_network_CLOSE";
Global.networkIPAddrHook = "&my_network_ip_addr";

//Global.enableCodeGeneration = false;

//var tskNdkStackTest = Task.create("&StackTest");
//tskNdkStackTest.stackSize = 0x1400;
//tskNdkStackTest.stackSize = 0x5000;
//tskNdkStackTest.priority = 0x5;

You should be able to build the test application NIMU_BasicExample_evmK2G_armExampleproject and NIMU_BasicExample_evmK2G_c66xExampleProject (Note, don't add nimu_eth.c and pbm_data.c to the project).

3. Please verify that you can ping the K2G with ping <IP> -l 1472 -n 1000 without any failure.

4. Finally merge your TCP server with the pdk_k2g_1_0_15\packages\ti\transport\ndk\nimu\example\helloWorld\src\helloWorld_k2g.c. The ping with -l 1472 should work, and Linux application sending 64KB should work.

Note, this is TI engineering build for debug purpose only, the official code will be delivered in the next Processor SDK RTOS package.

lib.zip

3173.helloWorld.zip

Regards, Eric

Hi,

Did you get a chance to try? Let me know the plan.

Regards, Eric

Thanks Eric for your Engineering build.

TCP is working for us in K2G with your new Engineering build changes.

I need one clarification. I observed following things in wireshark, while testing K2G(192.168.1.4) with linux PC(192.168.1.10) by sending and receiving 64K words,

Linux PC always splits packets into 1514 bytes while sending packets to K2G

Linux PC receives packets which is greater than 1514 bytes (please refer attached image).

So whether Jumbo packets enabled in K2G in this Engineering build, inorder to fix our issue?

Hi,

Thanks for the testing! Jumbo packets supporting need both NIMU and NDK change. We used the original NDK package for the testing, and it doesn't support jumbo. When we tried to ping with big packets, we saw the packet was split at 1514 byte boundary. 

I recall we worked before to enable jumbo packet supports in NDK by re-built it. Is it possible you used that NDK and saw the jumbo issue? Please use the original NDK. 

Also, is that correct you will use C66x for the TCP server? Please make sure your tested on the target core, although we verified on both A15 and C66x.

Regards, Eric  

Hi Eric,

We are using original NDK. We are using new development PC for this Engineering build.  Freshly we have installed CCS and pdk in this development PC. Other than Engineering build,  no changes done in this development PC.

I still don't understand how linux PC is receiving such large packets.  Can you please confirm that K2G is splitting the packets at 1514 byte boundary, while we use send() function in application layer?

We are going to use a15, anyhow I will test in c66x and will update you.

What is the NDK version you used? The NDK doesn't support jumbo by default. I will check send() function on the K2G side.

Regards, Eric 

Hi,

We don't have socket API example. But I have a NIMU_BasicExample_evmK2G_ARM/c66xExampleProject, it starts a UDP dtask_udp_hello(). There is a 

if( i >= 0 )
{
sendto( s, pBuf, i, 0, (struct sockaddr *)&sin1, sizeof(sin1) );
recvncfree( hBuffer );
}

Where i is packet size, I was able to change that i to different values and see what I captured on Wireshark. For example, i = 1000, 1400 , 1472, I have packets 1042, 1442 and 1514 bytes. When i > 1472, I don't have anything on the wire. So, I don't think NDK supports beyond 1514 bytes.

Regards, Eric

Hi Eric,

We are using ndk_3_61_01_01. 

Can you please verify TCP by sending packets greater than 1500 at your end by referring example present in C:\ti\ndk_3_61_01_01\packages\ti\ndk\tools\servers\newservers.c?

Since we are going to TCP in our target board.

Hi,

Can you send us the *.map file to double check?

Regards, Eric

Hi Eric,

Please find the attached .map fileNIMU_BasicExample_evmK2G_armExampleproject_map.zip

Hi Eric,

Is there any update? 

Hi,

We looked at the map file and didn't see anything wrong. So it is not clear why the jumbo packet was sent out by K2G. Is this an issue breaking the transfer with the PC so needs to be resolved or just an observation you want to understand?

Regards, Eric

Hi Eric,

If K2G is sending Jumbo packets it need to be resolved. Were you able to reproduce this issue at your end?

Hi,

No, I don't get chance to modify an example for socket application yet so I was not able to test. Is that possible you share a CCS project showing the problem, so I can reproduce? 

Regards, Eric

Hi Eric,

Attached the source code we have used in TI project NIMU_BasicExample_evmK2G_armExampleproject. You can copy the content of this file 3480.to_eric.zipin your project. and also attached the linux source code for TCP client

Hi,

Can you ping the K2G with packet size greater than 1514 bytes? Will you see K2G respond with jumbo packet or fragmented 1514 byte packets?

Regards, Eric

Hi,

When we try to send packets size greater than 1514 , there is no response from K2G.

Did you get a chance to try our TCP code?

Hi,

"When we try to send packets size greater than 1514 , there is no response from K2G." =========>This doesn't make sense to me. Ping with various packet size always worked for me with the fix. If basic ping test failed, there is no point to move to TCP test. Did the test fail on both DSP and ARM?  

Did you get a chance to try our TCP code? ========>Sorry, we were suggested to work for home and I don't have remote access to K2G EVM setup.

Regards, Eric 

Hi Eric,

Ping works fine for packet size greater than 1514 , when the mtu in linux configured as 1500.

But it fails, when the mtu size configured as greater than 1500 bytes.

Hi,

Thanks for the testing of ping.

>>> Ping works fine for packet size greater than 1514 , when the mtu in linux configured as 1500. But it fails, when the mtu size configured as greater than 1500 bytes.>>>>

This is expected. The K2G doesn't supports jumbo packets, it needs to be explicitly enabled in NDK driver and NIMU driver. By default, this is disabled. If Linux changes the MTU size greater than 1500, K2G has no way to receive jumbo packet from Linux machine and no way to reply back.   

The mystery is why K2G can send out jumbo packet when you do a TCP server test. I am checking with development team. 

Regards, Eric

Hi,

We don't think that can happen on K2G side. It could be WireShark presenting the packets as a single one? You may try this: https://osqa-ask.wireshark.org/questions/29028/question-about-fragmentation-packets. 

I expect we could look at the counters in the NDK to prove that the NDK is not sending the larger packets also. If you still can access the setup, please do a test (don't test too long, it is hard to count how many packets exchanged):

1. Collect a Wireshark capture with TCP packets between K2G and Linux PC, and have a rough number how many in each direction, assuming K2G sending out much less packets, but are jumbo packet! 

2. In the CCS expression window, halt the core and record NDK_tcps, NDK_ips. Please try to see those how many packets received and sent. Like below. Does counter supporting that NDK sending out jumbo packets?

Regards, Eric

Hi Eric,

We sent 4196 bytes of single frame from linux(192.168.1.10) to K2G(192.168.1.4) and then K2G(192.168.1.4) to linux(192.168.1.10).

Analysis for K2G to linux side:
In wireshark total packets captures are 7(2 DHCP, 5 TCP) , in which one packet captured with size greater than 1514 bytes.

But when i saw in NDK_tcps the SndTotal value is 6 and in NDK_ips both of the Total and Delivered value is 7.

So the NDK_ips values matches with wireshark captured values. With this can we confirm K2G is sending jumbo packets?

Attached captured screenshots for reference.

Hi,

Thanks for collecting this! I think it means the NDK sent out 7 packets on IP layer and Wireshark also recorded 7 packets, and one of them is a jumbo packet of 2974 bytes. I will check with NDK team and update here.

Regards, Eric

Hi,

From NDK:

NDK_ips.Total was 7, but that is the receive count (horrible name!!). NDK_ips.Delivered were number of packets delivered to the stack. So 7 makes sense. NDK_ips.Localout was 10. This is the number of IP packets sent. So it looks like the hardware or WireShark is combining. 

From NIMU and EMAC:

NIMU and EMAC driver does not combine multilple smaller packets into larger one.

So, the K2G sent out 10, the Wireshark showed 7. It is possible Wireshark is combining?

Also, you can look at K2G TRM (SPRUHY8I–January 2016–Revised March 2019) CPSW2G switch statistics to rule out any K2G issue. See 11.13.4.4.6.11 CPSW_2U Network Statistics

NOTE: Statistics start at physical address 0423 A000h for CPPI port 0.
Statistics start at physical address 0423 A200h for Ethernet port 1.

Regards, Eric

Hi Eric,

When I test between two linux PC, i didn't see such jumbo packets in wireshark.  I'm seeing jumbo packets only when i test between k2g and linux PC.

I will look at switch statistics and will update you.

Hi Eric,

Again i transmitted 4096 bytes of frame , in wireshark i observed 6 packets but the total tx packet count shows as 8 in K2G. I have also printed count for packet size greater than 1024 which is 6(Tx +Rx) but the total RX and Tx packets greater than 1024, captured in wireshark is 5.

Attached the captured screnshots

Hi,

Does this mean Wireshark combined the packets into jumbo, as the K2G CPSW stat showed more packets than the Wireshark saw? If you send a little bit more data, say 16K or 32K bytes, the difference will be much more? Also, there are total bytes sent counter in CPSW stat to understand the bytes sent out, it should be correct. And with total bytes and number of packets transmitted, you can confirm the average packet size, there are no jumbo from K2G? 

Regards, Eric