NDK TCP retransmit problem with C6678

Hi g.f.,

Could you send me a Wireshark capture file that shows the issue? You can get the latest version of Wireshark from here: https://www.wireshark.org/

Could you describe your network setup? Are the two boards connected to the same switch? Anything else connected to it?

Best regards,
Vincent

Hi Vincent,

Thank you for the reply.
I'm asking the Wireshark capture file to my customer. And also asking about the connection.
Please wait for a while.
I will be back as soon as possible.

best regards,
g.f.

Hi Vincent,

Sorry for the delay.
I got log file of Wireshark from my customer.
Please look at the attached file.

They are connecting C6678 Ether directly to the CPU board and it linkup with 1Gpbs.
No other device are connected.

best regards,
g.f.

1122.log_00023_20151112103046.zip

Hi g.f.,

Could they indicate which packets in the file correspond to the problem they are seeing? Just want to make sure we are looking at the right data. Also, when they are referring to the CPU board, I am assuming they are talking about their PC - is that correct? Do they have a small project on the C6678 board they are willing to share?

Also, it may be worth checking if they can upgrade to the latest NDK and verifying if this is something that has been addressed. The version they are using is more than 2 years old. NDK releases can be downloaded here:http://software-dl.ti.com/dsps/dsps_public_sw/sdo_sb/targetcontent/index.html. Note that NDK 2.24.03.35 was validated on:

SYS/BIOS 6.37.02.27

XDCtools 3.30.04.52

They can get BIOS and XDCTOOLS from the same page.

Best regards,

Vincent

Hi Vincent,

Thank you for the reply.

>Could they indicate which packets in the file correspond to the problem they are seeing?

Please look at attached pdf. I wrote the detail in the pdf.

>Also, when they are referring to the CPU board, 
>I am assuming they are talking about their PC - is that correct?

They are connecting to their custom board not the PC.

I will ask to my customer for the C6678 project and will ask upgrading to latest NDK are possible or not.

best regard,
g.f.

0044.NDK Packet Retransmission Problem.pdf

Hi Vincent,

My customer are trying to upgrade the NDK version, SYSBIOS, etc..
So, please wait for the result.

And I got small project from my customer.
Please check the attached file.

best regards,
g.f.

8461.project file.zip

Hi g.f.,

Thanks for the pdf and the project. I wasn't sure if the captured data matches the scenario in the original post since on the capture there are 6 or 7 ACKs sent by the CPU after the packet with seq #286161 was sent, whereas the original post only showed 2, so thanks for confirming. It seems an old packet is resent after a *random* number of other subsequent packets has been sent and ACKs received. Is this a good description?

Unfortunately our NDK expert is out on leave, so let's first see what the customer says regarding the upgrade before we dig further into this.

Best regards,
Vincent

Hi Vincent,

Thank you for the reply.
Yes, your description is correct.

I will be back as soon as possible.
Sorry, but please wait.

best regards,
g.f.

Hi Vincent,

My customer tried with newest NDK v2.24.03.35.
But the same problem still occur.
Can you please take a look at the customer's project which I attached previously?

And they have additional questions about NDK.
////////////////////////////////////////////////////////////////
They started communication TASK(Daemon) by DaemonNew().
Is it possible to start multiple TASK which communicate with the same IP but different Port number
by using DaemonNew()?

The 3rd argument of DaemonNew() is Port number,
so if calling DaemonNew() multiple time with inputting different Port number each time,
they guess it is possible to start multiple TASK. So they tried.
But it didn't start successfully.
Multiple TASKs got started and it is connected to the client successfully,
but data communication is available against only one port.

How to communicate with multiple ports by using DaemonNew()?
If it is not possible, can you give me an advise how to communicate with multiple ports?
////////////////////////////////////////////////////////////////

best regards,
g.f.

Hi g.f.

g.f. said:

And they have additional questions about NDK.
////////////////////////////////////////////////////////////////
They started communication TASK(Daemon) by DaemonNew().
Is it possible to start multiple TASK which communicate with the same IP but different Port number
by using DaemonNew()?

DaemonNew() creates a server thread.  It will create a socket that listens for new connections on the port that was passed to it as an argument.  So, you can call DaemonNew() with different ports.

g.f. said:

Multiple TASKs got started and it is connected to the client successfully,
but data communication is available against only one port.

So you are saying that they have client code that tries to connect to the IP address and port (corresponding to the other daemons), but the client is unable to connect?  Or can it connect, but it just can't transfer data on this connection?

Also, DaemonNew() returns NULL on error.  Are they sure that DaemonNew is succeeding in this case?

g.f. said:

After first three divided packet are send, ACK against these packet was send to C6678 from CPU.

Then next two divided packet are send, and CPU send ACK again.

But suddenly C6678 retransmit the packet which was already been sent

I thought of another possibility here.  It's possible that the 6678 never received the ACK for packet 3/16.  For example, if there were not any available RX frame buffers in the 6678 Ethernet driver, the frame containing the ACK could have been dropped at layer 2.  From the TCP layer's perspective, this would look the same as the other host never having sent the ACK.

(note that in this scenario, you would still see the ACK in Wireshark, so it would seem that all is well on the wire, but if the frame is dropped in the stack, and hence not reaching the TCP layer, this is harder to detect and not immediately obvious).

I believe the 6678 Ethernet driver has some statistics available.  You should be able to see a count of dropped RX frames in those stats.

You can check the driver code for some global "stats" variables, otherwise Vincent or someone from the 6678 driver team can probably help you find out how to check this.

Once you know the stat to check, just let us know if there are any dropped RX frames after this issue occurs.

Steve

Hi Steven,

Thank you for the reply and I'm sorry for the dely, I was out of office.

*About DaemonNew()
I understood as that it is possible to start multiple TSK.
Is there any sample program?

>So you are saying that they have client code that tries to connect to the IP address
>and port (corresponding to the other daemons),

Yes.

>but the client is unable to connect?
>Or can it connect, but it just can't transfer data on this connection?

It can connect to the client but it can transfer data against only one port.

>Also, DaemonNew() returns NULL on error. Are they sure that DaemonNew is succeeding in this case?

I will ask to my customer.

*About retransmit
>I believe the 6678 Ethernet driver has some statistics available.
>You should be able to see a count of dropped RX frames in those stats.
>You can check the driver code for some global "stats" variables,
>otherwise Vincent or someone from the 6678 driver team can probably help you find out how to check this.

>Once you know the stat to check, just let us know if there are any dropped RX frames after this issue occurs.

I will share with customer about the possibility of ACK receive.
and ask to my customer if there are dropped RX frames.
So, please wait for the answer.

best regards,
g.f.

Hi Steve,

My customer and I can't find the stat which indicate the dropped Receive Frame.
I'm sorry, we need a help.
Which stat should we check for the dropped Receive Frame?

And my customer are asking me following questions.
Can you give me answers for these questions?

1.Is there any method to check how many free space are left for Receive Frame Buffer?
2.Is there any method to check where the Receive Frame Buffer are mapped to?
3.Is there any method to decide whether ACK is being received?

best regards,
g.f.

g.f.

I again asked the driver team to respond to this thread with any insights on getting the count of dropped RX frames.  They should respond here soon.

Can you try looking at the TCP statistics?  These stats will show info on things such as the number of ACKs received.

You can view the stats easily within CCS.  If you halt the program, then open an expressions window, you just need to type "tcps" to see them.

Please look at the stats and see what the count is for ACK packets.  Also, look for any other stats that have a high count.

I'll attach a screen shot to show this, too.

Steve

g.f.
I heard back from the driver team. There should be a CSL API you can use to get statistics:

"[You] can get the CPSW statistics using "CSL_CPSW_3GF_STATS" CSL API ... CSL_CPSW_3GF_getStats() is the CSL API. CSL_CPSW_3GF_STATS [...] is the register structure/layout to hold those Tx/Rx statistics counters"

Also,

"[you] can check the port status through EMAC registers 0x02090000 and 0x020C0000"

"[you] can put these register address in CCS memory window to get the status of the port."

I would suggest trying the look at those register addresses first. You might try using a memory window first, and also the CCS register view (look for EMAC registers or some such), as it might be easier to read in the register view (if the EMAC registers are available in that view).

Steve

Hi Steve,

Thank you for the reply and sorry for the delays.

I will ask my customer to check stats by looking "tcps".

Before sharing the information with my customer,
may I ask about which EMAC registers to check for port status ?

Is it the registers of "Statistics (STATS) submodule" which are written
in Keystone I EMAC User Guide(sprugv9d) page.158 "3.5.4 Statistics (STATS) submodule"?
It seems that "CSL_CPSW_3GF_STATS" correspond to these registers.

From the description of RXSOFOVERRUNS Register,
I guess this is the register for checking the dropped recieve frame.

By the way, is there any method for the following questions?
1.Is there any method to check how many free space are left for Receive Frame Buffer?
2.Is there any method to check where the Receive Frame Buffer are mapped to?

best regards,
g.f.

g.f.,

I received the attached GEL file from the driver team.  I wanted to pass it along to you today.

You can initiate the "emac_stats" function in CCS like below.

CCS scripts -> EVMC6678L EDC Functions -> emac_stats

I will respond to your other questions tomorrow.

Steve

6180.evmc6678l.gel

Hi Steve,

Thank you for the Gel file and giving me PBM bufffers and API informations.

I send the Gel file to my customer and told them to look at NDK API guide for
checking where the Frame buffer are mapped to and how much the free space are left.
(It seems that combination of PBM_get API can be use for checking Frame buffer.)

best regards,
g.f.