AM389X: g_ether.ko(Ethernet connection and port reset) problem

Dear TI Team,

We are waiting for your response.

Whether the above mentioned issues are known issues?

We are running out of time. Please reply back as soon as possible.

Thanks in advance.

Best Regards,

Supreet

Supreet

We have not faced such condition during our testing. We perform iperf test with variable buffer sizes and normal ping test with PC.  Can you try with g_ether.ko as built-in to kernel.

Regards

Ravi B

Hi Ravi,

We tested g_ether.ko as built-in to kernel but still the problem persists.

We opened 10 cmd prompt window and did fast ping to the board with data size 65500 bytes. The arguments given to fast ping utility is : Fping.exe 192.168.1.4 -s 65536 -t 1 -c.

After some time, there is no response to ping(request timed out).

When we remove the USB cable and connect it back, it displays the below message 

rndis_msg_parser: unknown RNDIS message 0x80000004 len 28
g_ether gadget: RNDIS command error -524, 0/32
g_ether gadget: high speed config #2: RNDIS

After this if we ping, we get the reply but connection gets lost after sometime.

Can you please let us know the reason for the above issues.

Also can you confirm that whatever kernel version we are using is proper one? If not, Can you share the proper linux kernel files.

Waiting for your prompt reply.

Thanks.

Best Regards,

Supreet

Dear Ti Team,

Any update on above issues.

Please respond.

Best Regards,

Supreet

Supreet

10 multiple thread with fast ping of 65500 bytes from windows may lead too much ethernet packet flow on usb, device may be busy in receiving the data not having time to sevice them may lead to timeout. We have verified with 3-4 windows open with fast ping of 65500 bytes and seen timeout for one or two request in b/w in during ping transfer and console at target seems to be busy. 

You can use iperf to measure RNDIS performance/througput.

Regards

Ravi B

Hi Ravi,

Thanks for your response.

We are not testing RNDIS Performance/throughtput now. Our problem is when we transfer bulk data from host to  board or vice versa, connection between host and board is disconnecting.

Our final application will be doing bulk data transfer continously(say overnight) from host to Board and vice versa using Socket API Calls. Socket server application will be running on board and client application on host PC. Now when we are doing bulk data transfer, we are seeing Ethernet connection being disconnected.

So to isolate whether our application is causing problem or whether its problem in Linux driver, we used Fast ping utility. Ethernet Connection is getting disconnected when we do continuous ping using Fast ping utility too . So it confirms that there is some issue in Linux kernel driver only.

We are using linux kernel from PSP=v2.6.37_ti816xpsp_04.00.02.14 release. Can you confirm whether we are using proper kernel/ driver files???

Please provide us the solution for the same and whether its known issue???

Waiting for your prompt reply.

Best Regards,

Supreet 

Supreet

What is data rate transferred (Mbps) send/receive over the usb by the client application?

Regards

Ravi B

Hi Ravi,

Data is transfered at the rate of 4.5 MB/s. Note that Disconnection  issue is not seen only by our application but also with Fast Ping utility.

Please let me know if you need any further information.

Best Regards,

Supreet 

Supreet

What is meant by Disconnect,  whether the usb RNDIS network interface is complely disappear from windows host. Can you confirm from you application.

Can you confirm, you observe the disconnect with single cmd window with fast ping of 64K?

Regards

Ravi B

Hi Ravi,

Thank you for your prompt reply.

Our Host pc is running windows XP SP2.

Disconnect means connection between host and board is getting lost. i.e if we ping from either host to board or from board to host using ping command, there is no reply (request timed out).

USB RNDIS network interface does not disappear from windows host( When we check the properties of RNDIS Gadget in Device Manager,  it says "This device is working properly."). But if  we give the "ipconfig" command in the windows command prompt, prompt hangs without showing any ipaddress(until we remove and connect back the USB cable again). We are able to see the ipaddress on board side by giving "ifconfig" command. Once the ping fails, if i remove the usb cable and connect back the same again, ip address is assigned on the host PC and ping works .We get the error message on the board side as below when we remove and connect back usb cable after ping fails. 

rndis_msg_parser: unknown RNDIS message 0x80000004 len 28
g_ether gadget: RNDIS command error -524, 0/32
g_ether gadget: high speed config #2: RNDIS

We Tested fast ping on only single command prompt window with size 64k for 1 and half hour now (ping count 4 lakh), connection between host and PC was not getting lost (i.e we were getting reply to fast ping continously). Anyways i will keep this case overnight today and will let you know if it fails .

Note :  In your older posts dated "May 28, 2012 5:09 AM", you had posted "We have verified with 3-4 windows open with fast ping of 65500 bytes and seen timeout for one or two request in b/w in during ping transfer" , We are seeing that connection between host and board is getting lost (ping fails) after sometime (say 5-10 minutes) even with fast ping on only 3 windows. 

Please let me know for any further clarifications.

Thank you.

Best Regards,

Supreet

Supreet

The request timeout occuring is due to no response from the device due to device is heavily loaded. Make sure the host client application traffic is within the maximum RNDIS throughput. Use iperf to measure the rndis throughput or refer to datasheet for more details.

Regards

Ravi B

Hi Ravi,

We configured g_ether gadget in PIO mode(Previously it was DMA mode). When we tested Fast Ping in PIO Mode, we are getting request timeout(Board is being pinged from host PC) after sometime say(5000 times) with only 1 Fast Ping Window .

Can you please let us know why the above behavior is happening in PIO mode with only 1 Fast ping window also. 

Also please let us know solution for that.

Waiting for your quick reply.

Best Regards,

Supreet

Dear TI Team,

Waiting for your Response.

Best Regards,

Supreet

Dear Ti Team,

We tested USB Ethernet gadget in both DMA and PIO mode for fast ping issue on TI's TMDXEVM8168 kit . Same behavior( Request timeout for ping) is seen on that board also.

So Can you please let us know the root cause for it and solution for the same???

Our customer is waiting for this solution from past 2 months. We are not able to move forward.

Request you to address these issues ASAP.

Thank you.

Best Regards,

Supreet

Supreet

I have responded for this question in earlier reply. If you are looking for performance measurement either user iperf or ethernet performance thru IXIA kind of setup. You cannot use Fping for this. Fping from multiple command window will flood too much traffic system, due to this timeout you are seeing.

What is your use-case requirement? check for data sheet about perf. numbers whether matching with your use-case.

Regards

Ravi B

Dear Ravi,

We are not looking for Ethernet performance now.  Our problem is when we do fast ping from 3 command prompt  window to the board( In this case ethernet gadget is in DMA mode), we are getting connection timeout  with single command prompt window also(closed 2 other command prompt windows). 

So we configured Ethernet gadget in PIO Mode . When we tested Fast Ping in PIO Mode, we are getting request timeout(Board is being pinged from host PC) after sometime say(5000 times) with only 1 Fast Ping Window .

Why we are getting connection timeout for ping with only 1 fast ping window when Ethernet gadget is in PIO mode?

We are not concerned about performance now. Our main concern is why ping is failing after sometime in case of both PIO and DMA mode.

Please help us to resolve these issues.

Thank you.

Best Regards,

Supreet

Supreet

I too observe timeout if running Fping from multiple windows, but  no issues seen with single window Fping. This could be due high traffic packet flow & would be dropped by network layer or could be issue at application, but root cause is not known. We dont have bandwidth to investigate this issue this at this point. To prove this you can check Fping from multiple windows to ethernet interface and see it is working reliably.

In previous mail you have confirmed with single window Fast ping is working. 

Did you see any issue from use-case perspective?

Regards

Ravi B

Hi Ravi,

There are 2 scenarios here as mentioned below.

Case 1 : Ethernet gadget compiled in DMA mode - In this Case , there is no issue with single window Fping ( We get response continuously). But if we do Fping from 2 to 3 windows, We get connection timeout( no response to ping).

Case 2 : Ethernet gadget compiled in PIO mode - In this Case , we are getting request timeout(Board is being pinged from host PC) after sometime say(5000 times) with only 1 Fast Ping Window .

So the final observation is , When the ethernet gadget is in DMA mode, We are getting connection timeout when Fast ping is done with 2 -3 Fping window and when the ethernet gadget is in PIO mode , we are getting connection timeout with only 1 Fast ping window.

Can you please let us know why we are getting connection timeout in PIO mode with 1 Fast ping Window only?

Waiting for your prompt reply.

Thank you.

Best Regards,

Supreet

Supreet

Firstly the performance with PIO mode is expected to be much less compared to moving data with DMA(a lot of CPU cycles are spent in copying data) - so any congestion is the TCP/IP stack(leading to dropping of packets) is expected to be more pronounced when you are using the PIO mode (as experienced by you)

We are still not cclear about the intent of using FPING with such a large packet size for your testing. In using a large packet size(64k), you are forcing multiple fragments(each of MTU size, 1514bytes) to be created for every data packet. If there is congestion(buffer starvation, CPU being saturated) in network stack, and even if a few fragments get dropped you will experience a timeout.  So this is the expected behavior if you are bombarding the interface with large packet sizes without flow-control/re-transmission.

 I assume in your end application you should be prepared to implement flow control or re-transmission to overcome this.

I would suggest we check on the following

• run fping (still continue with floogd ping) with packet size < MTU
• When the timeout happens, please check if communication over the USB interface is still active - close all fping sessions, check with a simple from the board to the PC (do not disconnect and re-connect). Also look for network statistics at both the ends

Unfortunately when we ran tests at our end, we have been able to reproduce the timeouts(as expected) but none of the error messages/conditions as you have reported

Dear Srirama,

Thank you for the reply.

Our final application will be doing bulk data transfer continously(say overnight) from host to Board and vice versa using Socket API Calls. Socket server application will be running on board and client application on host PC.We are creating 16 threads with 16 client socket connected to board server socket application. Each thread will send 128KB of data from host PC to board and vice versa via socket.So when we run our application, we are seeing Ethernet connection being disconnected. once the connection goes down, even if we close our application and run normal ping, we are not getting reply.

So to isolate whether our application is causing problem or whether its problem in Linux driver, we used Fast ping utility. Ethernet Connection is getting disconnected when we do continuous ping using Fast ping utility too . So it confirms that there is some issue in Linux kernel driver only.

I tested the method suggested by you and still facing the same issue. I did the following things.

• I opened 3 fping window and pinged to board with size 1500 bytes.
• When the timeout happened, i stopped all the fping sessions and pinged from board to PC.Still I get the timeout here. Communication over the USB interface is DEAD.

Same behavior is seen in case of PIO mode with only 1 fping window only. i.e connection is completely down.

Request you to help us to resolve these issues as soon as possible.

Thank you.

Best Regards,

Supreet

Hello, Supreet Shanbhag

I am not sure that I'll be able to help, I don't have AM389x device, but I have DM816x and you are saying that the problem is present there also.  I tried to reproduce your problem, but everything works fine on my side. I enabled the ethernet gadget (conveniently as built-in, DMA enabled), I checked that I have ping through the gadget usb0 interface, and then I intensively pinged the board (using the gadget) from 4 terminals with packet size ~65K (fragmentation takes place) for 25 minutes +. I successfully sent back and forth more then 4500 packets and I didn't get the board's networking failure.

So I am unable to observe well your problem, but:

1. Is your gadget configuration done exactly how it is described here at section MUSB controller in gadget mode:

http://processors.wiki.ti.com/index.php/Usbgeneralpage

2. What Linux kernel are you using and how did you obtain it? Is it an appropriete one and is it recent? The one that worked OK for me was the 2.6.37 obtained from the official EZSDK 5_04 distribution. Would it be easy for to try it whit this one?

Meanwhile I'll try to get your problem on my side.

BR

Hi Vladimir,

Sorry for the late response.

Were you able to reproduce the ping issue at your side?

We downloaded the official EZSDK 5_04 distribution and used the kernel-2.6.37 present in it.

We enabled the ethernet gadget (conveniently as built-in, DMA enabled). And the board used to test is TI'sTMDXEVM8168 kit

We are still getting the problem with ping. i.e if we do fast ping to board with size 65535 from 4 command prompt windows,after sometime connection is doing down permanently.

We have taken the snapshot of the cpu version and EVM kit version from the board we have here. Please refer the attached document containing snapshots of both cpu and evm version details.

Are you using same version of CPU and EVM kit at your end or higher version?

Please let us know how to proceed further.

Request you to help us to resolve this issues as soon as possible.

Hello, Supreet

I wasn't able to reproduce it; I tried few times for about 20 minutes each time. I will try again. Perhaps it is a software issue, doesn't seem related to the EVM version, but I can check it. As I asked you before:

Is your gadget configuration done exactly how it is described here:

http://processors.wiki.ti.com/index.php/Usbgeneralpage#MUSB_controller_in_gadget_mode

There is one difference on my side: I am using Ubuntu as a host. I am pinging like that:

ping 10.0.10.20 -s  65507

What is yours now, exactly?

Is there any particular reason to use fping instead of ping(Windows built-in)? There is only one target.

We will continue to investigate.

BR

Hi Vladimir,

Thank you for your prompt reply.

Yes we have configured the gadget as per the link mentioned by you.

We are using windows 7 as a host. we downloaded the Fast ping  utility(Fping.exe) from internet and running it as mentioned below:

Fping.exe 192.168.1.4 -s 65536 -t 1 -c

Can you please test using fast ping utility(Fping.exe) and check whether you are able to reproduce the issue?

Reasons for using Fping:

Our final application will be doing bulk data transfer continously(say overnight) from host to Board and vice versa using Socket API Calls. Socket server application will be running on board and client application on host PC.We are creating 16 threads with 16 client socket connected to board server socket application. Each thread will send 128KB of data from host PC to board and vice versa via socket.So when we run our application, we are seeing Ethernet connection being disconnected. once the connection goes down, even if we close our application and run normal ping, we are not getting reply.

So to isolate whether our application is causing problem or whether its problem in Linux driver, we used Fast ping utility. Ethernet Connection is getting disconnected when we do continuous ping using Fast ping utility too . So it confirms that there is some issue in Linux kernel driver only.

Please let me know if you need any further clarifications.

Waiting for your prompt reply.

Thank You.

Best Regards,

Supreet

Hello, Supreet

Is it your board's networking that crashes or it is the one of the host (the Windows machine), when you say the connection goes down?

Do you need to restart the host to start it working again?

If it is the board:

        Is it not working until your restart the board or the ping works again when you replug the USB A to A cable?

BR

Hi Vladimir,

Its the Board's networking that crashes. If i ping from host to board or board to host, we are getting timeout for ping.

Ping works again, once we replug the USB A to A cable.

Thanks for your prompt reply. Let me know for any further clarification.

Best Regards,

Supreet

Hello, Supreet

You want to accomplish:

sending traffic: ~2MB/s

receiving traffic: ~2MB/s

16 x 128KB/s in each direction. Is that right? I am asking in order to know how to ping it.

Thank you and sorry for the inconvenience.

BR

Dear Vladimir,

We are not concerned about the throughput now. Our main concern is ethernet connection should not be down permanently when fast ping is done from more than 2 windows.

For Example, i will open 3 command prompt windows and do Fast ping with size 65536 to the board from all the 3 prompts. After some time connection goes down and there will be no reply to ping in all the 3 command prompt window. Then we will close all the 3 Fast ping command prompt and do normal ping to the board. At this point , when we do normal ping it should work at least which in our case is not happening.

When we do fast ping from 3 windows, if there is some packet loss, its not a problem but Ethernet connection should not go down permanently.

Please let me know for any further clarifications.

Thank you.

Best Regards,

Supreet

Hello again, Supreet

Yes, I understand that you have that error with only 3 pings, I know it is not about the throughput now, I asked about your goal.

I am still not able to reproduce your issue. I wasn't able to see it, to understand the behaviour.  But when you are saying "Its the Board's networking that crashes. If i ping from host to board or board to host, we are getting timeout for ping", this is not a clear evidence that the problem is in the board, it could be very easy on Windows' side as well. Or it could be from both of them combined.

1. Try to ping it form another Windows 7 machine. There could be something weird  in your current one i.e. some unwanted registry values, that could affect the driver.

2. Try to ping it using the Windows' built-in ping:

ping -l 65000 -t

from 3 command prompt windows. I don't thing you need the fping tool for what are you testing. And don't use more than 65000.

3. Try to ping using Ubuntu machine:

ping 10.0.10.20 -s  65000

4. Is there any message/error printed in the board's shell or in host's prompt; what is it?

The result of these would give some tips what is going on. Sorry that I am not of much help. We'll see what we could do.

BR