AM335x Ethernet over USB UDP throughput issue

Hi,

I will forward this to the USB experts.

Brain,

- Have you tried the USB Ethernet Adapter on PC to compare the UDP throughput?
- Please provide the adapter numeration log on AM335x console;
- Please try to to disable MUSB CPPI41 DMA in kernel config and re-run the test.

Hi Bin,

Q1 : I tried the USB Ethernet Adapter on PC and the UDP throughput is about 171Mbps
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.87.67 port 5001 connected with 192.168.87.70 port 4858
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-60.0 sec 1.20 GBytes 171 Mbits/sec 0.275 ms 35/874162 (0.004%)

Q2: Please refer to the usb related log as below.
......
musb-hdrc musb-hdrc.1.auto: MUSB HDRC host driver
musb-hdrc musb-hdrc.1.auto: new USB bus registered, assigned bus number 1
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: MUSB HDRC host driver
usb usb1: Manufacturer: Linux 3.14.26-gb9df364-dirty musb-hcd
usb usb1: SerialNumber: musb-hdrc.1.auto
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
musb-hdrc musb-hdrc.2.auto: MUSB HDRC host driver
musb-hdrc musb-hdrc.2.auto: new USB bus registered, assigned bus number 2
cfg80211: Calling CRDA to update world regulatory domain
usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: MUSB HDRC host driver
usb usb2: Manufacturer: Linux 3.14.26-gb9df364-dirty musb-hcd
usb usb2: SerialNumber: musb-hdrc.2.auto
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 1 port detected
cryptodev: driver 1.6 loaded.
EXT4-fs (mmcblk0p2): re-mounted. Opts: data=ordered
(stk) :ldisc installation timeout
(stk) :ldisc_install = 0uim:poll broke due to event 10(PRI:2/ERR:8)
usb 2-1: New USB device found, idVendor=05e3, idProduct=0610
usb 2-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
usb 2-1: Product: USB2.0 Hub
wlcore: wl18xx HW: 183x or 180x, PG 2.2 (ROM 0x11)
hub 2-1:1.0: USB hub found
hub 2-1:1.0: 4 ports detected
wlcore: loaded
wlcore: driver version: R8.5
wlcore: compilation time: Tue May 19 08:33:10 2015
usb 2-1.1: new high-speed USB device number 3 using musb-hdrc
usb 2-1.1: New USB device found, idVendor=0b95, idProduct=1790
usb 2-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 2-1.1: Product: AX88179
usb 2-1.1: Manufacturer: ASIX Elec. Corp.
usb 2-1.1: SerialNumber: 0000116B673930
ax88179_178a 2-1.1:1.0 eth1: register 'ax88179_178a' at usb-musb-hdrc.2.auto-1.1, ASIX AX88179 USB 3.0 Gigabit Ethernet, 00:11:6b:67:39:30
......

Q3: I disable the CONFIG_USB_TI_CPPI41_DMA in kernel but the test result is the same (< 10 Mbps)

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
CONFIG_USB_MUSB_HDRC=m
# CONFIG_USB_MUSB_HOST is not set
# CONFIG_USB_MUSB_GADGET is not set
CONFIG_USB_MUSB_DUAL_ROLE=y
# CONFIG_USB_MUSB_TUSB6010 is not set
CONFIG_USB_MUSB_OMAP2PLUS=m
# CONFIG_USB_MUSB_AM35X is not set
CONFIG_USB_MUSB_DSPS=m
# CONFIG_USB_MUSB_UX500 is not set
CONFIG_USB_MUSB_AM335X_CHILD=m
# CONFIG_USB_INVENTRA_DMA is not set
# CONFIG_USB_TI_CPPI41_DMA is not set
CONFIG_MUSB_PIO_ONLY=y
CONFIG_USB_DWC3=m
# CONFIG_USB_DWC3_HOST is not set
# CONFIG_USB_DWC3_GADGET is not set
CONFIG_USB_DWC3_DUAL_ROLE=y

Hi Bin,

Do you have any suggestion?

I also did TCP throughput test. The performance result for upload result is 102 Mbps and download is 86.9 Mbps.
The poor throughput only happened on UDP download.

Thanks.

Best Regards,
Brian

Brian,

Can you please try the latest TI 4.1 kernel to see if the issue still exists?

url = git://git.ti.com/ti-linux-kernel/ti-linux-kernel.git
branch: origin/ti-linux-4.1.y

I have an impression that the Asix chipset is not supported well in kernel. I had a Asix usb-ethernet dongle on my Linux PC which had some RX issues with Ubuntu 12.04.

Hi Bin,

Actually, we tested some kinds of USB Ethernet Adapter devices not only the Asix chipset device and their test result are the same (< 10Mbps) based on ti-processor-sdk-linux-am335x-evm-01.00.00.00 and ti-processor-sdk-linux-am335x-evm-01.00.00.03.

I also verify the latest tisdk version, ti-processor-sdk-linux-am335x-evm-02.00.00.00(kernel 4.1.6), and the UDP throughput looks good.
Upload: 165 Mbits/sec
Download: 120 Mbits/sec

However, we developed our product on ti-processor-sdk-linux-am335x-evm-01.00.00.00(kernel 3.14). It is a large changes from kernel 3.14 to kernel 4.1.
Would TI has a plan to fix UDP throughput issue on ti-processor-sdk-linux-am335x-evm-01.00.00.xx ?

Thanks.

Best Regards,
Brian

Brain,

If other USB-Ethernet dongle also has the same issue, likely the MUSB driver has a problem.

The SDK1.0.0.0 and 1.0.0.3 has almost the same kernel, only difference is other components in the SDK package.

The MUSB driver in SDK1.x has a known issue which is fixed by patch #8.1 in processors.wiki.ti.com/.../Sitara_Linux_MUSB_Issues. Can you please try if this patch fixes the UDP issue?

How long time would it take for you to migrate to the 4.1 kernel? It might be shorter than that for us to replicate the issue and find the root cause.

Hi Bin,

I tried the patch#8.1 but the throughput is the same poor result.

Thanks.

Best Regards,
Brian

Hi Bin,

We should take 1.5 months to migrate kernel from 3.14 to 4.1, including driver, application and testing.
I would like know how long can TI solve this issue and provide patch on kernel 3.14 ?

Thanks.

Best Regards,
Brian

Brian,

I am unable to commit how long it will take to solve the issue in 3.14, until we understood the root cause.

I got a Asix usb-ethernet USB3 dongle, which uses driver ax88179_178a:

Product: AX88179
Manufacturer: ASIX Elec. Corp.
SerialNumber: 000000000017<7

But at this moment I see this dongle can only get 20Mbps UDP RX in 4.1 kernel on both AM335x and AM437x, which have different USB controllers - MUSB vs. DWC3, so I would blame the Asix driver right now. Unfortunately the Asix driver has no any line of debug code, so I am unable to check what was happening.

What is the chipset of the dongles you have tested other than your Asix?

Hi Bin,

I tested a LTE module device, which uses driver qmi_wwan,

Product: HP lt4120 Gobi 4G Module
Manufacturer: Hewlett-Packard
SerialNumber: 0123456789ABCDEF

BTW, I tested UDP throughput with Asix USB Ethernet adapter device on Ubuntu 14.04 (kernel 3.19) and the test result is good for your reference.

Thanks.

Best Regards,
Brian

Hi Brian and Bin,

1. I did some test for this issue on our Lab server ( Ubuntu 12.04 Kernel 3.2.0 ), I think Asix adapter is supported well in Linux Kernel 3.2.0.
    My Asix dongle product name is AX88772B USB2.0 to Ethernet Adapter.
    Here is my test result on CP ( Windows 7 ) to server ( Ubuntu 12.04 ) :

wayne@wpi-ThinkCentre-M92p:~$ uname -r
3.2.0-68-generic
wayne@wpi-ThinkCentre-M92p:~$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.2.7.157 port 5001 connected with 10.2.7.46 port 59075
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 96.6 MBytes 80.9 Mbits/sec
^Cwayne@wpi-ThinkCentre-M92p:~$ iperf -s -u
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 224 KByte (default)
------------------------------------------------------------
[ 3] local 10.2.7.157 port 5001 connected with 10.2.7.46 port 53218
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.0 sec 114 MBytes 95.6 Mbits/sec 1.186 ms 3950/85445 (4.6%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
wayne@wpi-ThinkCentre-M92p:~$

It seems the Asix adapter and Kernel 3.2 would not cause the bad UDP performance.

2. I also did the test on BBB with Processor SDK 2.0 pre-built images, but the UDP performance is still not good ( < 1Mbit/s ).
    According to your discussions, this issue is " fixed " on Processor SDK 2.0.
    I could not confirm this statement. Is there any thing I miss?
    Here is my test result on PC ( Windows 7 ) to BBB (with Processor SDK 2.0 pre-built images) :

root@am335x-evm:~# uname -r
4.1.6-g52c4aa7
root@am335x-evm:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 169.254.197.11 port 5001 connected with 169.254.197.24 port 57904
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 96.8 MBytes 81.2 Mbits/sec
^Croot@am335x-evm:~# iperf -s -u
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 160 KByte (default)
------------------------------------------------------------
[ 3] local 169.254.197.11 port 5001 connected with 169.254.197.24 port 64406
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-12.3 sec 149 KBytes 99.8 Kbits/sec 589.628 ms 20649/20753 (99%)

Do you have any ideal with this result?
And can you provide the test log of the Processor SDK 2.0?

Best Regards,
Wayne Kuo

Hi Wayne,

You may refer to my test log as below:

am335x-evm:~# uname -r
4.1.6-g52c4aa7
am335x-evm:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.87.68 port 5001 connected with 192.168.87.70 port 1314
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 151 MBytes 126 Mbits/sec
^Cam335x-evm:~# iperf -s -u
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 160 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.87.68 port 5001 connected with 192.168.87.70 port 1315
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.0 sec 153 MBytes 128 Mbits/sec 0.446 ms 0/109177 (0%)

Thanks.

Best Regards,
Brian

Wayne,

Wayne Kuo said:

    According to your discussions, this issue is " fixed " on Processor SDK 2.0.
    I could not confirm this statement. Is there any thing I miss?

I have the same test result as yours. Only Brain claimed v4.1 works with his dongle.

I also tested v3.12 (SDK7.0) has no such issue with my Asix dongle, but figuring out which patch causes the regression from 3.12 to 3.14 will take me some time as I am currently working on another task. I will update once I have progress on this issue.

Hi Wayne,

I post my USB Ethernet adapter information for your reference.

usb 2-1.1: New USB device found, idVendor=0b95, idProduct=1790
usb 2-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 2-1.1: Product: AX88179
usb 2-1.1: Manufacturer: ASIX Elec. Corp.
usb 2-1.1: SerialNumber: 0000116B673930
ax88179_178a 2-1.1:1.0 eth1: register 'ax88179_178a' at usb-musb-hdrc.2.auto-1.1, ASIX AX88179 USB 3.0 Gigabit Ethernet, 00:11:6b:67:39:30

Thanks.

Best Regards,
Brian

Hi Bin,

Thanks for your information, my test on SDK 7.0 is also fine.
Please see the below information for test log.
I'll try to find what's different between Kernel 3.12 & 3.14.
If you have any update, please let us know, thank you.

Test result on BBB with SDK 7.0:

root@am335x-evm:~# uname -r
3.12.10-ti2013.12.01
root@am335x-evm:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 169.254.197.11 port 5001 connected with 169.254.197.24 port 49460
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 60.0 MBytes 50.2 Mbits/sec
^Croot@am335x-evm:~# iperf -s -u
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 160 KByte (default)
------------------------------------------------------------
[ 3] local 169.254.197.11 port 5001 connected with 169.254.197.24 port 51697
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.0 sec 62.8 MBytes 52.6 Mbits/sec 0.331 ms 203/44976 (0.45%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order

Best Regards,
Wayne Kuo

Brain & Wayne,

Please try the following patch on 3.14. It fixes the issue for my dongle.

commit b8ef95dd1dcc03dd13d6fda0736b2ac6ad3ceb3a
Author: Bin Liu <b-liu@ti.com>
Date:   Fri Oct 23 12:14:52 2015 -0500

    Revert "usb: musb: musb_host: Enable HCD_BH flag to handle urb return in bottom half"

    This reverts commit dd97dc60232a7c5019c968c7d5a27d7a2b11761e.

diff --git a/drivers/usb/musb/musb_host.c b/drivers/usb/musb/musb_host.c
index 409ddd9..7d6022f 100644
--- a/drivers/usb/musb/musb_host.c
+++ b/drivers/usb/musb/musb_host.c
@@ -2613,7 +2613,7 @@ static const struct hc_driver musb_hc_driver = {
        .description            = "musb-hcd",
        .product_desc           = "MUSB HDRC host driver",
        .hcd_priv_size          = sizeof(struct musb *),
-       .flags                  = HCD_USB2 | HCD_MEMORY | HCD_BH,
+       .flags                  = HCD_USB2 | HCD_MEMORY,

        /* not using irq handler or reset hooks from usbcore, since
         * those must be shared with peripheral code for OTG configs

Hi Bin,

Thank you for your replay.
This patch is not working good for me at the beginning, but after I turn on the "MUSB DMA MOD" in menuconfig it works good.
Please see the below information for the test result.

However, this patch dose not help Brian.
I'll sync with Brian to see why the patch is not useful for him.
Do you have any other idea for this situation?

Thanks for your Help again!

Best Regard,
Wayne Kuo

Test result for inserted your patch:

root@am335x-evm:~# uname -r
3.14.26-gb9df364-dirty
root@am335x-evm:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 169.254.197.11 port 5001 connected with 169.254.197.24 port 58661
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 100 MBytes 84.1 Mbits/sec
^Croot@am335x-evm:~# iperf -s -u
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 160 KByte (default)
------------------------------------------------------------
[ 3] local 169.254.197.11 port 5001 connected with 169.254.197.24 port 64558
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.0 sec 114 MBytes 95.6 Mbits/sec 0.276 ms 14/81363 (0.017%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order

Hi Bin,

I used "singlecore-omap2plus_defconfig" on tisdk 1.0.0.0 to apply this patch but it didn't fix the issue for me.

We might have different kernel components selected.

Could you provide your .config file for our reference ?

Thanks.

Best Regards,
Brian

Brain,

my .config is attached below.

_config-3.14.txt

Hi Bin,

After compared with your config file and mine, there are a lot of differences.

I checked the defconfig file on SDK 1.0.0.0 and found that you used "omap2plus_defconfig" instead of "singlecore-omap2plus_defconfig".
As I know, Wayne used "omap2plus_defconfig" as well.

That's why the patch couldn't fix the issue on my test environment.

Do you have any idea which kernel components would cause poor USB UDP throughput ?

Thanks.

Best Regards,
Brian

Brain,

For my dialy work I use the kernel tree from ti.com so singlecore-omap2plus_defconfig is unavaliable. Can you please use omap2plus_defconfig and enable/disable kernel configs according to you project?

Hi Bin,

OK. I see.

I just read some information from the link as below, it is suggested to use singlecore-omap2plus_defconfig on new TISDK released.
processors.wiki.ti.com/index.php

Thanks.

Best Regards,
Brian

Brian,

Yes, singlecore-omap2plus_defconfig is recommended, it is provided in the SDK package for convenience for customers. But please understand that singlecore-omap2plus_defconfig is derived from omap2plus_defconfig with some fine tuning for AM335x. For experienced customers, omap2plus_defconfig can be used too, then revised according to specific use cases.