• State Answer Suggested
  • Locked Locked
  • Replies 11 replies
  • Answers 2 answers
  • Subscribers 100 subscribers
  • Views 2864 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM335X 10MB/s ethernet issues

Alistair Higgins
Prodigy 110 points

Hi all,

 

We are using Beaglebone blacks as a starter for an AM335X-based design, I am having issues with 10MB/s Ethernet packets dropping out on Beaglebone black dev kits when sending traffic in both directions.

Is there a configuration option or kernel patch(?) I am missing?

I am using the meta-ti layer with the linux-ti-staging kernel v3.14 (more details near the end of the post).

  

To get the issue I am seeing to occur:

 -Set both ports on the PC and beaglebone to advertise only 10MB/s FD with ethtool:

ethtool -s eth<X> speed 10 duplex full autoneg on

 - running iperf server on the Beaglebone:

 iperf -s -u -i 1

 - running iperf client on my PC:

iperf -c 192.168.7.2 -b 64000 -i 1 -t 600 -d

 

The issue I am seeing is (as reported on the beaglebone via the debug serial port ttyO0)
where no packets were received on the beaglebone from 220 seconds until 247 seconds:

root@beaglebone:~# iperf -s -u -i 1
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.7.2 port 5001 connected with 192.168.7.10 port 38424
------------------------------------------------------------
Client connecting to 192.168.7.10, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  5] local 192.168.7.2 port 52644 connected with 192.168.7.10 port 5001
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0- 1.0 sec  7.18 KBytes  58.8 Kbits/sec   0.009 ms    0/    5 (0%)
<<<snip>>>
[  3] 215.0-216.0 sec  7.18 KBytes  58.8 Kbits/sec   0.031 ms    0/    5 (0%)
[  5] 215.0-216.0 sec  7.18 KBytes  58.8 Kbits/sec
[  3] 216.0-217.0 sec  7.18 KBytes  58.8 Kbits/sec   0.027 ms    0/    5 (0%)
[  5] 216.0-217.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 217.0-218.0 sec  8.61 KBytes  70.6 Kbits/sec
[  3] 217.0-218.0 sec  8.61 KBytes  70.6 Kbits/sec   0.038 ms    0/    6 (0%)
[  5] 218.0-219.0 sec  7.18 KBytes  58.8 Kbits/sec
[  3] 218.0-219.0 sec  5.74 KBytes  47.0 Kbits/sec   0.075 ms    0/    4 (0%)
[  5] 219.0-220.0 sec  8.61 KBytes  70.6 Kbits/sec
[  5] 220.0-221.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 221.0-222.0 sec  8.61 KBytes  70.6 Kbits/sec
[  5] 222.0-223.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 223.0-224.0 sec  8.61 KBytes  70.6 Kbits/sec
[  5] 224.0-225.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 225.0-226.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 226.0-227.0 sec  8.61 KBytes  70.6 Kbits/sec
[  5] 227.0-228.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 228.0-229.0 sec  8.61 KBytes  70.6 Kbits/sec
[  5] 229.0-230.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 230.0-231.0 sec  8.61 KBytes  70.6 Kbits/sec
[  5] 231.0-232.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 232.0-233.0 sec  8.61 KBytes  70.6 Kbits/sec
[  5] 233.0-234.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 234.0-235.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 235.0-236.0 sec  8.61 KBytes  70.6 Kbits/sec
[  5] 236.0-237.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 237.0-238.0 sec  8.61 KBytes  70.6 Kbits/sec
[  5] 238.0-239.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 239.0-240.0 sec  8.61 KBytes  70.6 Kbits/sec
[  5] 240.0-241.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 241.0-242.0 sec  8.61 KBytes  70.6 Kbits/sec
[  5] 242.0-243.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 243.0-244.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 244.0-245.0 sec  8.61 KBytes  70.6 Kbits/sec
[  5] 245.0-246.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 246.0-247.0 sec  8.61 KBytes  70.6 Kbits/sec
[  3] 219.0-220.0 sec  1.44 KBytes  11.8 Kbits/sec   0.079 ms    5/    6 (83%)
[  3] 220.0-221.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 221.0-222.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 222.0-223.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 223.0-224.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 224.0-225.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 225.0-226.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 226.0-227.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 227.0-228.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 228.0-229.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 229.0-230.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 230.0-231.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 231.0-232.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 232.0-233.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 233.0-234.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 234.0-235.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 235.0-236.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 236.0-237.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 237.0-238.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 238.0-239.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 239.0-240.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 240.0-241.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 241.0-242.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 242.0-243.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 243.0-244.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 244.0-245.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 245.0-246.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  3] 246.0-247.0 sec  0.00 Bytes  0.00 bits/sec   0.079 ms    0/    0 (nan%)
[  5] 247.0-248.0 sec  7.18 KBytes  58.8 Kbits/sec
[  3] 247.0-248.0 sec  1.44 KBytes  11.8 Kbits/sec   0.150 ms  151/  152 (99%)
[  5] 248.0-249.0 sec  8.61 KBytes  70.6 Kbits/sec
[  3] 248.0-249.0 sec  1.44 KBytes  11.8 Kbits/sec   0.144 ms    4/    5 (80%)
[  5] 249.0-250.0 sec  7.18 KBytes  58.8 Kbits/sec
[  3] 249.0-250.0 sec  5.74 KBytes  47.0 Kbits/sec   0.121 ms    3/    7 (43%)
[  3] 250.0-251.0 sec  4.31 KBytes  35.3 Kbits/sec   0.102 ms    2/    5 (40%)
[  5] 250.0-251.0 sec  7.18 KBytes  58.8 Kbits/sec
[  3] 251.0-252.0 sec  8.61 KBytes  70.6 Kbits/sec   0.074 ms    0/    6 (0%)
[  5] 251.0-252.0 sec  8.61 KBytes  70.6 Kbits/sec
[  3] 252.0-253.0 sec  7.18 KBytes  58.8 Kbits/sec   0.064 ms    0/    5 (0%)
[  5] 252.0-253.0 sec  7.18 KBytes  58.8 Kbits/sec
[  5] 253.0-254.0 sec  8.61 KBytes  70.6 Kbits/sec
[  3] 253.0-254.0 sec  8.61 KBytes  70.6 Kbits/sec   0.053 ms    0/    6 (0%)
<<<snip>>>
[  5]  0.0-600.3 sec  4.58 MBytes  64.0 Kbits/sec
[  5] Sent 3267 datagrams
[  5] Server Report:
[  5]  0.0-600.3 sec  4.58 MBytes  64.0 Kbits/sec   0.051 ms    0/ 3267 (0%)
[  3]  0.0-600.3 sec  4.35 MBytes  60.7 Kbits/sec   0.061 ms  166/ 3267 (5.1%)

Full logs from both my PC and the Beaglebone, and Wireshark in attached files.

There are no packets missed by my PC, but I have to run the test is bi-directional mode ('-d' option) to see the issue.

I have tried different data rates both faster and slower but the issue persists, the issue also occurs with TCP based tests (omitting the '-u' option from both command lines) where the test simply blocks until the connection is restored.

There are no errors (or any messages at all) reported via dmesg for the duration of the test.

 

I am building the OS via Yocto and have reduced the setup to a basic system with

 - poky,                                 git rev: "daisy:a4d8015687cf9ddd6ef563e29cf840698f81c099"
 - meta-openembedded, git rev: "daisy:662cf409c1175450699d498085f3c894e0fe81d0"
 - meta-ti,                            git rev: "daisy:df04699e3aae0ccfa88369117d6d20a9b46a6b26"

I have attached the only files I have modified for my setup:
 - /opt/r2/ethernet-issue/poky/build/conf/bblayers.conf
 - /opt/r2/ethernet-issue/poky/build/conf/local.conf
 - /opt/r2/ethernet-issue/poky/meta/recipes-core/images/core-image-minimal.bb.

files.zip
  • 0
    TI__Guru**** 393215 points

    Hi,

    Kernel v3.14 is not officially supported by TI. Latest supported kernel is v3.12. Have you tried the current EZSDK 7.0 release to see if the problem exists there?

  • 0 in reply to Biser Gatchev-XID
    Prodigy 110 points

    Hi Biser,

    Thanks for the quick response, I downloaded the latest EZSDK, v7.00.00.00, installed the defaul pre-built image and tried the exact same test in my first post, the results are identical:

    Randomly unable to receive packets for approximately 15-20 second periods, with wireshark confirming the packets are definitely being sent by my PC.

     

    Regards,

    Alistair

     

     

  • 0 in reply to Alistair Higgins
    TI__Mastermind 34805 points

    I tried various setups and Ethernet link rates. Only when the PC side is at 10 Mbps do I see the dropped packets on the server side (AM335x).  I also saw the silence period happening in almost the exact sime time, roughly 220 seconds.

    It didn't seem to matter what the link speed was on the AM335x.  I also tried using two TI devices and didn't see the problem.  At 100Mbps line rate I don’t see a problem with a PC running the test with 64K bit rate. I also tried ran two opposing clients and servers between the PC and BBB and did not see an issue with 10Mbps.

    The test I did with the two client/server pairs runnig simultaneously was trying to emulate a dual mode iperf test. I did see 1 dropped packet not the 20 plus seconds  of silence. Right now I think this might be an iperf issue with this bit rate which I'll admit does seem strange.

     

     

  • 0 in reply to Schuyler Patton
    Prodigy 110 points

    I am glad to see it is not just me that can reproduce the issue as reported :)

    To address your points:

    It didn't seem to matter what the link speed was on the AM335x.
    The link speed should only need to be set on one end of the connection as auto-negotiation should take care of the other, I only set both ends to ensure a correctly negotiated connection.

    I also tried using two TI devices and didn't see the problem.
    I am trying betwen multiple Beaglebone's now (the only ti devices I have at hand currently) - I will report back with what I find.

    At 100Mbps line rate I don’t see a problem with a PC running the test with 64K bit rate.
    At 100MB/sec I see no problems either, unfortunately for our application we cannot run at 100MB/s, so all communications will be limited to 10MB/s.

    I also tried ran two opposing clients and servers between the PC and BBB and did not see an issue with 10Mbps.
    With some experimenting I can still reproduce the issue here, although it is MUCH harder, I have a snippet of the log below.
    N    ote that I increased the test time out to 1200 seconds (although I don't believe test length is a contributing factor, it merely increases the liklihood of observing the issue).
    To reproduce the issue I had to start the client applications as near to simultaneously as possible and even then I can still only reproduce the issue in approximately 1/5 tests - it seems very sensitive to timing.
    The only potential difference here is that I started the iperf server on the Beaglebone from an SSH session - the client was started as usual from the debug serial port.
    This test was run on the pre-built EZSDK image as before.

     

    [  4] local 192.168.7.4 port 5001 connected with 192.168.7.10 port 33911
<<snip>>
[  4] 1067.0-1068.0 sec  8.61 KBytes  70.6 Kbits/sec   0.018 ms    0/    6 (0%)
[  4] 1068.0-1069.0 sec  7.18 KBytes  58.8 Kbits/sec   0.017 ms    0/    5 (0%)
[  4] 1069.0-1070.0 sec  7.18 KBytes  58.8 Kbits/sec   0.016 ms    0/    5 (0%)
[  4] 1070.0-1071.0 sec  5.74 KBytes  47.0 Kbits/sec   0.020 ms    2/    6 (33%)
[  4] 1071.0-1072.0 sec  4.31 KBytes  35.3 Kbits/sec   0.023 ms    0/    3 (0%)
[  4] 1072.0-1073.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1073.0-1074.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1074.0-1075.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1075.0-1076.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1076.0-1077.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1077.0-1078.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1078.0-1079.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1079.0-1080.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1080.0-1081.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1081.0-1082.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1082.0-1083.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1083.0-1084.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1084.0-1085.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1085.0-1086.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1086.0-1087.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1087.0-1088.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1088.0-1089.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1089.0-1090.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1090.0-1091.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1091.0-1092.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1092.0-1093.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1093.0-1094.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1094.0-1095.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1095.0-1096.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1096.0-1097.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1097.0-1098.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1098.0-1099.0 sec  0.00 Bytes  0.00 bits/sec   0.023 ms    0/    0 (nan%)
[  4] 1099.0-1100.0 sec  2.87 KBytes  23.5 Kbits/sec   0.091 ms  153/  155 (99%)
[  4] 1100.0-1101.0 sec  1.44 KBytes  11.8 Kbits/sec   0.086 ms    3/    4 (75%)
[  4] 1101.0-1102.0 sec  5.74 KBytes  47.0 Kbits/sec   0.075 ms    3/    7 (43%)
[  4] 1102.0-1103.0 sec  7.18 KBytes  58.8 Kbits/sec   0.064 ms    0/    5 (0%)
[  4] 1103.0-1104.0 sec  7.18 KBytes  58.8 Kbits/sec   0.051 ms    0/    5 (0%)
[  4] 1104.0-1105.0 sec  8.61 KBytes  70.6 Kbits/sec   0.039 ms    0/    6 (0%)
<<snip>>
[  4]  0.0-1200.3 sec  8.93 MBytes  62.4 Kbits/sec   0.020 ms  161/ 6532 (2.5%)
  • 0 in reply to Alistair Higgins
    Prodigy 110 points

    And to add, I cannot reproduce the issue between 2 Beaglebone blacks either, only when connected to a PC.

  • 0 in reply to Alistair Higgins
    TI__Mastermind 34805 points

    I tried the same test using two linux PC and got even worse results at 10Mbps. I will try the same test again perhaps using a different router tomorrow.

  • 0 in reply to Schuyler Patton
    Prodigy 110 points

    Hi Shuyler,

    I am not seeing this behaviour at all between PCs, I saw at worst 2 packets lost during a 5 minute test - I tried sending between 2 linux-based hosts, and between linux- to Windows-based hosts, both primary and secondary/USB NICs.

  • 0 in reply to Schuyler Patton
    Prodigy 110 points

    Hi guys,

    Is there any update on this issue? Or are you waiting on more data from me?


    Regards,

    Alistair

  • 0 in reply to Alistair Higgins
    TI__Genius 15510 points

    Could the problem be that the mi1_col and mii1_crs pins are not muxed? In arch/arm/boot/dts/am335x-bone-common.dtsi add them to the pinctrl for cpsw_default and see what happens. Rebuild the .dtb file and copy to your card.

    Steve K.

  • 0 in reply to Steve Kipisz
    Prodigy 110 points

    Hi Steve,


    The mii1_col and mii1_csr pins were indeed not explicitly muxed, and adding them to the dts file as inputs made no difference (which would e expected as we are running in full-duplex where these signals go unused(?) ).

  • 0 in reply to Alistair Higgins
    Prodigy 110 points
    To tidy up loose ends in case others are having this issue, Schuyler suggested a workaround by commenting out the 2 lines:

    else if (phy->speed == 10)
    mac_control |= BIT(18); /* In Band mode */

    in _cpsw_adjust_link() in the drivers/net/ethernet/ti/cpsw.c file.