cpsw dual emac mode freeze

Jaroslav Vozab

Hello,

we have a custom AM335x board and there is a problem with cpsw in dual emac mode.

When the board boots with both ethernet cables plugged in and nfs root is used the ethernet freeze after a while (10 seconds)

The problem never occured when the board boots only with eth0 cable plugged in and the eth1 cable is plugged in later.

The problem never occured when the board boots with both eth cables plugged in and nand flash root fs is used.

I tested it on AM335x Starter Kit too and the problem is the same.

We are using ti-sdk-am335x-evm-05.07.00.00 with 3.2.0 linux kernel

Is there any known solution for that?

Best regards

Jaroslav

over 11 years ago

0 Biser Gatchev-XID over 11 years ago

TI__Guru**** 393215 points

Hi Jaroslav,

Have you tested with the new SDK (v06.00)?

0 Jaroslav Vozab over 11 years ago in reply to Biser Gatchev-XID

Prodigy 70 points

Hi Biser,

Thank you for your reply. I have just finished testing SDK version 06.00 and the problem is still the same

Best regards

Jaroslav

0 Miroslav Kiradzhiyski XID over 11 years ago in reply to Jaroslav Vozab

TI__Mastermind 25235 points

Hi Jaroslav,

Can you provide a log of the ethernet freeze? Please attach it as a file.

P.S. Have you read the constraints of the cpsw dual emac mode? http://processors.wiki.ti.com/index.php/AM335x_CPSW_%28Ethernet%29_Driver%27s_Guide#Dual_Standalone_EMAC_mode

Best regards,
Miroslav

0 Jaroslav Vozab over 11 years ago in reply to Miroslav Kiradzhiyski XID

Prodigy 70 points

Hi Miroslav,

Thanks for your reply. The log is attached.

Yes I have read the constraints. The problem occurs even if eth1 is not configured in startup scripts.

Best regards

Jaroslav

Fullscreen 7558.log.txt Download

Starting kernel ...

Uncompressing Linux... done, booting the kernel.
[    0.000000] Linux version 3.2.0-gc604c51-dirty (vozab@vozab-desktop) gcc version 4.5.3 20110311 (prerelease) (GCC) ) #265 Mon Nov 18 11:46:44 CET 2013
[    0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c53c7d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] Machine: km_is3
[    0.000000] Memory policy: ECC disabled, Data cache writeback
[    0.000000] AM335X ES2.0 (neon )
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 65024
[    0.000000] Kernel command line: console=ttyO0,115200n8 nand0sz=128 nand1sz=128 root=/dev/nfs rw ip=dhcp nfsroot=192.168.1.26:/tftpboot/km_is3_fs
[    0.000000] PID hash table entries: 1024 (order: 0, 4096 bytes)
[    0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.000000] Memory: 256MB = 256MB total
[    0.000000] Memory: 254364k/254364k available, 7780k reserved, 0K highmem
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
[    0.000000]     vmalloc : 0xd0800000 - 0xff000000   ( 744 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xd0000000   ( 256 MB)
[    0.000000]     modules : 0xbf000000 - 0xc0000000   (  16 MB)
[    0.000000]       .text : 0xc0008000 - 0xc04ac000   (4752 kB)
[    0.000000]       .init : 0xc04ac000 - 0xc04e0000   ( 208 kB)
[    0.000000]       .data : 0xc04e0000 - 0xc0538ae8   ( 355 kB)
[    0.000000]        .bss : 0xc0538b0c - 0xc055c9d0   ( 144 kB)
[    0.000000] NR_IRQS:396
[    0.000000] IRQ: Found an INTC at 0xfa200000 (revision 5.0) with 128 interrupts
[    0.000000] Total of 128 interrupts on 1 active controller
[    0.000000] OMAP clockevent source: GPTIMER2 at 26000000 Hz
[    0.000000] omap_dm_timer_switch_src: Switching to HW default clocksource(sys_clkin_ck) for timer1, this may impact timekeeping in low power state
[    0.000000] OMAP clocksource: GPTIMER1 at 26000000 Hz
[    0.000000] sched_clock: 32 bits at 26MHz, resolution 38ns, wraps every 165191ms
[    0.000000] Console: colour dummy device 80x30
[    0.000207] Calibrating delay loop... 597.60 BogoMIPS (lpj=2988032)
[    0.056793] pid_max: default: 32768 minimum: 301
[    0.056937] Security Framework initialized
[    0.057053] Mount-cache hash table entries: 512
[    0.057466] CPU: Testing write buffer coherency: ok
[    0.078297] omap_hwmod: gfx: failed to hardreset
[    0.094501] omap_hwmod: pruss: failed to hardreset
[    0.095727] print_constraints: dummy:
[    0.096134] NET: Registered protocol family 16
[    0.098615] OMAP GPIO hardware version 0.1
[    0.101363] omap_mux_init: Add partition: #1: core, flags: 0
[    0.101729] The board is a AM335x km_is3.
[    0.101813]  omap_i2c.1: alias fck already exists
[    0.106630]  omap2_mcspi.1: alias fck already exists
[    0.106936]  omap2_mcspi.2: alias fck already exists
[    0.108070]  edma.0: alias fck already exists
[    0.108098]  edma.0: alias fck already exists
[    0.108122]  edma.0: alias fck already exists
[    0.129899] bio: create slab <bio-0> at 0
[    0.131796] SCSI subsystem initialized
[    0.134488] usbcore: registered new interface driver usbfs
[    0.134844] usbcore: registered new interface driver hub
[    0.135073] usbcore: registered new device driver usb
[    0.135237] musb-ti81xx musb-ti81xx: musb0, board_mode=0x13, plat_mode=0x3
[    0.135555] musb-ti81xx musb-ti81xx: musb1, board_mode=0x13, plat_mode=0x1
[    0.136678] omap_i2c omap_i2c.1: bus 1 rev2.4.0 at 100 kHz
[    0.138524] Switching to clocksource gp timer
[    0.156672] musb-hdrc: version 6.0, ?dma?, otg (peripheral+host)
[    0.156866] musb-hdrc musb-hdrc.0: dma type: pio
[    0.157221] MUSB0 controller's USBSS revision = 4ea20800
[    0.157249] musb0: Enabled SW babble control
[    0.158002] musb-hdrc musb-hdrc.0: USB OTG mode controller at d083c000 using PIO, IRQ 18
[    0.158185] musb-hdrc musb-hdrc.1: dma type: pio
[    0.158540] MUSB1 controller's USBSS revision = 4ea20800
[    0.158626] musb1: Enabled SW babble control
[    0.158823] musb-hdrc musb-hdrc.1: MUSB HDRC host driver
[    0.158902] musb-hdrc musb-hdrc.1: new USB bus registered, assigned bus number 1
[    0.159062] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    0.159082] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.159098] usb usb1: Product: MUSB HDRC host driver
[    0.159111] usb usb1: Manufacturer: Linux 3.2.0-gc604c51-dirty musb-hcd
[    0.159126] usb usb1: SerialNumber: musb-hdrc.1
[    0.160054] hub 1-0:1.0: USB hub found
[    0.160087] hub 1-0:1.0: 1 port detected
[    0.160734] musb-hdrc musb-hdrc.1: USB Host mode controller at d083e800 using PIO, IRQ 19
[    0.161208] NET: Registered protocol family 2
[    0.161410] IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.161745] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[    0.161935] TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
[    0.162048] TCP: Hash tables configured (established 8192 bind 8192)
[    0.162063] TCP reno registered
[    0.162077] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.162105] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.162310] NET: Registered protocol family 1
[    0.162588] RPC: Registered named UNIX socket transport module.
[    0.162604] RPC: Registered udp transport module.
[    0.162615] RPC: Registered tcp transport module.
[    0.162626] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.162880] NetWinder Float Emulator V0.97 (double precision)
[    0.163113] omap-gpmc omap-gpmc: GPMC revision 6.0
[    0.163136] Registering NAND on CS0 of size 0x8000000
[    0.163695] Registering NAND on CS1 of size 0x8000000
[    0.176201] VFS: Disk quotas dquot_6.5.2
[    0.176276] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    0.176714] JFFS2 version 2.2.01-2006 Red Hat, Inc.
[    0.177013] msgmni has been set to 496
[    0.180610] alg: No test for stdrng (krng)
[    0.181376] io scheduler noop registered
[    0.181393] io scheduler deadline registered
[    0.181471] io scheduler cfq registered (default)
[    0.182638] omap_uart.0: ttyO0 at MMIO 0x44e09000 (irq = 72) is a OMAP UART0
[    0.776664] console [ttyO0] enabled
[    0.781082] omap_uart.1: ttyO1 at MMIO 0x48022000 (irq = 73) is a OMAP UART1
[    0.788987] omap_uart.3: ttyO3 at MMIO 0x481a6000 (irq = 44) is a OMAP UART3
[    0.796803] omap_uart.4: ttyO4 at MMIO 0x481a8000 (irq = 45) is a OMAP UART4
[    0.804644] omap_uart.5: ttyO5 at MMIO 0x481aa000 (irq = 46) is a OMAP UART5
[    0.813029] omap4_rng omap4_rng: OMAP4 Random Number Generator ver. 2.00
[    0.830943] brd: module loaded
[    0.839913] loop: module loaded
[    0.844894] mtdoops: mtd device (mtddev=name/number) must be supplied
[    0.852479] mb85rs spi2.0: mb85rs1mt (128 Kbytes)
[    0.857602] Creating 1 MTD partitions on "mb85rs":
[    0.862650] 0x000000000000-0x000000020000 : "Fram Data"
[    0.869915] omap2-nand driver initializing
[    0.874561] ONFI flash detected
[    0.877970] ONFI param page 0 valid
[    0.881653] NAND device: Manufacturer ID: 0x2c, Chip ID: 0xf1 (Micron MT29F1G08ABADAH4)
[    0.890245] Creating 8 MTD partitions on "omap2-nand.0":
[    0.895796] 0x000000000000-0x000000020000 : "SPL"
[    0.902172] 0x000000020000-0x000000040000 : "SPL.backup1"
[    0.909253] 0x000000040000-0x000000060000 : "SPL.backup2"
[    0.916317] 0x000000060000-0x000000080000 : "SPL.backup3"
[    0.923440] 0x000000080000-0x000000260000 : "U-Boot"
[    0.930844] 0x000000260000-0x000000280000 : "U-Boot Env"
[    0.937879] 0x000000280000-0x000000780000 : "Kernel"
[    0.946627] 0x000000780000-0x000008000000 : "Root FS"
[    1.003914] ONFI flash detected
[    1.007356] ONFI param page 0 valid
[    1.011071] NAND device: Manufacturer ID: 0x2c, Chip ID: 0xf1 (Micron MT29F1G08ABADAH4)
[    1.019739] Creating 1 MTD partitions on "omap2-nand.1":
[    1.025289] 0x000000000000-0x000008000000 : "File System"
[    1.087089] CAN device driver interface
[    1.091143] CAN bus driver for Bosch D_CAN controller 1.0
[    1.138611] davinci_mdio davinci_mdio.0: davinci mdio revision 1.6
[    1.145064] davinci_mdio davinci_mdio.0: detected phy mask fffffff9
[    1.152829] davinci_mdio.0: probed
[    1.156388] davinci_mdio davinci_mdio.0: phy[1]: device 0:01, driver unknown
[    1.163774] davinci_mdio davinci_mdio.0: phy[2]: device 0:02, driver unknown
[    1.171321] PPP generic driver version 2.4.2
[    1.176405] usbcore: registered new interface driver cdc_ether
[    1.182695] usbcore: registered new interface driver cdc_eem
[    1.188782] usbcore: registered new interface driver dm9601
[    1.194643] cdc_ncm: 04-Aug-2011
[    1.198180] usbcore: registered new interface driver cdc_ncm
[    1.204109] Initializing USB Mass Storage driver...
[    1.209460] usbcore: registered new interface driver usb-storage
[    1.215725] USB Mass Storage support registered.
[    1.220833] mousedev: PS/2 mouse device common for all mice
[    1.227811] rtc-isl1208 1-006f: chip found, driver version 0.3
[    1.239424] rtc-isl1208 1-006f: rtc core: registered rtc-isl1208 as rtc0
[    1.246898] rtc-isl1208 1-006f: rtc power failure detected, please set clock.
[    1.254594] i2c /dev entries driver
[    1.259573] OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[    1.266115] _regulator_get: deviceless supply vdd_mpu not found, using dummy regulator
[    1.274911] cpuidle: using governor ladder
[    1.279780] cpuidle: using governor menu
[    1.283881] omap4_aes_mod_init: loading AM33X AES driver
[    1.289567] omap4-aes omap4-aes: AM33X AES hw accel rev: 3.02
[    1.296103] omap4_aes_probe: probe() done
[    1.300565] omap4_sham_mod_init: loading AM33X SHA/MD5 driver
[    1.306697] omap4-sham omap4-sham: AM33X SHA/MD5 hw accel rev: 4.03
[    1.319012] omap4_sham_probe: probe() done
[    1.325983] usbcore: registered new interface driver usbhid
[    1.331838] usbhid: USB HID core driver
[    1.335841] oprofile: hardware counters not available
[    1.341121] oprofile: using timer interrupt.
[    1.345597] nf_conntrack version 0.5.0 (3974 buckets, 15896 max)
[    1.352399] ip_tables: (C) 2000-2006 Netfilter Core Team
[    1.358066] TCP cubic registered
[    1.361468] NET: Registered protocol family 17
[    1.366120] can: controller area network core (rev 20090105 abi 8)
[    1.372682] NET: Registered protocol family 29
[    1.377316] can: raw protocol (rev 20090105)
[    1.381785] can: broadcast manager protocol (rev 20090105 t)
[    1.387730] Registering the dns_resolver key type
[    1.392737] VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 3
[    1.400752] ThumbEE CPU extension supported.
[    1.405271] mux: Failed to setup hwmod io irq -22
[    1.410903] Power Management for AM33XX family
[    1.415758] Trying to load am335x-pm-firmware.bin (60 secs timeout)
[    1.422463] Copied the M3 firmware to UMEM
[    1.426831] Cortex M3 Firmware Version = 0x18
[    1.431966] sr_init: platform driver register failed
[    1.440963] clock: disabling unused clocks to save power
[    1.460877] Detected MACID=1c:ba:8c:a5:d3:42
[    1.466448] cpsw: Detected MACID = 1c:ba:8c:a5:d3:44
[    1.474097] rtc-isl1208 1-006f: hctosys: invalid date/time
[    1.484386] net eth0: CPSW phy found : id is : 0x2000a211
[    1.497967] net eth1: CPSW phy found : id is : 0x2000a211
[    3.479302] PHY: 0:02 - Link is Up - 100/Full
[    3.489239] PHY: 0:01 - Link is Up - 100/Full
[    3.498579] Sending DHCP requests ., OK
[    3.519047] IP-Config: Got DHCP answer from 0.0.0.0, my address is 192.168.1.161
[    3.529389] IP-Config: Complete:
[    3.532764]      device=eth0, addr=192.168.1.161, mask=255.255.254.0, gw=192.168.1.254,
[    3.540905]      host=192.168.1.161, domain=, nis-domain=(none),
[    3.547173]      bootserver=0.0.0.0, rootserver=192.168.1.26, rootpath=
[    3.583417] VFS: Mounted root (nfs filesystem) on device 0:14.
[    3.589926] Freeing init memory: 208K
INIT: version 2.88 booting
Starting udev
Starting Bootlog daemon: bootlogd: cannot allocate pseudo tty: No such file or directory
bootlogd.
NOT configuring network interfaces: / is an NFS mount
Tue Sep 24 04:05:00 UTC 2013
INIT: Entering runlevel: 5
Starting system message bus: dbus.
Starting telnet daemon.
Starting nginx.
Starting syslogd/klogd: done
Stopping Bootlog daemon: bootlogd.

 _____                    _____           _         _
|  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
|     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
|__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
              |___|                    |___|

Arago Project http://arago-project.org am335x-evm ttyO0

Arago 2013.02 am335x-evm ttyO0

am335x-evm login: root
root@am335x-evm:~#
root@am335x-evm:~# [   49.958598] nfs: server 192.168.1.26 not responding, still trying
[   65.378632] nfs: server 192.168.1.26 not responding, still trying

0 Schuyler Patton over 11 years ago in reply to Jaroslav Vozab

TI__Mastermind 40090 points

Are the two ethernet ports connecting to the same subnet?

0 Jaroslav Vozab over 11 years ago in reply to Schuyler Patton

Prodigy 70 points

No, each port is connected to a different subnet.

eth0 is connected to 192.168.1.xx and eth1 to 192.168.9.xx

ifconfig output follows:

eth0      Link encap:Ethernet HWaddr 1C:BA:8C:A5:D3:42
          inet addr:192.168.1.161 Bcast:192.168.1.255 Mask:255.255.254.0
          UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
          RX packets:8307 errors:0 dropped:8 overruns:0 frame:0
          TX packets:5382 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6936846 (6.6 MiB) TX bytes:840140 (820.4 KiB)

eth1      Link encap:Ethernet HWaddr 1C:BA:8C:A5:D3:44
          inet addr:192.168.9.152 Bcast:192.168.9.255 Mask:255.255.255.0
          UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
          RX packets:35 errors:0 dropped:1 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5216 (5.0 ) TX bytes:1160 (1.1 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

0 Jaroslav Vozab over 11 years ago in reply to Jaroslav Vozab

Prodigy 70 points

I have discovered one more feature of the problem.

It doesn't happen if the command "ifup eth1" is called during the 10 seconds before freeze

Jaroslav

0 Jaroslav Vozab over 11 years ago in reply to Jaroslav Vozab

Prodigy 70 points

Hello,

the reason of the freeze is that number of free cpdma rx descriptors decrease to 0. Does anyone know why?

Best regards

Jaroslav

[   22.586373] cpsw cpsw: channel 32 (rx 0) state active
[   22.591465] cpsw cpsw:       hdp: 4a102060
[   22.595469] cpsw cpsw:       cp: 4a102040
[   22.599195] cpsw cpsw:       rxfree: 1
[   22.602651] cpsw cpsw:       stats head_enqueue: 1
[   22.607196] cpsw cpsw:       stats tail_enqueue: 748
[   22.611920] cpsw cpsw:       stats pad_enqueue: 0
[   22.616372] cpsw cpsw:       stats misqueued: 0
[   22.620644] cpsw cpsw:       stats desc_alloc_fail: 0
[   22.625460] cpsw cpsw:       stats pad_alloc_fail: 0
[   22.630184] cpsw cpsw:       stats runt_receive_buff: 0
[   22.635181] cpsw cpsw:       stats runt_transmit_buff: 0
[   22.640268] cpsw cpsw:       stats empty_dequeue: 0
[   22.644903] cpsw cpsw:       stats busy_dequeue: 246
[   22.649627] cpsw cpsw:       stats good_dequeue: 745
[   22.654352] cpsw cpsw:       stats requeue: 0
[   22.658441] cpsw cpsw:       stats teardown_dequeue: 0
[   22.663545] cpsw cpsw: channel 32 (rx 0) state active
[   22.668638] cpsw cpsw:       hdp: 0
[   22.672007] cpsw cpsw:       cp: 4a102040
[   22.675735] cpsw cpsw:       rxfree: 0
[   22.679192] cpsw cpsw:       stats head_enqueue: 1
[   22.683736] cpsw cpsw:       stats tail_enqueue: 751
[   22.688461] cpsw cpsw:       stats pad_enqueue: 0
[   22.692914] cpsw cpsw:       stats mised: 1
[   22.697185] cpsw cpsw:       stats desc_alloc_fail: 0
[   22.702000] cpsw cpsw:       stats pad_alloc_fail: 0
[   22.706725] cpsw cpsw:       stats runt_receive_buff: 0
[   22.711722] cpsw cpsw:       stats runt_transmit_buff: 0
[   22.716809] cpsw cpsw:       stats empty_dequeue: 0
[   22.721444] cpsw cpsw:       stats busy_dequeue: 247
[   22.726170] cpsw cpsw:       stats good_dequeue: 749
[   22.730895] cpsw cpsw:       stats requeue: 0
[   22.734986] cpsw cpsw:       stats teardown_dequeue: 0

0 Jaroslav Vozab over 11 years ago in reply to Jaroslav Vozab

Prodigy 70 points

I have found the problem. There is a bug in cpsw_rx_handler function CPDMA buffers are not handled correctly, when an ethernet interface was running and then stoped.

Best regards

Jaroslav

0 Boris-Ben shapiro over 10 years ago in reply to Jaroslav Vozab

Expert 1300 points

Hi Jaroslav.

I am facing a very similar issue to yours... do you maybe remember where did you found the bug in rx_handler??

Please Help :)

Boris-Ben Shapiro
RT Embedded Engineer
Elspec LTD

Processors

Processors forum

cpsw dual emac mode freeze