USB (in)stability on DM816x / EVM816x board

"I get some short periods of correct behaviour, then 'something' stops working."

I have seen similar behaviour with my USB keyboard  - sometime works, then stops and then works again .....

Tested on a custom board with EZSDK 5.02.02..60 -> Arago -> Internet Browser

Mouse works ok - in both USB slots.

Hi

There are additional usb patches on top of 4.0.0.12 is available at arago http://arago-project.org/git/people/?p=sriram/ti-psp-omap.git;a=summary . Starting from patch

commit : 61fcc607145bbdb62f408b5d912a1fa6e13e96f2
ti816x: Added CPU check revision for TI816X ES2.0

Regards

Ravi B

Robert

Please check whether usb keyboard  and mouse is enumerated succesfully. If event ineterface nodes for usb keyboard/mouse has been created (/dev/input/event0 and /dev/input/event1) then there is no issues from USB perspective. Check whether any issue from application layer.

Regards

Ravi B

Hi Ravi,

I believe the keyboard is enumerated ok. See my console output (first is mouse, second is keyboard):

root@dm816x-evm:~# usb 1-1: new low speed USB device using musb-hdrc and address 2
usb 1-1: New USB device found, idVendor=046d, idProduct=c518
usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 1-1: Product: USB Receiver
usb 1-1: Manufacturer: Logitech
input: Logitech USB Receiver as /devices/platform/musb-ti81xx.0/musb-hdrc.0/usb1/1-1/1-1:1.0/input/input0
generic-usb 0003:046D:C518.0001: input: USB HID v1.11 Mouse [Logitech USB Receiver] on usb-musb-hdrc.0-1/input0
input: Logitech USB Receiver as /devices/platform/musb-ti81xx.0/musb-hdrc.0/usb1/1-1/1-1:1.1/input/input1
generic-usb 0003:046D:C518.0002: input: USB HID v1.11 Device [Logitech USB Receiver] on usb-musb-hdrc.0-1/input1
usb 2-1: new low speed USB device using musb-hdrc and address 2
usb 2-1: New USB device found, idVendor=046d, idProduct=c316
usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-1: Product: Logitech USB Keyboard
usb 2-1: Manufacturer: Logitech
input: Logitech Logitech USB Keyboard as /devices/platform/musb-ti81xx.1/musb-hdrc.1/usb2/2-1/2-1:1.0/input/input2
generic-usb 0003:046D:C316.0003: input: USB HID v1.10 Keyboard [Logitech Logitech USB Keyboard] on usb-musb-hdrc.1-1/input0
input: Logitech Logitech USB Keyboard as /devices/platform/musb-ti81xx.1/musb-hdrc.1/usb2/2-1/2-1:1.1/input/input3
generic-usb 0003:046D:C316.0004: input: USB HID v1.10 Device [Logitech Logitech USB Keyboard] on usb-musb-hdrc.1-1/input1

root@dm816x-evm:~#

Hi Ravi,

Thanks for your response - it's really appreciated.   Unfortunately I've also already been tracking the arago repository regularly; I've seen and tested your changes, which do help operation with DMA to some degree, but still everything eventually dies.  The Arago tree also appears to have a broken NAND driver or some sort of problem with JFFS2, which makes it difficult for me to use that tree as a whole.

What classesof devices have been tested with this driver?  Any Wifi dongles or webcams?  I guess from your reply that you think USB support should be fairly reliable at this point in time already?  Might it be possible for us to send you some of the sorts of devices we're seeing big problems with?   I have asked for FAE support because USB support is so critical to the product we are developing, but there doesn't really appear to be anything for me here in the UK, just sales agents.

Many thanks again,

Ian.

 

Jeffray

We have tested with cameras with high bandwidth support. We have not verified with wifi dongle.  Can you explain the issue in detail what is not working. You can enable the usb driver log messages by "echo D8 > /proc/driver/musb_hdrc.0 or 1" and provide the dump of "dmesg" command.

Regards

Ravi B

Hi Ravi,

Thanks again for the reply.  Nice to hear you've had some high bandwidth transfers working.  I wish I had. :-)

'Reasonable' data transfer via USB causes it to stop, and the device must be pulled out before anything works again.   For example, with a wifi adaptor, doing simple 'pings' will be perfectly fine (I've left it doing that for hours, just fine), but it's not reliable enough to ftp even one file.  [The wifi driver has been well proven on other ARM and x86 platforms, and I'm just about to check it on Blackfin, which has an Inventra/MUSB host device].

I can't easily apply all your recent 29 change sets from the arago tree because there are some failed dependencies.  Currently trying resolve changes in omap_hwmod_81xx_data.c.

I can't use the latest arago tree as a whole, because it seems the NAND driver has been rather broken.  I've spent some hours trying to fix that, reverting the changes that seem to have been made since the 4.0.0.0.12 release, but I can't get it to work properly at all.   (All my Wifi configuration tools run from NAND).

Still trying to apply all your changes, then I'll try and provide some debug trace.

Ian.

 

Is this related to the "babble interrupts" that I frequently see?

Lee

 

Ian

In the first mail you have mentioned "Behaviour is better with a memory stick than it is with a Wifi device.".

1) Do you mean you face issue with memory stick. can you explain what is the issue.

2) With wifi device with ftp, is it consistent failure for every file transfer, or is it intermittent behavior? When issue occurs, can you provide the musb console logs (echo D8 > /proc/driver/musb_hdrc.0 or 1)  or USB bus capture, this could help me to understand what is the issue. 

3) With PIO mode you should able transfer the file (though performance is low). Do you see any issue here?

Regards

Ravi B

 

Ravi B said:

1) Do you mean you face issue with memory stick. can you explain what is the issue.

2) With wifi device with ftp, is it consistent failure for every file transfer, or is it intermittent behavior? When issue occurs, can you provide the musb console logs (echo D8 > /proc/driver/musb_hdrc.0 or 1)  or USB bus capture, this could help me to understand what is the issue. 

3) With PIO mode you should able transfer the file (though performance is low). Do you see any issue here?

Hi Ravi,

Thanks again for the reply.  I did actually manage to apply all your USB patches to the kernel last night and there's clearly some level of improvement, but performance is now very, very low.  My 300MBps wifi device transfers data at less than 1MBps, with or without DMA enabled.

To answer your questions:

1) We used many memory sticks for testing, and got best performance from a SanDisk Cruzr Contour - 18MBytes/sec.  This does prove that USB is able to give good throughput with some classes of device.

2) The failure happens immediately and consistently.   Before I applied your recent patches, even "iwlist wlan0 scan", to scan for access points, would fail.   Now that seems more reliable.

3) Both PIO mode and DMA mode seem the same with your recent patches - very, very slow transfer, but it is at least reliable now which is good progress.   I would not consider wifi to be truly "usable" still (under 1Mbit transfer rates) but your work has clearly improved something.

I have tried the 'echo D8' procedure, but no extra debug seems to appear - I see almost nothing about USB at all -- and I believe I have selected every USB and kernel debug option available in my kernel configuration.

We have ordered a BeagleBoard-xm, which has the same musb host, in order to be able to compare against operation there.   Will let you know our results.

Regards,

Ian.

 

Ian

Thanks for the reply. When you are taking performance do not enable the debug ("echo D8"). If you enable musb debug (echo D8), then you must use "dmesg" command on console to get musb logs.

Regards

Ravi B

Ian,

Beagle-xM uses Inventra DMA and not CPPI4.1 DMA.  You cannot compare the performance with Beagle-XM

Regards

Ravi B

To follow up on this some more, I'm still searching for a Wifi solution that will work with the DM81xx series.

I've now purchased and tested eight different USB wifi dongles and none work properly on this processor - a total disaster.

I occasionally see the following kind of errors:

musb-hdrc musb-hdrc.0: dma_pool_free buffer-2048, ffc14000/8ba66000 (bad dma)
musb-hdrc musb-hdrc.0: dma_pool_free buffer-2048, ffc15000/8b9ae000 (bad dma)
musb-hdrc musb-hdrc.0: dma_pool_free buffer-2048, ffc16000/8b9a7000 (bad dma)
musb-hdrc musb-hdrc.0: dma_pool_free buffer-2048, ffc17000/8b9ef000 (bad dma)

The only place the DM81xx mUSB host driver calls dma_pool_free is in cppi_dma.c:cppi_pool_free

Does this give anyone any insights?

(I have exactly the same problem on DM816x EVM and DM814x EVM so it's something very specific to CPPI DMA it would seem)

 

Jeffray

Most of the usb wifi dongles does not use standard CDC class.   We have not verified the usb wifi device on DM816x. I am currently working on wifi device (RaLink), will keep you posted on this.

Let us know what are the list of USB wifi dongles you have tried

Regards

Ravi B

Hi Ravi,

I guess this really should be a different named thread now, because since your recent great work for the 04.00.01.13 release, the DaVinci USB host has actually been "stable" but seemingly very underperformant for wifi devices.

The most success I have had has been with the rtl8712 driver, for Realtech 8192se devices.   (LMTechnologies LM005 andLM006) That's in the "staging" area for the old (current) 2.6.37 kernel.   I've also had some level of success with the rtl8192ce driver (unbranded devices, download from realtek.com as it's not integrated in to the 2.6.37 kernel).  I get around 10Mbit/sec sporadic transfer at maximum with these devices.  (compared to 40Mbit/sec on a slow old 200Mhz ARM9 board, and 94Mbit/sec on an x86 PC).    I have also tried several rt2xxx devices (Sweex LW324 and Edimax EW-7722UTN), some atheros based devices (carl9170 driver,  TP-Link WN821N) and attempted several others.   I have backported many fixes from upstream kernels to ensure the wifi drivers are as up to date as possible.

I have tested with several different access points and configurations, on DM816xEVM and DM814xEVM with the latest EZSDK/PSP releases and also with the latest fixes in the arago git.

My focus is on the rtl8712 driver for the 8192se chipset because that's shown the best performance and complete reliability on other platforms and the driver is integrated in to the kernel already and seems stable.   I have contacted the driver maintainer for suggestions and tried several things to check possible issues such as buffer alignment.

I have offered to buy our local TI rep some of these USB devices to test with, and I really appreciate that it's technically outside of your remit to resolve issues with third party hardware but it would really seem that since the problem can be almost 'proved' to be something specific to the CPPI DMA in the TI device, I do think it needs TI help to resolve this.

There is currently no performant 802.11n solution for DaVinci.  The best TI offers is the wl1271 "TiWi" device which is limited to 64Mbit - so we have to look to a USB solution.

Regards,

Ian.

The chance of this helping is rather small, but have you tried connecting the wifi dongle via a powered usb hub?  Just wondering if there is some usb power issue, since I could easily see a wifi dongle exercising thouroughly the power the usb port can provide.

 

Hi Jason,

We did try this.  Several USB powered hubs, and also cutting a USB extension cable apart and running the wifi device power from a strong lab supply.   Unfortunately, nothing helped.  These devices typically seem to draw very much less than the 500mA available from the USB host anyway.

Regards,

Ian.

 

I should probably move this to a new thread with a better name, but to continue my attempts to use with Wifi on Dm81xx...

I've tried the latest Arago git kernel today, with RaviB's latest changes for USB, including the Tx DMA changes which looked useful.

Unfortunately the driver appears to be *MUCH* worse now.  I was getting a horrible 10Mbit throughput on wifi before, and now it's down as low as 4Kbit at times, and only 500Kbit at the very best.  Pretty horrendous.

I guess I may try and back out the USB changes one by one to see what's made it go so bad... it's very odd.

How could we get some "proper" committed (paid?) TI help on this big problem?  Our local FAE hasn't been able to help.

 

By disabling DMA, we improve from 0.5Mbit/sec to 85Mbit/sec

I think this rather proves DMA is still very broken for DM81xx devices :(

 

Ian

You said you could get 10Mbps and after including the usb patches from arago, the perf. drops down to 0.5Mbps. Can i know the list of patches (with commit id) you applied which drops the performance to 0.5Mbps. Can i know make or model the wifi dongle device you have tested.

Regards

Ravi B

 

Hi Ravi,

Ok, I've spent some hours on this specific PIO testing, and the situation is pretty strange:

Using "LM Technologies" LM005 and LM006 adaptors (these have proved most reliable and the driver is fairly simple and self contained).  The chipset is Realtek RT8192SU which is supported by the "r8712u" driver in the arago/TI kernels in the 'staging' area.

On DM816xEVM (RevF):
With arago git + DMA : 3Kbit to 0.4Mbit  (sometimes fails to transmit anything at all)
With arago git + PIO : usually below 0.1Mbit, but have seen 21Mbit at best (and sometimes fails to transmit anything at all)
With latest release PSP + DMA : 3Kbit to 0.4Mbit (one run saw an initial 5 second burst of 54Mbit but that was unrepeatable)
With latest release PSP + PIO: 0.5Mbit at best

On DM814xEVM (PG2.1):
With arago git + DMA : 0.5Mbit at best
With arago git + PIO : Always over 70Mbit-77Mbit/sec (but with a loadav going up to 0.3 during extended testing)
With latest release PSP + DMA : 0.5Mbit to 1Mbit
With latest release PSP + PIO: Always over 70Mbit/sec (saw 89Mbit once)

On x86 PC (for driver verification and performance comparison):
94Mbit/sec sustained

On old 400Mhz ARM9 board: (for another ARM comparison)
40-44Mbit/sec sustained

Thankfully, our current product will use AM3894 (DM814x) so the fact it's always dreadful on the DM816x is acutally less of an immediate issue,

Sticking to PIO isn't really a great "solution" for us because we need all the CPU horsepower for processing work while transmitting.

PIO mode also completely kills the throughput for USB based discs and memory sticks... down from 20MBytes/sec to 8.2MBytes/sec :-(

 

FWIW, I've just taken delivery of an AM335x "Beagle Bone" board and the problem is exactly the same on this platform.

Wifi with TI USB seems to be just horribly broken.  :-(

 

None of BeagleBone linux releases from TI has supported wifi until  PSP 04.06.00.03 [Dec-1011] and so would advice to you read the release note first and try wifi if it is claimed to be supported.

Ajay

Thanks for the reply Ajay, but the point is that the USB host controller has a huge problem.  It's not especially important that any specific wifi functionality works, but the fact that *NO* USB driver I've tried works properly with DM814x/DM816x/AM355x TI devices, but they *ALL* work fine on x86 and ARM9 devices, does pretty much prove the TI USB host controller DMA is broken.  My worry is that this is a hardware fault and that we need silicon fixes...

 

Has the development of the TI Linux PSP for these platforms stopped now?

I've seen very little change for several months now, and specifically our local TI rep said my specific issue would be investigated.

Nothing has happened.  What's going on?  How do I get TI to actually do something, to provide proper support?

Well you always could do what we did, install a USB 2.0 PCIe card and forget about using TI's built-in USB.  To get this to work we had to rebuild the kernel having ECHI HCD (USB 2.0) support as a module.  It is my understanding the USB issues are fixed in the Rev 2.0 silicon.

Lee

 

Hi Ravi/Ajay,

If WIFI issue on DM816x Platform is resolved ( ie DMA Mode is working) ?

If yes, could you please share the commit id for fix patches .

Thanks in Advance.

With Regards
Ashish