AM3352 USB transaction error

Correction to the above post...

...suggests that it's NOT something like the DMA controller fetching...

Some additional info on testing:

Lot codes that were tested that fail:  58AHE4W, 58ADS2W, 58AHEFW, 58A5L9W, 58ASVOW, 58A95JW, 58A5N2W, 58AKPRW, 58AKQEW.  Lot code 58A5N4W has also seen failures as reported in the original post.

From a sampling of 879 boards that have 18 different lot codes, 2 units from each lot code were tested.  9 of the lot codes failed.  The test was to run the system once and then cycle power.  Repeat 100 times.  The typical failure point was within 10 power cycles but there were three outliers that required 34, 44 and 68 power cycles.  All 18 lot codes indicate August 2015 manufacturing date.

Replacing the processor on four boards that were failing 'fixed' the problem with those four boards.

Kevin,

As replacing the processor on the failing board fixed the problem, do we confirm this is HW issue instead of SW? 

Regards, Eric

Kevin,

"If the problem is software, again I would need some assistance to try to identify an area since we're using TI sample code."====> As it is inclusive whether HW or SW issue, can you explain what exact version of TI SW for USB driver was used? Then we can try to see any SW issue on TI EVM setup.

Regards, Eric

Kevin,

SW: I found 2.0.0.5 here http://processors.wiki.ti.com/index.php/StarterWare_OldReleases. As you made the changes for bug fix, was the change inside the library usblib.lib? Or the changes is inside the examples? There are several device examples, which is the one you based on? Is the code or library proprietary? I am not sure the scope of the code change, is it proper to post into the forum? 

I wanted to test the code on my setup here with an AM335x EVM as a USB device, to see if I have any errors to copy files with a PC via power cycle test. I don't have exact lot number of the chip on my EVM, so may or may not see the same problem. I hope to rule out the SW issue with the test. 

Regards, Eric

The changes were made to the usblib, not the examples.  Our software guy said one of the changes was in an interrupt routine that was hanging around waiting for an event to complete, meanwhile missing other things that were going on in the system.  He had to restructure the code somewhat to fix so it might not be compatible at a higher level with what you're running.  I've put our code on our Sharefile:  https://burroughs.sharefile.com/d-sea05552d8e449beb

@DK

"I've forwarded the issue to our production team to see if they can offer some insight into the lot codes you listed."

Did anything result from your inquiry above?

Is there anything that can be done to escalate this problem within TI?

Kevin,

There are many difference bewteen customer's usblib folder and TI 2.0.0.5 GA folder. I was not able to build the library with their code. Do they have a pre-build library and with compatible API to TI test code?

Regards, Eric

Eric,

Unfortunately, our software does not work with the unmodified usblib code and the modifications that we made to allow usblib to work break compatibility with software that you have that does work with the unmodified usblib code.

Another data point we came across on Monday is that unplug/replug the USB cable can be made to cause the failure.  So far though we haven't been able to show that unplug/replug of the USB cable can cause the failure to be 'fixed' from a broken state but there has only been fairly limited testing of that.  However, since reseating the USB cable can cause the failure, and not just power cycling, it does seem to remove several potential suspects such as boot up code, power supply sequencing etc.  What would seem to be left is hardware/software specific to starting up USB.

I still strongly suspect that this is a hardware issue due to the nature of the failure (i.e. data being shifted by two bits, then four, six and eight) but so far nobody at TI has been able to confirm or suggest additional testing that could resolve.  Maybe there is something PLL related that is occurring at USB startup and the PLL is taking longer than it should to lock to the proper frequency?  Or from a software angle, maybe there is some delay that is required that is not being honored?  Is there some specified delay that must be met after a USB connect prior to sending packet data that perhaps we're missing?  If so, could you reference?  Since we're able to cause a failure via human reaction timeframes, I wouldn't expect small number of milliseconds or less to be an issue, but larger timeframes could.

Kevin,

Can you provide the register dumps for working case and failure case:

0x44e1_0620

0x44e0_0004

0x44e0_001c

0x44e0_047c

In your test of case of unplug/plug USB cable to create the failure, can you explain: is the USB host a PC? failure is USB enumeration failure? Or the failure is data transmission failure but enumeration always happening? Do you have to use a scope to monitor data bit shift to declare a failure or you can use PC to copy files to find the failure?

Regards, Eric

Eric,

Will take some time to get the info on the register dumps, but I'll put it on the list of things to do.

The host in all cases is a PC (not just one PC).  When I refer to a failure, I'm referring to the 'Xact Error' that gets reported when the device is running a diagnostic that is simply transmitting known, canned data.  No enumeration problems.  In fact, even after the 'Xact Error', I'm able to communicate without issue as long as it is via messages that happen to result in 'small' USB packets (i.e. not the 512 byte packets that are occurring when running the diagnostic).

To get detailed data on the cause of the 'Xact Error', I have to use a hardware USB analyzer.  With that I can see all of the data packets and find the packet with the error.  Although 'Xact Error' could have multiple causes, what I'm seeing is data corruption in the packet which results in an invalid CRC.  I've captured 14 of those errors.  The typical running though simply needs to look on the PC for the 'Xact Error' to be reported, or for the diagnostic to complete.

Eric,

Register values you requested.  Values are the same both during a good power up and a bad one.

Register                       Address     Value

========                       ==========  ========

USB_CTRL0 REGISTER             0x44e10620  3C186004

CM_PER_L3S_CLKSTCTRL Register  0x44e00004  0000000A

CM_PER_USB0_CLKCTRL Register   0x44e0001C  00000002

CM_CLKDCOLDO_DPLL_PER Register 0x44e0047C  00000300

Some additional failure information.  One particular run of the test showed a number of different things:

- Apparent bit shifting by both an even and an odd number of bits (previously the bit-shifting phenomenon had always been 2,4 or 6 bits)

- Apparent dropping of three bytes of data from the data stream.  See packet 683768 below.  Both the analyzer and the PC saw this as a CRC error.  The PC did not ACK which causes the AM3352 to resend the packet.  On both the original transmission and the resend, the USB analyzer detected the same CRC being transmitted by the AM3352.  The resend however was not missing the three bytes from the first transmission so now the CRC that was transmitted was correct.  See packets 683768 and 683770 below.

- Six of the eight data packet errors below show the USB host sending an ACK to a data packet that the hardware USB analyzer had reported a CRC error.  On the other two CRC errors, the host did not ACK (which is the correct behavior) and the data was resent in the next transaction.

Firmware analysis:

Tracing was added in the HandleEndpoints() function (modified a bit from the published TI StarterWare) in order to log when the function was called and the current state of the Dma Pending Status.  Specifically, ‘pendreg’ was logged where ‘pendreg’ is:

                // Get the DMA Interrupt status

                pendReg = CppiDmaGetPendStatus(USB_INSTANCE);

The trace showed that the function was called 56 times.  All but one time, pendreg was set to 0.  One time it was set to 0x40000000 (CPDMA_TX_PENDING).  The first 55 calls to the function occurred within 1.484 seconds.  There was then a gap of 2.381 seconds to the next call.  The test that was running has a 5 second timeout after which the USB Transaction Error was reported.  It is not clear if there is a way to correlate the firmware trace timings with the hardware USB analyzer timings.  Given the amount of data that was transferred, only 56 calls seems light.

An example of a much smaller data packet of 39 bytes that was transmitted three consecutive times and failed every time.  All three times the PC did not respond with an ACK indicating that from it's view, the data packet was corrupt.

The correct data stream is the following:

09 02 27 00 01 01 00 C0 FA 09 04 00 00 03 FF FF FF 00 07 05 82 02 00 02 00 07 05 04 02 00 02 00 07 05 81 03 80 00 01 

This data stream was sent with a CRC16 of 72CC according to the USB analyzer (which did not flag a CRC error).  Since the PC did not ACK, the AM3352 resent the same data packet.  It was mostly correct.  The last six bytes in the packet (highlighted above), came out as 15 02 07 00 01 02 with a CRC16 of 3966 which the USB analyzer flagged as being incorrect.  Once again the PC did not ACK this packet, so the AM3352 sent the packet a third time.  The data packet and CRC16 were exactly the same as the first time it was sent and again the PC did not ACK the packet.  There was no fourth attempt to send the packet, not sure why.

Sure seems like the chip itself is flaking out and bit timing is slightly off somehow.  But if it powers up and works the first time, all is good until the next power up which rolls the dice again.

Eric,

On the hope that there would be a clue in the register settings, I dumped all of the USB registers and some of the possibly relevant clock module registers.  The table is shown below.  Here is a description of what is contained in each column

RegAddr(Hex):  Regiater address

RegData(Hex): Data read from RegAddr

Addr Match:  Two dumps of the register values were taken, one before the first DMA transfer and one after the DMA transfer had failed.  Comparisons of the address and data were made to see where something had changed between these to register dumps.  A value of 'True' for Addr Match indicates that the address is the same for both dumps (mostly this is a sanity check to make sure that the data is compare is comparing the correct things).

Data Match:  Similar to Addr Match but flags when the data has changed between the two dumps.

Good Match:  I also made a run that did not have any problems and dumped the register values as described above.  This column indicates whether the data from this 'good' run matches the data from the failed run.

Good Data:  This is the data from the 'good' run.

Register Name:  Over after the // is a reference to the register name and the section in the Technical Reference manual.

Difference:  Some of the register contents happen to change from run to run whether this was a 'good' run or not.  I went through all of the places where the data differed, and noted the bits that were different and whether it appeared to me to be relevant or not.  I didn't find anything that jumped out as a notable difference, but maybe somebody with more knowledge in this area of the part can review.

Can anybody suggest other fruitful areas of the memory that might be useful to dig into for differences?