Compiler/CC2640: UART Sending Spurious Data

FYI: Not a compiler issue. You can change that designation

Can this be closed?  Same as https://e2e.ti.com/support/wireless-connectivity/bluetooth/f/538/t/869514

Regards,

travis

Travis

I marked https://e2e.ti.com/support/wireless-connectivity/bluetooth/f/538/t/869514 as resolved and created this issue we are in

Here is a sample of the corruption. The setup is that teraterm sends a command to the LaunchPad, LaunchPad will echo its receipt and then forward the message to a BLE peripheral and then the peripheral will respond with its own message

You can see the spurious data in loop 2010 right after "id:" is spit out. Other analysis that we have done is from prior data we have sent. You can see that perhaps the data in the last sequence(handle 46) came from the handle: 45 sequence. Really weird. Not sure how to debug this

Current Loop Number 2003
{"optype": "call", "id": "bm-w", "handle":0, "data": "AABB010400C03E20", "secure": 0}
<{"optype":"status","id":"bm-w","handle":0,"error":0}
<{"optype":"event","id":"bm","handle":39,"data":"AABB03202020112525252524252627000102035252525200000000000000140200000200"}

Current Loop Number 2004
{"optype": "call", "id": "bm-w", "handle":0, "data": "AABB010400C03E20", "secure": 0}
<{"optype":"status","id":"bm-w","handle":0,"error":0}
<{"optype":"event","id":"bm","handle":40,"data":"AABB03202020112525252524252627000102035252525200000000000000140200000200"}

Current Loop Number 2005
{"optype": "call", "id": "bm-w", "handle":0, "data": "AABB010400C03E20", "secure": 0}
<{"optype":"status","id":"bm-w","handle":0,"error":0}
<{"optype":"event","id":"bm","handle":41,"data":"AABB03202020112525252524252627000102035252525200000000000000140200000200"}

Current Loop Number 2006
{"optype": "call", "id": "bm-w", "handle":0, "data": "AABB010400C03E20", "secure": 0}
<{"optype":"status","id":"bm-w","handle":0,"error":0}
<{"optype":"event","id":"bm","handle":42,"data":"AABB03202020112525252524252627000102035252525200000000000000140200000200"}

Current Loop Number 2007
{"optype": "call", "id": "bm-w", "handle":0, "data": "AABB010400C03E20", "secure": 0}
<{"optype":"status","id":"bm-w","handle":0,"error":0}
<{"optype":"event","id":"bm","handle":43,"data":"AABB03202020112525252524252627000102035252525200000000000000140200000200"}

Current Loop Number 2008
{"optype": "call", "id": "bm-w", "handle":0, "data": "AABB010400C03E20", "secure": 0}
<{"optype":"status","id":"bm-w","handle":0,"error":0}
<{"optype":"event","id":"bm","handle":44,"data":"AABB03202020112525252524252627000102035252525200000000000000140200000200"}

Current Loop Number 2009
{"optype": "call", "id": "bm-w", "handle":0, "data": "AABB010400C03E20", "secure": 0}
<{"optype":"status","id":"bm-w","handle":0,"error":0}
<{"optype":"event","id":"bm","handle":45,"data":"AABB03202020112525252524252627000102035252525200000000000000140200000200"}

Current Loop Number 2010
{"optype": "call", "id": "bm-w", "handle":0, "data": "AABB010400C03E20", "secure": 0}
<{"optype":"status","id":"bm-w","handle":0,"error":0}
<{"optype":"event","id":"bndle":45,"data":"AABB03202020112525252524252627000102035252525200000000000000140200000map","handle":46,"data":"AABB0320202

I'm routing it to the correct person now...

Thanks Travis. The thread is marked as TI THINKS RESOLVED, can that be changed?

There should be a button/pull down menu that you can click on for that post and select "did not solve my problem"

Sorry Travis, not seeing it

Sorry, I can't see your view, but I'm told on the post that is marked "TI Thinks Resolved", there should be a button for you to click that says "This did NOT resolve my issue".

I know the following update will NOT make sense. It doesnt to me

We've been able to identify a working model and a failing model

The failing model is when we update our devices using dslite.bat which is created using Uniiflash.

The working model is when we update the code image using SmartRF  programmer. In this test we download the same .hex file which is located in the zip file that UniFlash created.

After updating the LaunchPad using SmartRF programmer, I was able to get  over 1 million UART transfer without error. Using dslite.bat created by Uniflash.exe we are encountering a UART issues well under 25K transfers.

Wondering what your thoughts are at TI

Hello TI

is there anyone yet assigned to this?

Here is my update and I hope someone from TI can help soon.

If I run dslite.bat which was created using UNIFLASH, I eventually run into UART issues on my application.

If I connect to SmartRF Programmer (post dslite.bat execution), my device comes up as unreconginzeed and the program prompts for an update of the debug probe. I comply and allow the update of the probe, but  I  DO NOT then see any UART issues going forward. Please note that I am NOT downloading my image to the target, I am simply updating the debug probe.

I summize that something is being downloaded during the dslite.bat executing which contains this UART issue. I am eager to have TI weigh in on what could be going on.

Additionally, we have gathered USB traces using a Beagle 40 USB probe which in fact shows bogus  data coming from the device. Internal tracing of UART writes show the application is not requesting this bogus data transfer

Hi Paul

First of all, please let me know which version of the SDK you are using and what HW you are running on.

I also need to know which version of the Flash programmer you are running and what version of UNIFLASH you are using, as this sounds to be a tools problem more than an actual UART problem.

Since it is working when using the Flash Programmer, I recommend you to test to download the code with UNIFLASH, using the GUI, and not the Command Line Interface. Does it work as expected then?

Siri

Hi Siri

the SDK for the launchPad is simplelink_cc2640r2_sdk_2_40_00_32. I am running on the CC260R2 launchpad

SFLASH Programmer 2 is 1.8.1

Uniflash is 5.2.0.2519

ToolSet is xdctools_3_51_01_18_core

SO, here is some other information. I use uniflash to  create the DSLITE.BAT, and use that batch to download the image, I have UART issues. If I then launch FlashProgrammer 2 (after doing dslite.bat) my device comes up as unknown and it requests a firmware update of the debug probe. If I conduct that update without downloading the target code image, the UART issues is not seen at all

So I agree it is a tools or driver issue.

Paul

I will try to find someone familiar with the Uniflash tool to look into this.

Hi,

To eliminate if the problem is related to the flash programming or the the handling of the UART communication there are a couple of things that can be done.

• Dump the flash image that is programmed with the SmartRF Programmer and UNIFLASH to compare and see if there are any differences.
• Program the image with SmartRF Programmer. Then update the XDS firmware to the version that is used with UNIFLASH.
  You will find the firmware image under the Uniflash installation: .\deskdb\content\TICloudAgent\win\ccs_base\common\uscif\xds110.
  The firmware can be programmed with the "xdsdfu" tool. Normally that should be available after installing CCS. Hint: search "xdsdfu.exe" under the c:\ti\ccs_base folder.
   

Regards,

Øyvind

This is almost certainly an XDS110 firmware issue.

The working firmware is version 2.3.0.18. The failing firmware is version 3.0.0.5.  Because that is a major (and not backwards compatible change) between the two versions, each emupack must re-flash the XDS110 with the firmware it has. That's why you would see one tool reprogramming the XDS110 after using it on the other tool.

Otherwise, UART is handled separately from any tools or drivers in the XDS110.

What are the steps needed to reproduce the issue here?

What is the baud rate? (At a certain threshold, the UART routines switch to DMA transfers, so there are basically two different UART implementations in the firmware.)

Hi Paul

Someone in the Tools team has been assigned to look into this. It would be very helpful if you could provide us with as much information as possible so that we can try to reproduce the problem here. Are you able to see the same problem with a simpler program? Have you tried to use the uartEcho examples available in the SDK andset it up with your UART configuration and have it fail?

If no, do you have code that you can share with us (offline) so that we can have the same test setup as you?

The least we would need is your complete UART configuration

BR

Siri

Hello

I am currently out of the day but I will be able to look into this when I return to the office on Monday

Paul

Øyvind

can the version of XDS firmware be selected/changed when creating the "standalone" command package?

When generating the "standalone" command packets, the drivers on the host side that is compatible with the XDS firmware will be part of it, so to do that you also have to change everything in the "ccs_base" folder. Never tried but might work.

Øyvind

Hello TI Engineers

unfortunately I will be out of the office again on January 13th. However I have asked my colleagues to try and provide some of the information you require

Paul

Hi Edward,

The baud rate is 921600. However, we tried to run the same test at 9600 and still had the same issue. We noticed the problem started to occur when our payload size increased. As Siri suggests we can try to reproduce the problem on a simpler UART echo example with large payloads, If the issue is XDS110 firmware then this test should confirm that. 

Hello TI engineers

I think we have a finding which might be of interest to you. If we use UNIFLASH package 5.0.0.2289 instead of package 5.2.0.2519 the UART corruption does not occur

Additionally, after loading code using dslite.bat created by package 5.0.0.2289 we can then launch Flash Programmer and have the LaunchPad CC26040R2 be properly detected (by Flash Programmer). If you remember that after running dslite,.bat created using version 5.2.0.2519 the Flash Programmer doesn't correctly identify LaunchPad

I feel these are critical findings and look forward to your guidance concerning this recent finding.

Regards Paul

Hi Edward,

do you still require the information you requested on Jan 09 or is does my most recent post provide a different direction?

paul

Hi Siri,

We managed to run the TI-RTOS based UART Echo example from resource explorer and at payload about 70 bytes we are seeing the same malformed data. We running at 921600 baud and the UART setting are  

   uartParams.writeDataMode = UART_DATA_TEXT ;

   uartParams.readDataMode = UART_DATA_TEXT;

   uartParams.readMode = UART_MODE_BLOCKING; ;

   uartParams.readReturnMode = UART_RETURN_FULL;

   uartParams.readEcho = UART_ECHO_OFF;

   uartParams.baudRate = 921600;

I inserted a work document with screenshots of good and bad responses with increasing payload length

TI-Dongle-UART-ECHO-ISSUE.docx

Uniflash 5.0.0.2289 includes XDS110 firmware 2.3.0.18 which is working.

I'll try to duplicate the issue using the info in Anurag's post.

Edward

just to document this in the JIRA which version of XDS firmware is used in 5.2.0.2519 Uniflash?

The XDS110 firmware in Uniflash 5.2.0.2519 is 3.0.0.5 (which I got from the Jira).

The UART Echo example appears to have an issue itself and may not be demonstrating the problem.

I built the example per the instructions posted above and used TeraTerm to send a 60 byte and a 70 byte file at 921600 baud. The 60 byte file always worked, and the 70 byte file was truncated. However, when I loaded the XDS110 firmware we believed to be working, the 70 byte file was still truncated.

So, I tried just looping the TX and RX pins of the XDS110. In that case, the 70 byte file was sent and received correctly with both versions of the XDS110 firmware.

I then connected an FTDI UART to USB cable to the TX and RX pins of the CC2640 core. And with this setup, the 70 byte file was again truncated. The example code running on the CC2640 appears to not be keeping up with the data.

So it appears there is an issue with the example, and it's probably not sufficient to demonstrate the original issue.  Would it be possible to get the binary file used to reproduce the original problem?

Hi Edward, I can provide the binary file. Do you have a box location?

I assume you want to examine the binary file and not run our target code, is that correct?

I would need to run something on the CC2640 that duplicates the issue.  The UART Echo example seems to not suffice, certainly didn't reveal any difference between the 2.3.0.18 firmware and 3.0.0.5.  So I'm still at a loss on how I can debug this on my end.

Right, so running our target code isnt going help. The LaunchPad is used to communicate to a BlueTooth device and doesn't provide much functionality otherwise

Would it be beneficial to you if I created a command which got sent to the LaunchPad which would then simply echo that data back? I think it might be useful. I could test that is creates this specific issue

Paul

Yea, that might do it. We just need to pin down what's happening between the CC2640 and XDS110 -and- why the newer firmware isn't working properly. If you could get something that works on 2.3.0.18 and fails on 3.0.0.5 the same way, that'd be perfect.

So, in your use case, the XDS110 is not sending much data to the CC2640, most of the traffic is from the CC2640 to the XDS110 (and then on to the host)?

Hi Edward,

I am running the actual production code flashed using dslite with the failing XD110 version, but I am bypassing USB-UART emulation chip by connecting TX and RX pins of the core to TTL-USB converters. If we don't see any problem this will prove your theory. Once I hit 100000 loops in my test I would consider it to be passed.

Hi Edward

it is unclear to me exactly where the dividing line is between the XDS110 and the CC2640. We are running a simple central application where the computer sends JSON packets to the LaunchPad and the LaunchPad will send those JSON packets over BLE to a target. That target will respond with data and upon receipt of a BLE notification from the target we will then packetize that data into a JSON string and send it back over the UART.

Given the fact that we have found a version of Uniflash which works for us, we do believe we can proceed on our end. However I also think that it would be good for TI to understand and fix the issue. So what I am saying is that we are conducting this debug exercise so that TI can  understand the problem on your end to satisfy your end users needs. So please keep that in mind,.

Hi Edward,

So after running the above test, we had no failures if we bypass the broken XDS110 firmware and using a TTL-USB converter instead. So this test proves the point that that version of XDS110 fails randomly when our payload is large.

Alrightly, then.  Some further questions to help figure out how to reproduce this here:

How much data is being sent?

Is the data stream fairly constant, or are there pauses between individual messages?

If there is a pause, can you characterize how much and how much it may vary?

Is the XDS110 doing anything else while this is happening (e..g debug session open, other COM port in use, Energy Trace session)?

Is the XDS110 plugged directly into the host, or is it connected via a hub?

And is the software on the host running natively or in a VM?

Thanks.

Edward

in my test case, I am sending/receiving data with constant data, it does not vary. Obviously we can send varying data but for test purposes I thought it best to use the same packets. In the code I will provide, I will vary the "handle" if you wish so that we may easily show that the bad data being sent over the UART is stale.

In our application, we will send a message and then wait for a specific response. We will not send another request until we first receive a response from the first request. No delay is provided between the response receipt and the next transmit.

The XDS is not doing anything else, but we do have two tasks running in the OS.

We are plugged directly into a host and that host is running Windows 10 and is not running a VM

Here is a sample of a request and response

   sendln '{"optype": "call", "id": "bmap-write", "handle":0, "data": "1A05010400C03E20", "secure": '  SECURE_STR'}'
    waitregex '"event","id":"bmap","handle":0,"data":"1A0503202020112525252524252627000102035252525200000000000000140200000200"'

Hi Edward,

Data sent is about 94 bytes and data received on teraterm is about 125 bytes. (The Issue is when the received payload is this large, somewhere around one in 10000 transfers we see a malformed packet received on teraterm )

we send a request and wait for a response and do this in a loop on teraterm and its usually a few milliseconds between each transmission. If we receive a malformed packet from the TI dongle, the test on teraterm stops.

XDS110 is running normally not in a debug session, it is connected to USB hubs, No energy trace session.

And the teraterm software is running natively on the host system.

Anaurag

in your setup you are using a HUB, in my setup I am not.

Edward, also I do have Beagle USB traces of the issue

And at what baud rate has this been tested and shown to not work with the newer firmware?

Oh, and thanks again for providing the details needed to investigate this issue.

Issue can be produced at 9600 Baud or 921600

That's not good news.

At 921600, the code switches to using DMA to pump the incoming UART data into the USB buffers. That would likely have been the point of failure given what we've seen so far. 

But, at 9600, the DMA code is not enabled, and that blows up several theories I had working on what to check.

Is there any indication that messages being sent to the CC2640 are malformed?  Or is it only the data being returned?

Edward,

just the returned data is malformed

Anurag can you try 9600 again? Or have you already done that recently?

Hi Edward

I created code which would simply echo data send to the launchpad back to the testing PC via teraterm. I am writing/echoing a lot of data but well over 1million attempts resulted in zero corruptions.

I then went back to my original script (same LaunchPad) code and I did get UART corruption. That data is below. Please note that the data being sent and received is static but I cycle the HANDLES from 10 to 99 in order to discriminate varying packets. As you can see HANDLE:60 repeats and stomped on packet 61.

I could try varying the time from receive to re-transmit in the ECHO test case

Current Loop Number 21645
{"optype": "call", "id": "bmap-write", "handle":0, "data": "1A05010400C03E20", "secure": 0}
<{"optype":"status","id":"bmap-write","handle":0,"error":0}
<{"optype":"event","id":"bmap","handle":58,"data":"1A0503202020112525252524252627000102035252525200000000000000140200000200"}

Current Loop Number 21646
{"optype": "call", "id": "bmap-write", "handle":0, "data": "1A05010400C03E20", "secure": 0}
<{"optype":"status","id":"bmap-write","handle":0,"error":0}
<{"optype":"event","id":"bmap","handle":59,"data":"1A0503202020112525252524252627000102035252525200000000000000140200000200"}

Current Loop Number 21647
{"optype": "call", "id": "bmap-write", "handle":0, "data": "1A05010400C03E20", "secure": 0}
<{"optype":"status","id":"bmap-write","handle":0,"error":0}
<{"optype":"event","id":"bmap","handle":60,"data":"1A0503202020112525252524252627000102035252525200000000000000140200000200"}

Current Loop Number 21648
{"optype": "call", "id": "bmap-write", "handle":0, "data": "1A05010400C03E20", "secure": 0}
<{"optype":"status","id":"bmap-write","handle":0,"error":0}
<{"optype":"event","id":"bmap","handle":60,"data":"1A050320202011252525252425262700010203525252520000000000000014020000020"handle":61,"data":"1A0503202020112525252524252627000102035252525200000000000000140200000200"}

Hi Edward

I hope you are well. I was able to create code which does the echo of the data back to the host. Extended testing has not shown any UART corruption so I am trying to add some delay between the request and the retransmission

Without much though, I am using CPUdelay() to conduct this test. Presently I am up to 572 counts (which I think are machine cycles) of delay

Do you  have any other suggestions?

Did you ever get an answer on the re-test at 9600 baud?

Assuming this is a DMA issue, then it is most likely there's a race condition between the idle timer and the data coming in. The idle timer is to clear out any partial buffers when the transmission ends so that all data does get sent to the host. Otherwise, the DMA feature waits indefinitely until its buffer is completely filled (1K of data).

If there are no breaks in the incoming data, then there's no issue. The idle timer never times out, so it has no effect.

If the breaks are long compared the timer, then the idle timer can clear the buffer before new data comes in.

I suspect, but this is still just a guess, the delay is just enough that the idle routine is in the middle of cleaning up when new data arrives, and the process has a bug that causes the corrupted data.

We know that no breaks in the data and apparently a very small break (572 machine cycles wouldn't be very long) don't have a problem.

I would try to use a logic analyzer to examine the pattern of data, try to characterize how long the delay between incoming transmissions is, and then tweak the example to try to closely match that.

Edward

a retest of the 9600 Baud configuration showed no errors over the weekend. If I remember correctly I had enabled a mouse/move background task when doing the 9600 BAUD test which I believe caused a mouse entry into the teraterm command window I think that is what had happened during that testing. However, no mouse moves software is running under normal conditions when the error is seen at the higher baud rate.

I will resume testing with my echo code looking for a sweet spot for the delay. Also do you have any other recommendations?

I don't know what else to suggest to duplicate it on your end.  I am going to play around with the idle timer.  If I shorten it, a lot, then it may be possible to duplicate the issue easier here.  Assuming that's the problem.

I have a sample firmware I'd like for you to check out.  See if you can duplicate the same issue with this or not at 921600 baud. 

/cfs-file/__key/communityserver-discussions-components-files/538/firmware_5F00_3.0.0.9.bin

This copy of the firmware has the idle timer completely removed. The DMA code gathers a 1k buffer from the UART, and when that buffer is full dumps it to the USB connection to send to the host. But it always waits for a full buffer. The idle timer was designed so that after a pause in the data stream, we'd send the partial buffer to not lose the tail end of data. So the last bit of a continual test will probably not be sent, but all of the data up to that point should be good if the problem is the idle timer code.  Either way, this test could cut the potential problem area in half.