Fix needed for RTDX failure - seems to depend on data volume

Hi

I'm trying to use RTDX to capture sample channels into, between, and out of two DSP algorithms on our own TMS320VC5506 target system.  The sample rate is 8kHz and I can choose at build time how many channels to log (1 to 5).  It all works, up to a point in time, then fails.  The time it runs for before failiing seems dependent on the number of channels, i.e. the volume of the data going across the interface:

The DSP algorithms are called every 10ms i.e. on 80 samples of data.  Logging code can (via conditional build macros) be setup to log any of five channels (3 inputs, intermediate buffer, and output) channels to RTDX, also called in the same function as the DSP algorithms, every 10ms.
 
Logging 5 channels is 40,000 16-bit samples per second = 640,000 bits per second.  This consistently goes wrong after 11.4 seconds;

4 channels, 32,000 SPS, 512,000 bps, goes wrong after 14.8s;
3 channels, 24,000 SPS, 384,000 bps, 26.6s;
2 channels, 16,000 SPS, 256,000 bps, 37s;
1 channel,   8,000 SPS, 128,000 bps, no RTDX problems seen, up to the duration of the test (120s).

I’ve added timing code which makes the a GPIO output high, to show how long the logging calls take: Best case is 1 channel (0.48ms, i.e. 4.8% CPU) and 2.3ms (23% CPU) for 5 channels.  The output stays high in many ~155ms-long intervals when the logging breaks, so the DSP is waiting and is therefore aware of the hold-up in the logging system.

I have found two relevant settings in Code Composer Studio: Buffer Size (default 1024 bytes), and Number of Buffers (default 4).  Increasing these to 10000 and 64 made no difference.  The help says that the buffer size has to be bigger than the biggest message which, as my messages are 160 bytes, it is, at 1024 bytes.

CCS RTDX config claims that the JTAG link has a theoretical data rate of 6966.66kbytes/sec.  This is well above my maximum of 80k bytes/sec.

RTDX can be disabled from CCS even though the RTDX write code is still present and running in the target. In this case an identical 4.8% - 23% CPU usage remains (as seen on the GPIO output), but nothing fails (and, of course, no data is transferred!).

I’m wondering if anything in the PC (setups?) or a faster PC would improve the transfer speed to hard disk, if that’s the problem.  Looking at the RTD file size in explorer (keep pressing F5), it grows quite slowly (10 to 20 kB/sec) and keeps growing a long time after the logging has stopped (I realised that I should not halt the DSP until the file has stopped growing, or data is lost).  This reinforces my belief that something at the PC end is to blame for the data-size-related failure times.

Cheers

Gavin

 

  • Hi Gavin,

     

    Unfortunately RTDX is no longer supported so any fixes needed for it will likely not occur. But I have moved this thread into the BIOS forum where there is more expertise with RTDX in case there is a workaround for your problem.

    Thanks

    ki

  • Gavin,

    I didn't see the theoretical speed that you were quoting when I looked at my CCS3.3 install, but the number seems high.  You should only expect about 20 kbytes/sec on a C5500 DSP using the standard JTAG interface.  There was a Hi-Speed version of RTDX that was experimented with on the C6000 DSPs which used the EMU0/1 pins for addition BW.  It turned out to not be reliable for some reason.  Even with the extra bandwidth you could only expect maybe 50 Kbytes/sec.  This is nowhere near the 7 Mbytes/sec that you are quoting.

    Regards.

  • In reply to TommyG:

    Tommy,

    Many thanks for the quick reply. 

    I tried to attach a screen dump from CCS which shows the 6966.66 kbytes/sec in the RTDX Configuration Control Properties -> Port Configuration Page, but I'm not sure how.  If you let me know your email address I could send it.

    CCS scrren dump showing RTDX data rate (JPG)

    Anyway, the 20k bytes/sec rate fits with my findings. It also means that there must be a lot (3.5 to 6Mbits) of RAM buffering somewhere on the target (that I wasn't aware of - it only has the TMS320VC5506ZHHR and a 512kB flash) or in the Black Hawk USB560 JTAG pod.  I suspect the latter.  

    The buffering being in the JTAG pod rather than the target suggests that the 20 kbytes/sec bottleneck is in the USB link between the pod and the PC, rather than in the JTAG link between the target and the pod.  This is particularly disappointing, as even USB1 is capable of a much higher data rate than I need, let alone USB2, which I believe is what the USB560 pod has. 

    I just tried changing the JTAG TCLK frequency in the CCS Setup from "Automatic with 10.368MHz limit" to "Automatic with faster 35.0 MHz limit".  This has, strangely, reduced the "Theoretical Data Rate" in the RTDX config proprerties mentioned above, from 6966.66 to 5576.76 kbytes/sec.  The time I can log for without crashing has not changed though.

    I conclude that the only way to make RTDX do what I want is to get a JTAG pod with a (5x) bigger buffer.  I don't expect this will be available.  But, if you know different...

    Thanks again for your help with this. 

    All the best

    Gavin

  • In reply to Gavin Withers:

    Gavin,

    The BW limitation in RTDX is the TDI/TDO data link for the JTAG connection with the target board.  RTDX is an Idle thread task for BIOS that only runs if no other higher priority tasks are running.  If you have other RTA tools enabled, then they share the channel with RTDX. 

    The onchip memory of the DSP is used for the buffer on the target side.  The buffer configuration you mentioned originally is reserving target memory.  The data is streamed directly to the host PC when data is transfered from target to host.  There is no buffering in the JTAG pod.

    If you really need to transfer at a higher BW, then you might want to look at one of the other peripherals on the device and develop a custom interface / software to talk to the PC.  The USB client possibly has the BW you need.  If you use one of the DSP peripherals, then your application would have to share resources with the data transfer and would probably be more intrusive.  Not sure how much of the DSP resources you have left over.  That's the only thing I can think of at the moment.  Maybe someone else has some better ideas.

    Regards.

  • In reply to TommyG:

    Tommy,

    I'm afraid I cannot modify the hardware so adding a new interface is not practical.  This is an existing product, thousands of which are in the field, and I need to sort these algiorithms soon so we can send out a software upgrade.

    I can just about get by if I log three out of the five channels (only looking at one of the two DSP algorithms at a time - two inputs, one output).  This works for about 26 seconds, which I will have to live with.  

    It's only of academic interest now, but the buffering figures don't add up: There is 128kB on the DSP, not all of which is available for buffering, also the CCS buffer settings, even at their tiny defaults of 1024 bytes and 4 buffers, make no difference to the logging time before crashing.  If the link goes at 20kB/sec, then with three channels being logged, the input data rate is 2 * 8ksps * 3 = 48kB/sec, so the buffer fills up at 28kB/sec.  As I get 26 seconds before a crash, that means there must be 28 * 26 = 728kBytes of buffer available somewhere, on a DSP with 128K bytes of RAM? 

    Anyway, thanks again for your prompt help with this, and if we design a new Ti-based DSP board, I'll suggest a dedidcated high-speed logging interface!

    All the best

    Gavin

     

  • In reply to Gavin Withers:

    Gavin,

    I happened to be able to get in touch with one of the original RTDX target side SW developers and asked him about what you are seeing.  He didn't think that RTDX is causing the problem you are reporting.  If there is a buffer overflow problem on either the host or the target all that should happen is that the RTDX call should return with a FAIL status.  Here are his words:

    First there are two buffers of which he has to be aware

    ·         Target buffer

    ·         Host side buffer

     When he configures the host side buffers he does so by going into the RTDX plugin and sets up the buffer sizes for the *host* side only. There is a default buffer size for the target but he can adjust it by editing the file *buf.c* that is released with the RTDX target libraries. The reason we did this is so that the user could adjust the target side buffer based on the amount of memory on the target. If they include this file in their build, the linker will pick up this file and use it rather than the default that is in the rtdx.lib.

    The best way to isolate the problem is to *not* use BIOS but to merely use RTDX channels to see if the problem still occurs. He may already be doing that but I think he is getting confused with the host side buffers and target side buffers. Recall that RTDX is an *asynchronous* protocol and so if data is not consumed then on the host side writes and reads will return a *FAIL* status. There should not be a crash. Similarly when you do an *RTDX_write* in the target if there is no buffer space to store the data from the write this API will return a *FAIL* status.

    So I misspoke when I said the that CCS buffer configuration was for the target.  As stated above, it is for the host.  The default size on the target looks to be 256 words.  Are you doing any error checking on your RTDX data transfers?

    Regards.

  • In reply to TommyG:

    Hi Tommy (and David Friedland?)

    This sounds promising.  Perhaps I should explain more about the problems that I would like to solve:

    1) How to increase the amout of data that I can transfer in one minute?  I would like to transfer data at least 48,000 bytes/sec for at least one minute.  The best I have achieved so far is 48,000 bytes/sec for between 22 and 26 seconds.

    2) When RTDX goes wrong (I previsouly used the word crash to describe this) , I would like it to go wrong in such a way that it does not break the target.  What hapens is, the time spent in the RTDX_write() and while(RTDX_writing != NULL) code drastically increases, from ~2ms to over 100ms. As this code is in a thread which services the DSP algorithms, CODEC read/write etc., actions in this and other threads are not completed and cannnot recover.  I would really like to be able to anticipate or trap the write holdup and stop RTDX before this happens, so the target application keeps running. 

    I can try reading the return value from RTDX_write() and stop RTDX immediately if it returns a FAIL status.

    3) If the data volume cannot be increassed and we can sort out the "orderly" halt to RTDX when the buffers are full (problem 2), then I would like to be able to choose (offset) when the logging starts, so I can inspect the system behaviour at critcial points of time via a 22 - 26 second window.

    Regarding DSP/BIOS, the system uses BIOS for scheduling, pipes etc. so I don't know how to avoid it for RTDX.  The situation is complicated by the fact that I cannot configure BIOS in a GUI as there is no project file.  BIOS is configured from several text files and I am lucky that the original software designer included RTDX within the BIOS setup. I have not changed anything in the BIOS setup regarding this problem. 

    I have no "buf.c" file in my build.  Please tell me more about this, if adding it would improve the situation.  As I've said before, the maths makes me think that some of the buffering is in the JTAG pod as well as in the target DSP (which has less than 128kB of RAM available). 

    Can you explain exactly what the "size of buffers" and "number of buffers" settings do?  Is the "number of buffers" related to number of RTDX channels, or can several buffers apply to one channel, or more than one channel to one buffer?

    There is no automated error checking of the RTDX data, but it is obvious from the contents of the log (which I convert to WAV files, play back and listen to) that RTDX has gone wrong, and when it went wrong.

    Let me know your thoughts and ideas.

    Best Regards

    Gavin

  • In reply to Gavin Withers:

    Gavin,

    I'm surprised that you are getting as high an RTDX output as you are seeing.  There isn't really anything you can do to increase the RTDX output.  It is limited by the JTAG interface throughput and the application running on your target.

    Since RTDX runs in the IDL loop of BIOS, it is the lowest priority task in the system.  It was designed this way to have minimale (if any) impact on your main application.  I believe that what you are seeing when RTDX_write() time increases is a symptom, not a cause.  Something else in your system is taking more processing bandwidth so the BIOS scheduler sends the system back to the IDL thread less frequently. 

    Do you have RTDX installed in your system?  The rtdx_buf.c file should be in the folder \rtdx\lib.  You would need to add it manually to your project, then modify as you see fit.  Remember the target side RTDX uses that same DSP memory that your application does, so make sure there is enough applcation memory for you.

    I'm not sure what the impact of buffer config on host side other than 4 is the minimum.  The larger the buffer the more time the host has to process the information coming from the target.  I suspect that you need 4 due to double buffering the reads and writes.  However, the host side configuration shouldn't have any impact on target side performance.