C674x internal resource conflict exception

I would suspect program memory is being overwritten somehow, especially if you know this code is being executed correctly many times before the crash.  However, if this is the first time this code is being executed, and it fails on the first time through, the only thing I can think of is sin and cos have instructions that write registers after the function has returned.  This would be unlikely however, as the compiler does not allow instructions to write after the function in which those instructions are located has returned.   i.e. the compiler would make sure all instructions write their results before cos returned to its caller.

Hi Todd,

Thanks for your response. I did a quick diff between the pre-crash and post-crash dissassembly near the exception point, but didn't see any differences. Is there an easy way to verify the state of the program memory after the exception with the .out file in CCS? Could there be a issue with the cache?

I've been trying to find all the potential causes for a resource conflict exception in the documentation, but haven't found a complete list. Do you know what these are? This might help us pinpoint the cause.

Also, the dsplink default makefile has sploop disabled. Since we didn't know why, we left it in. However, we're linking the pre-compiled fastmath library, which may or may not use sploop. Is a possibility for issues here? 

Eric

Look also in the sin and cos functions for program memory changes.

There isn't a documented list, but it is basically things like two different latency instructions trying to use a functional unit's read or write port to the register in the same cycle.  Or, in the case of program memory corruption, you could have two instructions trying to operate on the same functional unit.

Should be no issues w/ SPLOOP and fastmath rts.

Hi Todd,

I did a diff between the entire contents of the .text section and the .bios sections in memory pre and post crash and didn't see any differences between them. Are there any other places I should look? Could there be any cache issues?

I also tried some experiments with SPLOOP and didn't find any effects, as you predicted.

We've also tried replacing all of the sindp and cosdp with single precision operations and so far haven't seen any crashes. We're not convinced that this is a solution to the problem, but this maybe a clue as the underlying issue. Any thought about this?

Thanks for your help!

Eric

Eric,

Thanks for reporting the issue with the assembly version of the fastmath trignometric functions. We are looking at resolving this issue and will try to fix this in the next release. In the meantime if you have not been able to fix this issue yet, I would recommend using the intrinsic C version of the sindp, cosdp and atandp functions included in the library. You can find the intrinics versions in the source provided with the library. The C intrinsic version of the same API adds a suffix _c to the original API. These functions will provide you almost the same performance as the assembly version but will not have the issue that you see with the hand assembly. Here are the performance estimates

8561.c67xfastRTS_Benchmarking.pdf

Unfortunately the issue that you are reporting did not show up in our test bench which is also provided in the library. Please let us know if this resolves the issue you are facing and let us know if the performance you get from intrinsic C version of the API suffices your application requirement.

Regards,

Rahul

For the next person to find this issue, we (Eric and I) tracked this down to the small answer return path of the sindp and cosdp functions in the fastmath library (they are identical with a few extra lines for the cosdp function).  The m1 unit isn't allowed enough delay before the return branch and is still writing to register A7 on the first operation after returning.  If this next operation writes to A7, an exception is issued.  Here's the code from sindp starting at the mpydp instruction:

LTpi:                                           ; label if no arg. reduction

        mpydp   .m1     A5:A4, A5:A4, A7:A6     ; F = A*A

||      lddw    .d2     *+B2[3], B5:B4          ; T = r8

||      extu    .s1     A5,1,21,A2              ; reduced exp

||      mvkl    .s2     1023-20,B1              ; MIN exp

        cmplt   .l1x    A2,B1,A2                ; V = true for small exp

||      extu    .s1     A1,31,0,A1              ; move sign to bit 31

        lddw    .d2     *+B2[4], A1:A0          ; S = r7

||      mv      .l2x    A1,B0                   ; sign = bit 31 position

   [A2] b       .s2     b3                      ; return small answer exit

|| [A2] or      .l1x    A5,B0,A5                ; set sign bit for small ans

        nop             5                       ; wait for F (or take b b3)

A single nop between lddw || mv and the branch call should be sufficient to fix the vulnerability.  As for why the multiply in conjunction with the sindp call is at fault, it may have something to do with code that compiles into using the A7 register on the return.  We use sindp everywhere in our code, but only had issues when doing double precision multiplies on the results.

Hope this helps,

Geoff

Hi Rahul,

Thanks for the suggestion. We're linking the fastmath library as a replacement for the standard RTS rather than directly calling the fastmath API function names. Is there a simple way to rebuild the library so that the linked functions use the intrinsic versions rather than the asm versions? I didn't see any obvious switches in the make files.

Eric

Eric,

The fastmath library does not archive the intrinsic version in the library so there is no apparent switch in the library. The easiest way to use them is to include the cosdp_c.c, sindp_c.c and atandp_c.c as source files to your project and then scan for the trigonometric functions(sin,cos,atan) in your code and replace them with the intrinsic APIs (cosdp_c,sindp_c, atandp_c).

But if you still wish to use the library, then you might have to remove the asm files from the library build project (given in the build folder ) and rebuild the library but that would still require you to rename the APIs in your code with the _c version of the RTS functions.

Best Regards,

Rahul

Hi Rahul,

Because our C code is generated from matlab code, we'd like to avoid using the C calls. I'd like to attempt to rebuild the fastmath library so that the intrinsic functions get called at the linker step. I've already modified the make file to build and archive the correct files, but I need to tell the linker to override the rts functions. What is the best way to do the equivalent of the following asm code for the C functions? 

        .if     __TI_EABI__

           .asg    divdp, _divdp

        .endif

        .global _divdp   ; entry labels

        .if (OVERRIDE_RTS = 1)

        .global __divd

        .endif

Eric,

You don`t need to worry about the __TI_EABI__ symbol as that is a symbol used only when you are building ELF binaries. Now in order to override the RTS functions, I think there are two ways to do this

Method 1: (Minimum modifications to source)

Easiest way is to create a header and use #define to replace RTS function calls to intrinsic function calls.

For Eg: #define cos(x)  cosdp_c(x)

Include the header in your source, to override the RTS functions 

Important: Inorder for this to work you will have to use --gcc option with the C6000 compiler.

If you are working in the CCS environment this option is available under Properties->C/C++Build->Settings->Compiler->Language Options

Method 2: (No modification to your source code but change in source of the intrinsic functions)

Rename the intrinsic functions to match the RTS function names/symbols in their source file before rebuilding library and continue to link the library the way you have been doing earlier (prelink the fastMath library before the RTS library)

I have not tried these solutions myself but I am reasonably confident this will work. Please let us know how this goes.

Rahul

Thanks Rahul,

I ended up doing Method 2 last night and that seemed to work smoothly. I'm in the process of testing our system with the intrinsic c library to see if there are any problems.

We're assuming that the C code will be more reliable than the hand asm code since there's less chance of subtle errors. However, we're a little worried that these functions haven't been vetted as much since most people probably just link the asm functions. How much testing has the intrinsic library gone through? Can we be confident that the library is safe? Given the problems that we've had so far, we're a little paranoid at the moment.

One more question: given the significant increase in performance for the fastmath library, why isn't this code used in the default RTS? 

Also, there's an error in line 142 of the c67xmathlib/build/Makefile: DIVDP should be DIVSP.

Thanks,

Eric

Eric,

The testing for each of the intrinsic functions in fastMath was done by creating a stand alone single thread applications where these functions are checked for accuracy and performance over 100 runs. In terms of accuracy the optimized C and assembly versions are bit exact and I have provided you the difference in perfromance estimates in an earlier post. 

Since these functions can also be use in an ARM+DSP implementation, we have also checked calling of these functions from the ARM side over the dsplink in a cache enabled implementation on the DSP where data comes from the external memory. It is unfortunate that you had to work through this issue but there are several other active developers who are working with this library and this is the first major issue that was reported with the library so far.

The fastMath functions removes some control code and checks from the RTS functions to improve efficiency, rewrites some parts of the code to allow the code to be pipelined and cuts down on the terms in the expansion series of some functions as an efficient tradeoff between accuracy and performance. Also,you may have seen not all the RTS functions are rewritten in this library so it continues to make sense to provide the normal RTS library with the full functionality and to provide a subset of optimized functions that are generally used to math intensive real time applications.

Thanks for your patience in dealing with this issue. Looking forward to hearing from you after you complete your testing.

Best Regards,

Rahul

Eric Chin said:

One more question: given the significant increase in performance for the fastmath library, why isn't this code used in the default RTS? 

It is my understanding (correct me if I am wrong Rahul) that the fastMath library does not handle certain exceptional cases in some instances.  e.g. Overflow, NaN, etc.  It is also less precise in some/many cases.

I have spent a long time trying to work out why a stand alone test program that a colleague wrote ran successfully, whilst when I ported it to run in a DSP/BIOS thread, I would get exceptions.

I held back from posting to the forum for some time as I did not want my first post to be a cry wolf!

http://e2e.ti.com/support/dsp/tms320c6000_high_performance_dsps/f/115/p/133991/482799.aspx#482799

In my case the exceptions occurred when sindp was called with an argument of zero, and the result was passed to another function.

For example:

x = sqrt(sin(y));

would cause the NMI HWI exception if y was zero.

I found that breaking the call down fixed the problem:

x = sin(y);
x = sqrt(x);

does not cause the excepetion when y is zero.

I am assuming that the compiler assigns registers differently in a DSP/BIOS environment, as the stand alone non-DSP/BIOS version did not have any apparent issues.

I have just rebuilt the c67xmathlib_2_01_00_00 library using the suggested changes courtesy of Geoff and Eric.

(I added a single nop to sindp.asm and cosdp.asm.)

I now have working code.

I am aware that I may encounter other bugs in the library if my code just happens to use the wrong combination of data at runtime.

For anyone following this post with the same issue wondering whether to edit the source code or edit the library, I would like to say that rebuilding the library was trivial.
It will take you approximately 5 minutes to rebuild the library and then update your own project.
In your own project it does not involve editing any header files or source files. E.g. no search and replace.

On the down side, there may be other bugs in the library that are missed - so use caution making your decision.

In CCS:

• select Project->Import Existing,
• browse to the build folder in the c67xmathlib_2_01_00_00 library on your harddisc
• import the project,
• edit the files (see Geoff's post above - 09-01-2011 5:07 PM)
  
• build the project.

Note that the resulting library is placed in the folder C:\c67xmathlib_2_01_00_00\build\C674x and is called c674xfastMath_rebuild.lib

Hence you need to modify your own project(s) to link in the rebuilt library:

Open the Build Properties -> C/C++ Build->C6000 Linker -> File Search Path

• locate the entry for the original library (by default this would be "C:\c67xmathlib_2_01_00_00\lib\c674xfastMath.lib")
• replace it with the new path and name (by default "C:\c67xmathlib_2_01_00_00\build\C674x\c674xfastMath_rebuild.lib")

If you are using DSP/BIOS you will (no doubt) have added the library into the Build Properties -> CCS Build->Link Order tab.
(If you have not done this then you will not be using the library correctly as DSP/BIOS inserts the standard rts library into the linker command file prior to any user specified libraries.)
So you will need to check that that is also updated.

I would like to take this opportunity to thank Geoff and Eric for posting the assembler fix.

Much appreciated.

Paul

For anybody following this post,

We tested the C intrinsic functions from the fastmath library as a replacement for the assembly version and found a number of serious problems, particularly around the edge cases. We've contacted TI regarding these issues, so hopefully there will be an updated library soon. In the mean time, I do not recommend using the C intrinsic or vectorized functions in the fastmath api. 

Eric