• Locked Locked
  • Replies 4 replies
  • Subscribers 98 subscribers
  • Views 957 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Help: BIOS hang problem in TSK_create() when linking FastRTS Library into DSP image

Serge Limondin
Intellectual 305 points


Hello,

I have been chasing a difficult problem that cause the DSP BIOS call TSK_create() to hang and never return, hanging the entire DSP application. This behavior is random and may not happen for hours, however i came up with a test case that repeatedly creates C++ object which in turn create 2 worker threads then delete the object and releases these two worker threads. The application runs forever with no issues (days) when the DSP Image is linked against the standard rts library (we use rts64plus.lib).

However when linking also first fastrts62x64x.lib, the test case witl invariably fail on the second iteration after having created about 20 threads (TSKs). SImply relinking the application without liking FastRTS and the problem disappears.

Here are the Platform and tools I am using:

DSPBIOS 5.41.06.21

CCS Studio v4.2.4.00033

CGT Tools 6.1.20

DaVinci DM6446 HW platform

Our application is 99% C++ with heavy use of templates. The particular code being optimized is a complex FPU based Machine Vision Algorithm (compiles to about 3 MBytes of object code optimized). Given that FastRTS accelerate this Machine Vision algorithm by a factor of 3x (amazing), I am not willing to give up just yet but I am running out of ideas.

Also please not that we developed our own IPC ARM/DSP communication scheme and are not using any of the Ti mechanism.

Our DSP Firmware is only dependent on the DSP BIOS and RTS.

I believe it might be a calling convention/linker/memory corruption of some sort but cannot prove it.

My attempts to recompile the FastRTS library with the CGT 6.1.20 also fails with the following error (eventhough I followed the FastRTS manual instructions to the letter).

$ mk6x fastrts62x64x.src -l fastrts62x64x.lib

Building archive "fastrts62x64x.src" --> "fastrts62x64x.lib"

"divdp.asm", ERROR!   at line 27: [E1000] Include/Copy file not found or opened

                .include divc30.asm     ; \

"divdp.asm", ERROR!   at line 28: [E1000] Include/Copy file not found or opened

                .include mpy62.asm      ;  > all 3 are called by "dpdiv1.asm"

"divdp.asm", ERROR!   at line 29: [E1000] Include/Copy file not found or opened

                .include flt64.asm      ; /

3 Assembly Errors, No Assembly Warnings

Errors in Source - Assembler Aborted

>> Compilation failure

>> Archive "fastrts62x64x.lib" not built

/cygdrive/c/dev/Earthbound/VS/JaguarDSP/App/TiLibs/FastRTS/lib

$ ls

adddp.obj  addsp.obj  divc30.obj  fastrts62x64x.src  fastrts64x-orig.lib  fastrts64x.lib  fastrts64xe.lib

/cygdrive/c/dev/Earthbound/VS/JaguarDSP/App/TiLibs/FastRTS/lib


Any help is very appreciated,

Kind Regards,

Serge Limondin
 
 
 

 

  • Intellectual 305 points

    I am trying to suspect that these assembly routine may not be setup correctly for the C++ compiler (read register allocation and usage) as our C++ code uses *exceptions* and *rtti*, and we have to use the plus RTS library for everything to link correctly.

    Just a thought ?

    Serge Limondin

     

  • TI__Mastermind 45025 points

    Hi Serge Limondin,

    Serge Limondin said:
    the DSP BIOS call TSK_create() to hang and never return, hanging the entire DSP application

    Where is the program counter at when you halt the DSP at this point?

    Is it possible that you have run out of heap?  TSK_create() must allocate memory dynamically in order to create the new Task at run time.

    Can you try using ROV to help?  You can see ROV in CCS by going to "tools->ROV" and it will give you information about DSP/BIOS (note that this tool is only available while the target is halted).

    Regarding your rebuild of the RTS library, I've brought this issue to the attention of the Code Generation Tools team; they should be able to help you with that part of it.

    Steve

  • in reply to Steven Connell
    Intellectual 305 points

    Hello Steve,

    I was using ROV before to check thet state of all the threads however since we are on the subject, How do I use ROV to see the stack traces of all the thread one at a time like is possible on the Visual Studio 2011 Debugger (by simply selecting the thread and bringing the stack trace window) ? Please note that I do not use any of the Ti tools in our DSP image (just BIOS and RTC), no XDC, DspLink, etc...None of this information is available to me in the ROV pane...

    However if the ROV TASK display shows the threads PC location then I will get you this information.

    New piece of data:

    After a long debugging binary search session today (I basicallyu unpacked by hand the FastRTS src archive and added each asm file to the project). I also made this Machine Vision algorhtm project a partial link project so I could control which FastRTS call I wanted to compile and link in.

    Well the culprit of the TASK_create() hang is divsp.asm and/or divdp.asm, if i exclude these two APIs from tbeing linked in, the problem disappears...

    So i have a few new questions for you now:

    1) Is it possible that these FastRTS div calls are preventing context switching (i.e. they are not interruptible) which might explain the randomness of the condition.

    2) It is not a memory issue as I have checked this and the Heap still has 50MB free when the condition occurs.

    3) Is it possible that the div APIs mock with HWI and SWI interrupts, hanging the scheduler in the BIOS ?

    4) Is the BIOS we use compatible with FastRTS? which BIOS was FastRTS validated on ?

    Considering the FastRTS div APIs are the most complex with multiple pages of assembler I could really use some help from the developer of these APIs and the assumptions he might have made.

    Regards,

    Serge Limondin

     

  • in reply to Serge Limondin
    TI__Mastermind 45025 points

    Serge Limondin,

    Unfortunately I'm not familiar with the FastRTS ... however I have forwarded your question to our CGTools team and they should be able to give you some more information on FastRTS soon.

    Regarding ROV, there is no stack trace available, unfortunately.  I looked up a previous post from you and I saw that I had already mentioned the stack trace that shows up within CCS itself, but sounds like this is not good enough.

    So what can you do to trace backwards?  One method that is used often is to backtrace using the B3 register.  When a new function is entered, the return address of where that function was called from is stored into the B3 register.  If you halt your program at the error point, you will be able to see where it is at in the dis/assembly view.

    If you then type "B3" into the address pane of the dis/assembly window and hit enter, it will take you to the instruction that made the call into the current routine.  You can put a break point there, and then run again until you hit the break point (assuming that the break point will hit just before the error condition).  Once the break point is reached, again use B3 to trace backwards.

    Using this method, you can "un-wind" to see how you ended up where the PC is when the error occurs.

    Steve