I have an intermittent problem with C++ exception handling when I use multiple tasks ("threads" in normal parlance).
The problem is intermittent, but I can reliably reproduce it with the code below. I am using CCS 3.1 with cgtools 5.1.12, and I have enabled exception handling with the --exceptions command line switch and throw exceptions from time to time. Usually, everything is fine. Every now and then I get a message in red in the Stdout pane:
Assertion failed in file "EDGRTS/throw.cpp", line 1312
(or some other line, but they all trace to the same function __throw() in the source file given). I looked through the RTS source for this file in the cgtools/lib directory, and the failed assertion occurs on this line:
check_assertion(__curr_eh_stack_entry == &curr_throw_stack_entry->throw_marker);
The code in that RTS file is rather complex, but you can see that this has something to do with the stack not matching the expected thrown-exception somehow. (FWIW, I notice that there is no substantive change to throw.cpp in cgtools 6.1.5, which I have access to on a different computer.) Can anyone give me any hints as to what this error could mean?
Below is a simple program the demonstrates the problem and, before that, an explanation of what it does. I am running on a TMS320C6713 processor, and I am using CCS 3.1.23, cgtools 5.1.12, DSP/BIOS 4.90, and IDE 5.90.100.9. I used the default DSP/BIOS configuration that is provided when I do "File | New | DSP/BIOS Configuration ..." for my processor, save that I edit it manually to link rts6700_eh.lib instead of rts6700.lib. An explanation of the program: main() creates 8 tasks of several different priorities. Each task runs the same function, Task(), which sits in an infinite loop and throws and catches exceptions. It makes calls into the standard library functions printf() and rand(), and because those two function maintain internal state (e.g., buffering and the current position in the pseudo-random sequence), I use DSP/BIOS semaphores, wrapped up in some non-throwing C++ helper classes, to synchronize access to them. Each task sometimes throws and catches an exception based on a value of rand(), and then each task sleeps for a random amount of time. All this pseudo-randomness is intended to recreate the indeterminacy in execution order of my production code, which is exhibiting this same problem. The higher priority tasks usually sleep longer in order not to starve the lower priority tasks. If you run the program, you'll see it humming along, throwing and catching exceptions like mad in various threads. After a few minutes of this the RTS assertion in EDGRTS/throw.cpp that I mentioned below shows up for no obvious reason and the program aborts. I have checked for stack overflows on the kernel and the tasks in the kernel object viewer, but none of them are even close to their respective limits. One other hypothesis is that the exception handling is not thread-safe. Can anyone give me any insight into this problem? Is there any work-around other than removing exception handling (not an easy option for our production code!)? Is the exception handling thread-safe?
TIA! --M
Here's the code:
#include <std.h> #include <tsk.h> #include <exception> // for exception #include <cstdio> // for printf() #include <cstdlib> // for rand() #include <new> // for bad_alloc using namespace std; // All synchronization operations are no-throw class Semaphore { public: Semaphore( const Int cnt ) : m_sem( SEM_create( cnt, NULL ) ) {} ~Semaphore() { SEM_delete( m_sem ); } void Reset( Int count=0 ) { SEM_reset( m_sem, count ); } void Post() { SEM_post( m_sem ); } void iPost() { SEM_ipost( m_sem ); } bool Pend( const Uns timeout = SYS_FOREVER ) { return SEM_pend( m_sem, timeout ); } Int GetCount() const { return SEM_count( m_sem ); } // Helper class to automatically lock and unlock the semaphore class ScopedLock { public: ScopedLock( Semaphore& sem ) : m_sem( &sem ) { (void) m_sem->Pend(); } ~ScopedLock() { m_sem->Post(); } private: Semaphore* const m_sem; ScopedLock( const ScopedLock& ); ScopedLock& operator=( const ScopedLock& ); }; private: // Data members SEM_Handle m_sem; // Disable copying Semaphore( const Semaphore& ); Semaphore& operator=( const Semaphore& ); }; class Mutex { public: Mutex() : m_sem( 1 ) {} void Lock() { (void) m_sem.Pend(); } void Unlock() { m_sem.Post(); } // Helper class to automatically lock and unlock the semaphore class ScopedLock { public: ScopedLock( Mutex& mutex ) : m_mutex( &mutex ) { m_mutex->Lock(); } ~ScopedLock() { m_mutex->Unlock(); } private: Mutex* const m_mutex; ScopedLock( const ScopedLock& ); ScopedLock& operator=( const ScopedLock& ); }; private: // Data members Semaphore m_sem; // Disable copying Mutex( const Mutex& ); Mutex& operator=( const Mutex& ); }; static Mutex s_randMutex; static Mutex s_ioMutex; int Rand() { Mutex::ScopedLock lock( s_randMutex ); return rand() * rand(); } void Printf( const char* const msg, int id, const char* const emsg = 0 ) { // Protect i/o Mutex::ScopedLock lock( s_ioMutex ); if( emsg ) { printf( msg, id, emsg ); } else { printf( msg, id ); } } struct MyException : exception { const char* const m_msg; MyException( const char* const msg ) : m_msg( msg ) {} virtual const char* what() const throw() { return m_msg; } }; extern "C" Void Task() { const TSK_Handle self = TSK_self(); const int priority = TSK_getpri( self ); const int id = int(self) | (priority << 20); Printf( "Task %x starting\n", id ); for( ;; ) { try { if( 0 == Rand() % 10 ) { Printf( "Throwing in (%x)\n", id ); throw MyException( "D'oh!" ); } } catch( const exception& e ) { Printf( "Exception (%x): %s\n", id, e.what() ); } catch( const bad_alloc& ) { Printf( "Exception (%x): bad_alloc\n", id ); } catch( ... ) { Printf( "Exception (%x): unknown\n", id ); } TSK_sleep( Rand() % (2*priority) ); } } const int N_TASKS = 8; TSK_Handle s_tasks[ N_TASKS ]; int main() { for( int n=0; n < N_TASKS; ++n ) { if( s_tasks[ n ] = TSK_create( (Fxn)&Task, 0, n ) ) { TSK_setpri( s_tasks[ n ], 1 + n / 2 ); } else { printf( "Unable to create task %d\n", n ); } } printf( "Main finished: %d tasks created.\n", N_TASKS ); }
Wow! This is quite a lot to go through. I'm by no means an expert when it comes to exception handling, so I may have to defer to someone a little more knowledgable. I will at the very least check around to see if there's any reason exception handling is not considered thread-safe.*edit* I am sure this is the first thing you did, but please make sure that _all_ files are built with the --exceptions option per section 6.5 in the Compiler Guide.
-Tim
---------------------------------------------------------------------------------------------------------Please click the Verify Answer button on this post if it answers your question.---------------------------------------------------------------------------------------------------------
FYI, there's an article on the eXpressDSP Wiki that may shine some light on the reason for the ambiguity of your error message.
Thanks for the quick reply!
Indeed, I did do that. In this test program, there is only one source file, though there are some files (a .c and a .s62 file) generated by the DSP/BIOS. To handle the .c file, I had to specify the -fg option to compile C files as C++, but that doesn't address the assembly file, though that is basically just some data, int vectors, and program boot code. Do you think this could matter? It doesn't seem likely to me.
M
That link doesn't work for me. I see the main page for that Wiki (http://tiexpressdsp.com/wiki/index.php?title=Main_Page), but I'm not clear on what you were pointing me toward.
The documentation seems to specify that only C++ files need to be built with this tag (because C and Assembly don't support EH). I do not think not having that tag for those files would make a difference.
mlimberThat link doesn't work for me. I see the main page for that Wiki (http://tiexpressdsp.com/wiki/index.php?title=Main_Page), but I'm not clear on what you were pointing me toward.
TimHarron I'm by no means an expert when it comes to exception handling, so I may have to defer to someone a little more knowledgable. I will at the very least check around to see if there's any reason exception handling is not considered thread-safe.
I'm by no means an expert when it comes to exception handling, so I may have to defer to someone a little more knowledgable. I will at the very least check around to see if there's any reason exception handling is not considered thread-safe.
Ok, Tim. I'll wait for a reply at least about thread-safety.
Many thanks!
An update: It appears to me that the exception handling code is not properly destroying exception objects. (This may be a symptom of a deeper problem, but it may just as well be the problem itself.)
Here's what I did. I lock a global mutex in my exception class's constructors and unlock it in the destructor. This means that only one MyException object can ever exist at a time, and in fact the program would deadlock if it ever tried to copy the exception object.** I also added print statements there to tell me when an exception object is created or destroyed:
Mutex g_ehMutex;struct MyException : std::exception{ MyException( const char* const msg ) : m_msg( msg ) { g_ehMutex.Lock(); Printf( " Create %x\n", (int)TSK_self() ); } MyException( const MyException& e ) : m_msg( e.m_msg ) { g_ehMutex.Lock(); Printf( " Copy %x\n", (int)TSK_self() ); } ~MyException() throw() { g_ehMutex.Unlock(); Printf( " Destroy %x\n", (int)TSK_self() ); } virtual const char* what() const throw() { return m_msg; }private: const char* m_msg; MyException& operator=( const MyException& );};
The output of my program looks like this:
Main finished: 4 tasks created.Task 213d2c startingTask 2138cc startingTask 1145ec startingTask 11418c startingThrowing in (213d2c) Create 13d2cCatching in (213d2c): D'oh! Destroy 13d2cThrowing in (1145ec) Create 145ecThrowing in (2138cc)Catching in (1145ec): D'oh! Create 138ccAssertion failed in file "EDGRTS/throw.cpp", line 1126
Notice the correctly paired Create and Destroy messages followed by two Create messages, which should be impossible (unless ~MyException() is not being called even though the exception handler has completed operations), followed by the internal RTS error. I haven't been able to reproduce this in a single-threaded environment, which adds evidence to my theory about exceptions and multithreading.
Any suggestions, hints, etc.?
**C++ savants might notice that this mutex makes MyException uncopyable, though exceptions must be copyable according to the C++ standard. I realize this, and in fact initially implemented this experiment with a recursive mutex that does not have this issue. Using print statements in the exeption class's copy constructor with no mutex at all, however, shows that at least for this puny example, the copy constructor isn't called anyway. So simplifying down to a non-recursive mutex should work fine. Actually, the recursive mutex class experienced the same root problem: namely, the destructor of MyException is not always invoked, so the mutex isn't unlocked for each of its recursions (in this particular case, there is no actual recursion anyway since the copy constructor is never called, so it should only lock and unlock once) and then only one thread can ever lock the mutex to create an exception.
Here's the complete updated source code (mostly unchanged except for the MyException class as noted above):
#include <std.h>#include <tsk.h>#include <exception> // for exception#include <cstdio> // for printf()#include <cstdlib> // for rand()#include <cassert>#include <new>class Semaphore{public: Semaphore( const Int cnt ) : m_sem( SEM_create( cnt, NULL ) ) {} ~Semaphore() { SEM_delete( m_sem ); } void Reset( Int count=0 ) { SEM_reset( m_sem, count ); } void Post() { SEM_post( m_sem ); } void iPost() { SEM_ipost( m_sem ); } bool Pend( const Uns timeout = SYS_FOREVER ) { return SEM_pend( m_sem, timeout ); } Int GetCount() const { return SEM_count( m_sem ); } // Helper class to automatically lock and unlock the semaphore class ScopedLock { public: ScopedLock( Semaphore& sem ) : m_sem( &sem ) { (void) m_sem->Pend(); } ~ScopedLock() { m_sem->Post(); } private: Semaphore* const m_sem; ScopedLock( const ScopedLock& ); ScopedLock& operator=( const ScopedLock& ); };private: // Data members SEM_Handle m_sem; // Disable copying Semaphore( const Semaphore& ); Semaphore& operator=( const Semaphore& );};class Mutex{public: Mutex() : m_sem( 1 ) {} void Lock() { (void) m_sem.Pend(); } void Unlock() { m_sem.Post(); } // Helper class to automatically lock and unlock the semaphore class ScopedLock { public: ScopedLock( Mutex& mutex ) : m_mutex( &mutex ) { m_mutex->Lock(); } ~ScopedLock() { m_mutex->Unlock(); } private: Mutex* const m_mutex; ScopedLock( const ScopedLock& ); ScopedLock& operator=( const ScopedLock& ); };private: // Data members Semaphore m_sem; // Disable copying Mutex( const Mutex& ); Mutex& operator=( const Mutex& );};namespace{ Mutex s_randMutex; Mutex s_ioMutex;}int Rand(){ Mutex::ScopedLock lock( s_randMutex ); return std::rand() * std::rand();}void Printf( const char* const msg, int id, const char* const emsg = 0 ){ // Protect i/o Mutex::ScopedLock lock( s_ioMutex ); if( emsg ) { std::printf( msg, id, emsg ); } else { std::printf( msg, id ); }}void Printf( const char* const msg, int id, const int i ){ // Protect i/o Mutex::ScopedLock lock( s_ioMutex ); std::printf( msg, id, i );}void Printf( const char* const msg, int id, const int i1, const int i2 ){ // Protect i/o Mutex::ScopedLock lock( s_ioMutex ); std::printf( msg, id, i1, i2 );}Mutex g_ehMutex;struct MyException : std::exception{ MyException( const char* const msg ) : m_msg( msg ) { g_ehMutex.Lock(); Printf( " Create %x\n", (int)TSK_self() ); } MyException( const MyException& e ) : m_msg( e.m_msg ) { g_ehMutex.Lock(); Printf( " Copy %x\n", (int)TSK_self() ); } ~MyException() throw() { g_ehMutex.Unlock(); Printf( " Destroy %x\n", (int)TSK_self() ); } virtual const char* what() const throw() { return m_msg; }private: const char* m_msg; MyException& operator=( const MyException& );};extern "C"Void Task(){ const TSK_Handle self = TSK_self(); const int priority = TSK_getpri( self ); const int id = int(self) | (priority << 20); Printf( "Task %x starting\n", id ); for( ;; ) { try { if( 0 == Rand() % 10 ) { Printf( "Throwing in (%x)\n", id ); throw MyException( "D'oh!" ); } } catch( const std::exception& e ) { Printf( "Catching in (%x): %s\n", id, e.what() ); } catch( const std::bad_alloc& ) { Printf( "Exception (%x): bad_alloc\n", id ); } catch( ... ) { Printf( "Exception (%x): unknown\n", id ); } TSK_sleep( Rand() % (4*priority) ); }}const int N_TASKS = 4;TSK_Handle s_tasks[ N_TASKS ];int main(){ for( int n=0; n < N_TASKS; ++n ) { if( s_tasks[ n ] = TSK_create( (Fxn)&Task, 0, n ) ) { TSK_setpri( s_tasks[ n ], 1 + n/2 ); } else { std::printf( "Unable to create task %d\n", n ); } } std::printf( "Main finished: %d tasks created.\n", N_TASKS );}
I don't have any definitive answers just yet, but we are looking into whether there are issues between C++ Exception Handling and DSP/BIOS. I'll be sure to get back to you when I have additional info.
Unfortunately, you have wandered into unsupported territory. C++ exception handling uses a data structure, not exposed to the user, to do its job. There is only one copy of this data structure in the system. The different tasks are all modifying this one data structure. Changes made by one task foul up the exception handling going on in another task. This information needs to be added to the C++ Wiki article.
Thanks and regards,
-George
TI C/C++ Compiler Forum ModeratorPlease click Verify Answer on the best reply to your question.The Compiler Wiki answers most common questions.Track an issue with SDOWP. Enter your bug id in the "Find Record ID" box.
GeorgemUnfortunately, you have wandered into unsupported territory. C++ exception handling uses a data structure, not exposed to the user, to do its job. There is only one copy of this data structure in the system. The different tasks are all modifying this one data structure. Changes made by one task foul up the exception handling going on in another task. This information needs to be added to the C++ Wiki article.
Hi, George.
Are you a representative of TI, and is this TI's official answer to this problem?
If so, is the "unsupported territory" exception handling in general (which contradicts the compiler manual and release notes) or the use of it in conjunction with DSP/BIOS tasks (a rather glaring omission from the manual, IMHO)? Are there any other unsupported areas related to exceptions that we should be aware of? Does TI consider this a bug, and if so, will it be fixed and when? Can you provide me with a workaround? E.g., can I recompile part or all of the RTS source in the lib directory with a mutex around the aforementioned data structure? Can I safely use exceptions in a single task with several other non-throwing tasks running "simultaneously"?
As I mentioned previously, I guessed an unprotected data structure might be the problem and tried to exclude multiple exceptions from being thrown simultaneously with a mutex in my code (see my second post with code above). If the problem is what you describe, why did my solution not work? In particular, how do you explain the failure to destroy the exception object and thus unlock the exception handling? AFAICT, the exception handling code in throw.cpp should not be invoked until until the temporary MyException object is constructed in the statement 'throw MyException( "" );' completes, which can't happen until my exception handling mutex is available. Hence, the internal data structure should not become corrupted by multiple tasks throwing exceptions.
FWIW, I have uploaded my project file, DSP/BIOS config, and the updated source code here on my TI community account.
Well, mlimber, you write some long posts. I may yet regret joining this thread. :-)
I work for TI. That's true for anyone who has that little red person looking thing under their avatar.
WRT "official answer" ... Of course not. It is just a forum post hoping to be helpful. The official answer will arrive in a readme file, or the documentation, or something like that. The way I think about it, the whole point of a forum is to get useful information out quickly, rather than making you wait for the official answer.
I should have specified "unsupported territory" more clearly. Here's another go ... C++ exceptions work. BIOS tasks work. They don't work together. More generally, C++ exception handling is not re-entrant, which means it does not work in combination with any RTOS feature that supports multiple threads of execution. This means BIOS, Enea, Wind River, Linux, etc.
More answers ... I'm not aware of other unsupported aspects of C++ exceptions. TI does think this needs to be addressed. I don't know when it will be addressed. There is no workaround. I do plan to add this to that Wiki article about C++ support.
The overwhelming majority of our C++ customers handle this by not using exceptions at all. After all, exceptions impose a noticeable performance penalty, even when no exceptions are thrown.
I see how you tried to lock out task switches between throw and catch. I am not familiar with the low level details of TI's exception handling. I know even less about BIOS. So I just can't comment. But it seems to me that settling this side issue is not necessary. Even if we figure out how to make it work, will folks settle for disabling task switches between throw and catch? I don't think so.
GeorgemWell, mlimber, you write some long posts.
What I meant by "official" is, Is this tech support's response to this problem? Are you working with Harry and Tim, who are my points of contact on this matter? Are they going to come back to me with more/different (or perhaps ultimately the same) information than you have provided here?
The C++ standard (at present) does not make any mention of multithreading, so it neither affirms nor denies that it will work. Nonetheless, ISTM that this falls in the "glaring omission" category on TI's part. If TI claims to support both features in its products, it is natural to assume they work together unless TI tells its customers otherwise. Can I use RTTI and threads? Can I use virtual functions and threads? Etc. The obvious answer is, "Yes, unless the vendor tells me otherwise." The documentation tells me that I can't use printf without protection, for instance.
In any case, can I use them together in restricted situations? For instance, can I throw exceptions in main()? Can I allow one of my four tasks to throw and catch exceptions within itself if I disallow (by programming convention) the others from doing so? I would suspect, yes, given your identification of the problem as corruption of an internal data structure, but could just task switching somehow corrupt this structure? I've run some tests, and I don't see this corruption happening, but it may be that the circumstances weren't just right to make it happen. This is the most important question in this email.
Ok. Is anyone else aware of other unsupported aspects, a timeline for addressing it, or a workaround -- particularly along the lines that I did above? (I ask because you said you aren't familiar with DSP/BIOS or TI's exception handling.)
That is irrelevant. You cannot and should not try to micromanage your customers' designs. Your compiler supports C++ exceptions, RTTI, virtual functions, etc. and you supply various libraries and facilities such as DSPLib and DSP/BIOS. Let us choose when it is appropriate to use them, and give us all the information we need to make an informed decision. Space/time overhead is only one of many considerations, as is incompatibility of a language feature with DSP/BIOS multitasking.
First note that my code doesn't disable task switching, though that may be an alternate approach, but rather precludes other tasks from creating MyException objects. I agree that this is not a final solution, but if something along these lines could be made to work, it would provide something we could work with until TI gets a full solution in place. Then we could use exceptions so long as we're careful not to throw more than one at a time (just like we have to be careful with printf). It isn't optimal, but sometimes something is better than nothing.
In any case, my locking code doesn't work. I'd guess that this is because the destructor for the MyException object is called before the exception handling is truly done processing. So the behind-the-scenes code looks like:
try { throw MyException( "D'oh!" ); } catch( const exception& e ) { Printf( "Caught in (%x): %s\n", id, e.what() ); // Compiler generated code: e.~MyException(); // A eh_internal_cleanup(); // B }
Lines A and B may actually be contained in some function in EDGRTS/throw.cpp, but only the order is important here. The problem is that A releases the lock and B continues using the data structure that was intended to be locked. If another thread jumped in threw an exception in between A and B, we could see corruption of the internal data structure. The same thing would happen if I called SWI_disable()/SWI_enable() in MyException's ctor/dtor to actually disable task switching. One could move the lock from the ctor/dtor to before the throw and after the catch block like:
g_ehMutex.Lock(); try { throw MyException( "D'oh!" ); } catch( const exception& e ) { Printf( "Caught in (%x): %s\n", id, e.what() ); // Compiler generated code: e.~MyException(); // A eh_internal_cleanup(); // B } g_ehMutex.Unlock();
This would presumably make it safe since two throwing threads couldn't run concurrently and stomp on the internal data until B has totally completed. Although one could automate this a little more, like you suggest, this is not a good general solution and is rather error prone in real code. It is essentially equivalent to saying only one task may use a try-catch block at a time.
Other possible work-around hacks are to find the eh data structure in the linker map and save and restore it when throwing (this might not work, depending on what that structure contains), or locally rebuilding the RTS to be exception safe by, say, inserting a mutex lock in the __throw() function and an unlock in whatever the appropriate place is or using thread-local storage to allocate the relevant data.
In any case, I am proceeding on the assumption that I can throw/catch exceptions in one and only one thread since it would be quite painful to alter all the code at this point and one thread has significantly more exceptions than the others. Please confirm that this is safe.