This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TM4C1290NCPDT: Task (Priority 4) Blocked on GateMutex Despite Sufficient Heap Space

Part Number: TM4C1290NCPDT

Tool/software:

Hello,

I am troubleshooting a critical issue in my TI-RTOS application. Here are the details:
Task Name: Operator Task
Task Priority: 4
Task Purpose: Feeds the watchdog every 50ms.
Issue Observed: The Operator task remains blocked on a GateMutex and fails to execute as expected, causing the watchdog to not be fed.

Observations from RTOS Object View (ROV):

1. The Operator task is blocked on a GateMutex (0x200377fc).


2. Heap Memory:
Total Size: 0x10000
Total Free Size: 0x7550
Largest Free Block: 0x74C0
• Heap memory appears sufficient.


3. Call Stack (Attached Screenshot):
• The Operator task appears blocked while trying to allocate memory.
• The failure seems related to a std::string operation in Var_Presenter.cpp, line 63, specifically during a std::replace operation.



4. Code Context (Attached Screenshot):
• The blocking point corresponds to a std::replace operation on a std::string object in the Var_Presenter::Update() function.


Debugging Steps Taken:
1. Verified heap memory in the HeapMem ROV module—free space and the largest free block seem adequate.
2. Checked the call stack, which suggests a memory allocation issue, possibly due to std::string operations in the Operator Task involved in manipulating a string used for a display update in the Var_Presenter::Update() function.
3. We looked at HeapTrack to view the timing of tasks, did not help with any Operator task based information.

Request for Help:
Are there specific recommendations for debugging or resolving GateMutex and heap allocation failures in TI-RTOS?


Any insights or guidance would be greatly appreciated!

Thank you!
Mahima

  • We can consider an alternate to the `std::replace`, but we're not expecting that function to actually be changing characters. Regardless, it appears a large amount of heap memory is still free to do the work.

  • Hi,

      I'm not too sure what is the problem. It is probably due to the priority inversion. An example would be that a low priority and a high priority thread share a mutex. The low priority first gets the mutex but it is preempted by a bunch of "MEDIUM" priority threads. In this case the high priority thread is held off for a longer time - thus inverting the priority between the low and high priority tasks. 

      I think what you want is in both the threads that use mutex to first set the priority. See below example if that helps. 

    gateKey = GateMutexPri_enter(gateMutexPri); // enter Gate

    cnt += 1;                                                             // protected access

    GateMutexPri_leave (gateMutexPri. gatekey)  // exit Gate

    What this does is that when the high priority thread runs (or preempts the low priority thread), the BIOS will force the low priority thread to have the same priority as the high priority thread, allowing the low priority thread to finish its access and then BIOS will demote the low priority thread to its original priority upon GateMutexPri_leave() is called. 

  • Thank you for the suggestion regarding priority inversion and the usage of GateMutexPri. Unfortunately, this approach doesn’t address the issue because the mutex causing the blocking is not explicitly managed by our application.

    The GateMutex in question is part of the malloc implementation from TI-RTOS, and the blocking occurs during a dynamic memory allocation call (std::string operation) in Var_Presenter::Update(). This means the mutex is internally handled by the TI-RTOS heap implementation.

  • • The blocking point corresponds to a std::replace operation on a std::string object in the Var_Presenter::Update() function.

    I have not used the replace function in the format you use before. It seems like you are trying to replace a NULL char with a SPACE? Below is an example on the internet. Can you try some experiments with a non NULL character or something simple and do you still the Mutex issue? For example, replacing a "A" with a "B" and does that work?

    // C++ Program to show, how to use
    // string::replace() for replacing the
    // multiple characters with single repeated
    // character
    #include <bits/stdc++.h>
    using namespace std;
    
    int main() {
        string str = "Hey World";
    
        // Defining the range
        auto first = str.begin();
        auto last = str.begin() + 3;
    
        // Replaces firts 3 character of
        // str with 3 copies of '!'
        str.replace(first, last, 3, '!');
    
        cout << str << endl;
    
        return 0;
    }

  • Thank you for the insights so far. I’d like to provide an update on our findings and further observations.

    Update:

    After reading the suggestion about our use case of  std::replace causing issues, we removed that line of code entirely. However, the failure persists and has now moved to a different point in the code. Specifically, the failure is now occurring in the Format_Value_As_Decimal() function, as shown below.


    Code Context:

    void Format_Value_As_Decimal(std::string &msg, bool editing, double val) {
       std::stringstream ss;
       std::string       sign;
       uint8_t           digits = Get_Display_Digits();
       bool              is_negative = false;
       if (val < 0) {
           val *= (-1);
           is_negative = true;
       }
       if (m_consts.Precision) {
           digits++;
       }
    // Add the sign indicator
       if (m_consts.Is_Signed() && (is_negative == false)) {
           sign = "+";
       } else if (m_consts.Is_Signed() && (is_negative == true)) {
           sign = "-";
       } else {
           sign = " ";
       }
    // Set fill character based on editing state
       char fill = ' ';
       if (editing) {
           fill = '0';
       }
    // Build the string with left-padded zeros
       ss << sign << std::fixed
    << std::setw(digits)
    << std::setfill(fill);
       ss << std::fixed << std::setprecision(m_consts.Precision) << val;
    // Copy string back
       msg = ss.str();
    }
    


    Observations:
    1. The failure occurs when the above function is executed, specifically during dynamic memory allocation operations involving std::stringstream and std::string.
    2. Heap Memory Details:
    • The HeapMem ROV view shows that the heap has sufficient space.
    Total Free Size: 0x7550
    Largest Free Block: 0x74C0
    • The heap free list indicates that some blocks are in use, while others are available, which suggests no complete exhaustion of memory.
    3. Call Stack Details:
    • The call stack points to a contention issue within the TI-RTOS malloc implementation during memory allocation, specifically while processing the std::stringstream and std::string operations.
    4. Also in Heapmem -> Freelist, we did notice that there were free blocks and couple in use blocks which does indicating space was free

    Concerns:
    1. Could the underlying issue still be related to memory fragmentation, even though the heap has free blocks available and appears to have sufficient space?
    2. Since the failure is now observed in a different part of the code (after removing std::replace), could this suggest a deeper contention or fragmentation issue within the TI-RTOS heap/malloc implementation?
    c. Are there tools or techniques within the TI-RTOS framework that you can recommend that provide more detailed insights into heap fragmentation or contention at runtime?


  • Hello,

    Thank you for your response and for sharing the related post on HeapMem and SysMem.
    While that information provides useful insights into how heap memory is managed, I need a structured way to start debugging this issue systematically from the watchdog failure back to the root cause.

    Right now, I know that:
    1. The watchdog fails, indicating the Operator Task (Priority 4) did not execute as expected.
    2. The Operator Task is blocked on a GateMutex that is internally used by TI-RTOS’s malloc implementation.
    3. The heap has enough free space, but the task is still getting blocked.

    Where Should I Start Debugging?

    I want to start from the watchdog breakpoint and work my way back to the root cause.
    Please, share all the steps you would take to debug this.
    Along with: 
    • What should be my first step when the watchdog trips?
    • What tools or debugging methods would help me determine why the task isn’t running?
    • How do I trace mutex contention or heap issues in TI-RTOS?
    • Are there common pitfalls in RTOS scheduling or memory allocation that could cause this behavior?

    I would really appreciate any guidance on where to begin. Let me know what information would be helpful to provide.



  • Hi,

      Please refer to this FAQ by Todd where under Debugging you can find various resources to debug TI-RTOS projects. 

    https://e2e.ti.com/support/processors-group/processors/f/processors-forum/840835/faq-where-can-i-find-ti-rtos-or-sys-bios-faqs/3109064#3109064

    This below presentation by Todd on Debugging TI-RTOS projects has some useful tips. Look under Memory Allocation where you can enable Heaptrack to debug dynamic memory allocation issues.

    TIRTOS_CCSDebugging.pdf