• State Resolved
  • Locked Locked
  • Replies 4 replies
  • Subscribers 101 subscribers
  • Views 2265 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Mailbox message processing hold off by low priority task

Ganz
Intellectual 770 points
Other Parts Discussed in Thread: SYSBIOS, CC2650

We have a application that the mailbox messages need to be process in a timely manner. But we found that the message processing could be hold off by lower priority task due to sysbios Mailbox implementation -- putting the message into the queue and posting the notification semaphore to the receiver is not atomic.

Scenario:

Task 1 -- low priority, message writer;  Task 2 -- medium priority;  Task 3 -- high priority, message writer;  Task 4 -- highest priority, message receiver

Using certified mail (mailbox size > 1)

Mailbox message 1 contains -- Msg1, notification semaphore 1(Sem1), response semaphore(RspSem1)

Mailbox message 2 contains -- Msg2, notification semaphore 2(Sem2), response semaphore(RspSem2)

1. Task 4 pends on mailbox messages.

2. Task 1 sends message1. After mailbox put message1 into  queue but before it can post out the notification semaphore, it's preempted by Task 2.

3. Task 3 preempts Task 2, sends message to Task 4,

4. Task 4 is unblocked by notification semaphore sent from Task 3. Then it get the message from the queue, which is from Task 1,  and process it,

5 Task 4 pends on mailbox message notification.

6. Task 3 yield.

7. Task 2 resumes and continues to run.

Now the message sent from Task 3 can not be processed by Task 4 because the notification is blocking by Task 2.

Question1:

How do we resolve the issue?

Question 2:

Why the mailbox action from putting the message to the queue to posting notification semaphore is not atomic?

GanZ

  • 0
    Expert 1820 points

    Hi


    What you experience is an example of priority inversion. There's a relatively simple solution, which is raising a task's priority to the maximum priority before posting to a mailbox (and resetting it afterwards).

  • 0
    Guru 16361 points

    I've not investigated this too deeply, but I think the mailbox should really be adding the item to the dataQue and posting dataSem atomically. Currently it does this:

    /* put message on dataQueue */
Queue_put(dataQue, (Queue_Elem *)elem);

/* post the semaphore */
Semaphore_post(dataSem);

    Each of those function calls performs an atomic operation using a Hwi_disable/Hwi_restore pair. The problem is that an interrupt can occur between the two. I think this change would fix the problem without keeping interrupts disabled for too long:

    key = Hwi_disable();

/* put message on dataQue (wrapped in Hwi_disable/Hwi_restore, so don't need Queue_put here) */
Queue_enqueue(dataQue, (Queue_Elem *)elem);

/* post the semaphore */
Semaphore_post(dataSem);

Hwi_restore(key);

    If the mailbox code is left as-is, then this scenario and the task priority workaround should be covered in the documentation. Currently the TI-RTOS Kernel User's Guide only mentions priority inversion in the context of gates.


    Edit (March 2016): Considering this further, I think it's only possible to get this situation if the low priority task gets interrupted by a HWI. The task scheduler wouldn't have chance to run between the Queue_put and Semaphore_post otherwise. Of course, that HWI could wake up a task with higher priority than the one currently in Mailbox_post...

  • 0 in reply to Robert Cowsill
    Guru 16361 points
    I've just checked the latest TI-RTOS for CC2650 (V2.16); this issue is still present in the latest code and not mentioned in the documentation.

    It'd be good to hear from the RTOS development team on this. Can the mailbox be made safe in the case of a task preemption during Mailbox_post? If not, would it be possible to warn of the issue in the documentation?
  • 0 in reply to Robert Cowsill
    TI__Mastermind 21645 points
    Hi Robert,

    Thanks for pointing this out. I have filed a bug to track this issue. In the bug I have asked we either make the operation of putting the message on the queue and posting the notification semaphore atomic or document this.

    Best,
    Ashish