• State Resolved
  • Locked Locked
  • Replies 10 replies
  • Answers 1 answer
  • Subscribers 99 subscribers
  • Views 681 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

DSPLink Shutdown Requirements?

Judson Wilson
Intellectual 620 points
Other Parts Discussed in Thread: OMAP-L138

I am trying to figure out how to properly shutdown DSPLink - specifically if I notice that the DSP is acting funny and I want to pull the rug out from under it and then restart it (while keeping Linux running continuously).  This is using the OMAP-L138 running Linux on the ARM.

In this situation, the DSP side would probably not be able to do any shutdown, and it seems as though the ARM has complete control over the DSP, so this shouldn't be a problem.

I have a project based on the MESSAGE example in the 1.65.00.03.  I modified the project such that the DSP continuously stuffs messages over to the ARM, and after receiving X messages the ARM decides to shutdown (while the DSP is still trying to send messages).

The shutdown routines are the same as in the MESSAGE example, although the DSP side does not get called.  The arm side appears to hang indefinately inside MSGQ_close(). However, if I send a shutdown message to the DSP right before this, and have the DSP call the MESSAGE example's shutdown procedure, everything seems to work fine.

I cannot find anything in the documentation that suggests that MSGQ_close() would ever hang, or that it has any requirements before shutting down that hasn't been met. Any help or pointers to some documentation?

Thanks.

  • 0
    TI__Genius 12025 points

    Judson,

    Unfortunately, DSPLink does not support fault tolerance. It is true that the ARM has complete control over the DSP, as you point out, however, the sticking point is in resource allocations. During startup, DSPLink will allocate some Linux resources and share them with the DSP. If the DSP corrupts these resources, or fails to release them, we have a resource leak on the Linux side. Some of these resources are allocated in the Linux kernel by the DSPLink driver. This makes it especially difficult to reclaim these resources and maintain a domain of integrity on the Linux side. Short of a system wide restart, the best you could do is to terminate the Linux application and remove the DSPLink driver.

    I know you are trying to avoid this, however, this would require fault tolerance support in DSPLink which it does not have.

    ~ Ramsey

  • 0 in reply to Ramsey
    Intellectual 620 points

    Ramsey said:
    Short of a system wide restart, the best you could do is to terminate the Linux application and remove the DSPLink driver.

    I am fine with this, if it will work.  So far I see the program hang and the terminal go unresponsive, I guess I can try another terminal to kill the application and remove the driver.

  • 0 in reply to Judson Wilson
    Intellectual 850 points

    Dear Ramsey:

     If we want to change DSP programs in normal condition, what's the proper steps to stop DSPLink resource such as MSGQ and re-initialize/use it? 

     

     

     

  • 0 in reply to CalvinTai
    TI__Genius 12025 points

    Calvin,

    The exact sequence of calls to restart DSPLink will depend on your application and which modules you use. You will always be using PROC and POOL, but other modules like MSGQ depend on your application. Look in the gpp/samples folder for examples on startup and shutdown code for the different modules. The first DSPLink call you make will be PROC_setup and the last call will be PROC_destroy. Here is an overview from the message queue example.

    Setup

    PROC_setup
    PROC_attach
    POOL_open
    MSGQ_open
    PROC_load
    PROC_start
    MSGQ_transportOpen

    Shutdown

    MSGQ_transportClose
    PROC_stop
    MSGQ_close
    POOL_close
    PROC_detach
    PROC_destroy


    ~ Ramsey

  • 0 in reply to Ramsey
    Intellectual 620 points

    Ramsey said:
    best you could do is to terminate the Linux application and remove the DSPLink driver.

    I tried this once, and the process was consuming 99% of the processing time, and could not be killed.  Also, the driver processes could not be killed with kill -9 or killall, etc.

    I guess the only thing to do is a full system reset?

  • 0 in reply to Ramsey
    Intellectual 620 points

    Ramsey said:
    Setup

    PROC_setup
    PROC_attach
    POOL_open
    MSGQ_open
    PROC_load
    PROC_start
    MSGQ_transportOpen

    Shutdown

    MSGQ_transportClose
    PROC_stop
    MSGQ_close
    POOL_close
    PROC_detach
    PROC_destroy

    The problem with this is that it does not shed any light on which order things have to happen with relation to the other processor, and how one might ensure these requirements are met. 

    Take, for instance, the sequence shown above, and the behavior I described earlier about the hang in MSGQ_close() on the ARM.

    On the ARM side, if the ARM is waiting in MSGQ_close() for something to happen on the DSP side, it isn't going to happen, because PROC_stop was already called, and the DSP is halted.  So somehow the DSP needs to signal to the ARM that it is ready to stop - but doing so requires the messaging services which it just finished closing, and are thus unavailable.

    Does this problem make sense? Is there a robust solution?

    Thanks,

    - Judson

  • 0 in reply to Judson Wilson
    TI__Genius 12025 points

    Judson,

    Yes, in these cases a full system reset is what I do. Once something has gone bad in the Linux kernel, there is little else to do.

  • 0 in reply to Judson Wilson
    TI__Genius 12025 points

    Judson,

    I understand your question now. You are correct, the startup and shutdown sequence on both the ARM and DSP must be coordinated. Unfortunately, we don't have this documented anywhere. Somehow, the master application, typically the ARM, will initiate a shutdown sequence. This can be done in-band by sending a final shutdown message to the DSP. The DSP would acknowledge the request by returning the message to the ARM and then begin the shutdown sequence on the DSP. When the ARM receives the return message, it would free the message and also begin the shutdown sequence.

    After the DSP sends the final message to the ARM, it can release the message queue it had located and close its own queue. After the ARM side has received the final message, it can free the message and then release the DSP's queue and then close its own queue. Only the ARM makes PROC calls to stop the DSP. Sometimes, we add a delay on the ARM side before calling PROC_stop to give the DSP time to complete its final MSGQ_close call, but there is no sure way to know when the DSP is done.

    The best I can offer is to follow the examples given in the sample folders for the particular modules your application uses. DSPLink does not have a robust solution for this.

    ~ Ramsey

  • 0 in reply to Ramsey
    Intellectual 620 points

    Ah, I implemented the ARM->DSP shutdown message in my app, and was thinking about adding the delay, but I see a lot of potential benefits in a DSP->ARM response as well, just to make sure the DSP got the message. I will investigate that further - hopefully I don't run into a cat-chasing-tail situation where now I don't know if the ARM can actually receive the message if the DSP begins shutdown shortly after! 

    Thank you for your help.

  • 0 in reply to Judson Wilson
    TI__Genius 12025 points

    When the DSP sends the final message, as long as the MSGQ_put returns success, you know that the message has been delivered, it is in a queue on the ARM waiting to be picked up. And as long as the heap from which the message was allocated is owned by the ARM, then the DSP can safely start the shutdown sequence without worry. Eventually, the ARM will pickup the last message and return it to the heap.

    ~ Ramsey