Reattach DSP to ARM using IPC on an OMAP-L138

I just spent some time on the "end run" approach and got nowhere.   After the DSP reboots, there seems to be nothing the ARM can do to get to a state where it can "attach" the DSP again short of rebooting. 

I've tried doing a Ipc_detach on the DSP proc ID, but the ARM detach gives up because the DSP's Ipc_Rserved->startedKey structure is in set to PROCSYNCFINISH.

I've tried Ipc_stop and then another Ipc_start on the ARM side, but that appears to fail as well, i.e., the second Ipc_start never starts.

So, is what I'm trying to do just impossible?   Is the proper response to the DSP rebooting to reboot the whole chip?   Should I just give up on IPC and go back to bare metal signal interrupts?

Jay,

Which processor is the owner of SharedRegion 0?

I'm curious what would happen if you made the other core the master since the master core will clear away the reserved portion of shared memory when you do an Ipc_start().

Judah

I believe the ARM is the owner of Shared Region 0, at least according to the commong .xs I include in both the DSP and ARM projects:

var SharedRegion = xdc.useModule('ti.sdo.ipc.SharedRegion');
SharedRegion.setEntryMeta(0,
    { base: SHAREDMEM,
      len:  SHAREDMEMSIZE,
      ownerProcId: 0,
      isValid: true,
      name: "shared_mem",
    });

 The ARM brings up my DSP and starts it, so it makes sense (to me) that the ARM is in charge of the IPC as well.  Perhaps going the other way (since I expect the DSP to be going up and down much more often in my development cycle than the ARM) is a good thing to try.   I would just change the ownerProcId to 1 and leave everything else the same, correct?

Yes, that is correct.

I tried switching the SR0 owner to be the DSP, and I still can't get the result I want.   I see that the DSP's Ipc_Start resets the shared memory region to 0 when the DSP restarts, which is an improvement, but there still seems to be too much "state" lying around on the ARM to re-establish communications, even though the ARM knows that the DSP has gone down and come back up again through an external flag in DDR.

For example, as far as I can tell, the "attached" reference count for the DSP cannot be reduced via Ipc_detach  to 0 because the "startedKey" of the internal module state of various things  is not in the right condition.   I tried just poking in a value of 0 for the attached reference and reattaching the ARM, but that failed as well.

Doing an Ipc_stop seems catastrophic:  it sets Ipc_sr0MemorySetup to 0 and nothing works after that.

So, am I barking up the wrong tree here?  Is there a known method for handling a processor going down and coming back up again besides resetting the whole system? 

It seems that the _pe9.c file generated for my application sets up module state variables in very particular ways.   Is there anyway to get back to this state short of a reload of the program?   It looks like it would be very difficult for my program to access the structures set up in the _pe9.c file. 

Jay,

I'm assuming you are running BIOS on both ARM and DSP correct?  IPC wasn't design to handle this sort of catastrophic going down of a processor.

When you are doing down on the DSP, do you know that you are going down?  Are you able to do anything before you go down or does it simply reset at some point?

I think at this point you are probably fighting a losing battle.  I think you've exhausted the possibilities.  Its something we might try to support in the future but we haven't had request for this sort of scenario.

Judah

Yes, I am running BIOS (6) on both processors.  

What I'm trying to address is my group's normal workflow where we get our image acquisition/replay system going, load and start the ARM via CCS5, which enables the DSP, and then work on the DSP vwhere the algorithms are (also via CCS5).  In this mode we bring the DSP up and down, sometimes loading new code on, sometimes just restarting, debugging, repeating that many, many times without ever touching the ARM again as it happily acquires images and notifies the DSP.   In the running code booting from flash I do not expect to see the DSP go down without also taking down the ARM.

Having to change the work flow to actually do a complete boot process, i.e., reload code onto the ARM, whenever I want to restart the DSP, significantly slows down my group's development.   To the point that I will probably back off of IPC and go back to bare metal signal interrupts, since I know that works the way I want it to.    I had high hopes that I could build on IPC beyond the simple notification to get more sophisticated message passing and make more effective use of the ARM, but that looks unlikely now without significantly slowing down our debug cycle.

I guess I have a feature request:  I want Ipc_stop to bring the IPC system to EXACTLY the same place it was in before the first Ipc_start, so that Ipc_start brings the system up in a "blank" slate.   If I had that, then everything would be fine, because I have a method to let the ARM core know that the DSP has restarted, I just can't get the ARM back to its starting point in regards to IPC.   A simple test would be the ability to do a Ipc_start -> Ipc_stop -> Ipc_start cycle succussfully.

Jay,

Are you using SysLink on the ARM? Reading through the post, I see you are using the ARM to initially load and run the DSP (with BIOS on both cores). How did you initially load and run the DSP, with SysLink or manually with CCS? If you are using SysLink, then there is considerable state on the ARM which would need to be reset.

In general, our fault tolerance is very limited. We have started adding fault tolerance support to SysLink (not yet available) but I don't believe IPC has any FT support.

From reading your post above, I believe your development flow is to have the ARM up and running, then be able to rebuild a new DSP executable and load/run it on the DSP "on-the-fly" without restarting the ARM. This would be difficult to support with IPC because there are data structures in memory used by both sides. When you relink an executable, the address of this data structure might change. When SysLink loads/runs the DSP, it extracts this address from the DSP executable (during the loading phase). In your situation, if the ARM does not reload the DSP, there would be a mismatch of the base address and no hope of successfully re-attaching.

One last idea, would it be possible to modify your development flow to have the ARM stop/unload the current DSP executable and then load/run the new DSP executable through a flag mechanism (use CCS to set a flag in memory). This would keep the ARM up and running. Reloading the DSP is supported by our software stack.

~Ramsey

Ramsey,

Thanks for your response.

We are not currently using SysLink on either the ARM or the DSP.    The documentation led me to believe that if both sides are SysBIOS, then the "proper" approach was to use the IPC library.   Was that the wrong impression?   With SysBIOS IPC the "shared memory" location is determined by the common configuration files, so at least that is fixed.

Your idea of a "flag" is an interesting one.   I'm not sure what the sequence would be, as it appears that Ipc_detach has to be a two sided thing, i.e., both sides have to call detach simultaneously or the connection is not broken.   Is that an incorrect statement?   I only believe this because when I try to do an Ipc_detach of the DSP on the ARM when the DSP is suspended, it never returns success, since there appears to be a check in the Ipc_detach code to make sure the DSP is in a "detaching" state, which it never gets to.    Is there a tested sequence of operations with SysLink that would allow this?

While fault tolerance is good, all I really need is the ability to "restart" the communications from scratch.   Can I do this with a released version of SysLink, i.e., do you have a unit test that shows a sequence on the "host" processor of

  start communications
  make connections (block until client core connects)
  stop communications (breaking connections, assume client core has been reset, and will be restarted, perhaps even with the restriction of using the same code)
  start communications (block until client core connects)
  make connections (block until client core connects)
    .
    .

It appears to me that the version of IPC I have (1.24.02.27) cannot do this, and my impression from the previous discussion is that no version of IPC can.   If SysLink can, then I would happily switch from IPC to SysLink to get this ability.

   Jay

I haven't given up on you guys yet.   For now I'm going to fall back to just raising the CHIPSIG2 interrupt via registers.   If we move to more sophisticated need for DSP/ARM communications, I'll check it again.

One thing that may be an alternative approach in the future:  instead of the Ipc_start/stop/start ability, how about an Ipc_detachForce call?   Right now, Ipc_detach just doesn't work at all if the other side has disappeared, i.e., it has to be part of a two sided negotiated dance.

How about you support the situation where Ipc_detach_force does this

1.  is the ID of the module being detached > the current ID (i.e., on an OMAP am I the ARM disconnecting the DSP)?
2. If so, reset the connection to detached (set the attached reference count to 0, clear out any bookkeeping in the internal tables, etc.), regardless of the state of the connection, because we expect that the "other" side has gone away and will be doing a complete reset from scratch.

The idea is that this would reset the result if Ipc_isAttached to false and allow a subsequent call to Ipc_attach(DSP) to act like the first call to Ipc_attach(DSP).  

With such a call (or equivalent), I believe I could get the functionality I want.  

   Jay