This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Ipc_detach not working on slave core even when master achieved detachment

Other Parts Discussed in Thread: SYSBIOS

Hi,

I am working right now on the shutdown path of an ARM/DSP application, using DM8168-z3 and ezsdk-5_05_01_04.

My DSP (Id 0) cannot detach from the ARM (Id 3), I looked into the API documentation, it says:

If called from the processor with the smaller procId, this function returns Ipc_E_NOTREADY while the processor with the bigger procId has not finished detaching. Once the processor with the bigger procId is finished detaching, this function deletes the instances created for communicating with the specified remote processor.

On the ARM side, I have:

remoteProcId = 0;

status = Ipc_control(remoteProcId, Ipc_CONTROLCMD_STOPCALLBACK, NULL);

vdbg("Ipc stop callback status [%d]", status);

And I achieved (log dump):

SvrCntl_Unlink: Ipc stop callback status [0]

Now, I found out that Ipc_control w/ the STOPCALLBACK argument goes down to Ipc_detach, the returned 0 is saying that it achieved detachment, while on the DSP side I have my while loop:

/* disconnect from remote processor */
do
{
status = Ipc_detach(3);

Log_print1(Diags_USER7, "Ipc_detach[%d]", (IArg)status);

if (status == Ipc_E_NOTREADY)
{
Task_sleep(400);
}

} while (status == Ipc_E_NOTREADY);

With the following dump:

N:DSP P:0 #:00247 M:xdc.runtime.Main S:Ipc_detach[-11]
N:DSP P:0 #:00248 M:xdc.runtime.Main S:Ipc_detach[-11]
N:DSP P:0 #:00249 M:xdc.runtime.Main S:Ipc_detach[-11]
N:DSP P:0 #:00250 M:xdc.runtime.Main S:Ipc_detach[-11]
N:DSP P:0 #:00251 M:xdc.runtime.Main S:Ipc_detach[-11]
N:DSP P:0 #:00252 M:xdc.runtime.Main S:Ipc_detach[-11]
N:DSP P:0 #:00253 M:xdc.runtime.Main S:Ipc_detach[-11]

Endlesly..

So I think I am following the rules of the API, dont know if I am missing something else. Are there any circumstances that make the detchment to go wrong? 

Note: Using also IPC Notify and IPC MessageQ, both services closed and unregistered by the time of the detachment.

Regards

-Jose Lopez

 

 

 

 

 

  • Hi Jose,

    I apologize for not responding sooner.  Is this still an issue for you?

    I've needed to wait until our local "Ipc expert" is back in the office, which should be on 6/12.

    From my understanding and internal code inspection, it appears that you're doing everything that needs to be done.  The DSP Ipc_detach(3) is waiting for a flag in SharedRegion 0 to be set properly, and that appears to never be happening here.  I wonder if perhaps the SharedRegions 0 is being deleted by the ARM host before the DSP gets around to Ipc_detach().

    Some general questions that would help our analysis:
        - who is the owner of SharedRegion 0?
        - what "sync" method is configured for Ipc, ProcSync_PAIR or ProcSync_ALL?
        - are other cores running Ipc too (such as VIDEO-M3 or VPSS-M3)?

    Regards,

    - Rob

  • Hi Robert,

    Not been too much around e2e.ti, thanks for the reply.

    Here is my ipc related config of my Firmware, have worked for me for some time, if you see something funny that may lead to bogus behavior please let me know.

    /* xdcruntime settings */
    xdc.useModule('ti.sysbios.xdcruntime.Settings');

    /* OS runtime environment setup */
    var osalGlobal = xdc.useModule('ti.sdo.ce.osal.Global');
    osalGlobal.runtimeEnv = osalGlobal.DSPLINK_BIOS;

    xdc.useModule('ti.sdo.ce.ipc.dsplink.dsp.Settings');
    var MultiProc = xdc.useModule('ti.sdo.utils.MultiProc');
    var settings = xdc.useModule('ti.sdo.ipc.family.Settings');
    var procNames = settings.getDeviceProcNames();

    MultiProc.setConfig("DSP", procNames);

    var SharedRegion_map = {};

    SharedRegion_map["SysLink: HOST<--->DSP"] = 0;
    SharedRegion_map["Ipc"] = 1;

    var SharedRegion = xdc.useModule('ti.sdo.ipc.SharedRegion');

    var syslinkSharedMem = Program.cpu.memoryMap["DDR3_SR0"];
    var ipcSharedMem = Program.cpu.memoryMap["DDR3_SR1"];

    var entry = new SharedRegion.Entry();
    entry.base = syslinkSharedMem.base;
    entry.len = syslinkSharedMem.len;
    entry.ownerProcId = MultiProc.getIdMeta("HOST");
    entry.isValid = true;
    entry.cacheEnable = true;
    entry.name = "SYSLINK";

    SharedRegion.setEntryMeta(
    SharedRegion_map["SysLink: HOST<--->DSP"], /* index */
    entry
    );

    var entry2 = new SharedRegion.Entry();
    entry2.base = ipcSharedMem.base;
    entry2.len = ipcSharedMem.len;
    entry2.ownerProcId = MultiProc.getIdMeta("HOST");
    entry2.isValid = true;
    entry2.createHeap = true;
    entry2.cacheEnable = true;
    entry2.name = "SR1";

    SharedRegion.setEntryMeta(
    SharedRegion_map["Ipc"], /* index */
    entry2
    );

    - who is the owner of SharedRegion 0?

    I think from the above the owner is the ARM processor. If you think that the SharedRegion deletion may be a cause, on my DSP code I only delete the Heap created over shared memory that is used by the MessageQ API, I think in this case we are talking about different SharedRegions.

      - what "sync" method is configured for Ipc, ProcSync_PAIR or ProcSync_ALL?

    Apparently none explicitly declared. I think some months ago I tried playing with this configs with no remarkable result. Right now the default (which one?) should be the one used.

        - are other cores running Ipc too (such as VIDEO-M3 or VPSS-M3)?

    I am pretty sure yes, when my z3 starts it will load the M3 firmwares, those ones use OMX, so I think they should be using from CE down to syslink and IPC. I like this question because IS VERY important that all firmwares can work together without bothering each other, at least regarding memory management there should not be any trouble, and obviously I do not want to harm the ipc subsystems that connects all this coprocessors with the ARM. 

    So actually came to my mind asking if it is safe to use Ipc_detach and Ipc_stop on the DSP firmware and at the same time not affecting the M3 linkage.

    Regards

     

  • Jose,

    You should be able to shutdown the DSP without disrupting the VIDEO-M3 or VPSS-M3 processors. The idea is that the ARM has separate connections with the M3 processors which are maintained. Only the ARM to DSP connection is shutdown.

    One possible issue is cache management. Make sure that the SR_0 cache configuration is correct for both the ARM and DSP. My guess is that on ARM, SR_0 is cached; on DSP SR_0 is non-cached.

    Can you post your DSP memory map. Or tell me the base address of the SR_0 region. Meanwhile, I'll try to find the address of the detach flag.

    ~Ramsey

  • Ramsey,

    Here is the map. According to the EZSDK memory map, I am using the region that is supposed to be for LINUX_MEM_2, Linux is not using it for this tests.

    var mem_normal =
    [
    ["DDR3_HOST",
    {
    comment: "DDR3: Memory reserved for use by ARM",
    name: "DDR3_HOST",
    base: 0x80000000,
    len: 0x0B000000,
    }
    ],

    ["DDR3",
    {
    comment: "DDR3: Memory reserved for use by DSP C674x",
    name: "DDR3",
    base: 0x9FC00000,
    len: 0x02800000,
    }
    ],

    ["DDRALGHEAP",
    {
    comment: "DDRALGHEAP: off-chip memory for dynamic allocation",
    name: "DDRALGHEAP",
    base: 0x98000000,
    len: 0x01400000,
    }
    ],

    ["DDR3_SR1",
    {
    comment: "DDR3_SR1: Memory reserved for use by Shared Region 1",
    name: "DDR3_SR1",
    base: 0x9A100000,
    len: 0x00100000,
    }
    ],

    ["DDR3_SR0",
    {
    comment: "DDR3_SR0: Memory reserved for use by Shared Region 0",
    name: "DDR3_SR0",
    base: 0x9F700000,
    len: 0x00200000,
    }
    ],

    ["LOGGERSM",
    {
    comment: "LOGGERSM: UIA shared memory for logging", /* USE ./loggerSMDump.out -d 0xA4C00000 0x20000 dsp*/
    name: "LOGGERSM",
    base: 0xA4C00000,
    len: 0x00020000, /* 128KB */
    }
    ],
    ];

    Now that you are talking about Cache, my application needs Cache and also I use UIA over shared memory to dump logs. The only way I could achieve that was using the MAR bitmasks that are available through C64p family cache module, so right now some portion of my DSP memory partition is cache disabled. 

    This is on my ./cfg script also:

    var Cache = xdc.useModule('ti.sysbios.family.c64p.Cache');

    Cache.MAR160_191 = 0x0;

    Regrettably, I think we do not have the MAR bitmask feature for the c674 target for more precise cache disabling.

    I have another memory layout without UIA and cache disabling (that use DSP-only related memory according to EZSDK memory map ), wonder if firmware load/unload several times may work on that one.

    Regards

  • Jose,

    I've studied your DSP memory map above. Here is what I think it looks like.

    base       len comment
    ----------------------------------------------------------------
    8000_0000  B00_0000 (176 MB) DDR3_HOST, used by ARM
    8B00_0000  D00_0000 (208 MB) --------
    9800_0000  140_0000 ( 20 MB) DDRALGHEAP
    9940_0000   D0_0000 ( 13 MB) --------
    9A10_0000   10_0000 (  1 MB) DDR3_SR1
    9A20_0000  550_0000 ( 85 MB) --------
    9F70_0000   20_0000 (  2 MB) DDR3_SR0
    9F90_0000   30_0000 (  3 MB) --------
    9FC0_0000  280_0000 ( 40 MB) DDR3, used by DSP
    A240_0000  280_0000 ( 40 MB) --------
    A4C0_0000    2_0000 (128 KB) LOGGERSM
    A4C2_0000    E_0000 (896 KB) --------
    A4D0_0000  B30_0000 (179 MB) --------
    B000_0000 1000_0000 (250 MB) --------

    ----------------------------------------------------------------
    A000_0000 2000_0000 (512 MB) DSP: non-cached

    It looks to me that both shared regions #0 and #1 are cached by the DSP. From your shared region configuration, it looks like you have cache enabled for both of these regions. So, I don't think that cache is an issue. Your cache configuration looks correct.

    I see that you have placed SR_0 at 0x9F700000, which is the same location used by the EZSDK memory map. So, this also looks correct. Do you have the following link describing the EZSDK memory map?

    http://processors.wiki.ti.com/index.php/EZSDK_Memory_Map

    The next step is to observe the IPC handshake flags in memory. Each processor has a separate handshake flag for each remote processor. During attach and detach, the local processor writes to its own flag and reads the corresponding flag from the remote processor. For your configuration, I believe the flags are located at the following addresses (all numbers are in hex).

    DSP handshake flags
    -------------------------
    9F70_0000  VIDEO-M3
    9F70_0014  VPSS-M3
    9F70_0028  HOST

    VIDEO-M3 handshake flags
    -------------------------
    9F70_0080  DSP
    9F70_0094  VPSS-M3
    9F70_00A8  HOST

    VPSS-M3 handshake flags
    ------------------------
    9F70_0100  DSP
    9F70_0114  VIDEO-M3
    9F70_0128  HOST

    HOST handshake flags
    -----------------------
    9F70_0180  DSP
    9F70_0194  VIDEO-M3
    9F70_01A8  VPSS-M3

    Flag values
    ----------------------
    1 = START
    2 = FINISH
    3 = DETACH

    For example, when the DSP is handshaking with the HOST, the DSP will write into its own HOST flag at 9F70_0028 and read from the HOST flag at 9F70_0180. At the same time, the HOST will be writing into its own flag (9F70_0180) and read from the DSP flag (9F70_0028).

    During detach, each processor should write a '3' into its own flag and wait to see a '3' in the remote flag.

    Use a memory window in CCS (on the DSP side) to see if there is a '3' at address 9F70_0028. You can also view memory from the DEBUGSS_DAP. This view will not be cached. It is a good check to see what's actually in memory.

    One last thought. If the DSP has attached to either M3 processor (I think this is unlikely), then the DSP must detach from the M3 processor before detaching from the HOST. But I don't think the DSP would attach to either M3 processor. You can verify this by looking at the corresponding handshake flags. They should be zero.

    ~Ramsey

  • Ramsey,

    Here I collected some info that may tell how we can fix this one, I am working with a custom board so no JTAG, and no CCS. I used devmem utility to read those registers you were talking about:

    Reads before Firmware Load and everything..

    DSP handshake flags
    / # devmem 0x9F700000 #VICP
    0x00000000
    / #
    / # devmem 0x9F700014 #VPSS
    0x00000000
    / #
    / # devmem 0x9F700028 #HOST
    0x00000000

    HOST handshake flags
    / # devmem 0x9F700180 #DSP
    0x00000000

    Reads after Firmware Load and attached proc on both sides

    DSP handshake flags
    / # devmem 0x9F700000
    0x00000000
    / #
    / # devmem 0x9F700014
    0x00000000
    / #
    / # devmem 0x9F700028
    0x00000002

    HOST handshake flags
    / # devmem 0x9F700180
    0x00000002

    Reads after Firmware Load and deattachment on both sides (first HOST)

    DSP handshake flags

    / # devmem 0x9F700000

    -/bin/sh: /: Permission denied (Funny)
    / #

    #NO VPSS reading

    / # devmem 0x9F700028 #HOST
    0x00000002

    / # 

    HOST handshake flags

    / # devmem 0x9F700180
    0x00000002

    Seems that flags did not change at all after the deatchment. Here I will paste you my deatchment routines:

    ARM Client (this one runs first)

    int CLNT_Unlink()
    {
    vdbg("Entry");
    int status = 0;

    /* invoke the SysLink stop callback */
    status = Ipc_control(Module.ServerProcId, Ipc_CONTROLCMD_STOPCALLBACK, NULL);
    if (status < 0)
    {
    printf("Ipc_control: stop callback failed\n");
    return status;
    }

    vdbg("DSP Firmware stop acknowledged");
    SysLink_destroy();

    return status;
    }

    Dump:

    CLNT_Unlink: Entry
    CLNT_Unlink: DSP Firmware stop acknowledged

     

    DSP Server: 

    int SVRModule_Unlink()
    {
    Log_print0(Diags_ENTRY && Diags_USER1, "Server_Unlink: Entry");

    Int status = 0;

    /* disconnect from remote processor */
    do
    {
    Ipc_detach(Module.ClientProcId);

    if (status == Ipc_E_NOTREADY)
    {
    Task_sleep(100);
    }

    } while (status == Ipc_E_NOTREADY);

    if (status < 0)
    {
    Log_print0(Diags_STATUS && Diags_USER1, "Failed to disconnect from remote process");

    return status;
    }
    Log_print0(Diags_STATUS && Diags_USER1, "Ipc_detach: Done");

    /* NOTE: Only one thread must call stop */
    Ipc_stop();

    Log_print0(Diags_USER1 && Diags_EXIT, "Server_Unlink: Exit");

    return status;
    }

    Dump:

    N:DSP P:0 #:00006 M:xdc.runtime.Main S:Ipc_detach: Done
    N:DSP P:0 #:00007 M:xdc.runtime.Main S:Server_Unlink: Exit

    Interesting that on both sides the logs showed up a successful deatachment, but there is no sign of that change on the memory registers. Wondering if there is someone around blocking write operations after the processor attachment. 

    Looking forward for your comments, thanks!

    -Jose L.

  • Jose,

    I'm confused by your last comment. The trace would indicate that both ARM and DSP have indeed completed the IPC detach. But I thought the problem is that the detach is not completing on the DSP. Please clarify this point.

    One more config param to check is Ipc.sr0MemorySetup. Please check that your DSP configuration contains the following statement.

    var Ipc = xdc.useModule('ti.sdo.ipc.Ipc');
    Ipc.sr0MemorySetup = false;

    I'm also wondering if you have any detach hooks registered.

    I cannot explain the results you are seeing in memory. If the detach really succeeded, you should see a value of 3 at those addresses. I'm wondering what the DSP does after it finishes to detach. Does it just wait to be halted and unloaded? Does the ARM halt the DSP after the detach has completed? Maybe if the ARM halts the DSP too soon, the DSP would not have a chance to write the final value into memory.

    ~Ramsey

  • Ramsey,

    I'm confused by your last comment..

    Yes you are right, sorry for the confusion. Seems that originally my dettachment call did not returned successfully, that is different from my current's week test. The reason I started this post was because I have to load/unload the firmware several times so I can test my services startup on the DSP.

    What happened is that I can achieve 1 firmware load and 1 firmware unload. When I attempt to load the fw the second time the attachment fails, and my guess was because previous detachment did not work correctly, seems to me that that one still is my best guess by now because of the processors handshake flags values.

    The short version is, I want to load/unload the firmware as many times I want instead of rebooting the board, sorry if the post title got outdated :).

    var Ipc = xdc.useModule('ti.sdo.ipc.Ipc');
    Ipc.sr0MemorySetup = false;

    Nope, dont have that one, will have to build and test to see what happens.

     

    I'm also wondering if you have any detach hooks registered.

    If you are talking about hook functions no. I am not using does sort of resources.

     

    I'm wondering what the DSP does after it finishes to detach.

    Before the detach the Firmware tasks are terminated by that time, so the BIOS scheduler should be idle. After that I run the firmware_loader ... stop ... command. I think I am leaving enough time for the DSP to write to memory.

    Last time I unloaded the firmware I got a backtrace dump, dont know if this may give hints. Thanks

     

    / # firmware_loader 0 dspsample.xe674 stop -i2c 0
    FIRMWARE: Memory map bin file not passed
    Usage : firmware_loader <Processor Id> <Location of Firmware> <start|stop> [-mmap <memory_map_file>] [-i2c <0|1>]
    ===Mandatory arguments===
    <Processor Id> 0: DSP, 1: VideoUnable to handle kernel NULL pointer dereference at virtual address 0000000c
    -M3, 2: Vpss-M3 pgd = cdd70000

    <Location of F[0000000c] *pgd=8ddc4031irmware> firmwar, *pte=00000000e binary file
    , *ppte=00000000<start|stop>
    to start/Internal error: Oops: 17 [#1] PREEMPT
    last sysfs file: /sys/devices/virtual/pvr/pvrsrvkm/dev
    Modules linked in: omaplfb pvrsrvkm ti81xxvin ti81xxfb vpss syslink
    CPU: 0 Not tainted (2.6.37-470419-0103.BA_xxx #1)
    PC is at Platform_stopCallback+0x138/0x508 [syslink]
    LR is at Memory_alloc+0x124/0x160 [syslink]
    pc : [<bf018f54>] lr : [<bf02af44>] psr: 80000013
    sp : cdd59de8 ip : 00000000 fp : cdd59e74
    r10: 00000000 r9 : 00000000 r8 : de754000
    r7 : 00000000 r6 : 00000000 r5 : bf0fe8a0 r4 : de754000
    r3 : 00000000 r2 : de754000 r1 : cdd59e20 r0 : de754000
    Flags: Nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
    Control: 10c5387d Table: 8dd70019 DAC: 00000015
    Process firmware_loader (pid: 174, stack limit = 0xcdd582e8)
    Stack: (0xcdd59de8 to 0xcdd5a000)
    9de0: 00000000 cdd59df8 00000000 de754000 00000001 cdd71328
    9e00: bfffe183 bfffe183 cdd59e3c cdd59e18 c00474f4 c0048a34 cdd59ec4 cdd59e58
    9e20: c03f1eb0 c00bde04 00000000 c03f1eb0 cdd59e5c cdd59e40 c0048a4c c0047494
    9e40: 9fca5000 60000113 cdd59fb0 babe0002 00000000 00000000 bf0feaac be8f6bb0
    9e60: be8f6bb0 00000000 cdd59e9c cdd59e78 bf04dda4 bf018e28 babe0002 00000000
    9e80: c014f497 d0a6b800 00000000 00000005 cdd59ef4 cdd59ea0 bf0881cc bf04dce8
    9ea0: c014f497 be8f6bb0 cdd59ec4 cdd59eb8 c008f27c dd500000 babe0002 00000000
    9ec0: 0000a24c bfffeffc c00c3ab4 d17f1ca0 d0a6b800 00000005 00000005 be8f6bb0
    9ee0: cdd58000 00000000 cdd59f04 cdd59ef8 c00df908 bf0880dc cdd59f74 cdd59f08
    9f00: c00e009c c00df8ec cdd59f3c cdd59f18 c00474f4 c0048a34 cdd59f74 cdd59f58
    9f20: c03f1eb0 c00d260c 00000000 c03f1eb0 cdd59f54 cdd59f40 c03f1eb0 c00474c4
    9f40: d0a6b800 00000005 cdd59f74 00000000 be8f6bb0 c014f497 00000005 d0a6b800
    9f60: cdd58000 00000000 cdd59fa4 cdd59f78 c00e013c c00dfb50 00000001 00000001
    9f80: 00000003 00046abc 00000000 00008d8c 00000036 c00450e8 00000000 cdd59fa8
    9fa0: c0044f40 c00e00f0 00046abc 00000000 00000005 c014f497 be8f6bb0 00000005
    9fc0: 00046abc 00000000 00008d8c 00000036 00000000 00000000 400a0000 be8f6b94
    9fe0: 00000000 be8f6b78 0002f098 402e6aec 20000010 00000005 00000000 00000000
    Backtrace:
    [<bf018e1c>] (Platform_stopCallback+0x0/0x508 [syslink]) from [<bf04dda4>] (Ipc_control+0xc8/0x14c [syslink])
    [<bf04dcdc>] (Ipc_control+0x0/0x14c [syslink]) from [<bf0881cc>] (IpcDrv_drvioctl+0xfc/0x5e8 [syslink])
    r7:00000005 r6:00000000 r5:d0a6b800 r4:c014f497
    [<bf0880d0>] (IpcDrv_drvioctl+0x0/0x5e8 [syslink]) from [<c00df908>] (vfs_ioctl+0x28/0x44)
    [<c00df8e0>] (vfs_ioctl+0x0/0x44) from [<c00e009c>] (do_vfs_ioctl+0x558/0x5a0)
    [<c00dfb44>] (do_vfs_ioctl+0x0/0x5a0) from [<c00e013c>] (sys_ioctl+0x58/0x7c)
    [<c00e00e4>] (sys_ioctl+0x0/0x7c) from [<c0044f40>] (ret_fast_syscall+0x0/0x30)
    r8:c00450e8 r7:00000036 r6:00008d8c r5:00000000 r4:00046abc
    Code: eb00421f ea0000bf e595302c e0833006 (e1d300bc)
    stop the firmwar---[ end trace bdd20dfeecf23659 ]---
    e
    ===Optional arguments===
    -mmap input memory map bin file name
    -i2c 0: i2c init not done by M3, 1(default): i2c init done by M3
    FIRMWARE: isI2cInitRequiredOnM3: 0
    FIRMWARE: Default memory configuration is used
    Firmware Loader debugging not configured
    Default FL_DEBUG: warning
    Allowed FL_DEBUG levels: error, warning, info, debug, log
    MemCfg: DCMM (Dynamically Configurable Memory Map) Version : 2.1.2.1