Undefined reference to ipc_gates and ipc_heaps

I found a combination of versions of xdc, ipc, and bios that builds without error. However it doesn't run properly, only Core1 seems to generate output, and it fails with the first attempt to notify. Never get anything out of Core0 or any of the others. Trying to dig into why that would be.
Mike

Was your previous undefined reference problem fixed by updating XDC?

Feel free to post your project and/or source code and I can take a look and advise about the issue.

Regards,

- Rob

I'm not completely sure what fixed it, but the XDC version is now 3.30.5.60 and Bios is 6.41.0.26   These are theoretically not supported by IPC 1.24.3.23 which is what I'm using, but this is the combination that I found that worked.  And, I closed and re-opened CCS.  It seems that occasionally helps when things are not making sense, but I can't prove it, I've not done careful, repeatable, testing.  I regularly get JVM Heap Low messages from Windows notifications, and when I do, I close CCS and reopen it.

In the .cfg there is this:

/*
 *  Since this is a single-image example, we don't (at build-time) which
 *  processor we're building for.  We therefore supply 'null'
 *  as the local procName and use MultiProc_setLocalId to set the procId
 *  at runtime.
 */
MultiProc.setConfig(null, nameList);

---

However, just discovered that nowhere in main() or anywhere else in the source is there a call to MultiProc_setLocalId()

Here is main():

/*
 *  ======== main ========
 *  Synchronizes all processors (in Ipc_start), calls BIOS_start, and registers 
 *  for an incoming event
 */
Int main(Int argc, Char* argv[])
{
    Int status;
    UInt numProcs = MultiProc_getNumProcessors();

    /*
     *  Determine which processors Notify will communicate with based on the
     *  local MultiProc id.  Also, create a processor-specific Task.
     */
    srcProc = ((MultiProc_self() - 1 + numProcs) % numProcs);
    dstProc = ((MultiProc_self() + 1) % numProcs);

    System_printf("main: MultiProc id = %d\n", MultiProc_self());
    System_printf("main: MultiProc name = %s\n", 
        MultiProc_getName(MultiProc_self()));
    
    /*  
     *  Ipc_start() calls Ipc_attach() to synchronize all remote processors
     *  because 'Ipc.procSync' is set to 'Ipc.ProcSync_ALL' in *.cfg
     */
    status = Ipc_start();
    if (status < 0) {
        System_abort("Ipc_start failed\n");
    }

    /*
     *  Register call back with Notify. It will be called when the processor
     *  with id = srcProc sends event number EVENTID to this processor.
     */
    status = Notify_registerEvent(srcProc, INTERRUPT_LINE, EVENTID,
                                  (Notify_FnNotifyCbck)cbFxn, NULL);
    if (status < 0) {
        System_abort("Notify_registerEvent failed\n");
    }

    BIOS_start();
    
    return (0);
}

Could this be the issue?

Mike

Mike Stich said:

In the .cfg there is this:

/*
 *  Since this is a single-image example, we don't (at build-time) which
 *  processor we're building for.  We therefore supply 'null'
 *  as the local procName and use MultiProc_setLocalId to set the procId
 *  at runtime.
 */
MultiProc.setConfig(null, nameList);

---

However, just discovered that nowhere in main() or anywhere else in the source is there a call to MultiProc_setLocalId()

MultiProc_setLocalId() iscalled by MultiProcSetup_init().

MultiProcSetup_init() is called from the XDC startup array, which is accomplished by this statement in <ipc>/ti/sdo/ipc/family/tci663x/MultiProcSetup.xs:
    Startup.firstFxns.$add(MultiProcSetup.init);
That statement is called if MultiProcSetup.configureProcId = true, which is the default value so make sure you're not setting that to false.

That statement results in the following generated content in <ipc>/ti/sdo/ipc/examples/multicore/evm667x/package/cfg/ti_sdo_ipc_examples_platforms_evm6678_core0/release/message_multicore_pe66.c:
/* --> xdc_runtime_Startup_firstFxns__A */
const __T1_xdc_runtime_Startup_firstFxns xdc_runtime_Startup_firstFxns__A[3] = {
    ((xdc_Void(*)(xdc_Void))((xdc_Fxn)ti_sysbios_heaps_HeapMem_init__I)),  /* [0] */
    ((xdc_Void(*)(xdc_Void))((xdc_Fxn)ti_sdo_ipc_family_c647x_MultiProcSetup_init__I)),  /* [1] */
    ((xdc_Void(*)(xdc_Void))((xdc_Fxn)ti_sysbios_hal_Hwi_initStack)),  /* [2] */
};

The "firstFxns" are called before main, so that is probably not the issue, and main()'s MultiProc_self() calls should be returning the right thing.

Your runtime problem is probably due to the user of newer, unsupported XDC & BIOS versions.  Can you go back to using the ones that were "supported" in IPC 1.24.3.32?  Those are XDC 3.23.03.53 and BIOS 6.33.05.46.  We can then focus on your original build problem if it's still there.  Make sure you clean what you can before pointing to those versions.

Re: your original build problem...

Those "undefined reference" symbols should be defined in the same place that they're referenced, in the notify_multicore_pe66.c file.  Can you search for one of the undefined reference symbols in that file, and see if the *definition* is present (in addition to the reference).

Regards,

- Rob

I'll try getting those releases.  The IPC release notes do say "or later". 

I have XDC 3.23.4.60 and BIOS 6.33.6.50, but when I try that combination in the RTSC configuration tab of the CCS General page in properties, there are no platforms that can be selected.  If I manually type "ti.platforms.evm6678", the problem window immediately comes up with :

XDC runtime error: org.mozilla.javascript.Undefined@187d6c5: no property named 'xdc.cfg.Program'    notify_multicore.cfg    /NotifyExample    Configuration Validation    XDCTools Configuration Marker

I do have an IPC 3.22.02.11, but that seems completely different.  Doesn't have the example in it, and the directory structure is all different.  Am I using the correct IPC for the 6678?

Thank you much for the help.

Mike

The sysbios download page link to 6.33.05.46 seems to be broken. It does indicate that 6.33.06.50 (which I have) is recommended with IPC 1.24.3.23

I've come a full circle to the original runtime error.

Got a clean build with SYSBIOS 6.33.06.50 and XDC 3.23.04.60   The problem was that I had accidentally checked "TI_RTOS" on the RTSC tab, and that apparently was including things that caused the errors.  

This example is supposed to send console output from each core sequentially, using IPC notify to trigger the next core in the sequence.  I get no output from any core except Core1, and it aborts.  This is the Console output:

[C66xx_1] main: MultiProc id = 1
main: MultiProc name = CORE1
tsk1_func: Received request #1 from CORE0
sendEvent failed

All the other cores appear to be running in the idle task.  Core1 does an abort exit.

I copied the notify_multicore.c and notify_multicore.cfg to my CCSv6 project and build and debug from there.  (I'd attach the 2 files, but I can't seem to find a button to click on to attach files to this post; the only options seem to be media files.  Should I paste the entire files?)

Thanks for the help.

Mike

Mike Stich said:

This example is supposed to send console output from each core sequentially, using IPC notify to trigger the next core in the sequence.  I get no output from any core except Core1, and it aborts.  This is the Console output:

[C66xx_1] main: MultiProc id = 1
main: MultiProc name = CORE1
tsk1_func: Received request #1 from CORE0
sendEvent failed

All the other cores appear to be running in the idle task.  Core1 does an abort exit.

All the cores should have the "main:" output at least.  And since CORE1 received a Notify event then CORE0 should have the "tsk1_func: Sent request #1 to CORE1" output.  Perhaps you see only CORE1's output because it failed and possibly did a System_flush() call during abort (just a guess at this point).

At this point of failure it makes sense that all the other cores are running the idle task, since they're all waiting for a Notify callback that will post a Semaphore.

Mike Stich said:

I copied the notify_multicore.c and notify_multicore.cfg to my CCSv6 project and build and debug from there.  (I'd attach the 2 files, but I can't seem to find a button to click on to attach files to this post; the only options seem to be media files.  Should I paste the entire files?)

No need to post those files, I have them in my copy of IPC 1.24.03.32 (unless you changed something, in which case you can just post what you changed).

The Forums recently got a facelift, so I don't know why posting source files is now difficult.  I can look into that.

So, what platform or device are you using?  Is this a pure "DSP" run or are the ARM cores also running SYS/BIOS?

Are you building with CCS or with command line "make" tooling?  How are you loading the cores?

Most likely something is wrong with your platform or device.  For a 6638, Notify should be using the NotifyDriverCirc driver, which looks like it can't fail when the 'waitClear' parameter of Notify_sendEvent() is TRUE and you're Notifying a different core, which it is in this example.

And to answer your questions from another post ... IPC 1.24.03.32 is fine for Keystone II, as are all later IPC versions up to the current IPC 3.23.  You're correct that IPC 3.x has quite a different directory structure and doesn't have this example, but it should have a similar one.  Is there a reason you want to use 1.24?

Regards,

- Rob

The device is a TMS320C6678, and the board at this point is the EVM for that platform. This I believe is a Keystone I. All cores are homogenous, C66 DSP devices, no ARM core involved. The only reason I'm using IPC 1.24.03.32 is because it had the example that the IPC documentation I have pointed to. I would gladly use the most recent version of all the tools, but I've been pretty confused as to what works with which device/bios/xdc/CCS versions. It has seemed to me that not all tool versions are backward compatible. I.e. new/old bios needs to be matched with new/old xdc. Not sure about IPC and other tools. What versions for these would you suggest for the Keystone I TMS320C6678 and CCSv6?

Discovered something interesting on the run time error. Loaded all the cores and ran them to main. Put a breakpoint on Core1 only, just before the send event that was failing. All the parameters looked fine. Removed the breakpoint and resumed. All the output appeared on the console correctly (well, all jumbled and out of order, but that is expected). So I'm guessing there is either some sort of timing problem or an output buffer is overflowing, or something along those lines with the CIO. Any idea what might fix this? Is some initialization not finishing in time? I'd like to understand it so I don't step into this particular mess with my application code. If it is just something involving the CIO, I won't worry too much.

Mike

Mike Stich said:

So I'm guessing there is either some sort of timing problem or an output buffer is overflowing, or something along those lines with the CIO. Any idea what might fix this

Yeah, CIO is probably getting in the way here.  It's accomplished by having an internal BP, so each System_printf() statement is stopping the core, doing some CCS memory access, and then restarting the core.

You can change the example to use SysMin instead of SysStd in notify_multicore.cfg.  When doing so, you would run your program and then use ROV to view the SysMin output buffer.  It is much less intrusive and doesn't involve internal BPs.

You could also change the example source code to use Log_printf() instead of System_printf().  SysMin is less intrusive than SysStd, but SysMin is still printf-based, whereas Log is really fast on the target and leaves the printf-type formatting to the host (CCS, in this case).

Regards,

- Rob

 

I did change to SysMin a while back so this problem happened with it. Not sure what you mean by using ROV, would have to remove the printf's to avoid the breakpoints, but then how would the data get to the SysMin output buffer to be viewed?

I'm not familiar with Log_printf(), I'll try and investigate how to use that.

Thanks much. Mike

Mike Stich said:

I did change to SysMin a while back so this problem happened with it. Not sure what you mean by using ROV, would have to remove the printf's to avoid the breakpoints, but then how would the data get to the SysMin output buffer to be viewed?

Hmmm, maybe this is the issue then...

SysMin causes System_printf() output to be rendered to a buffer that's internal to the SysMin module.  You won't see the output in the CCS Console, but instead need to view the output with the RTOS Object View, or ROV.  ROV is available in the Tools top-level menu.  You would open it and then select the SysMin module in the vertical list, and look at that module's content.

There are no BPs associated with SysMin.

So, perhaps you have SysStd configured for just CORE1 (somehow, even though there's usually just one .cfg file and one .xe66 to be used for all cores) and SysMin for the others and that's why you're seeing just CORE1 output in the CCS console.

I'm hoping that the issue is just with these CIO BPs messing up the timing such that the app failes on CORE1.

Regards,

- Rob

Well, that is definitely not what I'm seeing.  I used SysMin in a different, non-multicore project with a PDK platform_printf() that basically mapped to printf, and the output showed up on the console.  And in this example with System_Printf, if I put the breakpoint in Core1 and then remove it after it was hit and resume execution, all the output shows up in the console correctly, from all the cores. So it seems to me SysMin is writing to the CCS console.  Would the XDC version I'm using matter?

In any case, here is my .cfg file, if it helps.  There is no reference to SysStd in it. 

Thanks much.    Mike

---

/* 
 * Copyright (c) 2012, Texas Instruments Incorporated
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 * *  Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 *
 * *  Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 *
 * *  Neither the name of Texas Instruments Incorporated nor the names of
 *    its contributors may be used to endorse or promote products derived
 *    from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
 * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
 * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
 * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
 * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 * */
var MultiProc = xdc.useModule('ti.sdo.utils.MultiProc');
var SysMin = xdc.useModule('xdc.runtime.SysMin');
SysMin.bufSize = 2048;

/* 
 *  Get the list of names that the build device supports.
 *  I.e. ["CORE0", "CORE1", "CORE2" ... ]
 */
var nameList = MultiProc.getDeviceProcNames();

/* 
 *  Since this is a single-image example, we don't (at build-time) which 
 *  processor we're building for.  We therefore supply 'null' 
 *  as the local procName and use MultiProc_setLocalId to set the procId
 *  at runtime.
 */
MultiProc.setConfig(null, nameList);

var System   = xdc.useModule('xdc.runtime.System');
//System.SupportProxy = xdc.module("xdc.runtime.SysMin");
System.SupportProxy = SysMin;


/* Modules explicitly used in the application */
var Notify      = xdc.useModule('ti.sdo.ipc.Notify');
var Ipc         = xdc.useModule('ti.sdo.ipc.Ipc');
var BIOS        = xdc.useModule('ti.sysbios.BIOS');
var Task        = xdc.useModule('ti.sysbios.knl.Task');

BIOS.heapSize   = 0x8000;

var tsk0 = Task.create('&tsk0_func');
tsk0.instance.name = "tsk0";

/* To avoid wasting shared memory for MessageQ transports */
for (var i = 0; i < MultiProc.numProcessors; i++) 
{
  Ipc.setEntryMeta(
      {
        remoteProcId: i,
        setupMessageQ: false,
      }                              );
}

/* Synchronize all processors (this will be done in Ipc_start) */
Ipc.procSync = Ipc.ProcSync_ALL;

/* Shared Memory base address and length */
var SHAREDMEM           = 0x0C000000;
var SHAREDMEMSIZE       = 0x00200000;

/* 
 *  Need to define the shared region. The IPC modules use this
 *  to make portable pointers. All processors need to add this
 *  call with their base address of the shared memory region.
 *  If the processor cannot access the memory, do not add it.
 */ 
var SharedRegion = xdc.useModule('ti.sdo.ipc.SharedRegion');
SharedRegion.setEntryMeta(0,
						    { base: SHAREDMEM, 
						      len:  SHAREDMEMSIZE,
						      ownerProcId: 0,
						      isValid: true,
						      name: "MSMC_RAM",
						    }                             );

/* Create a semaphore with count 0 */
var Semaphore = xdc.useModule('ti.sysbios.knl.Semaphore');
Program.global.semHandle = Semaphore.create(0);

System.maxAtexitHandlers = 4;

Mike Stich said:

And in this example with System_Printf, if I put the breakpoint in Core1 and then remove it after it was hit and resume execution, all the output shows up in the console correctly, from all the cores. So it seems to me SysMin is writing to the CCS console

I looked into SysMin a bit more...

SysMin causes System_printf() to output to the internal buffer.  SysMin_flush() will fwrite() the buffer to stdout if the user doesn't set a SysMin.outputFunc, which is what causes the CCS console to be used.  This outputFunc is called by System_flush() when SysMin is configured, so probably a System_flush() is being performed when your program exits.  Having said that, I just looked at SysMin.c and it calls SysMin_flush() for aborts and exits.

So I think that explains why you're getting console output for just CORE1 when CORE1 aborts on the failed sendEvent, due to the SysMin_flush() called by SysMin_abort().  And when you add your CORE1 BP and then resume and the example finishes, then all the other cores' output comes on the console due to a System_exit() being performed when they successfully complete.  However, that doesn't explain why CORE1 is failing, though, since presumably there's been *no* CIO (i.e., console) output to that point.  But the fact that a BP on one core allows all the others to operate correctly indicates that the failure is due to a subtle timing issue.

How are you loading the DSP cores?  It might be important that CORE0 starts first.

Regards,

- Rob

Being new to the multicore environement, all I did was in CCSv6, load all the cores and run to main.  Then with the cores all set as a group, I "resumed" (pressed the green arrow") the entire group at once.  I'm guessing I could start the cores manually in sequence, but that would mean a lot of time between each one starting.  Not sure what the "right" way to start the cores is in this case.

Thank you.

Mike

I'll check out the return code on the sendEvent. Thanks for the info.
Mike

Rob,
The return code from Notify_sendEvent() was Notify_E_EVTNOTREGISTERED. This sounded a lot like Notify_E_NOTINITIALIZED, so I spun on the Notify_sendEvent() for either of those return codes and it now works. Not sure if that is the recommended way to synchronize the initialization, but it seems to work.

Thank you much for sticking with this. It has helped me understand. I find the whole multicore/RTOS environment pretty confusing.
Mike