C6000 Compiler version 7.6.0

The problem seems to be related to C runtime initialization order. I didn't find the exact root cause, but found the problematic global  C++ class instances. Moving their construction into a function which is waked from xdc.runtime.Startup.lastFxns solved the problem.

The DSP startup sequence is as following (http://rtsc.eclipse.org/cdoc-tip/xdc/runtime/Startup.html): 

1. CPU is initialized and initial C stack setup is performed.
2. The function specified by Startup.resetFxn is called. Startup.resetFxn is called only on platforms where reset is performed before running a program. For example, boot code for all TI targets invokes Startup.resetFxn, but this function is not invoked on Microsoft targets.
3. C runtime initialization is performed.
4. Functions from the array Startup.firstFxns are called.
5. All Mod_Module_startup functions (see Module Initialization below) are called in a loop until all such functions return Startup_DONE or the maxPasses threshold is reached.
6. Functions from the array Startup.lastFxns are called.
7. The function main is called.

So the initialization of those classes was moved from #3 to #6.

Vesa,
what's in that code that was moved from #3 to #6? Does it reference any RTSC modules, which are initialized in #5? Also, when you say that image does not boot, did you mean to say that your executable doesn't reach main()?

Sasha,

yes, in the booting I don't reach main(). It seems that there is more to this issue. The moving of the mentioned classes solved my problem only temporarily. Now I've pulled more symbols from RTS library (--export option in the linking of the baseimage), and the same problem happens again.

Found out that at leats this symbol triggers the problem: _ZNSt8ios_base5_InitEv ( std::ios_base::_Init()  ).

When I build the DSP image without --dynamic=exe option, it boots ok all the time. No matter how much symbols are exported. When the dynamic exe option is enabled is face this problem (only with compiler 7.6.0 version). With 7.4.7 it works fine.

One  more observation:

1. I compiled our code base with 7.6.0, but linked it with 7.4.7. The resulted image works fine.

Something has changed in the linking, but I'm not saying it's a linker bug :) It might be that I'm using the tool with wrong options.

You may have a problem with the loader (see SDSCM00042973) not zero-initializing all uninitialized memory before the program starts.  This sometimes results in global C++ objects failing to construct properly upon startup.  The object you mention is required for proper C++ I/O.  Try adding to the linker command file a fill value of 0 for all uninitialized sections to see if the symptom goes away.

One thing to be aware of is that when --dynamic is used, variables with initial values do not get those values via .cinit decompression; instead, the loader initializes them directly from the .out file, which contains an initialized image of their sections (.data, .fardata and so on). This is not necessarily intentional behavior -- I discovered it while looking into this query. However, this has been true since before 7.4.x, so any bearing on this issue is probably indirect. But it may explain why the program boots without --dynamic. It also could relate to Archeologist's point, in that without .cinit processing, zero-initialization may not be happening either. It could be that it worked by accident with 7.4.7.

I can't see any obvious change in the linker between 7.4.0 and 7.6.0 that would explain this. I think the use model is correct; that is, I don't think additional options are needed. Any program that loads without --dynamic should load and execute the same way with --dynamic=exe (subject to the caveats above). Use of --dynamic simply adds meta-data to the .out file that provides for post-link relocation and linkage with other modules.

It may be worth using ofd6x -v to examine the .out files resulting from the two versions, and seeing if there is any difference that could explain the change in behavior.

Thanks a lot for this information!

1. I've defined in the linker "--fill_value=0", should this achieve the same as "try adding to the linker command file a fill value of 0 for all uninitialized sections"?

2. Still looking to the outputs of "ofd6x -v ", so far found one difference (the code is compiled with 7.6.0 and linked once with 7.4.7 and once with 7.6.0 with --dynamic=exe):

With 7.6.0 it says to ".cinit"  "Loaded Onto Device:  Yes" and with 7.4.7 "Loaded Onto Device:  No". 

What does this mean?

-vesa

left side = 7.6.0, right = 7.4.7 ;)

I'm not sure what causes that difference but it should be innocuous. In cases the size of .cinit is 0, so the loader should ignore it regardless of the flags. I don't think that's our issue but we may want to come back to it if nothing else pans out.

As I mentioned --dynamic=exe suppresses initialization via .cinit. You can simulate the same effect without --dynamic by using the options -cr --cinit_compression=off. The -cr option means "RAM model initialization", that is, direct from the .out file. The other option disables compression. The reason to try this (without --dynamic) would be to see if there is something about this form of initialization that is causing the issue, or some other effect of --dynamic.

Do you know if any of the other startup steps you posted manipulate the memory system? The reason I ask is that "direct" (no .cinit) initialization happens when the file is loaded, whereas .cinit-based initialization happens when the program starts to run (just before main).

Of course all this could be a red herring.

With  "-cr --cinit_compression=off" instead of "--dymic=exe" my DSP image loads ok :)

But I cannot compile my dynamic libraries against it. The linker complains: 

fatal error: file "../dsp/Debug/DspBaseImage.out" could not be statically or
dynamically linked

when linked with 7.4.7, there is much more dynamic symbols (ofd6x -v):

What can cause this difference?

Right, without --dynamic you won't be able to link against it. That was just an experiment to isolate the initialization behavior, which is apparently not the issue.

So there must be some other change between 7.4 and 7.6 that is exposed by --dynamic. I'm not sure what that could be.

It may be useful to compare the ofd6x output from the two versions. Can you attach them? If that does not reveal anything it may be necessary to step through the startup sequence and see why things are going south.

(I am leaving the office soon so I will have to pick this up tomorrow.)

I've to ask permission to post those :)

One more difference that I noticed, is that in  "String Table ".dynstr"", there are no entries like this in 7.6.0 output:

 145  "__TI_exidx_linkto_scn_start_1"

In " String Table ".strtab"" these are in both:

Thanks a lot so far!

Btw, my device is EVMTCI6638K2K, and I'm loading the dsp image using mpmcl.
I've not suspected mpmcl because it has been working fine so far. Can there be something with 7.6.0 and mpmcl combination?

The presence of the __TI_exidx symbols in the dynamic symbol table (.dynsym section) and dynamic string table (.dynstr) should not affect anything at run time. They are symbols defined by the linker as part of the C++ exception handling mechanism. A change was made to the linker in 7.6 that removes linker-defined symbols from the dynamic tables if they are not exported, since there is no reason whatsoever to publish them.

I put together a simple example that uses C++, exceptions, and links with --dynamic, and compared the output using 7.4 and 7.6. I also spent some time looking at all the linker changes between 7.4 and 7.6 to see if anything stood out. Other than the things we have already explored I did not see any differences that would explain the behavior we are seeing. So I am also beginning to suspect that something is happening between mpmcl and the .out file that is exposed by 7.6 with --dynamic, or a spurious dependence on zero-initialization as Archeologist brought up. 

Can you try some things to isolate what's going wrong in the boot sequence? One thing to try would be to see if a simple C-only program when linked with --dynamic. If it does, gradually make it more complicated by adding C++, iostreams, etc.

Earlier you mentioned that std::ios_base::_Init() was a trigger somehow, but I did not understand the comment. I took it mean that you called that function explicitly by registering it as a RTSC lastFxn startup? I wonder if there is some interference or dependence between the RTSC startup and the C++ constructor calls.

The mechanism for calling functions before main() is to enter them in a table called __TI_INITARRAY_Base in the .iinit_array section. I presume that RTSC simply puts its Startup_exec() in that table. It may be useful to look at that table and see if there are any ordering differences between the working and non-working versions, either 7.4 vs 7.6 or --dynamic vs no --dynamic.

I am not as up to speed on RTSC and the MCSDK as I should be, so I may be of limited help beyond the linker itself.

In talking to Archeologist it seems that lack of zero-initialization could very conceivably cause the behavior you are seeing. In C++ the construction of global objects is guarded by a static variable, conceptually

int init_guard = 0;
init_fn() { if (init_guard == 0) { call constructor; init_guard = 1; }

If the guard is not properly initialized the constructor may not get called, causing subsequent initializations that depend on that object to crash.

Zero-initialization can be forced in the linker command file by adding fill = 0 to uninitialized sections containing C variables, typically .bss and .far. It would look something like:

SECTIONS
{
   .bss  > RAM, fill = 0
   .far   > RAM, fill = 0
}

The --fill_value option, as you proposed (above, May 01 at 16:59), is not enough: it only supplies a default value to use for filling; it does not cause uninitialized sections to be filled.

Let's try that and see what happens. 

Vesa, any update? 

No big news so far. I'll come back to this issue soon. I've verified that loading works fine with CCS even when linked with 7.6.0.

I'll try today the zero fill that you suggested.

No luck so far. 

Filling didn't help and also we updated the MCSDK  /Arago in hope that a newer version of the mpmcl could solve the issue. Didn't help either.

Currently I'm proceeding so that the compilation is done with 7.6.0 and linking with 7.4.7. I hope that we could find a solution to this issue, so no need for manual linking stage.

Hello,

I was able to generate an example project in which we can see the same behavior as with our project. See the attached archieve (DynamicExe7.6.0.tar.gz). Hopefully this will help to find out was the root cause of the problem :)

The things that trigger the problem in this example are:

1. The exe is linked with dynamic flag (--dynamic=exe)

2. export some certain symbols from rts library (see export_symbold.cmd in the project).

When loading this project with mpmcl, it never finished the load command:

  root@keystone-evm:~# mpmcl reset dsp1
  reset succeeded
  root@keystone-evm:~# mpmcl load dsp1 DynamicDspExe.out

Verified also, that when I link this project with 7.4.7, the loading succeeds with mpmcl:

  root@keystone-evm:~# mpmcl reset dsp1
  reset succeeded
  root@keystone-evm:~# mpmcl load dsp1 DynamicDspExe.out
  load succeeded

7115.8078.DynamicExe7.6.0.tar.gz

Addition to the point 2 above: the symbol must be C++ class related. I don't know what has to be in that class in order to the problem appear.

Here are also the out files for the example project. Both compiled with 7.6.0.

4137.6404.DynamicDspExe_7.4.7.out.tar.gz

0827.1307.DynamicDspExe_7.6.0.out.tar.gz

Vesa, thanks for the test case. I glanced at the .out files but nothing stands out. I suspect mpmcl. I am contacting the MCSDK team to see if they can confirm.

Hello Rex,

We have a C++ code base that does not compile with 7.4.7 because of (at least) two C++ issues with it. With 7.6.0 the code base compiles without problems.  One of the issues we faced with 7.4.x has bug reported: SDSCM00047480. For the another one I've not seen a report in clearQuest.

Currently we are working so that we compile with 7.6.0 and link with 7.4.7.  So the development can continue for now. 

-vesa

Vesa,

Can mpmcl load the binary compiled with 7.6, but linked using 7.4? Could I have that binary and try it?

Thanks!

Rex

Yes it can. Check my post on  May 13 2014 20:30 PM. There are two out files attached, both compiled with 7.6. One linked with 7.6 and another with 7.4.7.

Hi, Vesa,

Do you happen to have the .map files for both the .out files you posted? They will be helpful in debugging the mpmsrv. Thanks!

Rex

Hi Rex,

I am taking over this issue from our side since Vesa has moved to other challenges. Please find the attached .map files you requested. The compilation is done with C6000 7.6.0 compiler and the linking with either

1) 7.6.0 linker

8780.DynamicDspExe_7_6_0.map.txt

2) 7.4.4 linker

6557.DynamicDspExe_7_4_4.map.txt

Vesa was using linker from C6000 7.4.7 tools but it seems that .out file works fine as long as it is linked with any of 7.4.x linkers.

Regards
Marko

Marko,

Thanks. We'll see what we get. So far, this IR is targeted to be fixed in MCSDK 3.2.0 release, but we still need to root cause it first.

Rex