Binary Comparison for builds with same source mismatch

What is the version of the compiler you use?  Please see this wiki article for more detail about tools versions.

Thanks and regards,

-George

Based on what we know so far, we do not know the root cause of the bug, nor do we know exactly what might fix it.

I have two suggestions to consider.  They both involve upgrading the compiler to a more recent version.

We are aware of one bug which roughly meets this description.  The first release which fixes this bug is 8.0.x.  Are you able to upgrade to this release?  To understand what such an upgrade means, please see this wiki article.  Also, 8.0.x only supports EABI, and not the older COFF ABI.  If you build for COFF ABI, then you cannot use 8.0.x.

A lower risk upgrade is to change to version 7.3.23.  This is the latest release in the 7.3.x release stream.  The only difference from 7.3.1 is bug fixes.  There is no guarantee it fixes your particular problem.  And we know it has the problem that is not fixed until 8.0.x.  

Thanks and regards,

-George

George,

Thank you for your reply. Upgrading our CCS environment would involve many man hours of testing and integration. Perhaps, the situation we're seeing can be mitigated by another means, that would instruct, or trick the compiler/linker into working around it. This may be more optimistic, than realistic, granted.

Is there a link you could provide in your next reply to Texas Instrument's bug database that would hold a description of the issue you may think is related ?

Thanks in advance,

-Will

William Martin said:

Is there a link you could provide in your next reply to Texas Instrument's bug database that would hold a description of the issue you may think is related ?

We do have an entry in the SDOWP system.  But it needs some changes and additions to make it usable by general customers.  Rather than do that, it is easier for me to summarize it here.  A customer experienced the compiler generating different assembly output on builds that should see no difference.  It was difficult for us to reproduce and track down.  But we finally found the root cause.  The cause lies in an error deep in the infrastructure of the compiler.  Uninitialized memory was being read.  The compiler infrastructure has since undergone a significant change.  The customer agreed to upgrade to a version of the compiler which uses the new infrastructure.  Thus the bug in the old infrastructure was not fixed.

William Martin said:

Perhaps, the situation we're seeing can be mitigated by another means, that would instruct, or trick the compiler/linker into working around it.

Unfortunately, this is not practical.  The uninitialized memory read occurs in a processing phase that is many times removed from the C source.  It is not possible to say, with any certainty, how it can be avoided.

Thanks and regards,

-George

Hi George,

     Thanks. 

1.  Can you tell me which version of Code Composer Studio would hold the CGT tools version you mentioned above?

2.  Can you zip up the example project/source which can cause the issue to occur?

Will  

William Martin said:

Can you tell me which version of Code Composer Studio would hold the CGT tools version you mentioned above?

The bug was reported in C2000 compiler version 5.2.15.  All TI compilers, including those for C2000 and C6000, are implemented from a common code base, with customization as needed.  The bug is from the common code.  CCS ships on an independent schedule, and uses whatever compiler versions are most recent at the time of release.  This compiler version probably went with CCS 5.5, though it might be CCS 5.4.

William Martin said:

Can you zip up the example project/source which can cause the issue to occur?

It's a C2000 project.  And it is customer code that I cannot make available.

Given that you have focused tightly on this one bug, I should point out that we do not know if this bug is the cause of your problem.  It could be something else entirely.

Thanks and regards,

-George

George,

Does this issue occur only when optimization settings are turned on? I'm experimenting with my build to isolate the factors further and didn't see the binary comparison between two unoptimized builds change for 6 iterations. I wrote a script to get a larger sample.

Unoptimized settings are -g, clear the size, speed settings and then in the optimization tab, clear the setting for optimize (from -O5). 
 
thanks,

Will

I don't recognize ...

William Martin said:

C60_abi.c:356 internal warning #10282: (".bss")
C60_abi.c:356 internal warning #10282: (".neardata")
C60_abi.c:356 internal warning #10282: (".rodata")

I presume these appear in the Console window when you build.  It would be good to see the entire console output from a build.  Create the text file with the icon in the upper right bar of the Console view titled Copy Build Log.

To see these ...

William Martin said:

Undefined symbol
_________________
__c6xabi_unwind_cpp_pr0
__c6xabi_unwind_cpp_pr3
__c6xabi_unwind_cpp_pr4
__cxa_allocate_exception
__cxa_throw

... you must be building C++ code with exceptions.  Is that correct?  Those symbols are defined in the RTS library.  This library is not shipped with the compiler installation, but the linker should automatically build it for you when needed.  That must not be happening for some reason.  Seeing all the console build output should help explain what happened.

Thanks and regards,

-George

George Mock said:

C60_abi.c:356 internal warning #10282: (".bss")

The internal text for that error message is: "While defining Static Base symbol, .bss not placed. It must have a run address to allow definition of __TI_STATIC_BASE__."  Make sure your linker command file explicitly places the .bss section.

COFF ABI is not supported in the C6000 compiler versions 8.0.x and higher.

Thanks and regards,

-George

George,

    Thank you.  I believe your first answer pertained to our situation, where a person would stay at CCS v.5, using CGT 7.x.    COFF may fix the problem, or it may not. 

     I have had a chance to migrate our projects over to CCS v.6, but am waiting on a few outside libraries to be built with conformity to the toolset.  I'm getting linker errors stating I can't use these libs built with CGT 7.3x. 

Thanks,

Will  

Just another data point, I'm seeing the same CRC issue with COFF formatted output files. I switched to use COFF in my projects and its still there. No reply needed from TI support, just another factoid on my current situation. I still have not gotten all the CCS 6.x build put together. We'll have an answer next week.

Archaeologist,

From another post on Saturday(?), I can't see the posting in this view. I see you were trying to accumulate all the facts about the problem.

1.  Here's an example of the re-ordering (of registers only), from two actual builds.  There are much more changes in the Assembly dump, but this was the easiest to hand-copy from our closed system.   I've seen instructions actually be re-ordered as well.

File1.asm , build 13

CALLP                   .S2                         __mpyd, B3

||           MV                        .L2X                      A5, B5                                 ; | 219|

||           MV                        .L1                         A11, A4                              ; | 219|

||          MV                        .S1                         A10, A5                               ; | 219|

File1.asm, build 14

CALLP                   .S2                         __mpyd, B3

||           MV                        .L2X                      A5, B5                                 ; | 219|

||           MV                        .L1                         A11, A5                              ; | 219|

||          MV                        .S1                         A10, A4                               ; | 219|

2.  The object file sizes are different.  The dump of the assembly of those object files shows re-ordering of assembly, different units (in some cases), and different registers being used.

3.  I've reproduced the problem in both COFF/ELF formatted output now with CGT 7.3.1 (same listed above). 

4.  Still working on the CGT 8.x solution.  Sometime this week we'll have the answer.

5.  Optimization seems to make the problem happen.  I haven't seen the problem in an unoptimized build. 

That's all I can remember from your post on Sat.  Let me know what else you need to match this up to the SDO you have. 

-Will

Archaeologist,

During some analysis of assembly, I see types of data that either use math functions (e.g., sqrt) or are const float in a namespace get re-ordered in the generated assembly code most frequently.
E.g.,

filename: MyTypesFile.hpp

#ifndef guarded

#define guarded

namespace MyVariables
{
const float GENERIC_SPEED_OF_HYPERLOOP = 714.123; // mph
const float GENERIC_ALTITUDE = 200.0; // above sea level
const float AIR_VELOCITY = 1.13;
const float AIR_DENSITY = 1.225; // kg/m^3
const float FORCE_OF_DRAG = (AIR_VELOCITY * AIR_VELOCITY) * 1/2 * AIR_DENSITY; //not the complete equation

};

#endif
Then, in a class this is used, these get defined at run-time in STI__classname.cpp. At this point, I see the variables be defined in different locations when comparing between builds...sometimes 2-3 lines apart, and sometimes 10 lines.

After looking at the object file dump of these constants, some appear in the .cinit section, and others in .text.  

Is this something you might believe is occurring due to the SDO in question?

-Will

Archaeologist,

     I have two answers for you.  I turned on just those two options. 

1.   I have one file, in one build, that has exhibited no change in the src interlist.  I see the branch labels have changed, however.  E.g.,  C$73  in one file, C$74 in another, then you have B C$73 and B C$74, respectively.   Essentially, branching to a new label created by the compiler/optimizer.

2.  I have another build where two files are different.  The one in #1 shows up again, with same differences.  However, another file has very distinct, and noticeable, differences in the source interlist.  In some cases, it looks innocuous.  In one case, however full instructions are completely missing in the differences.  Posting an example:

So, where its "repeated" on the left, there is no equivalent on the right.  I am hand typing from a closed system, and putting "repeated" in cases where the tag increments, and the assembly looks similar to what I've typed with pattern ( C$N: = (const double* const) obj/this+X)).    These lines aren't just shifted down later in the second build.  They are not found later in the assembly code in the right file.  Just to be clear, I'm not making a statement about the correctness, or executability, of the code, only that you were looking for specific cases where they are different. 

What would you like me to try next?

-Will

Archaeologist, would you believe the binary comparison between a 7.3.0 build and 7.3.23 build would be exactly the same, after pushing through the hex6x tool to generate a flashable binary? I upgraded, am running into a new "warning" of NDK has some libraries in "COFF" format, so I'm not certain if that would affect the final binary comparison between 7.3.x and 7.3.23 or its just they would be different. From a quick glance at the outputted assembly for both, I see many differences (I've only compared one instance, of the 7.3.1 build to 7.3.23); .fields are now .bits, and c interlist/assembly is different.

Thanks,

Will

Agreed. Software management will ask the amount of variability one can expect with the upgrade. I'm satisfied that we know it will be different. The amount (time and $$) of testing and analysis will be thorough. Thanks.

I've performed 15 builds with consistent binary output.