Compiler: cl6x compiler to produce bitwise reproducible output

I have similar problem in https://e2e.ti.com/support/microcontrollers/hercules/f/312/t/743189. I have found the following workaround. I set "keep generated assembly language (.asm) file (--keep_asm, -k). in Assembler Options. This results in deterministic names.

I saw that request, but do not see any solution, and topic was locked, thus created my own (and I have issue with cl6x, not arm compiler).

Marvelous! That somewhat workarounds first issue I mentioned. Not a production though... Those assembly files will significantly increase build dir size (4 times of object file size). When I have thousands of objects (yeah, quite a big project), then I can count that in gigabytes...

In worst case, I'll just implement yet-another-cl6x-wrapper (among "line buffering" wrapper and similar ones), which will instantly remove those files afterwards. Or use --asm_directory=$(BUILDDIR)/trash, and remove it after build. Need to rethink...

Thanks anyway, that is some good initial approach until we get something production-ready from TI experts.

Now, when using --keep_asm, I also get some numbers, which are the same as last modification timestamp of the compiled *.asm file.

Looking at binary hexdump, I see strings like the following:
/path/to/asm/file.asm:$C$L6:1546604415

Looking at file.asm:
$ stat /path/to/asm/file.asm -c "%Y"
1546604415

So, it turned out that setting --keep_asm does not solve my issue...

A good summary on this topic is in this forum thread.  

Consider using the utility objdiff from the cg_xml package.  By default, it ignores the debug information and the symbols.  This reduces the constraints imposed on the build.

Thanks and regards,

-George

Hello George,

Thanks for the answer. It shed some light on the topic.

I agree that some aspects of build process are not compiler/linker responsibility (e.g. maintaining order of inputs), but some other are, and I think that my request address such things.
When I execute the same command, on the same host, in the same directory, I'd expect exactly same result (i.e. md5sum/sha256sum should match in both).
Or I'd expect at least some easy method to fake build environment, so that compiler gives predictable results...

In such case, I can keep only fingerprint (e.g. md5sum hash) of an executable + environment description (a few kilobytes), and compare rebuilt binaries with it, to assure I got exactly the same content (using tools that are available on any linux box). I cannot imagine how to achieve this efficiently with objdiff...

Argument that "we don't test something, thus not delivering" does not seem to be relevant in this discussion. It's not a matter of testing or not, but willingness to support this kind of use case, and actually start doing anything related to this. And, based on amount of similar questions to mine, it seems there are some people who would be interested in bitwise identical binaries.

So, maybe question should be: will you add tests (and support) for this?

The solution currently provided by TI compilers does not work this way ...

Bartlomiej Kucharczyk said:

When I execute the same command, on the same host, in the same directory, I'd expect exactly same result (i.e. md5sum/sha256sum should match in both).

Instead, some executable or library is established as the baseline, and then objdiff is used to test whether subsequent builds are the same.

Bartlomiej Kucharczyk said:

maybe question should be: will you add tests (and support) for this?

Unfortunately, that is not on our roadmap.

Thanks and regards,

-George

Hmm... that's sad news.

Can anything be done you add this topic into your roadmap?

Anyway, how I could compute a fingerprint (e.g. MD5 hash) of an executable/library, that could be used later to compare against newly built executable/library? 

It is also acceptable for me to get some way to strip those debugging symbols (strip6x tool did not work for me -- still some build paths were in the objects).

Bartlomiej Kucharczyk said:

I've made a tool which is erasing some of the useless data from the binary

Thank you for the contribution.  But I don't see the advantage of this approach over the one used by objdiff.  objdiff doesn't erase anything, it just skips over the "useless data".

Thanks and regards,

-George

For me:

• I'm able to compute and store fingerprint using md5sum (or other standard hash tool available on linux).
• It was easier to write it than examining objdiff source code and figuring out what is "important data", to compute hash of that value. Most probably working with some ELF library is better.

You are right, thanks for pointing that out.

However I do not understand how it is in contradiction with C standard. I haven't said anywhere that "all C source code will be 100% bitwise reproducible".:-)

We simply try to avoid those macros in source code (and other pitfalls that are causing builds to be irreproducible). In case someone is using it, we'll catch that immediately (md5sum will differ even no change in source code/build environment). 

But let's go one step back, and ask: do you have any reasons for using variable __DATE__ and __TIME__? Because personally, I cannot see any, so that could be good education for me. ;-)

Maybe that was useful when programming was done by carving rocks, and there was no git. ;-) I don't know...
If you use any version control system, revision string can be generated from version control, deterministically.

Topic is also discussed here:
reproducible-builds.org/.../
reproducible-builds.org/.../