placing binary resources in the linked executable

I'd like to chime in with a request to the linker people.  I am migrating a project from IAR, where a raw binary image can be linked in to the executable at build time. Over the years I have had several occasions when I need to do this for several different processors, and always had to kludge something to get around it.  Currently programming the ARM926 on L137 using TMS470 tools, earlier occasions were with c6000 tools.

It would be very helpful if I could specify something like this in the linker command file:

SECTIONS

{

    rawdata     : ALIGN(4) > RAM
                {
                sym1 = .;                      /* make a symbol that points to the beginning of this data */
                myfile.bin                     /* this is the binary data file, maybe specify with some flag? */
               }    /* raw binary image is copied into this section */

}

thanks for sympathy,

-Cameron

This topic is marked as answered, and the response says "I'm confident we can figure out a way to do it," but I do not see any method presented.

I am working with CCS3 on a C5506 firmware, and I need to be able to add a binary file's data to my .out file. This binary file is not in .obj or .lib format, so it seems that the linker will not read it. Can we figure out a way to do this and document it here?

Brian Willoughby said:

This topic is marked as answered, and the response says "I'm confident we can figure out a way to do it," but I do not see any method presented.

Since Mr. Sachs never responded further I cannot be sure, but I suspect my response lead him to a solution that worked for him.  As I said before, there is no canned solution for this problem.  

Brian Willoughby said:

I need to be able to add a binary file's data to my .out file. This binary file is not in .obj or .lib format

Here is an idea to consider.  Use the Unix od command (Internet search that term for background) to dump out the binary file contents in a format that is close to what's expected for a C array.  Then use a (choose one: Perl, awk, sed, Python, whatever) script to change the od output into the exact C array you need.  Then compile that array into your system.

Thanks and regards,

-George 

Thanks for the idea, George. I am actually already doing that. I have a C program which reads the file and converts it to source, complete with comments and data-dependent formatting.

My problem - that I did not mention in my original description - is that my data is constantly changing, so I have to keep running this reverse C compiler to alter the source and re-compile.  It would obviously be much quicker to link the binary in directly.

There was a hint that some method might be available to link in a binary, so I was hoping to learn a few more details along those lines.

Georgem said:

Here is an idea to consider.  Use the Unix od command (Internet search that term for background) to dump out the binary file contents in a format that is close to what's expected for a C array.  Then use a (choose one: Perl, awk, sed, Python, whatever) script to change the od output into the exact C array you need.  Then compile that array into your system.

Consider using the command xxd -i. Xxd comes with the VIM text editor.

Regards, Johannes

Georgem said:

This topic is marked as answered, and the response says "I'm confident we can figure out a way to do it," but I do not see any method presented.

Since Mr. Sachs never responded further I cannot be sure, but I suspect my response lead him to a solution that worked for him.  As I said before, there is no canned solution for this problem.  

[/quote]I did not find a solution. I gave up and found another way to do what I needed. I'd still like a way to link in binary data.

My gut feel is that the easiest approach would be to create a program that generates .obj files from binary data so that the linker is happy.

But long term, the linker should provide this.

I'm willing to file a request for this feature to be added to the linker.  But I need to know more about the overarching problem.  What is typically in such a binary file?  An image?  Or what?  How is it that such a file is available at build time?  Any other such information is welcome.

Thanks and regards,

-George

Georgem said:

I'm willing to file a request for this feature to be added to the linker.  But I need to know more about the overarching problem.  What is typically in such a binary file?  An image?  Or what?  How is it that such a file is available at build time?  Any other such information is welcome.

Thanks & a fair enough request:

>What is typically in such a binary file?  An image?  Or what?

Anything: an image, a table of calibration constants, metadata about the software build (build time, source control version number, machine name, etc.), the public key for a cryptographic certificate, the bus schedule for Peter Pan Bus Co. in New England. In my case it happens to be metadata about the software build, but I don't see why it would really matter as long as it can be programmed into the DSP and read out properly at runtime.

Does it make sense to use this approach in all applications? If you have a DSP on a circuit board and the binary data is likely to vary between boards, it doesn't make sense. But if the data is going to be the same for all programmed parts, and there's no other options for nonvolatile storage, putting it in flash memory can make sense, especially when the flash memory is relatively roomy in this DSP (at least compared to RAM space).

Jason is correct. The list of "typical" data is endless. Basically, any data that might appear in a constant memory array is a candidate for binary linking.

At build time, all the linker needs is a symbol address and a length. Given a file name, the length of the data can be taken directly from the length of the file. All that leaves is the symbol. Depending upon how this feature is implemented, there might even be a linker map command to associate a file name with a symbol or section. In that case, the standard linker command file syntax can be used to align the data and control where the data is placed in the run time memory map. My assumption here is that the exact memory address is not important, which is why the linker should be allowed the flexibility to relocate the binary data anywhere that is available and adjust the symbol address to reflect the placement, which is the whole job of the linker anyway.

Regarding Jason's comment about putting data in Flash memory:

1) I am working with a C5506 DSP that does not have on-chip Flash memory, so any new linker solution should not assume that the Flash memory is directly accessible in the DSP memory space. In my case, all data gets loaded somewhere in RAM. Linking in a binary simply saves me from the extra steps of converting the data into C source just so the compiler can turn it back into binary data.

2) If the firmware can access the data in Flash without loading it into RAM, then it seems like there are a couple of options. First, a serial Flash can be used by simply reading the data as a stream starting at some offset in the Flash address space, in which case the bootloader data could be separate in the Flash from the binary data. Second, a parallel Flash can be randomly addressed, so it truly does save RAM space. In both cases, it seems more useful to not link in the data with the linker, but rather to load the data separately into the Flash where the code can access the data programmatically at some known offset. Since I am not working with such hardware, I might not have an accurate picture of exactly what is needed, but it certainly seems like the RAM-saving Flash data should not be part of the bootloader, and thus probably should not be linked in with the main firmware image in the first place.

My point is that there are multiple scenarios for accessing data in Flash, and thus there are multiple ways to get the data into the hardware. A new linker feature is probably only useful in my scenario where the data must be linked into the firmware and included in the bootloader.

I never found a good workaround for embedding data into the .out file using existing tools, so I wrote my own.

This generates an .obj file for the C28xx processor, which you can link into your .out file, assuming this works for you. I haven't done rigorous testing but it appears to work for all the test cases I've tried.

Source code & jar download here: https://bitbucket.org/jason_s/file2obj28xx

If you have a binary file "foo.bin" and you want to create a variable intended for Flash memory called "myfoo" in section "mysection" with the contents of foo.bin, you can run this command to create foo.obj:

  java -jar file2obj-28xx.jar --obj --input foo.bin --section mysection --output foo.obj --format U8 myfoo

which should create something that is equivalent to

  extern const unsigned char myfoo[] = {...}

without having to generate a .cpp file and compile it.

=====

file2obj-28xx v0.1.0
  produces .obj (and .cpp and .asm) files from input data.

  Syntax:

    java -jar file2obj-28xx.jar [options] symname

  or

    java -jar file2obj-28xx.jar [args] @controlfile  (not presently supported)

  Options:

    --obj              .obj (object file)
    --asm              .asm (assembly source file) (not presently supported)
    --cpp              .cpp (C++ source file) (badly supported)

    --format {type}    controls data format (default = U8):
      Target data formats (const arrays):        U8          unsigned char
        S8          char
        U16_LE      packed (2 bytes/word) as unsigned short, LSB first
        S16_LE      packed (2 bytes/word) as signed short,   LSB first
        U16_BE      packed (2 bytes/word) as unsigned short, MSB first
        S16_BE      packed (2 bytes/word) as signed short,   MSB first
      Debug data formats: (stored as DW_TAG_CONSTANT)
        DW_FORM_STRING   null-terminated string
        DW_FORM_BLOCK    block data (embedded nulls OK)

    --section {name}    denotes which section (valid for target data only)
                        (defaults to .econst:{symname})
    symname             denotes the name of the data value
          For target data, this is the variable name, e.g. foo::bar
          For debug data, this sets the name of the constant tag
    --data {literal}    uses {literal} as the data value
    --input {path}      uses the contents of the file {path} as the data value
                        '-' is stdin
    --output {path}     sets output file ('-' is stdout)
    -i {path}           short for --input {path}
    -o {path}           short for --output {path}

=====

If someone else out there wants to make requests / suggestions / submit improvements (e.g. for other processors besides 28xx), I'd consider them, but my free time is very very limited right now.

Thanks, Jason. If you were to open-source this Java tool then folks would not need any of your very, very limited time for feature improvements.

I filed SDSCM00043204 in the SDOWP system to request this new feature in the linker.  Feel free to track it with the SDOWP link below in my signature.

Thanks and regards,

-George