This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Compiler: large stall in TI compiler builds

Other Parts Discussed in Thread: CCSTUDIO

Tool/software: TI C/C++ Compiler

we seem to have run into a long compiler stall with the TI compiler when building our TM4C129x projects.

We do not see the issue compiling smaller projects or libraries with just a dozen or so source C++ files.

It seems when we cross a certain number of source C++ files we hit a stall. The compiler seems to just freeze for almost 5 minutes!

After it finally recovers it continues compiling and linking with no errors displayed in the console window. The generated output files are fine.

We have verified this only to be an issue with the TI compiler as we have a mirror build for a PC target using most of the same source files (less the TM4C target specific files) using the gcc compiler.

It has no issues and does not stall.

We have tried several different versions of the TI compiler (including the latest TI v16.9.3.LTS) and all have the same symptom.

We have also tested on different versions of CCS on both Linux and Windows all show the same symptoms.

we do not see any disk activity when the stall occurs.

We initially thought it was do to the linker but after it recovers it continues to compile the last few source files and then links so I assume its not a linker issue.

If I just do an incremental build after touching just a file or two it does not stall. If I do a clean and build the stall reappears.

This is really impacting our whole development team to the point we are investigating moving the whole thing to the GNU Linaro compiler and junking TI's compiler altogether.

Any ideas?

  • I suspect one (or two) function take up most of the time.  To find that function, add the option --verbose and build again.  You get lots more output from the compiler.  It prints the name of each function as it is compiled.  If my guess is correct, you will see the name of the function that take so long.  Once you know that, here is how to submit a test case for further analysis.

    1. Preprocess the source file that contains the problem function
    2. Attach that file to your next post
    3. Indicate the name of the function
    4. Show the compiler version used
    5. Show all the build options exactly as the compiler sees them

    Thanks and regards,

    -George

  • verbose was no help. As I stated before it not related to any particular file but more the "number" of files in the project.

    Indeed if I do a full build (with the stall) and it stalls a file x.cpp and then I go back and touch file x and do an incremental build the project builds fast and fine.

    If I do a clean and Rebuild ALL the stall reappears. 

    The stall is between one file and the start of another:

    "../Generated/UIServer.cpp" ==> UIServer::initial(UIServer*, const QP::QEvt *)

    "../Generated/UIServer.cpp" ==> UIServer::HandleCommunication(UIServer*, const QP::QEvt *)

    "../Generated/UIServer.cpp" ==> UIServer::Enabled(UIServer*, const QP::QEvt *)

    'Finished building: ../Generated/UIServer.cpp'

    ' '      XXXXXXXXXXXXXXXXXXXXXXXXXXXX stalls here XXXXXXXXXXXXXXXXXXXXXXX

    'Building file: ../Generated/RATClient.cpp'

    'Invoking: ARM Compiler'

    "C:/ti/ccsv7/tools/compiler/ti-cgt-arm_16.9.3.LTS/bin/armcl" -mv7M4 --code_state=16 --float_support=FPv4SPD16 -me -Ooff --include_path="C:/ti/ccsv7/tools/compiler/ti-cgt-arm_16.9.3.LTS/include" --include_path="C:/Users/David Vescovi/workspace_v7/build/firmware/common/application" --include_path="C:/Users/David Vescovi/workspace_v7/build/firmware/common/application/Generated" --include_path="C:/Users/David Vescovi/workspace_v7/build/firmware/common/include" --include_path="C:/Users/David

  • I suspect it might just be the Java VM is just running out of heap space or something like that???
  • David,

    Here are some initial thoughts, suggestions and questions.

    DavidVescovi said:
    It seems when we cross a certain number of source C++ files we hit a stall. The compiler seems to just freeze for almost 5 minutes!

    How many source files does the project where you are experiencing this issue have?
    Have you tried importing and building the project in a new CCS workspace, just to rule out any issues with the workspace itself?

    Does the stall always occur between one source file compile and start of another? Does it happen at the same point (ie between same source files) during every rebuild?

    DavidVescovi said:
    I suspect it might just be the Java VM is just running out of heap space or something like that???

    You could experiment with the Java heap setting as described in this FAQ to see if it has any impact.

    If none of the above suggestions help, we'd have to consider the next steps to take. Ideally we'd like to be able to reproduce the issue locally. Is it practical/possible for you to share the project with us? If not, I will have to check with the development team if there are specific logs that will help them investigate the issue.

  • Can you suggest what to change?

    I see a ccs_properties

    ccs_config.xml

    ccsstudio.ini

    eclipse.ini

    all in the TI folder under ccsv7

    I tried several mods to -Xmx512m   but nothing seems to make a difference.

    To recap:

    Not related to ccs version ..does the same on v5 and 7

    Not related to TI compiler version ...tried several versions including the latest.

    Also note: this problem is affecting ALL of our developers (some 20 plus) on a whole range of computers from quad core 4.2GHz with 32Gb RAM all the way down to 2Ghz with 8GB RAM.

    This problem only seem to have appeared when we crossed some arbitrary number of C++ files in our project (as far as we can tell).

     

  • DavidVescovi said:
    This problem only seem to have appeared when we crossed some arbitrary number of C++ files in our project (as far as we can tell).

    Maybe running the Windows Sysinternals Process Explorer and/or Process Monitor tools would help to identify where the stall occurs, in terms of which process/program.

    Process Explorer is the easiest to start with, as you can see a tree of the processes running.

  • and look for what?

    ...not getting much help here.
  • DavidVescovi said:
    I tried several mods to -Xmx512m   but nothing seems to make a difference.

    David,

    The suggestion was to try changing the -Xmx argument inside the file ccstudio.ini, perhaps to 768m. But it sounds like you already gave this a try and it didn't make a difference.

    Could you please provide answers to these questions? They may help provide some additional insight.

    1) How many source files does the project where you are experiencing this issue have?

    2) Have you tried importing and building the project in a new CCS workspace, just to rule out any issues with the workspace itself?

    3) Does the stall always occur between one source file compile and start of another? Does it happen at the same point (ie between same source files) during every rebuild?

    4) Could you try building the project using command line instead of the CCS GUI? That could help isolate where the issue is coming from. For details on how to import and build a project on command line, please see this page: http://processors.wiki.ti.com/index.php/Projects_-_Command_Line_Build/Create
    I would suggest trying the command line import and build in a new workspace just to keep it separate.

    5) Finally, would it be practical/possible for you to share the project with us, so we can reproduce the issue locally? 

  • 1) I'll have to get back to you on #1   ...I do have the logs.

    2) I find it hard to believe all 20 of our developers have workspace issues all at once.

    3) No, as I stated before it moves between two file based on (what it appears to be) the number of files.

    its not always between the same two files ...it moves around.

  • When you have the issue, try to concatenate files to see if it is a number of files issue or a number of lines of source code/classes
  • I already stated it seem to be related to the number of files as it stalls between finishing compiling one file and the start of the next.

  • I have a pretty much stripped down project which still exhibits the issue.
    I can zip and send it to someone for analysis.

    Contact me via my account
  • DavidVescovi said:
    I have a pretty much stripped down project which still exhibits the issue.
    I can zip and send it to someone for analysis.

    I sent you a friend request. After you accept the request,you can start a private conversation with me and attach the zip file to it. Thanks!

  • I see you've gone to a private channel to exchange files. Just wanted to say, you have 700+ people interested in this thread, please let us know how it works out in the end!

    Thanks

  • The files to reproduce this were exchanged offline. For the benefit of others following this thread, here is the summary so far.

    The stall was happening when building one of the source files. I narrowed this done by building the project after disabling the "parallel build" option.  When parallel build is enabled, since multiple processes are running at the same time, it gives the impression that the stall is occurring between source files, when in fact it is happening during compilation of one of the files. This was also confirmed by selecting and building just that single source file.

    After narrowing it down to a specific source file, I then enabled --verbose compiler option and built just that single file on a command prompt. This helped narrow down the specific function that was taking the long time to build. The recommendation in such situations is to break up the function into smaller functions, which the original poster is going to look to and report back.

  • After reworking the offending function we were able to reduce the compile time from over five minutes to a reasonable 20 seconds or so.

    Thank you for all the help.

     

  • Can you tell us a little about what made the function take so long to compile? Was it thousands of lines long? Did it have a nested "if" with 57 levels? I'm sure many of your readers would like to know how to avoid having this happen to them. I've never heard of anything like this in the past.

    Thanks,

    Lloyd

  • we made several improvements to offending function but I believe the major improvement came when we removed some inline calls and removing some macro expansions which was happing in a header file.

    It might have been the macro expansion was happening with each scan of the header file.

  • Thanks very much for the info!