OMAP-L138: Intermittent compilation failures with Jenkins

By the way, all of our CCS projects are configured with the following CCS Build settings:

• Use standard build arguments
• Enable parallel build
• Use optimal jobs (8)
  • This particular virtual machine is configured with 8 virtual processors

With these settings, I see the CPU activity for all eight CPUs approach 100% for a few minutes while CCS builds one of the DSP core projects.

If I disable parallel build, I see the CPU activity for all eight CPUs stay below roughly 60% while CCS builds one of the DSP core projects. No surprise, it takes considerably more time to complete. (I did not time either scenario.)

I will disable parallel build for each project and commit and push changes and see what happens with an automated Jenkins build using our CI environment. 

Patrick,

Did disabling parallel build resolved the issue?

Michael

No. It did not completely resolve the issue. I still saw failures but not as many as I saw with parallel builds enabled.

Thank you for checking. Engineering is investigating.

I think I found a clue.

On one problematic Jenkins build agent virtual machine, let's call it VM_A, I observe the following:

• Random "Compilation failure" errors on cpp files in the project
• Application errors against opt6x.exe in the Windows administrative event log
  • This gives me the impression that opt6x.exe is crashing on some random file

On a different, reliable Jenkins build agent virtual machine, let's call it VM_B, I observe the following:

• No random "Compilation failure" errors
• No application errors against opt6x.exe in the Windows administrative event log

I did not find any content for "opt6x" in the TMS320C6000 Optimizing C/C++ Compiler v8.3.x User's Guide (sprui04f)

Are the options for "opt6x" listed somewhere? Is there an option for verbose output? Can opt6x tell me why it is unhappy?

I assume cl6x spawns opt6x processes. Right?

If I cannot come up with a better idea, I may try to re-install Code Composer Studio v12.2.0 on VM_A in the near future.

Patrick McCloskey said:

I did not find any content for "opt6x" in the TMS320C6000 Optimizing C/C++ Compiler v8.3.x User's Guide (sprui04f)

opt6x is the C6000 optimizer. Information on it is in chapter 5 of the above documentation.

Patrick McCloskey said:

Are the options for "opt6x" listed somewhere? Is there an option for verbose output? Can opt6x tell me why it is unhappy?

I assume cl6x spawns opt6x processes. Right?

I believe cl6x will run opt6x. I will bring this thread to the attention of the compiler experts 

Thanks

ki

Patrick McCloskey said:

Are the options for "opt6x" listed somewhere?

No.  There is no public interface.  Because the only way to run it is from cl6x.

A few years ago, the similar issue EXT_EP-10596 was reported.  That issue was never fixed.  It was reported against a old version that was no longer supported.  When the customer upgraded to a more recent version, the problem disappeared.  In that case, opt6x is not used, so it cannot be the problem.  A different compiler pass, named cg6x3, was the problem.  

Patrick McCloskey said:

Is there an option for verbose output?

I sent you an E2E friend request.  Please accept it.  That will allow us to use the messaging feature built-in to the forum.  That's how I will tell you how to run an experiment that may tell us more.

Thanks and regards,

-George

Hello George. Thanks for the response.

I am having trouble finding your friend request. Where should I look for it?

I am troubleshooting multiple Jenkins build agent virtual machines misbehaving like this. They have been in service so I have to work around jobs initiated by other folks. We are making slow progress with the following experiments:

1. Uninstall and reinstall CCSv12.2.0
   1. Preliminary observation: This seems to have made a difference on one agent. Testing in progress.
2. Replace one of the misbehaving build agent virtual machines with a clone of an agent that is not misbehaving
   1. No results to report yet

This thread has changed to private messages.  Just for now, I will close it.  Once the cause is found, a summary will be posted here.

Thanks and regards,

-George