Tool Qualification Plan for
TI C/C++ Compiler

Version:	0.8.6
Date:	2019-09-11
Status:	In Progress / Presented / Reviewed / Final
Author:	Oscar Slotosch
File:	TQP.docm
Size:	12 Pages
Maximal Safety Level	ASIL B
Qualification Method	Process Qualification

NOTICE: The following Tool Qualification Plan only serves as a template; it is the user’s responsibility to fill out, review, verify and follow the plan as completed by the user. This plan is intended to assist users in their efforts to comply with the requirements of certain safety standards. (Users may delete this Important Notice once the plan is filled out.)

Revision History:

Version	Date	Status	Author	Change
0.5	2017-01-27	Reviewed	Slotosch	Created template with example
0.6	2017-02-10	In Progress	Miller	TI Review of template
0.8	2017-06-09	Reviewed	Miller	TI Review of template
0.8.1	2017-08-03	Reviewed	Miller	TI Review of Template: confidence from use description generalized to ISO26262 formulation.
0.8.2	2019-08-07	Reviewed	Slotosch	Changed to 50657
0.8.3	2019-08-14	Reviewed	Slotosch	Added word variables for level and mode
0.8.4	2019-08-15	In Progress	Slotosch	Regrouped checks and results
0.8.5	2019-08-16	Reviewed	Miller	Reviewed
0.8.6	2019-09-06	Reviewed	Slotosch	Added TÜV Feedback: References und clarifications for confidence from use
0.9	Reviewed	<User>	Created & Confirmation reviewed document
1.0	Final	<User>	Finalized document

Contents

1	Scope of this Document	4

2	Validation Process	5

2.1	Qualification Mode Determination	5

2.2	Configuration and Qualification	5

2.3	Qualification Documentation	6

3	Checks	7

3.1	Configuration Check	7

3.2	Known Bug Check	7

3.3	Process Compliance Check (Mode=Process Qualification)	7

3.4	Confidence From Use Check (Mode=Process Qualification)	7

3.5	Validation Check (Mode=Qualification by Validation)	8

3.5.1	Use Case Definition	8

3.5.2	Obtain Instrumented Compiler	8

3.5.3	Measure Application Coverage	8

3.5.4	Submit Coverage Files	8

3.5.5	Compare Application Coverage with Qualification Coverage	8

3.5.6	Mitigations for Coverage Gaps	9

3.6	Anomalous Operating Condition Check (Mode=Qualification by Validation)	9

4	Qualification Results (Example)	9

4.1	Configuration Check (Example)	9

4.2	Known Bug Analysis (Example)	9

4.3	Process Assessment (Example)	10

4.4	Confidence From Use (Example)	10

4.5	Validation (Example)	10

4.5.1	Use Case Definition	10

4.5.2	Receive Instrumented Compiler	11

4.5.3	Measure Application Coverage	11

4.5.4	Submit Coverage Files	11

4.5.5	Compare Application Coverage with Qualification Coverage	11

4.5.6	Mitigations for coverage gaps:	11

4.6	Anomalous Operating Conditions (Example)	11

5	Qualification Analysis (Example)	11

6	Qualification Summary (Example)	11

7	References	11

Scope of this Document

This document describes how the TI C/C++ Compiler will be qualified for usage by <Tool User>. The tool has qualification needs according to the tool’s classification see [QKit_UG]. Therefore, the tool needs to be qualified. This document sets forth the plans for the tool qualification, defines the required steps from the tool qualification plan, and lists the results.

The compliance of the qualification is described in separate compliance reports. It depends on the used validation method. For ISO 26262 the qualification is applicable until ASIL B (qualification method = Process Qualification). If the Validation check (qualification method = Qualification by Validation) is performed successfully by the user, then the tool can also be used in ASIL D projects.

The document is structured as follows:

Section 2 describes the overall qualification process

Section 3 describes the checks required for qualification

Section 4 describes the qualification results

Section 5 describes analysis of the results

Section 6 contains the qualification summary

The qualification method and a glossary are provided in the Qualification Kit User Guide, see [QKit_UG].

Validation Process

The overall validation process consists of the following phases to be performed by the user:

Determination of qualification mode

Select configuration and perform qualification by applying the described checks

Write qualification report

These phases are explained in detail in the following subsections.

Qualification Mode Determination

The qualification mode depends on the safety standard. For ISO 26262, this also depends on the required safety level.

A prerequisite for both modes is that the user’s version of the TI C/C++ Compiler is the same version as described in the tool definition in [ToolDefinition].

The Process-based qualification mode can be chosen in case the system is developed according to ISO 26262 with ASIL A or ASIL B.

The Validation-based qualification mode must be chosen if the system is developed according to ISO 26262 with ASIL C or ASIL D, or if the system is developed according to IEC 61508 or EN 50657.

The qualification Mode for the current project is qualification is:

Process Qualification.

Configuration and Qualification

The selected configuration for the qualification is (for Example):

--mv=7R4 --opt_level=2 --float_support=vfpv3d16 -g --unaligned_access=off --code_state=16 -me

The selected Version of the compiler is: Version 18.12.3.LTS

Based on the chosen qualification mode, all required checks must be performed by the user as described in the corresponding section of the Tool Qualification Plan [TQP]. The results are documented by the user in the Tool Qualification Report (based on the template) [TQR] as described in the next section.

Qualification Documentation

Writing the Tool Qualification Report involves/has involved the following activities to be performed by the user:

Discuss all analysis findings. Let clearness and plausibility of the findings be confirmed by tool experts.

Suggest measures to detect or avoid the erroneous tool outputs.

Write the tool qualification report.

The tool qualification report will be created by the user by updating this document as follows:

Update Section 1 Scope of this Document.

Define the qualification mode and the configuration in Section 2.2

Remove the sections not relevant for the qualification mode or mark them as not applicable:

For qualification mode=Process Qualification, these are:

Validation: Description in Section 3.5 and results in Section 4.5

Anomalous Operating Conditions: Description in Section 3.6 and results in 4.6

For qualification mode=Qualification by Validation, these are:

Process Compliance: Description in Section 3.3 and results in Section 4.3

Confidence From Use: Description in Section 3.4 and results in 4.4

Below sections shall be completed:

4) Qualification Results

Description of qualification results from executing the selected checks

5) Qualification Analysis

Analysis of failed checks

Mitigations for failed checks

6) Qualification Summary

Recommendation on the usage of the TI C/C++ Compiler

And the document shall be reviewed by the user or safety expert.

Checks

This section describes all checks that need to be performed for the qualification of the TI compiler (depending on the qualification mode, not all need to be performed).

Configuration Check

The goal of this check is to ensure that the configuration selected in Section 2.2 was picked from the tool definition [ToolDefinition], hence it is a valid configuration.

Known Bug Check

The current open known issues for the selected configurations must be analyzed for relevance and mitigations must be provided if required.

The list of known issues is included in the TI C/C++ Compiler installation folder in file, Open_defects.html, or can be obtained from Texas Instruments.

See qualification results (Section 4.2) for an illustrative example of this check.

Process Compliance Check (Mode=Process Qualification)

The TI C/C++ Compiler has been developed according to a well-defined process. In has been assessed for compliance to DO-330 TQL-5 which is an international development standard (see [DO330]) for tool development and tool qualification, see [ProcComp].

Confidence From Use Check (Mode=Process Qualification)

The confidence-from-use check is done by the user in three parts:

Specification of the use case for the project by fixing:

Target platform

Compiler version and options

Project start (with the version of the compiler)

User’s internal issue tracker for compiler issues

Determination of confidence from own usage:

Which patch version has been selected (>=2)?

How mature is it (>= 6 month)?

How long has it been used in projects (>=6 months)?

Have additional issues been found within the project that block the usage? This should be answered at project end by checking for reported issues within the project. If the number / criticality is low then the confidence is increased based on this report.

Determination of confidence from other usages: Obtain download numbers from Texas Instruments and determine if they are sufficient and adequate for your use case (more than 10 users of the same basic version, more than one user of the same patch version).

The recent data on the download and user statistics can be obtained from

Texas Instruments.

See qualification results (Section 4.4) for an illustrative example of this check.

Validation Check (Mode=Qualification by Validation)

This section can be removed for the case where validation-based qualification is not required (see Qkit User Guide [QKit_UG]).

The validation was performed by Texas Instruments (“Pre-Qualified” approach) in many configurations (compiler option use cases). The configuration is checked within the configuration check, see Section 3.1, hence the main goal of the validation check is to ensure that the qualification covers the application of <Tool User>. This is done by comparing the code coverage within the compiler between compiling the application of <Tool User> and compiling the qualifications performed at Texas instruments.

Use Case Definition

The configuration and version described in Section 2.2 are used from <Tool User> to compile the following software components:

Component 1

Component 2

The check whether the qualification covers this use cases was done using the code coverage comparison approach in the following steps.

Obtain Instrumented Compiler

Obtain an instrumented compiler from Texas Instruments for the compiler version specified in section 2.2. Instrumented compilers can be obtained from the SafeTI_CQKIT “mySecure Software” portal where the QKIT installer was obtained.

Measure Application Coverage

Replace the compiler on your production system with the instrumented compiler for the coverage measurement (making a backup of the original compiler to restore it after the coverage measurement). Compile the software with the instrumented compiler and find the coverage files using the environment variable “COVERAGE_FILE”. Coverage testing instructions can be obtained from the SafeTI_CQKIT “mySecure Software” portal where the QKIT installer was obtained.

Submit Coverage Files

Submit the application coverage files to Texas Instruments.

Compare Application Coverage with Qualification Coverage

Coverage comparison is performed by Texas Instruments.

Mitigations for Coverage Gaps

In case coverage gaps are found, they must be listed together with explanations and mitigations.

Anomalous Operating Condition Check (Mode=Qualification by Validation)

The behavior of the compiler in anomalous operating conditions (Low CPU, low memory, low disk space, ..) has to be analyzed or it has to be ensured that the compiler is used within environments that this cannot occur or will be detected.

Qualification Results (Example)

The chosen qualification mode was process assessment. Therefore the following checks have been performed

Configuration Check, see Section 4.1,

Known Bug Check, see Section 4.3,

Process assessment, see Section 4.4,

Confidence from Use, see Section 4.2,

Validation, see Section 4.5 and

Anomalous Operating conditions, see Section 4.6.

Configuration Check (Example)

The configuration described in Section 2.2 was taken from the tool definition in [ToolDefinition]and hence it is a valid configuration.

Known Bug Analysis (Example)

The following issues have been received from Texas Instruments on November 11th and classified to have impact (see relevant_issues.xls for more details and the mapping to the mitigations)

CODEGEN-6469

CODEGEN-6070

…

The following work arounds are required for the known issues:

use work around in:

https://e2e.ti.com/support/tools/ccs/f/81/p/810598/3026599#3026599 or avoid exceptions/C++

avoid duplicated type definitions

…

The work arounds have been added to the tool safety manual

Process Assessment (Example)

The development process of the compiler has been assessed by Validas and documented in [ProcComp]. No additional steps have been performed.

Confidence From Use (Example)

We are using the selected configuration from Section 2.2

The confidence from use is

Patch version .3 from July 2019 is used (>=2)

.3 was released in July 2019, i.e. 7 months after the 18.12.0 which was released in December 2018 (>= 6 month)?

There are two projects using it since 4 months, hence we have now (Nov 2019) 8 months of experience with it (>=6 months)?

Only two issues have been found since that, one was wrong usage and the other was a bug already known, so no new issues and the tool justifies our confidence from use.

The following user statistics has been obtained from Texas instruments. It shows that many users are using this version of the compiler:

.

Validation (Example)

The validation has been performed by Texas-Instruments (“Pre-qualified” approach). This is described within this section.

Use Case Definition

Results from the user checking if the validation use cases from Texas Instruments were sufficient to cover the requirements from <Tool User>

(as generated by the user per [TQP] section 5).

The configuration and version described in Section 2.2 are used from <Tool User> to compile the following components:

Component 1

Component 2

The check whether the qualification covers this use cases was done using the code coverage comparison approach in the following steps:

Receive Instrumented Compiler

Has been received from Texas Instruments as of the 11th of August 2019 and is located on the development server.

Measure Application Coverage

Application coverage files during compilation of the listed components have been generated by <Tool User> on 16th of August 2019.

Submit Coverage Files

Application coverage files have been submitted to Texas Instruments as of 17th of August 2019.

Compare Application Coverage with Qualification Coverage

Coverage results comparison returned from Texas Instruments as of 24th of August 2019.

Mitigations for coverage gaps:

List out the mitigations for the user for any coverage gaps that were found.

Mitigation 1

Mitigation 2

Anomalous Operating Conditions (Example)

The behavior of the compiler in anomalous operating conditions is not relevant, since the safety manual requires to use the compiler in a stable environment, e.g. on a server or within a stable (and supervised) environment like a build tool that checks the results of the compiler calls.

Qualification Analysis (Example)

The qualification did not bring up any findings that need to be analyzed.

Qualification Summary (Example)

The tool has been qualified successfully using Process Qualification up to ASIL B, since all checks have been performed successfully.

References

[DO330] RTCA. DO-330: Software Tool Qualification Considerations 1st Edition 2011-12-13.

[ProcComp]: Process Compliance Check Result for TI C/C++ Compiler: ProcessComplianceCheckResult_TI_Compiler.pdf

[Tool]: User Manual for TI C/C++ Compiler: SPNU118W.pdf and SPNU151T.pdf

[ToolDefinition]: Tool Definition for TI C/C++ Compiler: ToolDefinition.pdf

[TestRep]:	Release validation test results: validation_results.pdf

[QKit_UG]: Qualification Kit User Guide: QKitUserGuide.pdf

[TQR]: Tool Qualification Report (Template) for TI C/C++ Compiler, document version 0.8.5: TQR.docm

[TSM]: Tool Safety Manual for TI C/C++ Compiler, document version 0.8: TSM.docm

In case this is the TQR will complete the TQP by adding the planned information.