If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

MSP430FR59941: Calculation error

Part Number: MSP430FR59941

Hi,

I'll perform some calculations in the CCS (Code Composer Studio). When a variable is declared as a float or double, calculations produce inaccurate results.

How to solve the problem.

alu_calculation = 7.7 + 2.4 = 10.1

alu_calculation  = 9.8 - 5.6 = 4.2000000000000011 (wrong)

alu_calculation = 5.9 * 2.7 = 15.930000000000001 (wrong)

This is code:

//*************************************************************************************

static unsigned int Status_ALU = 0;
double alu_calculation = 0.000;

alu_calculation = 7.7 + 2.4;

if(alu_calculation == 10.100000)
{
alu_calculation = 0.000;
alu_calculation = 9.8 - 5.6;

if(alu_calculation == 4.2)
{
Status_ALU = 1;
}
}
else
{
Status_ALU = 0;
}

if(Status_ALU == 1)
{
alu_calculation = 5.9 * 2.7;

if(alu_calculation == 15.93)
{
Status_ALU = 1;
}
}
else
{
Status_ALU = 0;
}

//************************************************************************************

Regard,

Vijay

• Welcome to floating point. floats and doubles are only approximations, i.e., they are not infinite precision, and you can expect round off and conversion errors. As a Kernigham and Plauger say: "Working with floating point is like moving sand piles. Every time you move one you lose a little sand and add a little dirt."

You need to work with the significant figures you have.

If you really want better precision with numbers like this, you might need to switch to fixed point, which is faster anyway.

https://c-faq.com/fp/index.html

• Just for the record I tried the same thing in C# with MSVS 2022.

Here is the relevant code:

```int Status_ALU = 0;
double alu_calculation = 0.000;

alu_calculation = 7.7 + 2.4;
richTextBox1.AppendText("=10.1 :: " + alu_calculation.ToString() + '\n');

if (alu_calculation == 10.100000)
{

alu_calculation = 0.000;
alu_calculation = 9.8 - 5.6;
richTextBox1.AppendText("=4.2 :: " + alu_calculation.ToString() + '\n');

if (alu_calculation == 4.2)
{
Status_ALU = 1;
}
}
else
{
Status_ALU = 0;
}

if (Status_ALU == 1)
{
alu_calculation = 5.9 * 2.7;
richTextBox1.AppendText("=15.93 :: " + alu_calculation.ToString() + '\n');

if (alu_calculation == 15.93)
{
Status_ALU = 1;
}
}
else
{
Status_ALU = 0;
}

alu_calculation = 5.9 * 2.7;
richTextBox1.AppendText("=15.93 :: " + alu_calculation.ToString() + '\n');```

And here are the results:

• Hi Keith,

I'll perform a fixed-point calculation.

Thank you for helping.

Regard,

Vijay

**Attention** This is a public forum