(MM club member)

2014-08-11 11:02 PM

Math Question

Am I missing something stupid here? Can someone explain to me why the following code results in an output of 1.4210854715202E-014 ?

Code:

$num = 729.95-629.96-99.99
? $num
get $

I would expect it to be simple 0.00.

Glenn Barnas Administrator

(KiX Supporter)

2014-08-12 04:44 PM

Re: Math Question

Ya think? $\:D$

Paste the value into Excel, then format the cell to "Number" and you get "0.00".

Even more strange is that

Code:

$N2 = 99.99
$N2 = $N2 - 99.99
$N2 ?

returns zero!

Glenn

Glenn Barnas Administrator

(KiX Supporter)

2014-08-12 04:50 PM

Re: Math Question

More info:

Code:

$num = 729.95 - 99.99 - 629.96

works and properly returns 0.

From a Libre-Office bug sheet:

Quote:

This is not an LO bug. I get the exact same result with a Python script,
a Ruby script, and Calligra Sheets.

It is a problem with internal representation of decimal numbers in any
computer system. This is a well known problem in scientific
computing/numerical analysis.

Also, reordering the addition produced different precisions. I think the
correct numerical analysis term for this is precision referring to how
closely the actual value is to the true value.

This is why I changed the order, and - to my surprise - got the expected value.

Glenn

Allen

(KiX Supporter)

2014-08-12 05:06 PM

Re: Math Question

How would you ever know you are running into this without testing every single value passed?

ShaneEP

(MM club member)

2014-08-12 05:08 PM

Re: Math Question

And unfortunately the FormatNumber() in kix doesn't even fix it.

Code:

FormatNumber($num, 2)

results in a long string of 0's, and not just 0.00 like it should. Having to use the Round() function to get around it for now.

BradV

(Seasoned Scripter)

2014-08-12 05:35 PM

Re: Math Question

Yes, I was going to post earlier, but can't from work. It has to do with how floating point numbers are stored in memory. It is usually just an approximation. You have to keep track of your significant digits. In your case, you probably want to multiply by 1000, convert to an integer, add 0.00 plus your variable, divide by 1000 to get a more accurate answer.

Code:

$num=....
$num2=$num*1000
$num3=int($num2)
$num4=0.00+$num3
$num5=$num4/1000

Of course, you can condense that. Just want to illustrate that this will give you 0.0. You can change your "-99.99" to "-99.97" and your answer will come out 0.02.

Henriques

(Fresh Scripter)

2014-08-12 06:03 PM

Re: Math Question

$num = 729.95 - (629.96 + 99.99)
? $num
get $

Will give you the right answer.

ShaneEP

(MM club member)

2014-08-13 01:06 AM

Re: Math Question

Thanks for the tips guys.

Would you say that it's a bug in FormatNumber() that it doesn't format the weird number down to 2 decimal places? Don't know why FormatNumber($num, 2) doesn't work.

Lonkero

(KiX Master Guru)

2014-08-16 02:28 AM