Page 1 of 1 1
Topic Options
#209406 - 2014-08-11 11:02 PM Math Question
ShaneEP Moderator Offline
MM club member
*****

Registered: 2002-11-29
Posts: 2079
Loc: Tulsa, OK
Am I missing something stupid here? Can someone explain to me why the following code results in an output of 1.4210854715202E-014 ?

 Code:
$num = 729.95-629.96-99.99
? $num
get $


I would expect it to be simple 0.00.

Top
#209407 - 2014-08-12 04:44 PM Re: Math Question [Re: ShaneEP]
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4282
Loc: New Jersey
Ya think? \:D

Paste the value into Excel, then format the cell to "Number" and you get "0.00".

Even more strange is that
 Code:
$N2 = 99.99
$N2 = $N2 - 99.99
$N2 ?
returns zero!

Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
#209408 - 2014-08-12 04:50 PM Re: Math Question [Re: Glenn Barnas]
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4282
Loc: New Jersey
More info:
 Code:
$num = 729.95 - 99.99 - 629.96
works and properly returns 0.

From a Libre-Office bug sheet:
 Quote:
This is not an LO bug. I get the exact same result with a Python script,
a Ruby script, and Calligra Sheets.

It is a problem with internal representation of decimal numbers in any
computer system. This is a well known problem in scientific
computing/numerical analysis.

Also, reordering the addition produced different precisions. I think the
correct numerical analysis term for this is precision referring to how
closely the actual value is to the true value.
This is why I changed the order, and - to my surprise - got the expected value.

Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
#209409 - 2014-08-12 05:06 PM Re: Math Question [Re: Glenn Barnas]
Allen Administrator Online   shocked
KiX Supporter
*****

Registered: 2003-04-19
Posts: 4441
Loc: USA
How would you ever know you are running into this without testing every single value passed?
Top
#209410 - 2014-08-12 05:08 PM Re: Math Question [Re: Glenn Barnas]
ShaneEP Moderator Offline
MM club member
*****

Registered: 2002-11-29
Posts: 2079
Loc: Tulsa, OK
And unfortunately the FormatNumber() in kix doesn't even fix it.

 Code:
FormatNumber($num, 2)


results in a long string of 0's, and not just 0.00 like it should. Having to use the Round() function to get around it for now.

Top
#209411 - 2014-08-12 05:35 PM Re: Math Question [Re: ShaneEP]
BradV Offline
Seasoned Scripter
****

Registered: 2006-08-16
Posts: 678
Loc: Maryland, USA
Yes, I was going to post earlier, but can't from work. It has to do with how floating point numbers are stored in memory. It is usually just an approximation. You have to keep track of your significant digits. In your case, you probably want to multiply by 1000, convert to an integer, add 0.00 plus your variable, divide by 1000 to get a more accurate answer.
 Code:
$num=....
$num2=$num*1000
$num3=int($num2)
$num4=0.00+$num3
$num5=$num4/1000
Of course, you can condense that. Just want to illustrate that this will give you 0.0. You can change your "-99.99" to "-99.97" and your answer will come out 0.02.

Top
#209412 - 2014-08-12 06:03 PM Re: Math Question [Re: BradV]
Henriques Offline
Fresh Scripter

Registered: 2007-09-13
Posts: 30
$num = 729.95 - (629.96 + 99.99)
? $num
get $

Will give you the right answer.

Top
#209414 - 2014-08-13 01:06 AM Re: Math Question [Re: Henriques]
ShaneEP Moderator Offline
MM club member
*****

Registered: 2002-11-29
Posts: 2079
Loc: Tulsa, OK
Thanks for the tips guys.

Would you say that it's a bug in FormatNumber() that it doesn't format the weird number down to 2 decimal places? Don't know why FormatNumber($num, 2) doesn't work.

Top
#209425 - 2014-08-16 02:28 AM Re: Math Question [Re: ShaneEP]
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22332
Loc: OK
Yes. Bug.
_________________________
!

download KiXnet

Top
#209436 - 2014-08-20 01:33 AM Re: Math Question [Re: Lonkero]
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11564
Loc: CA
Well if this is a math question....


Top
#209437 - 2014-08-20 01:45 AM Re: Math Question [Re: NTDOC]
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22332
Loc: OK
Yup
_________________________
!

download KiXnet

Top
#209439 - 2014-08-20 02:03 AM Re: Math Question [Re: NTDOC]
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4282
Loc: New Jersey
Dang - I knew I should have stopped at 25... ;\)
_________________________
Actually I am a Rocket Scientist! \:D

Top
#209440 - 2014-08-20 10:19 AM Re: Math Question [Re: Glenn Barnas]
Mart Moderator Offline
KiX Supporter
*****

Registered: 2002-03-27
Posts: 4641
Loc: The Netherlands
LOL Nice one Doc.
_________________________
Mart

- Chuck Norris once sold ebay to ebay on ebay.

Top
Page 1 of 1 1


Moderator:  Jochen, Allen, Radimus, Glenn Barnas, ShaneEP, Arend_, Mart 
Hop to:
Shout Box

Who's Online
1 registered (Allen) and 181 anonymous users online.
Newest Members
skygreen2031, amelindarķe, Nims123, davidscroegh, davidscroe
17626 Registered Users

Generated in 0.076 seconds in which 0.029 seconds were spent on a total of 13 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org