# Computer Calculator Question **URL:** \ **Category:** Factual Questions\ **Created:** [January 26, 2006, 9:19pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753 "2006-01-26T21:19:26Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![SiXSwordS](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@SiXSwordS](https://boards.straightdope.com/u/SiXSwordS)\ **Post date:** [January 26, 2006, 9:19pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/1 "2006-01-26T21:19:26Z")
I often use a calculator at work. Mostly, I add or subtract lists of simple numbers so I can go back to the calculator tape and quickly compare numbers. (Yes, my job is fulfilling.) Sometimes the calculator’s paper tape shows a much more complex equation than what I put in. e.g.: 39.49 - 31.99 = 7.500000000000004 FWIW, I am talking about the calculator on the computer. Why is a simple subtraction problem handled this way? ---
**Author:** ![ultrafilter](https://avatars.discourse-cdn.com/v4/letter/u/3d9bf3/32.png) [@ultrafilter](https://boards.straightdope.com/u/ultrafilter)\ **Post date:** [January 26, 2006, 9:23pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/2 "2006-01-26T21:23:13Z")
It’s because the computer represents numbers internally in binary. The only rationals that have a finite-length representation are of the form p/2[sup]n[/sup]; everything else is repeating. Because computers can’t store an infinite number of bits, they have to make an approximation. Now, you can write a calculator that operates exactly on rational numbers. There is a bit of a difference in speed, but for something like the Windows calculator, it wouldn’t be noticeable. Why did they do it that way? Who knows? ---
**Author:** ![Shagnasty](https://avatars.discourse-cdn.com/v4/letter/s/9dc877/32.png) [@Shagnasty](https://boards.straightdope.com/u/Shagnasty)\ **Post date:** [January 26, 2006, 9:25pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/3 "2006-01-26T21:25:32Z")
It is just handling the number of significant digits differently than you would like. I doubt it happens when you just add or subtract numbers. That is simple. However, if you mix some multiplication or division in, the program might end up with a long string of digits and just try to keep using them. Instead of wronging numbers to the Nth decimal place, it just keeps that result and that tiny fraction can pop up later when you do another operation. You may be able to set the caluculator (what program is it) so that it rounds numbers automatically and you don’t have to worry about it. It is really just a display problem. The fact that the extra digits are there should not have an adverse impact on your results. ---
**Author:** ![SiXSwordS](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@SiXSwordS](https://boards.straightdope.com/u/SiXSwordS)\ **Post date:** [January 26, 2006, 10:09pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/4 "2006-01-26T22:09:34Z")
> [@Shagnasty](#): > > I doubt it happens when you just add or subtract numbers. That is simple. However, if you mix some multiplication or division in, the program might end up with a long string of digits and just try to keep using them. Actually, I don’t mix multiplication and Division in with the addition and subtraction. Here’s another example I just put in all by itself. 118.1 - 116.75 = 1.349999999999994 There is obviously a fractional part of each number… > [@Shagnasty](#): > > You may be able to set the caluculator (what program is it) so that it rounds numbers automatically and you don’t have to worry about it. It is really just a display problem. The fact that the extra digits are there should not have an adverse impact on your results. There is no real problem, just curiosity. In fact, the “display,” just the part you would normally look at if you used a calculator to figure a problem, shows the correct number with the correct number of significant digits. I use the list of numbers so I have a “paper tape” which is a seperate window that prints out each calculation like an adding machine would. > [@ultrafilter](#): > > It’s because the computer represents numbers internally in binary. I’ll risk my ego here a little and ask a stupid question; why doesn’t this happen with the actual adding machine I have or the calculator that sits beside the computer? ---
**Author:** ![scr4](https://avatars.discourse-cdn.com/v4/letter/s/59ef9b/32.png) [@scr4](https://boards.straightdope.com/u/scr4)\ **Post date:** [January 26, 2006, 10:29pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/5 "2006-01-26T22:29:15Z")
I’d consider that a bug. The program is printing out 15 digits even though it’s only accurate to 13 digits or so. The inaccuracy probably comes from converting decimal to [floating point](http://en.wikipedia.org/wiki/Floating_point) and back again. ---
**Author:** ![Capt.Ridley\_s\_Shooting\_Party](https://avatars.discourse-cdn.com/v4/letter/c/cc9497/32.png) [@Capt.Ridley\_s\_Shooting\_Party](https://boards.straightdope.com/u/Capt.Ridley_s_Shooting_Party)\ **Post date:** [January 26, 2006, 10:32pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/6 "2006-01-26T22:32:38Z")
To expand on what **ultrafilter** said, the floating point format that most computers use is incapable of representing a _lot_ of numbers. An n bit binary string is capable of representing 2^n different numbers, but the number of real numbers in any interval is infinite so approximations are needed. This becomes a problem for some kinds of applications, and comparing floating point numbers must therefore be done indirectly, through use of an epsilon value. A good basic explanation of what is going on can be found [here.](http://www.aplawrence.com/Basics/floatingpoint.html) ---
**Author:** ![ultrafilter](https://avatars.discourse-cdn.com/v4/letter/u/3d9bf3/32.png) [@ultrafilter](https://boards.straightdope.com/u/ultrafilter)\ **Post date:** [January 27, 2006, 4:50am UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/7 "2006-01-27T04:50:02Z")
> [@SiXSwordS](#): > > I’ll risk my ego here a little and ask a stupid question; why doesn’t this happen with the actual adding machine I have or the calculator that sits beside the computer? The calculator is probably programmed to be more intelligent about rounding, or at least to swallow the extra digits. The adding machine operates internally in decimal, so it’s not an issue. ---
**Author:** ![TJdude825](https://avatars.discourse-cdn.com/v4/letter/t/dec6dc/32.png) [@TJdude825](https://boards.straightdope.com/u/TJdude825)\ **Post date:** [January 27, 2006, 5:44am UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/8 "2006-01-27T05:44:35Z")
I’m far too tired to even attempt the technical descriptions in this thread, floating points, etc. However, I have an idea. To calculate 118.1 - 116.75, type 11810 - 11675, then divide the answer by 100 (or “move the decimal point back to the left”). Would that give you an accurate answer? ---
**Author:** ![ultrafilter](https://avatars.discourse-cdn.com/v4/letter/u/3d9bf3/32.png) [@ultrafilter](https://boards.straightdope.com/u/ultrafilter)\ **Post date:** [January 27, 2006, 5:47am UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/9 "2006-01-27T05:47:19Z")
No, because the fractional part of the first value can not be represented accurately in limited-precision binary. ---
**Author:** ![engineer\_comp\_geek](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/engineer_comp_geek/32/504_2.png) [@engineer\_comp\_geek](https://boards.straightdope.com/u/engineer_comp_geek)\ **Post date:** [January 27, 2006, 7:36am UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/10 "2006-01-27T07:36:32Z")
> [@SiXSwordS](#): > > I’ll risk my ego here a little and ask a stupid question; why doesn’t this happen with the actual adding machine I have or the calculator that sits beside the computer? Quite honestly, because whoever programmed your calculator program didn’t do a very good job. The way calculators and good programs work is they store numbers as a certain number of digits (some store more digits than others), and possibly may have an expontent multiplier as well. Simple addition and subtraction ends up corresponding to a bunch of integer calculations inside the cpu (calculators and computers both have cpu’s). Storing numbers as a floating point value limits you to the precision of a floating point in the computer, which isn’t all that great and also leads to more weird rounding errors like what you are seeing. The reason the programmer used floating point was just so he/she could take advantage of the processor’s natural floating point ability. This saved them the effort of writing arithmetic processing routines, which they were either too lazy to do themselves or didn’t know how to do. Or maybe their boss gave them too tight of a deadline and they decided to cut corners. Who knows. ---
**Author:** ![Joey\_P](https://avatars.discourse-cdn.com/v4/letter/j/919ad9/32.png) [@Joey\_P](https://boards.straightdope.com/u/Joey_P)\ **Post date:** [January 27, 2006, 12:57pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/11 "2006-01-27T12:57:45Z")
Are you using a calculator or an adding machine. Eitherway, check to see what the decimal is set at. If you’re dealing with money, it should be set to 2, you might have it on F. ---
**Author:** ![Thudlow\_Boink](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/thudlow_boink/32/320_2.png) [@Thudlow\_Boink](https://boards.straightdope.com/u/Thudlow_Boink)\ **Post date:** [January 27, 2006, 1:03pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/12 "2006-01-27T13:03:56Z")
> [@SiXSwordS](#): > > FWIW, I am talking about the calculator on the computer. _Which_ “calculator on the computer”? That is, what calculator program are you using? Perhaps a different one wouldn’t have this quirk. ---
**Author:** ![SiXSwordS](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@SiXSwordS](https://boards.straightdope.com/u/SiXSwordS)\ **Post date:** [January 27, 2006, 4:06pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/13 "2006-01-27T16:06:19Z")
> [@Thudlow Boink](#): > > _Which_ “calculator on the computer”? That is, what calculator program are you using? Perhaps a different one wouldn’t have this quirk. I guess, in the final reckoning, it is just a quirk and I was just curious. It will take me a while to get through the floating point links. Thanks ---
**Author:** ![Whack-a-Mole](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/whack-a-mole/32/141_2.png) [@Whack-a-Mole](https://boards.straightdope.com/u/Whack-a-Mole)\ **Post date:** [January 27, 2006, 4:14pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/14 "2006-01-27T16:14:44Z")
Just to help confuse things further you need to pay attention to how the calculator does its order of operations. For instance, with Microsoft’s built-in calculator if you enter: 1+2\*3=9 If you use the Scientific version of Microsoft’s calculator you get: 1+2\*3=7 The Scientific calculator’s answer is the correct one as it waits for the whole equation to be input before generating an answer. The “basic” calculator does the calculation on the string just input each time a function key is hit (so 1+2_3 becomes 1+2=3, 3_3=9). The Scientific calculator sees the whole 1+2\*3 and correctly applies the order of operations. Just something to be aware of… ---
**Author:** ![SiXSwordS](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@SiXSwordS](https://boards.straightdope.com/u/SiXSwordS)\ **Post date:** [January 27, 2006, 4:23pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/15 "2006-01-27T16:23:31Z")
> [@Whack-a-Mole](#): > > \if you enter: > 1+2\*3=9 > > If you use the Scientific version of Microsoft’s calculator you get: > > 1+2\*3=7 I’ve noticed this too, and it bugs the crap outta me. I have to use four different calculators and each one is different. On one it’s 4 + 2 - to get 4 - 2 = 2 On another it’s 4 [enter] 2 - On another it’s the _basic_ 4 - 2 with no = an finally, just 4 - 2 = 2. Then you bring in the fact that on some, if you _don’t_ hit total or equal or sum or whatever between steps it’s interpreted one way and on others if you _do_ hit some button or another, the damn machine prints out the answer then clears the _[buffer_ so you can’t do other calculations! AARRRGHHH! maybe that should have been in the pit. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [January 27, 2006, 10:12pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/16 "2006-01-27T22:12:53Z")
> [@](#): > > On another it’s the basic 4 - 2 with no = an finally, just 4 - 2 = 2. I don’t follow this… How does it know you’re not about to hit another digit, thereby asking for (say) 4 - 20? ---
**Author:** ![Voyager](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/voyager/32/133_2.png) [@Voyager](https://boards.straightdope.com/u/Voyager)\ **Post date:** [January 27, 2006, 10:25pm UTC](https://boards.straightdope.com/t/computer-calculator-question/341753/17 "2006-01-27T22:25:03Z")
> [@Thudlow Boink](#): > > _Which_ “calculator on the computer”? That is, what calculator program are you using? Perhaps a different one wouldn’t have this quirk. I just checked a calculator on Solaris (CDE) and on my cellphone, and they both came out right. Either your calculator program is buggy, or you have some settings screwed up. Or else you’re running on a Pentium with the FDIV bug. (Sorry, cheap joke.)