Very odd thing with float numbers in Python

Factual Questions
21

For everyday bookkeeping usage in most businesses, the smallest unit available is the cent. The software that we use literally will not accept anything else. But if you’re a bank keeping track of interest, you go down to a lower level of precision, and only present the figures that have been run through a floor function with significance one cent.

I’ve never seen amortization schedules with amounts less than a cent; the difference is made up in the last payment. That doesn’t mean, however, that when banks decide how much interest to pay out or charge when payments have not already been agreed upon to a perfect schedule that they simply round - they keep track as necessary.

One place similar to a bank that I invest with is Prosper, and it’s quite important there since they’re dealing with loans sliced into $25 pieces that they need to be paid on all separately, and the fee they charge per payment is on the order of 2 cents, but needs to scale based on the loan balance remaining. While everything is usually displayed down to the cent, on the account transactions page they offer the ability to hover over amounts and see exactly what the payment or fee was down to 4 decimal places past the cent.

22

It can get even more interesting. I always defined a variable or constant epsilon, usually 0.00000001, and compared A with B by testing | A - B | < epsilon.

But what about the other comparisons? Have you figured out the correct way to do all these other tests?

  • A < B
  • A > B
  • A <= B
  • A >= B
  • A ≠ B
     
Click here for a hint.

In all of these tests, you must test for “nearly equal” first and if so, then treat the values as equal. If they fail the “nearly equal” test, then test for less than or greater than.

I worked for a company that did cash register and inventory management software. The principals of the company simply did not understand any of this. Their software was always coming up with rounding errors, especially problems with comparisons going the “wrong way”. Their solution was to compare numbers by converting them to character strings correct to two decimal places (there was a standard function for this), and then compare the character strings.

23

Worth pointing out that simple equality predicates only scratches the surface of the problems with numerical calculations. You can attend multiple semester courses that have a constant background theme of managing numerical stability. Loss of precision can bite you even in simple arithmetic. Seemingly correct arithmetic expressions can and do yield wildly wrong answers to perplex the unwary. Once you are solving systems with numerical tools your life revolves around this.

24

Python traditionally used the standard c math library, and still does mostly. The c math library is a little bit dumb about floating point numbers. The python maintainers understand that the c library works, and is correct, and if you want anything else – well, (now) there are a couple of arbitrary-precision libraries, that, as well as being arbitrary precision, include optimal handling of floating-point precision.

There is always a small error at the end of a binary representation of a (floating-point) real number. The simplest way to handle this is to accept and ignore it: your real numbers are accurate to the precision of the binary representation.

A slightly more complex way is to realize that, just as a decimal floating point number represents a range of binary floating point numbers, a binary floating point number represents a range of decimal numbers, with the math library free to pick which decimalization it uses.

Knowing that the true value of $4.510000027 is within the limits* 4.51000050 and 4.509999900, a sophisticated math library can choose $4.510000000, $4.51, as the decimal representation.

One way of doing that requires keeping track of the known error limits through the calculation: libraries that aren’t the c standard math library can do that. Another way is by using extra bits, or by using reduced bits: libraries that aren’t the c standard math library do that.

There is also the much simpler problem of just representing 0.10 and 0.25 with the optimal decimal representation. When I put that to the python mailing list 15 years ago, they knew so little about floating point numbers that they told me I didn’t understand floating point numbers. At that point I gave up in disgust.

*I didn’t work out numbers with accurate binary representation! I just pulled those numbers out of my rear! Unless by accident, those numbers do not represent realizable limits for 4.51

25

This “tonal” Bitcoin stuff appears to be the work of a lone kook (some Luke-jr) promoting the “tonal” number system. It doesn’t appear to have any relation to the mainline Bitcoin implementations.

The actual subdivision in active use is the satoshi, which is 1/100,000,000 of a bitcoin. Since only 21,000,000 bitcoins can ever exist in total, only 51 bits are necessary to represent any possible amount exactly. The blockchain itself uses a 64-bit number.

26

The financial world runs on Excel, which is entirely binary floating bit. That’s what spreadsheets are and do and are for.

Fortunately, you only need n+1 bits to ensure absolute accuracy at n bits, and you only need 3.3 bits to get 1 digit, so you only need ~ 3*X + 1 bits to get X digits accuracy out to ~ 10 digits, and double precision (as used by Excel) gives 52 bits, so it’s accurate out to, what, 15 digits.

You just limit the precision of your display to cents, and the numbers magically come out correct. Floating point errors are outside the range.

The standard contracts are specified at whatever (8 ???) because they were written for fixed-point minicomputers, so all your interest, and stock market, and foreign exchange contracts can be handled completely accurately by Excel and other floating point financial systems.

There are limits and exceptions. All financial transactions were originally written in a form suitable for hand calculation, and the change to fixed-point was incomplete and gradual. 25 years ago the American stock markets were in 12.5c points: 1/8 of a dollar: as in “Spanish Dollar” and “Pieces of 8” and “for 2 bits I’d knock your block off”.

Sums across and down a spreadsheet don’t match, as they are required to do (there is a word for that, but I can’t remember it), but they never did: that’s why bankers invented bankers rounding.

Fixed point is unable to handle, and floating point representations loose precision, when the number gets too big. This is a limitation that existed in hand calculation as well. India used to be, and for all I know still is, in multiples of 1000 or 10,000 because the rupiah is too small for handling large financial transactions. People and Fixed point and Single representation couldn’t handle that many significant digits. At some point, numbers have too many digits to be handled accurately even by double-precision numbers, but that is comparatively rare because people don’t want to handle that many significant digits: the finance world is mostly people making contracts, and they are happy with human-level or mini-computer levels of precision, and contracts are written that way.

27

That’s unnecessarily complicated: All you’re doing is changing the threshold.

Example: Suppose that you have a number x, and the specification is to do Thing A with it if it’s less than or equal to 100, and Thing B if it’s greater than 100. The naive code would be something like

if(x<=100.0) result=doA(x);
else result = doB(x);

But let’s say that a difference of 0.01 is “close enough” to count as equality. Your method would then say to code

if(abs(x-100.0) <= 0.01) then result=doA(x);
else if x < 100.0 then result = doA(x);
else result = doB(x);

But this is equivalent to saying

if(x <= 100.01) then result=doA(x);
else result = doB(x);

But of course, it’s even more subtle than that, because your argument would have the same objection to if(x<=100.01) that it did to if(x<=100.0) , and so move the threshold again to 100.02, and so on. The real answer is that in any sane system, your doA and doB functions should have continuity between them (where “continuity” is here defined not as “no discontinuities”, but “no discontinuities larger than the precision we care about”), such that doA(100.0) and doB(100.0) should give the same result (or close enough not to matter). In which case, any distinction between if(x<100.0) and if(x<=100.0) is irrelevant.

Alternately, suppose that, for reasons that are outside of the control of you, the humble coder, you do have a non-sane system that you have to maintain backwards compatibility with. Then, you need to figure out where the threshold really is, and use that, and the threshold you choose should be one that is never close enough to any of the actual inputs to introduce any ambiguity. For instance, if your inputs are amounts of cash tendered (which are integer multiples of a cent), but they’re coded in floating points because the legacy system that nobody wants to change was stupid, and you’re supposed to do something different for amounts less than or equal to a dollar, then the threshold you use should be 1.005 dollars: Anything that’s “actually” less than or “actually” equal to a dollar will be below your threshold, and anything that’s “actually” more than a dollar will be above it, and you’ll never have something that’s right at your threshold.

28

Holy smokes yeah, I wish I had seen this thread a year ago (using a time machine?!).

I went down a “rabbit hole” of trying to debug a simulation I wrote using set of linear inequality constraints to find the optimal solution (using MATLAB, YALMIP and GUROBI). In it, the sum total of one of the variables was adding up to an exact integer and this was a “no-no” in the sense that one of the constraints for that variable was apparently, thanks to the way the solver works, needing to be a strict inequality (< or > not <= >=). The error messages were not revealing; and it took me a long time to wade through it and discover why it was not working. Finally I added a tiny fudge factor to the value of these integers, not enough to affect the solution but enough to build a total value that would never equal the constraint limit. ARGH. Then everything worked fine.

(You see, I am not that well trained in CS but my dissertation has me trying out a number of algorithms and approaches that might best be left to an expert in CS…silly me)

29

It’s Python 3.9 on a 64-bit Windows 10 machine, with PyCharm being used as the editor.

Thanks everybody for answers, it’s clearer now - a matter of how decimal numbers are represented in binary. I’ll count with integer numbers of cents throughout and convert that to dollar figures only once the total is printed.

30

Right, so IEEE 754 (completely standard) double-precision floating-point binary has (almost) 16 decimal digits, and the corresponding floating-point decimal explicitly has 16 digits. You should (depending on what you are doing) be able to “count” amounts up to the trillions without experiencing any kind of magnitude mismatch or catastrophic cancellation.

In fact, I just tried it:

>>> x = 1234567890123.45
>>> y = 0.01
>>> x+y
1234567890123.46
>>> z = x+y
>>> z - x
0.010009765625

You may not like the “extra digits” in the last number, but ISTM there is not a true problem. Or am I overlooking something?
Cf

>>> from decimal import *
>>> x = Decimal('1234567890123.45')
>>> y = Decimal('0.01')
>>> z = x+y
>>> z
Decimal('1234567890123.46')
>>> z-y
Decimal('1234567890123.45')
>>> z-x
Decimal('0.01')
>>>
31

I couldn’t say for certain either, but order of operations is a good bet. Floating-point math is not associative. That is, (a+b)+c does not (in general) equal a+(b+c). The bits that get rounded off aren’t the same and so you get a slightly different result. If different Excel versions do the math in a different order, you could run into this.

I learned a similar lesson in computer graphics. One of the ways of simulating reflections is to shoot a ray at a shiny object, figure out the reflection vector from the bounce, and then shoot another way from where it hit. You then get the color you need from the reflected ray (or do another bounce, if necessary).

If you implement this in the obvious way, you get a nasty speckle pattern. The reason is that at the bounce point, the new ray ends up inside the object about half the time. Just by a tiny amount of course, but when you need shoot the reflected ray it ends up black because it hit the inside of the very object you were trying to bounce from.

The solution is to add a little epsilon value to move the bounce outside.

32

One might add that another source of problems is transiting values via textual form. Outputing a value as readable text, then parsing it back into a binary form can lead to a loss of precision. Even when you use exponential form. For things like spreadsheets, csv, xml, or for others json, yaml etc based interchange of data can thus silently lead to data being very very slightly different. Students of the origins of chaos theory might remember just such an issue.

And take a moment to feel for the poor fools coding in javascript. The language only defines number as the numerical type. There is no native integer. All integer values are represented as 64 bit float. It even uses floating point values as array indecies. :no_mouth: For values where the integer value is representable within the number of bits in the mantissa (53), the values are exact. But there is a point where you run out of precision. It is your problem to manage this. The language helpfully provides a constant: Number.MAX_SAFE_INTEGER is the largest integer you can use and rely on the language preserving the value. Nothing in the language or runtime will check if you ever run over this value at any time. 53 bits might seem a lot, but it isn’t that hard to find yourself needing to manage integers larger than this.

There is a new add on - BigInt - but it isn’t type compatible with number. You must explicitly construct new BigInt objects from number in order to do much, and cope with possible loss of precision on the way back.

33

It was also important to the development of chaos theory.
The observation was that a model that ran on a system with 16-bit floats eventually generated wholly different results to the same model being run on a system with 32-bit floats (these details are IIRC).

This was surprising, since the input data was nowhere near precise enough to require 32 bits floats to store, and the model was deterministic.

34

I understood that: After figuring out all the logic for doing all the “approximately” comparisons (where you test for “nearly equal” first), I realized that it was equivalent to what you say.

But as you also (almost) said, this issue is worse than having functions that are discontinuous at the supposed-to-be-equal point. The bigger problem arises when you really need to call distinctly different procedures depending on the comparison, that do entirely differently things.

But note this (bold added by me):

and this:

As far as I can see, this is the first mention here of “catastrophic cancellation”. Careless and unskilled use of floating arithmetic can produce drastic losses of precision that you would never expect, and it doesn’t necessarily result from the vagaries of binary notation.

Consider, for example, two decimal-coded numbers having seven significant digits, and you want to subtract them:
     3.141593
    -3.141551
    --------------
     0.000042
Well, surprise surprise! Your subtraction of two numbers with seven sigfigs each, gives a result with only two significant digits! Now imagine some heavy-duty number-crunching (e.g., an atomic explosion simulation at the molecular level, running for 40 hours on the most modern multi-megaflop supercomputer). What hope do you have of having any meaningful results by the end of that?

For this, we have those semester-long courses that Francis_Vaughan mentions, in which you spend the entire class studying numerical analysis and the techniques you need to do useful computational work to get good results.

Consider a trivial example: the computation: a * ( b - c )
The subtraction of (b-c) could very well produce the catastrophic cancellation shown here. Multiplying that result by a could just magnify the imprecision. It might be better to compute (a * b) - (a * c) to get a less-imprecise result. If you have a chain of computations, there could be a whole lot of re-arrangements you could to get the best possible results. (And beware of smart-ass optimizing compilers, which might do algebraic optimizations that ruin that! That’s why good compilers should have a “no-optimize” option.)

35

What problems with “very large prime numbers” do you think occur when calculating in binary rather than in decimal?

36

Let’s take an example from cash registers, because that’s what you said that you worked on. Suppose that a store is running a promotion, where the customer gets $5 off any order of $30 or more. That might seem like one of the cases you’re describing, where slightly different values lead to doing completely different things… but it shouldn’t be. If you implement it strictly as “the discount only applies if x >= 30.00”, then what happens with the customer whose order comes to $29? They’re not going to pay $29; they’re going to find some knickknack in the impulse rack by the cashier that costs $1, to bring their total to $30, and so pay only $25. And if the discount is programmed intelligently, this isn’t even necessary: For any order between $25 and $30, the cash register should do the equivalent of including a virtual knickknack, of just such a price as to bring the total to the threshold that qualifies for the discount. In other words, the behavior should be “if price before discount is in ($25,$30), then price after discount = $25, and if price before discount is $30 or more, then price after discount = price - $25”. Which brings it back to continuity, because now it doesn’t matter which interval I count the endpoints as being a part of.

There are worse ways to transfer values. Like, say, in textual form… that’s then printed out and OCRed back in.

37

Like this?

38

Even better - I have worked with old geological magnetic field survey data that had been plotted onto a contour map and then, years later, re-digitised from a physical printed map. They had lost the original data (or never actually had it). The noise was, shall we say, rather high.

39

No excuse for “all I know”, when a simple google search will tell you that :slight_smile:

  1. Rupiah is the currency of Indonesia, not India.
  2. Rupee, ₹, is the currency of India.
  3. In Indian numbering system; 1,00,000 is lakhs and 1,00,00,000 is crores. This system has been used since ancient times and it has nothing to do with " too small for handling large financial transactions"
    Indian numbering system - Wikipedia
  4. For comparison purposes, 1 USD is about 70 Indian Rupees, about 110 Japanese Yen, and about 90 Argentine Pesos (approximately based on today’s exchange rates)
40

A Fermats Library post showed up today which mentioned an Ariane 5 rocket oopsie that involved a conversion from a 64bit float to a 16bit signed integer which led to an overflow which led to an unplanned energetic disassembly of the rocket.

Some details in this article,

https://hownot2code.com/2016/09/02/a-space-error-370-million-for-an-integer-overflow/