# Traditional rounding 'bias'. I don't get it. **URL:** \ **Category:** Factual Questions\ **Created:** [June 5, 2020, 3:21am UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085 "2020-06-05T03:21:36Z")\ **Posts on this page:** 20\ **Page:** 4
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [June 12, 2020, 11:10am UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/61 "2020-06-12T11:10:31Z")
I mentioned the Jpeg tricks because they seemed interesting. We’re not talking of banks that misplace a few pennies, but of lossy compression methods (not just Jpeg) which get a big performance boost by understanding rounding bias. > [@DPRK](#): > > Perhaps you are more familiar with JPEG and related standards- I have not read any of them all the way through, so cannot say what “tricks” are legal according to the standard… > you must take into account that the decompressor may be any baseline JPEG and does not know how you might have massaged the quantization step. It’s safe to say that I have more than a passing familiarity with Jpeg. 🆒 The compression “trick” I mentioned was designed to operate with any ordinary decompressor. (And obviously, reduction of the frequency of ±1 would immediately yield a Huffman code optimization.) The decompression “trick” I mentioned operates with any ordinary compressor. (Jpeg files can include custom markers for compressor-decompressor cooperation to get further improvement, but the tricks I described don’t require them.) The tricks are so obvious that it may be a mystery why Jpeg did NOT explicitly incorporate them. I am aware of one company which kept its knowledge of these ideas secret to obtain a competitive advantage over “vanilla” implementations. (I had a hunch what they were doing and confirmed it with a code disassembler.) People on the Jpeg-2000 committee (some of who were aware of my own paper on this topic) did add a new quantization parameter: > [@](#): > > In JPEG 2000, there is some indication that using nz ~= 0.25 (i.e., dead-zone size about 1:5D) can provide a very slight decrease in MSE and generate more visually pleasing low-level texture reconstruction. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [June 12, 2020, 1:13pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/62 "2020-06-12T13:13:11Z")
If the company was keeping secret that they were doing this, then it couldn’t have been protected under patent, and anyone who managed to reverse-engineer their trick would be legally entitled to do whatever they wished with it (as long as they did so without help from any employees of that company, who are probably bound by their contracts). ---
**Author:** ![Pleonast](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pleonast/32/1183_2.png) [@Pleonast](https://boards.straightdope.com/u/Pleonast)\ **Post date:** [June 12, 2020, 1:35pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/63 "2020-06-12T13:35:18Z")
> [@septimus](#): > > I am so sad. ☹ I divulge little-known insights into lossy compression methods, which will improve image quality for any given compression ratio, and there is no response — Zero. The tricks are explicitly about _rounding_, the topic of the thread. Perhaps I should have worked the name of Kellyanne Conway or Monty Hall into the post! 🙂 I found your post very interesting, but I can’t reply to it other than to say that. I don’t like replying to GQ posts without adding to the discussion. ---
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [June 12, 2020, 2:29pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/64 "2020-06-12T14:29:31Z")
Thank you, **Pleonast**! Your kind words are appreciated. > [@septimus](#): > > (Disclaimer: At some point it **may** have been illegal for you to use the “tricks” described: they probably violated claims in some U.S. patents (now expired?). IANAL and I’m certainly N _your_ L.) I have a bad habit of tongue-in-cheekiness and this paragraph is a case in point. A few decades ago, there was a craze to patent software algorithms. Were they even patentable? Some patent attorneys insisted that the improper patent claim “I claim the algorithm which blah blah blah” could be rendered proper by simply adding four words — “I claim an apparatus which implements the algorithm which blah blah blah.” (Even though the apparatus wasn’t the inventor’s at all — just an off-the-shelf Pentium or whatever.) The U.S. PTO would eventually get tired of arguing and issue the patent, but would it stand up in court if challenged? This software-patenting craze did generate income for patent attorneys, so I guess it was at least as good as breaking windows. :smack: ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [June 12, 2020, 4:01pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/65 "2020-06-12T16:01:24Z")
Patents for methods or techniques, as opposed to physical objects, were well-established long before the age of computers. I don’t see why a method patent couldn’t apply to an algorithm. ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [June 12, 2020, 4:19pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/66 "2020-06-12T16:19:36Z")
Many patent jurisdictions specifically exclude computer programs, among other things. So one could argue that an algorithm is a description of a computer program and therefore not patentable, if it is not already excluded as an “abstract idea”. On the other hand, certain “practical applications” or “implementations” of a mathematical formula or algorithm are theoretically patentable… ---
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [June 12, 2020, 4:46pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/67 "2020-06-12T16:46:16Z")
There are dozens of court cases on the subject of patenting mathematical algorithms, several of which went to the Supreme Court. Consider _Parker v. Flook_ (1978): > [@Justice John Paul Stevens joined by 5 other Justices](#): > > " ‘A principle, in the abstract, is a fundamental truth; an original cause; a motive; these cannot be patented, as no one can claim in either of them an exclusive right.’ _Le Roy v. Tatham_, 14 How. 156, 175, 14 L.Ed. 367. Phenomena of nature, though just discovered, mental processes, and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work." 409 U.S., at 67, 93 S.Ct., at 255. > … > Respondent correctly points out that this language does not apply to his claims. He does not seek to “wholly preempt the mathematical formula,” since there are uses of his formula outside the petrochemical and oil-refining industries that remain in the public domain. And he argues that the presence of specific “post-solution” activity—the adjustment of the alarm limit to the figure computed according to the formula distinguishes this case from Benson and makes his process patentable. We cannot agree. > > The notion that post-solution activity, no matter how conventional or obvious in itself, can transform an unpatentable principle into a patentable process exalts form over substance. A competent draftsman could attach some form of post-solution activity to almost any mathematical formula; the Pythagorean theorem would not have been patentable, or partially patentable, because a patent application contained a final step indicating that the formula, when solved, could be usefully applied to existing surveying techniques. The concept of patentable subject matter under § 101 is not “like a nose of wax which may be turned and twisted in any direction . . . .” _White v. Dunbar_, 119 U.S. 47, 51, 7 S.Ct. 72, 74, 30 L.Ed. 303. One test, I think, is to eliminate any details of a mathematical formula and see if the result is vacuous. In our example, “I claim a process wherein said intermediate arithmetic value is altered in some way …” Sounds almost vacuous to me. But the Supreme Court has reversed itself and then reversed again since _Parker v. Flook_. Contact an attorney! ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [June 13, 2020, 8:42am UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/68 "2020-06-13T08:42:14Z")
Leaving aside JPEG and JPEG2k, there are many newer standards and proposed standards which include steps along the lines of computing the discrete cosine transform or wavelet transform of a block, followed by quantization and entropy coding. How do they handle the inevitable rounding? ---
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [June 15, 2020, 4:06pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/69 "2020-06-15T16:06:26Z")
JPEG2k was after my time, let alone anything newer, but since no one else answered I’ll summarize the basic difference in quantization rounding between Jpeg and Jpeg2k. Recall that ‘x’ in the following has already been multiplied by a scaling constant which depends on psychometrics and user-desired compression ratio. Jpeg: … -1.5 \< x \< -0.5 --\> x = -1 -0.5 \< x \< +0.5 --\> x = 0 +0.5 \< x \< +1.5 --\> x = +1 +1.5 \< x \< +2.5 --\> x = +2 +2.5 \< x \< +3.5 --\> x = +3 … The spec offers no flexibility beyond this. (One implementation increases the x=0 domain to -1.5 \< x \< +1.5 — i.e. eliminates all ±1’s which would be encoded — in image regions the user specifies as being of low interest.) Jpeg2k, default: … -2.0 \< x \< -1.0 --\> x = -1.5 -1.0 \< x \< +1.0 --\> x = 0 +1.0 \< x \< +2.0 --\> x = +1.5 +2.0 \< x \< +3.0 --\> x = +2.5 +3.0 \< x \< +4.0 --\> x = +3.5 … _ **This policy recovers most of the inefficiency that applied to Jpeg** _ (cf. my ‘tricks’ above); further meddling would yield only diminished returns. However IIUC two optional parameters are provided to further tune this quantization: The width of the zero region can be adjusted, and the reconstruction values can be moved slightly toward zero to reduce average squared error. (Never mind more complicated modes of Jpeg2k.) ---
**Author:** ![RaftPeople](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@RaftPeople](https://boards.straightdope.com/u/RaftPeople)\ **Post date:** [June 16, 2020, 4:49am UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/70 "2020-06-16T04:49:46Z")
> [@Chronos](#): > > Why would a cash register be using floating point for anything? Just track everything as an integer number of cents. You’re right that float isn’t appropriate for almost any business/accounting software, but neither is integer for currency and catch weight values. Fixed precision decimal/numeric data types are what is used. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [June 16, 2020, 12:46pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/71 "2020-06-16T12:46:32Z")
Fixed precision is the same thing as integer, just with different units. ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [June 16, 2020, 12:55pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/72 "2020-06-16T12:55:05Z")
It’s not like the financial programmer can just ignore rounding issues just because she uses decimal and/or fixed-point numbers. All the issues (possibility of incorrect rounding, etc) discussed in this thread still apply. ---
**Author:** ![RaftPeople](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@RaftPeople](https://boards.straightdope.com/u/RaftPeople)\ **Post date:** [June 16, 2020, 7:52pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/73 "2020-06-16T19:52:52Z")
> [@Chronos](#): > > Fixed precision is the same thing as integer, just with different units. I was assuming you were using the term “integer” to mean the software data type of integer because the context was software running on the cash register. Maybe you were using the term more generically. My point was that there are data types that are designed for this type of stuff as opposed to the integer data type which just requires a lot more work to achieve the same result as the decimal/numeric types. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [June 16, 2020, 9:58pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/74 "2020-06-16T21:58:45Z")
No, I mean that a decimal type is the same thing as an integer type. Like, measuring in dollars to two places after the decimal point is equivalent to measuring in integer cents. ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [June 17, 2020, 2:37am UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/75 "2020-06-17T02:37:52Z")
Not exactly the same; you need to store the number of digits after the decimal point also, 2 in your example, and keep track of it when multiplying, etc. ---
**Author:** ![Pleonast](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pleonast/32/1183_2.png) [@Pleonast](https://boards.straightdope.com/u/Pleonast)\ **Post date:** [June 17, 2020, 4:00am UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/76 "2020-06-17T04:00:17Z")
> [@DPRK](#): > > Not exactly the same; you need to store the number of digits after the decimal point also, 2 in your example, and keep track of it when multiplying, etc. If you’re storing the place of the decimal point, it’s floating point, not fixed point. ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [June 17, 2020, 4:10am UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/77 "2020-06-17T04:10:04Z")
> [@Pleonast](#): > > If you’re storing the place of the decimal point, it’s floating point, not fixed point. I meant fixed point. Say you have one decimal digit, and the operation is 1.2 x 3.4. The intermediate value is 4.08, and it must be correctly rounded to 4.1. On the other hand, 12 x 34 = 408. So, arithmetic is not quite the same. ---
**Author:** ![Moris](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/moris/32/2846_2.png) [@Moris](https://boards.straightdope.com/u/Moris)\ **Post date:** [June 17, 2020, 1:07pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/78 "2020-06-17T13:07:04Z")
There’s something else in rounding that bugs me. 2.3217 could be rounded up to 2.328. And ultimately to 2, if we need an integer. But 4.4449 could be rounded up to 4.445, and then 4.445 to 4.45, then to 4.5, then to 5. I’m not saying that’s happening very often, but if a multiple people are working on the same spreadsheet, it’s possible. I did it myself a few times, absentmindedly. I’ m just saying - rounding is a convention, and we should be careful following the rules of that convention. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [June 17, 2020, 3:49pm UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/79 "2020-06-17T15:49:35Z")
Under what circumstances would you ever be multiplying two amounts of money? And even if you do, if you have two amounts of money expressed in cents, and you multiply them, you’ll end up with a perfectly valid answer in square cents. Why are square cents any less valid of a unit than square dollars? ---
**Author:** ![RaftPeople](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@RaftPeople](https://boards.straightdope.com/u/RaftPeople)\ **Post date:** [June 24, 2020, 12:40am UTC](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085/80 "2020-06-24T00:40:32Z")
It’s not typically two amounts of money, it’s typically money (e.g. unit price, invoice sub total) multiplied by things like shipped weight (typically food uses catch weight systems), or yards (e.g. 3.07 yards consumed) or currency conversion (typically 7 decimal positions), or tax rates, or discounts, or prorating costs across multiple invoices or across lines, etc., etc., Manually tracking the decimal position during all of these calculations requires extra work and is more error prone than just using the software (and sometimes hardware) that already handles it. [Previous page](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085.md?page=3) [Next page](https://boards.straightdope.com/t/traditional-rounding-bias-i-dont-get-it/855085.md?page=5)