January 06, 2006
From the SoftwareAppreciation desk. This article on rounding algorithms is an excellent companion to FractalDesign.

I recall coming across my first lesson in rounding on the job (since it wasn't included in my jazz studies) working on a payroll application, wondering why I was getting an unexpected result of a round. That was when I was first introduced to Banker's Rounding (called “Round-half-even” in the above article).

The arithmetic rounding (“round-half-up”) I was taught in grade-school isn't a good idea in a case like the following:

original2.5 + 3.5 + 4.5 + 5.5 = 16
arithmetic rounding3.0 + 4.0 + 5.0 + 6.0 = 18
banker's rounding2.0 + 4.0 + 4.0 + 6.0 = 16

The extra interesting thing I learned in that particular case was that most of the core Delphi routines I was using passed the rounding off to the processor (although I found one which I couldn't figure how it did its rounding), and the processor had a global switch determining how it would do its rounding (in one of four ways, none of them arithmetic rounding), the default being Banker's Rounding. But, IIRC, anyone could set this flag on the processor with the right assembly code and it would affect every running application. Talk about hard to track down bugs.

tags: ComputersAndTechnology
comments powered by Disqus