Got more questions? Find advice on: SQL | XML | Regular Expressions | Windows
in Search
Welcome to AspAdvice Sign in | Join | Help

Dave Sussman's .NET Blog

writerus drivelus

God made integers, all the rest is the work of the devil

So, working with Yukon at the moment and wasn't having a very good week. Finally got some code working , but some of my conversions weren't giving the right values. This is GIS stuff, so I'm dealing with Latitude & Longitude, converting to decimal values, calculating distances etc. I couldn't work out why things weren't right, and spent hours debugging. Finally, with Alex's help, we realised I was using the wrong type - I should have been using decimal to preserve accuracy in calculations.

Now I understand rounding and the instrinsic problems of storing floating point numbers, but it's just so painful having to do lots of type conversions/casting just so you can get accurate numbers. I mean I'm only dealing with a few decimal places so you'd kinda expect things to be accurate, but oh no. As a good example, start a project in VS. Doesn't matter what type, but break into the debugger. View the immediate window and type 9.2-9 - what do you expect? By and large I'm a fairly optimistic guy, and although my maths skills are pretty poor, even I knew it should be 0.2. But I was wrong. Now call me a pedant. Call me stupid. Call me naive, but don't call me wrong for wanting to believe that such a simple calculation should give an incorrect answer. At what level should we expect rounding errors to occur?

On a side note (and perhaps not seriously, but then again perhaps I am serious), why is it that we have rounding errors at all? Why is there any need to store floating point numbers as actual floating point numbers? After all, they could be stored as integers, all calculations could be done on intergers and accuracy would be preserved. The decimal point is really only needed for display purposes. Of course, it would mean radical changes to every computing platform, but heck, there's no gain without large restructing of the world as we know it.

Sponsor
Published Sunday, July 18, 2004 3:04 PM by dsussman
Filed under: ,

Comments

 

dsussman said:

"Why is there any need to store floating point numbers as actual floating point numbers? After all, they could be stored as integers, all calculations could be done on integers and accuracy would be preserved. The decimal point is really only needed for display purposes"

That is essentially what the Decimal type does.

See the "What are Floating Point Numbers" topic in http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html for why floats are useful (if not always useful).

July 20, 2004 11:08 PM
 

dsussman said:

pedant!

Just kidding. I hear you. We all know computers are BAD at rounding etc, and that floats aren't true numbers... but sheesh, maybe letting them be accurate out to the 10s,100s, or event 1000th's place seems that it would be safe. (Alas it is not).

Frankly, i think only scientists (and politicians) probably ever knowingly use floats.

(oh, and i think floats behave they way they do because they all adhere to IEEE standards... see the following for more info:
http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html
)
July 20, 2004 11:23 PM
 

God made integers, all the rest is the work of the devil said:

December 5, 2007 4:28 PM
Anonymous comments are disabled