Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
33*25.4
#1
Hello,

I was using the IDLE today and typed in 33*25.4 and it evaluated to 838.1999999999999

I also tried the Python web shell and it gave me the same answer?

Does anyone know why this would happen? Have I found a bug of some kind, or is it just an operator error?

Thanks!
Reply
#2
That is the correct value. The binary value closest to 838.2. Computers work with binary numbers, not decimals. Most decimal numbers do not have a binary equivalent.
Reply
#3
Read through the python tutorial on this:
https://docs.python.org/3/tutorial/floatingpoint.html
Reply
#4
Thank you both for your help with this. I will keep this in mind with my programs and find ways to adjust when necessary. I suppose there isn't a way to set up a program to globally round all floats to 8 is there? I didn't see anything like that in the tutorial. They just said to round your final numbers to what you want.

Thanks again!
Reply
#5
Let me guess. You're converting inch to mm.

The slowest solution is to work with Decimals.

from decimal import Decimal



def inch2mm(inch, precision=None):
    value = float(Decimal(str(inch)) * Decimal("25.4"))
    # optional rounding if required
    if precision is not None:
        value = round(value, precision)
    return value
Another trick is to do the calculation with integers instead of floats.
print(33 * 25.4)
print(33 * 254 / 10)
My code examples are always for Python >=3.6.0
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#6
Yes, I am converting from inch to mm and mm to inch every day. I'm an engineer :)

I was also thinking about calculating it as an integer instead of a float, but ended up with this (see below), which I really wasn't expecting either. This base 2 stuff really blows my mind!

>>> format(33*(254/10),'.17g')
'838.19999999999993'
>>> format((33*254)/10,'.17g')
'838.20000000000005'

I now understand why I sometimes see these very small errors in SolidWorks, which is always rounded to 8 decimal places.

Thanks for your help!
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020