Oct-27-2020, 02:46 PM
Yes, I am converting from inch to mm and mm to inch every day. I'm an engineer :)
I was also thinking about calculating it as an integer instead of a float, but ended up with this (see below), which I really wasn't expecting either. This base 2 stuff really blows my mind!
>>> format(33*(254/10),'.17g')
'838.19999999999993'
>>> format((33*254)/10,'.17g')
'838.20000000000005'
I now understand why I sometimes see these very small errors in SolidWorks, which is always rounded to 8 decimal places.
Thanks for your help!
I was also thinking about calculating it as an integer instead of a float, but ended up with this (see below), which I really wasn't expecting either. This base 2 stuff really blows my mind!
>>> format(33*(254/10),'.17g')
'838.19999999999993'
>>> format((33*254)/10,'.17g')
'838.20000000000005'
I now understand why I sometimes see these very small errors in SolidWorks, which is always rounded to 8 decimal places.
Thanks for your help!