Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
miscalculation on "100*e-6"
#2
Many numbers cannot be exactly represented in the computer's floating point representation. So it stores as close as it can get.

You might be better off using format to display the number rounded to some decimal points.

>>> format(100*1e-6, "f")
'0.000100'
>>> format(100*1e-6, ".17f")
'0.00010000000000000'
>>> format(100*1e-6, ".20f")
'0.00009999999999999999'
or if you prefer exponential format...
>>> format(100*1e-6, "e")
'1.000000e-04'
>>> format(100*1e-6, ".11e")
'1.00000000000e-04'
>>> format(100*1e-6, ".15e")
'9.999999999999999e-05'
For background, https://docs.python.org/3.8/tutorial/floatingpoint.html
Reply


Messages In This Thread
miscalculation on "100*e-6" - by cools0607 - Jun-11-2020, 07:54 AM
RE: miscalculation on "100*e-6" - by bowlofred - Jun-11-2020, 08:19 AM
RE: miscalculation on "100*e-6" - by cools0607 - Jun-12-2020, 07:32 AM
RE: miscalculation on "100*e-6" - by Larz60+ - Jun-11-2020, 05:52 PM
RE: miscalculation on "100*e-6" - by DeaD_EyE - Jun-12-2020, 08:22 AM

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020