Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
miscalculation on "100*e-6"
#4
I appreciate your reply to my problem. Your code and background are very useful for me.
Although I am still puzzled with "format(100*1e-6, ".15e") -->9.999999999999999e-05"...

(Jun-11-2020, 08:19 AM)bowlofred Wrote: Many numbers cannot be exactly represented in the computer's floating point representation. So it stores as close as it can get.

You might be better off using format to display the number rounded to some decimal points.

>>> format(100*1e-6, "f")
'0.000100'
>>> format(100*1e-6, ".17f")
'0.00010000000000000'
>>> format(100*1e-6, ".20f")
'0.00009999999999999999'
or if you prefer exponential format...
>>> format(100*1e-6, "e")
'1.000000e-04'
>>> format(100*1e-6, ".11e")
'1.00000000000e-04'
>>> format(100*1e-6, ".15e")
'9.999999999999999e-05'
For background, https://docs.python.org/3.8/tutorial/floatingpoint.html
Reply


Messages In This Thread
miscalculation on "100*e-6" - by cools0607 - Jun-11-2020, 07:54 AM
RE: miscalculation on "100*e-6" - by bowlofred - Jun-11-2020, 08:19 AM
RE: miscalculation on "100*e-6" - by cools0607 - Jun-12-2020, 07:32 AM
RE: miscalculation on "100*e-6" - by Larz60+ - Jun-11-2020, 05:52 PM
RE: miscalculation on "100*e-6" - by DeaD_EyE - Jun-12-2020, 08:22 AM

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020