Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
miscalculation on "100*e-6"
#5
The floating-point is a representation in binary system, which cannot represent all decimals.
It's mathematically proofed.

Decimal could help, but keep in mind, that it's much slower than floating-point arithmetic.

from decimal import Decimal

print(Decimal(100) * Decimal("1e-6"))
print(Decimal(100e6) * Decimal("1e-6"))
Output:
Decimal('0.000100') Decimal('100.000000')
If you need floating-point with higher precision and fast, then try mpmath.
This will not eliminate the error, but you can get closer to a result.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply


Messages In This Thread
miscalculation on "100*e-6" - by cools0607 - Jun-11-2020, 07:54 AM
RE: miscalculation on "100*e-6" - by bowlofred - Jun-11-2020, 08:19 AM
RE: miscalculation on "100*e-6" - by cools0607 - Jun-12-2020, 07:32 AM
RE: miscalculation on "100*e-6" - by Larz60+ - Jun-11-2020, 05:52 PM
RE: miscalculation on "100*e-6" - by DeaD_EyE - Jun-12-2020, 08:22 AM

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020