Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Precision conversion
#1
I have an output file with a series of values. Some of these are high precision values...

1.05200D+09 1.05200D+09 9.94376D+31 3.66754D+10 7.52265D+31 7.52265D+31 4.99722-235

I want to read these into my Python program, maintaining the level of precision as much as possible.

Most of this is straight forward; I can look for the sign (+ or -) and the D, etc.

However, I am then wondering about how to store the values within Python. Simply taking each mantissa and multiplying it by 10 to the power of the exponent would work, but would Python be capable of holding such a small value as 4.99722-235 or as large as 1.01137D+34 (or larger)?
Always nice to post a question, then solve it because the problem never feels so isolated!

Doing exactly as I have stated and I have maintained the accuracy required.
def convert(oldValue):
	if oldValue.find('D') != -1:
		mantissa = float(oldValue[0: oldValue.find('D')])
	else:
		if oldValue.find('+') != -1:
			mantissa = float(oldValue[0: oldValue.find('+')])
			
		else:
			mantissa = float(oldValue[0: oldValue.find('-')])
			
	if oldValue.find('+') != -1:
		exponent = int(oldValue[oldValue.find('+'):])
	else:
		exponent = int(oldValue[oldValue.find('-'):])
	
	newValue = mantissa * (math.pow(10, exponent))

    return newValue
Reply
#2
Those are not high precision values. There should be no need to do anything special to read the values other than convert the "D" to an "e", and then use float() to convert the modified string to a number.

I think "4.99722-235" is invalid. I would interpret that as 4.99722 minus 235, an equation.
print(convert('9.94376D+31'))
print(convert('4.99722-235'))
print(float('9.94376D+31'.replace('D', 'e')))
print(float('4.99722D-235'.replace('D', 'e')))
Output:
9.94376e+31 4.9972200000000005e-235 9.94376e+31 4.99722e-235
Reply
#3
The problem is that the standard output for Fortran is to use the D, except where the exponent is 3 or more digits long; then the D is dropped.

So, in Fortran output 4.99722-235 is equivalent to 4.99722D-235 or 4.99722e-235; and not an equation.

Aside from that (and this only affects the first two lines and only in two results, but these cannot be ignored), your solution is far nicer than mine. Smile
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Gmpy2 Newbie Working on Precision charlesrkiss 5 573 Jan-23-2024, 04:23 PM
Last Post: charlesrkiss
  Precision with Decimal Type charlesrkiss 9 759 Jan-18-2024, 06:30 PM
Last Post: charlesrkiss
Photo Filtering data and precision during calculations Scientifix 0 1,790 Mar-30-2021, 01:00 PM
Last Post: Scientifix
  High-Precision Board Voltage Reading from Python into PD Liam484 1 2,094 Mar-29-2021, 02:57 PM
Last Post: Marbelous
  Decimal (Global precision setting) Valentina 3 3,658 Aug-23-2019, 11:53 AM
Last Post: Valentina
  python decimal scale precision massimo_m 5 5,447 Aug-22-2019, 11:47 AM
Last Post: perfringo
  Trouble with decimal precision Cassie 4 3,147 Feb-14-2019, 08:23 PM
Last Post: Cassie
  precision error chris64 1 2,249 Nov-29-2017, 09:40 PM
Last Post: micseydel

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020