python convert multiple files to multiple lists

[MCL169]

year = '1990'

filename = open('FF_abbreviated.txt')

for i in filename:
    print(i)

Output:
19260701    0.09    0.22    0.30   0.009

19260702    0.44    0.35    0.08   0.009

19260706    0.17    0.26    0.37   0.009

19260802    0.82    0.21    0.01   0.010

19260803    0.46    0.39    0.38   0.010

19260804    0.35    0.15    0.32   0.010

19260901    0.54    0.41    0.08   0.010

19260902    0.04    0.06    0.23   0.010

19260903    0.48    0.34    0.09   0.010

19270103    0.97    0.21    0.24   0.010

19270104    0.30    0.15    0.73   0.010

19270201    0.00    0.56    1.09   0.012

19270202    0.72    0.23    0.18   0.012

19270203    0.17    0.22    0.08   0.012

19270301    0.38    0.07    0.57   0.011

19270302    1.12    0.10    0.22   0.011

19270303    1.01    0.11    0.04   0.011

19280103    0.43    0.90    0.20   0.010

19280104    0.14    0.47    0.01   0.010

19280105    0.71    0.14    0.15   0.010

19280201    0.25    0.56    0.71   0.014

19280202    0.44    0.15    0.18   0.014

19280203    1.12    0.48    0.42   0.014

19280301    0.23    0.04    0.12   0.011

19280302    0.07    0.01    0.66   0.011

19280303    0.49    0.01    0.64   0.011


Process finished with exit code 0

I'm trying to create a list for each line using delimiter so i can sum totals of a column

[Larz60+]

the following will convert file contents to list

year = '1990'
filelist = []
 
with open('FF_abbreviated.txt') as fp:
    for line in fp:
        filelist.append(line.strip())

# check list contents
for item in filelist:
    print(item)

[snippsat]

I'm trying to create a list for each line using delimiter so i can sum totals of a column

This task can also be simpler if use Pandas,as eg sum a column is a very common task do in Pandas.
To give a example.

import pandas as pd

with open('FF_abbreviated.txt', 'r') as f:
    data = [i.strip().split() for i in f]

columns = ['Date', 'col1', 'col2', 'col3', 'col4']
df = pd.DataFrame(data, columns=columns)
df['Date'] = pd.to_datetime(df['Date'], format='%Y%m%d')
df.iloc[:, 1:] = df.iloc[:, 1:].apply(pd.to_numeric)

Look at data Types and the DataFrame.

>>> df.dtypes
Date    datetime64[ns]
col1           float64
col2           float64
col3           float64
col4           float64
dtype: object

>>> df
        Date  col1  col2  col3   col4
0 1926-07-01  0.09  0.22  0.30  0.009
1 1926-07-02  0.44  0.35  0.08  0.009
2 1926-07-06  0.17  0.26  0.37  0.009
3 1926-08-02  0.82  0.21  0.01  0.010
4 1926-08-03  0.46  0.39  0.38  0.010
5 1926-08-04  0.35  0.15  0.32  0.010
6 1926-09-01  0.54  0.41  0.08  0.010

As you see have i convert types Data(datetime64)(which could be just integers) and float64 for rest of columns.
Now can sum up.

# One column 
>>> df['col1'].sum()
2.87

# col1 + col4
>>> df[['col1', 'col4']].values.sum()
2.936999999999999

# All columns
>>> df.iloc[:, 1:].sum(axis=0)
col1    2.870
col2    1.990
col3    1.540
col4    0.067
dtype: float64

[MCL169]

I'm trying to create a list for each line using delimiter so i can sum totals of a column

This task can also be simpler if use Pandas,as eg sum a column is a very common task do in Pandas.
To give a example.

import pandas as pd

with open('FF_abbreviated.txt', 'r') as f:
    data = [i.strip().split() for i in f]

columns = ['Date', 'col1', 'col2', 'col3', 'col4']
df = pd.DataFrame(data, columns=columns)
df['Date'] = pd.to_datetime(df['Date'], format='%Y%m%d')
df.iloc[:, 1:] = df.iloc[:, 1:].apply(pd.to_numeric)

Look at data Types and the DataFrame.

>>> df.dtypes
Date    datetime64[ns]
col1           float64
col2           float64
col3           float64
col4           float64
dtype: object

>>> df
        Date  col1  col2  col3   col4
0 1926-07-01  0.09  0.22  0.30  0.009
1 1926-07-02  0.44  0.35  0.08  0.009
2 1926-07-06  0.17  0.26  0.37  0.009
3 1926-08-02  0.82  0.21  0.01  0.010
4 1926-08-03  0.46  0.39  0.38  0.010
5 1926-08-04  0.35  0.15  0.32  0.010
6 1926-09-01  0.54  0.41  0.08  0.010

As you see have i convert types Data(datetime64)(which could be just integers) and float64 for rest of columns.
Now can sum up.

# One column 
>>> df['col1'].sum()
2.87

# col1 + col4
>>> df[['col1', 'col4']].values.sum()
2.936999999999999

# All columns
>>> df.iloc[:, 1:].sum(axis=0)
col1    2.870
col2    1.990
col3    1.540
col4    0.067
dtype: float64

This is all really useful and I appreciate the input.I can't wait to get off work and go work on these things because I really want to like my professor says "own this" . Hopefully one day I can help others along the way.

[snippsat]

If this is basic Python training should also try to the task with Python,Pandas is a different beast🦄
To give a example,eg result of col1 2.87.

lst = []
with open('FF_abbreviated.txt') as fp:
    for line in fp:
        line = line.strip().split()
        col1 = line[1]
        lst.append(float(col1))

So now have list of col1 and can eg use sum().

>>> lst
[0.09, 0.44, 0.17, 0.82, 0.46, 0.35, 0.54]
>>> sum(lst)
2.87

Can also try to do it without using sum(),all this is good training for learning Python.

[MCL169]

If this is basic Python training should also try to the task with Python,Pandas is a different beast🦄
To give a example,eg result of col1 2.87.

lst = []
with open('FF_abbreviated.txt') as fp:
    for line in fp:
        line = line.strip().split()
        col1 = line[1]
        lst.append(float(col1))

So now have list of col1 and can eg use sum().

>>> lst
[0.09, 0.44, 0.17, 0.82, 0.46, 0.35, 0.54]
>>> sum(lst)
2.87

Can also try to do it without using sum(),all this is good training for learning Python.

Absolutely. Im late to the party so im looking to immerse myself in this language. I'm in a certificate course and I'm thinking about asking some of my fellow students if they want to start a study group. I follow some good Udemy courses but I really like to interactive Zoom approach maybe I'll find something that fits the bill.

[Iqratech]

Sure, if you want to calculate the sum of values in col1 without using the sum() function, you can achieve this by using a loop to iterate through the values in lst and accumulate the sum. Here's an example:

python
Copy code
lst = [0.09, 0.44, 0.17, 0.82, 0.46, 0.35, 0.54]

# Calculate the sum without using sum()
total_sum = 0
for value in lst:
total_sum += value

print("Sum of col1:", total_sum)
This code initializes total_sum to 0 and then iterates through each value in lst, adding it to total_sum. Finally, it prints the calculated sum.

This approach provides an alternative way to calculate the sum without relying on the built-in sum() function.

buran write Nov-25-2023, 07:15 AM:
Please, use proper tags when post code, traceback, output, etc.
See BBcode help for more info.