Python Forum

Hi,
I have multiple CSV files in the folder, I want to read them parallelly. Each file takes 1min to read and do some calculations. I want to read CSV files simultaneously (parallelly) and concate the results (for example First column mean from each file and put into col_mean_df)

Show your code so far, and where you think there is a bottleneck.

I use below code.

import os
import pandas as pd

df_result =pd.DataFrame()

directory = os.path.join("D:\\","\PythonCodes\inputmultifiles")
for root,dirs,files in os.walk(directory):
    for file in files:
        f = os.path.join(directory,file)

        if f.endswith(".csv"):
           ff=pd.read_csv(f)
           tmp = ff['Name']
           print(tmp)
           df_result= pd.concat([df_result,ff['Name']])
           
df_result = df_result.reset_index(drop=True)      
df_result.columns = ['New_col']