+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: How to read csv files parallay (/thread-30504.html)
How to read csv files parallay - Mekala - Oct-23-2020
Hi,
I have multiple CSV files in the folder, I want to read them parallelly. Each file takes 1min to read and do some calculations. I want to read CSV files simultaneously (parallelly) and concate the results (for example First column mean from each file and put into col_mean_df)
RE: How to read csv files parallay - Larz60+ - Oct-23-2020
Show your code so far, and where you think there is a bottleneck.
RE: How to read csv files parallay - Mekala - Oct-24-2020
I use below code.
import os
import pandas as pd
df_result =pd.DataFrame()
directory = os.path.join("D:\\","\PythonCodes\inputmultifiles")
for root,dirs,files in os.walk(directory):
for file in files:
f = os.path.join(directory,file)
if f.endswith(".csv"):
ff=pd.read_csv(f)
tmp = ff['Name']
print(tmp)
df_result= pd.concat([df_result,ff['Name']])
df_result = df_result.reset_index(drop=True)
df_result.columns = ['New_col']