Python Forum
How to read csv files parallay
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to read csv files parallay
#1
Hi,
I have multiple CSV files in the folder, I want to read them parallelly. Each file takes 1min to read and do some calculations. I want to read CSV files simultaneously (parallelly) and concate the results (for example First column mean from each file and put into col_mean_df)
Reply
#2
Show your code so far, and where you think there is a bottleneck.
Reply
#3
I use below code.

import os
import pandas as pd

df_result =pd.DataFrame()

directory = os.path.join("D:\\","\PythonCodes\inputmultifiles")
for root,dirs,files in os.walk(directory):
    for file in files:
        f = os.path.join(directory,file)

        if f.endswith(".csv"):
           ff=pd.read_csv(f)
           tmp = ff['Name']
           print(tmp)
           df_result= pd.concat([df_result,ff['Name']])
           
df_result = df_result.reset_index(drop=True)      
df_result.columns = ['New_col']
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to read in mulitple files efficiently garynewport 3 844 Jan-27-2023, 10:44 AM
Last Post: DeaD_EyE
  Read directory listing of files and parse out the highest number? cubangt 5 2,249 Sep-28-2022, 10:15 PM
Last Post: Larz60+
  Python code to read second line from CSV files and create a master CSV file sh1704 1 2,352 Feb-13-2022, 07:13 PM
Last Post: menator01
  Open and read multiple text files and match words kozaizsvemira 3 6,666 Jul-07-2021, 11:27 AM
Last Post: Larz60+
  code to read files in folders and transfer the file name, type, date created to excel Divya577 0 1,834 Dec-06-2020, 04:14 PM
Last Post: Divya577
  Read KML files, edit items, and rewrite files? Winfried 4 4,694 Aug-21-2020, 03:55 PM
Last Post: Winfried
  Python: Automated Script to Read Multiple Files in Respective Matrices Robotguy 7 4,119 Jul-03-2020, 01:34 AM
Last Post: bowlofred
  Read Multiples Text Files get specific lines based criteria zinho 5 3,050 May-19-2020, 12:30 PM
Last Post: zinho
  How to read multiple csv files and merge data rajeshE 0 1,917 Mar-28-2020, 04:01 PM
Last Post: rajeshE
  Read all csv files, and store the last line from each folder SriRajesh 4 5,453 Jan-17-2020, 12:13 AM
Last Post: SriRajesh

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020