Python Forum
Pandas, How to trigger parallel loop
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Pandas, How to trigger parallel loop
I have multiple CSV files in the folder, and I need to read each file and some calculation (like getting the first coulmn sum) and concate to result_df. Is there any method in python? to achieve this. Actually to ready and do some calculation, it is taking me 2mins, I need to wait longer time if there are many files.
Please show your code so far.
import os
import pandas as pd
df_result =pd.DataFrame()
directory = os.path.join("D:\\","\PythonCodes\inputmultifiles")
for root,dirs,files in os.walk(directory):
    for file in files:
        f = os.path.join(directory,file)
        if f.endswith(".csv"):
           tmp = ff['Name']
           df_result= pd.concat([df_result,ff['Name']])
df_result = df_result.reset_index(drop=True)      
df_result.columns = ['New_col']
if the file size is large, and it takes time, wait the previous iteration finish. Now I want to do like multiple threading to trigger all iterations at a time and combine the results from each iteration.
This is really a pandas question, not csv, as pandas reads in the data.
(Oct-28-2020, 02:21 PM)Larz60+ Wrote: This is really a pandas question, not csv, as pandas reads in the data.

Yes, but more generally I would put it this way: how to trigger multithreading

Possibly Related Threads…
Thread Author Replies Views Last Post
  for loop in dataframe in pandas Paulman 7 152 Yesterday, 12:15 AM
Last Post: bowlofred
  Pandas (Python) in powerBI loop abderrazak_rouchad77 0 168 Nov-08-2021, 12:54 AM
Last Post: abderrazak_rouchad77
  loop through python pandas data frame Johnse 4 1,408 Sep-02-2019, 01:45 AM
Last Post: Johnse
  How to trigger python script twice a day SriMekala 1 1,888 Aug-30-2019, 01:31 AM
Last Post: scidam
  Parallel iteration with for loop Josh_Python890 1 1,220 Jul-19-2019, 11:50 PM
Last Post: metulburr
  parallel for loop with multiprocessing dervast 0 1,142 Jul-04-2019, 03:16 PM
Last Post: dervast
  Why does my Switch trigger all Cases? IAMK 6 2,076 Apr-09-2018, 11:12 AM
Last Post: snippsat
  How to trigger a function by clicking the left mouse click? HelloBoyz 1 6,018 Jun-29-2017, 09:43 PM
Last Post: nilamo

Forum Jump:

User Panel Messages

Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020