Python Forum
Pandas, How to trigger parallel loop
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Pandas, How to trigger parallel loop
#3
import os
import pandas as pd
 
df_result =pd.DataFrame()
 
directory = os.path.join("D:\\","\PythonCodes\inputmultifiles")
for root,dirs,files in os.walk(directory):
    for file in files:
        f = os.path.join(directory,file)
 
        if f.endswith(".csv"):
           ff=pd.read_csv(f)
           tmp = ff['Name']
           print(tmp)
           df_result= pd.concat([df_result,ff['Name']])
            
df_result = df_result.reset_index(drop=True)      
df_result.columns = ['New_col']
if the file size is large, and it takes time, wait the previous iteration finish. Now I want to do like multiple threading to trigger all iterations at a time and combine the results from each iteration.
Reply


Messages In This Thread
Pandas, How to trigger parallel loop - by Mekala - Oct-23-2020, 01:05 PM
RE: How to trigger parallel loop - by Larz60+ - Oct-23-2020, 04:11 PM
RE: How to trigger parallel loop - by Mekala - Oct-28-2020, 11:40 AM
RE: How to trigger parallel loop - by Larz60+ - Oct-28-2020, 02:21 PM
RE: How to trigger parallel loop - by Mekala - Oct-29-2020, 12:58 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  export into excel, how to implement pandas into for-loop deneme2 6 2,437 Sep-01-2022, 05:44 AM
Last Post: deneme2
  parallel loop python caro 4 1,790 Jun-16-2022, 08:46 PM
Last Post: woooee
  How to trigger for loop after while quest 2 1,515 Mar-22-2022, 11:34 PM
Last Post: quest
  Increase the speed of a python loop over a pandas dataframe mcva 0 1,313 Jan-21-2022, 06:24 PM
Last Post: mcva
  For Loop Works Fine But Append For Pandas Doesn't Work knight2000 2 2,006 Dec-18-2021, 02:38 AM
Last Post: knight2000
  for loop in dataframe in pandas Paulman 7 2,746 Dec-02-2021, 12:15 AM
Last Post: bowlofred
  Pandas (Python) in powerBI loop abderrazak_rouchad77 0 1,272 Nov-08-2021, 12:54 AM
Last Post: abderrazak_rouchad77
  loop through python pandas data frame Johnse 4 2,841 Sep-02-2019, 01:45 AM
Last Post: Johnse
  How to trigger python script twice a day SriMekala 1 3,902 Aug-30-2019, 01:31 AM
Last Post: scidam
  Parallel iteration with for loop Josh_Python890 1 2,161 Jul-19-2019, 11:50 PM
Last Post: metulburr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020