Pandas, How to trigger parallel loop - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: Pandas, How to trigger parallel loop (/thread-30500.html) |
Pandas, How to trigger parallel loop - Mekala - Oct-23-2020 Hi, I have multiple CSV files in the folder, and I need to read each file and some calculation (like getting the first coulmn sum) and concate to result_df. Is there any method in python? to achieve this. Actually to ready and do some calculation, it is taking me 2mins, I need to wait longer time if there are many files. RE: How to trigger parallel loop - Larz60+ - Oct-23-2020 Please show your code so far. RE: How to trigger parallel loop - Mekala - Oct-28-2020 import os import pandas as pd df_result =pd.DataFrame() directory = os.path.join("D:\\","\PythonCodes\inputmultifiles") for root,dirs,files in os.walk(directory): for file in files: f = os.path.join(directory,file) if f.endswith(".csv"): ff=pd.read_csv(f) tmp = ff['Name'] print(tmp) df_result= pd.concat([df_result,ff['Name']]) df_result = df_result.reset_index(drop=True) df_result.columns = ['New_col']if the file size is large, and it takes time, wait the previous iteration finish. Now I want to do like multiple threading to trigger all iterations at a time and combine the results from each iteration. RE: How to trigger parallel loop - Larz60+ - Oct-28-2020 This is really a pandas question, not csv, as pandas reads in the data. RE: How to trigger parallel loop - Mekala - Oct-29-2020 (Oct-28-2020, 02:21 PM)Larz60+ Wrote: This is really a pandas question, not csv, as pandas reads in the data. Yes, but more generally I would put it this way: how to trigger multithreading |