Looping to read data in database - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: Looping to read data in database (/thread-29810.html) |
Looping to read data in database - CEC68 - Sep-21-2020 I am hoping someone could help me read in a master directory file which tells me what .csv to read through and save in memory and then move to next file to read until eof() first .csv has names CL NG GC those names tell me where in my c:\Data to get the file called CL etc etc. I have imported import pandas as pd import numpy as np import matplotlib.pyplot as plt import scipy as sp pd.read_csv('c:/QlData/Continuous/Daily/CL.csv')so this i guess is reading in the file named CL but once i review the huge amount of data in CL I want to then use a for loop to go to next line in master directory and read in NG and so on. I just cannot find anything online to help me do this besides static reading process like I have above? Any help would be appreciated RE: Looping to read data in database - scidam - Sep-24-2020 (Sep-21-2020, 01:32 AM)CEC68 Wrote: besides static reading process like I have above? Do you mean hard coded file names to be processed? If I understood you correctly, you want to find all *.csv files in a specific folder/subfolders and load them one-by-one (and do something with data). The following code snippet could help you to do this: import os for root, dirs, files in os.walk("C:/Data"): for name in files: if name.endswith('csv'): data = pd.read_csv(os.path.join(root, name)) # do something with data... |