Python Forum
Looping to read data in database
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Looping to read data in database
#1
I am hoping someone could help me read in a master directory file which tells me what .csv to read through and save in memory and then move to next file to read until eof()

first .csv has names
CL
NG
GC

those names tell me where in my c:\Data to get the file called CL etc etc.

I have imported
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import scipy as sp

pd.read_csv('c:/QlData/Continuous/Daily/CL.csv')
so this i guess is reading in the file named CL but once i review the huge amount of data in CL I want to then use a for loop to go to next line in master directory and read in NG and so on.

I just cannot find anything online to help me do this besides static reading process like I have above?
Any help would be appreciated
Reply
#2
(Sep-21-2020, 01:32 AM)CEC68 Wrote: besides static reading process like I have above?

Do you mean hard coded file names to be processed?
If I understood you correctly, you want to find all *.csv files in a specific folder/subfolders and load them one-by-one (and do something with data). The following code snippet could help you to do this:
import os
for root, dirs, files in os.walk("C:/Data"):
    for name in files:
        if name.endswith('csv'):
            data = pd.read_csv(os.path.join(root, name))
            # do something with data...
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Help with to check an Input list data with a data read from an external source sacharyya 3 318 Mar-09-2024, 12:33 PM
Last Post: Pedroski55
  How to detect abnormal data in big database python vanphuht91 5 1,064 Jun-27-2023, 11:22 PM
Last Post: Skaperen
  Database that can compress a column, or all data, automatically? Calab 3 1,120 May-22-2023, 03:25 AM
Last Post: Calab
  Pymodbus read and save to database stewietopg 3 1,767 Mar-02-2023, 09:32 AM
Last Post: stewietopg
  Correctly read a malformed CSV file data klllmmm 2 1,814 Jan-25-2023, 04:12 PM
Last Post: klllmmm
  Read nested data from JSON - Getting an error marlonbown 5 1,309 Nov-23-2022, 03:51 PM
Last Post: snippsat
  Basic SQL query using Py: Inserting or querying sqlite3 database not returning data marlonbown 3 1,304 Nov-08-2022, 07:16 PM
Last Post: marlonbown
  Read JSON via API and write to SQL database TecInfo 5 2,108 Aug-09-2022, 04:44 PM
Last Post: TecInfo
  Read data via bluetooth frohr 9 3,150 Jul-10-2022, 09:51 AM
Last Post: frohr
  Write and read back data Aggie64 6 1,812 Apr-18-2022, 03:23 PM
Last Post: bowlofred

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020