Python Forum
Read all csv files, and store the last line from each folder
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Read all csv files, and store the last line from each folder
#1
Hi,
I want to read all csv files and read each file and capture last line form each file.
I use the below code, but I struck.

import os
from glob import glob
PATH = "D:\data\project\*\*\InputFiles"
EXT = "*.csv"
all_csv_files = [file
                 for path, subdir, files in os.walk(PATH)
                 for file in glob(os.path.join(path, EXT))]
print(all_csv_files)
Reply
#2
with pathlib written here without testing, but I think it's right or close to it:
from pathlib import Path

p = Path("D:\data\project\*\*\InputFiles")
csv_files = [csvfile for csvfile in p.iterdir() if csvfile.is_file() and csvfile.suffix == '.csv']
for filename in csv_files:
    print(filename)
Reply
#3
I use below code. Here, the there will be 4 levels (subfolders) so I use

Exammple: Rootfolder/project/cataegory after this I do not know folder names, but there will be 4 more subfolders and in the last folder my file exist.

My file name is: .Dailycollection.log

If file exist after this : Rootfolder/project/cataegory, then I want to capture full path of the file.
p = Path("D:\Mekala_Backupdata\*\*\*\*\")
csv_files = [csvfile for csvfile in p.iterdir() if csvfile.is_file() and csvfile.suffix == '*.Dailycollection.log']
for filename in csv_files:
print(filename)
Reply
#4
I can't run the following code as I don't use windows, but this should list all csv files in all directories
within and below a root directory of D:\data\project\*\*\InputFiles (you will have to supply actual values for '*')
from pathlib import Path


def walk_dir(starting_dir):
    flist = []
    for path in Path(starting_dir).iterdir():
        if path.is_file():
            if path.suffix == '.csv':
                print(path)
                flist.append(path)
        elif path.is_dir():
            walk_dir(path)

    for file in flist:
        print(file)


if __name__ == '__main__':
    start_path = 'D:\data\project\*\*\InputFiles'
    import os
    os.chdir(os.path.abspath(os.path.dirname(__file__)))
    srcdir = Path('.')
    savefile = srcdir / 'allcsvfiles.txt'
    walk_dir(start_path, savefile)
Reply
#5
Actually the last layer (sub)directory does not know (not fixed). But the file will be saved into this last folder. Hence I want to use the wild card.
The file is available at :D:\Mekala_Backupdata\*\*\*\*\ --> here the file.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Compare folder A and subfolder B and display files that are in folder A but not in su Melcu54 3 466 Jan-05-2024, 05:16 PM
Last Post: Pedroski55
  Rename files in a folder named using windows explorer hitoxman 3 693 Aug-02-2023, 04:08 PM
Last Post: deanhystad
  Rename all files in a folder hitoxman 9 1,386 Jun-30-2023, 12:19 AM
Last Post: Pedroski55
  Python Serial: How to read the complete line to insert to MySQL? sylar 1 781 Mar-21-2023, 10:06 PM
Last Post: deanhystad
  How to read in mulitple files efficiently garynewport 3 845 Jan-27-2023, 10:44 AM
Last Post: DeaD_EyE
  How to loop through all excel files and sheets in folder jadelola 1 4,331 Dec-01-2022, 06:12 PM
Last Post: deanhystad
  python gzip all files from a folder mg24 3 3,812 Oct-28-2022, 03:59 PM
Last Post: mg24
  delete all files and subdirectory from a main folder mg24 7 1,530 Oct-28-2022, 07:55 AM
Last Post: ibreeden
  Read directory listing of files and parse out the highest number? cubangt 5 2,252 Sep-28-2022, 10:15 PM
Last Post: Larz60+
  Merge all json files in folder after filtering deneme2 10 2,253 Sep-18-2022, 10:32 AM
Last Post: deneme2

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020