Python Forum
Read directory listing of files and parse out the highest number?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Read directory listing of files and parse out the highest number?
#1
So i have a directory where im placing files after my script runs, each file currently has a unique number in the file name.

Example: igMess_'+ fileNumber+'_.csv'

Currently i have the script hardcoded with the fileNumber variable set manually before running the script, its ok now because the directory only has about 10 files, so visually easy to see what the next number would be...

What i would like to do is have a way to look in the directory for ONLY files that follow that naming convention and then determine the last/highest number so that my variable can be dynamic and use the next seq. number.

Ive seen examples of using max(), but that doesnt seem to do what i thought it would, or at least not using it correctly.

If my directory has the following files:

igMess_1_.csv
igMess_2_.csv
igMess_3_.csv
igMess_4_.csv
igMess_5_.csv
igMess_6_.csv
temp1.txt
temp2.txt
mydata.xlsx

i would like to ONLY consider the files that start with "igMess_" and end with "_.csv" and in the above example, the number i would like returned back would be "7" since that is the next seq number, so when my script runs, it names the new file as "igMess_7_.csv"

would i still use max() in some way, or do i need to using something else?
Reply
#2
new detail

So it would def need to look at ONLY files that start with igMess and contain 2 underscores ( _ )
seems we have older files that are formatted like this:
igMess_1.csv
igMess_2.csv

The newer format is:
igMess_1_.csv
igMess_2_.csv

So only care to look at these and return the next highest number.

I have this sample script im playing around with, but not sure how to only consider the files with 2 of those underscores.

# iterate over files in
# that directory
for filename in os.listdir(directory):
    f = os.path.join(directory, filename)
    d = filename
    # checking if it is a file
    if d.startswith("igMess_"):
        print(d)
Reply
#3
this code has gotten me closer, but still not sure how to get the max value.

# iterate over files in
# that directory
for filename in os.listdir(directory):
    f = os.path.join(directory, filename)
    d = filename
    # checking if it is a file
    if d.startswith("igMess_"):
        if d.count('_'):
            if d.count('_') == 2:
                #print(d)
                temp = re.findall(r'\d+', d)
                res = list(map(int, temp))
    print(str(res))
this returns the following in my console.
[1]
[2]
[3]
[]
So im assuming that its not techanically a "list" where i can use max()
Reply
#4
Just an idea: for every filename which ends with _.csv strip the specific characters, convert to int and find max value:

filenames = ["igMess_2_.csv", "igMess_3_.csv", "igMess_1_.csv", "igMess_5.csv"]

max_value = max(int(filename.strip('igMess_.csv')) for filename in filenames if
       filename.endswith("_.csv"))

# max_value would in this example be 3
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#5
well thats def simple and cleaner than what i was trying to do above.

This logic works against my test directory

filenames = os.listdir(directory)

max_value = max(int(filename.strip('igMess_.csv')) for filename in filenames if
       filename.endswith("_.csv"))

print(str(max_value))
Reply
#6
Another approach:
test conditions:
directory in same path as script, named 'filedir' (you can change as needed)
containing the following files:
Output:
igMess_9.csv igMess_10.csv igMess_11_.csv igMess_12_.csv igMess_13_.csv
from pathlib import Path
import os


os.chdir(os.path.abspath(os.path.dirname(__file__)))

def check_for_highest(StartsWith, EndsWith, Directory):
    dir = Path(Directory)
    filelist = [filename for filename in dir.iterdir() \
        if filename.is_file() and str(filename.stem).startswith(StartsWith) \
        and str(filename.stem).endswith(EndsWith)]
    largest_name = sorted(filelist)[-1]
    return filelist, largest_name

def testit():
    filelist, largest = check_for_highest('igMess_', '_', 'filedir')

    for file in filelist:
        print(file)

    next_available = str(int(largest.stem[7:-1]) + 1)
    print(f"next available number = {next_available}")

testit()
results:
Output:
filedir/igMess_11_.csv filedir/igMess_12_.csv filedir/igMess_13_.csv next available number = 14
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  uploading files from a ubuntu local directory to Minio storage container dchilambo 0 398 Dec-22-2023, 07:17 AM
Last Post: dchilambo
  change directory of save of python files akbarza 3 814 Jul-23-2023, 08:30 AM
Last Post: Gribouillis
  Using pyinstaller with .ui GUI files - No such file or directory error diver999 3 3,075 Jun-27-2023, 01:17 PM
Last Post: diver999
  parse/read from file seperated by dots giovanne 5 1,043 Jun-26-2023, 12:26 PM
Last Post: DeaD_EyE
  Monitoring a Directory for new mkv and mp4 Files lastyle 3 1,568 May-07-2023, 12:33 PM
Last Post: deanhystad
  Listing directories (as a text file) kiwi99 1 802 Feb-17-2023, 12:58 PM
Last Post: Larz60+
  How to read in mulitple files efficiently garynewport 3 845 Jan-27-2023, 10:44 AM
Last Post: DeaD_EyE
  python read iperf log and parse throughput jacklee26 4 2,648 Aug-27-2022, 07:04 AM
Last Post: Yoriz
  How to read python shortcut target profile directory of Chrome Ink file sunny9495 1 1,617 Apr-12-2022, 06:12 PM
Last Post: sunny9495
  How to save files in a separate directory Scordomaniac 3 1,755 Mar-16-2022, 10:17 AM
Last Post: Gribouillis

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020