short version of os.listdir()

[Skaperen]

is there a way to shorten os.listdir() such as to have it only read several (like maybe 32 to 256) names at a time? i need to scan through a massively huge directory and it is have trouble with it being so big. the directory has well over 70 million files.

[deanhystad]

Don’t use os.listdir. Use pathlib.iterdir

[Pedroski55]

Fun with generators!

from pathlib import Path
import sys

mydir = Path('/home/pedro')
filelist = (filename for filename in mydir.rglob("*") if filename.is_file())
type(filelist) # generator
sys.getsizeof(filelist) # returns 104
total = sum(1 for f in filelist) # takes a couple of seconds then returns 193820

# show some of the files
filelist = (filename for filename in mydir.rglob("*") if filename.is_file())
for f in range(32):
    print(next(filelist))

Apparently, in the latest Python, pathlib has .walk() just like os (I don't have the latest Python!)

import pathlib    
path = pathlib.Path(r"E:\folder")
for root, dirs, files in path.walk():
    print("Root: ")
    print(root)
    print("Dirs: ")
    print(dirs)
    print("Files: ")
    print(files)
    print("")

What do you want to do with 70 million files??

[Gribouillis]

In addition to pathlib.iterdir(), you can use more_itertools.chunked()

[snippsat]

Can also use itertools.islice to slice into a generator.
So here load only files eg 5-10 or 32-256 into memory.

from pathlib import Path
from itertools import islice

def generate_paths(directory):
    for path in Path(directory).rglob('*'):
        if path.is_file():
            yield path

if __name__ == '__main__':
    dest = r'C:\Test'
    # Slice into the generator to get files in range specified
    selected_files = islice(generate_paths(dest), 5, 11)
    for path in selected_files:
        print(path)

[Skaperen]

What do you want to do with 70 million files??

reduce it to about 700 files or maybe even fewer.

[Skaperen]

Don’t use os.listdir. Use pathlib.iterdir

it gives me only ONE (1) at a time. i guess that's what "iter" implies. this is going to take "forever". is there a way to get like 256 at a time, or at least do one input from the directory per block that the names are stored on?

[Skaperen]

the desire to get 32 to 256 at a time is not so i can have a loop do one at a time. it's so i can get all the names from a directory block with a single physical read operation. i created a test directory and was able to put 243 files into a single block of a directory.

re-phrased: my goal is to read the entire directory as fast as possible to acquire the list of names and write that list into a file. then i will run things to filter that huge list down to the few files i actually need, based only on the particular name fitting a collection of patterns. i don't need to open any of these files, yet.

hmmm, how to open a directory as a file in Python? trivial in C. never done this in Python. maybe os.open() and os.read().

[Gribouillis]

how to open a directory as a file in Python? trivial in C.

How do you do that in C? Isn't it a call to opendir() and a loop of calls to readdir() ?

Hm, ChatGpt told me one can read a directory in C with scandir(). In your case however it would use malloc to allocate 70 millions character strings. I don't see how you could get only chunks of 256 entries for exampl.

[Skaperen]

how to open a directory as a file in Python? trivial in C.

How do you do that in C? Isn't it a call to opendir() and a loop of calls to readdir() ?

to open a directory as a file you simply do the steps you would do if it is a regular file, open() and read(). doing "the file way" in Python, on a directory, raises IsADirectoryError.

Hm, ChatGpt told me one can read a directory in C with scandir(). In your case however it would use malloc to allocate 70 millions character strings. I don't see how you could get only chunks of 256 entries for exampl.

i would not use scandir() (i have never used it for any purpose). if i were doing this in C, i would use read(), if i opened it with open(). i think readdir() buffers whatever it gets when it does read() instead of the whole directory all at once, which could make it usable (instead of duplicating code to slice up a directory). i need to try more things with Python, first, before i drop back to C to do this. i have zero experience mixing C and Python.