short version of os.listdir()

[snippsat]

i would not use scandir() (i have never used it for any purpose).

Maybe not directly,but all directory iteration in Python inherit from this after PEP 471 – os.scandir() function – a better and faster directory iterator
Starting from Python 3.5, the implementation of os.listdir was modified to use os.scandir.
The same with eg Path.iterdir(), Path.glob, Path.rglob they all use the same C code underneath C speedups for scandir module.

---

Just to make clear with me code example it dos not matter that a directory has eg 70 million files as you mention.
Can read it in chunk of whatever size,the point is only read files specified into memory an not all.
Here i read files from eg 12000, 12005 in folder that has 15k files,eg with os.listdir it will read all 15k files.

from pathlib import Path
from itertools import islice

def generate_paths(directory):
    for path in Path(directory).rglob('*'):
        if path.is_file():
            yield path

if __name__ == '__main__':
    dest = r'G:\div_code'
    # Slice into the generator to get files in range specified
    selected_files = islice(generate_paths(dest), 12000, 12005)
    for path in selected_files:
        print(path)

Output:
G:\div_code\answer\ibm_env\Lib\site-packages\pip\_vendor\rich\bar.py
G:\div_code\answer\ibm_env\Lib\site-packages\pip\_vendor\rich\box.py
G:\div_code\answer\ibm_env\Lib\site-packages\pip\_vendor\rich\cells.py
G:\div_code\answer\ibm_env\Lib\site-packages\pip\_vendor\rich\color.py
G:\div_code\answer\ibm_env\Lib\site-packages\pip\_vendor\rich\color_triplet.py

[Gribouillis]

Can read it in chunk of whatever size,the point is only read files specified into memory an not all.
Here i read files from eg 12000, 12005 in folder that has 15k files,eg with os.listdir it will read all 15k files.

I don't agree. In your code example to access the slice 12000 to 12005, the code will call is_file() for at least 12005 paths, which means 12005 system calls to stat.

Besides, Python's scandir is not the same function as C's scandir. The code you linked uses the opendir-readdir-closedir pattern. See Accessing Directories in the GNU C library for example.

I wonder if opening a directory as a file with open() and use read() in C as @Skaperen does is documented somewhere. I doubt it is a safe/portable way to read directories. Also if this work, it may be possible to call the C library functions directly through the ctypes module.

I don't see the point in having directories with 7e7 files. You can store that many files in 4 levels of directories having 100 children each.

[Skaperen]

I don't see the point in having directories with 7e7 files.

no real point at all. it was an accident (missed typing a '/' where i should have) i worked to correct (got it done, now). the big issue is that when things read the next chunk of file names, something went back to the start of the directory each time, so each chunk was slower and iterating over chunks just got slower and slower while the disk stayed busy. i finally managed to make a full list of files (in C) and broke those into chunks to be copied to a new drive (in bash). took about 5 days, but i could watch the progress. now i have a few thousand subdirectories, with a few hundred files, each, about 6TB worth (and another 8 TB hard drive destined to become another backup space).