Python Forum
file tree recursion generator
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
file tree recursion generator
#1
this is a generator (usable as an iterator) that generates individual file system object paths in the tree topped by the given apex path. i plan the make a wrapper that yields tuples describing the file system object (type, size, times, owner, etc.). you can also download this source file from http://ipal.net/src/python/gen/ftrgen.py

from os import listdir,readlink
def ftrgen(path,**kwargs):
    """File tree recursion generator that yields all paths, """ \
    """optionally sorted by sortkey in key=.

Copyright © 2020, by Phil D. Howard - all other rights reserved

Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA, OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE, OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

The author may be contacted by decoding the number
11054987560151472272755686915985840251291393453694611309
(provu igi la numeron al duuma)
"""
    maxdepth = kwargs.pop('depth',2**62)
    key = kwargs.pop('key',None)
    if kwargs:
        raise TypeError('ftrgen: '+__name__+'() '+\
            'got an unexpected keyword argument '+\
            ','.join([repr(x) for x in kwargs.keys()]))
    if not isinstance(path,str):
        raise TypeError('ftrgen: '+__name__+'() '+\
            'top/apex path is not a string (only str)')
    work = [[path]]
    deep = 0
    while work[0]:
        if work[-1]:
            path = '/'.join([x[0] for x in work])
            yield path
            try:
                readlink(path)
                names = []
            except OSError:
                try:
                    names = sorted(listdir(path),key=key)
                except OSError:
                    names = []
                deep += 1
                if deep > maxdepth:
                    names = []
            if names:
                work.append(names)
            else:
                work[-1].pop(0)
            continue
        else:
            work[-1:] = []
            work[-1].pop(0)
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#2
How does this compare to os.walk()?
Reply
#3
it gets the files in the correct order. you can also provide your own sort key that will be applied to the list of names in each directory.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#4
Hey,

Do you mind explaining this code?

Usually I use os.walk, but I saw you are generating something like stuck? that makes instance?
Would love to see explanation for this so We can study from your skills.
Reply
#5
what do you want to know? it is a generator.

what do you mean by "generating something like stuck"?

skills? practice makes skills.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply
#6
Why do you use your own complicated implementation (which could not handle WindowPaths) instead of using os.walk?


os.walk example as generator
import os
from pathlib import Path


def walk(path, maxdepth=None, sort_key=None, with_files=True, with_dirs=True):
    for depth, (root, dirs, files) in enumerate(os.walk(path)):
        if maxdepth is not None and depth > maxdepth:
            return
        if with_dirs:
            for directory in sorted(dirs, key=sort_key):
                yield Path(root, directory)
        if with_files:
            for file in sorted(files, key=sort_key):
                yield Path(root, file)
If you need compatibility to Python 2.7 (hopefully not), omit pathlib and use os.path.join.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#7
DeaD_EyE Wrote:If you need compatibility to Python 2.7 (hopefully not), omit pathlib and use os.path.join.
Pathlib can be installed from pypi for python 2.7 code.
Reply
#8
i do not need compatibility with 2.7 other than making sure that if my code fails under 3.5 or lower, it will not delete any user files.

i tried os.walk(). i remember that the order it walked the directory was not the same order i get with other tools when sorted. i do not want to have to load the whole tree to sort it. i want it to be walked in correctly sorted order even if the tree is so big it cannot fit in memory.
Tradition is peer pressure from dead people

What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  file tree walk depth first generator Skaperen 0 3,004 Aug-12-2019, 07:02 AM
Last Post: Skaperen

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020