Python Forum
q re glob.iglob iterator and close
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
q re glob.iglob iterator and close
#1
I'm struggling to find any documentation that tells me how I should properly "short circuit" a glob.iglob iterator.

I have some code that needs to check that a directory has at least one file matching a given pattern. It seemed reasonable to handle this via a call to glob.iglob that then tested for a value:

itr = glob.iglob(os.path.join(publickeys, "*.key"))
if next(itr, None) is None:
    raise ValueError(f"publickeys {publickeys} has no *.key files")
I know that under the hood what glob.iglob is doing is yield'ing on a call to a 'private' function that ultimately calls os.scandir:

  1. https://github.com/python/cpython/blob/m...lob.py#L69
  2. https://github.com/python/cpython/blob/m...lob.py#L94
  3. https://github.com/python/cpython/blob/m...ob.py#L163
  4. https://github.com/python/cpython/blob/m...ob.py#L128
  5. https://github.com/python/cpython/blob/m...ob.py#L146

The os.scandir documentation (https://docs.python.org/3/library/os.html#os.scandir) does say it supports close to free acquired resources, and you can see in step 5 above that os.scandir is called using a with clause, which will close the os.scandir file descriptor once the entries have been looped over. But, of course, in my code I'm not looping over everything. I'm just checking the first entry...

I thought the following would be reasonable:

try:
    if next(itr, None) is None:
        raise ValueError(f"publickeys {publickeys} has no *.key files")
finally:
    itr.close()
and some testing shows that it does what I expect, the close'd iterator does properly reflect its closed state if I try and call next() on the iterator after I've closed it.

But a fellow developer is calling this try/finally into question and asking whether or not there is any risk if relying on the iterator close() method is a mistake, that it relies on "knowing that os.scandir is being called."

TLDR: Do I need to close a glob.iglob iterator that I'm not draining, or should I just let python's gc cycle 'handle it' whenever it decides it's safe to close the iterator?
Reply
#2
As long as an opaque object such as itr is in scope, there is a risk that it keeps some resources alive. The simplest opaque way to clear these resources is that the object disappears from scope. It seems to me that the best way is to encapsulate this in a function
if no_glob(os.path.join(publickeys, "*.key")):
    raise ValueError(f"publickeys {publickeys} has no *.key files")

def no_glob(pathname, recursive=False):
    return next(glob.iglob(pathname, recursive), None) is None
Reply
#3
itertools module documentation has recipes chapter and there is function consume for consuming iterator:

def consume(iterator, n=None):
    "Advance the iterator n-steps ahead. If n is None, consume entirely."
    # Use functions that consume iterators at C speed.
    if n is None:
        # feed the entire iterator into a zero-length deque
        collections.deque(iterator, maxlen=0)
    else:
        # advance to the empty slice starting at position n
        next(islice(iterator, n, n), None)
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  prime numbers with iterator and generator cametan_001 8 1,771 Dec-17-2022, 02:41 PM
Last Post: cametan_001
  resetting an iterator to full Skaperen 7 6,808 Feb-20-2022, 11:11 PM
Last Post: Skaperen
  |SOLVED] Glob JPGs, read EXIF, update file timestamp? Winfried 5 2,410 Oct-21-2021, 03:29 AM
Last Post: buran
  popping an iterator Skaperen 11 3,600 Oct-03-2021, 05:08 PM
Last Post: Skaperen
  [SOLVED] Input parameter: Single file or glob? Winfried 0 1,539 Sep-10-2021, 11:54 AM
Last Post: Winfried
  Problem with an iterator grimm1111 9 4,217 Feb-06-2021, 09:22 PM
Last Post: grimm1111
  Multi-class iterator Pedroski55 2 2,335 Jan-02-2021, 12:29 AM
Last Post: Pedroski55
  Listing files with glob. MathCommander 9 4,812 Oct-26-2020, 02:04 AM
Last Post: MathCommander
  is a str object a valid iterator? Skaperen 6 5,539 Jan-27-2020, 08:44 PM
Last Post: Skaperen
  discard one from an iterator Skaperen 1 1,957 Dec-29-2019, 11:02 PM
Last Post: ichabod801

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020