Using .hdf5 files only once they are finished writing

[pyhill00]

Thank you for the tip. That helped and now it somewhat works. Now the problem is when one file is emitted i.e. goes to the else part, afterward the code gets stuck in OSError loop and goes through the if else loop until the else's break. I tried a few things but nothing worked. It seems the problem is that after emitting the signal, there is no event at self._q.get() so it goes into OSerror until the loop is broken. If during this loop, there is a new file at self._q.get() the file can be emitted once the if/else loop is finished and the process starts again. Tersely: the problem now is the wait time during files. Do you have any idea how I can fix this?

        
while True:
            event = self._q.get()
            max_retry_count = 3500  # for test purposes now but want to set an upper bound on verifying a file is finished.
            retry_interval_seconds = .01  # every hundreth it will try the file to see if it finished writing
            retry_count = 0            
            while True:
                try:
                    file = h5py.File(event.src_path, "r")
                    file.close()
                except OSError:
                    if retry_count < max_retry_count:
                        retry_count += 1
                        print(f"h5 file <{event.src_path}> is locked, retrying {retry_count}/{max_retry_count}")
                        time.sleep(retry_interval_seconds)
                    else:
                        print(f"h5 file <{event.src_path}> reached max retry count, skipping")
                        break  # <--- looks useful here
                except Exception as err:
                    print(f"Got unexpected Error <{type(err).__name__}> while opening <{event.src_path}> ")
                    traceback.print_exc()
                else:
                    self.new_file.emit(event.src_path, os.path.basename(event.src_path))
                    break