pipeline.py

[Skaperen]

I just read what deadlock means. Asyncio module?

probably not.  the deadlock in this case is around process handling and how it affects i/o.  when a process does an exit() it does not cease to exist, yet.  it only ceases to exist when its parent process gets the exit status information from the process data structure (in the process table) via the parent doing a wait() syscall.  the process is in what is called zombie state (or defunct state) for this interim.  this is necessary so that the correct process information can be retrieved by PID (essential if the parent has 2 or more children) at a later (less than a millisecond in most cases) time.  any file descriptors that were open at exit() remain open until the process is reaped, leaving the zombie state.  for the pipeline module, the problem is this.  the pipe's write side is left open.  thus the reader does not get an EOF, yet.  the multiprocessing module does not do the needed os.wait() call until its process.join() method is called.  because of how the pipeline module structured its own methods (my design to make things easy without thinking far enough ahead to anticipate this) the caller won't know to call the pipeline.close() method (which calls process.join() which calls os.reap()) until it gets an EOF from the pipe it is reading.  in short, it won't do the action that is needed to get an EOF until it gets an EOF.

what asyncio can help solve is the ability to avoid being stuck waiting for data in a read() call, when that same process is going to be the cause of that data to written.  it can get stuck if the pipe has no data in it at the instant of the read() call.

[Skaperen]

i have now updated the code to handle EINTR in the rare cases it happens in Python < 3.5 ... see PEP 475.