Intermittent errors using Queue()

[Fredrik]

Hello

I don't know if this is the correct forum for this but General Coding Help sounded right.
I'm writing a multiprocess program and I communicate using queues.

I noticed that I was getting errors some of the time... intermittent... no pattern to it.

The error only occurs when I use exit() to terminate the program. I realize an exit() at the end of the program doesn't make any sense, but this is just minimal code to reproduce the error.

I can't find a way to shut down the manager server and the queues before exiting as neither the manager object or the proxied queue object seem to have any relevant functions.

The minimal code I use to reproduce it is:

import multiprocessing

if __name__ == '__main__':
    multiprocessing.freeze_support()

    ''' Create a Manager object to handle IPC queues '''
    manager=multiprocessing.Manager()
    work_q=manager.Queue()

    exit(0)

Most times this works and generates no error, but sometimes it generates the following error:

Error:
"C:\Program Files\Python35\python.exe" "S:/Code/Python/Video Recode/Video Recode/VideoRecode.py"
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x000002DFEAFCA2F0>
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x000002DFEAFCA2F0>
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x000002DFEAFCA2F0>
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable

Process finished with exit code 0

[nilamo]

I can't reproduce the error.  You mentioned it being intermittent, so I tried about a dozen times, each had a clean exit.

Does the error persist if you .join() the work_q before calling exit()?

Do you get similar errors if you use queue.Queue() instead of the proxied version from Manager?  What if you use a different syncronization mechanism from Manager instead of the Queue?  (just trying to find out where, exactly, the issue is)

[Fredrik]

Hello

Thanks for trying to help.

Reproducing:
It might be the platform or the Python version ?
I run Python 3.5 on Win 10 64-bit.
IDE is PyCharm (latest version)

join():
No difference if I call join() on it before exit(0). Still same error.
Calling shutdown() on the manager doesn't help either.

queue.Queue():
I thought I had to use the proxied queues to communicate between multiple processes safely.
I haven't be able to find any info on why there are three implementations of queues (multiprocessing.Queue, queue.Queue and Manager.Queue()).
I figured the use of proxied queue was best practice since it is usable over networks and between different interpreters on the same machine.

Cheers

---

queue.Queue():
queue.Queue() is not a drop-in replacement, I get a pickle error.
multiprocessing.Queue() is a drop-in replacement tho and can handle the message object I'm sending. It also doesn't give the weakref.py error message.

This solution works for this project but still doesn't help with future projects where I might need to rely on proxying queues over networks.