Is this a multiprocessing bug in Python - or am I doing something wrong?

[haimat]

Hello everyone,

I have a huge problem with one of my Python server programs, and since weeks we cannot figure out how to solve it. I really hope you can help me. So this Python program is a 24/7 server application, which controls a range of remote hardware devices, connected via the local network. We need to keep each of these remote devices fully separated from each other, so in our Python program we spawn a background process for each such remote device. Therefore when the application starts multiple Python processes are being spawned. This works fine.

Now sometimes we need to shut down such a process from the application and restart it again a bit later (while the application continues to run). The code to spawn a process is always the same (both when starting the application as well as later during runtime):

# Spawn context for multiprocessing
_mp_spawn_ctxt = multiprocessing.get_context("spawn")
_mp_spawn_ctxt_pipe = _mp_spawn_ctxt.Pipe

def spawn_proc(width, height, bpp):
    # Create shared memory
    mem_size = width * height * bpp
    shared_mem = shared_memory.SharedMemory(create=True, size=mem_size)
    image = np.ndarray((height, width, bpp), dtype=np.uint8, buffer=shared_mem.buf)
    parent_pipe, child_pipe = _mp_spawn_ctxt_pipe()
    time.sleep(1.0)

    # Spawn new process
    proc = _CameraProcess(shared_mem, child_pipe)
    proc.start()

class _CameraProcess(_mp_spawn_ctxt.Process):
    def __init__(self):
        super().__init__(name=f"CameraProcess", daemon=True)
        ...

Our issue now is that this logic works well when we start the application and for some time afterwards. However, after some days or weeks of runtime, when we then call this logic listed above to spawn a new process, the following exception is raised:

Error:
Jun 23 14:23:46 srv gunicorn[2717986]: Traceback (most recent call last):
Jun 23 14:23:46 srv gunicorn[2717986]:   File "<string>", line 1, in <module>
Jun 23 14:23:46 srv gunicorn[2717986]:   File "/usr/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
Jun 23 14:23:46 srv gunicorn[2717986]:     exitcode = _main(fd, parent_sentinel)
Jun 23 14:23:46 srv gunicorn[2717986]:   File "/usr/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
Jun 23 14:23:46 srv gunicorn[2717986]:     self = reduction.pickle.load(from_parent)
Jun 23 14:23:46 srv gunicorn[2717986]:   File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 110, in __setstate__
Jun 23 14:23:46 srv gunicorn[2717986]:     self._semlock = _multiprocessing.SemLock._rebuild(*state)
Jun 23 14:23:46 srv gunicorn[2717986]: FileNotFoundError: [Errno 2] No such file or directory

Interestingly, when we then shutdown and restart the whole application, then again the spawning of the processes with this logic above works fine again. So this issue really just occurs when the application has run for some days or weeks already. I thought that maybe it might be a bug in the multiprocessing module, so we installed the latest Python version 3.8.17 locally and ran the program with that, but without luck, we still hit the same error. By the way, we are running this application on an Ubuntu 20.04 system. Ohh and when this happens, both enough RAM and disk space is free, we double-checked this.

We tried several things already, but nothing helped. We always hit this issue after some time and cannot start the processes any more, instead we have to restart the whole application for a (temporary) fix.

Do you have any ideas how we could fix this?
Thank you very much in advance!

Larz60+ write Jul-26-2023, 10:00 AM:
Please post all code, output and errors (it it's entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.
This has been fixed. Please use BBCode tags on future posts.

[absalom1]

[b]Larz60+ write Jul-26-2023, 07:00 PM:[/b]
Please post all code, output and errors (it it's entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.
This has been fixed. Please use BBCode tags on future posts.

Could you let me know which commit is the fix for this issue?