Design inter-process communication using shared memory

[shang12]

I'm developing a video processing application using Python.
Capture (Producer) --> Processing (Consumer)
My design:

• Because of GIL ==> use multiprocessing (a capture process and a video processing process)
  
• Because image data is a large numpy array (~ 11 MB) ==> choose IPC: shared memory
  
• Use a single producer - single consumer ring queue to pass image data
  
• Ring queue contains n multiprocessing.RawArray + 2 semaphores.
  

My app works. Capture process have to copy data into queue and consumer process have to copy data from queue to fill into a numpy array.
But I want to find a design which doesn't use memory copy. Below is some problems:

• In producer, I use opencv-python and I can't ask opencv to capture frame into a pre-allocated buffer.
  
• In consumer, I don't feel safe if run computing on video frame which is on shared region (memory caching or ...).
  
• In method pop of my queue, I acquire an used-slot semaphore, fill slot data into numpy array, and release an empty-slot semaphore. If I want to process directly on frame data, I need to insert processing code between acquire and release semaphore. However, I can add acquire and release method in MyQueue class which other codes call them. This is still a bad design.
  

I think that only can easily get zero copy if work at the C/C++ layer. Could anybody give me some advice about this problem ? Thanks !

[shang12]

Sorry for re-post my thread. I have posted my question in some websites but nobody comments. Which site should I post my question (could pay some fee)?

I'm developing a video processing application using Python.
Capture (Producer) --> Processing (Consumer)
My design:

• Because of GIL ==> use multiprocessing (a capture process and a video processing process)
  
• Because image data is a large numpy array (~ 11 MB) ==> choose IPC: shared memory
  
• Use a single producer - single consumer ring queue to pass image data
  
• Ring queue contains n multiprocessing.RawArray + 2 semaphores.
  

My app works. Capture process have to copy data into queue and consumer process have to copy data from queue to fill into a numpy array.
But I want to find a design which doesn't use memory copy. Below is some problems:

• In producer, I use opencv-python and I can't ask opencv to capture frame into a pre-allocated buffer.
  
• In consumer, I don't feel safe if run computing on video frame which is on shared region (memory caching or ...).
  
• In method pop of my queue, I acquire an used-slot semaphore, fill slot data into numpy array, and release an empty-slot semaphore. If I want to process directly on frame data, I need to insert processing code between acquire and release semaphore. However, I can add acquire and release method in MyQueue class which other codes call them. This is still a bad design.
  

I think that only can easily get zero copy if work at the C/C++ layer. Could anybody give me some advice about this problem ? Thanks !

[micseydel]

I don't know what you mean when you say you "want to find a design which doesn't use memory copy". What's the problem? What behavior isn't what you want, and what is it instead of what you want?

[wavic]

The new version 3.8 has it.

multiprocessing can now use shared memory segments to avoid pickling costs between processes

[shang12]

The new version 3.8 has it.

multiprocessing can now use shared memory segments to avoid pickling costs between processes

Yes, Python 3.8 has a convenient SharedMemory class. In Python 3.6, I can use multiprocessing.RawArray and memory view to directly access on shared region. This is not my problem.

@micseydel:
I have a queue on shared region. Although, I can directly access to shared region with my code, I can't ask third-party libraries to fill data directly into shared region. Processes produce data into their own memory space, so I need to copy data into shared region (pickle + memory copy)