How to create waiting process?

[samuelbachorik]

Hi lets think i need to process image and i have function for it. How to create from this function process that will be always running and waiting for images from main script ?

My main script is busy with other things and i dont want to start always a new process. I just want to sent images to already running process and just get result from it. Is it possible?
I can not find any working example on internet.

[bowlofred]

It's possible, but I'm not sure I understand what problem you're trying to solve. In many cases, starting up a new process is much less complex (and is just as performant) as handing off to an already running daemon.

What problem are you trying to avoid by not just firing off a new process?

[samuelbachorik]

It's possible, but I'm not sure I understand what problem you're trying to solve. In many cases, starting up a new process is much less complex (and is just as performant) as handing off to an already running daemon.

What problem are you trying to avoid by not just firing off a new process?

Hi, Thank you for reply, exactly i have two Neural networks, and frames from webcam capture. I need to process these frames with both neural networks. I dont have time to do it serially. Let me show you example.

I need process that will be waiting for image.

this is what i have right now (just example),

for image in capture:
    #This take some time
    a = network1(image)
    #This also take some time
    b = network2(image)
    

And this is what i want.

By some way I want to have always running process that will wait for image and process it.
I need to process a and b concurently. But it is stupid idea to start new process every loop....

#"THIS IS MY ALWAYS RUNING PROCESS" 
def process(image):
    a = network1(image)
    
    return a 


for image in capture:
    
    #Now just by some way sent image to process
    a = sent image to process
    
    # And b will work on main process normally
    b = network2(image)
    

[ndc85430]

If you want to start a process and send it a request to do something, there are multiple ways to do that:

- interprocess communication (IPC) mechanisms provided by your OS (e.g. FIFOs on UNIX)
- message queues (e.g. RabbitMQ, ZeroMQ)
- others (e.g. writing a web app that you send HTTP requests, ...)

[bowlofred]

for image in capture:
    #This take some time
    a = network1(image)
    #This also take some time
    b = network2(image)

This doesn't directly spawn a new process. It waits for the data from each to return serially. What you could do instead is run it as a subprocess. Then the question becomes how you want to transfer the data between the processes. You could use pipes (stdin/stdout), or you could just have the subprocess read from a file and write it back out.

The advantage to this is you don't have to manage some daemon process that could fail. IPC can be simpler.

One way to do this is to imagine the processors just use the filesystem for I/O. We'd pass in the name of the file to process and the location of the output. Maybe something like this:

import subprocess
from pathlib import Path
cmd = "processor.py"
image_file = Path("path", "to", "image.img")

proc_a = subprocess.Popen(["python3", cmd, "--style", "network1", "--input", image_file], stdout=subprocess.PIPE)
proc_b = subprocess.Popen(["python3", cmd, "--style", "network2", "--input", image_file], stdout=subprocess.PIPE)

# image is processing here.  Can do some work that doesn't depend on the output

# now I need both outputs.  Wait for output to tell us the processing is done
output_a = proc_a.communicate()
output_b = proc_b.communicate()

# both images are done processing.  Read the output files to use.