Multiprocessing doesn't seem to work?

[t4keheart]

Having trouble getting multiprocessing to work as I had hoped it would. Ideally, I need these 2 processed to run at the same time. They could even open up their own separate console windows and run completely independently, as long as they both start up. Is multiprocessing what I'm looking for?

from Logger import Logger
from s3nd import beginSending
from r3ceive import beginRec
import multiprocessing

def send():
    beginSending()

def receive():
    beginRec()

if __name__ == "__main__":

    sender = multiprocessing.Process(name='sender', target=send)
    receiver = multiprocessing.Process(name='receiver', target=receive)
    sender.start()
    receiver.start()

Basically, only the second process starts (receiver)... what's going on that my first process gets skipped over?

EDIT:
If anybody would like to have a look, these are the sender and receiver modules. They work when they're started on their own in separate shells, but not when started from this 'main.py' program when I try to introduce multiprocessing.
Sender
Receiver

[Larz60+]

what does the sender do when data is exhausted?
Is it possible that the task is completed before you get to receiver.start()

[t4keheart]

what does the sender do when data is exhausted?
Is it possible that the task is completed before you get to receiver.start()

Unfortunately, I don't think that's the case. both of the processes are endless loops, and run as such when started manually in 2 separate python shells.

which is ultimately what I am trying to do, open 2 modules in 2 different shells, and I'm surprised at the difficulties I have been having! They don't need to run in exact parallel or anything, just want to start them both up in separate python shells.

[DeaD_EyE]

... I need these 2 processed to run at the same time.

You could do this with threading (io-bound tasks) or with multiprocessing (cpu-bound tasks).

They could even open up their own separate console windows and run completely independently

The module subprocess does this. It starts a new process.
The module multiprocessing uses subprocess to start the Process and it uses pickle for serialization between the processes. Since Python 3.8 you can use multiprocessing.shared_memory between the processes, which is a bit tricky.

Is multiprocessing what I'm looking for?

I guess no. A new process needs time to start.
A thread needs lesser resources as a process.
But remind, Python is single threaded. Using threads != multi threading
If nothing much happens in your functions, you better use threading.

Multiprocessing is useful, if you have less input (mean less serialization overhead) with a big calculation.
If you use blocking sockets, a thread is the better approach.
Much better for IO is asyncio, but this is difficult to explain.

Basically, only the second process starts (receiver)... what's going on that my first process gets skipped over?

Without code, we can only guess.
Maybe there is an error in the function beginSending().
Put some print functions inside the function or use logging.


IO-Bound in code:

...

while True:
    sock.recv() #  <-- Blocking call, it stays here until minimum one byte has been received

CPU-Bound in code:

while True:
    pass

Nothing is there, which gives other thread time to do their tasks.

If you sleep in your loop, then other tasks could do their work.

while True:
    time.sleep(0.001)
    # 1 ms

But CPU-Bound is also this:
math.factorial(2000000)
Nothing else is running, until this calculation has been finished, even if this runs in a different thread.
This is an effect of the GIL. Only one Python-Statement can run at the same time in the Python Interpreter.
Some extensions like numpy for matrix calculations do use multithreading, but they're written in C with a nice Python-Interface.

[t4keheart]

Thank you both for the explanations. I think i'm going to check out the subprocess and threading and see if either of those do the trick.

I've included codeshare links for the sender and receiver modules if you'd like to have a look- though they're each about 200+ lines of code. I do use logging, and it doesn't show anything at all from the module that apparently doesn't start.

[woooee]

Basically, only the second process starts (receiver)... what's going on that my first process gets skipped over?

Multiprocessing is not going to start one process and not the other. The problem is in the function, not with multiprocessing. Change to function to print every second, or something equally simple to show that multiprocessing is running it correctly.

[t4keheart]

Basically, only the second process starts (receiver)... what's going on that my first process gets skipped over?

Multiprocessing is not going to start one process and not the other. The problem is in the function, not with multiprocessing. Change to function to print every second, or something equally simple to show that multiprocessing is running it correctly.

I'll be sure to let you know once I figure it out... but I don't think you're right.

No issues with the function, when run on its own it runs without error (and it does print something every .3 seconds). AND if you flip the 2 around, make the sender program the second (last) entry on the main program that implements the multiprocessing, the OTHER program is started and runs... I know that whichever the other program is isn't running because it isn't doing what it's designed to do.

Something isn't right with the implementation of the multiprocessing, but I don't think there's any issue with the functions.

[DeaD_EyE]

Something isn't right with the implementation of the multiprocessing, but I don't think there's any issue with the functions.

No :-D
The implementation is well tested.

In your modules, you call the function beginSending() and in the other beginRec().
Don't do this. Remove the call from both modules or use the boilerpate code:

...

if __name__ == '__main__':
    beginRec()

If you use this code, the function beginRec() is not called, when:

• The module was imported
  
• Multiprocessing loads the module
  

Currently you call beginRec() twice and beginSending() twice.
Once the modules were imported and one time when Multiprocessing calls the target functions.

[t4keheart]

In your modules, you call the function beginSending() and in the other beginRec().
Don't do this. Remove the call from both modules or use the boilerpate code:

Hey, thanks for pointing that out. I had the 'beginRec()' and 'beginSending()' calls at the bottom from when I was creating each module, and running the modules themselves 'as main' for testing... I guess I just never removed them... but I didn't think it would cause any harm. I will remove them and see what happens.

I created a simple, reproducible test to test out the multiprocessing implementation, and it works as expected:

main.py

from func1 import runMe
from func2 import runMeToo
import multiprocessing

def func1():
    runMe()

def func2():
    runMeToo()

if __name__ == "__main__":
    first = multiprocessing.Process(name='first', target=func1)
    second = multiprocessing.Process(name='second', target=func2)
    first.start()
    second.start()

func1

import time

def runMe():
    while True:
        print("function 1!")
        time.sleep(1)
        

func2

import time

def runMeToo():
   while True:
        print("function 2!")
        time.sleep(.5)

output:

> function 1!
> function 2!
> function 2!
> function 1!
... etc.

However, Ideally I would like a solution that simply kicks off both 'processes' in completely separate terminal shells. Each of my modules produce output that is important, but unrelated to the other module's function, so I would like to keep their outputs separated amongst two different terminal windows. How can I code something to simple kick off each module independently in seperate terminals?

[DeaD_EyE]

Then use subprocess:

(I have not tested the code)

import sys
import time
from subprocess import Popen


def start_script(filename):
    proc = Popen([sys.executeable, filename])
    return proc

# s3nd and r3ceive are bad names
sender_proc = start_script('s3nd.py')
receiver_proc = start_script('r3ceive.py')

# don't forget to put the if __name__ == '__main__'
# into your modules bottom and call the sender and receiver.

time.sleep(10)
sender_proc.terminate()
receiver_proc.terminate()

The question is, what happens if receiver runs before sender?
Your problems goes deeper as you think. The whole structure is wrong.

I think a Queue can solve this problems.
Writing to a file and reading it from another process is very fragile.