Weird ProcessPoolExecutor initializer behavior

Weird ProcessPoolExecutor initializer behavior - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Weird ProcessPoolExecutor initializer behavior (/thread-39583.html)

Weird ProcessPoolExecutor initializer behavior - davetapley - Mar-12-2023

In the snippet below I demonstrate how passing an instance method to ProcessPoolExecutor's initializer argument results in the assignment being 'lost' in the spawned process.

Is this by design?

from concurrent.futures import ProcessPoolExecutor, wait
from os import getpid


class Handler:

  def initialize(self):
    print(f"initialize {getpid()}")
    self.test = "one"

  def handle(self):
    try:
      test = self.test
    except AttributeError:
      print(f"handle {getpid()} error")
    else:
      print(f"handle {getpid()} {test}")


handler = Handler()

# this works, presumably because "one" is pickled before process is forked
#handler.initialize()

print('start')
with ProcessPoolExecutor(max_workers=4, initializer=handler.initialize) as ex:
  jobs = [ex.submit(handler.handle) for n in range(4)]
  wait(jobs)

print('done')

RE: Weird ProcessPoolExecutor initializer behavior - deanhystad - Mar-13-2023

Is this windows, linux, mac?

You say "spawned" and "pickled" so I am thinking windows, but you also say forked, and the way the code is written it cannot possibly be spawned. To run on windows you would do something like this:

from concurrent.futures import ProcessPoolExecutor, wait
from os import getpid


class Handler:
    def initialize(self):
        print(f"initialize {getpid()}")
        self.test = "one"

    def handle(self):
        try:
            test = self.test
        except AttributeError:
            print(f"handle {getpid()} error")
        else:
            print(f"handle {getpid()} {test}")


if __name__ == "__main__":
    handler = Handler()
    # this works, presumably because "one" is pickled before process is forked
    handler.initialize()

    print("start")
    with ProcessPoolExecutor(max_workers=4, initializer=handler.initialize) as ex:
        jobs = [ex.submit(handler.handle) for n in range(4)]
        wait(jobs)

    print("done")

What is your target OS?

RE: Weird ProcessPoolExecutor initializer behavior - davetapley - Mar-13-2023

(Mar-13-2023, 04:30 PM)deanhystad Wrote: Is this windows, linux, mac?

You say "spawned" and "pickled" so I am thinking windows, but you also say forked, and the way the code is written it cannot possibly be spawned.
What is your target OS?

Linux. I tried both 'fork' and 'spawn' (via mp_context).

1. In both cases 'initialize' is called on the child pid (as expected).
2. With 'spawn' 'handle' is called on the parent pid (which seems wrong) actually I think this is just a quirk of getting the child pid
3. In both cases 'self.test' isn't set in 'handle' (which makes sense with spawn, but not with fork)

Thanks.

I also put it here: https://replit.com/@davetapley/ProcessPoolExecutorBug#main.py