Python Forum
Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor)
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor)
#1
Hello everyone!

The last code that I was working on was very taxing and slow, so I decided to optimize it by using multiprocessing. When I did this, I discovered that somehow class variables that have been changed in code before calling ProcessPoolExecutor aren't taken into account when instantiating the processes. As an example, I prepared the following code:
import concurrent.futures

class test():
    class_variable = None

    def test_method(self,iter):
        print(test.class_variable)
    
if __name__=='__main__':
    test.class_variable = 1
    inst = test()
    print(f"Class variable before pool: {test.class_variable=}")
    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.map(inst.test_method,[x for x in range(5)])
If you run this code, the results are:
Class variable before pool: test.class_variable=1
test.class_variable=None
test.class_variable=None
test.class_variable=None
test.class_variable=None
test.class_variable=None

As I said at the start, even if we change the class variable before using multiprocessing they will be ignored and the code will use the original value (None in this case).

I wanted to know if someone knows a way to make the processes take into account the change in class variables.
Reply
#2
You need to instantiate test class before using it.
line 10 should be after line 11.

import concurrent.futures


class test():
    class_variable = None
 
    def test_method(self,iter):
        print(test.class_variable)
     

if __name__=='__main__':
    inst = test()
    test.class_variable = 1
    print(f"Class variable before pool: {test.class_variable=}")
    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.map(inst.test_method,[x for x in range(5)])
Output:
Class variable before pool: test.class_variable=1 1 1 1 1 1
Reply
#3
I think this is a Windows/Linux thing. When I run Larz60+ code on my Windows 10 laptop the output is:
Output:
Class variable before pool: test.class_variable=1 None None None None None
On Linux a new process is created using fork. The new process is a copy of the existing parent process. In Windows a new process is created using spawn. The new process only has the resources needed to run the subprocess.

https://docs.python.org/3/library/multiprocessing.html
Quote:Contexts and start methods
Depending on the platform, multiprocessing supports three ways to start a process. These start methods are

spawn
The parent process starts a fresh python interpreter process. The child process will only inherit those resources necessary to run the process object’s run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using fork or forkserver.

Available on Unix and Windows. The default on Windows and macOS.

fork
The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic.

Available on Unix only. The default on Unix.

forkserver
When the program starts and selects the forkserver start method, a server process is started. From then on, whenever a new process is needed, the parent process connects to the server and requests that it fork a new process. The fork server process is single threaded so it is safe for it to use os.fork(). No unnecessary resources are inherited.

Available on Unix platforms which support passing file descriptors over Unix pipes.
I think you are running on Windows. Your processes print "None" because the code that sets test.class_variable = 1 has never un on any of the child processes. Windows loaded the module and class test() initialized class_variable = None. The code that modifies the class variable is protected behind
Quote:if __name__=='__main__':
so it never executes in the child processes. Global variables work the same way.

On Linux child processes are forked. They start out as exact copies of the parent process. If you changed a global or class variable in the parent process, the child process will have the modified value.
Tomli likes this post
Reply
#4
Windows has as mention None output,code make no sense as it is now.
Take a look at this Thread
So doing a useful task as downloading images fast,and this work fine in both Window and Linux.
Reply
#5
I also think you are running into problems with the GIL, which tends to limit multithreading/multiprocessing. I suggest reading the Python Wiki about this and it may give you a bit more understanding of your problem.

Notable is that with CPUs with more and more cores (such as the Apple M1) this is becoming a bigger issue and will likely be addressed further.
Reply
#6
(Nov-12-2021, 09:05 PM)jefsummers Wrote: I also think you are running into problems with the GIL,
Not the problem here as concurrent.futures ProcessPoolExecutor() bypass the GIL,the problem here is as mention that code dos nothing useful.
concurrent.futures Wrote:ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock
but also means that only picklable objects can be executed and returned.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to pass variables from one class to another hobbyist 18 1,631 Oct-01-2021, 05:54 PM
Last Post: deanhystad
  Problem with concurrent.futures. thunderspeed 3 548 Sep-01-2021, 05:21 PM
Last Post: snippsat
  concurrent.futures help samuelbachorik 2 573 Aug-22-2021, 07:20 AM
Last Post: bowlofred
  Acess variables from class samuelbachorik 3 575 Aug-20-2021, 02:55 PM
Last Post: deanhystad
  Multiprocessing, class, run and a Queue Object SeanInColo 0 720 Jul-12-2020, 05:36 PM
Last Post: SeanInColo
  Class variables menator01 2 905 Jun-04-2020, 04:23 PM
Last Post: Yoriz
  Question about naming variables in class methods sShadowSerpent 1 967 Mar-25-2020, 04:51 PM
Last Post: ndc85430
  Trying to understand concurrent.futures.ThreadPoolExecutor Laxminarsaiah 0 804 Dec-18-2019, 12:43 PM
Last Post: Laxminarsaiah
  Understanding Class Variables vindo 9 2,025 Jun-05-2019, 08:04 PM
Last Post: Yoriz
  What is the strategy for working with class variables? AlekseyPython 3 1,668 Feb-24-2019, 05:34 AM
Last Post: AlekseyPython

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020