Python Forum
Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor)
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor)
#1
Hello everyone!

The last code that I was working on was very taxing and slow, so I decided to optimize it by using multiprocessing. When I did this, I discovered that somehow class variables that have been changed in code before calling ProcessPoolExecutor aren't taken into account when instantiating the processes. As an example, I prepared the following code:
import concurrent.futures

class test():
    class_variable = None

    def test_method(self,iter):
        print(test.class_variable)
    
if __name__=='__main__':
    test.class_variable = 1
    inst = test()
    print(f"Class variable before pool: {test.class_variable=}")
    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.map(inst.test_method,[x for x in range(5)])
If you run this code, the results are:
Class variable before pool: test.class_variable=1
test.class_variable=None
test.class_variable=None
test.class_variable=None
test.class_variable=None
test.class_variable=None

As I said at the start, even if we change the class variable before using multiprocessing they will be ignored and the code will use the original value (None in this case).

I wanted to know if someone knows a way to make the processes take into account the change in class variables.
Reply
#2
You need to instantiate test class before using it.
line 10 should be after line 11.

import concurrent.futures


class test():
    class_variable = None
 
    def test_method(self,iter):
        print(test.class_variable)
     

if __name__=='__main__':
    inst = test()
    test.class_variable = 1
    print(f"Class variable before pool: {test.class_variable=}")
    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.map(inst.test_method,[x for x in range(5)])
Output:
Class variable before pool: test.class_variable=1 1 1 1 1 1
Reply
#3
I think this is a Windows/Linux thing. When I run Larz60+ code on my Windows 10 laptop the output is:
Output:
Class variable before pool: test.class_variable=1 None None None None None
On Linux a new process is created using fork. The new process is a copy of the existing parent process. In Windows a new process is created using spawn. The new process only has the resources needed to run the subprocess.

https://docs.python.org/3/library/multiprocessing.html
Quote:Contexts and start methods
Depending on the platform, multiprocessing supports three ways to start a process. These start methods are

spawn
The parent process starts a fresh python interpreter process. The child process will only inherit those resources necessary to run the process object’s run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using fork or forkserver.

Available on Unix and Windows. The default on Windows and macOS.

fork
The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic.

Available on Unix only. The default on Unix.

forkserver
When the program starts and selects the forkserver start method, a server process is started. From then on, whenever a new process is needed, the parent process connects to the server and requests that it fork a new process. The fork server process is single threaded so it is safe for it to use os.fork(). No unnecessary resources are inherited.

Available on Unix platforms which support passing file descriptors over Unix pipes.
I think you are running on Windows. Your processes print "None" because the code that sets test.class_variable = 1 has never un on any of the child processes. Windows loaded the module and class test() initialized class_variable = None. The code that modifies the class variable is protected behind
Quote:if __name__=='__main__':
so it never executes in the child processes. Global variables work the same way.

On Linux child processes are forked. They start out as exact copies of the parent process. If you changed a global or class variable in the parent process, the child process will have the modified value.
Tomli likes this post
Reply
#4
Windows has as mention None output,code make no sense as it is now.
Take a look at this Thread
So doing a useful task as downloading images fast,and this work fine in both Window and Linux.
Reply
#5
I also think you are running into problems with the GIL, which tends to limit multithreading/multiprocessing. I suggest reading the Python Wiki about this and it may give you a bit more understanding of your problem.

Notable is that with CPUs with more and more cores (such as the Apple M1) this is becoming a bigger issue and will likely be addressed further.
Reply
#6
(Nov-12-2021, 09:05 PM)jefsummers Wrote: I also think you are running into problems with the GIL,
Not the problem here as concurrent.futures ProcessPoolExecutor() bypass the GIL,the problem here is as mention that code dos nothing useful.
concurrent.futures Wrote:ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock
but also means that only picklable objects can be executed and returned.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Unchangeable variables in a class? Calab 12 1,402 Sep-15-2023, 07:15 PM
Last Post: deanhystad
  Concurrent futures threading running at same speed as non-threading billykid999 13 1,715 May-03-2023, 08:22 AM
Last Post: billykid999
  Weird ProcessPoolExecutor initializer behavior davetapley 2 1,576 Mar-13-2023, 06:49 PM
Last Post: davetapley
  PyRun_SimpleFile calling multiprocessing Python Class cause endless init loop Xeno 2 988 Sep-19-2022, 02:32 AM
Last Post: Xeno
  How to pass variables from one class to another hobbyist 18 10,376 Oct-01-2021, 05:54 PM
Last Post: deanhystad
  Problem with concurrent.futures. thunderspeed 3 1,988 Sep-01-2021, 05:21 PM
Last Post: snippsat
  concurrent.futures help samuelbachorik 2 1,706 Aug-22-2021, 07:20 AM
Last Post: bowlofred
  Acess variables from class samuelbachorik 3 1,865 Aug-20-2021, 02:55 PM
Last Post: deanhystad
  Multiprocessing, class, run and a Queue Object SeanInColo 0 1,491 Jul-12-2020, 05:36 PM
Last Post: SeanInColo
  Class variables menator01 2 1,957 Jun-04-2020, 04:23 PM
Last Post: Yoriz

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020