Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor) - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor) (/thread-35514.html) |
Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor) - Tomli - Nov-11-2021 Hello everyone! The last code that I was working on was very taxing and slow, so I decided to optimize it by using multiprocessing. When I did this, I discovered that somehow class variables that have been changed in code before calling ProcessPoolExecutor aren't taken into account when instantiating the processes. As an example, I prepared the following code: import concurrent.futures class test(): class_variable = None def test_method(self,iter): print(test.class_variable) if __name__=='__main__': test.class_variable = 1 inst = test() print(f"Class variable before pool: {test.class_variable=}") with concurrent.futures.ProcessPoolExecutor() as executor: executor.map(inst.test_method,[x for x in range(5)])If you run this code, the results are: Class variable before pool: test.class_variable=1 test.class_variable=None test.class_variable=None test.class_variable=None test.class_variable=None test.class_variable=None As I said at the start, even if we change the class variable before using multiprocessing they will be ignored and the code will use the original value (None in this case). I wanted to know if someone knows a way to make the processes take into account the change in class variables. RE: Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor) - Larz60+ - Nov-12-2021 You need to instantiate test class before using it. line 10 should be after line 11. import concurrent.futures class test(): class_variable = None def test_method(self,iter): print(test.class_variable) if __name__=='__main__': inst = test() test.class_variable = 1 print(f"Class variable before pool: {test.class_variable=}") with concurrent.futures.ProcessPoolExecutor() as executor: executor.map(inst.test_method,[x for x in range(5)])
RE: Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor) - deanhystad - Nov-12-2021 I think this is a Windows/Linux thing. When I run Larz60+ code on my Windows 10 laptop the output is: On Linux a new process is created using fork. The new process is a copy of the existing parent process. In Windows a new process is created using spawn. The new process only has the resources needed to run the subprocess.https://docs.python.org/3/library/multiprocessing.html Quote:Contexts and start methodsI think you are running on Windows. Your processes print "None" because the code that sets test.class_variable = 1 has never un on any of the child processes. Windows loaded the module and class test() initialized class_variable = None. The code that modifies the class variable is protected behind Quote:if __name__=='__main__':so it never executes in the child processes. Global variables work the same way. On Linux child processes are forked. They start out as exact copies of the parent process. If you changed a global or class variable in the parent process, the child process will have the modified value. RE: Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor) - snippsat - Nov-12-2021 Windows has as mention None output,code make no sense as it is now.Take a look at this Thread So doing a useful task as downloading images fast,and this work fine in both Window and Linux. RE: Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor) - jefsummers - Nov-12-2021 I also think you are running into problems with the GIL, which tends to limit multithreading/multiprocessing. I suggest reading the Python Wiki about this and it may give you a bit more understanding of your problem. Notable is that with CPUs with more and more cores (such as the Apple M1) this is becoming a bigger issue and will likely be addressed further. RE: Class variables and Multiprocessing(or concurrent.futures.ProcessPoolExecutor) - snippsat - Nov-12-2021 (Nov-12-2021, 09:05 PM)jefsummers Wrote: I also think you are running into problems with the GIL,Not the problem here as concurrent.futures ProcessPoolExecutor() bypass the GIL,the problem here is as mention that code dos nothing useful. concurrent.futures Wrote:ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock |