Python Forum
Updating variables in Multi-processing
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Updating variables in Multi-processing
#2
Your code is ok.
Just replaced one line for testing:
import multiprocessing
from multiprocessing import Value
 
error_value = Value('i',0)  #assign as integer type
 
def get_data(url):
    try:
        raise ValueError
 
    except ValueError as e:
       #if error thrown up simply count it
        error_value.acquire()
        error_value.value += 1
        error_value.release()
        pass # <- not required, does nothing
 
 
if __name__ == '__main__':
 
# ---- Perform multi-processing 
    p = multiprocessing.Pool(6)
    results = p.map(get_data,'argslist')
    p.close()
    p.join()
 
    print(str(error_value.value))
Output:
8
This is right, because the str "argslist" consist of 8 elements.


With a context manager it's easier to handle the locking:
import multiprocessing
from multiprocessing import Value
 
error_value = Value('i',0)  #assign as integer type

def get_data(url):
    try:
        # all even numbers -> ZeroDivisionError
        if url % 2 == 0:
            1 / 0
    except ZeroDivisionError as e:
       with error_value.get_lock():
           # read/write is not an atomic operatoion
           # lock is required to do it right
           # the context manager makes the use of it easier
           error_value.value += 1
           print("ZeroDivisionError -->", url, "/ 0")
 
 
if __name__ == '__main__':
    with multiprocessing.Pool(6) as p:
        results = p.map(get_data, [0,1,2,3,4])
        # usually you should consume the results
        # actually get_data does not return explicit something
        # so it returns implicit for each call None
        for result in results:
            ...
        # pro tip: 3 dots is Ellipsis, a placeholder used for numpy
        #          I use it often to have a placeholder for missing code
        print("Total errors:", error_value.value)
multiprocessing.Pool can also used with a context manager.
Then the Pool is automatically closed, if leaving the block.

Output:
ZeroDivisionError --> 0 / 0 ZeroDivisionError --> 2 / 0 ZeroDivisionError --> 4 / 0 Total errors: 3
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply


Messages In This Thread
Updating variables in Multi-processing - by WiPi - Feb-09-2021, 01:11 PM
RE: Updating variables in Multi-processing - by DeaD_EyE - Feb-09-2021, 02:34 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Strategy on updating edits back to data table and object variables hammer 0 1,238 Dec-11-2021, 02:58 PM
Last Post: hammer
  Multi-processing to communicate with microcontrollers Khoily 1 2,577 Mar-01-2019, 08:57 PM
Last Post: DeaD_EyE
  Multi-processing - problem with running multiple *.py files at the same time Antonio 5 3,956 Sep-12-2018, 01:08 PM
Last Post: volcano63
  multi-processing dR_Garuby 1 2,744 Mar-24-2018, 05:59 PM
Last Post: woooee
  How does multi-processing in python increase the execution time? kadsank 0 2,381 Jan-15-2018, 01:15 PM
Last Post: kadsank
  Running Class methods in a loop and updating variables. ujjwalrathod007 3 6,520 Oct-05-2016, 07:11 PM
Last Post: nilamo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020