Your code is ok.
Just replaced one line for testing:
With a context manager it's easier to handle the locking:
Then the Pool is automatically closed, if leaving the block.
Just replaced one line for testing:
import multiprocessing from multiprocessing import Value error_value = Value('i',0) #assign as integer type def get_data(url): try: raise ValueError except ValueError as e: #if error thrown up simply count it error_value.acquire() error_value.value += 1 error_value.release() pass # <- not required, does nothing if __name__ == '__main__': # ---- Perform multi-processing p = multiprocessing.Pool(6) results = p.map(get_data,'argslist') p.close() p.join() print(str(error_value.value))
Output:8
This is right, because the str "argslist" consist of 8 elements.With a context manager it's easier to handle the locking:
import multiprocessing from multiprocessing import Value error_value = Value('i',0) #assign as integer type def get_data(url): try: # all even numbers -> ZeroDivisionError if url % 2 == 0: 1 / 0 except ZeroDivisionError as e: with error_value.get_lock(): # read/write is not an atomic operatoion # lock is required to do it right # the context manager makes the use of it easier error_value.value += 1 print("ZeroDivisionError -->", url, "/ 0") if __name__ == '__main__': with multiprocessing.Pool(6) as p: results = p.map(get_data, [0,1,2,3,4]) # usually you should consume the results # actually get_data does not return explicit something # so it returns implicit for each call None for result in results: ... # pro tip: 3 dots is Ellipsis, a placeholder used for numpy # I use it often to have a placeholder for missing code print("Total errors:", error_value.value)
multiprocessing.Pool
can also used with a context manager.Then the Pool is automatically closed, if leaving the block.
Output:ZeroDivisionError --> 0 / 0
ZeroDivisionError --> 2 / 0
ZeroDivisionError --> 4 / 0
Total errors: 3
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
All humans together. We don't need politicians!