Oct-06-2019, 06:48 PM
from concurrent.futures import ProcessPoolExecutor with ProcessPoolExecutor() as executor: results = executor.map(my_func, args=args_list)multiprocessing.cpu_count() gives the number of the CPU cores. However, if you use all of them it can affect the whole system.
Map method will use all available cores.
I have somewhere a script which downloads in parallel more than 200 web pages and I tested it on 4 core system with a different number of processes. The best time I got was with around 30 processes set. If I found it, I can run the script here again to see what is going on and share it here. This system has 6 cores though.