Python Forum
How to use pool.map effectively?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to use pool.map effectively?
#5
from concurrent.futures import ProcessPoolExecutor

with ProcessPoolExecutor() as executor:
    results = executor.map(my_func, args=args_list)
multiprocessing.cpu_count() gives the number of the CPU cores. However, if you use all of them it can affect the whole system.
Map method will use all available cores.

I have somewhere a script which downloads in parallel more than 200 web pages and I tested it on 4 core system with a different number of processes. The best time I got was with around 30 processes set. If I found it, I can run the script here again to see what is going on and share it here. This system has 6 cores though.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply


Messages In This Thread
How to use pool.map effectively? - by AlekseyPython - Oct-06-2019, 11:15 AM
RE: How to use pool.map effectively? - by Larz60+ - Oct-06-2019, 12:00 PM
RE: How to use pool.map effectively? - by Larz60+ - Oct-06-2019, 03:08 PM
RE: How to use pool.map effectively? - by wavic - Oct-06-2019, 06:48 PM
RE: How to use pool.map effectively? - by wavic - Oct-07-2019, 10:09 AM
RE: How to use pool.map effectively? - by DeaD_EyE - Oct-07-2019, 08:20 AM
RE: How to use pool.map effectively? - by DeaD_EyE - Oct-07-2019, 01:01 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Multiprocessing Pool Multiple Instances How to Kill by Pool ID sunny9495 0 790 Nov-16-2022, 05:57 AM
Last Post: sunny9495
  Pool multiprocessing - know current status in loop? korenron 0 1,663 Jul-28-2021, 08:49 AM
Last Post: korenron
  pool mysql error - not catch by try\except? korenron 1 2,181 Jul-05-2021, 11:26 AM
Last Post: ibreeden
  Process (pool,map) strange behaviour maverick76 1 1,965 Feb-03-2020, 02:43 PM
Last Post: maverick76
  running multiple commands in a parallel pool Skaperen 6 4,023 Jul-30-2019, 05:49 AM
Last Post: Skaperen
  pool map cycle skorost5 5 3,870 Apr-07-2019, 09:21 AM
Last Post: skorost5
  mysql connection pool? MuntyScruntfundle 0 2,060 Oct-20-2018, 07:36 PM
Last Post: MuntyScruntfundle
  How to run different processes in a pool of 5 lravikumarvsp 0 2,378 May-29-2018, 09:43 AM
Last Post: lravikumarvsp
  How to restart the processes in the pool after the pool got completed lravikumarvsp 0 3,924 May-27-2018, 02:18 PM
Last Post: lravikumarvsp

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020