Python Forum
How to use pool.map effectively?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to use pool.map effectively?
#1
Python 3.7.3

if self.multiprocessing:
    quantity_process = multiprocessing.cpu_count()
else:
    quantity_process = 1
         
pool = multiprocessing.Pool(processes=quantity_process, initializer=initializer(), maxtasksperchild=1)
output_data = pool.map(count, self.input_data)
pool.close()
pool.join()
On 4 cores, I launched about 150 tasks using pool.map, but over time I saw in the system monitor that python processes are gradually ending, although the remaining processes continue to receive new tasks (as can be seen from the debugging information). In other words, tasks are not divided on the fly, but at the beginning pool.map, and therefore some processes get easier tasks and are performed faster, leaving all the difficult work to their "unsuccessful" ones.

How to avoid the fact, that not all process of pool work part time?
Reply


Messages In This Thread
How to use pool.map effectively? - by AlekseyPython - Oct-06-2019, 11:15 AM
RE: How to use pool.map effectively? - by Larz60+ - Oct-06-2019, 12:00 PM
RE: How to use pool.map effectively? - by Larz60+ - Oct-06-2019, 03:08 PM
RE: How to use pool.map effectively? - by wavic - Oct-06-2019, 06:48 PM
RE: How to use pool.map effectively? - by wavic - Oct-07-2019, 10:09 AM
RE: How to use pool.map effectively? - by DeaD_EyE - Oct-07-2019, 08:20 AM
RE: How to use pool.map effectively? - by DeaD_EyE - Oct-07-2019, 01:01 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Multiprocessing Pool Multiple Instances How to Kill by Pool ID sunny9495 0 790 Nov-16-2022, 05:57 AM
Last Post: sunny9495
  Pool multiprocessing - know current status in loop? korenron 0 1,663 Jul-28-2021, 08:49 AM
Last Post: korenron
  pool mysql error - not catch by try\except? korenron 1 2,181 Jul-05-2021, 11:26 AM
Last Post: ibreeden
  Process (pool,map) strange behaviour maverick76 1 1,965 Feb-03-2020, 02:43 PM
Last Post: maverick76
  running multiple commands in a parallel pool Skaperen 6 4,025 Jul-30-2019, 05:49 AM
Last Post: Skaperen
  pool map cycle skorost5 5 3,871 Apr-07-2019, 09:21 AM
Last Post: skorost5
  mysql connection pool? MuntyScruntfundle 0 2,061 Oct-20-2018, 07:36 PM
Last Post: MuntyScruntfundle
  How to run different processes in a pool of 5 lravikumarvsp 0 2,378 May-29-2018, 09:43 AM
Last Post: lravikumarvsp
  How to restart the processes in the pool after the pool got completed lravikumarvsp 0 3,924 May-27-2018, 02:18 PM
Last Post: lravikumarvsp

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020