Python Forum
How to use pool.map effectively?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to use pool.map effectively?
#1
Python 3.7.3

if self.multiprocessing:
    quantity_process = multiprocessing.cpu_count()
else:
    quantity_process = 1
         
pool = multiprocessing.Pool(processes=quantity_process, initializer=initializer(), maxtasksperchild=1)
output_data = pool.map(count, self.input_data)
pool.close()
pool.join()
On 4 cores, I launched about 150 tasks using pool.map, but over time I saw in the system monitor that python processes are gradually ending, although the remaining processes continue to receive new tasks (as can be seen from the debugging information). In other words, tasks are not divided on the fly, but at the beginning pool.map, and therefore some processes get easier tasks and are performed faster, leaving all the difficult work to their "unsuccessful" ones.

How to avoid the fact, that not all process of pool work part time?
Reply


Messages In This Thread
How to use pool.map effectively? - by AlekseyPython - Oct-06-2019, 11:15 AM
RE: How to use pool.map effectively? - by Larz60+ - Oct-06-2019, 12:00 PM
RE: How to use pool.map effectively? - by Larz60+ - Oct-06-2019, 03:08 PM
RE: How to use pool.map effectively? - by wavic - Oct-06-2019, 06:48 PM
RE: How to use pool.map effectively? - by wavic - Oct-07-2019, 10:09 AM
RE: How to use pool.map effectively? - by DeaD_EyE - Oct-07-2019, 08:20 AM
RE: How to use pool.map effectively? - by DeaD_EyE - Oct-07-2019, 01:01 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Multiprocessing Pool Multiple Instances How to Kill by Pool ID sunny9495 0 820 Nov-16-2022, 05:57 AM
Last Post: sunny9495
  Pool multiprocessing - know current status in loop? korenron 0 1,687 Jul-28-2021, 08:49 AM
Last Post: korenron
  pool mysql error - not catch by try\except? korenron 1 2,208 Jul-05-2021, 11:26 AM
Last Post: ibreeden
  Process (pool,map) strange behaviour maverick76 1 1,991 Feb-03-2020, 02:43 PM
Last Post: maverick76
  running multiple commands in a parallel pool Skaperen 6 4,065 Jul-30-2019, 05:49 AM
Last Post: Skaperen
  pool map cycle skorost5 5 3,950 Apr-07-2019, 09:21 AM
Last Post: skorost5
  mysql connection pool? MuntyScruntfundle 0 2,091 Oct-20-2018, 07:36 PM
Last Post: MuntyScruntfundle
  How to run different processes in a pool of 5 lravikumarvsp 0 2,402 May-29-2018, 09:43 AM
Last Post: lravikumarvsp
  How to restart the processes in the pool after the pool got completed lravikumarvsp 0 3,970 May-27-2018, 02:18 PM
Last Post: lravikumarvsp

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020