Oct-06-2019, 11:15 AM
(This post was last modified: Oct-06-2019, 11:19 AM by AlekseyPython.)
Python 3.7.3
How to avoid the fact, that not all process of pool work part time?
if self.multiprocessing: quantity_process = multiprocessing.cpu_count() else: quantity_process = 1 pool = multiprocessing.Pool(processes=quantity_process, initializer=initializer(), maxtasksperchild=1) output_data = pool.map(count, self.input_data) pool.close() pool.join()On 4 cores, I launched about 150 tasks using pool.map, but over time I saw in the system monitor that python processes are gradually ending, although the remaining processes continue to receive new tasks (as can be seen from the debugging information). In other words, tasks are not divided on the fly, but at the beginning pool.map, and therefore some processes get easier tasks and are performed faster, leaving all the difficult work to their "unsuccessful" ones.
How to avoid the fact, that not all process of pool work part time?