Python Forum
How does multi-processing in python increase the execution time?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How does multi-processing in python increase the execution time?
#1
I have tried simple piece of python code containing two for-loops using multi-processing pool.

First I set the pool=1, total execution time was= 22211 ms.

while for pool=40, total execution time for each of 40 processes was=

Time taken: 24045, Time taken: 24047, Time taken: 24072, Time taken: 24096, Time taken: 24117, Time taken: 24120, Time taken: 24158, Time taken: 24165, Time taken: 24215, Time taken: 24346, Time taken: 24395, Time taken: 24499, Time taken: 24619, Time taken: 24860, Time taken: 27499, Time taken: 31077, Time taken: 32225, Time taken: 33708, Time taken: 37483, Time taken: 37516, Time taken: 37520, Time taken: 37532, Time taken: 37555, Time taken: 37575, Time taken: 37579, Time taken: 37590, Time taken: 37636, Time taken: 37639, Time taken: 37671, Time taken: 37672, Time taken: 37715, Time taken: 37780, Time taken: 37799, Time taken: 37805, Time taken: 37821, Time taken: 37832, Time taken: 37849, Time taken: 37958, Time taken: 38080, Time taken: 38086 (all time in ms.)

System info: It was ran on machine of 56 CPU cores with hyper-threading. My code is:

import time
from multiprocessing import Pool
def compute(da):
    start_time = int(round(time.time() * 1000))
    sum = 0
    for i in range(10000):
        for j in range(50000):
            sum = sum + j
    print (sum)
    print ('Time taken: '+ str(int(round(time.time() * 1000))-start_time))

#start_time = int(round(time.time() * 1000))
data1 = ['1']
data40 = ['1','2','3','4','5','6','7','8','9','10','11','12,'13','14','15','16','17','18','19','20','21','22','23','24','25','26','27','28','29','30','31','32','33','34','35','36','37','38','39','40]
# FOR ONE PROCESS
p = Pool(1)
p.map(compute, data1)
# FOR 40 PROCESS
p = Pool(40)
p.map(compute, data40)
As we could see the execution time increases gradually from 24045(1st process) to 38086(40th process).

I would like to know, Why does this happen? Thanks in advance.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  kill python execution program lebossejames 0 182 Mar-16-2024, 11:16 AM
Last Post: lebossejames
  Confused about python execution jpezz 4 1,324 Oct-09-2022, 06:56 PM
Last Post: Gribouillis
  How to change UTC time to local time in Python DataFrame? SamKnight 2 1,527 Jul-28-2022, 08:23 AM
Last Post: Pedroski55
  Increase the speed of a python loop over a pandas dataframe mcva 0 1,288 Jan-21-2022, 06:24 PM
Last Post: mcva
  mysql.connector.errors.ProgrammingError: Failed processing format-parameters; Python ilknurg 3 5,456 Jan-18-2022, 06:25 PM
Last Post: ilknurg
  How to increase the size of a png picture for the heatmap of the correlation? lulu43366 9 3,383 Oct-06-2021, 04:15 PM
Last Post: deanhystad
  Using SoX in Python to Increase mp3 Bitrate DRT 1 1,714 Jul-10-2021, 08:41 PM
Last Post: DRT
  Why recursive function consumes more of processing time than loops? M83Linux 9 4,128 May-20-2021, 01:52 PM
Last Post: DeaD_EyE
  SimpleHTTPRequestHandler ( verses ) Files Python Processing JohnnyCoffee 0 1,750 Apr-29-2021, 02:47 AM
Last Post: JohnnyCoffee
  How to measure execution time of a multithread loop spacedog 2 2,835 Apr-24-2021, 07:52 AM
Last Post: spacedog

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020