Python Forum
How to run different processes in a pool of 5
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to run different processes in a pool of 5
#1
Hi,

I had a python script which executes different processes in parallel.
How can I modify this script so that it can run the parallel processes in a pool of 5 at a time?
Please help
from multiprocessing import Pool
from multiprocessing import Process
import glob
import os
import csv
from threading import Thread
from multiprocessing import Pool as ThreadPool
import time
import multiprocessing
patterns = ['ACM','ACX','AW','BC','XU0', 'DRM', 'DHD', 'CR', 'GSK', 'DMS', 'BLS']
	
def process(pattern=[]):
	
	base_path = '/ai2/data/dev/admin/inf/*{}*'
	print("ID of process: {}".format(os.getpid()))
	print "Name of the process is :" ,pattern
	
	

	search_path =  base_path.format(pattern)
	for f in glob.glob(search_path):
		print("-----------------------")
		print ("The directory path is:")
		print f
		print("List of files in the directory are:")
		os.chdir('/ai2/data/dev/admin/inf/')
		os.chdir(f)
		cwd = os.getcwd()
		for subdir, dirs, files in os.walk(cwd, topdown=True):
			for file23 in glob.glob('*.err'):
				print file23	
if __name__ == "__main__":  # confirms that the code is under main function
	patterns = ['ACM','ACX','AW','BC','XU0', 'DRM', 'DHD', 'CR', 'GSK', 'DMS', 'BLS']
	procs = []
	#proc = Process(target=process)  # instantiating without any argument
	#procs.append(proc)
	#proc.start()
	
    # instantiating process with arguments
	for pattern in patterns:
		#print(name)
		proc = Process(target=process,args=(pattern,))
		procs.append(proc)
		proc.start()
	
Output:
ID of process: 5789 Name of the process is : ACM ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_ACM_pvt List of files in the directory are: dsplit.err ID of process: 5793 Name of the process is : XU0 ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_XU0_pvt List of files in the directory are: t_itm.err ID of process: 5791 Name of the process is : AW ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_AW_pvt List of files in the directory are: aware.err ID of process: 5796 Name of the process is : CR ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_CR_pvt List of files in the directory are: issue.err ID of process: 5792 Name of the process is : BC ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_BC_pvt List of files in the directory are: run_ingest_BC_daily_1249.err ID of process: 5797 Name of the process is : GSK ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_GSK_pvt List of files in the directory are: gs_attribute.err ID of process: 5790 Name of the process is : ACX ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_ACX_pvt List of files in the directory are: accountshighfocus.err acm_access_log.err ID of process: 5795 Name of the process is : DHD ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_DHD_pvt List of files in the directory are: retpaid.err ID of process: 5794 Name of the process is : DRM ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_DRM_pvt List of files in the directory are: call_report_y9clookup.err ID of process: 5798 Name of the process is : DMS ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_DMS_pvt List of files in the directory are: run_ingest_DMS_daily_1235.err ID of process: 5799 Name of the process is : BLS ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_BLS_pvt List of files in the directory are: run_ingest_BLS_daily_1241.err
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Multiprocessing Pool Multiple Instances How to Kill by Pool ID sunny9495 0 766 Nov-16-2022, 05:57 AM
Last Post: sunny9495
  processes shall be parallel flash77 4 1,119 Sep-20-2022, 11:46 AM
Last Post: DeaD_EyE
  Sharing imported modules with Sub Processes? Stubblemonster 2 1,512 May-02-2022, 06:42 AM
Last Post: Stubblemonster
  Killing processes via python Lavina 2 2,625 Aug-04-2021, 06:20 AM
Last Post: warnerarc
  Pool multiprocessing - know current status in loop? korenron 0 1,640 Jul-28-2021, 08:49 AM
Last Post: korenron
  pool mysql error - not catch by try\except? korenron 1 2,151 Jul-05-2021, 11:26 AM
Last Post: ibreeden
  How to share a numpy array between 2 processes on Windows? qstdy 0 2,169 Jan-29-2021, 04:24 AM
Last Post: qstdy
  sharing variables between two processes Kiyoshi767 1 1,872 Nov-07-2020, 04:00 AM
Last Post: ndc85430
  2 or more processes on the write end of the same pipe Skaperen 4 3,881 Sep-27-2020, 06:41 PM
Last Post: Skaperen
  Errors using --processes parameter sonhospa 3 2,400 Jul-01-2020, 02:24 PM
Last Post: sonhospa

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020