Oct-27-2020, 08:43 AM
Hi, I am a python beginner, I am starting to learn multiprocessing. I have created this program to simply calculate the squares of a big list of numbers.
The program takes a start_number and a end_number, separates the numbers into groups, then uses multiprocessing to calculate the results.
Each process puts the results in a List, and form a dictionary with their sequence number as key, then put into a Queue.
At the end, the program then combine the results as a list and print out the result list.
The program works fine with the start_number = 1, and end_number = any number between 1 to 12_998.
However, the program does not work when end_number = 12,999. It may work with certain end_number and not others passing 12_999. For example it works with 13_000 thru 13_005 but not 13_006 and not 13_007. It works with 13_008 thru 13_012 but not 13_013 ..... and so on.
I have tried running the same program in different computers with different CPU counts, and both windows and linux. The results are the same. I am using python 3.6.9
I have learned later that multiprocess.Pool is easier to use in this scenario. But I am interested to know what was wrong with my multiprocess.Process program, if anyone is kind enough to have a look.
Thank you.
-----------------
My codes:
-----------------
The Result when end_number = 12_999 is used.
----------------------------------------------------
The program takes a start_number and a end_number, separates the numbers into groups, then uses multiprocessing to calculate the results.
Each process puts the results in a List, and form a dictionary with their sequence number as key, then put into a Queue.
At the end, the program then combine the results as a list and print out the result list.
The program works fine with the start_number = 1, and end_number = any number between 1 to 12_998.
However, the program does not work when end_number = 12,999. It may work with certain end_number and not others passing 12_999. For example it works with 13_000 thru 13_005 but not 13_006 and not 13_007. It works with 13_008 thru 13_012 but not 13_013 ..... and so on.
I have tried running the same program in different computers with different CPU counts, and both windows and linux. The results are the same. I am using python 3.6.9
I have learned later that multiprocess.Pool is easier to use in this scenario. But I am interested to know what was wrong with my multiprocess.Process program, if anyone is kind enough to have a look.
Thank you.
-----------------
My codes:
-----------------
# This program hang when end number = 12_999 , 13_006, 13_007, 13_013.. etc
from multiprocessing import Process, Queue
from multiprocessing import log_to_stderr, get_logger
# divide the range of numbers into groups. so that each group can be processed with different Process.
def dividegroup(minno, maxno, nof_groups):
total_range = maxno - minno + 1
remain = total_range % nof_groups
group_range = total_range // nof_groups
print(total_range, group_range, remain)
lof_groups = []
for i in range(nof_groups):
lof_groups.append( range( minno + (i * group_range), minno + ((i + 1) * group_range )))
if remain != 0 : lof_groups.append(range(minno + ((i + 1) * group_range), maxno + 1))
return lof_groups
# square the numbers and put in Queque
def square(seq, numbers, q):
answers = [x * x for x in numbers]
results = {seq: answers}
q.put(results, block = False)
# main program
def main():
log_to_stderr()
logger = get_logger()
logger.setLevel(20)
print('\033c')
start_number = 1
end_number = 12_999
number_of_groups = 8
list_of_groups = dividegroup(start_number, end_number, number_of_groups)
print(list(list_of_groups))
q = Queue(maxsize=0)
process_seq = 0
processes = []
for i in list_of_groups:
process_seq += 1
process = Process(target = square, args = (process_seq, i, q))
processes.append(process)
for process_s in processes:
process_s.start()
for process_j in processes:
process_j.join()
result_dic = {}
while not q.empty():
result_dic.update(q.get())
result_list = []
keylist = list(result_dic.keys())
keylist.sort()
for i in keylist:
result_list += result_dic.get(i)
print(result_list)
if __name__ == '__main__':
main()----------------------------------------------------The Result when end_number = 12_999 is used.
----------------------------------------------------
Output:12999 1624 7
[range(1, 1625), range(1625, 3249), range(3249, 4873), range(4873, 6497), range(6497, 8121), range(8121, 9745), range(9745, 11369), range(11369, 12993), range(12993, 13000)]
[INFO/Process-3] child process calling self.run()
[INFO/Process-1] child process calling self.run()
[INFO/Process-3] process shutting down
[INFO/Process-2] child process calling self.run()
[INFO/Process-4] child process calling self.run()
[INFO/Process-3] process exiting with exitcode 0
[INFO/Process-7] child process calling self.run()
[INFO/Process-1] process shutting down
[INFO/Process-1] process exiting with exitcode 0
[INFO/Process-8] child process calling self.run()
[INFO/Process-9] child process calling self.run()
[INFO/Process-6] child process calling self.run()
[INFO/Process-4] process shutting down
[INFO/Process-8] process shutting down
[INFO/Process-4] process exiting with exitcode 0
[INFO/Process-5] child process calling self.run()
[INFO/Process-9] process shutting down
[INFO/Process-7] process shutting down
[INFO/Process-2] process shutting down
[INFO/Process-5] process shutting down
[INFO/Process-6] process shutting down
[INFO/Process-9] process exiting with exitcode 0
[INFO/Process-2] process exiting with exitcode 0
[INFO/Process-8] process exiting with exitcode 0
[INFO/Process-7] process exiting with exitcode 0
[INFO/Process-6] process exiting with exitcode 0