Python Forum
Python Parallel Programing
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python Parallel Programing
#3
Hi Mr,
first i would like to express all my appreciation for your help.
i have followed your suggestions step by step and i modify my code so as it fit your recommendations, the new code becomes as follow:
import numpy as np
from time import time
import multiprocessing as mp

                        # Prepare data
np.random.RandomState(100)
arr = np.random.randint(0, 10, size=[200000, 5])#print(arr)
data = arr.tolist()
data[:5]
data1=[data[i] for i in range(0,1000)]
data2=[data[i] for i in range(1001,2000)]
data3=[data[i] for i in range(2001,3000)]
data4=[data[i] for i in range(3001,4000)]

                        # Define and function creation
def howmany_within_range(row, minimum, maximum):
    """Returns how many numbers lie within `maximum` and `minimum` in a given `row`"""
    count = 0
    for n in row:
        if minimum <= n <= maximum:
            count = count + 1
    return count

                        # Define funcThread
def funcThread(minimum, maximum):
    howmany_within_range(data1, minimum, maximum)
    howmany_within_range(data2, minimum, maximum)
    howmany_within_range(data3, minimum, maximum)
    howmany_within_range(data4, minimum, maximum)

                            # Block: Paralleization
        # Step 1: Init multiprocessing.Pool()
pool = mp.Pool(mp.cpu_count())
        # Step 2: `pool.apply` the `howmany_within_range()`
results = [pool.apply(funcThread, args=(4, 8))]
        # Step 3: Don't forget to close
pool.close()
pool.join()
        # Step 4: Result
print(results[:10])
the execution of my program gives the same error:

Error:
Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\109_personel\113_ARIMA_Tmin\Test_3.py", line 33, in <module> pool = mp.Pool(mp.cpu_count()) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 119, in Pool context=self.get_context()) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py", line 176, in __init__ Traceback (most recent call last): File "<string>", line 1, in <module> self._repopulate_pool() File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py", line 241, in _repopulate_pool File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main w.start() File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 112, in start exitcode = _main(fd) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 114, in _main self._popen = self._Popen(self) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 322, in _Popen prepare(preparation_data) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 225, in prepare return Popen(process_obj) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__ prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path _check_not_importing_main() File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main run_name="__mp_main__") File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 263, in run_path is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable. pkg_name=pkg_name, script_name=fname)
is there other modifications or do i make something else in my code in order to solve my problem
thank you a lot Mr and your help is highly appreciated.
Reply


Messages In This Thread
Python Parallel Programing - by wissam1974 - Feb-23-2019, 11:09 PM
RE: Python Parallel Programing - by stullis - Feb-24-2019, 12:03 AM
RE: Python Parallel Programing - by wissam1974 - Feb-24-2019, 12:02 PM
RE: Python Parallel Programing - by stullis - Feb-24-2019, 04:59 PM
RE: Python Parallel Programing - by wissam1974 - Feb-24-2019, 05:30 PM
RE: Python Parallel Programing - by stullis - Feb-24-2019, 06:04 PM
RE: Python Parallel Programing - by wissam1974 - Feb-25-2019, 08:48 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  How to run existing python script parallel using multiprocessing lravikumarvsp 3 4,823 May-24-2018, 05:23 AM
Last Post: lravikumarvsp

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020